Devblog 184 - Post Mortem

Build server issues, monument corruption and world mismatches.

3 November 2017
What Happened?

You may have noticed that some servers were having pretty major issues with yesterday's update. The backstory is that the last build we did before the update got corrupted on the build server, which broke the powerplant monument. We noticed this and started a new build for a hotfix, but servers that had already generated a map with the broken powerplant monument had a messed up world file. We've thought about our options and since only the minority of servers are affected we will not be forcing a wipe. We will however force the hotfix to be installed on all servers and server owners who are affected by the corrupted maps can decide if they want to go through with a wipe.

Official Servers

The official servers were updated last night right after the hotfix was released and eight of our official servers were showing corrupt worlds from the broken build. We're in the process of removing any floating or underground objects this has caused, but we will not be wiping the affected servers since the problem is limited to only a couple of areas.

Lessons Learned

We've had similar fuckups in the past, but they've been getting less and less frequent over the years as we hired more QA staff and expanded our testing procedure before updates. It would be easy for us to blame this on the build server issue, but if we're being honest this absolutely came down to us adding too many things in the last minute which reduces the time we can spend on testing things before they go live. We'll do our best to prevent this from happening in the future.

Linux Server Crash

The recent EAC changes have introduced a Linux server crash. This will be fixed in a hotfix just for the Linux server owners later today.

Mailing List

If you want to follow this project you can sign up to the mailing list.

We'll only update you about this project, we won't spam you about other stuff or sell your email address.