The server had a memory leak. A pretty bad one. It lead to me writing a script that wrote stats to a CVS file every minute. Then I imported the file into Excel and made pretty graphs.
As you can see, the MemUnity goes apeshit. This is memory used by Unity on the c++ side (I'm lead to believe). Memory consumed by Mono is MemMono, this is managed memory that is garbage collected. The big blue line going through the middle is the amount of sleepers, that is, the amount of players that have joined the server and then left. The Yellow is the amount of players on the server.
So because MemUnity is going crazy we know it's not something to do with garbage collection. To cut a long story short each player has an Audio Source to play their mic input. They're not actually used on the server, or uninitialized in any way. But the fact that they exist on the gameobject cause the memory to go crazy. So I write a script to cull them on the dedicated server.
That'll do for now.