Borg Cache Synchronization Problem

The other day just another Borg milestone was about to be reached. It was mostly about refactoring and removing all necessary player, client, character caches stored inside many different Borg components either as dictionaries or lists.

The main problem was data synchronization. If a client disconnects the Network Server can detect that easily and clean up its internal client cache, unfortunately there are many other caches to be cleaned up as well: player cache of the client state manager, playing character cache owned by the world server etc.

After the system has grown and new components where added keeping all that data in sync become a major pain. It was very easy to forget notifying a component about a change and the state got inconsistent which was not quite obvious until an error showed up. Even worse was the fact that adding a cache to a new component would require passing the components reference or interface to all other parts so they can communicate back any changes.

To tackle that problem we decided to do what we should have done at the first place: a central cache accessible from all interested parties. I added a MasterPlayerTable object which was basically a collection of MasterPlayerRecords and a bunch of dictionaries allowing look up per accountId, characterId, IP address etc. Since the Borg host process creates a ServerContext instance passed to all components I just added the MPT (MasterPlayerTable) to the context and everyone was happy.

Well not quite, since now all components where reading/adding/deleting records stored in the MPT it become a thread contention bottleneck. The act of adding and removing records was not a problem - only one single component can actually add new records after a new player connects. Not so many can (or have to) remove records from the MPT. The problem was the nature of cache usage inside Borg. Most parts have to do an enumeration like:

foreach (MPR playerRecord in MPT)
{
       // do something
}

which obviously is not yet a problem. But the parts of Borg which are removing some records (disconnected players, log-outs, bans, kicks, invalid protocol etc) couldn't just remove a record from the MPT since the enumerator would become invalid and all other threads which happen to be inside a foreach would get the "Collection was modified after the enumerator was instantiated" exception.

The classic approach to such problems:

1. store all invalid items in a removal list,

2. decide when to do a cleanup, lock the whole collection and remove all items found in the removal list,

3. clear the removal list and unlock the collection

Well...this was a disaster I had the misfortune to actually implement and test myself. In order for this to work all parts/threads accessing the MPT must acquire exclusive access (lock). There are many situations you need the MPT especially now that the local cache copies where gone. The TurnManager just to mention one is the main consumer executing all the player actions per turn in a foreach manner. It must lock the whole MPT for the most of its execution time. The client state manager must at least once per turn do a cleanup and  acquire the MPT for itself. Note that a server turn executes many times per second. All the other helper threads where starved to death waiting on the lock!

The problem escalated to a magnitude that it was obvious what's happening just from the console output without any need to run the concurrency profiler.

I had to do something and  I had to do it soon...the story will continue in my next post