Silkroads: The Serealizable Dictionary

The Dictionary is a type of data structure included in the .NET framework that works like a list, except that you can search through its elements using a ‘key’ rather than just a number (although you can use that too, with System.Linq’s ElementAt() function). And keys can be anything; objects of one of your own classes, for example, or enum types. As such, I can write:

// Declare the dictionary and add some elements.
Dictionary<Fruit, float> fruitSupplies = new Dictionary<Fruit, float>();
fruitSupplies.Add(apple, 5.0f);
fruitSupplies.Add(banana, 2.0f);
 
// Find an element by key.
Debug.Log("I have a banana supply of: " + fruitSupplies[banana]);

When I discovered this data structure at the start of my project, its application to a data-heavy grand strategy game seemed obvious. Say I have 200 provinces in the game, and each had an opinion (out of 100) of the other. It’d be compact and elegant if I were able to request, anywhere in my code:

// The lower rome's opinion of a state,
// the less efficient trade between the states.
rome.tradeEfficiency[carthage] = -rome.opinions[carthage] + 100;

Supply and Demand Values

Such cases where searching by my own key, be it a resource, a unit type, a map region, will be abundant in this type of game. So, it became clear to me from the outset that I’d be making heavy use of dictionaries.

I assumed this was a blessing on my progress. Here’s a neatly packaged data structure that does exactly what I need, ready for me to use. Maybe this won’t be so hard after all, I chortled. What a fool I was. The crucial caveat of the all-powerful Dictionary, I soon discovered, is that it cannot be serialised. In my previous post, I explained that serializing data means having it set and available before runtime, that ‘to serialize’ is essentially ‘to save’, and that I can either use Unity’s own API, JSON, XML, or any combination of the three, to successfully save and load – serialize and deserialize – my game data. But because Dictionaries are generic classes that can work with any object or data type – at least, I think that’s the reason – they cannot be serialized.

To introduce the specific problem I started off with, and how it was relevant to this dilemma: in my Silk Roads game, I will have a variety of trade goods, represented by objects of the Good class. Silk, Glass, Silver, Gold, Ivory, Lapiz Lazuli, Cinnamon, and Nutmeg, are the goods I’m using right now. No doubt I’ll add more. Every trading node in the game has a Market, and each Market needs to track the local supply/demand of each resource, currently represented as a number between 0 and 1. So, while the port of Harmozia is rich in Cinnamon from Southeast Asia, the markets of Antioch are more abundant with Roman Glass. Exploiting these local supplies and demands by buying where things are cheap and selling where they’re expensive is the game’s core mechanic, so in short, each market needs a number (a supply rate) for every trade good. Those numbers need to be specified before the game’s runtime, i.e. they need to be serialised.

All was not lost. While Dictionaries technically cannot be serialized, they can be reduced to two Lists on serialization – one for the keys and one for the values – and reconstructed into a Dictionary on deserialization. In fact, this is such a common issue that Unity provide their own mouthfull of a solution – the ISerializationCallbackReciever interface – for which the example of use is creating a serializable Dictionary.

*** I’m going to link to the documentation here because I’m sure it works for some people. But be aware, if you’re reading this in search of help, I’m about to explain why it did not work for me. ***

Interfaces

As mentioned, the ISerializationCallbackReciever is an interface (you can tell because it starts with ‘I’). Now if, like me a few weeks ago, you don’t know what an interface is; it’s basically a packet of functionality that you can tack on (or ‘implement’) to your classes. Say I have a Bomb class, and I want to give it timer functionality, because a bomb very much does contain a timer. I can create an ITimer interface, and implement it on the Bomb class. I can also add it elsewhere: clocks, microwaves, pulsars, and I can implement multiple interfaces on a single class, unlike inheritance. The syntax works like this:

// The interface.
public interface ITimer
{
    int time
    {
        get;
        set;
    }
}
 
// Interface is applied here with a colon.
public class Bomb : ITimer
{
    // Value to 'implement' or represent the interface's property.
    int iTime = 0;
 
    public Bomb(int _time)
    {
        iTime = _time;
    }
 
    // Getter and setter implementation of the property.
    public int time
    {
        get { return iTime; }
        set { iTime = value; }
    }
}

I’m still a little unclear on them, and don’t know why they have to have properties instead of variables, but they work.

Unity’s ISerializationCallbackReciever (I’m just going to call it ISerCallback for now) gives you two ‘callbacks’ (functions that are called automatically from elsewhere, that act like events for your class to respond to, like a Start() or Update() function). These are OnBeforeSerialize(), and OnAfterDeserialize(). As can be imagined, the idea here is to tell your Dictionary to become two lists when OnBeforeSerialize fires, and then to become a Dictionary again when OnAfterDeserialize fires. The basic code for Unity’s example looks like this:

// SDictionary inherits from a regular Dictionary.
// SDictionary takes a 'Type Key' TK and a 'Type Value' TV.
public class SDictionary<TK, TV> : Dictionary<TK, TV>, ISerializationCallbackReceiver
{
    public List<TK> keys = new List<TK>();
    public List<TV> values = new List<TV>();
 
    public void OnAfterDeserialize()
    {
        // For each key or value, add a key-value-pair to the dictionary.
    }
 
    public void OnBeforeSerialize()
    {
        // For each dictionary element, add a key and a value to the lists.
    }
}

At first, this seemed like the perfect solution. In fact, by making my keys and values serialize with [SerializeField], doing the same with the SDictitionary class’s instances, and making the class itself [Serializable], I was able to have my supply values appear in the inspector, ready for me to edit. With some fiddling around, I even customised the editor’s appearance to use sliders, and made gizmos to display the supply values as a bar chart.

This type of easy editing is why I was so excited to spend time on this class: it allows me to easily edit the dozens of trade nodes in the game, and makes it easy to read how the supplies for different trade goods are spread out geographically.

Problems With Unity’s Serialization

If that solution was as stable as I’d assumed, I’d have ended the post here, and finished my game’s supply distribution two weeks ago. But I think my commit history following that first SDictionary attempt tells its own story:

It’s difficult to pinpoint exactly what the central issue is with this type of serialized Dictionary, because following this first triumph, it was one gruelling problem after another. Values I had set were on several occasions overwritten. References to the different types of goods were constantly lost, broken, or duplicated. From what I can gather after two weeks of firefighting, the ISerCallback interface is not very robust – especially when using in-editor scripts (which is precisely what I’m doing with all this serialization) – for two reasons.

1. Unity serializes and deserializes very often. In all fairness, the documentation for ISerCallback does mention this. However, I wound up with serialization every frame, while deserialization didn’t seem to fire when it should at all. It was such a complicated mess, falling completely beyond my grasp, that all I can really say is this; you cannot know, at any time, if your SDictionary will be in a serialized state (where you should talk to its Key/Value lists) or its deserialized state (where you should talk to its Dictionary). The amount of NullReferenceExceptions is substantial and nerve-wracking. I could honestly never solve this issue.

2. Serializing/deserializing breaks the reference value of your key. Values vs. references is a common trope in programming. Suffice it to say, somewhere in my dictionary’s serializing/deserializing, the trade good keys which were supposed to represent references to the central list of goods was broken, such that the key for “Silk” in Jerusalem no longer equalled the Silk in the static Goods class, so I couldn’t ask for Jerusalem’s Silk value anymore. See the diagram.

There may have been even more issues; after two weeks, I gave up. This technique is extraordinarily unstable, in my experience, and I’d advise against it if you’re after a serialized Dictionary that can be interacted with in the editor. I found another way, in some ways less elegant, but certainly more stable.

The Solution

I took a step in the right direction after finally asking myself “why do I even need to make a dictionary before runtime?” The alternative Dictionary seems simple at first: store the dictionary as lists in edit time, and do not consult Unity’s ISerCallback events for when to build the Lists into a dictionary. Only build the Lists into a Dictionary when I want – that is, on Start() or Awake(). There were, however, a few issues I had to solve.

The first is how to have an event fire on the game’s start. “Easy” I hear you cry, “use void Start().” void Start(), I retort, is only available to Monobehaviour classes. My SDictionary is already inheriting from a Dictionary, so it can’t inherit from a Mononehaviour too (this aside, that’d be unwieldy for a simple data structure). It is possible, however, to define your own callbacks – like Start() – using delegates. Normally, that’d be incredibly simple: create a singleton Monobehaviour called “EventManager” or something, give it a delegate, and have all of the SDictionaries ‘subscribe’ to the delegate. But since the SDictionaries will be attempting to subscribe to the delegate event on compile time, and a singleton in Unity seems to be a runtime type of pattern, this is a no-go. Instead, the final structure took three objects, and looked like this:

A static class stores the delegate subscribers (the CreateDictionary() functions). A Momobehaviour game script calls the static object’s events. It’s a little messy, but it shouldn’t cause any havoc down the line.

The second issue I had to solve was much more difficult. As mentioned earlier, deserializing a dictionary renders the key references broken, so that Silk in Jerusalem no longer refers to Silk in the rest of the game. I had to think of a way to turn goods, which are runtime objects, into keys, and back again, without losing any references. After much toil, I believe I found a sound solution, with the use of interfaces.

Essentially, by assigning every object that can be used as an SDictionary key with a special name, a “label”, that string can be used as the serialized key. Then, when the Dictionary is being built, it can find out what that string refers to by searching a provided ‘stock list’ of master-objects. Thus when Market.supplies enters runtime and builds its dictionary, it takes its list of strings – “Silk”, “Glass”, “Silver” … – consults a stock list that was given to it on creation – the static Goods.goods List – and assigns the appropriate keys. Since this is so mind-boggling, I’ll post the entire source code.

using UnityEngine;
using System;
using System.Collections;
using System.Collections.Generic;
using Trade;
 
namespace DataUtility
{
    // Interface for labels.
    // Labels are implemented on objects that can be used by the SDictionary as keys.
    // This is because the SDictionary turns object keys (references) into string labels (values) on serialization,
    // and turns them back on deserialization. Objects that need to be used as keys in an SDictionary should
    // implement ILabel, and typically just set the label to their name (e.g. the silk object is labelled "Silk").
    public interface ILabel
    {
        string label
        {
            get;
            set;
        }
    }
 
    // The custom dictionary class, that can be serialized as two lists, and builds a dictionary on runtime.
    // Note that it requires the keys to use the ILabel interface.
    [Serializable]
    public class SDictionary<TK, TV> : Dictionary<TK, TV> where TK : ILabel
    {
        // Serialized keys (uses key labels as strings).
        [SerializeField]
        public List<string> keys = new List<string>();
        // Serialized values.
        [SerializeField]
        public List<TV> values = new List<TV>();
        // The key stock is the reference used to convert strings back to the proper types on creating the dictionary.
        protected List<TK> keyStock = new List<TK>();
 
        // Constructor, which requires a specified keyStock.
        public SDictionary(List<TK> _keyStock)
        {
            EventManager.objectStart += CreateDictionary;
 
            keyStock = _keyStock;
        }
 
        // Build the dictionary from the lists. Should be called on Start().
        public void CreateDictionary()
        {
            Clear();
 
            for (int k = 0; k < keys.Count; k++)
            {
                Add(LabelToKey(keys[k]), values[k]);
            }
        }
         
        // Using the keyStock, find the key object which the stored string is supposed to represent.
        private TK LabelToKey(string key)
        {
            TK stockKey = keyStock.Find(k => k.label == key);
 
            if (stockKey == null)
            {
                throw new Exception("The label: " + key + " could not be found in the stock list: " + keyStock);
            }
            else
            {
                return stockKey;
            }
        }
    }
}

The plan following this design is to implement ILabel to any object that will be used as a key. These objects will be limited, but might include Countries, Languages, Technologies, Units, Currencies, and of course Goods. Then I’ll set up a central ‘stock’ list for these SDictionaries to refer to. The result is safe, robust data storage and retrieval.

Why does this require an interface? You’ll notice that generic classes, such as SDictionary, don’t handle a specific data types. This means, however, that you can’t access the data of your generic types within the class (because an int doesn’t have a Transform, while a Monobehaviour doesn’t have a .ToString()). You can use restraints, with the ‘where’ keyword, to tell your class that it can be sure of some type members being available. As such: “where TK : ILabel” is read as “where Type Key has a label”. There is not, to my knowledge, another way of doing this.

The Fruits of My Labour

Following this long-winded firefight, during which I introduced myself to abstract classes and interfaces, I received my reward: a dictionary that could be serialized. This made it very easy, ultimately, to set up my local supply values across the Middle East, in a matter of minutes:

The SDictionary has already proven useful elsewhere. I’m also using it as the ‘hauls’ variable for my Caravan units, describing how many of what Good is being carried in a Caravan, in kilograms. This structure, and similair structures, will prove invaluable as I build a complex system of interrelated values, allowing me to easily search by key through any “x for each y” collections. Hopefully, the time I’ve spent will be easily redeemed.