How To Make Classes Serializable

A large fraction of Zeek’s classes are serializable, i.e. instances can be converted into a (machine-independent) binary format and back. This is a short summary of how to make a class support serialization.

Note that it’s quite important to add serialization support to new classes derived from already-serializable ancestors; and also to adapt the serialization methods of existing classes when some of their internal structure changes (e.g., new data members are added). Otherwise, everything involving serialization will break quite horribly, specifically data persistence and remote communication.

The general layout of the serialization framework is as follows. There are a couple of base classes that are serializable (e.g, Val, Expr, Connection, …). They provide Serialize()/Unserialize() methods to turn them into a binary representation. These methods transparently handle all derived classes, i.e. to serialize a StringVal which is derived from base Val, you’d call Val::Serialize(). In other words only the base class’s serialization is directly accessible while serializing derived classes remains internal.

(Note that in this text the term base class does not correspond to a top-most ancestor in the C++ class hierarchy but to a logically separate branch of Zeek’s class hierarchy).

Adding Serialization Support to a Class

Let’s assume that class Foo is directly derived from class Bar which already includes serialization support (this case should be the most common one: most of Zeek’s classes already implement serialization). Bar itself is a (direct or indirect) descendant of a base class Base.

Follow these steps:

1. Add a constant SER_FOO to SerialTypes.h which represents Foo‘s type:

   const SerialType SER_FOO = n | SER_IS_BASE;
   

   n is a number which is unique among all children of Base, and SER_IS_BASE is a type-mask defining Foo as being derived from Base.

2. Insert DECLARE_SERIAL(Foo) into the protected section of the class definition.

3. If it doesn’t already exist, add a (preferably protected) default constructor to Foo (if you are able to make sure that it is not used by any other party than the serialization code, you do not need to initialize data members which the deserialization process sets (see below); but that’s not Good Style(tm) of course…)

4. If Foo is not abstract, add IMPLEMENT_SERIAL(Foo, SER_FOO) to Foo‘s implementation file.

5. Add two methods to Foo‘s implementation (they are implicitly declared by DECLARE_SERIAL):

   bool Foo:DoSerialize(SerialInfo* info) const
       {
       DO_SERIALIZE(SER_FOO, Bar);
       <...Foo specific serialization code...>
       return <boolean indicating success or failure>;
       }
   
   
   bool Foo:DoUnserialize(UnserialInfo* info) const
       {
       DO_UNSERIALIZE(Bar);
       <...Foo specific unserialization code...>
       return <boolean indicating success or failure>;
       }
   

   (Don’t rename info!).

   The two methods have to implement serialization/unserialization code for all data members of Foo that are to be stored/restored. When DoUnserialize finishes all members must be fully initialized.

   Within the Foo specific code, usually all of Foo‘s data members are serialized/unserialized subsequently. There are two types of data members: atomic values and objects which are instances of serializable classes themselves.

   For atomic values, a couple of macros are defined to handle their serialization/unserialization:

   • For a value x of type int, uint16, uint32, char, double, bool and string use SERIALIZE(x) and UNSERIALIZE(&x) respectively.
   • For a string s of type const char*, use SERIALIZE_STR(s, len) (in which len is the length of the string), and UNSERIALIZE_STR(&s, len_ptr) (in which len_ptr is a pointer to an int receiving the length of the reconstructed string; pass 0 as pointer if you’re not interested).
   • In the special-case of a null-terminated string s, you may either pass 0 as length to SERIALIZE_STR or use SERIALIZE(s) instead. If s may be a null-pointer, use SERIALIZE_OPTIONAL_STR(s) and UNSERIALIZE_OPTIONAL_STR(&s).
   • For single bits b inside a bit field, use SERIALIZE_BIT(b) and UNSERIALIZE_BIT(b).

   For objects which are themselves serializable, call their base class’s Serialize() and Unserialize() methods passing the info object as parameter. For object pointers which may be null, use SERIALIZE_OPTIONAL(ptr, Base::Serialize(info)) and UNSERIALIZE_OPTIONAL(ptr, Base::Unserialize(info)).

   The macros SERIALIZE, SERIALIZE_STR, SERIALIZE_BIT, UNSERIALIZE, UNSERIALIZE_STR, UNSERIALIZE_BIT return false on failure. Similarly, on failure the ::Serialize() and ::Unserialize() methods return false or null respectively. Don’t forget to check the return codes and, if something goes wrong, pass the error up.

   The OPTIONAL macros automatically return to the caller in case of error.