Software Design Blog

Journey of Rinat Abdullin

Lokad CQRS - Using Protocol Buffers Serialization for Azure Messages

Lokad CQRS, just like any other Application Engine, can use multiple serialization formats to persist and transfer messages. We’ve tried various options, starting from the XML serialization and up to BinaryFormatter and WCF Data Contracts with binary encoding.

They all had their own issues. Serialization format that had performed best in our production scenarios is called Protocol Buffers.

Protocol Buffers

Protocol Buffers are a way of encoding structured data in an efficient yet extensible format. Google uses Protocol Buffers for almost all of its internal RPC protocols and file formats.

ProtoBuf.NET is a great implementation of ProtoBuf serialization for .NET by Marc Gravell (you probably saw him on the Stack Overflow).

Lokad.CQRS uses ProtoBuf serialization internally for transferring and persisting system message information. ProtoBuf serialization is also the recommended approach for serializing messages in Lokad.CQRS:

  • format is extremely compact and fast, better than XML Serialization, Data Contracts or Binary Formatter (see below);
  • format is evolution-friendly from the start (renaming, refactoring or evolving messages gets much simpler);
  • format is cross-platform.

Here’s how the performance looks like, when compared to the other .NET options (details):

Proto Buf Performance

However, in Lokad.CQRS you should not worry about these specifics and potential problems that have been taken care of. You just define message contracts with ProtoBuf attributes:

[ProtoContract]
public sealed class PingPongCommand : IMessage
{
  [ProtoMember(1)] public int Ball { get; private set; }
  [ProtoMember(2)] public string Game { get; private set; }

  public PingPongCommand(int ball, string game)
  {
    Ball = ball;
    Game = game;
  }

  PingPongCommand() { }
}

and switch to this serialization in the domain module:

.Domain(d =>
{
  // let's use Protocol Buffers!
  d.UseProtocolBuffers();
  d.InCurrentAssembly();
  d.WithDefaultInterfaces();
})

More information is available in ProtoBuf in Lokad.CQRS documentation. Additionally, Sample-04 (in the latest Lokad.CQRS code), shows implementation of Ping-Pong scenario with ProtoBuf.

Starting from this sample, we’ll be using Protocol Buffers as the default serialization in our samples.

BTW, side effect of using fast and compact ProtoBuf serialization is that it increases overall performance. Smaller messages are less likely to exceed 6144 byte limit of Azure Queues. App Engine handles such messages by saving them in Azure Blob Storage. This essentially allows to persist messages as large as a few GB. Yet, second round-trip to Blob is something that we would want to avoid, if possible. ProtoBuf serialization in Lokad.CQRS helps to significantly improve our chances here.

By the way, if you still think that ProtoBuf makes it hard to view and debug failing messages, check out the next article!