Roadmap: Lokad Cloud and CQRS for Windows Azure

This is not a formal article on efficient development, but rather a short note to give you some perspective on how Lokad Tier 1 Open Source projects (Shared Libraries, Cloud and CQRS) might be evolving in the near future, as brainstormed with Christoph.

Lokad.Cloud and Lokad.CQRS are going to get much better interoperability and integration. This is needed to accomplish new exciting tasks that lie ahead of Lokad, as there is a need to push our services to higher levels of scalability and flexibility with Windows Azure Platform.

This interoperability is going to be achieved by smarter and more thoughtful approaches to the development and collaboration, as opposed to investing more development resources. In fact, one of the goals is to reduce the development and maintenance efforts on these projects (which is essential for the small team of Lokad), while making them more reliable and robust.

Here are a few primary directions.

Lokad Message Format

We are planning to improve message-level interoperability between Lokad.Cloud, Lokad.CQRS and potential helper projects in the ecosystem. This is achieved by settling down on the common Lokad Message Format, which is a rather simple, cross-platform and efficient binary encoding of the messages. It is implemented on top of Google Protocol Buffers (we are currently using protobuf-net by Marc Gravel) and will allow to:

• Seamlessly exchange messages between Lokad.Cloud and Lokad.CQRS roles, allowing to have better integration between reliable enterprise workflows and auto-scalable distributed computations.
• Reuse message inspection tools from the Lokad.CQRS infrastructure (these tools are not in the Open Source domain, yet) for working, debugging and maintaining Lokad-based solutions for the Windows Azure.
• Provide a standard way for integrating with message brokers outside of Windows Azure Cloud Computing platform and even outside .NET ecosystem (even Apache QPID might benefit from automatic scalability sometimes).
• Provide a consistent way of handling evolution of message schemas in the ongoing enterprise projects (especially in the rapidly changing environments).

This format is merely a transport-level specification for passing messages around Lokad-enabled projects in a uniform way that enables infrastructure to perform some tasks efficiently. Actual content serialization (Data Contracts, Binary, XML etc) is still configurable by the developers.

By agreeing on the common format we establish a common way of handling situations like:

• size limitations of Azure Queue Messages;
• potential duplication of Azure Queue Messages and handling it;
• uniform message operations based on the message transport headers;
• content-based load balancing, affinity and routing;
• AMQP protocol semantics.
• batch message operations and simple event storage systems (message streams) in the world of changing event schemas.

Lokad.CQRS trunk already contains first bits of code that implement new message format. We'll blog more on that, as the specifications get stabilized and field-tested by first production scenarios.

Reusable Modules

Introducing and using similar configuration syntax across the projects. This would allow to reuse modules in Lokad-powered projects in the uniform manner.

Thus, Lokad.CQRS App Engine handlers will eventually get native access to flexible storage access routines from Lokad.Cloud; while Lokad.Cloud distributed services will be able to use things like ProtoBuf serializers or automatic NHibernate session and transaction management. More integration scenarios are in the requirements.

With the help of some syntax sugar it'll be trivial to enable new .NET configuration options by just adding reference to the appropriate assemblies.

Lokad.Shared.dll already features ISyntax and ISyntax interfaces that are needed for defining such extension modules in a way friendly to the ideas of component-driven development.

Existing modules are planned to be migrated within the next few months, while the new extensions will be written with the new syntax in mind, from the start.

Compatible Infrastructure

Actual extensibility implementation leverages the best IoC Container for .NET - Autofac, created by Nicholas Blumhardt. In fact, both Lokad.Cloud and Lokad.CQRS have infrastructure based on this container (as well as multiple other projects within Lokad).

In order to keep modules reusable between the projects in a sensible way, rules of the component composition within these projects have to be aligned. This normally does not require any special development effort, just keeping the additional constraints in mind, while designing and evolving components.

The approach also includes designing worker hosts for Lokad.CQRS and Lokad.Cloud in such a way, that their roles would be compatible between each over. For example, we might be interested in running Pub/Sub Manager in the Lokad.Cloud or adding self-tuning scalability manager for the message handler in Lokad.CQRS.

Lokad.Shared Libraries already have got the common interfaces required for such interaction. Lokad.CQRS roles implement them. It's a matter of time before the infrastructure of Lokad.Cloud will follow.

Additionally, such approach makes it easier for the developers to create their own roles, that would be reusable between Lokad.Cloud and Lokad.CQRS application engines.

Separation of Concerns

Although Lokad.Cloud and Lokad.CQRS both target Windows Azure platform, they still focus on completely different application scenarios. So it is necessary to keep them this way, while avoiding duplicate development, where it is possible.

Here's a quick outline on the separation of concerns between these two projects:

Summary

This was a quick overview of expected future evolution of Lokad Tier 1 projects, as seen from the high-level perspective and mid-term planning horizon.

Stay tuned to the updates!

Comments

[Sean Kearon]

Exciting stuff indeed! I have a question...you said this this will:

Provide a consistent way of handling evolution of message schemas in the ongoing enterprise projects (especially in the rapidly changing environments).

Can you give some more details about this yet?

[Rinat Abdullin]

Hi Sean,

Basically that's just Lokad Message Format (infrastructure-level interoperability and some conventions for smarter handling and reuse of tools and code) + evolution-friendly efficient .NET serializer for the message bodies (currently ProtoBuf) and some shared experience on defining these messages, upgrading their schema and message handlers in a way that does not incur a large development tax and downtime (zero downtime for the entire project life in the ideal scenarios).

BTW, I think that this thread at Azure forums might also interest you within the context of this discussion.

[Sean Kearon]

Thanks Rinat, that makes sense. In my scenario in my day work we have an ESB whose surface accepts messages through web services. Each message is XML and its version is defined by its schema. Not rocket science !

However, this raises the problem of how you manage changes to each message's structure? For example, the purchase order message version 1 has been in use for two years and there are 15 clients sending purchase orders through this integration point. Requirements have now changed and we need to modify the message structure accordingly, giving us purchase order message version 2. So how do we maintain the functionality provided by the ESB? We can...

[1] Migrate all client all at once?
[2] Add a new integration point for v2 messages - SendPurchaseOrder2?

[1] is not possible in a corporate environment and [2] will very quickly lead to an unmanageable mess.

Our approach is to design each integration point to handle multiple versions of the same message. So, it looks like this:

string SendPurchaseOrder(string);

The parameter and return are XML documents as strings. Behind the method we plug in handlers for v1 and v2. Hence we can easily support multiple versions of the same message until all clients are upgraded (if they all ever are!!!). The cost is some added configuration and deployment at each integration point.

I am wondering if this is a scenario that you have faced and what your solution would be?

Regards

Sean

[Rinat Abdullin]

Sean,

Our API (public services) cases at Lokad are a bit simpler for the time being, since the actual services are kept extremely simple themselves. So versioning does not pose a big problem.

Explicit message versioning, that I've been referring to with Lokad Message Format, is needed in our case for the event log (Time Machine) and persisting some partially schema-less data, that's going to stay around for really long time (entire life of the application). Thus we really need make sure, that changing message schema does not ever break any existing data (if change is breaking, then that should be a new contract) and ideally contracts will be fully forward and backward compatible. Existing .NET serializers of BCL just don't make this thing easy.

In a sense this problem somewhat intersects with your per-method versioning problems of exposed services and it actually resembles some article of Microsoft, where they had some complex and versioned SOA (it went really messy).

I haven't yet implemented public API that needs to stay available around, while evolving at granular level and still staying backwards compatible with existing clients. Yet, in the REST/CQRS world this could be implemented as a simple approach, where REST URL matches with the contract name of the command (name and simplified namespace, which could include version info).

POST http://api.company.com/accounts/create_v1

{
  AccountId : "GUID";
  Email : "some@domain.com"
  Name : "John Doe"
}

As long as changes in message schema involve adding new optional fields, we don't need to change the contract. If new command is incompatible, we define it under the new name(namespace): CreateAccount_v2.

Sending a command is just posting content (in whatever serialization formats are supported: json, xml, protobuf, data contracts, etc) to the appropriate URL. Wiring from the REST API to the command handlers could be probably handled by the infrastructure itself (name of the contract command and definition of command schema are logically enough for exposing the proper API). Thus, the friction of managing such API is mostly about managing .NET command classes and their handlers, which should make it easier to manage multiple versions of method at once.

NB: although I would try to keep things separate and CQS compatible (either we post commands or query views), real-world app might require getting actual response from the remote method (RPC). Return values are managed and versioned in the same way as the incoming message bodies.

What do you think?

[Sean Kearon]

Hi Rinat

I think that there is a lot of mileage in controlling how the message structure changes over time. Your approach to this is interesting and has validated the approach I am taking in my new product.

So, if I have read you right, you are saying that by managing changes to the message structure you can handle messages v1 and v2 through the same integration point. (By "integration point" I mean a REST URL, a web method, a file drop location, etc.) Further, Where the message structure changes mean that this is not possible you add another version of the integration point. Over time this may leads to something like:

CreateAccount_v1 integration point handles messages NewAccountMessage_v1, NewAccountMessage_v2, NewAccountMessage_v3.
CreateAccount_v2 integration point handles messages NewAccountMessage_v4, NewAccountMessage_v5.

So, managing the message change reduces the impact of change on the API surface - that is, we have 2 create account integration points, not 5 in the above. Our API is therefore easier to manage!

I am using a very similar approach to this in the data import/export features of the above product as I need to maintain compatibility over time. I work in a large government organisation that has nearly 900 servers and a large variety of business systems on those servers. The reason that this problem is interesting to me is that we have to be able to guarantee operation of these systems over time. Also, we do not have the ability to change the messages that clients are sending (sure, we can control that at implementation, but thereafter we have to preserve service until the supplier can deliver an update). I think your approach would work well in this scenario also as it helps to control increases in the complexity of the API. In either case, each integration point has to be able to deal with more than one message version, so the work behind the integration point remains the same.

Thanks for your feedback!

Regards

Sean

[Rinat Abdullin]

Hi Sean,

Just a few follow-up thoughts.

I did check up on the link of your product. Wow, the domain looks really complex. Are these screenshots displaying a stand-alone application or the UI to the web services that you expose? Is the UI using DXperience?

Regarding versioning and managing message structures. Basically the idea here is to architect and develop messages in such a way that (a) most probable changes to come do not introduce breaking changes (b) non-breaking changes are backward and forward compatible. For example, if we have messages (from your sample) that evolved from v1 to v3 in nonbreaking way, then this would mean, that v3 schema could be used to deserialize messages from v1 (missing fields would be defaults). v1 schema could be used to deserialize v3 objects as well (extra unknown fields will be stored as extension objects). ProtoBuf and WCF versioning logic capture the topic in greater detail.

Also, while versioning, we cold essentially version message schema implementation along with the message handling code, in the same assembly. If we go Udi style (up to the extreme point of having handler per dll), updating versions or adding new message handlers to the surface could be as easy as dropping these new dlls into the bin folders. I've never tried it this way, yet.

Well, since managing and manually distributing these numerous changes across the farm(s) of heterogeneous servers, located across the globe, might be too much of a burden, we could simply keep binaries in some storage (i.e.: cloud blob) and push the update by sending out XMPP message to the servers: "Update is ready, please update this dll".

[Sean Kearon]

Hi Rinat

Making the messages backward and forward compatible is an excellent idea. I will think about adopting this myself. Having any version being able to deserialise from any message version is rather compelling. We do register the actual message handlers into a generic web service that basically just unpacks the message, checks its namespace and sends it off to the dll that is registered to handle it. I have not considered updating farms of servers as our environment is all hosted internally (currently!).

As for my product, this is not my day job and it does not use services, it's a desktop application. Well spotted that it used DXperience - a fantastic set of UI controls indeed. You are right, the domain is all about some very complicated business rules - and they change regularly too!

Cheers

Sean