Anatomy of Distributed System à la Lokad

This is an update of my previous post on Building Blocks à la Lokad. It is based on improvements of the understanding and terminology thanks to Vaughn Vernon. It also shamelessly borrows from the subject of my continuous fascination: biology and nature itself.

Bounded Contexts

Let's start with the high-level overview of a Software-as-a-Service company (SaaS). This company could employ a range of systems needed to deliver its services to customers. Systems will be separate to reduce coupling and reduce the friction, as company evolves and grows (smaller the company is, more important this becomes).

Each system is a separate Bounded Context (BC term comes from DDD), which could have different implementation details: teams, technologies, language, lifecycle, deployment and maintenance specifics. Below are some examples of BCs in a SaaS company.

Please keep in mind, that each bounded context is a modeling concept and not an indication of how system is implemented. For example, Client Portal could be implemented as a stateless web farm with ASP.NET MVC3, redis-based state server and cloud-hosted processing logic, accessible by desktop and mobile clients. Or, it can be a single Windows Server with IIS, file-based state and a few console processes running in background.

This is similar to how a human body is composed of a set of biological systems, where each one serves specific purpose and can be built from specialized organs, cells and tissues.

Events: Forming Digital Nervous System

These separate bounded contexts need to communicate with each other. It happens by publishing events by each bounded context to an outside infrastructure. Each event is just a serializable message that complies with certain guidelines and is routed by messaging systems. Each event tells about something that has already happened: Invoice Created or Invoice Possibly Expired.

Events streams form digital nervous system of a company, where bounded contexts act as biological system.

Image above reflects event flows in an imaginary company. It might look complex in this specific representation, however, this doesn't always need to be perceived this way.

While publishing, sender does not even know about existence of its recipients, however recipients can subscribe to any publishers, they are interested in. This traditional PUB/SUB approach simplifies the picture a lot, since we can focus on specific bounded context with its dependencies. It also makes it similar to information flow through biological neuron.

Each bounded context can subscribe to events in two distinct ways: by declaring event receptors or with view projections.

1. Event Receptor

Event Receptor is a simple declaration of events that the specific bounded context is interested in and will react to by sending commands to application services of this context.

In code such receptors are usually grouped together in classes, according to their purpose:

public sealed class ReplicationReceptor
{
    // 'Domain' is the name of the primary Bounded Context
    // in this system
    readonly DomainSender _send;
    public ReplicationReceptor(DomainSender send)
    {
        _send = send;
    }

    public void When(SecurityPasswordAdded e)
    {
        _send.ToUser(new CreateUser(e.UserId, e.Id));
    }
    public void When(SecurityIdentityAdded e)
    {
        _send.ToUser(new CreateUser(e.UserId, e.Id));
    }
    // more receptor methods skipped

2. View Projection

View Projection subscribes to events, which are projected to a view (or persistent read model) that is structured in a way that will be easy to query by components within this specific bounded context.

In code, projection elements are grouped together in classes based on the view they keep updated.

public sealed class InvoiceDeliveryProjection
{
    readonly IDocumentWriter<unit, InvoiceDeliveryView> _docs;

    public InvoiceDeliveryProjection(IDocumentWriter<unit, InvoiceDeliveryView> docs)
    {
        _docs = docs;
    }

    public void When(CustomerInvoicePaymentRequested e)
    {
        var mark = new InvoiceDeliveryMark
            {
                AccountId = e.Id, 
                Created = e.RequestedOn,
                InvoiceId = e.InvoiceId
            };
        _docs.UpdateEnforcingNew(view => view.Invoices[e.InvoiceId] = mark);
    }

    public void When(CustomerInvoiceClosed e)
    {
        _docs.UpdateEnforcingNew(view => view.Invoices.Remove(e.InvoiceId));
    }

Both View Projections and Event Receptors are intentionally made extremely simple and easy to change. This is required, since they are coupled to external bounded contexts which can be controlled by different teams with different level of agility and technological capabilities.

Actual business behaviors and complex logic reside within Application Services and Tasks, which are safely isolated from the change shocks by projections and receptors.

3. Application Service

Application Services are interfaces which are remotely accessible for calls within their bounded context. Normally these calls happen by sending a command towards one of these application services (this command can be either explicit command message, or it can be carried by infrastructure in form of remote procedure call).

So each application service is just a set of command handling methods, which are executed when a specific message arrives. These command handlers are grouped together according to intent and design guidelines. Each one deals with one command and publishes events afterwards. It can call upon various service clients in the process as well.

Examples of service clients that are usually used by command handlers:

• SQL client;
• NoSQL database client (e.g. key-value store with views that are projected within this bounded context);
• Connection to an Event Store;
• Integration endpoint with 3rd party;
• Business Logic client;
• Command sender to an application service within the same bounded context.

Implementations of command handlers within application services can be different:

• Stateless processing of incoming commands into events (functional style).
• Handling commands by executing certain behaviors and calling domain services (e.g.: CRUD-style Aggregate Roots).
• Aggregate Roots with Event Sourcing (AR+ES) and Domain Services (favorite).

Here’s an example code for AR+ES implementation:

public class CustomerApplicationService
{
    // domain services initialized from constructor
    IEventStore _store;
    IPricingService _pricing;
    IInvoiceCalculator _invoicing;

    // handler for command SettleCustomerInvoice
    public void When(SettleCustomerInvoice cmd)
    {
        DispatchAndMergeConflicts(cmd.Id, 
            cust => cust.SettleInvoice(cmd.InvoiceId, _pricing, _invoicing))
    }

    // skipped other command handlers

    // helper method that dispatches call to an aggregate root loaded
    // from event stream. If there were any concurrent changes, we’ll 
    // check server changes for merge conflicts and try to rebase our changes
    void DispatchAndMergeConflicts(IIdentity id, Action<Customer> action)
    {
        while (true)
        {
            var stream = _store.LoadEventStream(id);
            var agg = new Customer(stream.Events);
            action(agg);

            try
            {
                _store.AppendToStream(id, stream.Version, agg.Changes);
                return;
            }
            catch (EventStoreConcurrencyException e)
            {
                // verify our changes for merge conflicts
                foreach (var clientEvent in agg.Changes)
                {
                    foreach (var serverEvent in e.StoreEvents)
                    {
                        if (ConflictsWith(clientEvent, serverEvent))
                            throw new ConcurrencyException(e);
                    }
                }
                // there are no conflicts and we can rebase
                _store.AppendToStream(id, e.StoreVersion, agg.Changes);
            }
        }
    }
}

4. Task

Task is the last element of a distributed system (a la Lokad). It essentially is a method call that is executed at specific moments in time. It can call upon various service clients (e.g.: query views or check up on 3rd party integration systems) and publish events. Task implementation is generally similar to application service, except for the trigger part. Here’s an example of task that continuously checks on list of invoices, detecting invoices that need additional action.

var remindIn = TimeSpan.FromDays(15);
while(!server.IsCancellationRequested)
{
    var pending = _docs
       .Get<InvoiceDeliveryView>().Invoices.Values
       .Where(x => (x.CreatedOn + remindIn) < DateTime.UtcNow);
    foreach (var x in pending)
    {
        _events.Publish(new InvoicePossiblyExpired(
            x.AccountId, x.InvoiceId, x.CreatedOn));
    }
    server.WaitHandle.WaitOne(TimeSpan.FromMinutes(15));
}

Example

In the snippet above, we actually handle piece of bigger invoice delivery and reminder process that would probably be implemented in bounded context, using all 4 elements: Event Receptor, View Projection, Application Service and Task.

For instance, you can track invoices as part of customer application service. This application service would use customer repository and currency converter service as its dependencies in order to handle commands like:

• Create Invoice
• Add Payment To Invoice
• Expire Invoice
• etc

Some of these commands will be sent to application service by receptors of bounded context in response to events:

• When Invoice Payment Arrived then tell customer application service to Add Payment To Invoice
• When Invoice Possibly Expired then tell customer application service to Expire Invoice

On picture this would look like:

We can also keep track of all open invoices by creating a projection for Outstanding Invoice View. This view will be used by Invoice Expiration Tracker task, which will once now and then rescan list to detect outstanding invoices that were created too much time ago. For each one, it will publish invoice possibly expired event.

Purpose

Purpose of all this separation is simple: to have a set of simple building blocks from which more complex systems can be composed in a reliable fashion that allows change without much friction and regressions. Rules of such composition should be rather straightforward and hopefully clear. I will be talking about them and implementation details in following materials.

This approach is what I have arrived (so far) while working on various projects at Lokad. This is also what shapes future development of our small but rather interesting Software-as-a-service company. Even if it keeps on growing in complexity, I see no big problem in following up with the supporting infrastructure. After all, this has already been done by nature billions of times.

As you have probably noticed, I keep on shamelessly borrowing concepts and approaches from her, because it is all well-documented and:

Human subtlety will never devise an invention more beautiful, more simple or more direct than does nature because in her inventions nothing is lacking, and nothing is superfluous.

Leonardo da Vinci

Update

This discussion continues in another blog post: Bird's-eye view of a Distributed System - Context Map, which tries to take into consideration real world environment around such system.

Published: March 31, 2012.

🤗 Check out my newsletter! It is about building products with ChatGPT and LLMs: latest news, technical insights and my journey. Check out it out