Software Design Blog

Journey of Rinat Abdullin

How Do You Decompose Complicated System Into Components?

The answer to the problem of finding a proper way to decompose existing complicated system into smaller components is two-fold:

  • Use known methodologies for finding potential boundaries;
  • Iterate a lot to see if system system could be divided across these boundaries.

Let’s start with examples of methods for finding potential boundaries:

  • Look at the domain model from the perspective of Domain-Driven Design, seeking out bounded contexts and aggregate boundaries there (although classical DDDish BCs and aggregates could often be split in smaller focused parts as well).
  • Separate domain-side problems (need domain expert) from implementation problems like scalability (in some projects performance might be part of the domain, too);
  • Use domain events to capture state changes in some parts of your system; then identify related events and group them - they could be a core of a new component; non-related events could probably be put into separate components;
  • Look from the perspective of separating project between multiple teams and individual developers;
  • Look at transactional boundaries in your system. If there are that changes must absolutely happen together (or fail together) then they might be candidates for being put into a single component. If some changes don’t really affect the others, then you could probably consider keeping them separately.
  • Search for processes which must be coupled tightly from the temporal perspective - they might belong to one component. At the same time, presence of queues might indicate potential component boundary.

And, above all, simply keep on asking yourself one simple question - what things can be taken apart? How can we separate our software into atomic things that can be composed to solve the problem.

Decomposition is not simply about breaking some project into a bunch of tightly coupled modules. The idea is to identify the boundaries and invest effort into evolving them.

Methods for finding potential boundaries (like the ones listed above) only provide you with some hints and ideas. Your objective (as a designer) would be to play with these ideas, iterate and evolve system, while finding options that truly make it more robust, evolution-friendly and simple (put your priorities here).

How Micro-Services Approach Worked Out in Production

Last week we released a new version of business backend at Lokad.

Previous version was based on message-driven design with heavy use of CQRS principle, event sourcing and queues. There is a case study on that at Microsoft CQRS Journey. As I’ve discovered soon after, this architecture was a big pain in the ass due to being too monolithic and dogmatic. There was a lot of unnecessary ceremony involved in the simplest actions. Back then it didn’t feel as such, since literally everything in .NET world involves a lot of ceremony. Yet, after gaining initial exposure to the world outside of .NET, my perspective has changed. The concept of decomposing a complicated system into a bunch of tiny and simple components that work together and yet evolve separately - was refreshing and liberating.

Current version of business backend at Lokad grew upon these ideas, while adapting them to the .NET reality (making things slightly more monolithic, where it was absolutely necessary). Still the result is quite nice:

  • 13 backend components which have on average 4-5 classes and the same number of public methods;
  • 2 thin UI components (simple ASP.NET MVC web sites);
  • public API : JSON over HTTP; ability to subscribe to events from components;
  • no overreaching architecture in each component, although event sourcing with in-memory views shows up quite often;
  • all components are designed in a way which allows to have multiple instances running behind load balancer.
  • almost no unit tests, since the components are extremely simple.

If I were to draw a map of all backend components, it will look nothing like my previous monolithic designs:

Temp

arrows point in direction from upstream component (provides services/data) to downstream component (uses services/data).

Many of these components were rewritten multiple times from scratch, as the system was gradually evolving from legacy monolithic design towards deeper insight. This often happened in situations when we felt that there was a better way to decompose business capabilities into components. Sometimes “better” meant “simpler”.

The fact that previous version of the system was running event sourcing allowed to migrate functionality to new design in small steps. For example, at some points in time events were flowing from the old version to the new one. At other points in time, legacy code in the old system was calling directly newly established components that were already running within the boundaries of the new system.

Basically, effort to decompose existing business capabilities into small chunks started paint off immediately : it became easier to think and reason about evolution of the design. It also became possible to break down work in really small steps (which minimised risks), while still maintaining reasonable development throughput (because multiple components were developed in parallel).

I quite like the final result - so far the system is extremely simple and there were surprisingly few problems with the migration (due to the fact that the system is quite simple).

Performance is not an issue. Current configuration could easily handle the load even if number of Lokad users increases a few hundred times (if it goes beyond that - we’ll need to ask Azure to deploy one more server instance). This happened because performance of each component is measured along with its usage. In cases when components were frequently used and reasonably slow (as proven by stats from the production), they were tuned to better performance.

Ultimate performance tweak for reads was about serving reads from an in-memory cache which is kept up-to-date via persistent TCP subscription to event stream with changes (just an event projection that subscribes to event store and stores projected results in memory). Ultimate performance tweak for writes was about putting command messages to a queue and processing them in background (work is shared across all nodes).

If you remember my previous posts about Lokad.CQRS building blocks, you would recognise some of the patterns. The biggest change from “Lokad.CQRS architecture” is that there is no longer a uniform architecture in this new design. There are a few design constraints (e.g.: home components communicate or max size of the components), yet the internal implementation details are private to each component.

Such an approach leads to situation, where each component can be tailored specifically for the job at hand. It does not need to follow a unified way to access data or handle failures. This might lead to a greater diversity. Yet, the lack of unnecessary ceremony allows to get right to the core of the problem in the most simple and efficient way possible. Hence components can be diverse and simple at the same time, just like cells in human body.

Probably, the system could be made even more simple, if it were taken outside of common .NET stack.

What infrastructure do we use?

There is no real infrastructure, we mostly use some external libraries for our needs:

  • ServiceStack for hosting app services (web services made easy)
  • Metrics.NET for performance measurement
  • Event Store of Greg Young to store events and to push them to all subscribers
  • Windows Azure to host backend workers and frontend
  • ASP.NET MVC 4 for Web UI

Studying Go Language (Golang)

Over the course of the last few days I’ve been studying Go language (or golang), as a part of my job at happypancake.com. These days helped me to realise that requirement to learn new things is one of the best job benefits in my mind.

Here are a few things that I’ve discovered about go:

  1. Golang programs can be compiled to run on FreeBsd, Linux, Max OSX and Windows. Once compiled, you just distributed a single binary and run it.
  2. Golang is a compiled static typed language with garbage collection. It’s performance is within a reasonable speed margin of C.
  3. Concurrency primitives are built into the language. They are somewhat similar to erlang (but without erlang scheduling and supervisors, though).
  4. There is a nice type inferrence (better, than C#).
  5. There is a uniform formatting guideline for the language: go fmt enforces it.
  6. You can refactor go programs from command line.
  7. There is a nice ecosystem for building various backend and web tier servers.
  8. You can use Sublime as IDE (vim works as well):

Screenshot 2013 12 16 16 45 05

Below is an example of a simple web server that we hacked together with Tom Janssens together today (neither of us had any prior knowledge of go). This server maintains a list of accepted jobs, which are served for GET requests. You can POST new jobs to the server, which will process them asynchronously (simply adding to the list):

package main

import (
    "fmt"
    "net/http"
)

var queue chan string
var joblist []string

func init() {
    queue = make(chan string, 10)
    joblist = make([]string, 0)
}

func handler(w http.ResponseWriter, r *http.Request) {
    switch r.Method {
    case "POST":
        val := r.PostFormValue("job")
        fmt.Fprintln(w, "VALUE: ", val)
        queue <- val
    case "GET":
        fmt.Fprintln(w, joblist)
    default:
        fmt.Fprintf(w, "Not supported")
    }
}
func projection() {
    for req := range queue {
        joblist = append(joblist, req)
    }
}
func main() {
    go projection()

    http.HandleFunc("/", handler)
    http.ListenAndServe(":8080", nil)
}

Here are a few more interesting links:

From .NET to Erlang : Starting Work With Large Free Dating Website

10 years spent in .NET were fun, but it is time to try something radically different. As you probably already know, I’m moving to a new project at HappyPancake - large free online dating website (largest one in the North Europe, if I’m not mistaken). I’ll be working with outstanding people there: Tomas Roos and Tom Janssens.

I’m a completely newbie in this development stack (summarised by Tom):

Migrating a large free dating website from the Microsoft stack to Erlang/Linux...  
  Some keywords:
  - Programmer anarchy!!!
  - Erlang/Python/C/
  - FoundationDB / Phonegap / flightjs / responsive design
  - Chatroom devops
  - Remoting
  - CQRS / DDD / Eventstorming
  - Wercker / DigitalOcean / Github / BrowserStack / PhantomJs / Behave
  - Campfire / Mindmup / Google docs / Dropbox / Google Hangout
  - And many more!

I hope to be learning fast enough to keep up with these guys, so that they will not kick me out of the team due to the sheer stupidity, of which I’m guilty on more than one occasion. So far the experience has been truly amazing.

We’re going to have RnD blog for this project.

I’m still going to stay with Lokad in a support role, however my time spent there is going to be limited. The company needs somebody to take on some of my responsibilities. If you are an experienced .NET developer with desire to get immersed into Windows Azure devops, Big Data, Business Analytics and event sourcing, there is an opportunity for you in Paris (I’ll try to help as much as I can in this process). Please get in touch via contact@lokad.com.

Stop Overdoing Things

During the course of the last years, software development somehow took a wrong turn and got vastly overcomplicated.

These days we just tend to over-design and overdo things. Consider projects that never get shipped because stakeholders keep on drawing UML diagrams for the requirements. Consider web sites that never surface the public web because of ever changing UI designs. Consider implementations so convoluted that development teams spend their entire lives playing “hot potato” with JIRA bugs that keep on showing their ugly heads up. Or consider software architectures so entangled that even ivory tower architects have their eyes bleed when they see implementations of them in the real world.

0654 V2 0 ARCH IMAGE FOR CODEPLEX thumb 2E61DCFD

Things don’t need to get complicated simply because everybody else seems to do that. Break down complex into simple, release early, iterate quickly and learn from feedback.

Here’s an example of a web site that just works:

Screenshot 2013 11 25 13 17 31

A blog theme that just works:

Screenshot 2013 11 25 13 18 27

Here’s an example of IDE that just works:

Screenshot 2013 11 25 13 16 12

There is no point in trying to be perfect from the very start. You can get a better chance of succeeding if you just get the smallest possible thing done, make it real and then iterate from there.

Sometimes you might even find out that good enough software works good enough and does not need to be any more perfect.

Separation of Contract and Implementation in Software

A bit earlier I posted a sample drawing with evolution options for a component in a big software application. This diagram is merely a convenience snapshot of performance optimisations and tech options available at any given point in time. Knowing about them in advance helps in planning future development.

Each node on the tree of component evolution represents a design pattern that has its own advantages and costs. Costs almost always include increased complexity. Sometimes such design pattern also has certain affinity with other patterns, making it simpler to evolve component to the next level.

For example, consider a component that happened to arrive at the following design during its evolution (after multiple iterations):

2013 10 11 Component

We can say that this component has a public contract and an internal implementation. Public contract could say:

  • Component implements a certain documented API with JSON, XML and ProtoBuf formats; these API interactions can be scripted or tested using tools on a variety of platforms;
  • we can expert 99.9% uptime of the component; query response times under 50ms in 99% of the cases; commands are acknowledged synchronously in 500ms in 99% of the cases;
  • we expect API to have throughput of 1500 transactions per second, if deployed in single-node configuration (all transactions extra would be rejected with 503 Retry Later);
  • Queries have eventual consistency of less than 1000ms in 99% of the cases.

All of the above is easily achievable, for example, using .NET on Windows Azure with multi-worker deployment configuration and an efficient Event Store.

Yet, please note, that the public contract does not say anything about the implementation details. It’s normally up to the team to decide what these should be (better if that team is also follows the mantra “you build it, you run it”). This means, that at any given point in time, internal implementation might change in order to accommodate new requirements. Implementation might also change if requirements get relaxed and we can actually degrade the performance and get rid of some complexity in exchange.

If component boundaries are defined well (as driven by strategic design and its evolution), then public contract will not change often. In such case development challenges are merely constrained to deal with the implementation details, shifting it along a well-known evolution path in order to achieve well-known benefits. We could actually fine-tune specific components to meet certain requirements.

We might even say that with this natural approach, large scale software design emerges as a by-product of design process driven by two distinct feedback loops:

  • evolving strategic vision which deals with business capabilities and how they are implemented by composing together components, defined by their contracts.
  • fine-tuning component implementations to fulfil their contracts.

However, while doing all this evolution, it is really important to break such process into small steps, which can be handled separately. Doing work in tiny bites provides you with opportunity to step back, acknowledge feedback, reflect upon the design at strategic and implementation levels. I think that one would find more opportunities for such separation if we:

  • decompose software into small focused components;
  • handle component contracts separately from their implementations;
  • plan evolution process in advance, where possible.

Many Ways for an Emergent Design in a Component

Recently I mentioned 6 steps of an evolutionary design in software development. These steps describe iterative process aiming at continuous improvement. Such an improvement process can happen at two distinct levels:

  • High-level view of the entire system involving components and their interactions;
  • implementation details of a component.

While high-level system evolution is covered pretty well in methodologies like domain-driven modelling (strategic design), implementation level can be more project-specific and hard to explain in uniform fashion. May be that’s because there is no generic approach to describe evolution of components or services in a real-world system. Each element might need to evolve in a unique way to order to reach the best balance between complexity, performance and capabilities.

For example, let’s consider an evolution path that a single component can go through in a startup team focused on emergent design, rapid iterations and .NET stack:

2013 10 30 stacks

Team dynamics, past experience and current political situation might lead to a design approach, where each component starts as a simple console app and then evolves towards more complicated design in order to fulfil specific requirements. We try to keep things as simple as possible, but no simpler. If a component is kept simple and focused (which is a task of a strategic design), then at any point in time it could be rewritten from scratch.

Evolution tree below is merely a visualisation of existing design approach inside a given team, serving as a way to make design options more explicit and allow better communication. Any change in team, business priorities or design methodologies could affect this evolution tree.

2013 11 24 evolution paths

Here is a bigger version of this image.

Please note, that at any evolution along such design tree is a specific optimisation that comes at the cost of complexity. Sometimes it is better to delay paying that complexity cost and keep your options open.

Things I Learned Recently

For the last few weeks I’ve been really busy with development at Lokad, without time to reflect about things publicly. This development was (and still is) focused around refactoring of Lokad backend infrastructure and a few standalone products that integrate with it.

Previously, all systems were based on regulated Lokad.CQRS design:

  • large system is broken down to one or a few bounded contexts;
  • each bounded context is captured in the code using one of a few predefined blocks:
    • Aggregates with event sourcing
    • Application services (wired to receive messages from queues)
    • View Projections
  • all communication happens via messaging (command messages and event messages);
  • events are stored in dedicated event store and then published around.

This approach has multiple advantages, yet is extremely limiting. For example, UI latency is the most “tangible” disadvantage:

2013 10 11 Monolyth

Yet, this was only the tip of the iceberg. Real problem was in the design approach that was completely wrong. I tried to come up with a few reasonably good design blocks and then mechanically assemble everything from them. This does not work when you are trying to analyse and capture a real world in your code - there are too many oddly shaped pieces that refuse to fit in a rectangle.

Hence, after learning a little bit about the ways of Christopher Alexander I started shifting towards much simpler design approach that consist of the following steps:

  1. Decompose problem domain into small components, you can call the resulting plan “town map”.
  2. Decide how these components will communicate (JSON over HTTP works really well, esp with a little bit of REST, where it makes sense).
  3. Implement each component in a brutally simple way (you want to release early).
  4. Get feedback!
  5. Either optimise individual components or adjust your “town map” based on feedback and insight.
  6. Iterate (while reducing friction and effort of each iteration).

2013 11 04 6 steps of design process

So far it works pretty good, I might add.

How to Produce a Superb Software Design?

It’s impossible to arrive at a perfect design right from the start. There are far too many uncertainties and unknowns involved in the process. Hence, you can safely assume that some sort of iterative learning process would be required.

Software development is a learning process. Working code is a side effect.
© Alberto Brandolini, Model Storming Presentation

Lessons learned could come in various forms:

  • discovered requirements and constraints;
  • implemented features or fixed bugs;
  • written documentation (wiki articles, email threads, diagrams, napkin drawings, whiteboard photos, software guides etc).

One of the most important goals of this learning process is to keep iteratively integrating newly discovered knowledge into already captured body of knowledge, while enriching it. This task might be more challenging than it seems at a first glance.

Let’s step away from software for a moment. If you think of it, continuously adding stuff to any existing storage container could eventually lead to a situation when we not only ran out of the free space, but it becomes really challenging to do anything with things that are already stored there. Consider filling cabinet with papers or adding more tools into a small closet. Unless we somehow reorganize the space (add more filing cabinets, introduce more shelves in a closet), it will be hard to work with all the accumulated stuff. Another option would be to come with smarter way to organize papers (e.g.: alphabetically) or tools (e.g. put the ones that are used most frequently - upfront).

Same principle can apply to software design. Even though software size is rarely a limitation these days, our mind is quite limited in the number of things it can handle at once without the support of some sane mental model.

…number of objects an average human can hold in working memory is 7 ± 2.
© Wikipedia, The Magical Number Seven

That’s what happened in far too many projects that I’ve seen: new features were continuously added to the project without adjusting software design to handle increased complexity. Such situations lead to the point where software got so fragile that developers would spend more time fixing bugs than adding new features. Quite often resolving one bug would release a horde of new ones.

One of the most frequent solutions to this problem sounds like this: “There is too much legacy. We can’t add any more features to our software till we rewrite everything from the scratch in version two.”

In reality, approach of rewriting complex software from the scratch can be more problematic than it appears:

  • rewrite costs time and money;
  • new bugs can be introduced in the process, since we are changing existing and working software for something new;
  • some really important requirements and features can be lost.

Ideal solution to the problem of increasing complexity would be to avoid such bottlenecks in the first place. This can be done by:

  • keeping software complexity as low as possible;
  • continuously evaluating fitness of design as more requirements and features are introduced;
  • continously adjusting design to stay meaningful and simple despite new features being added.

In this case we can talk about iteratively integrating new insights (features, requirements etc) into the software, while also adjusting the design to handle increased complexity. At each step software would stay healthy enough to keep on pushing forward its evolution without much friction. I believe, this constitutes a superb software design.

Hence: Arrive at healthy design by evolving it through a series of enriching transformations.

This might sound like a lot of work. However, if we design for such continuous evolution upfront, then at each step we would need to deal only with a limited scope of change. This would reduce risk of introducing unexpected bugs or loosing an important requirement. Developers would stay sane, too.

Note: This blog post is a draft of one story for an ebook on healthy design and patterns. Stay tuned, if interested. All comments are welcome.

Evolving

Enriching Transformations

It’s impossible to arrive at a perfect design right from the start. There are far too many uncertainties and unknowns involved in the process. Hence, you can safely assume that some sort of iterative learning process would be required.

Software development is a learning process. Working code is a side effect

Model Storming Presentation
— Alberto Brandolini

Lessons learned could come in various forms:

  • discovered requirements and constraints;

  • implemented features or fixed bugs;

  • written documentation (wiki articles, email threads, diagrams, napkin drawings, whiteboard photos, software guides etc).

One of the most important goals of this learning process is to keep iteratively integrating newly discovered knowledge into already captured body of knowledge, while enriching it. This task might be more challenging than it seems at a first glance.

Let’s step away from software for a moment. If you think of it, continuously adding stuff to any existing storage container could eventually lead to a situation when we not only ran out of the free space, but it becomes really challenging to do anything with things that are already stored there. Consider filling cabinet with papers or adding more tools into a small closet. Unless we somehow reorganize the space (add more filing cabinets, introduce more shelves in a closet), it will be hard to work with all the accumulated stuff. Another option would be to come with smarter way to organize papers (e.g.: alphabetically) or tools (e.g. put the ones that are used most frequently - upfront).

Same principle can apply to software design. Even though software size is rarely a limitation these days, our mind is quite limited in the number of things it can handle at once without the support of some sane mental model.

…number of objects an average human can hold in working memory is 7 ± 2.

The Magical Number Seven
— Wikipedia

That’s what happened in far too many projects that I’ve seen: new features were continuously added to the project without adjusting software design to handle increased complexity. Such situations lead to the point where software got so fragile that developers would spend more time fixing bugs than adding new features. Quite often resolving one bug would release a horde of new ones.

One of the most frequent solutions to this problem sounds like this: “There is too much legacy. We can’t add any more features to our software till we rewrite everything from the scratch in version two.”

In reality, approach of rewriting complex software from the scratch can be more problematic than it appears:

  • rewrite costs time and money;

  • new bugs can be introduced in the process, since we are changing existing and working software for something new;

  • some really important requirements and features can be lost.

Ideal solution to the problem of increasing complexity would be to avoid such bottlenecks in the first place. This can be done by:

  • keeping software complexity as low as possible;

  • continuously evaluating fitness of design as more requirements and features are introduced;

  • continously adjusting design to stay meaningful and simple despite new features being added.

In this case we can talk about iteratively integrating new insights (features, requirements etc) into the software, while also adjusting the design to handle increased complexity.

Tip
Arrive at healthy design by evolving it through a series of enriching transformations.

This might sound like a lot of work. However, if we design for such continuous evolution upfront, then at each step we would need to deal with a limited scope of change. This would reduce risk of introducing unexpected bugs or loosing an important requirement.