My productivity system

My productivity system I've been reflecting a lot about the personal productivity systems. It turns out I actually have one. If writing things down or keeping a journal works for you, then you might also find the story interesting. The reflection was triggered by a couple of tweets. The first one was from Matthias Verraes : Bloggers: "I'll demonstrate DDD by building a Todo app" Actual DDD: "We looked at how people are using Todo lists, and we figured that our model was wrong. Besides TODO and DONE, we needed to afford for POSTPONE, WON'T DO ANYWAY, and TOO LATE. A quick experiment showed that..." It evolved into a healthy discussion about personal systems, handling weekly TODO lists and aligning them with big aspirations. There is even a mention of a Life OS built with Notion and Miro . I tried multiple similar approaches in the past but always got stuck. I always end up with a TODO list that never gets completed. It makes me feel guilty, so I just back off from the system to never come back. In retrospect, most of these productivity systems need regular reviews to keep working. Reviews need time. Up to one hour for the weeklies. We have two little kids and a pug. Life with little kids is a challenge on its own, but you can also throw in COVID and relocation to Austria for extra fun. Things are better now, but I still remember weeks and months with no spare time and motivation at all. Without time and motivation, tasks and pending reviews start piling up. To get back on track, I would have to deal not only with low motivation but with even more cleaning up to do. That would be impossible. Guilt would grow, and I'd just pretend that the system never happened. However, there were a few small things that kept on coming back to me throughout these years. Healthy exchanges with Laura , Tamara and Beau helped me to recognize these surviving bits and the system behind them. It doesn't work always, but all my major work-related achievements in the past 6 years were supported by it. In retrospect, the approach is simple: the productivity system should adapt to good and bad times; it should be based around writing because that is how I think. The system is a small application. When the motivation is high, it works like a Journal. The system tracks my notes and tasks across various topics. The user experience is designed to present each topic as a story, encouraging me to keep it going. Whenever I neglect the system, it backs off. It hides all the low-level details and incomplete backlogs, showing me a list of stories and an empty canvass. Every time is a fresh start that builds upon past aspirations and achievements. When my motivation is high, it works like a simple Journal I write down my actions and notes in the journal, putting them in one of the topics. Examples: Personal health, Robotic hand, DDD Katas, DSML Book, Wireguard, etc. If there is no matching topic, I create a new one. I don't call them "projects" because that feels like an obligation to handle. There is too much "work" in that word for me. After the note is added, the system extracts the headers and pushes them to the Week Outline and Topic Outline . Both are high-level. Week Outline tells a story of the things I've done that week. Topic Outline is like a table of contents for the entire topic, telling a story from a high level. The list of Recent Topics also gets updated. These are my aspirations and interests that arise naturally. I can do just a little thing like saving an interesting thought or a link, but this is immediately acknowledged by the system and the related story. The system updates all views and shows the latest topic outline. This puts me back into the strategic view and encourages me to think about the next steps. How can I continue the story? If I have time, I could work just a little bit more. If there is no time, I could write a note to myself for the next weekend: "take a look at this project!" This will be my TODO that will show up in the week and topic outlines. It will also become a part of the story. When my motivation is so low, that I start neglecting the system, it turns into a storybook . Daily notes and TODOs fade away from the Week Outline because they were bound to a specific date. They are still safely stored by the system. I could dig them up if needed. Should I ever want to open the system after a period of neglect, there will be no guilt. Just an empty canvass and a list of stories from the past. My interests and problems will never go away. Eventually, there will be a break and I'll do something worth remembering. Journal notes are guaranteed to be safe . Immutability, redundant backups , and CAS for the binary data handle that. So if I have a bit of information to store, I would dump it into the Journal without thinking. So I go back to the Journal to add a quick note. All user interactions start with a list of recent topics. This pulls me back to the big picture - these were the past interests and big challenges, at a high level. As soon as I hit "Save", the journal would present me with the outline view that includes the last added topic. Last incomplete TODOs will be balanced with the weeks and months of small accomplishments before them. No fluff or guilt, just the big picture . Telling me: "hey, we had some fun with that story 6 months ago. Do you want to continue it?" Usually, the answer is yes, even if that is just to spend 10 minutes figuring out what to do next or cleaning up the code. But, hey, it gets me thinking and working on that story. There is comfort in knowing that the system will continue chipping away at problems for years. Should there be a challenging period, it will eventually get me back on track. There will even be a story about all that afterward. To summarize my system : Capture notes and tasks in a journal. When saving them to a topic, extract the headers and build an outline (table of contents). Display outline in the weekly view, this will pull you back to the big picture. Display current week by default, hiding past tasks as time goes on. This will allow guilt-free fresh start after a period of neglect. Display topic as an outline by default - whenever you come back, this will tell a story from the high level. Maintain a list of recent topics to surface aspirations and problems. Use reliable system with backups. Reliability will make you come back to save things. The system will then show the big picture and nudge you to continue writing stories. If you enjoy writing or leverage it to get through difficult times, then try building a personal productivity system around that. It could provide additional support and comfort on your journey.


Event Now!

Event Now! Current Event KanDDDinsky 2025 Recent Events LLMs in Marketing Hosted by the Data Lab at ESSCA Research, organized by Chaaben Kouki Click here to join the meeting (MS Teams) Links Lokad.com - better supply chain decisions Trustbit - Large Language Models for the companies Joannes Vermorel - blog Chaaben Kouki ESSCA Research Agenda FAQs, materializing the institutional knowledge Security/privacy and AI strategy for business. Generate product listings on marketplace (using text + vision) Automatically fix errors in product catalogues Write high-quality marketing texts for products Intent-driven search on a marketplace (recurring pattern) Questions and responses from audience. Large Language Models in Supply Chain With Joannes Vermorel from Lokad.com Coding with AI - by TrustBit Hosted by private customer Links: Replit - online IDE ChatGPT quickstart for developers How to get into ML for a developer? Extra: Trustbit LLM Benchmark ML under the Hood Newsletter AI in Software Development Kata Hosted By Christian F. and Erol A at Spar ICS, Salzburg. Links: Replit - online IDE ChatGPT quickstart for developers How to get into ML for a developer? Extra: Salzburg Software Craftsmanship Meetup page Trustbit LLM Benchmark


10 Steps To Become Better .NET Developer Here's a list of things you might want to learn about in order to become a better .NET developer. Better developers are eligible to higher paychecks , exciting projects and more freedom in their lifestyles. Learn Unit Testing Get familiar with distributed Version Control Systems : Git or Mercurial . Learn about IoC Containers : Autofac , Windsor Container and StructureMap . Contribute to Open Source Projects of your choice. Learn about Continuous Integration and try setting up TeamCity , Hudson or CruiseControl.NET . Learn about NHibernate Object Relational Mapper, but be prepared to outgrow it for NoSQL persistence like CouchDB , MongoDB or RavenDB . Check out the buzz about cloud computing , try Windows Azure Platform and also give a shot to running your .NET App with Mono on Linux ( some free offers to try). Familiarize yourself with messaging architectures and try NServiceBus , MassTransit or RhinoQueues ; then learn about alternative to MSMQ - AMQP and RabbitMQ . Read Domain-Driven Design and apply it in your projects. Get Started with CQRS Architectures and practice it in a learning project. As you can see, the list is not endless. There are just ten distinct topics structured in the recommended order of learning and practicing. Diligently walking through them is certain to get you way above the level of Senior Developer, opening a whole set of opportunities all around the world. And if you happened to learn, understand and master all these steps and still don't have interesting job or project offer - drop me an email ; I might get you in touch with folks desperately looking for developers with this set of skills . PS: On August 2012 we started a new podcast to help aspiring developers. Check it out .


Blog fully migrated to Obsidian

Blog fully migrated to Obsidian Last weekend I finally finished my great blog refactor v6! It is such a relief and good feeling, that I really want to share it here. TLDR; is now better integrated into Obsidian. This enables to move articles around, use wikilinks and wikilink-style images and benefit from the knowledge graph. As I side effect, I discovered and pulled aside collections of articles on event sourcing , opinionated tech and robotics . All that done is done mostly without breaking any existing (and very messy links) 🙂 Implementation Details Here is how the website currently works. I write my blog posts in Obsidian. When there is a time to publish, Python script runs. It will: Grab all markdown files and extract YAML-frontmatter from them. YAML contains just a few config entries: uid - unique auto-generated id; title - in case I want to have special symbols in the title; url - canonical (latest) url for the article. Run markdown through Markdown and Pygments (for code snippets). Run through wikimedia link parser to convert links like [[2017-12-17 Black Friday load|Black Friday]] to proper <a href . Generate static html, wrapping markdown content with an html layout ( Flask and Jinja ). Load url_map.tsv and generate redirects from various url versions to current canonical representation. rsync resulting folder to a remote server. Caddy will serve it there. That code used to be open source, but unfortunately isn't any more. Codebase is currently a mess 🤷 and is going to get even more messy. I want to bring back the comments and flatten the stack . That will require another rewrite. Did you know that automatic HTTPS from Caddy is available as a golang library ? However, if you are interested in blogging, setting up a blog or just writing in general, please don't hesitate to reach out to me. Writing is an important part of staying connected, productive and healthy in a remote-first culture. I'd be glad to help! ✌ ❤


New experiment in 2026

New experiment in 2026 This year I've decided to run one of the biggest experiments in my life - stop working as Head of ML & Innovation at TimeToAct Austria and take the risk of starting an independent business again. It felt to me that staying in a large company (anything larger than ~20 people feels large in retrospect) is a bigger risk in 2026 than leaving a secure job in an age of turmoil. There are just too many opportunities and ideas to explore, and large companies are notoriously slow to adjust and pivot, especially now. The way a recent PwC report frames it - workflow redesign is the “hidden multiplier” for the companies, while bureaucracy is essentially a tax on speed. Setup The experiment has started on February 1st, 2026, when my contract with TimeToAct ended. Here is the summary of things that happened since then. First, we took a week of vacation. It is pretty much obligatory in Austria, where travel opportunities for parents are always aligned with the school holidays. My wife is a product manager who’s spent more than a decade running remote teams across five continents. She’s exactly the person you want designing processes for a two-person startup in 2026 with clean slate and good tooling. We sat down together at the beginning of the second week and figured together the following schedule for me: Four creator blocks, 2.5 hours each (Mon - Thu) 1 hour for journaling and reflection on Monday 1 hour for maintenance and paperwork 1 hour for community management (while I'm waiting for the kids in the swimming pool) 4.5 hours in total for calls, insights and research 2 hours for hacking 2 slots to go to the gym + my regular Kendo practice with my son 2 hours for review and planning - at the end of the week A few hours to deliberately learn German It was important to balance obligatory tasks and boring routine with the other activities that provide energy, perspective and a chance to create. So, given 4 creator blocks, we've decided to split them between two activities: Create something fun that leverages my experience in creating fun collaborative challenges, presence of an awesome community of engineers and recent hype around personal agents. Finally finish and publish an English course on patterns and practices of deploying LLM in business for high-impact scenarios. We've run that for 3 sprints/weeks so far. First week was mostly figuring out - how do I work and setting up a laptop from scratch. So ~6 days focused on one of these initiatives with some time limit. While the schedule is not prescribed, it creates a welcome and diverse routine. Most importantly, it gives structure and introduces sport as a routine back into my life (something I haven't had time to do for years). Outcomes Here are the outcomes. 1. First boot of new agentic environment First, I've started designing a new challenge for the agents, targeting the simulation shape that corresponds to personal agents, as inspired by the Claw implementations and Homai architecture. By now I'm pretty confident that I will be able to pull things off, and also have an idea of tasks that could be fun to include in the challenge. Plus the new architecture will be able to scale better, to support all the participants running their agents. I also managed to solve a long-standing problem of creating and maintaining SDKs for the agents in multiple languages (people in the community tend to code in Python, JS/TS, C# and sometimes Kotlin/Java), all that while avoiding the mess that is OpenAPI/Swagger. The answer was Buf/Connect together with Codex, which quickly iterated through a couple of integration approaches until we arrived to a pleasant architecture that also happened to reduce code surface for me. Fast feedback is pretty important, so yesterday I was able to spawn my first lightweight VM and connect a mock agent to it via remote protocol that will be used in the challenge. Also, another fun thing. A while ago (in Better performance with DOD ) I mentioned as my goal: " NixOS, the power of declarative VMs and per-project system dependencies ." As you can see in the screenshot, these things are an essential part of my toolbox right now. Next steps: build up a tiny demo challenge, ship it with the SDKs to people and see what breaks. 2. Setting up a community We've also started working to figure out and shape the actual competition - how to organize the event, how to set up the timeline, register participants, connect them together in cities, stick to the privacy policy and terms of service, introduce code of conduct etc. Yes, organizing an event is suddenly a lot of paperwork. Thankfully ChatGPT and Claude can work as helpful assistants to streamline research, drafting and analysis. Things got slightly out of hand along the way. By the end of the month, we had a BitGN platform , April 11 as the date of the first challenge, 341 registered engineers that come from 68 cities around the world (still missing a few continents, though). We also have 5 hubs around the world (while the competition is remote-first, people tend to want to gather together and have fun together) with the HQ hub located in Vienna. Next steps: try to get more hubs into the competition. 3. Labs "Labs" is the codename for my Course platform. This is where I made the least progress (being too excited by the things that happen around BitGN). Still, it was something. First, I started working on the new website to serve the course, using the same stack as the one that powers BitGN community. DB structure, migrations, solution architecture, engineering harness and other tools were first designed in this project (codename "Labs") and then also deployed to BitGN. Among the most fun parts: DB migration framework that just works (meaning that Codex and Claude make no mistakes in evolving schema and I have no surprises when I deploy these changes to prod). Zero downtime deployment process for my stack (uses socket activation managed by SystemD within NixOS to drain connections to the old instance before starting the new one) Login with Google (it turned out to be simpler than I thought, reducing the need for me to deal with lost passwords and account sharing) Fun way to handle legal documents like ToS and Privacy Policy. Now I can change them in markdown files (if needed) and all users will immediately see the changes and will be required to accept them to continue using the site (acceptance timestamps will be recorded). Not that I plan to ever change them, but if there is a legal need - there is a zero-friction way. Engineering harness structure for my projects that makes it easier to track changes, decisions and supporting documents and communicate them to the coding agents. As you can see, most of these changes focus on reducing friction and allowing to scale human effort with the technology. These tiny improvements compound. For example, while writing this blog post, I came up with an idea to display a table of new cities that joined our community: It was a fun feature, not worth my time, so it was done completely by Codex (with my two follow-up requests): Yes, the table could be done better. Yes, styles are slightly inconsistent. Enterprises can come up with a thousand reasons to not deploy it. But it is good enough in the startup mindset! And since the deployment and DB versioning pipelines are already in place - it went live immediately. We can clean things up later, if needed. While these approaches work best of all with the code (since it is easier to verify quality), that can also be applied to the other areas of human life. 4. Personal knowledge base I always envied people that maintain elaborate and well-designed knowledge bases and personal productivity systems. I tried far too many times with tools like OmniFocus, Notion, Evernote, DevonThink, Obsidian etc. Most of these attempts stall due to my lack of patience in maintaining them. However, there was just too much noise with OpenClaw/MoltBot personal assistant, that I had to try something. The noise, I believe, happened because a noticeable portion of the population was able for the first time to experience what it feels like to have a personal AI assistant. And that experience was radical enough to fuel that media wave. I tried installing OpenClaw on a dedicated VM (for security). It felt refreshing in its freedom and ability to try things proactively. At the same time I didn't like lack of control and precision. Since I've been benchmarking too many LLMs and architectures, I just know too much about their failure modes. So what do you do, when you want to figure out something, but don't know how to approach it? You talk to a rubber duck. I had a fun session with ChatGPT Pro, trying to figure out how I can get a feel for OpenClaw's capabilities in my daily workflow, but in a more structured way. The resulting architecture involved a git repository that contains knowledge in markdown format, intermixed with agentic instructions on how to handle things. This repo is browsable with Obsidian and is connected to the remote (cloud) Codex. Structure at a high level looks like this: . ├── 00_inbox ├── 01_capture ├── 02_distill ├── 04_projects ├── 07_rfcs ├── 89_images ├── 90_memory ├── 99_process ├── AGENTS.md └── CLAUDE.MD It was designed by Codex in order to attempt solving my problem of experimenting with agents and giving agents more autonomy, while giving me the ability to ask weird things and still have some control. We've tried the following experiments: Automated knowledge capture and distillation . That surprisingly works and is the most used feature. I tend to throw URLs or voice transcripts to Codex, asking it to file the information. It will record, file, distill into a card (brief) summary. For example, yesterday I had an idea of connecting the concept of a personal agent with BitGN Challenge, and also my personal OS and linking with meta research on agentic trajectories. The idea came in a gym, so I opened up Codex on a smartphone, did a memory dump and asked to file it. The fun part was not the capture or distillation, but an afterthought. My Codex environment is configured with SOUL.MD (Be good, seek knowledge) and also an instruction for an agent to connect the dots on its own, exploring synergies, collecting insights. So here is what it wrote, reflecting back my own thoughts in a new light (which gives a few new ideas): <!-- AGENT_EDITABLE_START:reflection --> Novel vs recent cards: this is a concrete design leap from “agent workflow improvements” to a collaboration substrate where merge semantics become first-class product/API behavior rather than a tooling afterthought. The strongest compounding idea is dual-use architecture: one API contract can power both benchmark/challenge telemetry and real household/enterprise assistant operations, reducing duplicated infra bets. Treat semantic edit operations as governance boundaries (not only UX sugar): once operation types are explicit, authorization/reconciliation policies become enforceable and auditable per operation class. <!-- AGENT_EDITABLE_END:reflection --> Yes, I understand, that using Codex (which is a programming environment with its own sandbox) to download PDFs, capture sites or reason is an overkill and waste. But it works for me. The only problem is that this collection of insights lives in a GitHub repository, so it might be an inconvenient place for the collaboration. So I've done another interesting thing. I started up Codex and asked it to inspect the repository, the nature of my last changes (~20 last commits), all recent captured insights and come up with an idea of how I can keep all features but make the knowledge base accessible to others (ideally, readable by ChatGPT Desktop/Claude Cowork via MCP). The entire thing had to be usable by the agents (I'm not editing these files by hand, only viewing), while still giving me the ability to review any changes and roll them back as needed. Plus, if there are any conflicts - it should be easy for the agents to handle the merges, not me. So Codex came up with a fun and weird design of semantic text management repository with CLI/API interface (pluggable as MCP) that makes it easy for agents to manage on behalf of their controlling humans. My next step in this experiment is: "Hey, I like this design. Now go ahead and write me a thorough RFC on how to implement this semantic storage and completely migrate this repository to it." I wouldn't be surprised if it succeeds (not after seeing yesterday how Claude Desktop can manage org data in Notion after being connected to MCP). I wouldn't be surprised if that task fails today, because it is too complex to be solved in one step. Either way, this will be another interesting and fun experiment that will teach me something new. And it will potentially create one more tiny compounding effect that will slightly reduce friction for all the other work that happens in the upcoming months.


Aggregate Design and Security

Aggregate Design and Security Answering a question from the community: How do you handle security concern of sending aggregate id to the web client? What'd you do to ensure I can't, from the browser or another client, manually send aggregate id that belongs to another client to either get some data about it or send some commands to mess things up? Each specific system scenario would lead to the specific design. If our specific scenario is: business logic is hosted on Application Server (e.g. hosted in Lokad.CQRS) which we control trust; client UI is hosted on Web server UI which we control and trust; Web server accesses application server (by sending commands and polling views) and renders HTML to user's browser. Then the solution to the security concerns problem is: We don't trust client browser anything (except from keeping unique token) we trust Web server UI to authenticate user (ensuring that he is who he claims to be). In this web App we will ensure that client sends only commands to the aggregates he is allowed to access (while checking commands as well). In app service we don't do any deep security checks and just carry out the commands. This scenario works well in simple situations where we have an app server and a few Web UI servers (alternatively with REST API servers), managed by the same team and hosted in the same controlled environment. If we have a different environment (e.g. application server can't trust web server), then a different separation of responsibilities might be needed. For example, we might need to perform authentication within the methods of Application service (checking that user is who he claims to be ), while pushing authorisation down to method calls on an aggregate ( checking that user can do what he attempts to do ). In this scenario, authorisation might be encapsulated within an instance of domain service , passed down by application service to a method on aggregate . This service would have access to permission maps (or any other way of representing role/permission information), aggregate would call it's methods to find out if the specific user can perform requested actions on the requested resources. Naturally, in this case, Web UI must still ensure that we don't send illegal commands to the server. The difference from the first scenario is - we don't trust the web server to be diligent. These and other specifics of project environment, team organisation and trust can have a great impact on development process and product design.


Analysis Paralysis

Analysis Paralysis How many times did you want to start a new project and implement some really exciting idea? I've been there multiple times myself. Most of my attempts failed in the very beginning, because I was trying to think or plan too much in advance: How do I plan for future extensibility and adding new features? What if I need to switch between databases - how do we abstract this away? How do we scale out for 1000000 users? I need some formal process around features and releases. If project becomes successful and grows, new team members should have no difficulty joining in. These are the most sane of the imaginary requirements that used to come to my mind (more exotic ones included terms: "Neural networks", "Linux kernel", "ARM processor support" and "should make good expresso"). All this felt like something good, as if I were planning for every feature and problem in advance. However, in practice this somehow used to turn simple and exciting projects into challenging sets of problems, that had to be solved all at once. Most of the times, these sets were so complicated, that I had to stare them in awe without any slightest idea of where to start and what to do next. This state is often called analysis paralysis (or the worst way of dreaming). As you might guess, almost all of such projects were dropped, while the other half failed later during careful planning and execution. We can wish to be prepared for a lot of problems and features in advance. But do we really need for them to happen all at once? That's really hard to achieve. Life is simple. You can't walk 1000 miles at once. There has to be the first step, and then the one that will come after. There is an approach that helps to move forward with development in such situations (first time I heard it from Gregory Young). It can be really hard for developers, since we all are inherently perfectionists. Instead of trying to plan for the entire project in advance, we just take the smallest bite possible . You can call it a " prototype ", " minimum viable product ", " let's give it a try or " dirtiest and hackiest code I've ever written ". This attempt will be fast and deal with the core idea . If it fails - it will fail fast; if it makes at least some sense - it will only get better from this point. We can focus on the most painful problem that makes this idea shine (it will be easy to prioritise) and solve it. Then the next one and the next. The idea is just to start walking towards the goal, instead of burning yourself down by an attempt of 1000 mile jump (only to discover that you jumped in the wrong direction). The approach becomes even more valuable, when there are multiple stakeholders involved in the project. It is much easier to arrive at collaborative analysis paralysis, when everybody keeps on throwing their dreams in : " we want this ", " it should do this ", " what if this happens? ". The most simple solution to the core problem provides team with a starting point for discussion and planning. It makes discussions more real, than juggling with wishes and fears in the abstract problem space. This approach also helps to prioritise further progress - you focus on the most painful thing first. Life and projects can overwhelm with problems. Keep it simple and focus on making next most important step. Step by step, you can walk around the world.


Anatomy of Distributed System à la Lokad This is an update of my previous post on Building Blocks à la Lokad . It is based on improvements of the understanding and terminology thanks to Vaughn Vernon . It also shamelessly borrows from the subject of my continuous fascination: biology and nature itself. Bounded Contexts Let's start with the high-level overview of a Software-as-a-Service company (SaaS). This company could employ a range of systems needed to deliver its services to customers. Systems will be separate to reduce coupling and reduce the friction, as company evolves and grows (smaller the company is, more important this becomes). Each system is a separate Bounded Context ( BC term comes from DDD), which could have different implementation details: teams, technologies, language, lifecycle, deployment and maintenance specifics. Below are some examples of BCs in a SaaS company. Please keep in mind, that each bounded context is a modeling concept and not an indication of how system is implemented. For example, Client Portal could be implemented as a stateless web farm with ASP.NET MVC3, redis-based state server and cloud-hosted processing logic, accessible by desktop and mobile clients. Or, it can be a single Windows Server with IIS, file-based state and a few console processes running in background. This is similar to how a human body is composed of a set of biological systems , where each one serves specific purpose and can be built from specialized organs, cells and tissues. Events: Forming Digital Nervous System These separate bounded contexts need to communicate with each other. It happens by publishing events by each bounded context to an outside infrastructure. Each event is just a serializable message that complies with certain guidelines and is routed by messaging systems. Each event tells about something that has already happened : Invoice Created or Invoice Possibly Expired . Events streams form digital nervous system of a company, where bounded contexts act as biological system. Image above reflects event flows in an imaginary company. It might look complex in this specific representation, however, this doesn't always need to be perceived this way. While publishing, sender does not even know about existence of its recipients, however recipients can subscribe to any publishers, they are interested in. This traditional PUB/SUB approach simplifies the picture a lot, since we can focus on specific bounded context with its dependencies. It also makes it similar to information flow through biological neuron . Each bounded context can subscribe to events in two distinct ways: by declaring event receptors or with view projections . 1. Event Receptor Event Receptor is a simple declaration of events that the specific bounded context is interested in and will react to by sending commands to application services of this context. In code such receptors are usually grouped together in classes, according to their purpose: public sealed class ReplicationReceptor { // 'Domain' is the name of the primary Bounded Context // in this system readonly DomainSender _send; public ReplicationReceptor(DomainSender send) { _send = send; } public void When(SecurityPasswordAdded e) { _send.ToUser(new CreateUser(e.UserId, e.Id)); } public void When(SecurityIdentityAdded e) { _send.ToUser(new CreateUser(e.UserId, e.Id)); } // more receptor methods skipped 2. View Projection View Projection subscribes to events, which are projected to a view (or persistent read model) that is structured in a way that will be easy to query by components within this specific bounded context. In code, projection elements are grouped together in classes based on the view they keep updated. public sealed class InvoiceDeliveryProjection { readonly IDocumentWriter<unit, InvoiceDeliveryView> _docs; public InvoiceDeliveryProjection(IDocumentWriter<unit, InvoiceDeliveryView> docs) { _docs = docs; } public void When(CustomerInvoicePaymentRequested e) { var mark = new InvoiceDeliveryMark { AccountId = e.Id, Created = e.RequestedOn, InvoiceId = e.InvoiceId }; _docs.UpdateEnforcingNew(view => view.Invoices[e.InvoiceId] = mark); } public void When(CustomerInvoiceClosed e) { _docs.UpdateEnforcingNew(view => view.Invoices.Remove(e.InvoiceId)); } Both View Projections and Event Receptors are intentionally made extremely simple and easy to change. This is required, since they are coupled to external bounded contexts which can be controlled by different teams with different level of agility and technological capabilities. Actual business behaviors and complex logic reside within Application Services and Tasks , which are safely isolated from the change shocks by projections and receptors. 3. Application Service Application Services are interfaces which are remotely accessible for calls within their bounded context. Normally these calls happen by sending a command towards one of these application services (this command can be either explicit command message, or it can be carried by infrastructure in form of remote procedure call). So each application service is just a set of command handling methods, which are executed when a specific message arrives. These command handlers are grouped together according to intent and design guidelines. Each one deals with one command and publishes events afterwards. It can call upon various service clients in the process as well. Examples of service clients that are usually used by command handlers: SQL client; NoSQL database client (e.g. key-value store with views that are projected within this bounded context); Connection to an Event Store; Integration endpoint with 3rd party; Business Logic client; Command sender to an application service within the same bounded context. Implementations of command handlers within application services can be different: Stateless processing of incoming commands into events (functional style). Handling commands by executing certain behaviors and calling domain services (e.g.: CRUD-style Aggregate Roots). Aggregate Roots with Event Sourcing (AR+ES) and Domain Services (favorite). Here’s an example code for AR+ES implementation: public class CustomerApplicationService { // domain services initialized from constructor IEventStore _store; IPricingService _pricing; IInvoiceCalculator _invoicing; // handler for command SettleCustomerInvoice public void When(SettleCustomerInvoice cmd) { DispatchAndMergeConflicts(cmd.Id, cust => cust.SettleInvoice(cmd.InvoiceId, _pricing, _invoicing)) } // skipped other command handlers // helper method that dispatches call to an aggregate root loaded // from event stream. If there were any concurrent changes, we’ll // check server changes for merge conflicts and try to rebase our changes void DispatchAndMergeConflicts(IIdentity id, Action<Customer> action) { while (true) { var stream = _store.LoadEventStream(id); var agg = new Customer(stream.Events); action(agg); try { _store.AppendToStream(id, stream.Version, agg.Changes); return; } catch (EventStoreConcurrencyException e) { // verify our changes for merge conflicts foreach (var clientEvent in agg.Changes) { foreach (var serverEvent in e.StoreEvents) { if (ConflictsWith(clientEvent, serverEvent)) throw new ConcurrencyException(e); } } // there are no conflicts and we can rebase _store.AppendToStream(id, e.StoreVersion, agg.Changes); } } } } 4. Task Task is the last element of a distributed system (a la Lokad). It essentially is a method call that is executed at specific moments in time. It can call upon various service clients (e.g.: query views or check up on 3rd party integration systems) and publish events. Task implementation is generally similar to application service, except for the trigger part. Here’s an example of task that continuously checks on list of invoices, detecting invoices that need additional action. var remindIn = TimeSpan.FromDays(15); while(!server.IsCancellationRequested) { var pending = _docs .Get<InvoiceDeliveryView>().Invoices.Values .Where(x => (x.CreatedOn + remindIn) < DateTime.UtcNow); foreach (var x in pending) { _events.Publish(new InvoicePossiblyExpired( x.AccountId, x.InvoiceId, x.CreatedOn)); } server.WaitHandle.WaitOne(TimeSpan.FromMinutes(15)); } Example In the snippet above, we actually handle piece of bigger invoice delivery and reminder process that would probably be implemented in bounded context, using all 4 elements: Event Receptor, View Projection, Application Service and Task. For instance, you can track invoices as part of customer application service . This application service would use customer repository and currency converter service as its dependencies in order to handle commands like: Create Invoice Add Payment To Invoice Expire Invoice etc Some of these commands will be sent to application service by receptors of bounded context in response to events: When Invoice Payment Arrived then tell customer application service to Add Payment To Invoice When Invoice Possibly Expired then tell customer application service to Expire Invoice On picture this would look like: We can also keep track of all open invoices by creating a projection for Outstanding Invoice View . This view will be used by Invoice Expiration Tracker task , which will once now and then rescan list to detect outstanding invoices that were created too much time ago. For each one, it will publish invoice possibly expired event. Purpose Purpose of all this separation is simple: to have a set of simple building blocks from which more complex systems can be composed in a reliable fashion that allows change without much friction and regressions. Rules of such composition should be rather straightforward and hopefully clear. I will be talking about them and implementation details in following materials. This approach is what I have arrived (so far) while working on various projects at Lokad . This is also what shapes future development of our small but rather interesting Software-as-a-service company. Even if it keeps on growing in complexity, I see no big problem in following up with the supporting infrastructure. After all, this has already been done by nature billions of times. As you have probably noticed, I keep on shamelessly borrowing concepts and approaches from her, because it is all well-documented and: Human subtlety will never devise an invention more beautiful, more simple or more direct than does nature because in her inventions nothing is lacking, and nothing is superfluous. Leonardo da Vinci Update This discussion continues in another blog post: Bird's-eye view of a Distributed System - Context Map , which tries to take into consideration real world environment around such system.


Behavioral Analytics

Behavioral Analytics What is Behavioral Analytics? Behavioral analytics focuses on how users behave, and why. This covers a wide range of applications, starting from online shops and up to connected devices in your smart home. Deep understanding of the behavior allows to optimize for future circumstances, serving customers better or optimizing the use of resources . There is a lot of good that can be done here. For example, a smart restaurant might use weekly visitor patterns, combined with information about reservations and some forecasts to plan for the daily cooking routine and product replenishment. Smart online store could provide personalized product recommendations, just like Amazon does (or do it even better). Smart houses can help to reduce energy consumption or warn on various dangerous situations (e.g. unexpected spike in energy consumption at night or unusual person entering from the window). Design Requirements Behavioral analytics is one of these areas where multiple interesting software requirements are present at the same time: responsive and high-load (we don't want to degrade performance during the holiday shopping rush); personalized (we need to track events at level of a single user); large scale data mining is required, since we have to process all the accumulated date in order to create new models, verify them and track their performance); user interactions have to be captured and quantified properly in order to have some data to process. All this might sound high-tech and complicated, however it is quite easy to start approaching such problems. Devil is still in the details, though. It all starts with domain events. Role of Domain Events User interactions with a system could be represented as a sequence of events in a stream. For example, in online store we could have such sequence: {{% img src="events.jpeg" %}} The methodology of capturing such events is well-known and documented around the DDD related articles: event-storming, domain modeling sessions with experts, event-driven use cases etc. The story of 👍 HappyPancake covers all benefits of that in great detail. If a system emits events, these could be captured and persisted in an event store, suitable for further processing and integration. Towards a Deeper Insight With event store, it is quite easy to start using events for greater benefit (aside from integrating different modules and 3rd party systems). First of all, we could to set up a dashboard with various reports derived from such events and updated in real-time. Or easier yet - we could project events to some OLAP system and let managers slice and dice the data as they see fit. Even Excel has such capabilities. {{% img src="system.jpeg" %}} Given this initial insight, domain experts could come up with interesting ideas for improving user experience and getting more revenue out of the system. Then, we could run batch processing across the event streams to verify new theories or fine-tune existing models. These could later be expressed in form of rules that will run in real-time and interact with users, reacting to their behaviors. Back in my economics R&D days I used to run evolutionary algorithms across historical datasets to capture dependencies between available data. This information was used to remove less relevant data from datasets and build refined models. They were used for forecasting and running various "what if" scenarios. Not only we could execute various rules in real-time (with extremely low latency), but we could adjust their behavior and verify them through A/B testing . This approach, perfected by Amazon, involves splitting user base into multiple groups and giving each group slightly different experience. Nature of the experiment is recorded along with the user behavior captured in events. These are later compared to pick the most efficient approach. With A/B testing you could verify various theories: What is the best location of an advertisement on a page? Which promotion offers for users buying product A drive more sales? Which room brightness is the most comfortable for people in the evening? Evolutionary process of iterating over theories, models and real-time rules is a reflection of a usual software development process to a slightly different field. The purpose of this process is still the same though: gain deeper insight into the domain . One might even claim that the principles of domain-driven design could still apply here. To be continued Behavioral analytics and its application to various aspects of our life is something that interested me for a long time. I hope to continue this topic in a series of posts. If you have some comments, insights or interesting ideas to share, please don't hesitate to get in touch.


Behind the Scenes of DDD/ES Project - Salescast2 A few posts back I dropped a screenshot of Visual Studio solution structure for the Salescast2 of Lokad. Jonathan Oliver (author of EventStore and also partner in crime in the distributed podcast ) immediately left following comment: Ouch! That's a lot of projects. Granted, it's not as many as NServiceBus, but the pain in .NET land I'm feeling right now comes from compile times and JIT startup times on anything over 3-4 projects. I personally, I wouldn't consider Salescast2 solution to be really large, given what this baby can do. So let's do a quick tour, shall we? Salescast2 is a platform for pulling diverse sales data from retail networks (lots of it), performing some rather smart data processing (that's what we specialize in at Lokad), producing business intelligence reports and passing resulting reports back. Here's a case study and product page about the previous version of this platform. This sounds like something simple, so far. However, there is more. We pull data either via custom adapters or rather flexible database schema which could be located on Oracle, mySQL, MS SQL or PostgreSQL server. Each of these databases can have not only rather different setup (e.g. different columns for inventory products), but also different performance. For instance compare clustered Oracle setup versus some tiny mySQL database running in a shared environment. Obviously, Salescast2 should be able to automatically identify type of the database, it's configuration and performance capabilities, adjusting to all that . There are a lot of things that can go wrong. Some of these are critical (products table does not exist), while some are less important (exports table exists but is not writeable). Problems should be handled automatically if possible , exposing helpful information about the nature of information if something goes wrong. Printing exception stack is not enough. Solving all these problems was achieved via use of Event Sourcing applied to Domain-Driven Design . Granted, it was not easy, but fortunately we had our domain model established in the first version. Server-side functionality for all that is hosted in 5 projects: Contracts - message and persistence contracts, language of our system; Domain.BC - actual code that coordinates everything, starting from discovering projects and up to generating excel reports; Adapter.BC - messy code of SQL integration; Wires - some infrastructure-level implementations (e.g. sending emails or persisting documents); Worker - console to run app server locally, it also acts as a worker role that can run on Windows Azure cloud. That's it. This is the only project-specific code of the application server that runs the system . All the rest is less important. However database integration is only part of the big picture. This server code does much more . We also need to aggregate data, clean it up, run some analysis, calculate some formulas, start even more complex process of forecasting, retreive results, run additional formulas, arrange available data and skip missing bits, producing nice Excel reports or exporting data back to customer's database. Fortunately, during this process we don't need to deal with the most complicated part of analytics (producing classical and quantile forecasts). This aspect (and core of our technology ) is handled by a separate set of systems. Still, while doing all these calculations, we need to handle relatively big data . 300k products and 80m sales entries were an initial target for a single project (customer can have any number of projects). AFAIK, that's more than 500MB of data if stored in compressed format. We achieved that, and also done a bit more: This dataset can be processed on a single thread on a laptop under one hour (old version used to require day and a large server), obviously in production we can use as many threads as we need. data processing (e.g. map reduce) use efficient streaming operations that require rather small amount of memory. entire system can natively run either on a file system without ANY external dependences except .NET (xcopy deployment is fine) or it can be deployed to Windows Azure, using efficiently it's capabilities. One of the reasons behind that is - SQL server and especially Microsoft Azure Stack is horribly slow on my tiny Mac Book Air, which has to run a bunch of Visual Studios in a VM with 1.5GB of RAM. So they had to go, in order to provide comfortable development experience. Nobody regrets the decision. In order to achieve that, we used some of the old storage abstractions from Lokad.CQRS and also enhanced them with a few new ones (to be ported to open source). Obviously, this is still just a tip of the iceberg. This server, in addition to pure processing and business logic, also hosts some code for Web UI projects, providing denormalized read models (views), which are convenient to bind to ASP.NET MVC views. This server tracks any changes in it's own code and rebuilds views automatically, should it be required. It also accepts commands from various clients and provides additional integration capabilities with 2 more systems. These Web UI clients on their own are quite simple (since all complicated logic is done on the app server). They add a few more projects to the solution: Admin.BC - projections and read models for the Admin UI Admin.Web - Admin UI with ASP.NET MVC Client.BC and Client.Web - idem for Client UI Both Web UI projects, obviously, display relatively nicely both on desktop browsers and on mobile devices. Albeit, UI Design itself is not as nice as it could've been (if it'll be worth it - we could just hire a PHP or Ruby team to redo that UI in a few weeks). Rest of the projects in this solution is just support: Cqrs.Portable and Cqrs.Azure - copies of Lokad.CQRS source code to provide actual infrastructure capable of doing big data, cloud integration and all the other things; Dsl - helper util to generate command and event classes on-the-fly ; Admin.Deploy , Worker.Deploy , Client.Deploy - just deployment projects for Azure Audit - a copy of event stream viewer from Lokad.CQRS a bunch of test projects (not enough). Obviously, there is a bit more: Capability for zero downtime upgrades; Capability for immediate backup and replication of data (event streams make it easy); Almost-infinite scalability (processing partitioning is dead-easy here and storage is partitioned by default); Low-friction evolution - it is easy to add new behaviors or functionality to application server; UI changes are never a problem - server will rebuild the views as needed (and all queries take just one round-trip to Azure storage); Any server-side failure can be easily reproduced most of the time, including complete state of the corresponding business object at that moment; Writing and evolving long-running business processes is marginally more complex than dealing with projections (and these are handled automatically); Ability for fail-over between the clouds and availability zones (read-only mode and full functionality). That was something I really missed for Salescast on February 29th. So, all in all, I think this big-data BI platform has some decent capabilities packed in a rather small and simple project (even though it includes it's own infrastructure and tooling in source code). As you probably already know, we hate friction and unnecessary complexity at Lokad . Knowledge and experience learned during it's development was pushed in A+ES chapter in Vaughn's IDDD book (chapter is still not visible on Safar, yet), Lokad IDDD Sample and is being used to gradually improve Lokad DDD/ES Sample Project . So far I'm pretty happy with how this little project went (it will be released to public soon). Massive kudos, respects and thanks to Lokad team for pushing effort in this direction. Plus all the people behind CQRS/DDD and ES communities. What to do next? If you are developer interested in this technology - please feel free to do what you want with the materials from this blog on Salescast or source code from any of Lokad Sample Projects (no stupid restrictions apply). We'll be updating in the upcoming months to share some more code and practices. Don't hesitate to ask questions and share problems. Hopefully this little bit of sharing could help you to move forward personally. If you are an existing customer or developer with some Lokad-specific questions - please don't hesitate to drop me an email. I'm especially interested in cases, when some stupid big-data query takes hours to compute or when you need to budget millions of euros just to store and process your data. The latter is a huge waste of resources, you know (probably it all fits on a single laptop or even a smartphone). If you are a Lokad team member - we're just getting started :) By the way, do you know what was the most difficult part along this journey? There were two problems: Actually understanding what this product really has to do and how (Domain-driven design helped here); Stepping away from all these widely-accepted technologies, practices, databases and service buses; then unlearning them and doing something much simpler. This journey was clearly worth all the effort. And if I can do a single one thing, I would like you to encourage you to try a similar one . This does not need to be same stack or set of approaches, but just exploring new directions and sharing results. On such journey not only can you find something new for yourself, but you can also serve the greater good of helping to push forward the community. That's how we all learn, move forward and push the state of the art together.


Being the Worst Updates

Being the Worst Updates This is an update post to being the worst podcast on software design, which we started with Kerry Street at the end of this summer. A few things have changed since then. Module 1 (or Season 1, if you wish) is almost finished . In the second one we will switch to slightly more complicated domain (Factory is too simple). Further episodes will base on previously covered material with additional focus on: Production experience of a SaaS project run by one man or a small team; Collaboration between team members (or with external outsourcing parties); Collaboration between multiple sub-domains; integration with external systems; Deeper level of Domain-Driven Design; Patterns of Client UI development (including mobile clients, web UI). We'll see how it goes. So far, we are still the very worst and excited about staying this way. Growing community and support it provides - encourage us to keep moving forward personally while sharing the lessons learned: 250 subscribers milestone reached and still counting. Tonight I plan to record first episode of Being The Worst in Russian with Anton Vinogradenko . It will mirror english version, benefiting from its (relatively) coherent viewpoint, heavily commented samples and all additional reference implementations that will be added later. Tom Janssens contributed sample code in erlang for one of our episodes. There is work in community to provide Java equivalent for that, as well. Sharing knowledge became even deeper part of Lokad approaches (which are presented in this podcast). The same materials are reused in development training. Additionally our new Lokad Data Platform initiative shares theoretical foundation with BTW podcast. It is currently being introduced to largest retail companies in Europe as a way to enable high-speed data integration between formerly locked systems. Approaches and experience behind Data Platform will be covered in later modules of "Being The Worst" along with all these "big data", "real-time", "cloud computing" and "business intelligence" topics. Besides, I'm personally just curious, what would happen to the development world if some educational equivalent of "Advanced Distributed Systems Course with DDD and Event Sourcing" would be made available for free to the community (along with coherent set of additional study materials, assignments and samples). This can be an empowering social elevator for the people from undeveloped regions or poor families . What do you think about the podcast so far?


Bird's-eye view of a Distributed System - Context Map In one of my previous posts on Anatomy of Distributed System à la Lokad we were zooming into the design of a system that could be built as a part of SaaS company. Such system was built with and around CQRS/ES concepts. This approach looks like a nice by-the-book design: bounded contexts that are based on a uniform design and connected together to deliver some functionality. This is the kind of drawing that could be produced by an ivory tower architect or a consultant that is "too expensive to ever touch the codebase". However as we know, nothing exists in isolation from the real world . Every entity (either biological or digital) is always connected to much larger environment (and is affected by it). In software projects, for instance, you would have other departments to cooperate with, customers to serve and partners to integrate with. You might also have Microsoft and Apple OS to curse about. Even political rivalry between departments A and B in an organization could matter (especially if they share a database). So there is an immensely complex and diverse ecosystem that surrounds every single software project . There are different technologies, languages, concepts, mentalities and resource constraints. These things do not necessarily belong to the project directly, but could have an extremely strong impact on risks and costs, potentially leading to the difference between success and failure . Good software developer will always consider the most important of these strategic factors, while making tactical decisions. One of the ways to take some of these factors into consideration is by changing perspective of your system and by looking from bird's-eye view at it and other systems that it is linked to. Let's zoom out of our CQRS/ES-based set of systems in a department (which was discussed in Anatomy post ) and look at a bigger picture. We'll include other departments, some of the customers and partners. This view will be based on "Context Map" approach of DDD but slightly expanded to cover ecosystem within and around the company. I will keep CQRS/ES-based bounded contexts marked with orange border, while "external" ones will have borders of other colors (just like countries on a map). Ok, now that's a subtle change. We get a lot more variety in colors and shades, which represent differences: outside elements have technologies and context specifics that differ greatly from ours; our own elements turn out to have different shades of CQRS/ES implementations as well. In other words, each bounded context is different. These differences can be caused by a huge amount of internal and external factors. For example, consider these two contexts, which happen to be owned by different organizations: Obviously, these differences would make different approaches more or less successful than the others . And it actually is perfectly OK to have different approaches and technologies living in the same organization or a project, if this is justified by the environment. Unless, of course you are a developer from a Soviet Russia that has to employ absolutely identical set of tools everywhere (i.e. SQL everywhere, NHibernate everywhere, NServiceBus everywhere etc). A few vivid examples of ignoring the strategic context, while charging into the battle were clearly demonstrated by Napoleon and Hitler. They came over to visit Russia and stayed over for the winter with such extended and fragile supply lines. So, please, don't repeat mistakes already covered by history books (these are the most shameful ones) and: Take into consideration real world that surrounds this project, trends and risks that surround in future. Split your environment into set of bounded contexts, that could be distinguished by similar language, match with organizational boundaries or technology manifestations. Map the terrain to make it more understandable. Before considering to apply CQRS/ES (or any other tech) to a given bounded context - consider and compare it with other approaches , their risks and costs. Delay non-essential decisions as much as possible. Push for approaches that allow you to capture feedback from real life as fast as possible (lower development friction, faster iterations, simple architectures). Keep it simple and separated (KISS). Let me reiterate. Domain-Driven Design with its approach of bounded contexts that can be joined into a strategic context map - is one of the ways to visualize and represent environment surrounding and affecting a project. This approach can help to take into account all important factors and figure out the tactical details of most implementation, architecture and technology. Sometimes CQRS/ES approach would be the best fit, however more frequently you would have a different set of conditions that require a different solution to the challenge ahead. Pick your weapons wisely, for they might affect the outcome of the battle and war.


Better performance with DOD

Better performance with DOD Time flies! Black Friday Experiment has been running for 4 months, according to my git commit history. When I was starting, I wanted to scratch my itch and try improving on the past event-sourced designs. I'm quite happy with the code performance so far. It is faster than the LMDB-based inventory engine I used to work on in the past. In 3 minutes I can replay a story of a growing company: 500k nested locations, 498k SKUs, 690k orders, more than a million entities and events. All orders are resolved against strongly consistent availability engine. 1 thread, running in a local-only mode without remote event store replication. With EC replication, the speed will be the same. If we go consistent, then the latency/throughput will depend on the streaming/storage engine setup. Curiously enough, this exercise quickly moved from " exercise in event sourcing " to an " exercise in writing inventory constraint solver engine ". In order to make things fast, I had to finally start thinking about how data is laid out in the memory and how CPU cores go through that . If you think of it, classical Object-Oriented-Programming (OOP) lays its data in memory like Spaghetti bolognese - objects and pointers are all mixed up. This is fine until you want to do some intensive computations with your data. For example, detecting early, if moving items between two warehouses will negatively affect any pending orders. Given that you have a few hundred thousand locations and products already in the system. If your object graph is done in OOP, then CPU pre-fetcher will have jump back and forth between random memory addresses. Performance is probably the reason why deep neural networks are never implemented in OOP style . If each neuron and axon were represented as a distinct object with its fields, then exact memory location of a given neuron could be non-deterministic. Instead, neural networks are represented as arrays, matrices and tensors. This happens to be more natural and fast for CPU or GPU to work with. Fortunately, there is a good old programming approach called Data-Oriented Design . It says: " Dude, do whatever you want, just structure your data to use CPU efficiently ". One approach for that is about laying out data in continuous blocks of memory , so that it will be easy for CPU to prefetch all required data and just chew up through it. Just refactoring the codebase to achieve that was good enough for the performance gains I mentioned earlier: group all inventory locations for a given product close in memory; represent them as structs and arrays. // ProductStock captures stock quantities and reservations // of a single product across all locations and warehouses. // it represents a tree that is flat in memory. The root // always contains totals. // We can easily fit stock availability for 100 products in CPU cache // allowing us to resolve availability or run kit calculations type ProductStock struct { // location IDs in the system locs [] int32 // pointer to the parent location. -1 for root parentIdx [] int16 // reservation and availability counts reserved [] int32 onHand [] int32 } This means: whenever the code execution touches any stock, then all locations for that product are probably already in the CPU cache. So calculations across multiple locations will be cheaper. Main memory reference is as expensive as 20x L2 cache references or 200x L1 cache references. The impact of the resulting code is interesting - I need to maintain event stream, durable aggregate state and in-memory state that is follows data-oriented design. The complexity is manageable by treating in-memory objects as immutable. All of these states are considered to be transient - only the event stream matters . However for the sake of performance we need to cache things close to the processor. Always replaying the event stream will be horribly slow. There could be more interesting patterns to play with, like Entity-Component-System from the game development. But current implementation is good enough. No new features are going to be added to the codebase. I just need to make existing specs pass. Ultimately the final version of specs and a spec runner will be made open-source as a part of Trustbit katas. As for my Black Friday implementation - I'll keep it closed source for now. The domain is a toy project, but the engine, tooling and infrastructure are good enough for production and simple enough for a full handover to another team. Perhaps it will come in handy one day. Next : NixOS, the power of declarative VMs and per-project system dependencies .


Black Friday Experiment - Report 1

Black Friday Experiment - Report 1 This is a first report on the Black Friday experiment : let's rebuild and improve event-sourcing designs that I have worked on in the past decade. about the experiment; which features I wanted to keep and introduce; how the code looks like (features and specs); performance; simulation; tooling and development experience; possible next steps. About the experiment Black Friday is a special day in a retail business - a great deal of yearly revenue is made on that day . It also is a challenging day for engineers and operations. There is lot of stress on sales and inventory systems . Load goes up, cloud providers go down, things go haywire. This uncovers weird bugs, breaks assumptions, and teaches a lesson about cascading failures (under the Black Friday load failures never happen alone). In this world, upgrades have to happen with zero downtime under a sustainable load . So I wanted to see if I could design an event-sourced system to handle the Black Friday load while being simple to maintain and extend . At least, more simple than what we had achieved at SkuVault . Another way to look at the experiment - is about trying to scale event sourcing implementation to the extreme in performance, capabilities, and ease of supporting multiple engineering teams. From this perspective, developer tooling and experience become very important . In the recent past, I've worked on projects where finding and onboarding new engineers is a difficult process. This could relate to the domain (data science and machine learning in international logistics), but some of the lessons apply well to any software project. Picking the battles Here are the features that I want to keep from my past experiences: Event-sourcing - because that simplifies development and zero-downtime upgrades under sustainable load. Event-driven specifications to define and manage features that are linked to the user stories. Fast test runs - being able to run 1000s of specs per second on a local machine. Simulated development environments - be able to run a server or the whole cluster locally, if needed. High availability - killing any single server (Load balancer, application, or database) shouldn't bring the system down; Zero downtime for any operations. Here are the features I want to introduce: Tenant isolation . Each tenant has to run in its own tiny VM, while allowing to pack thousands of tenants on a single node. Previously: multiple tenants from a single partition would share a process. Design for live tenant migration and resource allocation . Previously - tenants were fixed to their partition. All tenants in a partition shared CPU, RAM, and storage. Use golang language - it is easy to pick up, has fast build times, and compiles to a single static binary that can be easily deployed. Previously: C# .NET Use gRPC to define APIs and contracts - because there is so much tooling around that. Previously: code-first contracts and custom IDL for service definitions. Use SQLite for the aggregate state - because there is so much tooling and ongoing work. Previously: LMDB database which has great performance but a smaller ecosystem. Design for event store compaction , since I want to keep primary event streams small and enable fast schema migrations. Previously: event streams were continuously growing without a bound. Enable per-tenant customizations , to reduce overall system complexity. Previously: features were developed and deployed for everybody, even if they were needed only for a few customers. Enable per-tenant encryption and encrypt all data (in flight or on disk) with a tenant-specific encryption key. Codebase so far At high level, the architecture looks like this: There is a single-tenant application that contains a single event-sourced aggregate. It is possible to have multiple tenants per application, multiple aggregates, too. However, this increases code complexity and discards some nice capabilities. At this point I just want to see how far "one event sourced aggregate per tenant" can take me. As you can see the tenant has two interfaces: Request/Response interface for API and Web. Publishing events to the event stream for replication (also loading these events by replicas). That interface would be my testing boundary for features. I have re-implemented specification testing framework in golang. Specifications follow common "Given events - When request - Then expect outcome and events". Check out this section on Event Sourcing for more details. Specifications test aggregate logic and look like this: When specs run, they produce an output like that on failure: I started with specifications because they are more important than the actual implementation. They define the behaviors of a service, capture edge cases, and prevent regressions. If you have good tests, then you could reliably refactor the actual implementation. You could merge, split or even completely rewrite actual codebase. See High Availability and Performance for more details. There is a portable demonstration of that approach on Replit. It contains a subset of specs and a spec runner. Last two specs are intentionally broken. The unit of development in this approach is not an aggregate, but a feature. Features could relate to specific Web/API methods (e.g.: create location, move location, reserve inventory) or be more fine-grained (e.g.: run custom script when reserving stock for an incoming order). It all depends on user stories. I usually keep one feature in a single file , with a separate specifications file nearby. That file would normally be a single method that covers everything from API/Web to DB access to event generation and application. Let's take a look at a simple feature. It is an API method that allows to move location to any new location. For example, we could have a incoming container that is being moved to the unloading zone in a warehouse. All logic happens within a single transaction that is managed by the framework. Within that transaction we are free to: Query the database . Queries are extremely cheap and generally take microseconds each. Make decisions and generate new events . Events will be applied to the database immediately, also taking microseconds. Return with error at any point . This will abort the transaction and roll the state back. Return a valid response . Framework will take care of appending events to the event store and completing the transaction. Feature-oriented project layout works surprisingly well. It is a relief from projects when adding a single field required changing half a dozen files. Or a dozen, if that involved entities living in different aggregates. In order to see how well this approach handles business complexity, I'm adding features that weren't on the table before (in my past projects): tenant-specific scripts; flexible inventory location structure (e.g. warehouse-bin or region-warehouse-zone-row-shelf-bin with movable containers and palettes); ability to reserve stock at any given location level (this reduces item availability); ability for tenants to create inventory reports that provide fast and consistent view into products quantities at predefined locations (e.g. by warehouse or in all containers-in-transit). We'll see where does this break. This is how custom scripts look like , by the way. They are installed in aggregate via a normal event, which allowes them to be testable without any extra scaffolding. Performance Test performance is good. I can run more than 10 thousand specs per second (using perf command to run existing specs in a loop): > make && bin / bf perf - sec 5 Matched 30 specs out of 30 Speed test with 4 core ( s ) for 5 s OPERATION TOTAL OPS / SEC KHZ SEC PER OP run spec 64050 12810 12 . 8 78 . 064 µ s apply event 172935 23716 23 . 7 42 . 164 µ s executed 64050 specs This also includes specs that involve custom tenant-specific scripts in Lua. That logic has a bit of overhead to start a new Lua VM, load its state, run custom scripts, and return control to the API call. Tests run on an empty database (using a combination of :memory: , WAL, and transactions to keep things fast), they don't indicate overall performance in a real world. To keep development grounded in the reality, I started building tooling for the stress tests. Here are the first results of a simple simulation scenario run with a stress command. Numbers are a testament to the performance and efficiency of golang and SQLite . > make && bin/bf stress Start simulation Setup simulated network Dialing 'sim' Serving on 'sim' DURATION DB SIZE LOCATIONS PRODUCTS ON-HAND RESERVED 432 ms 4.4 MB 511 1000 4934 16109 422 ms 4.5 MB 1022 2000 9486 32597 424 ms 4.7 MB 1533 3000 14116 48624 424 ms 4.9 MB 2044 4000 18717 64874 430 ms 5.1 MB 2555 5000 23356 81562 433 ms 5.3 MB 3066 6000 28643 98009 429 ms 5.5 MB 3577 7000 33492 114569 426 ms 5.7 MB 4088 8000 38785 130953 428 ms 5.8 MB 4599 9000 43605 147313 435 ms 6.0 MB 5110 10000 48865 164267 436 ms 6.2 MB 5621 11000 53694 180734 431 ms 6.4 MB 6132 12000 58368 196699 477 ms 6.6 MB 6643 13000 63468 213129 509 ms 6.8 MB 7154 14000 68682 229472 527 ms 7.0 MB 7665 15000 74143 245777 452 ms 7.2 MB 8176 16000 79378 261941 452 ms 7.4 MB 8687 17000 84434 278851 459 ms 7.6 MB 9198 18000 89124 295007 493 ms 7.7 MB 9709 19000 93776 311512 559 ms 8.0 MB 10220 20000 98624 328051 Simulation Stress test tool above uses simulated network connection. This is possible because gRPC SDK in golang relies on net.Listener interface: // A Listener is a generic network listener for stream-oriented protocols. type Listener interface { // Accept waits for and returns the next connection to the listener. Accept () ( Conn , error ) // Close closes the listener. Close () error // Addr returns the listener's network address. Addr () Addr } Even an onion could match this interface, if it knew how to return an instance of net.Pipe() (synchronous, in-memory, full duplex network connection). net.Pipe() allows to test the whole stack without actually touching network devices, until needed. This is as far as the simulation goes in this project, though. I couldn't figure how to control golang scheduler and simulate the passing of time, like it was possible in .NET (see Simulate CQRS/ES Cluster ). Tooling and development experience While specs are enough for the most of the development, in some edge cases I need more insight into application state . explore command helps here, it runs a single test specification, exporting the final state into a standalone database. This is good for fixing weird bugs, tuning queries or even developing new features. After starting to use this command, I noticed a nice side-effect. GoLand picked up on the schema (because I used it to run some queries) and started intelligently completing and highlighting inline SQL. Speaking of the development experience, the full cycle of building the code and running tests isn't that fast . It takes roughly 3 seconds on my laptop (Intel Mac Book) for that. I blame my CGO dependency for that (SQLite). Running go install -v doesn't help. > time make build test make build test 2.96s user 0.88s system 112% cpu 3.427 total I also took a look at running self-hosted server (with gitea ). It had server-side post-commit hook that would trigger the build inside a VM and send any failures to Telegram. So whenever I git push , there will be a success or failure notification arriving within 3 seconds. It was an interesting experience. Next steps I have a few ideas of where to take this next. Below is a list without priorities. There are too many ideas, and only one or two will get a follow-up. Switch the spec runner to work through the simulated gRPC interface. This will let to compare multiple different implementations of the same features, even if they are written in different languages. Christian Folie is looking into F# implementation! Explore adding more domains to the same codebase (JIRA, ML Pipelines library, Community blog with comments). Recover experiments with Terraform and FoundationDB to deploy golang codebase in a cloud environment against a clustered event store. Launch stress command and then start manually killing nodes at random. Continue exploration of FirecrackerVM (it is awesome) to streamline the process of building application as a bootable VM image. Then check out, what it takes to have a few hundred of them on a single server. What would the CI/CD process with stages look like? Improve the performance a bit more: prepare and cache SQL queries, switch Lua VM to use native LuaJIT, and play with request batching (trades off latency for throughput). Add web UI and web specs, so that features are more realistic. Set up a proper demo user map with an issue tracker and CI/CD cycle, then bring over a few friends to see what they think of the development experience. I'm personally interested in the FirecrackerVM line of research. What do you think?


Bounded Context is a Team Working Together Let's talk about architecture-level decisions that drive development and evolution of systems. As Stacy noticed recently: I notice your architecture is at the BC level. I certainly can see some advantages. But I wonder about things growing into a big ball of mud as requirements dictate more aggregates or larger ones. By designing at the aggregate root level, things stay small an agile as the business requirements grow and change. Although the tradeoff is more message chatter between them. Just wondering what drove you to the BC's as components, rather than aggregate roots? I used to focus just on aggregates (taking the route of "your aggregate is your bounded context"). This looked plausible in theory, but it didn't work for me in practice that well. Reason behind that - Aggregates are the most important building block of my systems, since they capture core business concepts and deal with complex persistence tasks (via event sourcing). However, focusing only on aggregates can lead to less efficient designs , that lack some synergies and capabilites. Such approach encourages putting too much responsibility in one place, which significantly raises complexity and cost of change (if you seen recent "Avengers" movie, you have an idea of what happens, when you put a team out of outstanding heroes - most of the time they argue and deliver collateral damage). Besides, this actually leads to rather chatty implementations. Aggregates, no matter how significant they are, still need to cooperate and exchange information (e.g. consider the case with "Customer" and "Invoice" aggregates, where old invoice has to be cancelled, whenever an override invoice is issued for the same customer). Solution to that problem of mine was to start thinking in terms of bounded contexts, rather than individual aggregates . Bounded Context (BC) is part of a domain model, unified by a common ubiquituous language, shared and connected concepts. It is like an organ of a human or a plant - composed from multiple separate cells, but with distinct boundaries (or, you can consider BC to be a country, if you prefer my war analogies ). As such, Bounded Context consists of multiple distinct building blocks that work together to simplify achieving common goals of capturing core business concepts (and aggregates are one of these building blocks). Implementation-wise, Bounded Context is a highly specialized team of focused soldiers (e.g. A-team ) that work together. They tend to be deployed together as well (which aligns them well with the development and operations). Software solution can be composed rom multiple bounded contexts. Each arcitecture style has it's own set of building blocks. Common guidelines for picking them are: blocks have to be relatively simple and coherent on their own in theory; development, management and evolution of this building blocks should be practical . In the case of doing Domain-Driven Design a la Lokad , we have arrived to the list of building blocks: Application Services - groups of command handlers, defined on a single class. Most of the time, these command handlers are responsible for loading aggregates and executing their methods (although, other uses are quite common, as well). Aggregates implemented with event sourcing ; they capture behaviors and core business concepts in an expressive way that frees us from bothering about complex persistence, its testing and versioning. View Projections - event handling classes that denormalize events into persistent read models called "views". These views are complextely disposable and can be automatically rebuilt by server, if corresponding projection code changes. They are used for bringing information together in a way, that is easy to consume by: aggregates, clients, tasks etc. Event Ports - simple event handling classes that declare event subscriptions and act as ACL (previously called "Event Receptors", but switched the name to match "Ports and Adapters" by Alistair Cockburn). They make boundaries and relations - explicit. Tasks - long-running processes (in essence, just a while(server.IsRunning) { DoSomething(); Sleep(server, 5.Mins()); } ), that allow the system to actively invoke actions; keep track of timeouts and schedule actions. In practice, not every bounded context has all of these blocks (for example, my client BCs tend to have only view projections). However, having them at hand (along with the available practices of development, deployment and operations) is what makes a lot of scenarios possible: starting from simple SaaS billing system and up to self-diagnosting big-data platform for integration (which integrates directly via databases with flexible schema) and business intelligence. Please, keep in mind, that this choice of building blocks is by no means something final, officially approved, certified (hell, I'm merely bringing together and exposing knowledge of more experienced practitioners). So it is just a selection that has (so far) proven to work the best in Lokad and some other small result-oriented companies. These are the companies, where software designers and architects have wear the burden of responsibility about their decisions and choices for years. This is exactly the environment, where operations crush and crumble nice theoretical ideas, forging some other solutions that ain't that pretty. Yet, in practice they work. Each block has a whole story behind. I hope eventually to share it (dude called Vaughn Vernon even strongly suggests on writing a separate book on that). However, what matters the most right now - this is minimal set of blocks that we would currently need to build a system in a case, which requires application of Domain-Driven Design (with or without capabilities for cloud deployments and big data processing). More is certainly possible, but that would be less. This approach to structuring Bounded Contexts would certainly evolve and change, as simpler approaches are found and new tools show up in this area of knowledge (some of the future improvements in development landscape are already accounted for). Takeaway Aggregates are just one of the building blocks for expressing Bounded Contexts in the code. Aggregates implemented with event sourcing shine in capturing rich behaviors that might require both complex persistence AND rapid evolution of underlying schema. Domain-Driven Design and use of other building blocks is an essential prerequisite for this approach to succeed.


Building Blocks in CQRS world à la Lokad Once you have broken your systems into separate subsystems (bounded contexts) and introduced Command-Query Responsibility Segregation inside each one, there are multiple paths to take from there. Each path depends on your background, familiar technologies and available developers. Two Primary View Points I think that all views on building distributed systems in .NET world can be organized in two groups. Developers from Udi Dahan/NServiceBus world tend to structure systems with CRUD aggregates (persisted in SQL or RavenDB) and SOA Services, implement business workflows with sagas that look almost like aggregates. Choice of persistence usually is restricted to transactional systems (MSMQ being the star here). There is a smart usage of caching and internet infrastructure to deal with scaling. People with preference for Greg's ideas tend to express business behaviors with event-sourced aggregates, avoid use of "orchestration" term and structure flows with either state machines or stateless document sagas. Views are usually rebuilt by replaying events. Choice of persistence does not matter and transactions are much less important. à la Lokad I'm personally closer to Greg's side, but a lot more constrained (startup environment and hybrid hosting environments took their tall). Primary differences (for the latest projects) is that I do not use any sagas (there is no such term) and there are no future messages. Let's highlight primary domain building blocks that are used in post-CQRS world according to Lokad: Aggregate Roots Workflows Projections Domain Services Processes Aggregate Roots Aggregate Roots with event sourcing (or AR+ES) - have complex behaviors, are easily tested and persistence - ignorant. They serve as consistency boundary and are partitionable by Id. When command arrives to the server, it is passed to the command handler, which loads the aggregate root (and any relevant services) and executes an action against that aggregate root. That action results in events that both change state of the aggregate and are published as messages to all subscribers. One of the simple mental patterns of implementing AR+ES entities revolves about splitting state and behaviors in two distinct classes: aggregate state and behavior. Aggregate State class contains structural representation of AR+ES, that can be mutated only by applying events to it. public class CustomerAggregateState { public CustomerAggregateState(IEnumerable<IEvent<IIdentity>> events) { foreach (var @event in events) { Apply(@event); } } public int Version { get; private set; } public bool ConsumptionLocked { get; private set; } public void Apply(IEvent<IIdentity> e) { RedirectToWhen.InvokeEventOptional(this, e); Version += 1; } public void When(CustomerLocked e) { ConsumptionLocked = true; } public void When(CustomerUnlocked e) { ConsumptionLocked = false; } // ETC... } Actual aggregate class contains behaviors that are usually executed in response to commands sent. In order to carry them out, aggregate uses it’s own state and any available domain services. Any changes are passed down to state as events. They will also be saved in unit of work, then - committed to event store (which will publish them afterwards). public class CustomerAggregate { Action<IEvent<CustomerId>> _unitOfWork; CustomerAggregateState _state; public void LockCustomerForAccountOverdraft(IPricingModel pricing) { if (_state.ManualBilling) return; var balance = pricing.GetPaymentThreshold(_state.Currency).Convert(d => -d); if (_state.Balance > balance) { Context.Explain("Balance {0} is above threshold of {1}. Don't lock", _state.Balance, balance); } else { LockCustomer("Overdraft"); } } public void LockCustomer(string reason) { if (_state.ConsumptionLocked) return; Apply(new CustomerLocked(_state.Id, reason)); } // ETC void Apply(IEvent<CustomerId> e) { _state.Apply(e); _unitOfWork(e); } } Command handler deals with bringing all this together: /// This command handler can be replaced by a set of lambdas public class CustomerHandler { IAggregateStore<CustomerId, CustomerAggregate> _store; IPricingModel _pricing; public void When(LockCustomer c) { _store.Update(c.Id, a => a.LockCustomer(c.Reason)); } public void When(LockCustomerForAccountOverdraft c) { _store.Update(c.Id, ar => ar.LockCustomerForAccountOverdraft(_pricing)); } // ETC... } Domain Services IPricingModel from the code above is actually a sample of domain service . It is acquired by command handler and then passed to aggregate root to provide all sorts of rich functionality. If Aggregate Roots (with their complex behaviors and advanced structural persistence) are brains, then domain services are actually the muscles. Here are some samples: index lookups; pricing calculators; mail messaging; integration with payment systems. Workflows Workflows are the corner stones for interactions in our bounded contexts (or sub-systems). They subscribe to all sorts of events that happen in the environment both outside and inside the bounded context and define immediate reactions to these in form of commands that will be sent to players inside. public sealed class BillingWorkflow { readonly IFunctionalFlow _flow; public BillingWorkflow(IFunctionalFlow flow) { _flow = flow; } public void When(CustomerBillChargeAdded e) { _flow.ToCustomer(new WriteCustomerInvoiceForBill(e.Id, e.BillId)); } public void When(CustomerInvoiceWritten e) { _flow.ToCustomer(new RequestCustomerInvoicePayment(e.Id, e.InvoiceId)); } public void When(InvoicePaymentReceived e) { _flow.ToCustomer(new AddCustomerInvoicePayment(e.CustomerId, e.InvoiceId, e.GrossAmount, e.PaymentCode, e.PaymentId)); } // etc This code is extremely simple and serves one and one purpose alone - to explicitly define events that this bounded context reacts to. These reactions will then be carried out by our command handlers which will load appropriate aggregate roots (brains) and let them operate domain services (muscles), while remembering what happened for future generations. Explicit difference from sagas is: Term "saga" is completely overloaded. Initially it was introduced for managing long-lived database transactions and then got hijacked by Udi Dahan and NServiceBus. Sagas usually allow complicated business logic structured around internal state. Workflows usually don't have any logic and serve merely as a way to explicitly define events which this Bounded Context subscribes and reacts to. If we bring all this into one picture, that's what will show up: Projections OK, this picture is nice, but dealing with events outside of Aggregate Roots can be complicated (unless you enjoy querying event streams). That's where projections come into play. They allow to project event streams into any structural representation (view or persistent read model). This view is eventually consistent and persistence - ignorant (in other words, it can live in any key-value store with decent consistency guarantees). Projections are usually way too simple to require any testing. public class CustomerInvoicesProjection { IAtomicWriter<CustomerId, CustomerInvoicesView> _writer; public CustomerInvoicesProjection(IAtomicWriter<CustomerId, CustomerInvoicesView> writer) { _writer = writer; } public void When(CustomerInvoiceWritten e) { _writer.UpdateEnforcingNew(e.Id, i => i.AddInvoice( e.InvoiceId, e.Totals.Total, e.Header.CreatedUtc)); } public void When(InvoicePaymentReceived e) { _writer.UpdateOrThrow(e.CustomerId, i => i.AddPayment( e.InvoiceId, e.GrossAmount, e.PaymentId)); } public void When(CustomerInvoiceClosed e) { _writer.UpdateOrThrow(e.Id, i => i.CloseInvoice(e.InvoiceId, e.Reason)); } } [DataContract] public class CustomerInvoicesView { [DataMember(Order = 1)] public IDictionary<long,CustomerInvoice> Invoices { get; set; } public CustomerInvoicesView() { Invoices = new Dictionary<long, CustomerInvoice>(); } public void AddInvoice(InvoiceId invoiceId, CurrencyAmount total, DateTime createdUtc) { Invoices.Add(invoiceId.Id, CustomerInvoice.Create(invoiceId, total, createdUtc)); } public void AddPayment(InvoiceId invoiceId, CurrencyAmount grossAmount, string paymentId) { Invoices[invoiceId.Id].ApplyPayment(grossAmount, paymentId); } public void CloseInvoice(InvoiceId invoiceId, string reason) { Invoices[invoiceId.Id].Close(); } } On our diagram they would look like this: Where the service (that provides query capabilities over the view) could be located in the client (Web UI) or inside server-side bounded context. For instance I could maintain an eventually consistent list of all outstanding invoices in my payment processing BC, so that when a payment comes, I could map it to the invoice by reference number. Processes These 4 building blocks can be combined in various ways to define a system that passively reacts to external events (which could be caused by user) and publishes responding events outside. How do we proactively do something to drive world around us? Let's code us our virtual user that will be running loops over the projected data and carrying out actions, when he finds something interesting. These actions will come either as events published to everybody ("FraudTransactionDetected") or commands to a command handler within the same bounded context ("TryClosePendingInvoice"). In some other teams people would use "sagas with state machines and future messages" in order to implement this simple user. In our case, this is just a simple process that accesses some service in a loop. The service can be our own read model or some 3rd party system that we are polling. while (!token.IsCancellationRequested) { try { var doc = _reader.GetOrNew(); foreach (var cmd in GetOverdraftSuspects(doc)) { _endpoint.ToCustomer(cmd); } token.WaitHandle.WaitOne(waitPeriod); } catch (Exception ex) { Context.Debug(ex); // to avoid overloads token.WaitHandle.WaitOne(TimeSpan.FromMinutes(5)); } } But, unlike sagas, this process is not a pain in the neck, when matter comes to debugging, maintenance and upgrades in real world. Bounded Context All these 5 blocks from above can be combined together within a bounded context to express any kind of desired behavior. For instance, let's say we are defining a BC for managing access of users to some cloud service. This BC would be responsible for: telling service fabric, which users are granted access to to it (and with which keys) receiving fine-grained usage statistics from service fabric; aggregating these statistics on a periodical basis into consumption bills for actual billing process. Such BC would have following components: Workflow that subscribes to outside account management events (from other BCs) and passes them as commands to internal command handlers (which will have aggregates for consumption tracking); these command handlers would also call domain service for service fabric management API to add/remove access keys for users; there will be a continuously running process to poll management API of service fabric for any new consumption details and pass them to consumption aggregates. there will be a projection that maintains a list of all accounts along with their respective billing periods. this projection will be scanned once in a while by a process that will look for accounts that should have their consumption bills settled. For each of these, it will send a command to internal aggregate, instructing it to close them. Somewhere in bounded contexts far far away, there could also be: a billing process in some other bounded context, that is interested in events about closed cnosumption bills (using them to charge customer's balance) Web Admin UI that displays a view with last 500 resources consumed and top consumers (kindly populated by corresponding projections) Web client UI that displays for the customers detailed break-down of the resources they have consumed within the last billing period. etc... But these are different bounded contexts, with different story, purpose, tech requirements and combination of these 5 core building blocks. Update This article has been improved upon with: Anatomy of Distributed System à la Lokad