Event Sourcing - Versioning

Event Sourcing - Versioning A while ago we've started talking about the basics of event sourcing and reasons when to use it (or not to use it). Let's proceed to the most frequent question related to this subject: How exactly do we handle changes in event contracts: renaming fields, adding new members or even performing a complete refactoring. Contracts Event contract is simply a definition of event, that could explain how to serialize some object into a transportable data structure and then rehydrate the object back. Usually contracts are defined as POCO classes. For example, here's one option: [DataContract] public partial class RequestPaypalPayment : ICommand < PaymentId > { [DataMember] public PaymentId Id { get ; set ; } [DataMember] public InvoiceDefinition Definition { get ; set ; } [DataMember] public Email Billing { get ; set ; } } Logical meaning behind the contract (how the name is chosen, what fields are used etc) comes from the world of Domain-Driven Design, applied to CQRS and Event sourcing. The problem comes from the fact, that it's nearly impossible to get these contracts right from the first attempt in building a system. And even if you manage to, there is a significant chance that our own perception of domain model will change, creating the need to adjust the contract as well. So contracts change . These changes can break our ability to read previously saved events (and previous history can easily go years back), or cause logical data corruption. Let's go over various types of changes in contracts and ways to handle them. Minor Corrections The easiest kind of change to event contracts deals with correcting typos and changing naming of class members. All these are completely painless and transparent, if your contract serializer is evolution-friendly. For instance, in you can use Google ProtoBuf Serializer, which does not care about names, relying on integral tags instead. In .NET world POCO contract would look like: [ProtoContract] public partial class RequestPaypalPayment : ICommand<PaymentId> { [ProtoMember(1)] public PaymentId Id { get; set; } [ProtoMember(2)] public InvoiceDefinition Definition { get; set; } [ProtoMember(3)] public Email Billing { get; set; } } By the way, there is an interesting trick with this serializer. Instead of using attributes from ProtoBuf.dll , you can leverage attributes of DataContractSerializer , which is a part of .NET BCL. This will make your contract libraries decoupled from non-BCL code (reducing change of dependency hell). [DataContract] public partial class RequestPaypalPayment : ICommand<PaymentId> { [DataMember(Order = 1)] public PaymentId Id { get ; set; } [DataMember(Order = 2)] public InvoiceDefinition Definition { get ; set; } [DataMember(Order = 3)] public Email Billing { get ; set; } } From now on, when we will be talking about event contracts, simplified contract definition syntax will be used. For instance the above contract could be represented as: PaypalPaymentReceived(PaymentId id, string paymentId, invoice, account, amount) This explanatory syntax is not only more compact, but it could also be used to build actual contract classes for any platform, given a DSL parser. Alternatively, you can just pass them to a Junior Developer and he'll be able to code them in. So the first type of changes to event contract deals with minor corrections (renaming members). Good serializer will handle this natively. If you don't have such serializer, or the change is more deep - you can use in-memory upgrader, discussed below. Enriching Events More common versioning scenario with event contracts involves enriching events by adding new fields. Consider following event: UserDisabled(Guid id, string reason) Imagine we have a system in production, that uses this event. However, at some point a request comes from the business to create a web view that would list names of disabled users for each account. There are multiple approaches to solve this problem. Some of them involve writing complex read-models (capturing a wide variety of events in order to persist some additional info). However, it will be much easier if we were to publish this event: UserDisabled(Guid id, string reason, string name) Given this definition, writing a view handler would be much easier. It is also quite easy to start publishing these events from the domain model. However, if we were to replay history to rebuild views, we would have events with null name. That's where domain in-memory upgraders come to the rescue. They are responsible for encapsulating this upgrade logic. Upgraders are defined like implementations of this interface: public interface IUpgradeDomainEvents { IEnumerable < IDomainEvent > Upgrade ( IDomainEvent @event , string messageId ); DateTime DefinedOn { get ; } } Where DefinedOn is the hard-coded date on which this upgrader was implemented (this is used to order upgraders within the chain). Actual upgrade method returns an IEnumerable to handle cases, when a single event is split into multiple events, or when we are compressing multiple events into one. Implementations of these upgrade methods are usually located and distributed within the contracts library (so they are part of the schema). Any code (or a node in distributed system) that stores history events and reprocesses them later - could easily locate and load these upgraders by a simple reflection. Potential implementation might look like: public class EnrichUserDisabled : IUpgradeDomainEvents { readonly IDictionary < Guid , string > _userNames = … public DateTime DefinedOn { get { return new DateTime ( 2011 , 10 , 4 ); } } public IEnumerable < IDomainEvent > Upgrade ( IDomainEvent e , string id ) { // if this is event with info , capture it . var named = e as UserNamed ; if ( null != named ) { _userNames [ named . Id ] = named . Name ; yield return e ; yield break ; } // if this is target event - enrich it var d = e as UserDisabled ; if ( null != d ) { if ( string . IsNullOrEmpty ( d . Name )) { yield return new UserDisabled ( d . Id , d . Reason , _userNamed [ d . Id ]); yield break ; } } // otherwise just pass it through yield return e ; } } Please keep in mind, that upgraders are just a convenience short-cut that prevents views (or any other event consumers) from being "contaminated" by logic to handle versioning of event contracts and their upgrades. Since upgraders are a shortcut, we can discard them altogether This is done by taking existing event streams, running them through the upgraders and overwriting source streams with results. Deep refactoring There are scenarios, when event has to be replaced completely or even split into multiple events. For example, after evolving our system, we might find out that AccountRegisteredEvent now becomes three distinct events: AccountCreatedEvent; UserAddedEvent; UserActivatedEvent. IUpgradeDomainEvents can handle such cases as well. You just return three different events in this case: yield return new AccountCreatedEvent ( e . AccountId , ... yield return new UserCreatedEvent ( userId , e . AccountId , ... yield return new UserActivatedEvent ( userId , e . AccountId , ... Upgrader approach can also be used to merge events , if really necessary. You just need to push first event on the stack, when it comes, and then replace the second event with the result.


Why Event Sourcing?

Why Event Sourcing? Event Sourcing is a concept that becomes increasingly popular day by day. Even ThoughtWorks has brought it into it's latest Technology Radar . Let's do a quick overview of ES one more time. Why Event Sourcing? In essence event sourcing is about persisting data in a way that preserves every single bit of information . It's about representing objects as a sequence of events that took place through the time and led them to the current state. For instance, if I were to persist information about my pocket money (i.e.: 67 EUR), I could simply save the latest state somewhere in a variable or database: Balance : 100 EUR Now, whenever there is a change, we would overwrite this value with the new value (discarding the previous one). Then at some point in time we will have something like this: Balance : 67 EUR Simple and elegant (and works perfectly in a large number of scenarios). However, we are performing a logical compression here (lossy one) and discarding some information. Let's see what would happen if we were to preserve all the changes : Got from ATM: 100 EUR Bought metro tickets: -12 EUR Grabbed a lunch: -8 EUR Found a coin: 1 EUR Took taxi: -14 EUR Obviously, if we have such a sequence of events, we can always "reconstruct" the current balance, by doing a total: Balance : 100 - 12 - 8 + 1 - 14 = 67 EUR In essence, the final state (Balance) is a left-fold function of the previous states (equivalent of IEnumerable.Aggregate in .NET, std::accumulate in C++ or array.reduce in JavaScript). Now, you might ask yourself a question - What's the point in storing all these intermediate steps, when you can just save the final balance? This way of persistence via event sourcing possesses some really interesting features. Below are some of these. By the way, if you are asking yourself about the performance of such an approach to storage, don't worry. It can easily beat relational databases in both scalability and throughput (ceteris paribus). Storage Ignorance It is really easy to save data as a stream of events. All we need is to define POCO ( Plain Old CLR Object ) classes (one for each event) and then serialize them to one of many formats available: Google ProtoBuf, JSON, Binary, XML etc. Now, before you say that it takes to much code to define events: GotMoneyFromAtm! (amount, transaction, time) BoughtMetroTickets! (count, amount, machine, time) GrabbedALunch! (amount, cost, time, menu, place) FoundACoin! (amount, gps, time) TookTaxi! (amount, rideDuration, taxiCompany, route, time) That's how the definitions could look like for C# language, if you used some T4 of Visual Studio (see code contracts explanation). Flexibility Given the sequence of events, we can project them to any desired structural representation. This is an extremely important feature. For instance, we could write a projection that would summarize all our expenses and produce the latest balance. However, we can do much more: Generate list of cities, in which coins can be found most often. Get a list of taxi companies that are the cheapest ones or fastest ones. List TOP 5 favorite places for having a lunch on Monday. What's more interesting, we don't need any really complex queries in order to do that. Writing event projections (at least in C#) is something that is quire boring. Try doing that if you have just a single field of Balance, or even if you had a list of changes (credit/debit). However, as long as you have a steam of events, you can project it to any form, even a conventional SQL database. Messaging Capabilities Events represent a serializable and immutable data structures that are appended to an append-only stream. As such, they share all the capabilities of messages. So we can: Multiplex them to multiple subscribers (i.e.: have a list of 5 last restaurants visited by Lady Gaga; while keeping that list up-to-date and continuously replicated around the globe to load balance the pressure of fans). Implement redundancy and reliability (i.e.: write events to 3 different locations at once, where we have 1 master and 2 slaves). Support load-balancing (competing consumers) and parallel processing. Support incremental sync that fits in a few lines of code. Here are these lines of code from the production system (we are replicating from remote to cache ): var next = _cache . GetCurrentVersion (); while ( true ) { var items = _remote . ReadRecords ( next , BatchSize ); if ( items . Length == 0 ) break ; next = items . Max ( m => m . Version ); _cache . AppendNonAtomic ( items ); logger ( string . Format ( "Loaded {0} records" , items . Length )); } Of course, in a more conventional system (that does not employ event sourcing) you can leverage something like SQL Replication or Microsoft Sync Framework . Improve performance Truth to be told, performance and scalability aspects are a by-products of inherent capabilities offered by event sourcing approach. In essence, we can get almost-infinite scalability on reads with blazing throughput and no deadlocks. All this is attributed to the following facts: events can be published as soon as they happen; it is dead-easy to precompute read models from the events (do a projection to some desired state); events can be processed in parallel; events can be multiplexed to multiple subscribers, where each one could maintain it's own copy of some read model. What is a read model? "Balance" is one sample of a read model; "List of favorite Gaga's restaurants" is another one. Essentially read model is some view (precomputed result of an SQL Query in SQL World). Since we have more flexibility with projecting events and passing them around, we can easily do more interesting things, reaching up to the speeds of LMAX (which was described by Martin Fowler ): Keep read models (precomputed query results) directly in memory in some MemCached instances (if server goes down, we can always repopulate cache from the history of the events). Run multiple processing units in parallel (if any one goes down - switch to the backup). Benefit from the real-time signal processing knowledge of robotics and critical systems (imagine, what could do an FGPA / ASIC optimized for parallel processing of events). Simplify Developer's Life There are a few more interesting aspects of event sourcing: Simplifies deployment and maintenance (less SQL, upgrade scripts and versioning). Reduces expenses on both the hardware and software (no need to have extremely powerful and redundant servers or commercial databases). Integration between systems is more straightforward (all Enterprise Integration Patterns apply here directly). Since no data is ever lost, we gain full audit (plus the ability to go back at any point in time) and excellent debugging capabilities. Event Sourcing is a natural fit for software developers that want to capture the essence of business domains (especially the most complex ones) while staying decoupled from platform complexity or hard performance issues. ES approaches help to provide clear answers to some of the new challenges brought to us by the market and technology: Cloud Computing, Big Data processing, Mobile and occasionally connected systems, real-time business intelligence. There also are some financial and political benefits for project stakeholders to be interested in. They all revolve around the ability to have better flexibility in project delivery, managing resources and risks. Ability to keep things simple, defer important decisions and adapt business solutions can be a powerful enabler in large conservative organizations. Smaller companies (such as lean startups) can also gain more competitive advantage and reduce time-to-market with such approaches. However, here we are getting already in the area of synergy effects with CQRS/DDD methodologies and their practical application to distributed environments (esp. clouds). This is a topic for a different blog post or a talk. Downsides Obviously, Event Sourcing is not a silver bullet , it is just a different approach to think and represent changes and data. If you are a C# or C++ developer, then this feels like going back to assembler. If you are a project manager - it's like consciously going back from Microsoft Project Server to task lists and custom budgeting software. This explains why there are quite a few problems with this approach. Defining these events is a complex art of it's own, which requires skills in domain modeling (hint: if you have a lot of events with following words in their names, then you are doing something wrong: Create, Insert, Update, Delete, Set, Change, Add). Domain-Driven Design (as both a book and a body of knowledge) is an entry point into this skill. There is little software and hardware to support event sourcing . Luckily, we need much less of that (as compared to SQL/NoSQL), but still. In the next few years we will see interesting solutions in this field. At the time being, there is even bigger lack of information and guidance on this body of knowledge (to be fixed within the next year). Since we have limited information, acceptance and software, naturally there is a limited number of experienced developers with true DDD/ES skills. All these downsides are quite surprising, since the actual principles behind event sourcing are extremely old; they have been discovered and applied in multiple areas over and over again. Even the replication of SQL (transaction logs) uses similar principles. Concerns There are a few additional concerns that might look like downsides of approach, but in fact are not that important. Extra storage costs - are usually negligible, when compared to the business value that might be created. For instance, cost of storing 200k events in the cloud is roughly 10 cents per month. Oh, and I've counted this one 10 times just for the sake of having 10 replicas in different data centers for redundancy. If this negligible cost would have saved me at least a few days of development, then this could be a bargain. However, event sourcing saved much more than that. Slower performance is not an issue, since we can optimize IO via snapshotting and persistent read models. And leveraging push-based nature of events, we can get immediately invalidating caches. In short, there are multiple technical solutions that could be plugged later , if there is such need. Fragility (loosing an event in the past causes the entire stream to be corrupt) is not an issue, since you can determine yourself the levels of SLAs to go for (via replication and redundancy). Corruption in any single replica can be reliably detected using git's approach: event includes SHA1 signature computed against it's contents and signature of the previous event. Versioning is sometimes perceived as a problem, since our systems tend to grow and event contracts (schemas) can gradually evolve to new formats that are incompatible with the old saved events. Yet, if approached consciously this can be solved (and solution is more elegant and simple than SQL migration scripts). I use a combination of 3 elements here: Using serializer that is evolution-friendly (i.e.: Google ProtocolBuffers is not only among the fastest serializers, it also allows you to rename class members without breaking anything). Using in-memory upgraders which are simple classes stacked on top of the event streams for replays and accessing history. They can split, merge, convert events or even fill in missing fields (if this data is available in the other events). Upgraders are just a lazy and quick replacement for rewriting the entire event stream history completely. Using careful event modeling (which is usually based on the Ubiquitous Language coming from DDD) to define events in a way that they will rarely change a lot (they will be based on concepts that do not change, no matter how their relations evolve). Further Reading If I haven't scared you enough with the downsides of Event Sourcing, here is some further reading. CQRS Info by Greg Young : Events as a storage mechanism Building Event Storage CQRS and Event Sourcing A few of my own articles : Event Sourcing a la Lokad Scenario-based Testing for Event Sourcing (and old article, to be replaced by a proper one on BDD testing). Some more materials : Event Sourcing and CQRS, Let's use it by Jérémie Chassaing (recommended read for C# guys) Really nice EventStore framework for .NET by Jonathan Oliver. Listen to the distributed podcast . Stories As nicely said by Mike Nichols in DDD/CQRS mailing list: My experience has been that ES and its promotion of business semantics over technological terms has a way of bending my mind toward modeling behaviors rather than essence. It also lets me avoid the ceremony of modeling state that doesn't contribute to behaviors. I see this as a good thing and as a side effect I probably use language the business person understands more. I just can't find a reason to use ORM in my domain anymore. ES seems to let me more rapidly model ... having a change log/audit trail is about the furthest thing from my mind when I reach for it.


Example of Self-documenting Unit Test with Event Sourcing One of the biggest advantages of event sourcing approach is it's inherent capability to turn unit tests into a living documentation . Below is an example of specification that I've worked my way through today (that's how NUnit prints it out). registrations: duplicate email fails - Passed Environment: index includes email("contact@lokad.com") When: Register 'd6e64e': Customer Name: Lokad Contact Email: contact@lokad.com Date: 2011-12-27 Expectations: [Passed] Registration 'd6e64e': Customer Name: Lokad Contact Email: contact@lokad.com Date: 2011-12-27 [Passed] Registration 'd6e64e' failed: Email 'contact@lokad.com' is already taken. And here's how actual NUnit code looks like: public Specification duplicate_email_fails() { var info = new RegistrationInfoBuilder("contact@lokad.com", "Lokad").Build(); var index = new MockUniquenessService(); return new RegistrationSpec(index) { Before = {() => index.includes_email("contact@lokad.com")}, When = new CreateRegistration(reg, info), Expect = { new RegistrationCreated(reg, info), new RegistrationFailed(reg, new[] { "Email 'contact@lokad.com' is already taken." }) }, Finally = index.Clear }; } All was achieved without any special magic or even fancy tools. I've just pulled over sources of SimpleTesting and CompareObjects for additional readability. For those who are interested in RegistrationSpec class, it is just a simple snippet wiring together dependencies of aggregate root to a strongly-typed specification deriving from TypedSpecification in SimpleTesting : public sealed class RegistrationSpec : AggregateSpecification<RegistrationId> { public RegistrationSpec(IRegistrationUniquenessService service) { Factory = (events, observer) => { var state = new RegistrationAggregateState(events); return new RegistrationAggregate(state, observer, service, new TestPassword(), new TestIdentity()); }; } } Explicit strong-typing of aggregates (as described in bliki ) works all the way back in unit test specification by allowing to benefit from compiler-time checking and IntelliSense support. In other words: you don't need to navigate through hundreds of messages to figure out which ones are actually applicable in test.


Experience with CQRS+ES/DDD in Current Project Just a quick summary of tech I'm using in the current project. So far it works out rather nicely. Project targets Windows Azure . However, just like our other recent projects at Lokad, it is developed to be cloud-ignorant (can be deployed anywhere where Microsoft .NET can be run). Development machine : MacBook Air 2011 13' with intel i5 CPU (1.7Mhz), 4GB RAM and 128 GB SSD. OSX (Lion) with Windows 7 running in Parallels. I used to boot it in BootCamp (for native performance) a few times and then switched to pure VM mode. Development IDE : Visual Studio 2010 with ReSharper 6 (solution-wide analysis is OFF). Windows Azure SDK is installed (with Emulator and Storage) However it's not used at this project at all, due to being rather inefficient with resources of my MBA. File system is used for both queues and storage instead. In production they will be reconfigured to use Azure-specific adapters (abstractions provided by Lokad.Cqrs). Architecture : Distributed system with "CQRS approach" where aggregates are implemented using Event Sourcing . Business processes are stateless routing rules (as opposed to more classical sagas), with more complex workflow logic and state being pushed to aggregates or dedicated entities. The latter makes perfect sense with event sourcing and deployments in cloud environments. Read-models are implemented using document model for now (each view is a single document being updated atomically). Still no SQL. Huge lists will be handled on per-platform basis (if they ever become a problem worth solving). Frameworks : Domain logic (the most complex part) does not really depend on anything except Base Class library (System. namespace). Infrastructure (essentially, configuration code) relies on ProtoBuf/ServiceStack for serialization and Lokad.CQRS with Autofac for hosting everything server-side. Web UI - ASP.NET. Testing : NUnit used to wire custom specifications (code copied and adjusted from SimpleTesting). Custom tools : using side helper to generate command/event contract classes (got lots of them) out of compact DSL. C# code is generated whenever I do Ctrl-S and is immediately picked up by R# (this is slightly improved version if T4-driven CodeDSL. Improvement - it no longer depends on T4 and hence is much easier to change. Actual DSL generation is completely within the project now. details are in gist . This custom tool can be dropped at any point (completely along with the DSL files), contract files will stay. What makes development complex : it's hard to get domain models and bounded contexts right (however coding or testing itself is no longer an issue. Scalability and performance - even more so); hard to communicate finer-grain detail to people used to SQL/ORM; I still haven't figured how to explain (graph) business processes. So far sequence diagrams work the best; Event sourcing plumbing had to be implemented manually (EventStore was too much of a dependency, something more simple was needed). For the record, this project is a replacement of the project that was using following stack: NHibernate + FluentNHibernate Newtonsoft JSON Autofac Windows Azure Microsoft SQL Server Protobuf Lokad Shared Libraries Migrating existing system to ES requires an additional custom "Reverse Engineering" app, which scans database and generates events for the new deployment (roughly 300000 events). Best lessons learned : NHibernate and ORMs in general are a pain (especially in systems that has to live through more than 1 deployment). Although we are so used to that pain, that it does not feel. Well, until you have to go back and work on projects that have not been migrated, yet. Specification tests (and ability to use them as a living documentation readable by non-dev people) create some interesting additional opportunities. Simpler the code is - better. Abstraction from the persistence not only reduces friction, but also allows to have multiple deployment options (in cloud, on-premises, in memory etc). If domain model is defined (i.e.: there is BC and specifications for an aggregate), then actually coding Aggregate root with event sourcing (AR+ES) feels like a monkey coding - you just need to make all tests pass. Helland's principles (of building distributed systems in eventually consistent world) can be enforced in CQRS/DDD world, if we get rid of sagas in classical interpretation (something that has state, subscribes to events and publishes commands), replacing them with stateless routing rules and pushing state to entities (aggregates). Underlying framework becomes much easier (no need to explicitly manage and correlate saga state). Dynamic scalability - too. I'm using Event Sourcing even for simple things (like replicated user logins), simply because introducing Sql (or any other storage) would add too much complexity and friction without any visible gains. Using specifications for testing AR+ES has another side advantage - I added a single unit test, that gathers pre-populated commands and events from these specifications. This data is round-ripped through a few serializers and compared. This ensures that message contracts (and their use) stays compatible with all primary serialization options (ProtoBuf, JSON). Helps to catch use of DateTimeKind issues (which is generally not supported by JSON+PB). Obviously, this is highly subjective experience (and really rewarding). It might change as we are pushing the project to the first release (and further). Your mileage may vary. If you are really interested in long-term production experience - please ask me about that in a few years . PS: This is not the first project where ES has been used. But it combines the best lessons learned in lean development at Lokad .


From .NET to Erlang - Starting Work with Large Free Dating Website 10 years spent in .NET were fun, but it is time to try something radically different . As you probably already know, I'm moving to a new project at HappyPancake - large free online dating website (largest one in the North Europe, if I'm not mistaken). I'll be working with outstanding people there: Tomas Roos and Tom Janssens . I'm a completely newbie in this development stack (summarised by Tom): Migrating a large free dating website from the Microsoft stack to Erlang/Linux... Some keywords: - Programmer anarchy!!! - Erlang/Python/C/ - FoundationDB / Phonegap / flightjs / responsive design - Chatroom devops - Remoting - CQRS / DDD / Eventstorming - Wercker / DigitalOcean / Github / BrowserStack / PhantomJs / Behave - Campfire / Mindmup / Google docs / Dropbox / Google Hangout - And many more! I hope to be learning fast enough to keep up with these guys, so that they will not kick me out of the team due to the sheer stupidity, of which I'm guilty on more than one occasion. So far the experience has been truly amazing. We're going to have RnD blog for this project . I'm still going to stay with Lokad in a support role, however my time spent there is going to be limited. The company needs somebody to take on some of my responsibilities. If you are an experienced .NET developer with desire to get immersed into Windows Azure devops, Big Data, Business Analytics and event sourcing, there is an opportunity for you in Paris (I'll try to help as much as I can in this process). Please get in touch via contact@lokad.com .


Get Better Understanding

Get Better Understanding I know BizTalk and would not hesistate to use it. Enterprise Architect As it seems, a lot of problems in software domain stem from a single cause - we don't truly understand a lot of things and fail to acknowledge that. This leads to complex designs and half-baked solutions. How many times have you encountered a marvelous new technology that has promised you better life and all the riches in this world? How many times, have you discarded this technology after a few days or years of trying? What about Aspect Oriented Programming, Hibernate or tools that calculate code metrics? I believe, this highlights one of the most essential problems we have in the development today: we try to substitute true understanding of something with smart technologies and tools that promise to take care of us and our tasks at hand. In fact, they can, but their applicability is more limited than it is advertised. And they can lead you into really dark valleys, while trying to hide some of their inherent issues. Do these sound familiar to you: SQL upgrade scripts, WCF configuration code, NServiceBus endpoint management or making your favorite IoC container work with some framework. If you want decent results a tool can help. However, if you want to push your limits beyond the limitations of your tools and technologies - go get your hands dirty. What I'm trying to encourage you to do - to dare to try and get rid of some tech or tool in your next project (whether it is work or a hobby). It could be something like IoC Container, service bus framework, ORM tool or entire storage engine stack. It certainly would hurt, just like any exercise does. But you might also find some deeper understanding there. The one that would let you push your limits beyond the limitations of these tools and technologies.


Get Real and Move Forward

Get Real and Move Forward These are a few interesting facts about Lokad, that you might find entertaining. In last 10 days team of 3 devs at Lokad delivered 9 official releases. This covers 4 different projects developed in parallel. Tenth release goes live tomorrow: an upgrade of Salescast. It brings online final elements of infrastructure needed to deliver quantile forecasts that represent major breakthrough in inventory optimization done by our amazing analytics R&D team. As you know, one of our customers managed to save million of euro at monthly cost of 150 EUR, and this was before quantiles surfaced. All this business intelligence is cloud-capable from the start and keeps on getting further reductions in development friction, risks and wasted efforts. We just keep on moving forward faster, adapting to changing environment on the go. This acceleration happens not via the increase of development teams but by working smarter. While doing that we don't hide everything behind the curtains but rather share as much secrets as possible with the community (check our open source projects ). This is rewarding in many ways. When I hear about development teams and companies that spend time moving at slower pace than that - it does not feel right. We are living in amazing times, where speed of change keeps on accelerating, as new disruptive technologies come into play. Cloud computing, mobile devices, personalized large-scale data processing, real-time intelligence and global socialization were unheard of by our parents and now they are a reality. Can you guess what things your kids would play with? I can't even imagine, but I can try to help building that different future. It will come fast. So get out of greenhouses with slow monthly releases and reality distortion field of ivory-tower architects and unlimited budgets. These walls are not going to hold on for much longer; things like "economical crisis", "technology changes", "political reforms" and "new market players" are probably already breaking through into previous safe heavens. Get out of the house and go play outside. Find things that you enjoy and push them to their limits. You can work from the ocean beach at Bali, travel around the cities for beers with friends or just explore the world along the way. This would also help to fuel inspiration and motivation on the go. Get real. Challenge authorities and people who dare to tell you what is wrong and what is right without proving their words by practice. Even if they prove - still challenge them, you might be able to do better. You should be able to do better - that's how humanity pushes forward state of the art. It's also fun.


Getting Rid of CQRS View Rebuilds

Getting Rid of CQRS View Rebuilds We all know that one of the coolest reasons of event-centric architectures is: you can always discard your persistent read models and rebuild them from scratch , optimizing for the specific UI. For instance you can replace that paginated list of users in your system with a list of the most active users along with their latest activities. You can also leverage this capability to change underlying technologies, introduce replication or in-memory caching. We've been using this approach a lot recently, even created an in-house tool for that. It discovers projections in the code (projection is a class that is responsible for transforming an event stream to a specific type of view) and runs an event stream through them. While doing that, first we save all views in memory (just makes things faster, especially when you have hundreds of thousands of events). When the entire rebuild is complete, we "dump" in-memory views to the actual persistent storage, overwriting the old version of views. For .NET developers we've published a sample source code for that in Lokad.CQRS Sample Project (you are interested in 'Audit'). Let's say, I have an event stream from production and want to see how my updated views will look like in local environment with the brand new UI: Start Audit tool. Point it to the event stream for the specific bounded context. Open views tab and pick the views I want to rebuild (more views to rebuild - longer it takes). Wait for the rebuild to finish copy materialized views from temp folder to the storage Start system. This sounds good and useful, isn't it? Now you can let your UI developers go wild, developing various denormalized view models, that could be used for really non-conventional views, while staying extremely responsive. However, there is a catch. Such functionality is a great enabler (it shortens UI-focused iterations), and its get's pushed to the extreme. Extreme is: you iterate UIs and and underlying views so often, that manually rebuilding views become tedious and time consuming. If I were to rate ORM+SQL experience on that tediousness scale, I would need a logarithmic one. So a few days ago I went ahead and pushed further the idea of self managing projection servers. I discarded rebuild functionality completely in one of our projects. Instead, server is tasked with duty of tracking his own projections and rebuilding only the ones that have changed. In normal production this would happen only once after a deployment that touches projections. In rapid development this happens whenever I edit some projections and restart the server. It will: Check if views exist. And if they don't - rebuild in memory and update before starting to process messages. If views exist - check if projection code have changed since views were touched last time. If they have - rebuild corresponding views in memory and replace them in the stores. Obviously, server does not process any new events, while rebuilding views. So I no longer have to worry about updating views either in production or in development - server tracks this for me. This is a major simplification. And since server rebuild only projections that have changed - it is always as fast as possible. How do we detect if projection code has changed? Answer is simple: we cheat by teaching the server how to decompile itself with the help from awesome Mono.Cecil (this is a trick I've learned from Greg and Svein and their awesome work on MightyMoose ). While doing that, to keep everything simple, we assume that no messy IoC containers are used and we explicitly wire and instantiate out projections the stupid way . Given that, we just need to generate a decompiled snapshot of a projection from the very code that is being executed (while discarding IL offsets) and store it whenever we generate views. Decompiled snapshot is just a list of instructions: Void DomainIndexProjection::.ctor(IAtomicWriter`2<unit,DomainIdentityVector>) IL_0000: ldarg.0 IL_0000: call Void Object::.ctor() IL_0000: nop IL_0000: nop IL_0000: ldarg.0 IL_0000: ldarg.1 IL_0000: stfld IAtomicWriter`2<unit,DomainIdentityVector> DomainIndexProjection::_writer IL_0000: nop IL_0000: ret Developers that use Lokad.CQRS Sample project for their dark production purposes, would immediately notice familiar class names. Actually the simplest code to generate something like this can start as: var builder = new StringBuilder(); foreach (var @class in projections.OrderBy(p => p.GetType().Name)) { var type = @class.GetType(); builder.AppendLine(type.ToString()); var typeDefinition = mod.GetType(type.FullName); foreach (var md in typeDefinition.Methods.OrderBy(m => m.ToString())) { builder.AppendLine(" " + md); foreach (var instruction in md.Body.Instructions) { // we don't care about offsets instruction.Offset = 0; builder.AppendLine(" " + instruction); } } } return builder.ToString(); So we just generate our representation of the executable code of the projections and compare it with the existing one. If some projections don't match - they need to be upgraded. In practice it's slightly a bit more complicated, since I have to walk down the referenced classes in order to grab things expressed inside referenced anonymous methods. Once the approach is stable I'll push it to the Lokad.CQRS Sample. Once you have this thing delegated to the server, life suddenly becomes more simple and practical. Both the development and deployment experience are improved. There is even no need to track which projections have changed within the iteration, since server can do this better. This especially becomes useful, when you need to deploy to the cloud. [0001070]: Engine ready. Running rebuild [0001071]: Projections in hub-domain-view are up-to-date [0001109]: Projections in hub-client-view are up-to-date [0001112]: Projections in hub-apiops-view are up-to-date [0006684]: Started Version - 2572 (Hub.Engine) Now if I could only figure the dead-simple simple way to automate VIP swaps between projection versions, so that they could be upgraded with no downtime at all… PS: Yes I did use decompiled snapshot of projection .ctor in this listing. Methods are slightly longer and wouldn't make a big difference.


Golden Rule of Technology

Golden Rule of Technology The rule says: 1 new technology goes into codebase only after 1 old technology goes out . This is a simple empirical rule that we've coined with Joannes , while talking about business and managing technology. As it turns out, this rule makes a really nice guideline for planning future development. The rule can apply to all sorts of things: new technological concepts, frameworks, ideas, tools and software. Basically, a technology in this list is anything that a new Junior Developer will have to learn about the project, before he can productively work the assignments . This approach encourages technology pruning and explains how to structure technological evolution of your company, almost turning this process into a game. This is quite important, since otherwise it will be hard to "embrace the change" and leverage new technologies and tools, as opposed to being hit hard by every new "revolution". Essentially it helps to define your own Technology Radar at the project level. For instance, we have the following evolution path for the User Interfaces in the world of Microsoft .NET Framework: Windows.Forms Silverlight/WPF Windows 8/HTML5/some Metro UI If you treat these UI technologies as something to focus completely on, then every new change might be quite expensive (i.e. from Silverlight to HTML5), because of the way the problem is being structured: We are going to develop new eCommerce suite with Silverlight. However, treating technologies as something transient and replaceable can reduce costs and even improve architecture (making it more resilient to risks and changes). Consider this way to structure a problem: We are going to deliver eCommerce suite, while using Silverlight for UI for the time being. Obviously, this approach requires an architecture that favors decoupling and simplicity (i.e.: CQRS/DDD/ES methodology ). However, this is not limited by the software architecture alone. Here are some samples: Frameworks Version Control Systems Testing Approaches Project Management Concepts Integration Servers Deployment Strategies Serialization Formats Another interesting side effect of this Golder Rule of Technology is that it encourages developers to simplify and refactor in order to be able to try some new cool things in the project. And sometimes, when you get rid of an old technology, there already is so much improvement, that you don't even want to add anything instead. One example of such replacement in place is gradual migration from traditional SQL-driven persistence to Event Sourcing at our place . It allows us to discard quite a few items from the current stack: NHibernate with all the related binary dependencies SQL Servers along with the entire concept of relational database and object-relational impedance mismatch Evolutionary databases and managing them (including upgrade scripts and unit tests working against mock databases to verify behaviors on continuous integration servers) Another practical sample would be - planned replacement of API implemented as REST via WCF (with SOAP and all sorts of weird configuration problems) towards a dead-simple implementation on HttpListener. An interesting thing is that I never really considered complexity of technologies related to a feature till starting to think about that in terms of "Return on Technology Introduced". As it turns out: Usage of technologies is extremely overrated (sometimes attributed to the marketing hype related to certain buzz-words). Usage of additional technologies tends to increase both burden and complexity of a project at hand. Emprical evidence suggests that linear increases in these - lead to exponential rise in project costs. The process of complexity increase can continue till the project becomes so expensive that it stalls. By reducing the number of non-essential technologies used, we can have better management of overall complexity (fighting delays, quality problems and costs). In essence, simplicity is a technology of its own . But it does not count as one.


Greg's Event Store and Windows Azure

Greg's Event Store and Windows Azure About Event Store As you probably already know, Greg Young built a file-based Event Store with a team in Ukraine. The product was publicly announced last September and now they are moving towards the second version. Single-node is open source and free to use (and it's a wonderful learning material for C# developers). If you want to have more failure-tolerance, then multi-node version is available. It uses negotiation protocols to ensure that cluster of Event Store nodes will stay up and keep all the information as long as half of the machines in it are alive (if I recall it properly, they actually use chaos monkey testing to ensure that, among other things). This store can get you thousands of persisted transactions per second on rather slow notebook. There is a javascript projection host built into this event store (currently it's in beta and hence has to be enabled manually). Alternatively you can define your own chasing projections and connect them to the store. ES offers HTTP REST API (designed to be scalable for reads using commodity building blocks like reverse proxies) and high-performance TCP API. With Windows Azure Here's what I currently know about behaviour of Greg's Event Store on Windows Azure (this question seems to be the recurring theme). Event Store works really closely file system, where the data is kept broken into chunks. Whenever possible these chunks are loaded into unmanaged memory to speed up read operations (which are really cheap). If you want to ensure certain degree of write durability, then these writes will be almost as fast as you can flush to your disk controller. This setup makes ES really performant, and also prevents it from getting the same level of performance in single-node configurations on Windows Azure. Single Event Store node can run on Azure in 2 major configurations: Data is stored on a local hard-drive . In this case, writes will be fast (thousands of tx) but there are no guarantees, that data on local drive will not disappear at some points. Local drives can serve only as temporary caches. This scenario is fast but less reliable . Data is stored on page blob mounted as VHD drive . In this case writes will be passed to the operating system and then flushed to Windows Azure Blob storage (where they will be replicated between multiple machines). Azure VHDs are guaranteed to stay there, however this comes at a cost ( reliable but slow ): you can expect roughly 125 writes per second to page blob on a single thread (afaik, ES currently needs multiple writes to commit a single transaction, which will yield even less than 125 transactions per seconds). when EventStore starts up (e.g. after a node is restarted by Fabric Controller), it needs to read from VHD, which can take minutes, during which the store will be unavailable for writes or reads. Luckily, for small scenarios: Windows Azure does not restart machines that often (how often do you need to roll out Windows update which needs a machine restart?) You can run Event Store on a Linux machine, which requires even less restarts. Depending on your business case, resulting performance might be enough for your scenario, or more tweaking would be needed. Only measurements will tell. Getting Better Performance You can theoretically get higher tx write throughput with some crash guarantees by: running event store with a local drive. adding a small background process which will incrementally copy ES to the blob. on a startup, before even starting ES, if local drive is empty - grab latest backup of event store from Azure. Alternatively, with a single node scenario, you can try to optimise performance of event store on Azure via enabling host caching for your VHD data disk (it's enabled for durable OS drives, but disabled for additional data drives). If the business grows, then it also might eventually be beneficial to sign-up for paid version of event store (putting event store data on transient disks in different fault domains). In this case, replication will be managed and guaranteed by Event Store engine. And Windows Azure will try to guarantee that nodes in different fault domains would not go down together.


Handling Big Data in Cloud

Handling Big Data in Cloud Big Data is one of the new hype terms that is gradually gaining popularity. No big surprise - amount of data around us is gradually growing and by properly mining it we can get competitive advantage and eventually make more money. Money attracts money. Usually by term big data we mean a collection of datasets that are so large, that they become too hard to be processed on a single machine . This data can even be so big, that it takes hours or days to process it in a data warehouse. Fortunately, cloud computing comes to the rescue. It provides following resources: nearly unlimited scalable storage capacity for data (i.e. Azure Blob storage or Amazon S3) elastic compute capacity to process this data (i.e. Azure Worker Roles or Amazon EC) network capacities and services to transfer data and coordinate processing (queues and actual networks). These resources are elastic (you can get as much as you need) and paid-on-demand. Latter is really important, since you pay only for what you use. I'm assuming here, that in your business model, more data processed means more money made. There are three distinct approaches in handling big data, based on the specific challenges. Batch Processing First one is about batch processing , where you do not need extremely fast computation results but have terabytes and petabytes of data. This is essentially about implementing MapReduce functionality in your system, often resorting to hacky but extremely practical tricks. Below I'll talk about some of the lessons learned in this direction in Lokad (we provide forecasts as a service without any hard limit on amount of data to process). Obviously, storage of this data becomes a primary concern with this approach. Fortunately starting companies can leverage already existing cloud computing resources, starting from Google's GFS and up to Azure Store . Or if you are big enough (and this is worth it), you can roll a data center of your own. Idem for computing and network capacities. Stream Processing Second approach deals with cases, where you need high-throughput and consistently low latency on gigabytes of data. High-frequency trading is one of the most known domains here (various real-world telemetry systems being the second one). Abusing event streams and ring buffers (like in LMAX ) along with partitioning - helps to deal with the challenge. Due to latency requirements, cloud computing might be not the best choice for latency-sensitive solutions. It is more efficient to roll out fine-tuned infrastructure of your own including specialized hardware and soft (things like InfiniBand and ASIC or even Mixed-signal chips). However if you are OK with a bit of latency and are more interested in high-throughput continuous computation, then the flexible network and CPU resources provided by cloud are a good match. Realtime Batch Processing Third approach to Big Data involves dealing with more complex requirement - providing real-time capabilities over vast amounts of data (so we have to be both fast and capable of handling petabytes of data). Twitter is a vivid example here - it needs to provide (almost) real-time analysis over billions of tweets around the world (although fortunately it does not to go at microsecond level). This challenge can be solved by actually mixing first two approaches: we would use slower batch-processing for dealing with actual bulk of data (tricks like preprocessing, append-only storage and MapReduce work here), while latest data will be handled through the real-time streams (stream processing and continuous computation). Results are merged as they come out of these two data pipes. Over the course of time (i.e. daily) latest data is pushed from "hot" zone into the bulk data. More often than not, real-time data processing is done by simplified approximating algorithms that deal with subset of data. Batch processing is more through and precise (at the cost of being slower). By pushing real-time data back to the bulk data, results of the computations are actually corrected, as they are recomputed by batch algorithms. Twitter's Storm project features nice overview of this approach.


Handling Concurrency in Cloud Views

Handling Concurrency in Cloud Views Let's expand on the topic of views in the Cloud Architectures (CQRS) by introducing concurrency handling. This is an update to the original article on Views (it was changed accordingly) that comes as a response to Olav's question (thanks!) How do we handle concurrency conflicts with the views: when the same view being accessed by more than thread simultaneously ? Actually we don't care a lot about cases with multiple readers and a single writer at once, since cloud storages providers generally ensure atomicity at this level. There are two major approaches for handling concurrency while updating views by multiple writers: simplistic and logical. The simplest approach is too keep updates of a single view entity limited to a single thread. For example, you can start with a single worker thread processing all view updates. As your application grows, increasing load and availability requirements, you can split updates of the different view types and entities between different threads and workers. In other words, you will partition views by type and/or view ID. Note, that we don't need to scale out actual view persistence, since it is generally handled by the cloud storage provider, to start with. However, such scaling tends to be limited by the world region (i.e.: North Europe) and we still might need to enable CDN or manually replicate data between multiple cloud centers. This is relatively easy to do just by streaming domain events to these data centers. As long as you pass entity ID in message headers (recommended in distributed systems), it will be easy to route domain event messages between different queues/workers. Just a quick clarification of terms. View Type is a message contract or the POCO class, while view entity is a single instance of this type, as identified and tracked by its unique identity. This identity serves as a primary key used to store and retrieve the actual view data. In the case of singleton views we have a single entity per type. Eventually you might encounter the need to allow multiple concurrent threads (whether hosted in a single worker or different ones) to be able to update the same view entity at once. This is where optimistic concurrency control comes in. We just need to modify our view writer implementations to keep track of blob's ETag (analogue of version ID), while downloading it. Then, after executing local update, we upload it back, while passing last known ETag back to the server. Azure Blob Storage (just like any other cloud storage) is aware of this feature and will update view data only if the ETags match. If they don't (somebody else managed to concurrently update our view) - update will fail and we'll get an exception. This exception will captured by the service bus, which will retry the actual update operation later. If such an update fails more than 4 times at once, this would mean some heavy concurrency issues probably coming from really complex update operations. This article is a part of R&D series on Practical Distributed Architectures and Clouds (aka CQRS in Cloud). You can subscribe to the news feed to stay tuned for more updates.


Handling Multiple Event Streams in Event Store Writing and reading 300000 events is fast even in VM running on MacBookAir (as fast as appending ProtoBuf serialized data with SHA1 hash on top). However the problems start, when you get thousands of individual aggregate streams to be written along the way. This keeps me thinking about the potential design options for an event store that could get decent performance even on low-end machines (i.e. these ARM-powered platforms that Kelly Sommers had been playing recently with). The culprit is that we want to have data reliably on disk as fast as possible. At the same time, we might be pushing thousands of event-sourced Aggregate Roots (AR+ES) to a single event store partition (served by one thread). And we want to keep these AR+ES instances separate. As you probably know, switching between files can reduce performance a bit (unless you are writing directly to disk, although even then would be seek penalty). The simplest solution is actually dead-simple: "What if we introduce a slight delay between the moment data is persisted on disk and moment it actually is persisted in a separate file"? The tricky part might be in reading events back per stream (since we would need to replay them for AR rehydration). One of the solutions is to route them to servicing threads that read individual entity streams. We can afford blocking read queries a little bit, if individual stream has not been updated to reflect the latest data. We can easily figure this out by checking up on absolutely consistent in-memory pointer which is updated by the ring writer (essentially just a dictionary of persisted streams and their latest versions). Interesting culprit is that we can actually speed up the performance of writer by configuring it (and all the slaves) to persist the main buffer in the memory. In this case, if master fails or crashes - we can always fail-over to one of the slaves (while letting the former master to catch up)


Helping non technical people understand the data I discovered this interesting question in Enterprise Architecture group in LinkedIn: I'm curious what some of the best practices are to share what information is stored in our database. I am aware of an ERD. I am looking for a business friendly solution that I can show to non technical employees to make them aware what data is available. Assuming that we are talking about relational databases, database representation is just a historical way to store data in a way that is optimised to save expensive disk and RAM space . Unfortunately this optimisation makes data schema overly complicated by trying to reduce information duplication and used space (we normalize, overwrite and fragment data). Curiously enough, prices on disk and RAM have dropped significantly during the last years. For example, Rackspace charges 0.15 USD for storing 1 Gigabyte of Data for 1 month on a hard drive (for that amount Windows Azure would even keep multiple replicas for you). Yes, we still try to store our data in databases, while accepting all the imposed limitations and cryptic storage schema . Slightly better way of storing business data (and sharing it) is to capture it as a sequence of documents which capture important business events. For example: NewCustomerCreated { Id : 37659, SSN : 0274178654, FirstName : "Rinat", LastName : "Abdullin", ... Manager : "John Big", ManagerId : 707, CreatedOn : 2013-06-12 16:34:19, CreatedAtLocation : "Paris-345 SE HQ", CreatedAtLocationId : 921 } AccountOpenedForCustomer { CustomerId : 37659, AccountId : 227461382600000045, Currency : EUR, Type : Deposit ... ApprovedByManager : "Merry Shea", ApprovedByManagerId : 802, ... } You can put as much information to these documents, as you want without worrying about the schema, as long as you keep them documented. Storage is cheap, so we can even add some extra data that might be useful later. This data is already more understandable by non-technical people than a schema in third normal form. People are used to documents and can read them. A sequence of such business documents can be automatically represented as a readable activity stream for a specific entity (e.g: "facebook wall of a customer"). Human understanding story does not end here, though. When you need to help another team to use your data, all you need is to: Provide them with the dictionary of these change documents, where each document has a short summary describing its purpose and a detailed explanation of all the fields. Provide team with the web service endpoint, from which they can retrieve a sequence of all events (for which they have access to) from any point in time and then stay updated in real time. If the team in question needs a database to work with that data, then they can easily transform these business change documents (or business events) into SQL schema that is populated with the data and then kept up-to-date. In essence they would have a persistent and up-to-date SQL cache of customer data on their side. Obviously, a team might be interested in doing something more interesting like cross-referencing or scanning data for new fraud detection patterns. Then they could transform these business events into batches for Map Reduce jobs in Hadoop or into star schema for OLAP analysis. In my experience, writing such transformations is an extremely simple task. Such team might eventually discover some new fraud patterns and would want to provide real-time alerts about these. They would just need to consume all new events in real-time, sending a notification when a match is detected. I wouldn't go in detail into performance forces behind such approach. It's enough to say that LinkedIn was forced to switch some systems from Oracle to streams of change documents just for the sake of performance (and lower license costs). It's needless to say that Twitter uses streams of events as one of the cornerstones of it's architecture. And very few systems can drink from it's firehose without drowning. At Lokad we had related experience through the use of Lokad Data Platform - Cloud-hosted server for storage and querying gigabytes of business events. Exposing business data as sequences of documented events (with business semantics) seems to work in enterprise world much better than trying to share database schema. It definitely helps to unlock "data silos" and enable inter-team collaboration within the enterprise.


How do you decompose complicated system into components? The answer to the problem of finding a proper way to decompose existing complicated system into smaller components is two-fold: Use known methodologies for finding potential boundaries; Iterate a lot to see if system system could be divided across these boundaries. Let's start with examples of methods for finding potential boundaries : Look at the domain model from the perspective of Domain-Driven Design , seeking out bounded contexts and aggregate boundaries there (although classical DDDish BCs and aggregates could often be split in smaller focused parts as well). Separate domain-side problems (need domain expert) from implementation problems like scalability (in some projects performance might be part of the domain, too); Use domain events to capture state changes in some parts of your system; then identify related events and group them - they could be a core of a new component; non-related events could probably be put into separate components; Look from the perspective of separating project between multiple teams and individual developers ; Look at transactional boundaries in your system. If there are that changes must absolutely happen together (or fail together) then they might be candidates for being put into a single component. If some changes don't really affect the others, then you could probably consider keeping them separately. Search for processes which must be coupled tightly from the temporal perspective - they might belong to one component. At the same time, presence of queues might indicate potential component boundary. And, above all, simply keep on asking yourself one simple question - what things can be taken apart ? How can we separate our software into atomic things that can be composed to solve the problem . Decomposition is not simply about breaking some project into a bunch of tightly coupled modules. The idea is to identify the boundaries and invest effort into evolving them . Methods for finding potential boundaries (like the ones listed above) only provide you with some hints and ideas. Your objective (as a designer) would be to play with these ideas, iterate and evolve system, while finding options that truly make it more robust, evolution-friendly and simple (put your priorities here).


How I Stopped Worrying and Learned to Love the WWW and UNIX Way Lokad provides big data analytics for retail as a service . For a long time, in order to deliver this to our customers, we used to invent new technologies and frameworks to fit our "custom needs". More than 20 public github repositories are still out there, including custom Azure execution framework, ORM and message bus. This was a good journey with a lot of learning about distributed systems, event centric designs and big data processing. There were many challenges, too. However, eventually we got tired of unnecessary challenges and became lazy ("we" as in "I"). This lead to one simple realisation: why do we even need to invent so much, if there already exists largest distributed system that we can learn from and reuse what others built? It is called World Wide Web. Surely, underlying principles might be not as sexy as some brand new "Enterprise stuff" (like AMQP or Azure Service Bus), yet they seem to work. Besides, WWW: has a huge amount of documented experience (more than any "Enterprise" software); wide variety of tooling; is frustratingly simple. In my current and limited experience, more we shift our design towards underlying principles of WWW (and away from latest sexy tech), more it feels like a huge relief and falling into the pit of success. Current project that we are working on in Ufa office (rewrite of business and SaaS backend of Lokad) is nothing like the previous systems. It's composed of relatively small and stand-alone applications which communicate over simple protocols (JSON over HTTP) using constructs aligned with the established domain model. A lot of complex technology is gone. For example, we ditched use of Azure Queues for communication between various components, replacing that with one-way RPC calls via JSON over HTTP (queueing can be plugged internally). All of a sudden, this: reduces software complexity (e.g.: your backend server is only accepts one-way commands in JSON over HTTP and publishes events as JSON entities in ATOM feeds); provides much better debugging and development experience (all of a sudden you can use tools like Fiddler or curl to interact and play with your backend server); allows scaling writes without complicated message topologies (just queue up all your one-way HTTP PUT/POST/DELETE requests); allows scaling reads by using dead simple force multipliers like reverse proxies; actually allows to use more efficiently services provided by Windows Azure while reducing vendor lock-in. In fact, in this project we stopped using all of Azure storage and messaging capabilities, since they are no longer needed. The only thing still used is Azure hosting model: instances of Worker and Web Roles which are managed by Azure fabric and run behind load balancers. However, should there be need it would be easy to move to a different cloud provider or managed hardware. Here are a few other new cool possibilities that opened: now it's possible to create integration tests for backend API by recording HTTP traffic with Fiddler and then replaying that; backend API just became self documented and accessible via XML, JSON, SOAP, thanks to ServiceStack; it's possible to rewrite some component using a completely new technology and nobody would ever notice; design is suddenly more friendly for things like reactive programming and single-page applications; it's much easier to deliver features by first implementing contract using hacky approach and then swapping that implementation for a proper code, whenever necessary. Things work out in such a way, as if we were trying to steer closer to UNIX Philosophy : This is the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface. In Unix world, small programs can be composed together to perform more complex tasks, communicating via the pipe or text files. Here's a beautiful example of installing Ruby Version Manager from a terminal: curl -L https://get.rvm.io | bash In case of distributed systems, we can think of small and focused applications which communicate over simple and human-friendly protocol (e.g. one-way JSON over HTTP in a RESTful way). If you align messages of this protocol with the domain model , like they do in Domain-Driven Design methodology, you'll have something that can withstand change pretty well . Keeping these applications really small and focused (like UNIX programs) would allow to reduce cost and friction of changing implementations so much that they could be easily thrown away and rewritten from scratch. Add immutability to the list of underlying design principles of these components and suddenly you get nice and predictable scalability and fault tolerance . All of a sudden, my head hurts less.


How Micro-Services approach worked out in production Last week we released a new version of business backend at Lokad. Previous version was based on message-driven design with heavy use of CQRS principle, event sourcing and queues. There is a case study on that at Microsoft CQRS Journey . As I've discovered soon after, this architecture was a big pain in the ass due to being too monolithic and dogmatic. There was a lot of unnecessary ceremony involved in the simplest actions. Back then it didn't feel as such, since literally everything in .NET world involves a lot of ceremony. Yet, after gaining initial exposure to the world outside of .NET, my perspective has changed. The concept of decomposing a complicated system into a bunch of tiny and simple components that work together and yet evolve separately - was refreshing and liberating. Current version of business backend at Lokad grew upon these ideas, while adapting them to the .NET reality (making things slightly more monolithic, where it was absolutely necessary). Still the result is quite nice: 13 backend components which have on average 4-5 classes and the same number of public methods; 2 thin UI components (simple ASP.NET MVC web sites); public API : JSON over HTTP; ability to subscribe to events from components; no overreaching architecture in each component, although event sourcing with in-memory views shows up quite often; all components are designed in a way which allows to have multiple instances running behind load balancer. almost no unit tests, since the components are extremely simple. If I were to draw a map of all backend components, it will look nothing like my previous monolithic designs: arrows point in direction from upstream component (provides services/data) to downstream component (uses services/data). Many of these components were rewritten multiple times from scratch, as the system was gradually evolving from legacy monolithic design towards deeper insight. This often happened in situations when we felt that there was a better way to decompose business capabilities into components. Sometimes "better" meant "simpler". The fact that previous version of the system was running event sourcing allowed to migrate functionality to new design in small steps. For example, at some points in time events were flowing from the old version to the new one. At other points in time, legacy code in the old system was calling directly newly established components that were already running within the boundaries of the new system. Basically, effort to decompose existing business capabilities into small chunks started paint off immediately : it became easier to think and reason about evolution of the design. It also became possible to break down work in really small steps (which minimised risks), while still maintaining reasonable development throughput (because multiple components were developed in parallel). I quite like the final result - so far the system is extremely simple and there were surprisingly few problems with the migration (due to the fact that the system is quite simple). Performance is not an issue . Current configuration could easily handle the load even if number of Lokad users increases a few hundred times (if it goes beyond that - we'll need to ask Azure to deploy one more server instance). This happened because performance of each component is measured along with its usage. In cases when components were frequently used and reasonably slow (as proven by stats from the production), they were tuned to better performance. Ultimate performance tweak for reads was about serving reads from an in-memory cache which is kept up-to-date via persistent TCP subscription to event stream with changes (just an event projection that subscribes to event store and stores projected results in memory). Ultimate performance tweak for writes was about putting command messages to a queue and processing them in background (work is shared across all nodes). If you remember my previous posts about Lokad.CQRS building blocks, you would recognise some of the patterns. The biggest change from "Lokad.CQRS architecture" is that there is no longer a uniform architecture in this new design. There are a few design constraints (e.g.: home components communicate or max size of the components), yet the internal implementation details are private to each component. Such an approach leads to situation, where each component can be tailored specifically for the job at hand. It does not need to follow a unified way to access data or handle failures. This might lead to a greater diversity. Yet, the lack of unnecessary ceremony allows to get right to the core of the problem in the most simple and efficient way possible. Hence components can be diverse and simple at the same time, just like cells in human body. Probably, the system could be made even more simple, if it were taken outside of common .NET stack. What infrastructure do we use? There is no real infrastructure, we mostly use some external libraries for our needs: ServiceStack for hosting app services (web services made easy) Metrics.NET for performance measurement Event Store of Greg Young to store events and to push them to all subscribers Windows Azure to host backend workers and frontend ASP.NET MVC 4 for Web UI


How sticking to HTTP and REST makes your life simpler As I discovered recently, stepping back from unnecessary technologies can lead you to a more happy path. For example, consider a simple system running business systems for a small startup. Previously I'd try to design it as a composite backend server and a bunch of front end UI applications. They would send commands via some messaging middleware, while retrieving read models from some cloud storage. This was pretty scalable and seemed rather simple. Still, there were a few problems: Special tools are needed to diagnose and inspect the system (custom test clients, cloud storage explorers etc) Special tools are needed to interact with these systems and script them. As Greg Young once said, the need of special tooling might indicate a bigger problem . Sometimes you don't even realise that there is a problem, till you try an approach that avoids it . Taking a business backend and switching from some middleware (which was not justified in that case) to use JSON over HTTP for communications suddenly gave much deeper insight into the system. For example, we could use Fiddler to see what's happening inside the backend: Automagically you get a plenty of existing tooling for logging requests, performance tracking, analysing sessions etc. I'm not even talking about ability to leverage existing internet infrastructure with all the intermediaries. It is a huge enabling factor to have an ability to take any given request, edit and replay it at will. Of course, we can do all that with special middle-ware tools (or write out our own). However, they will probably be less polished and numerous than open source tools for debugging HTTP requests.


How to Deal with Unexpected Problems

How to Deal with Unexpected Problems We live in a cruel world, where a lot of unexpected things could happen. So people that build reliable software tend to be really pessimistic about it and plan for the worst. That costs a lot (think of all the atomic bunkers built during the cold war). However, real life is much worse than any plans. It always proves our assumptions to be wrong. Expected bad things rarely happen, but unexpected weird things do happen. Here are some samples from software world: database connections timing out; Virtual Machines going corrupt; customer bringing in 100x of load that you planned capacities for; 3rd party services replying at the speed of 1 byte per second; somebody pressing the wrong button. SLAs, probabilities and estimates always lie. They are nothing more than expectations based on the assumption, that worst case scenario would be as bad as the one you've seen earlier (or just little bit worse). It's impossible to plan in advance. Get used to it. However, here's the approach that you can bet on (in fact it has been proven to be successful by thousands of years of evolution that brought fourth Homo Sapiens specie). Stop spending time trying to prepare for everything in advance. This will be waste of time and resources (think of all the dinosaurs that didn't survive Ice Age, although some were quite packed with muscles and claws). Instead, learn to evolve and adapt fast: reduce development and delivery friction and ensure that it is easy and cheap to make small changes with confidence . This will give you ability to react fast when life throws something unexpected at you. Make sure that you know your world well - deploy early and have means to keep a steady eye on your deployments, to know what is happening inside . Evolution is about survival of the fittest. So ditch ivory-tower dreams of being prepared for everything and just stay real and fit. Who knows, you might even buy yourself some time to enjoy the sight of competition dying out due to being unprepared for the unexpected.


How to produce a superb software design? It's impossible to arrive at a perfect design right from the start. There are far too many uncertainties and unknowns involved in the process. Hence, you can safely assume that some sort of iterative learning process would be required. Software development is a learning process. Working code is a side effect. © Alberto Brandolini, Model Storming Presentation Lessons learned could come in various forms: discovered requirements and constraints; implemented features or fixed bugs; written documentation (wiki articles, email threads, diagrams, napkin drawings, whiteboard photos, software guides etc). One of the most important goals of this learning process is to keep iteratively integrating newly discovered knowledge into already captured body of knowledge, while enriching it. This task might be more challenging than it seems at a first glance. Let's step away from software for a moment. If you think of it, continuously adding stuff to any existing storage container could eventually lead to a situation when we not only ran out of the free space, but it becomes really challenging to do anything with things that are already stored there. Consider filling cabinet with papers or adding more tools into a small closet. Unless we somehow reorganize the space (add more filing cabinets, introduce more shelves in a closet), it will be hard to work with all the accumulated stuff . Another option would be to come with smarter way to organize papers (e.g.: alphabetically) or tools (e.g. put the ones that are used most frequently - upfront). Same principle can apply to software design. Even though software size is rarely a limitation these days, our mind is quite limited in the number of things it can handle at once without the support of some sane mental model. …number of objects an average human can hold in working memory is 7 ± 2. © Wikipedia, The Magical Number Seven That's what happened in far too many projects that I've seen : new features were continuously added to the project without adjusting software design to handle increased complexity. Such situations lead to the point where software got so fragile that developers would spend more time fixing bugs than adding new features. Quite often resolving one bug would release a horde of new ones. One of the most frequent solutions to this problem sounds like this: "There is too much legacy. We can't add any more features to our software till we rewrite everything from the scratch in version two." In reality, approach of rewriting complex software from the scratch can be more problematic than it appears: rewrite costs time and money; new bugs can be introduced in the process, since we are changing existing and working software for something new; some really important requirements and features can be lost. Ideal solution to the problem of increasing complexity would be to avoid such bottlenecks in the first place. This can be done by: keeping software complexity as low as possible; continuously evaluating fitness of design as more requirements and features are introduced; continously adjusting design to stay meaningful and simple despite new features being added. In this case we can talk about iteratively integrating new insights (features, requirements etc) into the software, while also adjusting the design to handle increased complexity. At each step software would stay healthy enough to keep on pushing forward its evolution without much friction. I believe, this constitutes a superb software design. Hence: Arrive at healthy design by evolving it through a series of enriching transformations. This might sound like a lot of work. However, if we design for such continuous evolution upfront, then at each step we would need to deal only with a limited scope of change. This would reduce risk of introducing unexpected bugs or loosing an important requirement. Developers would stay sane, too. Note: This blog post is a draft of one story for an ebook on healthy design and patterns. Stay tuned, if interested. All comments are welcome.


Hunger for the integration builds

Hunger for the integration builds I want more different types of integration builds with every passing day. Every integration build checks for something, that could not be checked within the IDE or another build. Integration builds are especially good in detecting dormant issues that hide between the different sub-solutions of a development project, between internal and external code-bases or in some other integration joint. Obviously, increasing number of builds will increase the probability that something will break. And this is good, since that means more consistency and stability in the codebase and less dormant issues. The are constraints, though: Adding too many integration builds, will make the development extremely stable and extremely slow (good for mature projects with the established architecture and logic) Complex integration builds tend to have higher hardware requirements (exponential growth, as usual)


Implementing C# projections for Event Store In one of my previous posts I mentioned migration of some Lokad systems to dedicated Event Store . One of the steps in this migration process involve switch from legacy Lokad.CQRS View projections to new C# projections. New version of view projections does not have smart in-memory replay managed by the system, but they are inherently faster due to batch processing nature. Design is really simple: we subscribe to events from the storage (from last known checkpoint) and pass to projection, batching events together for better performance. After each batch, we update checkpoint. Here's how one projection implementation (maintaining list of comments per account) might look like: public sealed class AccountViewProjection : BatchProjectionFor<AccountView> { public AccountViewProjection(IKeyValueStore store) : base(store) { RequestEventsInBatchesOf(100); } public override void HandleEventsIdempotently(ICollection<object> events) { var comments = events .OfType<AccountCommentAdded>() .ToLookup(c => c.AccountId); if (!comments.Any()) return; // This batch change is fast, but it must be idempotent // for this specific projection type, since underlying // storage does not support transactions spanning multiple keys // Probably we could make this async... comments .AsParallel() .ForAll(g => Store.UpdateEnforcingNew(g.Key, view => { foreach (var added in g) { view.AddComment(added.Comment, added.Manager.FullName); } })); } } This design is shaped by the constraint that we need to work efficiently with dead-simple key-value storage like Azure blob storage (but support simple migration to any database engine). This is caused by the fact that existing Lokad.CQRS projections run use this storage. What we actually do here - for each projection: Start a projection manager as separate runtime task (can be implemented as Task that is retried on failure). Calculate projection version from the codebase. If projection code has changed, then kill all the cached views and reset checkpoint. Subscribe to specific event stream starting from the last known checkpoint. Feed retrieved events in batches to the projection. After processing batch, update the checkpoint. Please, keep in mind: this is the very first version that is not optimised. I'm following mantra: first, make it work, then make it beautiful, then make it fast . Additional notes: Projection can actually specify batch size it's willing to accept. Detection of changes in projection's codebase is done using the same code as in projection rebuilder of Lokad.CQRS (via Mono.Cecil). Since each projection subscribes to event store individually , they are completely independent. However, this could mean a lot of traffic going through. Simple solution would be to have an event store cache per node , which is shared by all projections on the same node. This cache could be either in memory or on disk. See mantra, though. In case of projection failure, it will blow up it's manager. Runtime will restart this task and projection will resume from the last known checkpoint. If we have too many repetitive failures - circuit breaker will kick in to let the system cool down. What do you think? How do you implement your view projections in C#?


Importance of Execution

Importance of Execution Ideas are worth nothing , no matter how smart they might sound like. They are just theory (even if they are based on real-world practice). And we all know, that in theory there is no difference between theory and practice . While in practice such difference exists . The crucial bridge between theory and practice is called execution . Good execution is the key element that can take a simple idea and turn it into profitable and inspiring business. Bad execution can easily ruin a brilliant idea and turn it into a pile of debts and bunch of burned out people. Ideas expressed in this and following texts could be dangerous or reckless, if considered in isolation, without taking a closer look at the execution. For instance, let's talk about rapid delivery at Lokad , where multiple releases to cloud per day are not an outstanding exception. For us it is a way to outrun competition, rapidly respond changing market or solve some unexpected problems . It is one of the reasons customers give us testimonials like this one: Lokad improved the accuracy of our planning process significantly. The immediate impact was a stock reduction of almost 1 million EUR at a monthly cost of 150 EUR. Thomas Bremont , Head of Supply Chain Bizline This might sound like an impossible idea for people coming from enterprise background with highly formalized environment, predictable monthly milestones and precise technical specifications. In fact, this approach is barely applicable in their case (which strengthens our competitive advantage even further). However, while thinking about reliability of these ideas, please consider their execution aspects at Lokad: we test complex behavioral code thoroughly with things like specifications that also serve as living documentation ; even more complex forecasting code has a dedicated multi-machine cloud deployment and benchmarking infrastructure that continuously cross-tests changes in the code against large data library; this tracks performance of forecasting models and allows our brilliant analytics team to push state of the art in forecasting ; they even have their own stack for Windows Azure ; our master branch (in git source control repository) of integration systems is always close to being stable (large changes happen in separate branches); releases are tagged, backed up and deployed according to deployment protocol ; core data is immutable and append-only (persisted as event sourcing streams for behavior-based entities) and hence it is inherently easy to back up or revert any changes; a lot of systems include sanity checks and self-diagnostic routines that help DevOps to detect any potential problems or edge cases; some even have self-recovery logic; most frequent changes in customer-facing systems deal with user experience, and UI is simple in DDD/CQRS systems , especially when changes to data structure behind UI are managed automatically by cloud servers; all of our new systems inherently support hybrid cloud deployments. Newest designs even support real-time replication of data based on event sourcing, since we must become more reliable and secure than any single cloud . As you can see, we simply took brilliant ideas from Greg Young and other members of CQRS community and try to diligently execute them by applying lessons from companies like 37Signals, github and Twitter. In the following articles I will try to address both aspects in parallel: ideas and execution .


Importance of Good Domain Models

Importance of Good Domain Models You might find that I will be writing less about the technical side of event centric systems (with all the CQRS, ES and cloud aspects) along with the problems of scaling and improving their performance. This is because all that is a solved problem in the industry (the only question is about costs and risks of any given solutions). There is a more important problem at hand - coming up with proper domain models that actually represent useful abstraction of the real world and allow event centric systems to be delivered. Or, in other words, coming up with a description of problem domain, that is structured in such a way, that you could easily build a system with that. DDD might sound like less exciting topic for discussion as compared to elastic scalability, clouds or why LMAX can get millions of messages. However the former is just the implementation detail, if you have adequate model. Unless you are doing something really specific with the technology, all problems you would face - have already been solved by many people. That's pure engineering (albeit not the one with a lot of documentation on top of that). You can have more-or-less predictable results, given the initial design in almost any field (aside from areas where technology is so important that it starts melding into the domain field - there really cool things start showing up). Design of domain models is something more of an art or a craft. Given the domain field, you are not guaranteed to arrive at satisfactory model. You are not guaranteed anything at all. Hence it is more of a problem that needs solution. However, if you arrive at something satisfactory, you would get domain model that could easily outlast any technological changes, while even making them less relevant (less expensive and less risky)


Importance of Tooling and Statistics in CQRS World Tooling is extremely important in debugging and managing application. This is especially true for solutions based on Command-Query Responsibility Segregation ( CQRS ) or any other message-based enterprise architecture. That's how, for example, we approach the problem in Lokad: Or, if we take a closer look at the event message (rendered from the binary Lokad Message Format into the human-readable text): Topic: ReportCreatedEvent ContractName: ReportCreatedEvent Sender: https://salescast.queue.core.windows.net/salescast-log Identity: 2c2e7f77-a62d-4540-8079-9db000a2bc47 CreatedUtc: 5245830405928397257 { "ReportId": "646211a6-fe78-4ccc-9536-9db000a2bb9e", "SolutionId": 251, "ReportType": 3, "Name": "Excel Report", "StorageContainer": "https://salescast.blob.core.windows.net/salescast-solution-000251/", "StorageReference": "report-646211a6-fe78-4ccc-9536-9db000a2bb9e.xlsx", "ContentType": "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet", "FileName": "SalescastReport.xlsx", "ReportSize": 8746, "Stats": [ { "Key": 71, "Value": 475 }, { "Key": 72, "Value": 8746 } ] } Tooling helps us here to: better understand and visualize dependencies and interactions within the solution; derive custom reports and run queries against the events that happened during the application lifetime; debug and troubleshoot potential issues that happened in the past; replay commands, script environment and automate certain tasks; capture information needed for discovering and eliminating performance bottlenecks (Stats structure holds primary execution statistics associated with the processing of the command message). The last bullet-point actually replies to the question I've promised to answer a long time ago - on tracking statistics related to the message lifecycle as it passes through the system. So we manually capture the necessary statistics in the context of the message and append them to the appropriate events. Statistics could include things like: Number of records processed; CPU resources used; Upload/download speed; Merge-diff statistics. Etc These statistics might not make a lot of sense on their own. For example, download speed means different things, when we are retrieving information from SQL Azure in North-Europe or mySQL database on shared hosting in Moscow. That's why we are capturing and persisting them along with the domain events, which are bound to the time-line and provide context of the situation. When the event is consumed by some component (i.e.: to update denormalized UI View) we would probably ignore these statistics - they will play out later.. Events are stored in the domain log available for the queries a la Time Machine . Then, when we need to actually figure out some performance bottle-neck or understand specifics of some situation, it'll be just a matter of doing event stream analysis (there's plenty of literature on that one already) and writing proper queries. Once we have statistics in the context of actual domain events, nothing prevents us from getting answers to the questions like: What's the average item retrieval speed from mySQL databases? How often do we encounter timeouts and deadlocks? How many seconds does it take to sync 100k products from SQL Azure database in the same data-center? What's the average upload speed to Lokad Forecasting API for datasets larger than 10k series, after that API upgrade in the last iteration? Theoretically, if we combine these statistics with Time Machine Queries (or continuous queries) and real-time logging - we should be able to do some nice things with CQRS solutions: Monitor health state and attach notifications to the key performance indicators of our distributed solutions. Detect potential problems in real-time or even ahead of time (if coupled with the forecasting). Analyze the real impact of performance optimizations. All in all this helps to better understand the realm of the solution, as it evolves in ever changing reality. This allows to reduce resources (developers, time and budget) required for the delivery, while keeping high quality of project deliverables. This article is a part of xLim 4 Body of Knowledge . You are welcome to subscribe to the updates and leave any feedback!