People Don't Think in Tables

People Don't Think in Tables Within the last few years I've seen major change in UI approaches in the systems I build. The change is caused by a shift from relational databases as persistence to something that does not require an SQL server (or DB server at all). Take a look at this old UI of mine. It's pretty powerful (huge amount of buttons proves that), is intuitive for geeks and scary for normal people. What's more curios, this UI is shaped by the relational persistence model that was created in mainframe days, when storage was expensive, memory even more so, and data had to be heavily normalized. However, technology has advanced slightly since then, and we no longer need to fit our UIs to something that is so spreadsheet-like. We can prevent user from overexposure to underlying data complexity, especially when data itself is simple, but just happens to be stored in a computer-optimized way. Think of it again. We turn UI experience of users in sudoku game just because we know how to store data in a way that was good 20 years ago . No wonder Apple is making so much money - at least they don't try to turn every UI into spaceship control panel: People don't think with tables . They would prefer something that is more simple, even if it does not benefit from a fancy styled UI framework. Something that does not involve solving sudoku puzzle every time you need to find a simple answer to your question or do your job. or where we have the most simple way to search complex structured info, as patented and proven to be useful by google: It is surprising, how underlying technologies affect what we build with them , is not it? As you can guess, first 2 screenshots depict UI that runs on some sort of SQL, while the other UIs were influenced by a simple NoSQL storage model (to be precise: CQRS/DDD+ES model that uses for UI persistence blobs for cloud persistence and plain files for on-premises deployments). BTW, I know that quite a lot of people are using Lucene to provide full text search capabilities for their read models. However in the simplest case (esp. with cloud deployments), it's easier to write a few lines of code as opposed to dealing with one more dependency. Another side effect is that with the change of underlying tech from SQL (and more importantly - thinking model), there is less need to resort to data-aware UI frameworks (mostly used on conjunction with desktop apps) in order to provide rich user experience. This allows to have web-based applications that don't force framework installations (good luck with getting .NET or Java on all machines in some large organization with a lot of momentum) or complex upgrade routines (i.e. ClickOnce auto-update or automatic MSI updates). I'm not talking about development simplicity and cloud-scalability - these come for free, and we don't really value things unless we struggle for them. Obviously, as you can see from screenshots above, my UI skills are still below that of a kid with a box of crayons but lately Bootstrup and jquery have provided a really nice experience, especially, when you can reuse some great ideas and frameworks. Wait, till we start getting into mobile experience with HTML5. It actually comes almost for free as well, since latest web layout frameworks provide a lot of mobile experience out-of-box. NB: In no way I'm implying that this second approach is anywhere near close to being reasonably simple. And it's obviously bad looking (especially UIs that I hack together), but I think this is going in the right direction. At least, less time is wasted on writing UI docs and providing support.


Practical CQRS/Cloud Samples and an eBook: Does This Interest You? One of the things I've learned from the production experience was the extreme value of getting feedback for your project as soon as possible. This might save you wasting a lot of effort, time and resources later on (or even save the project). There is thing that I wanted to share for the same reasons - to get an early feedback. As probably some of you know, in addition to my Lokad work (focusing on Windows Azure, Lokad.CQRS, high scalability and all sorts of fun stuff) I'm currently playing with various learning CQRS+Cloud projects in my free time (or of what remains of it). The goal is to learn continuously and keep personal development ahead of what is required by Lokad projects (this allows to work around potential problems and capture some nice opportunities). Besides, real-world experience also helps to straighten the theoretical conclusions by polishing them with the practice. Some things might look great on paper and diagrams but be completely useless in the production. The only way to find them is by trying. Preferably the first practice should not happen in high-importance production projects, so this is another reason to try and keep learning separate from production. Each logical step or concept that is figured out, helps either to make the next step in scalability, complexity and flexibility of your projects. Or you can just make your existing projects simpler and less expensive. Salescast project, that I've been mentioning a couple of times in this blog, benefits from both. Just like a few others. For example, compare concepts from xLim 1 of year 2007 (at the bottom of the xLim section ) and the latest articles ( xLim 4: CQRS in the Cloud ). I think there was some subtle progress made by figuring out the theory, field-testing it in production and learning from it. Right now I'm thinking about developing a few quick projects , that target some real-world business scenarios in a quick, dirty, but cheap and reliable way. Something like "Practical Cloud with a dash of CQRS". No complex ivory-tower theoretical constructs but rather implementations on top of this logic with hacks, where this makes sense. Eventually I'm planning to release these projects as open source . In parallel I'm also thinking about composing an eBook on the subject with detailed dive into these projects , explanations of how, why, theoretical references and recommendations on handling extreme scalability and complexity cases (backed by the actual Lokad production experience). Basically, a quick learning course about practical Cloud/CQRS the way I would've given it if I were back at the university. For the projects I'm deliberately trying to pick some niche where it's easy to get an application running and there is potential for lots of visitors coming for some free service (hence, creating real random stress, testing and verifying the underlying foundations of CQRS and Cloud development). It'll be probably non-Azure implementation as well (Azure is currently a perfect fit for Lokad, but for an ISV with extreme constraints on resources and friction it currently would not work out that well due to a number of reasons). eBook itself will come for a price and will be more convenient, detailed and guided alternative to finding your way through the xLim/CQRS articles as they jump through various topics (original articles and materials will still be published and shared in the usual manner). Again, just like a learning course. BTW, those that want to start learning CQRS/Cloud right now - you can just check out CQRS references section (links to various videos, blogs, articles and blogs that I've gathered), dive into first samples and tutorials from Lokad.CQRS for Azure and work you way through the xLim series (from bottom to the top). Greg's documents, video and project are probably the most valuable reference of them all . Now these are just thoughts. Nothing is in motion (except for the first learning project which was an inspiration and foundation for the latest articles on CQRS and xLim series). I wonder: What do you think? (even Yay or Nay would be helpful) What questions about practical CQRS/Cloud are the most pressing and important for you know? What do you think about projects? Would such samples (published as open source) help you out in your endeavors and how? Do you have any ideas about areas, where a simple CQRS application would help to solve some problem and would face a bit of scalability stress while dying so (preferably the stress would be dynamic)? There are quite a bit of materials on CQRS. Would you still be interested in such eBook that explains and walks over the theory, practice and experience in a structured and organized way fit for learning (focusing on practice and implementations)?


Processing Big Data in Cloud à la Lokad

Processing Big Data in Cloud à la Lokad Let's talk about a simple approach to visualise, model and deliver complex large-scale data processing tasks. Such tasks would deal with datasets that are so large, that they don't fit into the memory of a single machine and would also take ages to compute on a single machine. These datasets can often be referred to as "BigData". Such tasks would benefit from distributing out the work and storage load between relatively cheap machine instances that are made available in the cloud (either public or "private"). We would also want to optimize our consumption costs by get these resources only when they are needed and releasing afterwards. Let's also assume that such processing task, requires complex sequence of steps in order to complete (more complex than a mere MapReduce). Some steps must be processed before others can start, while others can work in parallel batches. Actual steps of the job are idempotent, messages can be delivered more than once or simply fail. Here is an example of how such processing graph could look like: That's how I would approach such problem in the situation, when: development resources are limited; data processing model is not formally established and is likely to evolve; team (or a single developer) is familiar with event sourcing. I would split the problem domain into two separate bounded contexts: Orchestration and Data Processing. Bounded Contexts Orchestration Bounded Context will be responsible for navigating data processing graph and orchestrating the individual jobs. Behaviors for that will be captured inside an aggregate root that uses event sourcing for persistence (AR+ES) for better testing and getting persistence mismatch troubles out of the way. Deployment-wise, this aggregate can live in a separate machine and would be configured in such a way, that all commands to this aggregate are synchronized and executed on a single thread (just a routing rule for messages). AR+ES just schedules batches of tasks that can be executed an parallel, and issues second batch only when the first one is complete. Should there be any message duplication (always a possibility in the cloud environments), AR+ES can easily track and drop duplicates by keeping hashes of already completed task batch identifiers. Data Processing Bounded Context will be implemented using a set of command handlers that consume work commands from an input queue and process them. These handlers would operate upon data that is stored somewhere in the cloud and is considered to be immutable for the duration of the specific big data process. Commands and events can contain meta-data, parameters and references to this immutable data. In essence, command handler is a function that will take as input a command (which could contain a reference to larde immutable data blob), perform certain operations and publish an event (optionally saving some large processing data into another data blob). Multiple instance of command handlers would be picking commands from the input queue in this bounded context. In essence, they would be competing for the jobs, just like clercs in the bank "compete" for customers standing in line (customer is handled by only one clerk). However, we would be better than a bank, since if we can always massively increase the number of command handlers handling the load, by instructing cloud fabric to provision more machines. Both bounded contexts subscribe to all important domain events of each other. Important: we should differentiate between actual data (which is so large that it does not fit into a single machine/process) and behavioral metadata. The former is accessed only by data processing bounded context, while the latter is passed within the messages between both bounded contexts. For example, number of time series in dataset is metadata, while actual values within these time series are actual 'raw' data. Orchestration Aggregate would use that metadata to make decisions that 'drive' the process through the graph. Flow of work Let's say we have a ProcessAggregate that contains orchestration logic for our complicated MapReduce process. When this aggregate starts, it simply publishes X events that say something TaskAScheduledEvent(processId = 1, taskID = guid) Note: there is more elegant way to do this, but that would require going deeper into DDD These events would be received by Receptor (or Port) in the second bounded context, which would translate them into instances of ProcessTaskACommand . These command messages would be passed into the queue from which multiple worker machines pick their jobs. When command handler finishes processing the task it sends TaskAProcessedEvent , which will get routed back to the ProcessAggregate as ConfirmTaskAResults(task ID = guid) Within the aggregate we: Mark task as confirmed (if it hasn't already been reported due to message duplication). If this task completes some batch and enables further processing, we schedule more tasks for cloud execution. We can also define a timeout view that simply lists all tasks that are currently running. Timeout manager (a simple process) regularly checks this view and sends "TryTimeoutTaskX" to the aggregate. Aggregate checks with it's internal state, and if task indeed has not been processed, decides to either reissue the task or terminate the whole process (yes, we essentially implement our timeout tracking as a business process within Lokad.CQRS architecture style ) Gotchas Advantages of this approach (esp. if aligned with Lokad.CQRS architecture style): no need to worry about persistence of complex object that represents our graph decision logic; orchestration logic can be explicitly tested with the specifications (and documented as such); we can easily migrate between multiple versions of the data process without downtime or stopping processes; process can easily be developed and debugged on the local machine, while being deployed to any cloud afterwards; we use same approaches and ideas that are used within Lokad.CQRS architecture style for modeling more conventional business concepts (this lowers learning barrier and allows to reuse answers to some common problems). Drawbacks of this approach are: It requires certain development discipline (and familiarity with cloud computing and AR+ES); At the moment of writing, there is no prepackaged infrastructure for event sourcing that would work out-of-the-box; performance of this approach would be somewhat inferior to finely tuned functional style map-reduce process implementation; this is a batch-processing approach, which is not fit for real-time processing (yet). In short, with this approach we trade some performance for development and deployment flexibility. This enables us to rapidly model and implement big data process (especially when requirements are still changing). After the process is formalized, we can always fine-tune and optimize bottlenecks. Although frequently you would find that it is cheaper to add another server (worth 100 EUR per month) than waste multiple development days of brilliant developers on performance optimizations. Heads up : the entire infrastructure does not need to be really performant with one exception - if you are doing hundreds of thousands of messages within a single process, then it's worth to invest effort in messaging infrastructure (e.g. direct communication with ZeroMQ), otherwise latency will kill everything. Event stream for actual process aggregate can be simply cached in memory. Deployment Options Below are some deployment variations that could be used within this approach. We can implement our core processing logic without any coupling to specific deployment environment and then deploy in various configurations. The latter would require just re-configuration and optionally providing some specific adapters implementations (for messaging, event sourcing and large BLOB streaming). Alternatively you can have the same project prepared for multiple deployment options from the start. Local development machine: orchestration bounded context runs as one thread; multiple data processing command handlers run either as parallel threads or as multiple instances of a single console app; file system is used for both message queueing, persistence of large binary files and event streams for aggregates. Windows Azure Cloud: Orchestration bounded context runs in a single worker role (e.g. instance of Lokad.CQRS-based engine); data processing handlers run as additional Windows Azure worker roles (you can configure them to run on X different threads within Y worker role instances); Large data is streamed to Azure Blob Storage, just like event streams for AR+ES entities; Azure queues are used for messaging. Amazon Elastic Compute: Orchestration bounded context is a single VM, while Data processing command handlers run within multiple replicas of another VM. We scale by adding or dropping instances of that second VM. Amazon S3 storage is used for persisting large binary data, while local instance of RabbitMQ is used for messaging; event streams could be persisted locally within orchestration VM. Obviously, these are just some of the few options. You can have completely different scenario, based on the specific resources, requirements, risks and constraints within you project. In each of these cases, elastic scaling can be done by implementing a simple task that would watch upon the amount of messages waiting in the command queue of data processing bounded context, adjusting number of command handler instances accordingly. Final Words This approach is in not a silver bullet. It just summarizes some limited experience gained while developing and maintaining non-realtime big data processes that could be hosted both in the cloud and on-premises. As such, it can have numerous applicability limitations (especially if you are working within constrained enterprise environment). Some alternative approaches and references worth mentioning are available in the reading list on Big Data . However, if you need to quickly deliver out some scalable multi-step data process with one person, no money for expensive software licenses and just a few weeks of time, then this approach might give you some ideas. If you want to read more along these lines, here are a few more relevant posts: Anatomy of Distributed System à la Lokad Bird's-eye view of a Distributed System - Context Map Software War Starts with a Map, Context Map DDD: From Reality to Implementation DDD: Evolving Business Processes a la Lokad


Reading List on Big Data

Reading List on Big Data This is purely theoretical blog post to summarize my last few days of studies into the big data (which were triggered by one homeless guy and sequence of highly unprobable events that actually took place). No fancy intro, just assuming that dealing with big data processing is really cool or at least has an outstanding financial reward potential (given the trajectory taken by modern IT and economics). Current Approach at Lokad Or at least - simple part of it, that is not touched yet by secret fairy dust of our outstanding analytics team. In short, big data is bulky, complex, requires a lot of CPU and does not fit in RAM. So we've got to break calculations into smaller batches . We want to process everything as fast as possible, so we push these batches to separate machines . Concurrent processing can cause a lot of problems with race conditions, so we cheat : Batches of big data are immutable once written (that should scare anybody who is coming from the old world where storage was expensive and SQL ruled the world). So we can share them. We keep bulk processing as dead-simple immutable functions, that are woken up by a message, consume some immutable data, do heavy CPU job and then produce output message (which could include reference to a newly created immutable block of data) These immutable functions construct a processing graph of (extremely simplified) map reduce implementation, that can actually do a lot of things. When graph elements need synchronization, we do that via messages that flow to aggregate roots controlling the process. They don't do any heavy-lifting (they don't even touch actual data, but just metadata level information), but encapsulate some complex behaviors to navigate execution through the computational graph. We don't have problems with implementing this seemingly complex part (or testing it), since Greg Young and Eric Evans provided us with CQRS/DDD toolset and some event sourcing . This is a poor-man's architecture that can be implemented by a single developer from scratch in a month. It will work and have decent elastic scaling capacities, provided that you have abundance of CPU, Network and Storage capacities (which are provided by cloud environments these days). This approach is explained in a lot more detail in Processing Big Data in Cloud à la Lokad Potential caveats: Domain modeling should be done carefully from the start. Lay out the computations and think through the algorithms. Cloud queue latency and complexity would be your bottlenecks in the cloud (all the other limitations are solved by adding more VMs) This is a batch-processing approach, which is not fit for real-time processing. Yes, this is a hand-made implementation of MapReduce. How can we improve that? So what are the limitations of the previous approach? Complexity Messaging latency Absence of real-time processing First complexity limitation can be handled by separating the system into separate elements. No, I'm not talking about layers (these will do more harm than good), but rather separate bounded contexts, that have clear: boundaries; language; contracts for exchanging data and communicating with other bounded contexts; whatever choice of technology that is fit. Second limitation of messaging latency can be worked by saying au revoir to any solution with man-in-the-middle architecture (in cloud these implementations are called "cloud queues" or "cloud service buses"). Broker-based architectures are a logical dead-end for the large-scale distributed computations (just like relational databases are for persistence). They limit our scaling capabilities. So we need to rewire our brains just a little bit and leave the world of ACID, transactions and tables (wake up, Neo): eventual consistency ; brokerless architectures ; gossip protocols ; Or, if you want a simpler path, just use Hadoop . Third limitation is lack of real-time processing . You see, this approach to big data is still good old batch processing. It grabs a chunk of data and then takes some time to process it. If we are working with real-life information, then by the movement we finish processing history, we'll have some fresh data that requires recomputation. If this were end of the road, Twitter would never exist. But they cheat. They provide real-time capabilities by incrementally processing new data using some crude and rough algorythms. They are not as precise as batch processing, can make mistakes, can handle only a little bit of data, but they are fast . So whenever a user loads up his twitter profile, he gets result that is composed from thoroughly processed batch data plus whatever changes happened since this slow process started. The latest bits might not be precise (i.e. they could miss a few mentions from fellows in the other hemisphere and also include a few spam-bots), but they are almost real-time, which is what matters. And actually as the time goes, results will be magically corrected , because a little bit later batch map reduce algorithms will get there and replace fast approximations with slow but thorough results. Twitter Storm incorporates end explains this dual approach to big data in greater details. Edge Cases People say that there are no silver bullets (which is actually wrong, you can buy that stuff on internet ) and hence one approach to big data will not fit all cases. Let's ignore total absence of logic in this statement and focus on potentially specific edge case of big data that might benefit from a separate approach. I'm talking about event streaming. Enterprise companies consider event streams this to be a rather complex scenario, when "a lot of events come in a short amount of time". Such complexity even created niche called Complex Event Processing which sounds like really complicated and expensive field. Partially this is justified because events often require proactive reaction to something that happens in real-time. Yet this reaction could depend upon the events that happened millions of events back (this means "a lot of" data has to be stored). From now on, please replace "a lot of" with the term that you will hear in 12th episode of distributed podcast which was recorded this weekend. Let's see if there is a simple poor man's approach to handle "a lot of events". We will need following ingredients. First ingredient is fast storage , that is dead-simple, provides immediate access to any data by key (at most two disk seeks for cases, where entire key index does not fit in memory). We don't ask for ACID or query language here, but ease of replication is definite plus. Basho bitcask and Google's SStable provide extensive guidance here. By the way, IndexDB in WebKit uses LevelDB, which is based on SSTable. Second ingredient is distributed architecture design that would support arbitrarily large amount of nodes operating in a single prepartitioned ring-shaped hash-space, where nodes can come, fail and go as they wish. Data should survive no matter what (and be easily available), while scaling can be elastic. If this sounds a bit scary for you, don't worry, there is plenty of material (and source code) from: Basho Riak Core Project Voldemort Father of them all - Amazon Dynamo Third ingredient is cloud computing , that would provide CPU, Network and computing resources needed to power that architecture, while not requiring any upfront investments. I've been blogging about this aspect for quite a few years for now. At this point in time we already have a number of cloud computing providers with some competition between them. For example: Windows Azure Cloud; Amazon AWS; Rackspace Cloud. Competition in this area already drives prices down and rewards efficient use of resources. Fourth ingredient is about lower-level implementation principles that fit into the distributed environments, append-only persistence and eventually consistent world that even tolerates conflicts. Great head-start in this area is provided by ZeroMQ tutorial and filled by the mess of patterns and practices united by the name of CQRS architectures . Since these CQRS-based principles actually provide a uniform mental model for dealing with events and events streams in distributed world (unless you are playing it old-school with SQL) along with techniques of designing and modeling such abstractions (known as DDD ), they already give solutions to most common problems that you might face, while bringing together in practice distributed architectures, fast streaming key-value stores and cloud computing. Fifth ingredient is the Force. May it be with you, in this new but certainly exciting field :) [2012021402321337237189] Memory dump complete...


Recent Lessons Learned in Lokad.CQRS

Recent Lessons Learned in Lokad.CQRS As you probably already know, Gabriel Schenker has started a series of blog posts on How we got rid of the database . One or two concepts in Gabriel's approach originate from Lokad.CQRS Sample Project . It was really inspiring to see these ideas put to some practical use and validated outside of Lokad. It also provided a nice retrospective which helped to see, what changed in Lokad.CQRS recently. Given that, I want to share some of the changes that actually happened to Lokad.CQRS approach (mostly based on DDDSummit 2012, work with Vaughn Vernon and also recent experience in new projects). Automated projection management is a huge life-saver (rebuilding views that have a projection code changed or creating new views). It vastly simplifies life both in development and production, enabling new scenarios. For example, it becomes possible to get immediate feedback for your UI tweaks in development. Joining command handlers into Application services (similar to how we join multiple event handlers into a projection class) helps to make them more consistent. This makes solution more structured, when you get dozens of commands. Development workflow noticeably speeds up, when you auto-generate (and auto-update) handler interfaces for your aggregates and their states, based on some contracts DSL (provided you use event sourcing there). DSL sample in Lokad.CQRS already does that. As your projects grow (in complexity and load) it becomes apparent, that handling event streams as individual files/blobs might not be efficient. For instance, try copying 100000 individual event stream files from the cloud for some debugging (hint: latency will be the pain). So at Lokad we gradually shift towards operating with event streams per bounded context . These are equivalent to domain logs in older Lokad.CQRS or to transaction logs in other database solutions. In essence, bounded context stream is an aggregation of all event streams within a given bounded context, where individual event streams are identified by a name but are still stored together. This makes it extremely simple to replicate or to write (we can also preallocate some disk space to reduce enormous fragmentation). Reading individual streams can be problematic, unless you store in memory either full caches or at least indexes. In other words, we just slightly change storage approach to keep all separate event streams together. First draft implementations of this approach are covered in stream implementations provided in IDDD sample . They are based on Riak Bitcask. Since publishing them, we have slightly improved on that in out internal code. I hope to bring all these improvements back into Lokad.CQRS soon (when starting to implement proper DDD sample). These changes are are well aligned with both Greg's future event store (he's making awesome product, by the way) and future book by Vaughn. Having said that, per-file event stores are still a valid and nice approach for local development purposes and low-volume production scenarios. Domain Services become really nice tool to exchange information between different aggregates in eventually consistent way. Before, I would try to somehow wire all information from one aggregate to another via event-command chain (making them really bulky and sending around a lot of useless info). Now, I tend to define view projection that is used within core bounded context. This projection is auto-managed (meaning I can change it any way, and server will rebuild it as necessary) and is populated by events coming from some aggregates. Another aggregate accesses it via a domain service that knows how to query this view. Value objects become really important in expressing domain logic. It is amazing, how simply writing a descriptive concept class (i.e.: email, forecast settings, service level variable etc) makes the rest of the code more compact and robust. As you can probably see, majority of changes affect either DDDesign principles on top of Lokad.CQRS or some subtle implementation details underneath. Core abstractions and ideas (i.e. document storage, Aggregates with event sourcing) seem to stay relatively stable and practical.


Recommended Reading on Big Systems

Recommended Reading on Big Systems Here's a list of videos and resources I've been studying from recently (more valuable - on top): Immutability Changes Everything by Pat Helland at Ricon 2012 Real-Time delivery architecture at Twitter by Raffi Krikorian at Strange Loop 2012 Runaway complexity in Big Data, and plan to stop it by Nathan Marz at Strange Loop 2012 Condos and Clouds - patterns in SaaS applications by Pat Helland at ACM Webinar Advancing Distributed Systems by Eric Brewer at Ricon 2012 Other Ricon 2012 talks Big Data architecture at LinkedIn - an interview by Sid Anand On the overall, I'm really impressed by the amount of innovation and sharing that happens in the industry.


Redirect Tcp Connections in Windows Azure I've just published a quick and extremely simple open source project that shows how to redirect TCP connections from one IP address/port combination to another in Windows Azure. It is sometimes helpful, when dealing with SQL Azure, cloud workers, firewalls and the like. Lokad Tcp Tunnel for Windows Azure | Download Usage is extremely simple: Get the package. Configure ServiceConfiguration to point to the target IP address/port you want to connect to (you can do this later in Azure Developer's Portal). Upload the Deployment.cspkg with the config to the Azure and start them. Connect to deployment.cloudapp.net:1001 as if it was IP:Port from the config. If you are connecting to SQL Server this way (hosted in Azure or somewhere else), then the address have to specified like this in Sql Server Management Console (note the comma): deployment.cloudapp.net,1001 Actual Azure Worker config settings should look similar to the ones below, when configuring TCP Routing towards SQL Server (note the 1433 port, that is the default one for SQL): <ConfigurationSettings> <Setting name="Host" value="ip-of-your-SQL-server" /> <Setting name="Port" value="1433" /> </ConfigurationSettings> The project relies on rinetd to do the actual routing and demonstrates how to: Bundle non .NET executable in Windows Azure Worker and run it. Deal with service endpoints and pass them to the processes. Use Cloud settings to configure the internal process. Since core source code is extremely simple, I'll list it here: var point = RoleEnvironment.CurrentRoleInstance.InstanceEndpoints["Incoming"]; var host = RoleEnvironment.GetConfigurationSettingValue("Host"); var port = RoleEnvironment.GetConfigurationSettingValue("Port"); var tempFileName = Path.GetTempFileName(); var args = string.Format("0.0.0.0 {0} {1} {2}", point.IPEndpoint.Port, host, port) File.WriteAllText(tempFileName, args); var process = new Process { StartInfo = { UseShellExecute = false, RedirectStandardOutput = true, RedirectStandardError = true, CreateNoWindow = true, ErrorDialog = false, FileName = "rinetd.exe", WindowStyle = ProcessWindowStyle.Hidden, Arguments = "-c \"" + tempFileName + "\"", }, EnableRaisingEvents = false }; process.Start(); process.BeginOutputReadLine(); process.BeginErrorReadLine(); process.WaitForExit(); Tcp Tunnel for Azure is shared by Lokad in hopes that it will save a few hours or a day to somebody.


Replicating events to remote EventStore

Replicating events to remote EventStore At the moment of writing majority of event-sourced systems at Lokad run at simplified event store from Lokad.CQRS library. We are migrating to Greg's Event Store . In our setup it is hosted on Azure . Migration process must ensure that we don't break any existing systems. This means that we can't simply jump and switch all code to use new storage. Instead, the migration process has to be incremental , like navigating deep waters. After all, running single node on Azure-backed VM, is not supported scenario (although it works). So we implement change in the least important part of the system, deploy it and observe. There will be some failures and lessons learned. Then, we'd incorporate these into the code and make the next step. So far sequence looks like this: Replicate events from production to dedicated event store server on Azure; switch non-critical projections to chase streams on dedicated event store server; migrate aggregates to use dedicated event store server; where possible, discard server-side command handlers and host aggregate logic directly in Web UI. One of these initial steps involves replication of events from production event store to new server . This can be achieved by continuously polling local event store for changes and pushing any new message batches as individual events to remote event stream. For the time being to keep things simple I'll be pushing all events to the single stream. Later on we could relink them to individual event streams. Actual replication code runs in a separate thread. It is just a "while(true)" loop: using (var conn = Connect(_config)) { conn.Connect(); connectFailure = 0; var version = GetLastReplicatedEventFromRemoteStore(conn, stream); var lastReplicatedEvent = version.LocalVersion; var remoteVersion = version.RemoteVersion; while (!token.IsCancellationRequested) { if (lastReplicatedEvent == _store.GetCurrentVersion()) { // no work to do, so sleep and continue token.WaitHandle.WaitOne(500); continue; } var keys = _store.ReadRecords(lastReplicatedEvent, 10000).ToList(); var remoteEvents = keys.SelectMany(ConvertToNewEventFormat).ToList(); conn.AppendToStream(stream, remoteVersion, remoteEvents); lastReplicatedEvent = keys.Last().StoreVersion; remoteVersion = remoteVersion + remoteEvents.Count; } } In order to deal with possible transient failures of the remote event store (this is possible, since I'm running it as single-node for now), we put wrap entire replication process into another loop: void RunReplicationProcess(CancellationToken token) { int connectFailure = 0; while (!token.IsCancellationRequested) { try { var localStoreIsEmpty = _store.GetCurrentVersion() == 0; if (localStoreIsEmpty) { token.WaitHandle.WaitOne(TimeSpan.FromSeconds(30)); continue; } using (var conn = Connect(_config)) { conn.Connect(); connectFailure = 0; /* replication from snippet above */ } } catch (Exception ex) { if (connectFailure == 0) { SystemObserver.Notify("Event store connection failure {0}", ex); } connectFailure += 1; token.WaitHandle.WaitOne(TimeSpan.FromMinutes(1)); } } } Once this code is in production and production event streams are replicated, we could migrate less important projections (like the ones backing up Admin Web Apps used by Lokad team members) to new event storage.


Roadtrip-2011 Lessons Learned at

Roadtrip-2011 Lessons Learned at In short - this spontaneous Event centric weekend in Kiev was totally awesome and rewarding . Check out tweets , if you are interested in what others feel (people are talking about sparkles in the eyes and #eventcentric cult :) This event was organized as a part of Road-Trip 2011 with Greg Young. There were quite a few other adventures that happened along the way earlier - too much to write about. So I'll just focus on the last weekend and lessons learned. Event Centric weekend in Kiev was a two day class held for free in Kiev. Greg Young was doing his well-known CQRS/DDD course, I've got to share some of my practical experience with such systems "in the wild" (including various cloud options). Room was packed with 90+ people from multiple Ukrainian cities and also from a few cities in Russia. Luxoft Ukraine helped us immensely by providing with the place to do that. Irina Odina of Luxoft deserves a medal (or two raises in salary) for staying with us through the weekends and helping with the coffee breaks. Mike Chaliy and really friendly Kiev Alt.NET community were helping with afterparties, logistics and coping with unexpected. Here's what I've learned during this event: One of the best ways to strengthen your understanding in a field is to speak and share . For instance, I've learned a lot simply by explaining aggregates and sagas to the audience and then seeing that not everybody gets a clear and coherent picture. Or, when you get an question from an audience, and you know an answer to that question, but can't explain this immediately in clear and concise language immediately. Then Greg comes in with his explanation and reveals missing pieces and steps in my own explanation and understanding. Showing my own hacks to the theory and practice (and sharing how everything works really nicely in distributed teams and cloud environments) is always a pleasure of its own. There were a few smaller realizations of its own along the way. The most important one of them was about more clear and focused picture of bringing multiple complex systems together in a way to limit complexity and facilitate development (obviously, with limited time and resources). Basically that's the a-ha understanding of how to separate elements of a distributed systems (bounded contexts, sagas, services, cloud scalability points), while keeping each element highly specialized (and relatively simple). In essence, this is a direct analogy with human body, where: Bounded contexts (matching aggregate roots in extreme) == neurons that are completely isolated from each other, but can exhibit complex behaviors and have a short-term memory; Sagas (stateful or stateless) == glial cells that protect and insulate neurons from each other, while connecting them; Stateless services (integration, number crunching, file processing etc) == specialized organs ) in a body that can get some job done, but tend not to have any memory on their own. They react to impulses and send back information about the events that happened to them. This analogy (coupled with the basic principles of biology) actually explained me better how to build and evolve really interesting IT systems, while keeping core cells simple. In short - that's just like how evolution does this. That was a really important realization since before that I simply could not understand how to bring together behavioristic CQRS-driven systems with elastically scalable functional elements of big data crunching or messy technology-dependent integrations. This understanding comes just in the right time - as we are starting yet another project at Lokad these days. I'll definitely blog about that more of that later. A few smaller realizations were coupled to Sagas (explaining them better, to be precise) and transition from domain models to Aggregate Roots with Event Sourcing via TryDo-Do logic (nothing essential to the production but still helps to see and explain better) By the way, the road-trip is not other yet. There is still a lot to learn and do. So far, thanks to everybody for helping to learn and have some wonderful time!


Rule of Time Limiting

Rule of Time Limiting I don't know who has passed this feedback, but I'm extremely grateful to this person. It's relatively easy to get a good feedback, but constructive ways of improving yourself are priceless. The cardinal rule of any opportunity to present is this: Thou shall stick to your time limit. If it is being pointed out that you are overtime, then you will stop immediately. Anything else shows disrespect and has the aura of "I am more important than you, just suck it up" Thanks, man. This will also help to stay focused with BTW Podcast.


Rx works nicely with DDD and Event Sourcing Yesterday I finally got to the point of trying to build Command-Query Responsibility Segregation solution for Cloud Computing environment that uses Domain Driven Design and Event Sourcing ( quick overview of these buzz words ). The whole purpose of the project is to learn, so there's a freedom to experiment upon the accepted patterns and practices. Mark Nijhof put together really thorough article on the Domain Events in CQRS. It has a lot of code and information packed in. Let's take it from there, but with a slightly different and highly experimental route. Usually Aggregate root implementations have all sorts of interfaces and methods helping the surrounding infrastructure to pick them up and handle. This makes the domain code look a bit complex for me. However it seems that these AggregateRoots are a native candidate for leveraging Reactive Extensions for .NET . AR might need to inherit and implement just a single interface (snapshots might require the second one, though): public class ProjectAggregateRoot : ISubject<IEvent,Change> Where ISubject is an interface from System.Reactive that merely says: public interface ISubject<in T1, out T2> : IObserver<T1>, IObservable<T2> How does it affect our aggregate design? Not much, we just slightly extend the "Apply Event" method, naming it "OnNext" and adding the ability to accept IEvent and publish Change : Subject<Change> _subject = new Subject<Change>(); public void OnNext(IEvent value) { EventInvoker.Apply(this, value); Interlocked.Increment(ref _version); _subject.OnNext(new Change(_version, value)); } public IDisposable Subscribe(IObserver<Change> observer) { var subscribe = _subject.Subscribe(observer); return subscribe; } Note: As you know, Reactive Extensions were designed for the asynchronous operations (i.e.: cloud computing interactions or UI events). Thus they work tightly with PFX. However, since I don't know anything about this integration and side effects, I'm dispatching and executing everything in sync so far. Change is just an event that was applied and thus has a version number: public sealed class Change { public readonly long Version; public readonly IEvent Event; public Change(long version, IEvent @event) { Version = version; Event = @event; } public override string ToString() { return string.Format("r{0:####}: {1}", Version, Event.ToString()); } } What does this give us? We can drop all event subscription and management interfaces and functionality, letting Linq-2-Events to do all the handling in rich and tested way: LoadFromHistory methods are equivalents of IObserver[TEvent] GetChanges functionality is fulfilled by subscribing to IObservable[Change] What's more important, our aggregate can avoid referencing any custom interfaces, since both IObservable and IObserver are in .NET 4.0 BCL. This also provides a wide range of extension methods available. Given that, it becomes rather simple task to write store that works like this: // somewhere in IoC init var store = new HybridStore(serializer, storage); store.RegisterEventSource<ProjectAggregateRoot>(); // somewhere in the handler we perform atomic update // if there is a concurrency problem, service bus will be // responsible for reapplying changes later store.Update<ProjectAggregateRoot>("project123", e => { e.AddTask("t1", "Satori Project"); e.AddTask("t2", "Build Simple Domain"); e.AddTask("t3", "Build Simple Sync reader"); e.CreateTaskReference("t2", "t1"); }); For the sake of consistency, here's how the actual persistence (highly prototypical) looks like: public void Write(Type type, object key, AddEntityDelegate addEntityDelegate, UpdateEntityDelegate updateEntityDelegate) { var item = MapTypeAndIdentity(type, key); Func<object, ISubject<IEvent, Change>> factory; if (_factories.TryGetValue(type, out factory)) { var condition = StorageCondition.None; var domain = factory(key); var changes = new List<Change>(); using (domain.Subscribe(changes.Add)) { try { item.ReadInto((props, stream) => { var source = (Change[])_serializer .Deserialize(stream, typeof(Change[])); foreach (var change in source) { domain.OnNext(change.Event); } condition = StorageCondition.IfMatch(props.ETag); }); } catch (StorageItemNotFoundException) { } var version = changes.Count; using (domain.Subscribe(_subject)) { updateEntityDelegate(key, domain); } if (version == changes.Count) return; } try { item.Write(stream => _serializer.Serialize(changes.ToArray(), stream), condition); } catch (StorageConditionFailedException ex) { var msg = string.Format( "Record was modified concurrently: '{0}'; Id: '{1}'. Please, retry.", type, key); throw new OptimisticConcurrencyException(msg, ex); } } } Since we are forced to deal with the event streams, it's easy to subscribe and do things like: using (store.Subscribe(Console.WriteLine)) { store.Update<ProjectAggregateRoot>("project123", e => { e.CreateTaskReference("t2", "t3"); }); } The listener subscription above will reveal us that creating task reference in this case actually results in two events: r5: Domain.TaskRemovedFromParentTask r6: Domain.TaskAddedToParentTask Since Reactive Extensions for .NET are built upon the event streams I think that doing all sorts of related operations operations (writing event denormalizers, merging streams, writing behavioral unit tests) might be simplified in Domain Driven Design with the Event Sourcing. Such abstraction allows to separate different concerns rather clearly. For example the actual implementation of the underlying storage passed to HybridStore could be FileStorageContainer or BlobStorageContainer for Windows Azure from Lokad.CQRS since both allow atomic updates and optimistic concurrency locking (or any reasonable RDB or NoSQL). It's interesting to note, that these storage container implementations were actually developed for the project without any hint of DDD or ES. Yet, since we have such a nice persistence ignorance in CQRS/ES architectures, they could be plugged in without problems. Also, it is rather simple to implement partitioning in this scenario, since all access to the storage goes with the entity key (identity). If you add into the mix cloud infrastructure capable of doing dynamic repartitioning (i.e. Windows Azure storage) and providing computing capacities on demand - you'll get foundation for building almost-infinitely scalable solution (although message queues still need proper support for transactional sending and ACKs). CQRS will provide high-level approaches for shaping the architecture and evolving it in scalable and cost-effective manner; while DDD holds the methodology for designing and managing the business core of the solution. All in all, future looks quite exciting , doesn't it?


Salescast - Scalable Business Intelligence on Windows Azure Yesterday we finally released first version of Salescast . There is an official announcement from Lokad. In this article we'll talk a bit about how it was built , focusing on the technology, Windows Azure and what this means to customers in terms of business value . What does Salescast do? This web application offers smart integration between various inventory and sales management solutions and Lokad Forecasting Services . Basically, if you have some eCommerce shop that you want to run through some analytics and get forecasts on the future sales, Salescast can help out and handle the routine. It will detect your solution, retrieve the data, process and upload it to Lokad Forecasting Services and assemble the results into nice reports. This decision support comes with some extra integration experience on top of that to automate and streamline the process further: Reduce inventory costs and over-stocks. Improve customer satisfaction. Increase overall sales. Ease relationships with suppliers. How was it Built? Salescast is running on Windows Azure platform and was architected to take full advantage of the cloud computing features it provides. For Lokad this meant coming up with the architecture principles, development approaches and frameworks that could allow to leverage all the potential efficiently. Principles were based on the adaptation of CQRS and DDD principles for the Windows Azure, great tools and frameworks that Microsoft provides with it. In order to fill some (temporary) gaps in this ecosystem, custom Enterprise Service Bus for Azure was created. We are planning to share experience, technological principles and frameworks with the community, just like it has been done with Lokad Shared Libraries , Lokad.Cloud and the other projects. For those, who are following xLim line of research and development , this will match with version 4: CQRS in the Cloud . At the moment, let's focus briefly on the business value all this technology creates for the customers. Handling any Scale Salescast has implicit capabilities of scaling out . It does not matter, how large is the inventory of customer or history of the sales. Starting from a few hundred products, up to hundreds of thousands and above - Salescast can handle it. So if you are a large retailer, you don't need to sign a large contract in order to just try how the solution works for your network of warehouses. Neither you need to wait for development teams to scale architecture and procure the hardware resources. It's all already there, powered by the scalability of CQRS architecture, ESB for Azure and virtually unlimited cloud computing resources of Windows Azure. Anticipating the Unexpected Salescast can work reliably and incrementally with any data sources. If sales and inventory management solution is persisted within SQL Azure - great. If it is an eCommerce shop running on mySQL in shared environment that tends to timeout from time to time - we are designed to handle it without stressing the other end. We understand that unexpected things can happen in the real world . More than 95% of possible problems will be handled automatically and gracefully. If something goes really wrong (i.e.: eCommerce server is changed or firewall rules - changed), then we'll know about the problem, will be able to handle it on case-by-case basis and then continue the process from where it stopped. Principles of enterprise integration and reliable messaging with ESB on top of Azure Queues help to achieve this. Designed for Evolution Salescast is designed for evolution. We understand, that customers might use rare systems, custom setups or unique in-house solutions. They could need some specific integration functionality in order to handle specific situations. In order to provide successful integration experience in this context, Salescast will have to evolve and adapt, get smarter. In fact, future evolution of this solution is already built into the architecture and implemented workflows. For example, if there is some popular eCommerce solution that we didn't think of integrating with, we'll teach Salescast how to handle it, for free. The next customer that attempts to optimize sales managed by a similar solution, will get it auto-detected and integrating instantly. This applies for the new versions and flavors of these systems as well. Basic principles of efficient development, Inversion of Control , pluggable architecture and some schema-less persistence helped to achieve this. Domain Driven Design played a significant role here as well. Cost-Effective Salescast is designed to be cost-effective . In fact, it's effective to the point of being provided for free. This comes from the savings that are passed to the customers. They are based upon: environment allowing to have efficient and smart development that is fast and does not require large teams; efficient maintenance that is automated and requires human intervention only in exceptional cases; elastic scaling out that uses only the resources that are needed; pricing of the Windows Azure Platform itself. Obviously you still need to pay for the consumption of Lokad Forecasting Services. But their pricing is cost-effective , as well (to the point of being 10x cheaper than any competition). So there are some tangible benefits for the money being spent. Secure and Reliable Salescast, as a solution, is based on the features provided by the Microsoft Windows Azure Platform. This includes: Service Level Agreements for the computing, storage and network capacities. Hardware reliability of geographic distribution of Microsoft data centers. Regular software upgrades and prompt security enhancements. Lokad pushes this further: Secure HTTPS connections and industry-grade data encryption. Redundant data persistence. Regular backups. Reliable OpenID authentication. Summary This was a quick technological overview of Salescast solution from Lokad, along with features and benefits it is capable of providing just because it is standing on the shoulders of giants . Some of these giants are: Windows Azure Platform for Cloud Computing and the ecosystem behind. Time-proven principles of development and scalable architecture. Various open source projects and the other knowledge shared by the development community. Lokad will continue sharing and contributing back to help make this environment even better. From this point you can: Check out official public announcement of Salescast and subscribe to Lokad company blog to stay tuned in for a company updates. Subscribe to the updates of this Journal on Efficient Development Check out xLim materials to see what is already shared within this body of knowledge on efficient development . I'd also love to hear any comments, thoughts or questions you've got!


Scalable and Simple CQRS Views in the Cloud In this article we'll talk briefly about the CQRS views (read model) implementation for the Cloud Computing environment. This material is based on a research and development at Lokad (which took more than a month for this case), followed by the actual production usage of the concepts. So this is something validated by real project. The article continues Cloud CQRS (xLim 4) R&D articles. Here are some of the previous highlights (see xLim page for a lot more references): CQRS Architecture and Definitions Command Handlers with Various Levels of Reliability Theory of CQRS Command Handlers: Sagas, ARs and Event Subscriptions CQRS Lifehacks from Lokad Production - Part 1 CQRS Lifehacks from Lokad Production - Part 2 CQRS Lifehacks from Lokad Production - Part 3 Let's get started with the basics. Basics CQRS Views are also sometimes called "denormalized views" or "read models" in the world of distributed systems. They usually represent data that is: stored somewhere (anywhere, starting from SQL tables to cloud-based key-value storage); optimized for reads from the perspective of simplicity, scalability, or both ; is populated by event handlers that subscribe to events coming from the domain (and hence is eventually consistent in most of the cases); is a logical dual of data caching mechanism (in classical layered architectures), with the primary difference being: here we don't retrieve information on first request, but rather pre-populate view store immediately after the corresponding event is published; this potentially leads to better performance and less staleness, as compared to caching; optimized for the reads and hence reduces performance and complexity stress on the write side . It is important to note, that in order to get these benefits, you don't necessarily need to employ messaging infrastructure or have full-blown CQRS architecture. For example, even in a classical 3-tier architecture you can do something like: have a separate set of SQL tables optimized for reading, that are populated along with saving changes to the data tables (implementation will be somewhat tangled and complex, but it might be worth to improve performance of a few select tight spots); push JSON files to CDN for direct consumption by the browser client; have a distributed cache that is kept up-to-date by subscribing to data change notifications being published from the write side. However, only by explicitly decoupling our business logic from the read models by domain events, can we achieve overall simplicity, while not sacrificing the scalability opportunities. Views with Cloud Flavor Cloud computing environment creates additional challenges and benefits for the distributed systems. We gain immense benefits of: development flexibility (ability to procure and deploy any combinations of systems); cost efficiency (paying only for what you use); elastic scalability (storage, queues and virtual machines can automatically scale); reduced operational burden (things like scaling, backups, configuration - are managed by the cloud service provider). This, for example, allows a small company like Lokad to do really interesting things with a tiny development team. The primary limitation of the approach - it requires a shift from the DB-oriented mindset towards the mindset of distributeds system in eventually consistent world . Fortunately, CQRS Architecture approach creates a simple foundation for such way of thinking. So, when we combine Cloud Computing environment with the CQRS Views (read models), one implementation option could be: Cloud Views are stored in Windows Azure Blob Storage (Amazon S3, or any storage with key-value capabilities) as simple deserialized files with arbitrary schema. View Handlers (domain event consumers responsible keeping views up-to-date) are just some running server processes subscribed to certain domain event messages. They are hosted within Azure Worker Roles, Amazon EC2 instances, Rackspace VMs or whatever. Client-side, in order to display cloud views, just needs to know view contracts and be able to read them from the storage (usually available via REST in the cloud environments). Technology: Azure, JSON, Simplicity In Lokad production scenario implementation details are: Azure blob storage, being populated by Lokad.CQRS Event handlers running in Azure Worker Roles. Clients, consuming cloud CQRS views(Web and desktop) perform user authentication on their own and directly access Azure Blob storage (more elaborate security models could easily be added as needed). Serialization format - JSON (just for the sake of readability and simpler debugging), it could easily be swapped for ProtoBuf for more compact encoding. Logically views are cheap and disposable. Development infrastructure automates complete and partial view regeneration (performance is not an issue here). Whenever a view structure is changed or a new one is added - just repopulate the corresponding data. Code implementation makes view handling logic part of the development language, it was designed to simplify automated view repopulation, maintenance and operations (just an attempt of proper OO programming that worked surprisingly well). Since cloud views are based on the key-value abstraction (i.e.: given a key, you get a view, but you can't list or query them), some concerns had to be handled explicitly. So secondary indexes and lists are just eventually consistent views as well. From the mental perspective, transition from NHibernate-hosted views resulted in: there is more code, but it is simple, clean and completely decoupled; development friction reduced significantly: no need to handle SQL update scripts for the read side in production/staging/development, simpler deployments, more freedom to refactor and experiment for simplicity; since views are disposable, and it is extremely easy to rebuild them for the production (a few mouse clicks), client UI is easier to evolve towards more usable implementations; less worries about operational costs (Azure blob storage is roughly 100x cheaper that SQL Azure for this kind of scenario); absolutely no worries about the scalability - Azure blob storage scales automatically and is optimized for the reads (plus we can always plug the CDN around the globe or replicate views across the datacenters, if needed); reduced complexity on the clients (no need to use some weird API or bring in NHibernate, it's just true POCO) and the server. The primary problem with the new approach - since now I have reduced development friction, I want to roll out production upgrades more often. 15 minute upgrade delay for Azure Worker Role becomes a frustrating experience. Implementation Details Actual implementation is dead-simple. It's not even fair to call it a framework - just a few interfaces and a simple wiring. Design and Windows Azure do the actual heavy-lifting. Although eventually "framework" will be included into the Lokad.CQRS for Azure open source poject, here are some details to give you better perspective. Views are defined as simple POCO classes , that can have any structure as long as they are serializable: public class UserDetailView : IViewEntity<long> { public virtual long UserId { get; set; } public virtual long AccountId { get; set; } public virtual string Username { get; set; } public virtual string Email { get; set; } public virtual string RegistrationToken { get; set; } } Base view interfaces are just a convenience for stronger typing and automated regeneration: /// <summary> /// View entity that has an identity (there can be many views /// of this type) /// </summary> public interface IViewEntity<TKey> : IViewBase {} public interface IViewBase{} /// <summary> /// View singleton (there can be only one document). /// </summary> public interface IViewSingleton{} Serialized view will look just like a file "view-userdetailsview/150.json": { "UserId": 150, "AccountId": 74, "Username": "abdullin", "Email": "some email", "RegistrationToken": "some identity" } View entities are stored in folder named after type of the view (derived automatically), singleton views (have only instance per view type) are grouped together in a single folder as well. Secondary indexes are just serialized dictionaries (simplest thing that works) saved as view singletons. If there ever will be performance problems with index sizes, there are multiple dead-simple improvements to make. So far we stick with: public sealed class UserByRequestIndex : IViewSingleton { public readonly IDictionary<Guid, long> Index = new Dictionary<Guid, long>(); } Event handlers responsible for populating views are dead-simple and auto-wired: public sealed class UserDetailViewHandler : ConsumerOf<UserCreatedEvent>, ConsumerOf<UserActivatedEvent>, ConsumerOf<UserDeletedEvent> { readonly IViewWriter<long,UserDetailView> _operations; public void Consume(UserCreatedEvent message) { var view = new UserDetailView { AccountId = message.AccountId, UserId = message.UserId, Username = message.Username, Email = message.Email, RegistrationToken = message.RegistrationToken, }; _operations.AddOrUpdate(message.UserId, view, x => { }); } Accessing views is dead-simple as well. Say, in Web client, powered by ASP.NET MVC 2, you have this hook to IoC Container and IViewReader: public static class AzureViews { static Maybe<TView> Get<TKey,TView>(TKey key) where TView : IViewEntity<TKey> { return GlobalSetup.BusInstance.Resolve<IViewReader<TKey, TView>>().Get(key); } static TSingleton GetOrNew<TSingleton>() where TSingleton : IViewSingleton, new() { return GlobalSetup.BusInstance.Resolve<IViewSingletonReader<TSingleton>>().GetOrNew(); } Then querying a view by a primary key just becomes a matter of: public static Maybe<UserDetailView> GetUser(long userId) { return Get<long, UserDetailView>(userId); } Or, if you need to access it by an eventually consistent index: public static Maybe<UserDetailView> GetUserByRequest(Guid requestId) { return GetOrNew<UserByRequestIndex>() .Index.GetValue(requestId) .Combine(GetUser); } If the syntax looks a bit weird - just check out Lokad Maybe helpers . For the sake of completeness, here's the design of view reader and writer interfaces (they are simple but took the most time to get them right, will likely evolve further): public interface IViewWriter<TKey, TView> where TView : IViewEntity<TKey> { void AddOrUpdate(TKey key, Func<TView> addFactory, Action<TView> update); void AddOrUpdate(TKey key, TView newView, Action<TView> update); void UpdateOrThrow(TKey key, Action<TView> change); bool TryUpdate(TKey key, Action<TView> change); void Delete(TKey key); } public interface IViewReader<in TKey, TView> where TView : IViewEntity<TKey> { Maybe<TView> Get(TKey key); TView Load(TKey key); } public interface IViewSingletonWriter<TView> where TView : IViewSingleton { void AddOrUpdate(Func<TView> addFactory, Action<TView> updateFactory); void Delete(); } public interface IViewSingletonReader<TView> where TView : IViewSingleton { Maybe<TView> Get(); } These interfaces make it extremely simple to implement automatic view discovery and wiring for the purposes of infrastructure and management: Here's how the tree was built: Inspect the assemblies for all event handler types. Since ConsumerOf[T] inherits from IConsume, it is just a matter of selecting non-abstract types deriving from the consumption interface. Select only view handler types (they have constructor with a single argument: strongly-typed IViewWriter or IViewSingletonWriter). Get the generic type argument out of these argument types - this will be the type of the view handled by the argument class. Actual interface implementations are simple. Inversion-of-control Container (Autofac) configuration just exploits generic registrations. Server-side module: public sealed class ViewWriterModule : Module { protected override void Load(ContainerBuilder builder) { builder .RegisterGeneric(typeof (AzureViewContainer<,>)) .As(typeof (IViewWriter<,>)) .SingleInstance(); builder .RegisterGeneric(typeof (AzureViewSingletonContainer<>)) .As(typeof (IViewSingletonWriter<>)) .SingleInstance(); builder .RegisterType(typeof (ViewContainerInitialization)) .As<IEngineProcess>() .SingleInstance(); } } Where ViewContainerInitialization is just a start-up task for Lokad.CQRS engine, that creates view folders, if needed. Client-side module: public sealed class ViewReaderModule : Module { protected override void Load(ContainerBuilder builder) { builder.RegisterGeneric(typeof(AzureViewContainer<,>)) .As(typeof(IViewReader<,>)) .SingleInstance(); builder.RegisterGeneric(typeof(AzureViewSingletonContainer<>)) .As(typeof(IViewSingletonReader<>)) .SingleInstance(); } } Concurrency How do we handle concurrency conflicts with the views: when the same view being accessed by more than thread simultaneously ? Actually we don't care a lot about reading and writing at the same time, since cloud storages providers generally ensure atomicity at this level. There are two major approaches for handling concurrency while updating views: simplistic and logical. The simplest approach is too keep updates of a single view entity limited to a single thread. For example, you can start with a single worker thread processing all view updates. As your application grows, increasing load and availability requirements, you can split updates of the different view types and entities between different threads and workers. In other words, you will partition views by type and/or view ID. Note, that we don't need to scale out actual view persistence, since it is generally handled by the cloud storage provider, to start with. However, such scaling tends to be limited by the world region (i.e.: North Europe) and we still might need to enable CDN or manually replicate data between multiple cloud centers. This is relatively easy to do just by streaming domain events to these data centers. As long as you pass entity ID in message headers (recommended in distributed systems), it will be easy to route domain event messages between different queues/workers. Just a quick clarification of terms. View Type is a message contract or the POCO class, while view entity is a single instance of this type, as identified and tracked by its unique identity. This identity serves as a primary key used to store and retrieve the actual view data. In the case of singleton views we have a single entity per type. Eventually you might encounter the need to allow multiple concurrent threads (whether hosted in a single worker or different ones) to be able to update the same view entity at once. This is where optimistic concurrency control comes in. We just need to modify our view writer implementations to keep track of blob's ETag (analogue of version ID), while downloading it. Then, after executing local update, we upload it back, while passing last known ETag back to the server. Azure Blob Storage (just like any other cloud storage) is aware of this feature and will update view data only if the ETags match. If they don't (somebody else managed to concurrently update our view) - update will fail and we'll get an exception. This exception will captured by the service bus, which will retry the actual update operation later. If such an update fails more than 4 times at once, this would mean some heavy concurrency issues probably coming from really complex update operations. Summary In this article we've briefly covered concepts of CQRS Views (also known as eventually consistent read models or denormalized views ) as applied to the Cloud Computing environment for the additional benefits of simplicity, scalability and cost-efficiency. There also were some high-level details describing experience of the actual implementation in the Windows Azure environment. This article is a part of CQRS in Cloud (xLim4) Research and Development series. You can subscribe to the news feed to stay tuned for more updates. All comments and questions are welcome and appreciated!


Scenario-based Unit Tests for DDD with Event Sourcing I'm still in the process of learning Domain-Driven Design for CQRS coupled with the Event Sourcing. One of the things I really like about this approach is the ability to thoroughly and reliably unit test even complex domains. As I've tried yesterday in a learning project, a unit test is basically a simple scenario (txt) file that looks like this: Given .ContextCreated { "ContextId": "10000000-0000-0000-0000-000000000000", "Name": "Name", "Rank": 12 } When .RenameContext { "ContextId": "10000000-0000-0000-0000-000000000000", "Name": "New Name" } Expect .ContextRenamed { "ContextId": "10000000-0000-0000-0000-000000000000", "Name": "New Name" } Or if we are expecting an exception: Given When .RenameContext { "ContextId": "10000000-0000-0000-0000-000000000000", "Name": "New Name" } Expect .InvalidOperation { } Basically each unit test ensures that: Given certain events in the past (they determine state of the Aggregate Root) When we call a single command (that's the behavior we are testing) Expected outcome is expressed as either 0..N events or an exception. These events are determine both the state of the AR and what is published to the bus (for further consumption by CQRS architecture). These tests are generated by hand (or from the UI or by recording sessions). You just drop them into the folder of your liking inside the test project (this feels like a reduced dev friction as compared to .NET unit tests). There is even no need to launch Visual Studio and add item references. Build will pick scenarios up automagically via the means of of slightly modified project file: <Target Name="BeforeBuild"> <CreateItem Include="Scenarios\**\*.txt"> <Output ItemName="EmbeddedResource" TaskParameter="Include" /> </CreateItem> </Target> The output will be a nice Project.Tests.Scenarios.dll that contains all the scenarios as embedded resources . Afterwards, you just need to generate tests (one test for each resource). In NUnit you can do something like this (note how Rx makes things so elegantly simple): [Test, TestCaseSource("LoadScenarios")] public void Test(Scenario scenario) { if (null != scenario.LoadingFailure) throw scenario.LoadingFailure; var observer = new Subject<Change>(); var root = new SolutionAggregateRoot(observer.OnNext); foreach (var @event in scenario.Given) { root.Apply(@event); } var interesting = new List<Change>(); using (observer.Subscribe(interesting.Add)) { DomainInvoker.RouteCommandsToDo(root, scenario.When); } var actual = interesting.Select(i => i.Event); ScenarioManager.AssertAreEqual(scenario.Expect, actual, scenario.Description); } TestCaseSource is NUnit native attribute that generates a unit test for each argument that is passed by the referenced collection factory. The latter could look like: public static IEnumerable LoadScenarios() { var assembly = Assembly.GetExecutingAssembly(); var clean = "Kensho.Domain.Scenarios.Scenarios."; foreach (var name in assembly.GetManifestResourceNames().OrderBy(n => n)) { using (var stream = assembly.GetManifestResourceStream(name)) using (var reader = new StreamReader(stream)) { Scenario scenario; var testName = name.Replace(clean, "").Replace(".txt",""); try { scenario = ScenarioManager.Parse(reader); } catch (Exception ex) { scenario = new Scenario {LoadingFailure = ex}; } var data = new TestCaseData(scenario) .SetName(testName) .SetDescription(scenario.Description); var failure = scenario.Expect.OfType<IFailure>().FirstOrDefault(); if (failure != null) { data.Throws(failure.GetExceptionType()); } yield return data; } } } Given all this, a folder like that: Would be translated into NUnit test suite like this: I'm still not sure if it is possible to affect NUnit tree structure via the attributes, making it more sensible and clean. A few notes: Latest Resharper does not work properly with these attributes (they probably haven't upgraded to the latest NUnit engine, yet). In this case NewtonSoft JSON serializer is used to persist messages for the scenario purposes. Adding new tests is really low-friction task. It's easy to have lot's of them or delete unneeded ones. Running unit tests is extremely fast and it does not require anything other than the AR assembly and message contracts assembly. Persistence ignorance is at rule here. Essentially unit tests form specifications for the domain here. Theoretically they could be captured with the help of UI (when some undesired behavior occurs) by people that are not familiar with the specific programming language. In essence we might be separating domain knowledge from the platform/language used to actually code this AR. This should provide some development parallelization and outsourcing opportunities that compliment native CQRS features in this area. AR unit tests in this case are not really fragile (they test only explicit behaviors) and should protect against regressions really well, while allowing to move forward incrementally and reliably with developing AR implementations for complex scenarios (something that I've been stuck with so far). All in all this experience is just based on my attempt to learn CQRS/DDD/ES by the means of running a learning project. You need to stay ahead of the current development requirements in a fast-paced environment like Lokad . Yet so far this scenario-based approach (inspired by using Fiddler recording sessions to unit test REST APIs) seems to look quite good, despite the fact that it was formalized and implemented just last night. Yet I'm really interested if there are any ways to improve the experience (esp. reducing friction-per test). So here are a few questions to the readers : Does anybody know any simpler serialization format for messages, than JSON? How do you test your domains? Are there more simple ways to organize, manage and run tests? What naming and organizing conventions to you use? How do you organize your tests (both code-based and scenario-based) and keep them in sync with the big-picture requirement descriptions? Any hints on improving this testing experience? PS: You can check out xLim 4: CQRS in Cloud series for any latest materials on this topic.


Self-Improvement Process

Self-Improvement Process Lately I've been doing a little bit of new and well-forgotten old things, while trying to become a better me. They also help me to reduce work pressure by offering something new for the brain to chew on. I just wanted to share these things (at least for the sake of writing down the links). We'll talk a little bit about: git; vim; LMAX and Martin Fowler; VerseQ; Lokad.CQRS and functional approach. Git I've started using git on a number of new learning projects. As I've mentioned it on a couple of occasions already, I start liking git more than mercurial, despite its steeper learning curve. By the way, Google Code has finally added git to their project support. Although they are way too much behind github both in usability and commits. Vim Subj, the good old editor coming from the terminal era. Exactly the one, where "the best way to generate a random string is to ask a student exit vim" (by the way, it is ":x" to exit with save-if-modified). I find using it more and more for common text editing tasks. That's where I'm writing articles and blog posts these days. And I'm getting really tempted to start replacing IDEs with vim in non-.NET environments. Developer's mind is better than any IDE anyway, especially when he or she deliberately keeps projects dead-simple. Now, if you got interested in vim, then that's what your future will look like . If this didn't scare you, here's a quick intro from the eyes of guy from VS environment. LMAX and Martin Fowler It seems that all this CQRS/ES stuff is getting mainstream, since Martin Fowler wrote a bliki post on CQRS and really good overview of LMAX architecture: the latter is essentially an in-memory circular buffer that also acts as a queue for the event sourcing messages; processed by a single writer and multiple readers. All workers have access to each other location pointers and this avoid any locks altogether. Predefined buffer size and one-writer-only approach allows to achieve high hardware affinity (in other words, CPU likes it a lot). The study is highly recommended for a read . By the way: bliki is term for versioned blog/wiki coined by Martin Fowler a while ago. These days people just use github with markdown (for example, this Lokad.CQRS Study isn't a page, but just a markdown file in a github repository). Don't read Martin's pounding on branch-per-feature. I believe he got it wrong (and got quite a bit of pounding from the community afterwards). VerseQ To keep up with the learning process, I've stepped back on improving my keyboard typing skills. I believe it would be a wise investment of my time to improve on something that is so tightly related to the job and thinking as the typing is. This (when coupled with vim) feels like reducing VS development friction by introducing ReSharper. For this task I'm currently sticking to VerseQ , which was recommended to me by Vsevolod . VerseQ tutor simply keeps on generating pseudo-random strings for you to type. However these strings are not entirely random: they include chords common to the language you practice (i.e.: "jg" is not common to English, while "li" is more common); next random string is based on the statistics captured so far. It will force you to practice your slowest keys and key combinations (or most erroneous ones). Recommended to give it a try, if you are typing more than 50 lines of text per day (emails or code alike). Functional Stuff In the previous post I've mentioned that it is possible to get a whooping number of messages per second on a single thread in Lokad.CQRS using lambda dispatchers . This comes from a simple realization that everything is either an aggregate/saga or a function. For example, handlers are just functions partially resolved from the container and executed against the message. Now, to be true with this kind of development you can easily achieve much higher throughput, if you switched to some other platform. I'm currently looking at akka for the small specific cases where extremely high performance would be needed (i.e.: message routing for Lokad.CQRS or doing some high-frequency data transfer). Fortunately enough, envelope and data formats of Lokad.CQRS are relatively cross-platform (protobuf to the rescue). With akka you can get something like a few millions of messages processed per second . The latter might be another life-saving option for devs and start-ups (constrained on resources) who are using Lokad.CQRS and are hitting some throughput limitations on elements of a decoupled/cloud system that are a bit hard/expensive to scale (load balancers and routers, for instance). Obviously I'm not planning to switch anything in Lokad.CQRS to akka and dead-simple functions editable in vim (no matter how tempting this might look, there always are real-world costs associated with such dreams). Yet, where possible to simplify things (i.e.: by throwing another framework out of the window) the experience and ideas will be reused in the approach. Especially, if they provide a clear and simple path of massively boosting performance of certain nodes in a decoupled app by switching them to the non.NET stacks. PS: if you post a comment and it does not show up within a day, please drop me a line . I get the feeling Squarespace has messed up it's spam protection badly again.


Separation of contract and implementation in software A bit earlier I posted a sample drawing with evolution options for a component in a big software application . This diagram is merely a convenience snapshot of performance optimisations and tech options available at any given point in time. Knowing about them in advance helps in planning future development. Each node on the tree of component evolution represents a design pattern that has its own advantages and costs. Costs almost always include increased complexity. Sometimes such design pattern also has certain affinity with other patterns, making it simpler to evolve component to the next level. For example, consider a component that happened to arrive at the following design during its evolution (after multiple iterations): We can say that this component has a public contract and an internal implementation. Public contract could say: Component implements a certain documented API with JSON, XML and ProtoBuf formats; these API interactions can be scripted or tested using tools on a variety of platforms; we can expert 99.9% uptime of the component; query response times under 50ms in 99% of the cases; commands are acknowledged synchronously in 500ms in 99% of the cases; we expect API to have throughput of 1500 transactions per second, if deployed in single-node configuration (all transactions extra would be rejected with 503 Retry Later ); Queries have eventual consistency of less than 1000ms in 99% of the cases. All of the above is easily achievable, for example, using .NET on Windows Azure with multi-worker deployment configuration and an efficient Event Store. Yet, please note, that the public contract does not say anything about the implementation details . It's normally up to the team to decide what these should be (better if that team is also follows the mantra "you build it, you run it"). This means, that at any given point in time, internal implementation might change in order to accommodate new requirements. Implementation might also change if requirements get relaxed and we can actually degrade the performance and get rid of some complexity in exchange. If component boundaries are defined well (as driven by strategic design and its evolution), then public contract will not change often. In such case development challenges are merely constrained to deal with the implementation details, shifting it along a well-known evolution path in order to achieve well-known benefits. We could actually fine-tune specific components to meet certain requirements . We might even say that with this natural approach, large scale software design emerges as a by-product of design process driven by two distinct feedback loops : evolving strategic vision which deals with business capabilities and how they are implemented by composing together components, defined by their contracts. fine-tuning component implementations to fulfil their contracts. However, while doing all this evolution, it is really important to break such process into small steps, which can be handled separately. Doing work in tiny bites provides you with opportunity to step back, acknowledge feedback, reflect upon the design at strategic and implementation levels. I think that one would find more opportunities for such separation if we: decompose software into small focused components; handle component contracts separately from their implementations; plan evolution process in advance, where possible.


Shelfcheck - Cloud Scalability Challenge for CQRS from Lokad If you have been following Lokad.CQRS docs , you've seen that we openly publish use cases of internal projects that are based on this architectural approach. Another massive project is going to be added to this case library in due time. Shelfcheck, on-shelf availability optimizer has been announced by Lokad today. You can read the official news blog entry, or just bear with me for more technical details. Retail industry (all kinds of shops, stores, hypermarkets etc) has one long problem that has plagued it for years: out-of-shelf problems . You as a customer can frequently discover them while doing grocery shopping and discovering that store is out of your favorite brand of beer, preferred snack or some specific type of diapers for the newborn. This small frustration can cause you to get a substitute product, delay the purchase or even go to the competing store. That's a small problem that roughly accounts for 100 billion USD losses for the retail industry worldwide . Solving that problem, among all other options, requires tracking and analyzing data at every point-of-sale (shop, hypermarket, etc) to detect out-of-stock issues as soon as possible. Previously this seemed to be impossible due to the sheer amount of number crunching involved . In order to run almost real-time analysis for thousands of stores in a retail network, where each store can have inventory of 10k-100k products, you either need a huge data-center... or a cloud. Now, that's where Lokad comes in with Shelfcheck , on-shelf availability optimizer . We plan to deliver highly affordable service that will be provided as a subscription with pay-as-you-go pricing. Naturally, the project will be deployed into the cloud, in order to handle all the load; with elastically scalable architecture. This would allow really efficient use of resources to keep the operational costs and prices extremely low. Technologically the project does not seem to be extremely challenging, given Lokad.CQRS and Lokad.Cloud to support this endeavor. However it should provide better insight into designing, building and running systems of such scale in the cloud. I'm personally really excited about the scope of the problem we are trying to solve for the industry with Shelfcheck . Stay tuned for more details.


💊 Simulation

💊 Simulation Once upon a time I got fascinated with an underrated piece of software called FoundationDB. This was before it was acquired by Apple, taken offline and then made Open Source again. You can read more about FoundationDB is cool in FoundationDB is Back! . One of the amazing things about FoundationDB was how well it was tested. The team was running the entire distributed system (cluster) inside a virtual environment. They could simulate network failures, power outages and other problems that are hard to reproduce. The entire test suite could run years of simulated runtime in minutes, while still allowing to deterministically reproduce that specific fault condition. I strongly recommend watching this 40 minute video by Will Willson. Needless to say that I got fascinated by the idea and wanted to learn and re-implement it. This section is a collection of exercises and materials that helped me to make some progress in this area.


Simulate Ring of Actors

Simulate Ring of Actors This is a naive implementation that explores the idea of simulating a ring of N actors that send the messages to each over M times. Concept of the ring benchmark and simulated actors is taken from the gist by Preetam Jinka. The implementation is quite simple and allows us to focus on core concepts: messages used for the communication; actors representing individual processes; simulation environment simulation loop . Implementation Let's start by defining a message and a simulation environment to hold a global inbox. public struct Message { public readonly int Recipient ; public readonly object Body ; public Message ( int recipient , object body ) { Recipient = recipient ; Body = body ; } } public sealed class Env { readonly Queue < Message > _messages = new Queue < Message > (); public void Send ( int recipient , object message ) { _messages . Enqueue ( new Message ( recipient , message )); } public bool GetNextMessage ( out Message msg ) { return _messages . TryDequeue ( out msg ); } } Actor is a class that has a recipient and can send a message to it: public class Actor { readonly int NextActor ; int _counter ; readonly Env _env ; readonly int _m ; public Actor ( int nextActor , Env env , int m ) { NextActor = nextActor ; _env = env ; _m = m ; } public void HandleMessage ( object message ) { _counter ++ ; if ( _counter <= _m ) { _env . Send ( NextActor , message ); } } } We can join everything together by the actual simulation loop. It sets up a ring of actors and then sends the first message. class Program { static void Main ( string [] args ) { const int n = 1000 ; const int m = 1000 ; var actors = new List < Actor > (); var env = new Env (); for ( int i = 0 ; i < n ; i ++ ) { var next = ( i + 1 ) % n ; actors . Add ( new Actor ( next , sim , m )); } env . Send ( 0 , new { hello = "world" }); var watch = Stopwatch . StartNew (); while ( env . GetNextMessage ( out var msg )) { actors [ msg . Recipient ]. HandleMessage ( msg . Body ); } Console . WriteLine ( watch . Elapsed ); } } Given N=1000 and M=1000 , the result on my machine is: dotnet SimRing.dll 00:00:00.0260680 Questions How would you implement a ring benchmark in golang? This .NET Core implementation is way faster than Erlang. What does Erlang spend time on? Why is this implementation faster than FoundationDB Flow? What extra work could the Flow do?


Simulate with Async

Simulate with Async This project demonstrates plugging into async / await capabilities of .NET Framework to run an application in a simulated mode. The simulation will inject random faults and fast-forward time. Source code is available on github . About SimAsync The purpose of the simulation is to automatically find expensive bugs that are otherwise hard to catch or reproduce. In order to fulfill this promise a simulation needs to exhibit a few specific properties. In the previous projects we've already explored Determinism and Time acceleration . This sample project introduces two more capabilities: Fault injections and Simulation of the parallel processes . For the purpose of this exercise we focus only on a single-node deployment without any complex storage or configuration. Determinism This simulation is deterministic . Two runs with the same Seed parameter will yield identical results. You can use random Seed to discover new failure scenario. You can also re-use Seed from the past to reproduce a simulation run. Time Acceleration The simulation fast-forwards time . Although the logic uses random delays to represent disk delays, freezes or network outages, the CPU doesn't actually need to sit idle during this delay. We control the scheduler and we could fast-forward to the next interesting moment in time. Years of the simulated time could pass in hours of real-time. Fault Injections Random faults are injected along the way to represent some of the bad things that could happen in reality to our code. After all, we want to capture, debug and fix some of these issues before the code is deployed to the production. Parallel processes Parallel execution is simulated (even though the simulation itself is single-threaded). Thanks to the state machines generated by await / async we could actually suspend execution of one execution path for some time, while other paths will continue running. For example, Actor 1 could experience a storage freeze for 10 ms while reading from the database. Scheduler will switch to the other pending tasks (e.g. message handling by Actor 2 ) before coming back to Actor 1 to continue execution. Assignments Questions This simulation reveals a problem with the application. What is the root cause and how would you fix it? This simulation uses an in-memory dictionary as a DB. What would it take to implement something real? What kinds of failures could you inject into a simulated storage? Tricky Questions How would you simulate: your favorite database? your favorite commit log or message bus? a load balancer? faulty disk controller? fail-over between two data centers with eventual replication? Bonus After fixing the first problem discovered by the simulation you are likely to hit one more (it might require running longer simulations). How would you handle it? Remember, that the any simulated logic will eventually be replaced by the production code. They have to match.