Business Amnesia

Business Amnesia At Lokad we focus on big data analytics and consulting for the retail. This is about doing some small things (sometimes simple, sometimes not) that would companies to be more efficient in their business (e.g.: reducing lost inventory, getting better service level, optimizing replenishment for an inventory of 500k products, etc). More often than not, I observe two patterns: First, companies have really refined and deep business models . They are really good in what they are doing. Second, business processes of these companies tend to be hindered by their software and it's limitations coming from the previous century. This hindrance comes in following forms: Business practices in a company don't even have a chance to evolve and adapt simply because legacy software is so fragile and limiting. Something that was once designed to support and serve now resists change by being expensive and risky to change. A lot of money and resources is wasted inefficiently to work around such limitations. Companies can still be extremely profitable, but a few billions of euros less than they could make. A lot of vital business data is lost irreversibly, simply because before for years it was too expensive or impossible to persist it within the existing setups. So we are arriving at an interesting paradox here. On one hand, a company can be really good in what it is doing in business. On the other hand it can continuously suffer from the progressing amnesia - loosing fine-grained information about actions and decisions linked to individual customers, supliers, products or promotions. The fact that you can't process your business data today, does not mean, that you will not be able to benefit from it later. It might be useless or it might increase your revenues. However, if you don't save it - you will loose it forever. Imagine what a sad world we would live in, if health records of every patient were discarded. Detailed history of business operations is such a set of health records. It can be used to find problems or explore opportunities for improvement, which would be unique for each unique business. You just don't throw out such vitual information. Such general approach to discarding extremely valuable data can be even more surprising, given the fact that these days it becomes increasingly cheap to store data for future reference . Prices start from 15 cent per GB per month (any cloud provider) and go down to 1 cent per GB per month ( Amazon Glacier ). Keep in mind that 1 year of sales history of 1000 supermarkets can easily fit on a single smartphone. More often than not, you don't need any rocket science to store your business history for the purposes of the learning and analysis. So please, do it - this a cheap and low-risk way to potentially derive additional value later. Business amnesia, on the other side, can be an extremely risky thing, especially in competitive markets.


Challenges of the Code Documentation

Challenges of the Code Documentation Here's the interesting problem. There are numerous situations, when code contains a lot of important information . This important code can change really frequently too. Let's say that we need to relay this important information for somebody who is not intimately familiar with the codebase. For example: Researchers depending on the conventions and transformations in some data pumping project. New users being introduced into some project via articles with a lot of samples. Managers , requiring knowledge of some business constants and rules. 3rd party Developers , that have to integrate with some API, while having the access to the latest samples, restrictions and constraints. Needless to say, that important code pieces could be scattered across multiple projects, adding friction to people that need to have a look at them fast. We do want to have this friction at minimum ! This way we increase the chances that some questions could be resolved by looking at the documentation, instead of wasting time and potentially involving somebody else into this quest for the answers. Saved time essentially translates in reduced expenses and faster reaction of an organization (resulting in improved ability to compete on the market). There also might be some important contextual information about this code. It might or might not be valuable for the certain party, but developers would want to write it somewhere (enabling them to forget details and free Brain RAM for the other tasks). Comments usually help here, but they have to stay with the code and are limited to the plain-text (no graphs, images, tables or even bold). One common way of relaying this information (in some specific context) is to document the code in external docs , while including the latest snippets. However, the code tends to change a lot . This is especially true for fast-paced environments with tight feedback loops and low-friction development (and deployments). So we have got ourselves a problem here: we either need to waste time and concentration on updating the documentation after every significant code change (i.e.: a few times a day); or we have to accept the fact that the documentation is out-of-date and essentially useless; or we have to include links like: "for the actual details look in the method DoomsdayMachine.RefreshWorld() and any other methods it might call" . We'll also need to remember to update the links, should the class be renamed or moved. One logical solution is to have auto-generated documentation that could be compiled from some text, while automatically linking to the code sources. And it has to survive refactoring and class renames. I know that Lokad researchers use LaTeX with some scripts for such tasks. However the whole LaTeX thing looks a bit of overkill here, plus I'm not sure it can bind to some MSIL-level markers within the .NET code, while providing common publishing functionality. Ideally this would work like this: Project has documentation files stored and versioned side-by-side with the sources (ideally in the same solution). These documentation files are expressive enough to contain graphs, images, tables and all the other nice publishing things, while referencing some code blocks in the project. Editing the documentation would be WYSIWYG-friendly , while the original document format would be friendly to the version control (and seeing the changes). Changing the original code (i.e.: adding a few lines in the beginning of the file, or moving method around) should not break the documentation. Whenever needed (or continuously on the integration server) these separate doc files are assembled and rendered to the desired publishing format (i.e.: online docs or PDF). Any document-level compilation problems are detected immediately (i.e. when building documentation). Does anybody have similar problems and ways of solving them? What do you think?


Cloud CQRS Lifehacks From Lokad - Part 2 Let's talk about some more practical lessons learned at Lokad, while running and evolving Cloud CQRS system deployed at Windows Azure. In the previous Cloud CQRS Lifehacks article we've talked about: "Real-time" CQRS Server Debugger Immediate Email Error Notifications Detailed Error Logs Domain Log History Excel + Domain Log Exploring your own domain In this article I would like to continue the topic and focus on the recent tricks that helped to improve performance, scaling, stability and development efficiency of our systems. Publish Performance Stats in Domain Events This is one of the best things I've done recently: Putting as much performance statistics into the domain events as possible (YAGNI does not apply here). Making these stats human-readable within the domain log . By the way, note how we are using human-readable translations of digits as well. "5 minutes" might be less precise than "00:05:17.1803", but it surely is much easier to read. This saves a few brain CPU cycles along the way and really helps in the long run. These performance stats are available within seconds after being processed in the cloud. They are captured in the code manually via a few simple helper classes (no voodoo magic here): using (stats.MeasureMs(SyncStat.UploadDatasetPart_UploadMs)) { UpsertTimeSeriesParallel(client, task.Dataset, history, false); } using (stats.MeasureMs(SyncStat.UploadDatasetPart_CommitMs)) { operation.MemorizeDatasetInput(history); operation.Commit(); } These stats aren't actually human-readable in the real world, since they are encoded with ProtoBuf-net serializer (fastest and most compact .NET binary serializer). However, for the display purposes I'm using Json formatter to make message contents look readable. This required defining a few custom Json Converters for classes that do not look nice in plain JSON, but this essentially was it. Use Consistent Color Coding It is really amazing, how a simple color coding can speed up data comprehension by a human brain. In essence visual elements (when they are not excessive) serve as mental hints or some sort of the index; they allow our conscience to skip actual reading and see high-level patterns. My life became a little bit simpler after color coding was added to the domain log. These colors directly map to the legend from my previous CQRS articles like: Domain-Driven Design, Event Sourcing, Rx and Marble Diagrams Theory of CQRS Command Handlers: Sagas, ARs and Event Subscriptions The legend is: Real-time domain log monitor follows the same pattern as well: Essentially with this approach, I do not need to read names of the commands (and recall their intents) in order to see the bigger picture. Optimize Management UI for the Touch This was unexpected, but recently I had to do a bit of monitoring via an iPad connected to a work-station via RDP. I don't have a habit of carrying a laptop around on weekends, while iPad is rather lightweight and is usually within the reach. However, native desktop interfaces are not necessarily a good fit for working with via Tablet devices. All this can add a bit of friction to the experience: context menus; keyboard shortcuts; unnecessary text-based controls blocking operations. You can really feel the friction as you struggle with your own UI on iPad. Natural urge is to reduce this friction in the areas, where you spend a lot of time. Surprisingly enough, while design gets "optimized" for touch interfaces, it also becomes more clear and explicit, serving as an explanation to itself. Monitor Stale Messages Unexpected things can happen in software systems. In the distributed cloud deployments they always do happen. Actively looking out for them and reacting immediately is the only way to handle the unpredictable. In addition to reporting poison messages directly to the email inbox, I've recently added another life-saver (as it turns out) feature to our systems. There is a scheduled task that checks all queues every few minutes. When it finds any with stale or delayed messages (essentially messages that were not processed within the designated amount of time) - an email is dispatched to support. Message can be stale for a number of reasons: full queues, threading issue, deadlock, low performance etc. However, if you know that such issues will be detected and reported fast, a certain amount of peace and tranquility will come to you. Of course, all these lifehacks are still separate hacks that do not compose a true self-healing and self-tuning Cloud CQRS system , but we will get there eventually. Stay tuned! PS: The next article in the series is: Helpful Domain Logs of CQRS . It takes color-coding idea one step further.


Command Handlers without 2PC and with Various Levels of Reliability In the last post we've tried to start reasoning out CQRS theory by making first parallels between Rx for .NET, almost-infinitely scalable systems as advocated by Pat Helland . I've been thinking a lot about this recently and so far the logic seems to hold. Let's continue with talking about persistence and message dispatch in command handlers. We will do that without worrying too much what exatly resides within the command handler. Valid options would be: aggregate root with a state repository; aggregate root with event sourcing; saga. Note that we are not talking about relational databases for the persistence in this scenario. Working with RDB would require 2PC, distributed transactions and all sorts of complex stuff. We don't need this. I would like to express my thanks to the DDD/CQRS group for helping to clarify this logic and especially Jonathan Oliver . Command Handler Command handler is a process that resides within a partition and knows how to apply incoming commands (delivered via the durable message queue) to entities that reside within the partition. Commands always have a recipient (we need to be able to route them to the partition), so applying commands is a matter of: loading the entity uniquely identified by it's identity which is mentioned in the command (from memory cache, snapshot, persisted state or event stream); applying the command to the entity (by calling a method); in case of success we persist changes to the store and acknowledge command to the incoming durable message queue (actually deleting it). Change persistence within the partition could happen in form of: event sourcing - we save events that describe the change and allow to reproduce it; essentially that's domain-aware transaction log that keeps all the information. state change - we just discard the intermediate information and persist only the latest state to the store, appending it. In both cases we need to persist information about the incoming command(s) that caused this transaction. This way we will be able to ensure command idempotency . The latter allows us to handle rare but possible scenarios when: environment duplicates some messages (this can happen in the cloud or partitioned systems). process crashes between the moment we've committed the change and ACKed the incoming message. Obviously appending changes to the disk is not the only outcome of the command handler. It might need to send messages (i.e.: aggregate publishes events and saga sends commands). This is done by the async process that runs in parallel and checks the storage for any newly committed changes. If it discovers them, then the associated messages are also published. In case of the event sourcing we simply publish all events, in case of the state persistence we just look for the explicit messages. Anyway message dispatcher within the partition will know how look for messages to send. By the way, incoming command message might actually include multiple commands. This could be used in CQRS scenarios when we want to achieve all fail or succeed together behavior (command composition is a lazy way to avoid writing large commands for each combination that might need to be atomic). In this case all caused changes are perceived as a single unit of work and are committed together as one. If the changes can't be persisted in a single write operation (i.e.: too big or storage does not handle atomic updates), you can first write all the changes in as many passes as you need. The write could be finished by the SHA1 signature of the changes. This way the transaction is considered to be completed (by the event dispatcher or anybody else who's reading it) if the signature is correct. Otherwise we have abandoned transaction that just needs to be cleaned up (or ignored). Reliability and Redundancy We can have to levels of reliability and redundancy in this scenario (we are talking about the simplest scenario where changes are written to disk and we need to handle reliability and redundancy explicitly). As long as we stay within the partition (a single machine or a small cluster) we can have 100% logical redundancy. Basically we can persist a change to multiple locations at once. Transaction will be considered as completed if the change is valid on all locations. And since all locations are within the partition, reads are guaranteed to be fast and consistent. One can implement this manually or just trust RAID to do that. If one of the writes fail - then the entire transaction is considered to be failed. This approach allows us to have a variable level or reliability in the persistence (we pick that level). In fact one should be able to move critical processes to highly-reliable partitions (dedicated hardware and data centers) and keep non-critical entities in the commodity environment (i.e.: in the cloud). Second level of reliability comes at the moment we cross the partition boundaries . We would eventually want to do this, since data is more safe, when stored in multiple locations (earth-quakes, floods, political issues etc). However, since we cross boundaries, the reliability would be eventual, giving us a small time window (from a few milliseconds up to seconds or longer), during which hot data will reside only within a single partition. Basically RAID arrays and redundant writes are just needed to reduce risk of fresh data corruption within this time window and increase the chance of successful replication it somewhere else. Replication comes here in form of yet another simple async process that merely picks up our changes and sends them across the partition boundaries to long-term storage (preferably there will be multiple locations). If our recipient is not available at the moment (network failure, repartitioning in the process or 2012), then all we can do is to raise alert, retry shipping changes outside of the partition boundary till we succeed (the infrastructure might also provide alternate shipping locations). By the way, the very process of shipping changes across the partition boundaries could also be used to replicate them across the globe, feeding to various read models (they are eventually consistent anyway). Basically we could combine these two simple approaches (writing to multiple locations within the partition and shipping changes outside of the partition) to achieve a desired degree of reliability and fault-tolerance. Besides, we can always position various partitions (based on their business importance) in environments with various reliability settings to achieve the maximum ROI and handle risks in cost-effective manner. There also is a third approach , that involves sending data in a transactional manner across the partition boundaries on the actual write. This one is slightly more complex and would probably require using some storage engine for the purpose. However in essence this would work like: storage sequentially writes the value to multiple locations within various partitions, sending ACK only if the last write is successful (or there is a quorum or whatever tech is used to figure out decision across the consistency boundaries). I doubt this is much better than 2-step redundancy approach described earlier. This simplicity and ability to tune reliability settings (and failure probabilities) on per-partition level is what I like about the approach. Simplicity allows to scale and evolve, variable-level reliability allows to be really efficient and cost-effective. NB: please keep in mind, that for me this just a theory at the moment (logically ruled out of the properties of the distributed and scalable environments and some cloud experience at Lokad projects). I haven't tried this specific approach in production (this might be fixed soon, though). The whole purpose of this exercise is to figure out and understand underlying theory so that future systems will be developed without architectural decisions blocking almost-infinite scaling by enforcing certain constraints based on the theory. All feedback is welcome! Now if only somebody could help to me figure out how to distribute events to subscribers in partitioned environment without feeding everything to a single message bus (complicates and creates single point of failure). I just can't figure out the logical transition from the published event (going outside of the partition) via the subscription and into the command that has a recipient and goes inside the partition.


Concurrency Programming is Like Thinking in 5 Dimensions Concurrency is about thinking in 5 dimensions at once. Let's consider a single code class to be a 2D space. In the end, it's flat and fits on one display. File organization (either flat or with multiple folders) add another dimension - third one. These are static dimensions, just like the height, width and depth in the real world. Fourth dimension, time, has it's own analogue in the code world - execution of the code within a single thread. This is were the code comes to life and shines in its dynamics. Concurrency brings another dimension to the mix - the fifth one, since you have to deal with multiple threads being executed. They are just like parallel dimensions in the real world. That brings us somewhat close to 5D. So delivering reliable concurrent code is about thinking in 5 dimensions: it takes a bit of time and exercise to get used to to it; it is rather hard to visualize; you may want to keep the basic logical pieces as simple as possible in order to manage and evolve the entire system efficiently in your head. Adding distributed systems could pump the dimensionality count up a little bit as well, increasing the complexity and excitement of the delivery. NB: thinking about an entity (software system, in our case) from completely different angles helps to find simpler solutions to existing problems and challenges in the code.


Const vs. readonly vs. static readonly in C# .NET Here are the differences between C# .NET const , readonly and static readonly fields. Constants : Static by default Must have compilation-time value (i.e.: you can have "A"+"B" but cannot have method calls) Can be used in attributes Are copied into every assembly that uses them (every assembly gets a local copy of values) Could be declared within functions The compiler performs some optimization by not declaring any stack space for the field Readonly instance fields : Are evaluated when instance is created Must have set value by the time constructor exits Static readonly fields : Are evaluated when code execution hits class reference (i.e.: new instance is created or static method is executed) Must have evaluated value by the time static constructor is done You really do not want to put ThreadStaticAttribute on these (since static constructor will be executed in one thread only and it will set value for its thread; all other threads will have this value uninitialized) Class that has only constant or readonly instance fields is considered to be prepared for the concurrency challenges of parallel computing, since it has no mutable state. BTW, there is a simple unit-test to enforce that your classes are immutable . Additionally you might be interested in some tips on writing the event-handling code in .NET C# from the theory category.


Continuous Learning at Lokad

Continuous Learning at Lokad I've been asked a few times recently about how Lokad approaches development from the high-level perspective. After giving it some thought, I've assembled a small article that provides a bit more detail on the subject. I hope you find it interesting or at least amusing. While reading, please keep in mind that these are merely personal thoughts or a person who has just started to learn the trade. If you need some solid professional advice or a guidance on the matters at hand - please go to Microsoft, Oracle or any consulting company. Lokad , once again, is a tiny SaaS company focused on providing forecasting services, while using some bleeding edge tech to deliver them in an efficient and cost-effective way (as in " much cheaper and smarter, than any competitor would be able to come up with ", " scalable beyond the wildest dreams of any customer " and " don't trust just my words on that and try-it-yourself-for-free "). Heavy research background helps us a lot here to stay efficient and move forward. This works because the majority of people of the company got involved into PhDs or secondary education at some point. PhD is a nice practice of continuous self-guided research and study process. Continuous learning is actually what we are doing on everyday process here (this is in addition to work done by the interns and people who are actually working on their PhD within the walls of the company). You'd be amazed to know, how much you would learn from you work in a continuous manner, if you perceived every task at hand as an opportunity to learn and improve . This approach is reflected in our formal practices (i.e. writing weekly internal blog posts on challenges encountered, things accomplished and planned for the next week). This is also reflected in informally adopted practices as well. Company culture encourages blogging (internal and public) which is a nice way to structure your own thoughts and do the planning (for example, blogs of Joannes , Matthias and Christoph ). Talks and presentations are also encouraged along with sharing non-core knowledge with the community in various forms (i.e. See our open source strategy ). Even development projects are shaped around vigorous learning approach as well. In development we try our best to avoid heavy planning and management approach (waterfall, PM, Agile, XP, Scrum or whatever other letter combination that is being sold by consultants these days). The reason being - we don't really care about the process or formalities of dev planning and management (as it is usually advocated by the thought leaders). The only trustworthy guide for us here is the real world - being able to deliver services and software that do the job, survive the beating and stress in the cloud environment, help to move the company move forward. They have to require as few man-hours as possible, of course. No matter how good development practice is, it does not guarantee that the project will have a financial success in the real world. However, if the project did succeed and is capable of evolving efficiently, nobody would really care about the "properness" of the development practice employed to achieve that success (aside from ivory-tower architects and managers). These are the additional reasons (fine-print for the previous statements), why our ignorance of any "proper formalized development planning" actually works out for us: we are still very disciplined in what we do at Lokad and how we approach planning and management; it just happens that we use approaches that are closer to the scientific fields with a lot of experimenting and unpredictable future; projects are deliberately broken down and structured to make this approach work; we avoid development and technology choices that would conflict with this approach; multi-hat roles and a lot of interpersonal communication help to soften the rough edges as needed. Let's dive into a bit more detail. Projects and systems are broken down into smaller tactical apps (Joannes wrote a bit more about the tactical approach ). Actual development is preceded by a series of rapid prototypes (spikes), if there are some unanswered technical questions. These prototypes stay forever in the trunk of the originating project for any future references. Usually the first version of the system is just a prototype that has gone too far. We don't involve any external consulting companies whenever we need to solve an unknown dev problem. First, this would be hideously expensive and slow, compared to our own standards; second, I doubt it would bring us anywhere (i.e. given what is preached by consultants in enterprise and cloud systems, for example). Instead, usual scientific research process is being used, tapping on the infinite practical knowledge shared by the community (all, who have kindly and patiently answered our numerous questions on forums and emails - I thank you!). Results are structured, implemented and shared back with the community where possible, in a form of open source projects, blog posts and published articles. The latter is not only a way to give back something, but also a way to gain additional insights, feedback, ideas and inspiration to move forward. First project iterations are pushed to production as soon as possible (often with few or no tests). This helps us to get real world feedback and correct the project at the stage, where correcting the project, business idea (or even discarding them altogether) is at it's cheapest - in the very beginning. After the first release, new discoveries and insights are incrementally incorporated into the codebase in the order of importance. This importance is prioritized by the business in form of Return on investment (ROI) in the mid-term (adjusted according to the priority of the project and it's positioning). Such factors as "reducing development friction" or "refactoring" play on the same table as "business features", since they all determine the capabilities of the project to survive, adapt and evolve with the minimum effort possible. I've seen some remarks by thought leaders along the line of " who put developers into the position to decide what technology they should use? " (the context was about the event sourcing or doing any major refactorings at the cost of short-term feature development). Well, at Lokad we have a clear separation of concerns. Business decides and determines, what should be done ; it comes up with the challenges and problems; it picks the direction (and is also encouraged to share wild dreams and things that may seem to be impossible to do these days). However, question of " how it should be done to achieve the specified result with the maximum efficiency possible " is handled by the development. It is the responsibility of the development to find the fastest and cheapest route towards the business objective , suggesting alternative routes if needed. Hence such things as introducing a new tech, if it really proves worth it, are never the problem. Here are some examples that made a quick way in, because they were a tool to solve some problem at the point: elastic cloud deployments; distributed version control systems; different IOC containers; new continuous integration platforms; new versions of SDK or frameworks etc. These days we are making it into distributed cloud engines, event sourcing, new types of UI. A lot more cool stuff is considered down the road as well. However, development does not run anarchist in it's work. We don't pick every single technology or jump at every opportunity. As you recall, it is the task of the dev to solve business problem in the most efficient way . "Most efficient" should, obviously, apply not only at the scope of a single project, but should cover the whole project portfolio of a company considering it's well-being, managing the risks and optimizing returns over both the present and foreseeable future. This forces Lokad development to be self disciplined and self-constraining. This also makes it an interesting challenge to balance business problems with the technical and resource constraints , overlapping them over the major paths that the future might take us. Probabilistic planning and balancing of resource pools and tech choices can send your head spinning sometimes (try visualizing effect of a choice over the possible scenarios of future, weighted by their math expectation of ROI and colored by the distance from the currently selected course ), but it allows development to be really efficient with resources and time , while helping to set what is believed to be the best course for the company. Here are some examples of the self-inflicted development constraints , that would seem to be illogical choice if considered with the scope of a single project only. However they are really justified (as we want to believe), when the entire business is factored along with the major future scenarios and our real-world constraints: we still use svn and mercurial for version control (we know, that svn can't even be compared to dvcs and thar git is more flexible than git); one of the current projects under work still uses ASP.NET web forms for the UI (although it's a pure drain of development time and effort); we continuously push effort into open source projects (well, this is easily financially justified); we make the effort to make systems at least 100 times more scalable, than the current actual stress (almost-infinite scaling is obviously the ultimate goal); we gradually migrate from SQL to noSql and event sourcing (easily justifiable, but for some reason it's so hard to do in some other companies); sometimes developers are assigned to projects and tasks that hold no real business value (aside from letting them learn and gain experience in preparation for projects planned down the road); a lot of times we restrain from using some tech approaches and solutions that would make our life easier (these tend to be the very same things that led us to big bowl of complex and expensive spaghetti code in the past). By the way, below are some widely accepted names, that we are either moving away from or ignoring altogether. Please, keep in mind, that this is a highly opinionated list . With a lot of these items we were in deep love before (and still share a deep appreciation for their authors). But we eventually discovered that in our hands these do a lot more damage than good (this is probably caused by my lack of technical expertise and the lazy approach of trying to do as less work as possible). NHibernate or any ORM; all mocking frameworks; anything more complex than basic NUnit for the unit testing; anything that requires an SQL database to work with; Azure compute and storage emulators for the cloud development, Azure bus and caching; Reusable frameworks (getting rid of Lokad.Shared libraries was a major pain) Any IOC container that is more complex than Autofac (although ideally we would boil to lambda-based IoCs); Silverlight, WPF or any other ancestor that would show up in Windows 8; MSMQ, or anything that has to do with DTC and System.Transactions; anything that has to do with Aspect-oriented programming and code weaving. In addition to selective ignorance of widely used technologies, we try to find our own way around the architectures and development principles as well. Modern enterprise development by the books (i.e.: as in N-Layer architectures or patterns and practices ) might work for large enterprises that could dedicate multiple developers (even dozens) to work on a single project for multiple months (even years). We don't have this luxury and should seek something dead-simple, dead-cheap and extremely efficient (for examples see CQRS and xLim series of articles within this blog). This process of trial and error does not end up with the development only and covers the other areas of running a business - marketing, sales, customer support, product planning etc. The most interesting pieces are reflected in articles of this blog, as well as the blogs of the other members of Lokad team . As you have probably guessed, I find deep satisfaction and pride in working in such interesting environment. Man should be challenged, and all challenges should be answered. By the way, we are in a lookout for young talents and researchers who are interested in learning and working in such productive and efficient environment, continuously learning and pushing the state of the art. Not just for the money, but also because it is fun and helps to make the world a better place. Internship opportunities are offered in Paris, France (developers, statisticians and sales/marketing) and Ufa, Russia (strong focus on dev, but other options might also apply). And there is always a possibility of negotiating something more permanent if we get along really well. Even if you just need advice or have a question in the field of forecasting, organizing and running start-ups, efficient development (CQRS/DDD/ES/Cloud etc), please don't hesitate to ask a question in our community or drop us a line in private. We will answer.


Contributing To Open Source Is So Much Better Than Taking A Class Let's say you want to really learn cloud computing, unit testing, inversion of control or CQRS. You can either: pay for a course, read some books, do exercises and get a certificate at the end; participate in an open source project or two in your field, getting real-world practical knowledge and being forever recorded as a contributor; nothing to pay and skills will be up-to-date. Guess which option will score more victory points, when you apply for a job or start building your own ISV startup? My personal biased bet will be on Open Source , so let's talk about it. BTW, my many thanks to Muneeb for sending a question on the topic. Also thanks to Jonathan Oliver for helping to clarify the title. Open Source projects are something we all know about - projects where people collaborate to develop a product, providing it at no cost along with documentation and source code . In .NET community you probably came across a project like NUnit, Autofac, NAnt or NHibernate. These are open source. Their analogues on the other platforms are usually open source as well. You can get the source codes, build them and run. Certain restrictions may apply , depending on the licensing terms. However generally you can do pretty much what you want as long as the credit is given and fair sense is applied. We can clearly see the commercial benefit of Open Source Software (OSS) in various projects - you don't need to pay licensing costs and usually have faster development model. However, there is much more in OSS! In my previous post of 10 Steps To Become Better .NET Developer there was this seemingly simple line: Contribute to Open Source Projects of your choice. Actually this was one of the most important items in the self-improvement list. Let's talk how exactly open source projects can help you to become a better developer. If you would like to recommend a .NET Open Source , or need help in finding one that fits your field of interest - please read towards the end of the article and leave a comment . Learn from Open Source First of all, you can benefit from the Open Source by just downloading the source code and learning how smart people get things done in software . I loved reading through and being inspired by NAnt , OmeaReader of JetBrains , MindTouch and StructureMap , when I was just starting with .NET. Such an activity helps to: practically see how people organize code and do all sorts of small things: name classes and variables, throw exceptions, write sanity checks and document code; learn about testing projects, building and integrating them - this is real-world stuff; see various ways to organize projects, supply documentation and version dependencies; practically understand what steps are actually involved in delivering a project. Of course, there are no perfect projects or developers. After checking out a few similar projects you will notice that they tend to have different strengths and focuses. Some might have better codebase, the other - better documentation, third - more efficient development environment and faster feature deliveries. By comparing them you will see tradeoffs made by people running them. They had to focus on the most important things (in their opinion) while using limited resources at hand. Resources are usually limited by their own free time and motivation. And since the Open Source project is active and being used, you can already say that they were successful in making these choices and executing them. Hence, you can learn a lot. Besides, if you focus on a set of open source projects in a certain niche, you will be able to learn the technology and the primary principles behind. You will practically see the choices and trade-offs being made in the code. For example , less than a year ago I knew little about messaging and nothing about service buses. Yet, when there was a need to build a service bus for Windows Azure with a specific set of requirements, it was a rather straightforward process. I simply had to: Learn the theory and read a lot of articles; Find open source projects dealing with the similar problems (NServiceBus, Mass Transit and Rhino Queues), reading through them; Start working on Lokad.CQRS , while adapting the most important ideas from these projects to the specifics of Windows Azure; whenever I had a problem at hand, there were already 2-3 options of solving it. The end result is that now I can say a couple of words about using and building service buses in cloud computing environments and on-premises (esp. in CQRS architectures and Windows Azure). Plus this brings real business value at Lokad. As you can see Open Source projects can give you a lot merely by providing real-world something to learn from. However you can gain a lot more by actively participating in them. Participate in Open Source Participating in Open Source project allows you to: actually practice your development skills: applying design patterns, writing unit tests, managing complexity and doing all sorts of other things essential for any project; get feedback on what you do and potentially some helpful guidance - it will come from the people involved in the project and hence having practical experience in the field; this is far more better than spending time in university (for which you will need to pay) and listening to teachers that don't necessarily have up-to-date practical knowledge and experience; learn and practice things like: issue trackers, wikis, version control systems and change management, integration tests; you will also learn how to use them in order to work with the other people spread all across the globe. How Can You Get Involved in OSS project? First you need to pick projects in the field that really interests you . The "really interests" part is extremely important, since personal motivation can make a huge difference between personal endeavor pushed to success and something we just spend time on and eventually put aside. Second, you learn as much as you can about it : read documentation (and especially the FAQ); download the source code, try building it and running; walk through the samples, if there are any; sometimes other people will blog about the project - google these articles up. Already at this point you probably have some questions or came across an odd behavior. Share these discoveries ! ask questions in the community; blog about successful and not-so-successful experience (while doing that - please try to be more professional, than I tend to be, say, when writing about Windows Azure); try to figure the problems on your own, propose solutions and submit patches. I can already tell you that almost every single open source project (i.e.: .NET framework or tool) would love to have: better documentation and tutorials; samples; unit tests and small bug fixes; small features that some people want but don't have the skills to do; people using their projects and providing detailed feedback; So you can check out issue trackers (they could also be named as tickets, bugs or feature requests) in a project. They will list problems and feature requests that project owners would love to be done. Yet for some reason they are not done; the reason being - lack of time to do that. So if you step up for the challenge and offer your help - you'll be a hero of the day. If you are just a beginning developer and don't see something you can handle, just go the community and say the words: "Hi, I love your project and would like to contribute. Tickets are too complex for me at the at the moment, but I really want to learn this field. How can I help?". I would be surprised if you'll be able to get away with less than few things to do and recommendations. The truth is: owners of these projects run them in their spare time and will gladly accept any help you can offer . Besides that, they are passionate about their projects and just love to talk, teach, help and share (how many university teachers do love answering questions on evenings and weekends?). Community members usually are no better either - they just love talking about the project, field and improving it, while doing all sorts of things to make this happen. Since there always are some really smart people out there (with a practical experience), this is way better than your average class for learning, sharing and having some fun. That's the nature of the open source. So if you would love to learn some practical skills (and understand that universities might just serve some outdated theory), feel free to check OSS out. For example , in .NET you can start with: For Inversion of Control: Autofac IoC Container ( 22 tasks in the issue tracker) and Windsor Container (more than 50 unresolved issues ) For Cloud Computing and Windows Azure: Lokad.Cloud for Windows Azure ( 23 tasks in the issue tracker) For Continuous Integration: Cruise Control .NET (more than 200 open issues ) For Unit Testing: AutoTest.NET (just 3 issues , but they will have more, plus participating there can get you a beta of Mighty Moose ) or NUnit (38 open bugs in 3.0 version) For service buses: NServiceBus and MassTransit NB: This list is by no means complete . It just serves the purpose of giving the overall perspective. If you want to recommend an open source project to participate or look for one in a specific field - please, read to the end of the article and leave a comment . After participating in Open Source projects, you can also take the next step - start your own . There are always places in .NET community and outside, where the tooling is less than perfect or there only are expensive options. By taking this step, you will learn an additional set of skills which a lot of companies look for in resumes (we definitely do at Lokad ). This will also will provide you with additional background for managing development projects at companies or starting your own business. In fact, an open source project you are passionate about, can become the foundation for the business. But that's a story for another post. Go for it! So if you liked what you heard - I encourage you to go and take a closer look at open source projects in the fields you are passionate about. You could learn a lot of practical and inspiring things, get a good line for the resume (actually companies tend to look for active contributors in the open source projects that they rely on) and give something back to the community. The latter will give you this warm fuzzy feeling of doing good stuff and being part of some bigger effort. I also strongly encourage you to comment below if you : look for an open source project in a specific field with a friendly and welcoming community willing to help you to learn and contribute back ; please indicate field of study/interest . run an open source project in some field (or know such projects) that would love to get more help and interest; please indicate focus of the project and give a few samples of how a novice can help . Contact info would also help. You can also retweet this article and share it with people and communities to encourage diving into open source projects (you might have one or two specific ones in your mind already) and getting real-world experience while helping them to move forward. Links for sharing are below. I'll keep this article updated. PS: As you can see, an innocent-looking item of "Contribute to Open Source Projects of your choice" in 10 Steps To Become Better .NET Developer turns out to be far more exciting and valuable than it might look on the surface. I'm planning to dive in a few more topics from this list later as well. Stay tuned !


CQRS Architecture and Definitions

CQRS Architecture and Definitions CQRS principle, in it's core form, merely introduces separation of reads from writes. This simple approach brings following benefits: denormalized query persistence is optimized for the reads (which usually make up the most of the persistence IO) resulting in better performance and user experience; we can optimize our read side for the needs of the UI (i.e.: fetching dashboard for the user in a single query) which will result in better development experience and less risk of breaking something on the write side. read side can be put on some cloud storage, which is inherently optimized for the reads, could be partitioned, replicated and even distributed via CDN; by offloading data reads to synchronous queries we automatically increase the performance of the write side - now it has lower stress and lower probability of hitting a deadlock (which you should still account for). Deeper introduction and more learning materials are available for study in CQRS Starting Point What about things that you hear in any CQRS talk: commands, events, DDD, eventual consistency and almost-infinite scalability? These are distinct architectural patterns with their own benefits and peculiarities. These patterns play so nicely with CQRS principle (separation of reads from the writes), that they are often perceived as one thing . So when we say "CQRS" this often means: "CQRS Principle" + "Message-driven architecture with commands, events and queries" + "Domain-Driven Design Methodology". This combination is one of the most frequent variations of "CQRS Architecture" (sometimes Event Sourcing is included there by default as well). Success of this variation is the reason why there is so much buzz and hype around the original CQRS term. So here's what we have here: CQRS - buzz-word that could mean a lot of things; also name of the "cult" . CQRS Principle - principle that dictates separation of reads from writes in your system. CQRS Architectures - specific architectural designs based upon the CQRS Principle and a few other time-proven methodologies and approaches. They usually come with a clear evolution path enabling migration of live application to more elaborate design, if needed. DDDD (Distributed Domain-Driven Design) - one of the CQRS Architectures , as presented by Greg Young. It is based upon "CQRS Principle" + "DDD" + "Message-based architecture" + "Event Sourcing". Documents on the CQRS Info site cover this in greater detail. Obviously, multiple architectural designs could be established on top of CQRS Principle, DDDD is just one of them (see my CQRS Roadmap for some other possibilities). I believe, when the CQRS book finally comes out, a few designs will be covered there. Meanwhile, here are the options that I'm aware of: DDDD - CQRS Principle + DDD + Message-based architecture + Event Sourcing Cloud CQRS Architecture - CQRS Principle + DDD + Message-based architecture + Elastic Scaling + NoSQL persistence on top of cloud storage. CQRS with Relational DB - CQRS Principle + Service Bus (NServiceBus/MassTransit) + ORM (NHibernate). Each category has it's own variations and possibilities as well. For example, while evolving your legacy application towards the DDDD Architecture, you could have at some point Event Store and RDB+ORM persistence, where the latter could be dropped in some areas of the system a bit later on. In fact, enterprise system, being build with real-life approach for CQRS, would probably contain a mix of various architectures. We would be picking whatever makes the most sense for each specific business case and subsystem (basically optimizing for the highest mid/long-term business value for the development effort invested). Better categorization of real-life CQRS architectures, their peculiarities, challenges and recommended evolution paths would probably require a bit more research. This could include gathering feedback from people practicing various CQRS flavors and structuring it on top of the theory with the formal methodologies used in universities during the PhD studies (for both learning and presenting). Maybe some time later I'll be up to this challenge. Meanwhile, you can stay tuned to this blog and also RSS feed of CQRSInfo.com (it will get more lively a bit later).


CQRS Lifehacks From Lokad Production

CQRS Lifehacks From Lokad Production So far I've been talking purely about CQRS theory and attempts to settle it down in a logical way for projects of various scalability and reliability requirements. Here's what the recent posts were talking about (check CQRS and xLim sections for even older articles): The Best Way to Learn CQRS, DDD and Event Sourcing Scenario-based Unit Tests for DDD with Event Sourcing Domain-Driven Design, Event Sourcing, Rx and Marble Diagrams Command Handlers without 2PC and with Various Levels of Reliability Theory of CQRS Command Handlers: Sagas, ARs and Event Subscriptions Now it's time to switch back to the real world and this wonderful thing called production. I've seen a lot of CQRS/DDD/ES theory articles and abstract snippets out there (and attempted to contribute to this myself), but I can't recall any posts describing real-world production systems, their problems, challenges and various life-hacks. By the way, if you know such articles or happen to share your experience, please drop a comment or twit me. I'll be sure to include reference to such material, so that everybody could benefit. Real-world systems are rarely pretty, they tend to contain a lot of hacks and miss potentially good improvements. Yet, they are real and they keep on surviving the best test out there - "natural selection" or "survival of the fittest". Theory can only prove itself to be correct by being implemented in such system and surviving long enough to evolve further. I'll start by sharing a few recent CQRS-oriented development discoveries that helped me a lot on the battle-field within the last two weeks. There will be less of nice diagrams and more of screenshots exposing ugly UIs that I hack for myself. I'm probably reinventing the wheel here by trying something done already done by the other people. If you have something to add from your experience, please - share in the comments or in your blog. This will benefit the community immensely. "Real-time" CQRS Server Debugger Essentially this is just a mere console that aggregates and gets all messages (events and commands alike) from the production system. Actual system could be deployed somewhere in the cloud and have access is protected by the HTTPS+DevKey authorization. Since we are reading the past history (with the slight delay) from the cloud persistence (inherently scalable and optimized for reads), this has little impact on the performance (close to none). At the same time it provides almost real-time insight into distributed server-side processes as needed. Each line is basically a string representation of a single message. Some messages get "ToString()" overloads to make them simpler to read. Other's just print their name. public override string ToString() { return string.Format("Send mail ('{0}' to {1})", Subject, To); } For those of you that have been working with Windows Azure and using Trace display of the Azure Dev Farbic, this is almost the same experience. But it works with the production deployments in the cloud and I use it a lot more than IntelliTrace in Azure. Error Notifications One of the crucial differences between ivory-tower theoretical architectures (which might look good in spikes and lab tests) and abused production deployments is the simple fact: unexpected problems happen in real world . Since they are unexpected , we can't avoid them 100%. Yet we can do our best to learn about problems as soon as possible, have the information to fix and be able to deploy the fix as soon as possible. Wiring email notifications to the poison queues is the simplest way to learn about errors fast. Such emails tend to help stabilize system really fast. This works especially well with the fragile integration points or external 3rd party systems that start behaving badly (timing out or returning unexpected results). If you start investigating issue as soon as possible, there is a chance to still catch such system at the crime scene. This makes it's easier to isolate the issue and prevent it from happening ever again by adding some smart behavior (i.e.: saga). Error Log Details Once there is information about error, you'd probably would want to fix it fast (preferably before this starts having impact on the customers). Detailed error log, referenced by the email notification, could help. BTW, you might be tempted to send the entire exception details by the email. I strongly recommend to avoid this path, since it could accidentally (just like the recent padding oracle in ASP.NET) expose confidential information outside the system. It's better to provide unique message identifier in the exception notification, while keeping the actual details in the persistence in a secure way. Once you've got exception details from the secure storage, you just need to copy exception stack trace, paste it to ReSharper ("Explore Stack Trace" feature) and jump to the failing point in the codebase. Domain Event History Sometimes information about the exception (no matter how detailed it is) is just not enough to solve the mystery of the problem at hand. Full domain message log (which comes native with the CQRS architecture) and append-only persistence are one of the best tools for post-mortem analysis . At Lokad we tend to record all available information into the events, just because it is extremely easy to do. This includes (but is not limited to): performance statistics (time spent, number of records processed); explicit details about the security context; denormalized information from the AR. This yields following benefits at almost no cost: easier to write view denormalizers; we've got perfect audit logs; easier to track performance statistics and tendencies of the system. The latter part is extremely important, since CQRS systems tend to be rather complex and dynamically evolving (just because it is so easy to evolve them without hitting any complexity barriers). This forces the system to encounter various real-world problems and scalability limitations as it rapidly grows from the prototype and into the scaled full-blown solution integrating with dozens of various internal and external systems. As long as we track all information in the events, we could data-mine captured knowledge for hints of problems yet-to-happen. Reporting over domain log will help us here. Excel + Domain Log Microsoft Excel 2010 is one of the best tools for analyzing complex data with just a few lines of code. We can take our domain log, write some denormalizing queries, run them through the history and dump resulting state directly into the Excel spreadsheets for charting, pivoting and looking for trends and potential problems. Since such reports are extremely easy to create and run, this encourages for exploration and experimenting and leads to better understanding the system. In the end assumptions about the real-world behavior (in my case they tend to be off-scale, especially when I'm trying to assume bottlenecks and performance impact of some things) are replaced with simple knowledge. For example, in order to mine all history for the report of all mail messages (presented above), one would just need to write a query like: var messagesPerDay = history.Messages .Where(m => m.Content is SendMailCommand) .GroupBy(me => me.GetCreatedUtc().Date, c => (SendMailCommand)c.Content) .Select(x => new { Date = x.Key.ToString("yyyy-MM-dd"), Internal = x.Count(m => m.To.Contains("lokad.com")), Public = x.Count(m => !m.To.Contains("lokad.com")), }); Infrastructure, reflection and some OpenXML will do the rest. Exploring your own domain Sometimes, in order to resolve the issue we would need to get really hacky and send raw messages directly to the system via some endpoint (at least SSL + dev key are recommended to secure such endpoint). Home-grown UI utils, organizing commands and events in a nice way, will help to navigate all the messages and automate sending the right ones. Another use for such functionality is to resend the last failing message from the poison queue back to the command handler, after deploying fixes to production. I used to rely on such functionality a lot while fixing various integration issues. What do you think? What hacks and tools do you use to evolve your systems past new scalability and feature requirements? PS: If you are interested, you might also the next article in the series . It shows how to "teach" Visual Studio a new language (in our case - DSL for specifying message contracts) PPS: You can also jump directly to the next part of Lokad Lifehacks .


CQRS/ES Road-trip lessons learned at

CQRS/ES Road-trip lessons learned at NetworkUA conference at Kharkiv was amazing! It felt a bit unofficial and practical all way through. Attendees, organizers and final feedback were just great. I couldn't even wish for a better start for our CQRS/ES road-trip . There was a outstanding panel of speakers - Sander Hoogendoorn (no frameworks), Ayende Rahien (no-SQL), Greg Young (DDD/CQRS). Note, how all these talks are focusing on keeping things simple and practical. I was presenting on event-centric development (we had the same setup with Greg as in Vienna, where he was doing crash course which I follow up with small bits of practical experience and applying all that in various projects). Obviously, my talk was far worse than presentations of Greg or Sander (well, that's just 2nd time I'm doing this presentation, as opposed to 50th of Greg :), but certain things come only with the experience. Next time it will be better. Sheer lack of time felt a lot: rushing through slides to fit into 45min talk. I was probably talking too fast. But we did catch up with some Questions and Answers in the corridor afterwards (blocking the entire corridor with a small crowd for quite a bit :) Lessons learned: Prepare core structure of the presentation and work it out (i.e.: enough for 20 minutes of a talk). Identify pieces of presentation that could be gradually added or excluded, depending on the lack or abundance of time. Make sure that you are heard well (not all conferences will have head-set mics). I need to finish the FarleyFile sample on CQRS/ES (or, at least, finally publish it as it is). Of course, in addition to the actual presentation-related skills, there were quite a few things learned during the after-party and unofficial hang-outs. But that's more technical (I'll blog about some things later, as promised). Speakers arrangement was amazing - small group, passionate about development and all agreeing on learning and keeping things simple. This made the final panel quite interesting as well - sharing motivation, inspiration and vision of future. This actually makes an interesting rule for the future, when deciding whether to participate in a conference or not. Feedback after the conference was incredible. People were sharing emotions and thanks to the organizers. Well, I'm still receiving emails and DMs. The most important thing: developers are interested in exploring, trying and learning. So they probably got something out of the conference. If I were to help in organizing European CQRS conference some time later (as discussed in the community) - I know much better now how it should look like to be educational, inspirational and also beneficial for the community's progress. Fun included. Thanks to Akvelon and Sigma Ukraine for providing an amazing atmosphere. To speakers - for providing the company, sharing ideas and experience (the morning is a bit rough after the last after-party :). To Microsoft - for providing a great framework and tooling for exploring all these ideas. In short - I've learned a lot already, making this CQRS/ES road-trip really educational already. And this is just the beginning. We'll have an Event-centric IT Cafe in Kharkov today. Then on Monday - we'll start to move slowly in the direction of Kiev towards Event-Centric weekend (depending on how things work with the developers out at the cafe).


Data Anonymization

Data Anonymization Various organizations, such as hospitals, social agencies or commercial companies, sometimes need to release potentially sensitive information to the other parties. They might want to do that in order to gain additional insight into the domain or run a specific research for which they don't have resources. While releasing such information, privacy of people and businesses should be protected. We don't want risk the chance of somebody stealing that data and abusing information in it. This is achieved via data anonymization process - removal of all personally identifiable information while preserving patterns essential for research. Anonymized data can then be shared with the people outside of organization without endangering confidentiality or leaking business secrets. {{% img src="data-anonymization.jpeg" %}} For example, consider a business domain that captures all relevant events. These events, after anonymization process, might look like the one below: { date: "2014-09-04T23:02:00", group: [ { ref: 1, delta: 1 }, { ref: 2, delta: 11 }] } As you can see, there isn't much personal information there. This event tells only that on September 9th something happened, involving two groups of records, with some deltas. It is just a data point now. Can you guess, which event or type of business we are talking about? This is how good anonymized data might look like. It contains enough information to run the necessary behavioral analytics , however all personally identifiable information was erased from it. This prevents cross-referencing of that information and tracing it back to the origins. This anonymized event was produced from the original event by: erasing all human-readable information; reducing precision of times or numeric values; replacing identifiers and tags with sequential numbers (1,2,3); altering field names to decouple event from the domain; distorting some values by introducing random noise (perturbation). For the reference, original event could have looked like this before destructive data anonymization: { tenant : "contsco-ebay", id : "contsco-2024-09-10-0334", time : "2014-09-04T22:57:55", status : "pending", shipping : { country : "Russia", zip : "450075", line1 : "Ufa ul.Zorge 66-61", to : "Abdullin Rinat", carrier : "usps" }, products: [ { id : "ksm6573er", name : "Kitchen Aid 6573 Empire Red", quantity : 1, price : "200 EUR" }, { id : "ka1234", name : "Plastic shield guard", quantity : 10, price : "7.99 EUR" } ] } In some cases, original domain could capture events on a really fine-grained level, allowing to apply data generalization - merging multiple events together into a more generic one. For example, we could sum all product sales in a day, producing a table of daily sales. This significantly reduces information quality but still keeps statistical analysis possible. Leaking sensitive information Diligence is required in data anonymization process. It is possible to leave some personally identifiable information in the data , even though it might not look like that at first sight. For example, hospital records without patient names but with birthdays and cities can be cross-referenced with other data sources, potentially leaking identities. In this case, it is better to weigh the risk of disclosure and invest extra effort into data anonymization. For example, birthdays could be randomly shifted by 30 days or rounded down to years. City names could probably be discarded, unless geographical location is required by the research. Losing valuable information It is possible to apply too much anonymization and discard valuable research information in the process . For example, aggregating individual sales into daily sums leads to loss of information on the shopping habits of customers. Researchers will not be able to analyze them, coming up with models for promotions and personalized discounts. Instead of aggregation, it could be possible to hide individual customers behind artificial numeric identifiers (1,2,3...). If extra caution is needed, it would be possible to reduce the risk of cross-referencing by applying various data transformations. For example: replace each sale of "Google Nexus 5" with 1 "Luxury phone" and 3 "post cards"; whenever a person buys 1kg of candies, multiply the amount by 2. Applying these transformations allows to keep important patterns in the anonymized data without making it useless for research. These are oversimplified examples, of course, but they can serve as an example of the general approach. We remove some confidential data and 'encode' the rest in research-friendly way, while keeping the key secret . Summary Data anonymization allows organizations to share private information with external researchers. This process has to be done carefully, balance kept between the two extremes: leaking personally identifiable information and removing bits valuable for the research. There are no general rules of thumb here.


DDD - Evolving Business Processes a la Lokad As you already know, there are multiple ways to express any given core business concept in the code via domain modeling (we discussed this topic in previous article ). These ways usually depend on the architecture style selected for the bounded context, in which we are currently working. For now, let's focus on one of such domain concepts: long-running business processes . In a cloud-based SaaS company, we could have following business processes (among many other): if invoice has a non-zero amount and was not paid within 15 days, then send customer a reminder. if customer balance stays below -5EUR for more than 30 days, then issue a lockdown. if distributed computing process has not finished processing all data batches within 1 hour, then restart it once (except cases, when it was already restarted - then issue a termination alert) As you probably already noticed, these examples share a few similarities : they are aware of the passing of time and deal with it; these processes express rather complex precondition that is based on current state of the system and leads to one or more then outcomes. Let's assume that we are dealing with a distributed system, where information about current state is shared with events. In such case, our business process might resemble a piece from complex event processing and would look this from the logical perspective: How can we implement this "Business Process" box? There are multiple alternatives, depending on the architecture style you have chosen. For example, you can use a state machine, where each instance of state machine would correspond to a specific process instance that you are tracking. Events would then be used to navigate an instance of the state machine across the nodes. It will also use external timer service to send messages "to future" (where message is put on hold till certain time comes). State machines are good for formalized domains. You can learn more about such approaches in the materials provided by Gregory Young and Udi Dahan. However, when we are dealing with business processes, that are rich with fuzzy logic, uncertainty and also happen to evolve rapidly, then a more simple solution might be needed. Especially, when you have almost no development time to spare. What is the most simple solution in case with locking customer balance for overdrafts? For instance, we can project all events to a view, which will track all active customers that used our services and went below the threshold at some point. Then our execution will be responsible for regularly checking this view and sending "Lockdown" to every customer that had his balance below the threshold for too long. This component would also need to keep in mind that certain customers require special handling and investigation before being locked out, while others can be locked right away. Naturally, these rules will be changing really often. What is the fastest and most flexible way to implement such component in a rapidly growing and changing environment? You simply wire view to the UI, attach a button to send "lockdown command" and ask a person from the business department to spend half an hour per week processing all late customers . This will save dev department hours on implementing these complex execution rules, testing them and then changing (as business discovers new corner cases). Essentially we let the rules evolve and change in the environment that shapes them: in the minds of business managers. In other words, at this point we avoid large development effort with a little bit of human time . Please, keep in mind, that once business processes are established and we have so many cases, that manually processing them takes too much time (that should be a profitable company by then), we can always rewrite these lockdown rules as a continuously running server-side task (rules would be mostly established by then). We could still keep the projection and a corresponding view. At this point we invest a fixed amount of development to automate a large portion of manual work . This gradual evolution of business processes is currently the recommended approach within Lokad.CQRS architecture style for delivery of non-formalized and rapidly changing business rules.


DDD From Reality to Implementation

DDD From Reality to Implementation This is yet another post in a series that were triggered by fruitful discussions with Vaughn Vernon over content for his DDD book. I think, one of the sources of confusion in DDD/CQRS world is that we often mix terms and concepts that belong do absolutely different layers (and that we don't know how to go from one to the other). Let's start by introducing the following separation: {{% img src="ddd-from-reality-to-implementation/2012-04-17_DDD-reality.jpg" %}} For the new readers, DDD stands for Domain Driven Design, which was introduced by Eric Evans in the book with the same name. CQRS stands for Command-Query Responsibility Principle, which is often associated with architecture styles for implementing systems with DDD and optional Event Sourcing. The term was coined and explored by Greg Young . Reality Reality is that thing around us, which we perceive through our senses and continuously try to understand. In the context of business and software, reality contains core business concepts which are important for the competitive advantage of our business. We want to capture them and then somehow express in code for automation purposes. For example, business concepts could involve things as: Customer Registration Process Customer Subscription Invoice Invoice Payment Cycle I'm taking examples from the environment of Software-as-a-Service (SaaS) company, since that's what I'm mostly familiar with. Domain Model As we learn more about reality and business concepts, we could distill our understanding into the domain model , which contains all things that are relevant and important in the current situation. For the sake of simplicity, we will break down the entire model into set of bounded contexts (BCs) which are separate by the natural boundaries we've discovered in the real world. In SaaS world we could highlight BCs like: Customer Subscriptions and Billing Client Portal Reporting Product 1 Product 2 Cloud integration etc. Each of the bounded contexts in this model stems directly from our understanding of the reality and the natural boundaries that we have identified ( read more ). If we dive inside one of these bounded contexts, we could discover more fine-grained concepts: Ubiquitous Language Aggregate and Aggregate Root Consistency Boundary Business Process Please keep in mind, that these are purely logical concepts, that have (yet) nothing to do with the implementation and all the less important details! The process of identifying BC boundaries can take into consideration things like: teams, skills, available resources and technologies. However, at this level we still don't care about technical details like : frameworks, databases, message middleware, service buses etc. We just create foundation for making conscious choice later down the road. Architecture Styles and Implementation Only after we have identified bounded contexts, we can focus on each BC and start thinking about implementation matters, while considering project specifics. Result of this exciting process would be a choice of key elements for the specific bounded context: development process - how do we organize and manage our development. architecture style - how do we structure and design software implementation. technology stack - what technologies do we use and how do we get them resource allocation - how do we get resources (budgets, people, knowledge) for the project delivery. For instance, if our teams are familiar with the SQL/RavenDB and NService Bus, we can pick architecture style described by Udi Dahan ( blog ), where: aggregates are persisted with SQL+NHibernate or RavenDb; command handlers and application services are hosted by NServiceBus; business processes are implemented with NServiceBus sagas; consistency boundaries happen within the transaction scope; views are created either by in-memory events or via projected audit logs; development process will be aligned towards Waterfall or Agile. If environment requires event sourcing or teams are hyped with AR+ES architectural style of Greg Young ( blog ), we can: persist aggregates with event sourcing in event streams; host command handlers in custom message dispatchers that use something like AMQP or direct socket communication; business processes are mainly implemented with state machines hosted within event handlers or via document-based state machines (where state is persisted in messages); views are disposable and are projected from event streams to whatever technology that is needed; consistency boundaries are within aggregate; use Agile development process. At Lokad we are mostly using Lokad.CQRS architecture style ( sample ) that is derived from Greg's but is fine-tuned to: fit cloud computing environments while supporting on-premises deployments; reduce development friction and development effort at the cost of higher requirements for team skills and discipline ; support rapid domain evolution in rapidly changing business environment. This architecture style involves following technical choices: aggregates are persisted with event sourcing in event stream; message handlers are hosted in custom message dispatchers provided by Lokad.CQRS sample project (with adapters for on-premises and cloud deployments); business process transitions are implemented as part of the aggregate behaviors (they could be triggered by user interactions, stateless event handlers sending commands in response to events or tasks that sending commands on a schedule); views are disposable and are projected from event streams, using dead-simple key-value persistence for the majority of cases (with adapters for on-premises and cloud); consistency boundaries are per entity (aggregate or view instance); rapid development process is used for multiple releases per week/day. Obviously, these are just a few options of implementing a given bounded context. There are more architecture styles available out there (and each architecture style can have multiple implementation options and variations). DDD Modeling Process Now, for the most fun part. This trajectory from reality to architectural style is just a happy path scenario that happens only in dreams. In reality you might need to iterate from reality to implementation multiple times : {{% img src="ddd-from-reality-to-implementation/2012-04-17_DDD-iterate.jpg" %}} These iterations are the foundation of the approach to explore and capture core business concepts in domain model (or, one of the approaches). Approach is totally attributed to Greg Young. We start by sketching out domain model without even trying hard to be precise from the start. Then we try to implement it in the code using the most hacky approach possible . Lokad.CQRS style with file-based persistence and messaging works for us, because it is designed for rapid iterations and has the least friction (I'll need to do a quick video on that process). One of the goals here is to build a set of unit tests that use specifications to verify behaviors of aggregate roots with event sourcing (AR+ES). These specifications can be printed out as use cases in human-readable way. More often than not, a lot of problems and questions show up during this implementation phase. At this point we usually forget about the implementation (while keeping our use cases) and go back to the domain experts with these questions (more often than not, for me this boils to going and talking to the mirror). Then we adjust the domain model according to the lessons learned. Depending on the nature of the adjustment, implementation is either discarded completely or refactored to fit (AR+ES specifications are usually kept between these iterations, to make sure that we don't loose any captured use case). The process is repeated till we distill our domain model to the point that it captures all required business concepts in a way that can be easily implemented in the code. At this point we can print out specifications, reconfigure implementation to use adapters for production environment (e.g. Azure), push it to production and try to call it a day (only to have next challenge handed to us). PS: If you are interested in this topic, next article in the series might interest you: DDD: Evolving Business Processes a la Lokad .


DDD Summit 2012 Summary -

DDD Summit 2012 Summary - Some time ago I was honored (probably because of some sheer mistake) to be invited to DDD Summit 2012 organized by Eric Evans. The event took place in Portland over the course of last week and it was an absolute blast for me. From left to right: Cameron Purdy - VP of development at Oracle [ Blog , twitter ] Daniel Gackle - CoFounder at SkySheet Randy Stafford - A-Team at Oracle [ About ] Jimmy Nilsson - Factor10, Author of Applying DDD Design and Patterns [ Blog , Twitter ] Rebecca Wirfs-Brock - Wirfs-Brock and Responsibility-Driven Design [ Wiki ] Vaughn Vernon - Consultant at ShiftMethod, Author of the next DDD Book [ Blog , Twitter ] Andreas Brink - Software developer & coach at Factor10 [ Blog ] Alberto Brandolini - Avanscoperta , DDD Instructor and master of tomato [ Blog , Twitter ] Eric Evans - Father of DDD and founder of Domain Language [ Twitter ] Dan Bergh Johnsson - consultant and partner at OmegaPoint [ Blog , Twitter ] Paul Rayner - DDD Instructor at Domain Language [ Twitter , Blog ] Martin Fowler - needs no introductions [ Wiki , Twitter ] Patrik Fredriksson - Consultant at Citerus AB, DDD Instructor [ Blog , Twitter ] Rinat Abdullin - accidental visitor, Tech Leader of Lokad In essence, we were discussing various aspects of DDD in small focus groups and all together, going out on hikes and dinners, talking, talking and talking. There was a wide variety of topics covered during these three days. For me, this was an amazing opportunity to dive deeper into DDD nuances, practical experience and ideas shared by the leading thought leaders. Outcomes and Commitments Long story made short, there are some outcomes of this event that should benefit of the community (including some of my personal commitments). There is going to be more activity on the DDDCommunity.org site , that might have felt a little neglected over the course of last year (with a lot of good materials being hidden by accident). Things will start improving within the next few weeks (more regular updates and more structured community). I personally hope to contribute some small practical experience bits of community building from projects and groups on DDD/CQRS (i.e. CQRSGuide and CQRS Beers with awesome communities around EU) , Lokad projects and Distributed Podcast , maybe even bringing all these closer together. Practical samples are a big thing for demonstrating DDD. We plan to bring them together on a github and refresh a little bit (may be even adding Scala/Akka implementation!). My own Lokad.CQRS Sample Project is essentially a sample of DDD (with all the latest tech stack, cloud portability, practical things and even some DDD modeling toolset) that currently lacks the most important thing - proper domain model. I plan to fix this last problem by the mid of June, in addition to committing to things I've been delaying far too long (i.e.: articles and videos about the sample project). More articles and materials about the Domain-Driven Design are expected be published; some of the topics were already outlined by the summit participants. There are plans to establish better environment for reviewing and shepherding such materials, which should definitely help. I personally plan to continue writing about various aspects of DDD/CQRS+ES. Hopefully these new articles will start making slightly more sense due to focus not only on disposable technical details (which CQRS+ES is, despite all its coolness and cloud-portability), but also on strategic and design decisions governed by DDD. Besides, some more peer pressure is expected on them :) There are even more cool long-term plans about pushing DDD and state of the art in the community, however they are long-term. Meanwhile, you can stay tuned for the updates via #DDDesign (I'll make sure that any official DDDCommunity news will be auto-published on twitter via this tag as well). How does this look?


DDD/CQRS Challenge - Integrating Distributed Systems Let's have a look at the relatively simple DDD/CQRS challenge in integrating elements of a distributed system composed of a different bounded contexts and deployed across different hosting environments . Let's imagine a small Software-as-a-Service company which provides some subscription-based service while charging customers per consumption on pay-as-you-go basis. Software infrastructure of such company could consist of only 3 bounded contexts (a major oversimplification on my part, bigger view might be more complicated ): Subscriptions - subscription management system, that keeps track of all customers, their active plans, billing information, invoices, monthly service consumption and available login keys. This system is architected as NoSQL solution with event sourcing and is deployed on a dedicated server (with plans to redeploy it to Azure some time later). Cloud Services Integration - massively scalable set of services deployed in Windows Azure (e.g. using some big data processing design ). Among the other things, these services expose API to 3rd party companies and even products of the same company. This API is secured by user tokens, which are replicated from the subscriptions BC. This project is stable and does not change frequently. Product 1 - a new product being delivered by the company. It is developed as a standalone set of systems that enhance user experience, using Cloud API. This product leverages authentication and user management capabilities from "Subscriptions" and interoperates with API. Here are some examples of the interactions between these system: If new user is added to the subscriptions, it's auth credentials should be immediately (within 1-2 seconds) replicated to Cloud Services, to enable access via API. If customer's account is locked out due to balance overdraft, then all related users should be locked out of the API. When services consumption is detected in the API, it should be within 5 minutes reported to subscriptions portal. Naturally all these systems have to work independently in such way, that if one of these is down, the rest will continue doing their part (at the very least by providing read-only UI, at best - doing everything that is not dependent on the other systems). For example, if subscriptions are down for maintenance or Cloud Services and Product 1 should continue working as they were (all pending changes should be replicated after system comes back online). Additional constraints: Resulting design (with inherent implications) should be relatively easy to explain to a Junior dev. It should also be relatively straightforward to deploy and run systems both locally (xcopy deployment of .NET code) and in the cloud. systems should be able to change independently and rapidly as they follow their individual DDD evolution paths (for example, weekly releases with new business processes but without breaking any relations). no more than 3 people per project to develop and maintain it. Note, that we are focusing here only on the integration between the systems. Internal design of each system might affect such integration, but is less relevant in this case. Still it would be nice, if integration patterns shared natural affinity with internal design of each bounded context (this tends to create systems that are more robust and practical). How would you approach this problem?