Keeping Mind of .NET Developer Open

Keeping Mind of .NET Developer Open Yesterday I've bought a new laptop, replacing the old one (video output to LCD got fried). I used to stick with DELLs (robust and can take almost any beating), yet this time, there wasn't much time or pocket cash to spend on the upgrade. So, without even thinking a lot, I went for Lenovo laptop with the slowest i7 processor, 4 GB of RAM and a home upgrade to 120GB Corsair SSD (Force) that came a week earlier. This upgrade (and OS reinstall) was an opportunity for me to start rethinking my development approaches and the whole CPU MHz race (in which any dev with VS 2010 and R# eventually ends up). I used to be jealous of blog posts and videos of developers working on blazingly fast Linux IDE environments and still staying really productive. At the same time, development story at Lokad has been going in a slightly opposite direction. Our build, integration and deployment scenarios are becoming so complex, that the big picture and evolution path tend to blur. Even usual everyday handling of dev and integration tasks on a dev box adds a bit of a mental burden and friction. Windows Azure with it's "lightweight" build and deployment process does not make things easier. My gut feeling, based on the limited development experience, came up with the rule of the thumb: If you need to keep solving same problem (in slight variations) over and over again, then you are going in the wrong direction. Think outside the box. Look on the other side of the fence, if you need any hints. Here's an example - optimizing database calls was rather a common routine for me a year ago. I used to spend days on tasks like writing smart SQL bulk inserts with high performance, adding small caching layers and optimizing queries. Needless to say, that all these "features" took a lot of time, increased complexity and were eventually thrown out of the codebase later. Perf problems were solved at the conceptual level by a simple paradigm shift from coupled data-driven models to message-based cloud architectures. I believe, same could be achieved with the distributed development targeting cloud architectures as well. Classical approach of having high-end dev machine, build server with regular deployments and central version control system - is too coupled and, well, centralized. We should be able to improve the big picture both in simplicity and friction. The latter is extremely important to me for two reasons: Reduced friction means faster reaction time (and ability to get rich feedback loop). For the business, having low-friction development means ability to execute faster, eventually outperforming competitors and getting the most out of the opportunities showing up. Reduced complexity leads to better decisions . Less moving parts you need to keep in mind, easier it is to come up with solutions that consider all primary factors and future probabilities (risks and opportunities alike). This starts from the development, but, for a software-oriented company, goes all the way up to the organizational and policy levels. In a sense, switch to the Cloud Computing and Cloud Architectures (CQRS + DDD + etc), was a big step for Lokad in the last few years. Lot's of lessons learned. Yet there are more potential reductions to be gained in order to keep progress steady and stress levels - reasonable. In order to keep my mind open to such opportunities, while learning as much as possible, I've decided to switch to Linux as my primary day-to-day OS (currently - Ubuntu 10.10), while keeping .NET Microsoft Development isolated within VMs. Integration dev will still stay where it is now - remote and cloud servers. A few insights from less than a day of using Ubuntu with VirtualBox-ed Win7: It takes a few minutes to install Linux distro to HD and Win7 in a VM, provided you use SSD. Linux forces you to be ready to learn new things (acceptance). Seamless mode of VirtualBox is an interesting experience, never thought this to be possible. I tend to use mouse less, which is a bit frustrating (all the keystrokes to learn), but speeds things up in a way similar to R# jedi tricks. Mercurial is really painful and slow for cloning large repositories. I was thinking about upgrading to 8GB of RAM. I'm not sure any more if it's even worth the effort of going to the shop. Linux host stays as responsive as ever (meaning, I can work emails, documents, code) even when Win7 in VM tries to stress machine by doing large compile or NGening .NET 4.0. Needless to say, that I gave away 75% of CPU and RAM to Win7. Thinking "How can I keep evolving projects to simplify development without VS10?" already brings up ideas on organizing, testing and integrating large multi-project solutions in a simple and straightforward way within the VS10 stack (before that, there was no coherent picture). Nothing really new - just the common principles of proper component development, applied at the project ad solution levels. It feels good to have a plan, though. Organizational structures and processes layer on top of that in a rather straightforward manner as well. PS: The discussion continues (with some nice comments) in the following posts: Linux Setup Tweaks of .NET Developer with SSD Why Even Bother Trying Linux


Key CQRS Ingredient

Key CQRS Ingredient Recently there was a bit of feedback around CQRS (mostly with the regard to Lokad.CQRS for Windows Azure ), where developer gets along with the architecture, design concepts and all the other exciting little things of CQRS, but then he just gets confused. There's too much tech information at once and too little ideas of where to go next. Questions tend to be like this: How do I design these commands and events? Why can't I just have one command handler that processes all requests? I have a problem I want to solve with Windows Azure and your framework. I did the tutorials. What do I do next? I recall being in the same place as well: elements of CQRS architectures make sense, but aside from that it's all too overwhelming and confusing. You don't understand why is everybody so excited with all this mess. You don't see the clear learning path forward . Let's see what we can do to simplify further learning and avoid possible disappointment. First, we'll take a little step back and walk over the basic concepts again. We'll start with the purely technical ones (excerpt from here ): Command-Query Responsibility Segregation Principle is about separation of reads from writes. Circular Architecture (another term that comes up in the groups; don't confuse it with the onion architecture ) is about structuring these reads and writes in a decoupled way with the help of asynchronous messages (namely commands and events) and synchronous view queries. Event Sourcing is another idea - persisting objects as a stream events that happened to them. These are just technical concepts that might look cool but aren't extremely exciting on their own. These are actually the things that have been used over and over in various systems for ages with various degree of success. Understanding these tech ideas alone is not enough for efficiently building valuable and scalable systems in the cloud and on-premises. There is the key ingredient missing from this picture: Domain-Driven Design . DDD is about understanding, explaining and evolving your domain model (that abstract thing that we are trying to express with the code) in a way that: model would focus on the most important characteristics of the problem at hand (while putting less important things aside, for the sake of preserving the sanity of everybody); the model could evolve and still stay in sync with reality; model would help different people with various backgrounds to work together (i.e.: users, sales people and hard-core developers); model would let you avoid costly development mistakes (it could even help to deliver new exciting features as a simple logical extension of what has already been implemented). Isn't that a lot for "model": something vague that's not even a code or final architecture document of 200+ pages? Well, DDD is not just "model" , but the methodology and principles to create it, distill and keep evolving afterwards. It's way of thinking, learning and talking about the business problem in a manner that implementing everything would be rather simple, despite the initial complexity of the actual problem. CQRS architecture approaches build upon this way of thinking and created models: here we extend the DDD methodology and add new exciting ways to implement everything as a scalable and cost-effective software. We make heavy use of terms like ubiquitous language , bounded contexts , context maps , linking these directly to the technological and architectural constraints of CQRS Architectures. That's actually what Distributed Domain-Driven Design is - DDD principles enhanced with specific CQRS-based patterns-n-practices and a clear way of migrating existing systems towards better complexity management, scalability and integration capabilities. The same is with the Cloud CQRS Architectures (or xLim 4 ) - Domain-Driven Design and understanding of your domain model are the important milestone to go for. Once you have it figured out - things like messages, routing, partitions and elastic scalability will just unfold on top of that model in a rather straightforward fashion without the risk of creating non-scalable, overengineered or expensive cloud computing system. So as you can see Lokad.CQRS for Azure is not extremely important or outstanding on it's own - it is just the framework to help using DDD/CQRS principles for building systems powered by Windows Azure Cloud Computing Platform. So if you got caught by this "CQRS hype" with all it's cool technological promises and then got stuck, I recommend to proceed as: Read the Domain-Driven Design by Eric Evans . Go through the materials on Cqrsinfo.com site (to cover the transition from DDD to DDDD). Start a small learning project using DDD and the most familiar technologies and architectures. Gradually walk through migrating this project towards the CQRS architecture (either cloud implementation or not). These references are already available in CQRS Starting point , but I wanted to highlight the importance of DDD one more time.


Legacy Code should still be Agile

Legacy Code should still be Agile We all have faced numerous situations with the legacy code and brownfield applications. General feeling towards these is "there is too much development friction, it would be easier to scrap the code and start a new version". For a long time I've been a believer in this approach as well. This "long time" actually goes towards the beginning of my development career. Yet. despite the urge to rewrite everything, most of the times the situation went to restraining myself and gradually evolving the codebase. This path initially feels to be much harder than a simple rewrite, simply because there are more variables software architect has to consider (as we all know a person can keep in mind only 5 +/- 2 distinct entities at a time). There are a few solid advantages, though: Code stays in production-ready state. This keeps the business value at a steady level and does not break the communication loop. Evolving code guarantees that there will not be any missing features or other small-but-important details that tend to go missing in the first rewrite version. Evolving code will stay within the feedback loop of quality assurance teams and customers. This would make it much harder for the codebase to go astray (which tends to happen when we, developers, close up in our dens to write the next version from scratch). Pushing existing codebase is quite hard, due to all the momentum complex systems accrue. However, it creates a much better understanding of domain and applied technology at hand. So the resume is "dropping legacy code is bad, evolving is better". How do we do it ? . Here's a simple approach I've came to: Isolate logical portion of the code that has to be evolved in the iteration. Ensure that the scope of iteration is limited and understandable. You might want to use Project Management templates while documenting the scope. Documentation part might be extremely important (even if it is merely about a few paragraphs), since it helps to clarify and settle down vision of the iteration. Written document would also serve as a nice restraining factor preventing the scope creep . Ensure that it is covered with unit tests and other types of tests, if applicable. Implement the prototype This might take some limited time (and a few rewrites), since we will be seeking for the right ideas and implementation concepts Component-driven development will help. Merge the prototype Write down for yourself somewhere: Lessons learned Perspective Basically, code itself does not really matter. Ideas and logic behind it are the real value. I'm not talking about the "business value" at this point (which is the development driver and focal point), but rather about architecture, patterns and composition. They are unique for every single domain.


Less is More

Less is More One of the best lessons I've learned recently is about creating value by removing things that you don't need . This might sound like a controversy in our modern world that is obsessed with hoarding things: features in products, possessions, friends in social networks or frameworks and tools. However, consider this: Apple built an iPhone by removing all buttons except one and putting really restrictive constraints on applications. 37Signals have created quite a reputation by building products that focus on simplicity. Japanese people created beautiful poetry in form of Haiku, that is extremely limited in it's form. Twitter has created wildly popular service by limiting people to 140 characters. In each of these examples, nothing prevented people from avoiding these limitations and going wild. However they choose something non-intuitive - they discarded everything that is not important and focused on the very core. And then a small wonder happened - they created something outstanding that would be remembered by the others. This is an extremely valuable ability and skill: to be able to focus on less in order to gain much more . It is about self-control and self-discipline. This is a way of planning the future that looks a bit deeper than short-term hoarding of features, ideas or things. I'm not asking you to reconsider your current points of view upon the world. However, if you ever come to a crossing point, just ask yourself one of these questions: What if we discarded all frameworks that are not utterly essential for the system? How much would we gain or lose if we discarded half of the features in our system? What is the smallest step needed to solve the problem? What if we try to work smarter instead of hiring more people as the company grows? You can find more interesting questions in Greg's post . By continuously asking yourself these questions, you might be able to stumble upon precious insights and revolutionary discoveries that otherwise would be hidden by clutter and blurred by complexity. Simplicity is the most valuable feature. And it is the hardest one to achieve. It's much easier to add code, functionality or technology, than remove. However as we add them, we lose focus, freedom and flexibility. We might be even so encumbered that we would not be able to avoid problem or jump after new opportunity. More is less. Less is more.


Lessons Learned - Development Methodology Software development is a really curious thing to study. It is all about rapidly changing environment, endless risks and people (the most important thing here) that try to hit moving targets in that mess. There are numerous thoughts on the web about the things that should be done in order to succeed in this game. Here's the list that I currently agree with: Adopt efficient development methodology. And keep doing that. There is no best methodology; only continuous improvement will give the development (and the business) a chance of staying efficient. Communicate efficiently . Use meetings, teleconferences, newsgroups, flash movies in emails - anything that's needed to find problems and make decisions quickly. Efficient and stable communication channel saves resources. Additionally there is the problem of communication between the knowledge domain boundaries (i.e. business to development). Well, in this case either get better communication channel and media, or eliminate the problem by putting both ends into the head of a single person. Know your field . The majority of wrong decisions were made because there is not enough information about the problem and the alternatives. Wrong or suboptimal decisions lead to wasted resources. Spending some resources to prevent chance of wasting a lot of them, could be a good deal. This applies to everything: picking development tools, technologies, platforms, market niches etc. Be flexible and open to changes . Statistically it is more efficient to operate with risks, chances and probabilities, rather than to try to nail the Plan that was carved in stone. Minimize the waste . That is it. Saved resource unit does not differ from the resource unit that was earned. It is all about the people . Right people in right positions can move mountains. Basically, if the rules of the game change frequently and you neither can predict those changes, nor influence them, then the only way to succeed is to adapt to those changes. Evolve. Faster - the better.


Lessons Learned - Diagnostics

Lessons Learned - Diagnostics Bigger the projects are is, harder it is to make sure that everything is running smoothly and there are no unknown bugs. This becomes even harder if the software is highly flexible and configurable. One way of dealing with this challenge (in addition to the UnitTests and Continuous Integration) is adding some more run-time diagnostics to software. For example you could ship some Unit Tests with your product (Kudos to SSW for the idea) or run built-in diagnostics. One of the easy ways for implementing Diagnostics - just sprinkle self-checking code in critical places of the system, and then run every validation (check out the Validation Application Block, for example) method while building the visualization of your system.


Query Multiple Aggregates from Event Stream Given a domain log (containing all events and commands of some context) it's really easy to load any aggregate (or any group of them) and perform some query, given that you are using event sourcing . Last night I've got a problem with one of the systems running on staging grounds in Rackspace cloud. This system performs some big data processing and analytics for OSA analysis . One of the required steps is to assemble data and then run massive Map Reduce operation on top of it, involving some 3rd party services. Roughly 8 reduce batches (out of 300) failed with the timeouts caused by that external service. This morning I have to figure out the exact data problem and find ways to 'scavenge' the results without rerunning long operation again. Luckily I'm using good old CQRS/ES to manage all behaviors and integrations (actual data is never referenced or managed by AR+ES entities). Also, to facilitate debugging (and view rebuilds), systems record all passing messages into a separate event stream called domain log . So the task is reduced to getting the latest version of this stream (which is just an append-only file) and loading it in a snippet: var dataSerializer = ProtoBufDataSerializer.LoadAndScanDomains(); var envelopeSerializer = new ProtoBufEnvelopeSerializer(); var envelopeStreamer = new EnvelopeStreamer(envelopeSerializer, dataSerializer); // load all messages var stores = new FileTapeStream(@"C:\temp\domain.tmd") .ReadRecords(0, int.MaxValue) .Select(b => envelopeStreamer.ReadAsEnvelopeData(b.Data)) .SelectMany(b => b.Items.Select(i => i.Content)) // pick only events for an aggregate bound to StoreId .OfType<IEvent<StoreId>>() // group them and load into the state objects .GroupBy(b => b.Id.Id) .Select(events => new StoreAggregateState().With(events)); foreach (var store in stores) { var missingForecasts = store.Forecasts.Values.Where(f => !f.Delivered); foreach (var forecast in missingForecasts) { Console.WriteLine("Missing {0} with key {1}", forecast.Dataset, forecast.ApiKey); } } That gives me enough data to check out the missing batches and deal with the situation. Important: obviously the purpose of this snippet is to facilitate the debugging. In production you would almost never use such queries across the entire domain log . Projections are a better fit here. This approach relies on strongly-typed identities and their binding to aggregate roots in order to simplify querying of event stream. However this is not essential.


Lokad Topologies

Lokad Topologies Let's talk about system topologies of Lokad.CQRS application. System topology describes flow of messages between elements of a single application or distributed cluster. If you remember classical CQRS triangle (client-application server-projection host), this is an example of topology. Normally this should be an episode in Being the worst podcast. However this is a complicated topic for first episodes, and I don't want to make some of the advanced Lokad.CQRS users wait for a few months. There are multiple ways to organize topology, depending on your scenario and technology. I'm going to present you one of such ways, that is partially based on CQRS principle with some support for multiple BCs and scalability. While reading that please keep in mind, that the core focus of this topology is to keep things as simple and uniform as possible. That approach would work with minor modifications for basic projects, however more complicated scenarios might require stepping back from the starting template. We'll start with revisiting basic elements. Here are some assumptions and terminology: Currently our building blocks are: Application Services, Ports, Projections and Tasks. Application Service is just a class that hosts multiple command handlers (methods that are called when command message or remote procedure call arrives to server). Application services contain core business logic and can either be stateless (functions that take command, do some action and publish result as an event) or stateful. Stateful application services usually deal with entities which are uniquely idenfitied by an identity. In Lokad.CQRS projects the majority of stateful application services host aggregates with event sourcing (or A+ES). Currently in Lokad.CQRS the only way to call application service is by sending a command, which will be put to a persistent queue. Message dispatcher on the server will pick this command and dispatch it to the corresponding command handling method on an application service. Application services hosting A+ES do not publish any events directly; instead, they append events to the event store. Message publisher will send them out later. This functionality is being introduced in IDDD branch of Lokad.CQRS and helps to solve the problem of two-phased-commit (if you don't know what this is - don't worry, that's legacy from SQL times). Application Service with A+ES Application service with A+ES and an event store looks like the bit below. We can scale this out between multiple threads on the same system by paritioning. Our partitioning logic will assume that entities with the same ID will always be handled on the same machine. For instance, we can use such router code: ICommand<IIdentity> cmd; IQueueWriter[] queues; queues[cmd.Id.GetStableHash % queues.Length].Put(cmd); In this case, if we have 2 queues, then commands to aggregates with IDs like 0, 2, 4 will always go to the first queue, while IDs 1,3,5 etc will be handled on the second. These application service instances can be located on different threads within the same machine or located on other machines. This approach allows us to parallelize execution, while still ensuring, that all commands of a single aggregate instance will always be handled within the same thread. Such thread affinity allows to simplify a lot of otherwise complicated synrhonization and concurrency scenarios at the cost of some idle threads (or even machines). Fortunately, hardware evolution favors low-power multi-core systems (think about all these energy-efficient ARM chips), which will benefit from this architecture even more. Functional Service With functional services everything is a bit simpler. They handle commands in idempotent fashion (preferrable) and produce events but without any state affinity. This makes them easy candidate for hosting some resource intensive logic (like number crunching). Since we don't have any state to synchronize access to, we can scale such tasks by distribute commands around in any way (e.g. having 8 functional threads on 4 different virtual machines). Round-robin and random distribution are simple but rather efficient approaches to balance the load more-or-less evenly. Besides, you can always add multiple functional command handlers to the same queue, making them compete for work. You might go even further and have these functional command handlers on a separate VM, increasing or decreasing number of VM instances according to the load (e.g. number of unprocessed messages in the queue). You can have these functional services publish events by writing them to the same event store, or you can throttle load on that event store by publishing messages to the "event recorder queue", handler or which will write events to the store. Client interactions Presence of command routers and functional recorders come in handy, when you need to bring client applications in picture (e.g.: web client or smartphone client). These applications would need a way to send commands (rename user) or report events (user logged into the web UI). Obviously, these would go to the same "command router" or "event recorder" queues. Note: you can easily have one single "router" queue, which will either route commands or record events, but I found this to complicate the implementations. It's easier to be more explicit. Bringing this all together Having said all that, let's have a look, how all these elements come together for one possible configuration of topology. This picture might look complex (it is actually a topology for Lokad.Salescast2 product), if you try to "swallow" all of it at once. If you break it down into individual components - things should get easier. Please note, that implementing this full topology is an overkill for a lot of scenarios . Often you can live with just a few queues and handlers. However, sometimes it helps to know in which direction you can evolve your topology, should requirements become more demanding.


Long story made short - Project Report for the xLim 1 xLim 2 is a set of principles/guidelines that come from some experience and serve specific purpose of efficiently building flexible and light distributed information management systems. Long before the xLim 2 there was xLim 1 (or simply "Lim" at that time). It was purely a research/study project undertaken in the spring of 2007 to explore efficient ways of building distributed Smart Client applications for information management purposes. It took 52 days to get from the project initialization phase to the proper close-out. As you may know, one of the PM principles is to capture all the "lessons learned" on the close-out before moving to another project. That's how it was done with the Lim 1 as well. If you are interested in the history, then you can download xLim 1 Project Report . It is a short 15 page summary that highlights major areas that constitute the body of knowledge captured in the course of xLim 1 project. Additionally there is an archive of screenshots (referenced as Finals.zip in the report document). Here are some pictures from it to give you the idea: Right now this old report seems to be quite naive and incomplete (it actually is), since there has been so much learned with xLim 2 (it looks like it would be extremely hard to keep the xLim 2 report under 50 pages). Yet, it was an important step. Without it xLim 2 could not have started and branched off several commercial implementations. Any comments, feedback or questions are welcome. PS: I definitely wish that other development teams (esp. DevExpress XAF team) published reports like this one (or in any other manner that attempts to capture information essential to the efficient development and delivery of frameworks/platforms and their implementations).


Maintenance and Monitoring Infrastructure for the Enterprise Software Systems Enterprise application software systems are about the software that organizations buy or develop to solve their business specific problems . Usually these systems start rather small. However, over the time they tend to grow more and more complex, increasing the number of subsystems involved and developing numerous interactions and synergy effects between them. For example, a company SmartForce might get started with a simple intranet web site and a database behind it. That's the most common scenario of getting started with an in-house enterprise system, that I've seen. That's the easiest one, too. As the company evolves, so does the software solving its problems. So over the time complex business rules and long-running workflows might be added to our system of SmartForce . These workflows might involve integrating with the external information systems or scheduling document processing for the nights. End-users might get interested in receiving custom notifications or scheduled tasks along the way. One of the most common implementations for this scenario (I'm generalizing here) is the introduction of automation server(s) to our web site deployment with a database backend. Then, the complexity of the enterprise software system might increase even further, as project stake-holders decide to introduce rich business functionality, integration with office tools or offline capabilities to the external client applications. This might lead us to Smart Client architecture implementations (whether they are done as desktop or browser-hosted applications). This kind of architectural change, generally requires some sort of the public API to be deployed. External integration scenarios also require such an API. The overall setup looks slightly more complex already. As if it were not enough, we might eventually get to the point of having multiple API versions to maintain and support (that's what usually happens to any API that is exposed to public). So the overall enterprise software architecture at this point might look like the one displayed on the picture below. Obviously, things might get a lot more complicated, but we are not showing them on the picture in order to keep it small and clear for the next part of the article. Yet, one can clearly imagine that the system might expand in size and complexity due to the following requirements: introducing multiple file storage servers integrated into the system; handling the traffic spikes by setting up a few API endpoints and web servers; integrating automated issue reporting and deployment updates into the system; distributing CPU-intensive tasks between multiple automation servers (or even delegating these tasks to the cloud); provide additional setups of the entire infrastructure for people from development, QA and marketing to play with. Such an enterprise software system definitely requires some efficient maintenance and health monitoring in order to function properly despite all the problems. By saying efficient I mean maintenance that does not require having a dedicated team in order detect problems before they happen (that's when fixing them is most cheap), simplifies and speeds up the resolution of issues encountered. Information is critical for this kind of maintenance (you have to know all about your patient in order to keep him healthy). We need a deep insight into the system, information on its behavior and its deviations under different conditions (as we all know, production reality may be quite different from the development despite all attempts to reproduce real-life stress and behavior profiles in testing scenarios). For example, we might need to know such basic things as: uptime and failures of subsystems; CPU and memory consumption on the machines hosting subsystems; utilization levels of the services and resources that our infrastructure provides; repository (database) statistics and performance network latency and response times of the external services; various exceptions captured and handled (or unhandled, which should be a rather exceptional scenario) within the code, failed sanity checks and warnings from the internal subsystems; low-level performance indicators in the code-base (i.e.: execution time, counts or loads of various methods, number of database dead-locks encountered and resolved by the reliability layer, custom counters). It is not enough just to capture these indicators. If we want to be efficient we could also: persist indicators in the operations database in order to see how system performance evolves over the time; present the data in form of real-time reports, making it more easy to spot problems or check the state; create and receive custom notifications when certain indicators or their aggregations go outside their threshold. For example, system going offline should result in an immediate notification to all involved parties, while slight performance degradation is just worth mentioning in the daily report. Obviously this entire setup has to be implemented in an efficient manner that allows to: change any indicator threshold in minutes; modify or add report in minutes; automatically version and deploy all changes. Note, that when providing time estimates, I'm taking into the consideration just the human labor required for the change. Automated deployment processes on their own might take anywhere from 5 minutes and up to 24 hours. This depends on the change propagation policies and processes employed by the specific development infrastructure. Let's take our previous picture of the enterprise system composition and add maintenance-related subsystems and interactions to the mix. One of the possible results might look like this: Basically, at this point we are talking about the infrastructure for performing system-wide maintenance and monitoring related tasks . This sample design works like this. Every primary subsystem exposes a standard monitoring API that could be used by the monitoring server to capture information. Internally every subsystem is instrumented with simple performance counters, capturing low-level execution (and failure) information and making it available via the monitoring interface API. Additionally, you may have your external client applications report exceptions directly to your issue management and monitoring systems. Maintenance monitor then aggregates all the information, applies any notification and warning rules available (some of these may result in notifications being sent out), runs custom reports and publishes them to the dashboard. That's not that hard to implement with existing open source tools. And having an ability to check how many seconds on the average does that complex repository call run against production db in real-time scenario - makes you wonder how could you live without such functionality before . It is like the ReSharper addiction . If you couple this kind of statistics with the domain events from the CQRS world, you would be able to answer tricky questions like: What's the average item retrieval speed from mySQL databases? How often do we encounter timeouts and deadlocks? How many seconds does it take to sync 100k products from SQL Azure database in the same data-center? What's the average upload speed to Lokad Forecasting API for datasets larger than 10k series, after that API upgrade in the last iteration? More on this topic... By the way, it may seem logical to extend the picture above by adding the maintenance server that would: aggregate information from the monitoring subjects across the infrastructure; match that information with the development and deployment reports in order to provide better OLAP analysis of the infrastructure performance across the: resource consumption growth : How do our subsystems handle spikes? Do we provide graceful degradation as resource consumption growth (or is it an exponential curve indicating some critical bottleneck that should be handled soon)? development process : How do new versions affect the overall performance? Do they improve or degrade it? Host and run SLA rules to watch for the resource consumption and provision/surrender resources depending on their consumption levels. But that's another story, that gets closer to the topic of cloud bursting (aka using cloud computing provider to handle spikes in the resource consumption). By the way, there is a post showing, how an enterprise overview report might look on an iPad dashboard . This document is an essential part of xLim 3 body of knowledge . What do you think about this post? : Do you have monitoring infrastructures in your enterprise application system? If yes , is your implementation different from the provided simple architecture? What would you like to add to it, if you were able to? If not , do you think, that it is worth having such an infrastructure?


Make Code Explicit and Stupid

Make Code Explicit and Stupid There is one thing that surprises me in software development. We, as developers, go at great lengths, explaining how event-centric architectures are so better than CRUD-based ones: you don't need to reverse engineer behaviors from state, since you express them explicitly (as events) and can project to any structural form. At Lokad we leverage this heavily for web UIs, that tend to be slightly more intuitive because of that. However, when things get to the code, we suddenly change our minds to the opposite and vastly favor implicit conventions and smart heuristics to discover and wire components in an application. Maybe this is because, we are so good with solving complex problems, that we see them in every single problem ? (this is only a part of the picture, component instantiation is in another class). We say, that such approach is smart, generic and reduces friction (when such approach is pushed to extreme, it is called Inversion of Control Container ). That it generally is a better alternative than this: Seriously, is it? An important remark : Both pieces of code were written by me. At the moment of writing of each one, I was extremely proud of the approach being used :)


Making Sense of Insane Software Requirements Writing is a thought you can analyze (c) Tim Ferris Have you ever found yourself looking at a pile of software requirements without any clue what to do with them? Some of them could make sense, while others - little sense or even contradict to each other. All in all this would leave this nagging feeling of "big ball of mud", that you have to sort out. This happens quite often to me. I used to hate such situations before. However these days such problems are simply tasks that usually have step-by-step solution. Let's see why these big piles of requirements (or specifications) even show up. They are probably caused by the fact that it's rare for a single person to have a clear vision of all requirements in his head. Usually these come from different people at different points in time: Different people specify different "I want to have this awesome feature" ideas for the same project. Somebody might provide one suggestion ("Make it green, it'll look awesome!"), forget about it, then provide completely contradicting solution a week later ("Make it red, it'll look awesome!"). This happens to busy people all the time. People might provide feature requests without full knowledge about how things are already implemented ("Please add this helicopter landing pad on top of our building", while the building is simply a bungalo on the beach) The most interesting part is that it's duty of software developer to accomplish all that in the best possible way, while turning down things that are too expensive or risky ("You know, adding underground parking garage to this existing building would probably make it collapse"). One approach to accomplish that is to simply start organizing these requirements, giving them some sort of structure. You would try to bring closely related concepts together and look at them in isolation from the others (hey, these 2 requirements try to give button "X" different colors). You might need to try different approaches to structure and represent requirements, before you find the one that fits your sutiation (put them on cards and shuffle around, add as lines to Excel, draw UI mockups etc). The big goal here is not to try build links and relations in your head - it'll take a lot of energy just to remember them. You can instead try to bring them some place out (even if the first attempt is a mess) and then improve on that structure on a step-by-step basis. At some point in the process you might even discover contradicting requirements (e.g.: making the button red and green at the same time), and this time you will be able to reason about them and explain "why" it is so. So next time you encounter a bunch of complex and contradicting requirements - just try putting them on a sheet of paper and shuffling around. Ultimately, in the world of complex business relations and behaviors, you can go further and try to code your domain model while exploring numerous specifications. Code can provide much stronger logical and reasoning support than simple writing. You'll be able to clarify your understanding by encapsulating some complexity inside value objects, hiding business behaviors inside aggregates and isolating UI-related details inside projected views and client-side interactions. This will give you rather structured high-level representation with the ability to dive into any single detail. Better than that, you could send it for somebody to review and improve. If coded domain model happens to capture everything perfectly - it can serve as a basis for production software that will do something useful with it. Otherwise, it's just a written model that you can improve on. Tim Ferriss once said "Writing is a thought that you can analyze". In the world of software design we can push it even further and say that "Code is a representation of the problem that you can analyze". If you happen to encounter complex business problem that needs solving, then may be capturing it in the code would help you to understand it and reason about it. Just give it a try, even if the code itself would never run in production.


Management Lifehacks

Management Lifehacks First of all, I would like to give my apologies for being sometimes slow on responses to emails and other sorts of communication. Last weeks were throwing new challenges, dealing with which takes a little bit of time. I'll catch up. In case you are interested, here are a few random ideas that have proven to be helpful in handling such situations. These ideas were reiterated so many times, that it's no longer possible to point to the original author . Don't try to be perfect in your decisions . You are human and small errors are inevitable. Besides, every single decision is worthless by itself (just like any "brilliant" idea). Only through the continuous and careful application of effort, something worthy can be achieved. Continuous chain of good decisions will beat perfect plan any day (simply because you can adapt and keep on going). Likewise, good execution can be more important than any great idea alone . Delegate . No matter how smart and talented we consider ourself to be, alone we are not able to handle and achieve as much as a team. Hence it is our duty to ensure that incoming challenges are balanced against the entire team. Our purpose is not to keep everybody under 100% load (e.g. by assigning tasks to people who are less efficient in handling them), but to ensure maximum efficiency of the entire unit. Focus . Keeping multiple projects in your head at once is likely to drive you insane, cause insomnia, burnout or do worse. So try breaking entire problem field of your division or company into separate contexts. They can be really diverse: starting from accounting, HR management and up to long-term tech RnD. More often than not, you will find that tackling a specific problem involves just a single context. So you can keep only one context in your head most of the time, switching between them, as your day moves forward. Arrange all tasks within this context in a queue, putting most painful ones upfront. Don't be afraid to drop tasks or change their priorities, if environment changes. If you are tired but don't have time to rest - switch contexts. If you are exhausted but don't have time to rest - stop complaining and find some time. Full personal burn-out is far more expensive than a little bit of rest. Adapt to changes on the battlefield. You would be surprised by the amount of resources wasted on projects even after they are doomed. Wars are not won by the sheer force, unless you are USA (and even States encountered some issues while bringing democracy to countries which didn't have serious fire-power but had population willing to take benefit of every hill, forest and trick in the book). By being willing to accept that our initial plan is imperfect and adapt, we can reduce risks, save resources and potentially leverage new opportunities. Think "Instagram" or "Apple". People are the only reason things happen around , don't forget about it. Universe was empty without purpose till humanity showed up bringing purpose along (with a noticeable degree of chaos). It might be tempting to forget about people and focus on a single technology, idea, concept or code. However, by doing that we risk missing the whole point and actually undermining our own efforts. For example, writing complex code without unit tests or documentation is often perceived as a sign of outstanding hacker or even a guru. Sometimes this even is worth it. However, more often than not, other people will have to maintain such code for years. If it is the case, then it is egoistic not to think about them. Code can have stronger positive impact if we put putting additional effort in making it helpful and friendly. It is much harder to do that, than delivering egoistic "easy to write and hard to read" code. Result will have greater long-term impact, though. Same principle applies to all the other things we do in our everyday lives. This is not even an altruism, but merely common sense and adequate long-term thinking. And the last one - Don't stop and keep pushing . PS: I don't follow these quotes every single day (not strong enough for that). However, continuously trying to do that - helps to keep moving. Update : Daniel have mentioned, this post can also be summarised with Viking laws


Many ways for an Emergent Design in a component Recently I mentioned 6 steps of an evolutionary design in software development . These steps describe iterative process aiming at continuous improvement. Such an improvement process can happen at two distinct levels: High-level view of the entire system involving components and their interactions; implementation details of a component. While high-level system evolution is covered pretty well in methodologies like domain-driven modelling (strategic design), implementation level can be more project-specific and hard to explain in uniform fashion. May be that's because there is no generic approach to describe evolution of components or services in a real-world system. Each element might need to evolve in a unique way to order to reach the best balance between complexity, performance and capabilities. For example, let's consider an evolution path that a single component can go through in a startup team focused on emergent design, rapid iterations and .NET stack: Team dynamics, past experience and current political situation might lead to a design approach, where each component starts as a simple console app and then evolves towards more complicated design in order to fulfil specific requirements. We try to keep things as simple as possible, but no simpler. If a component is kept simple and focused (which is a task of a strategic design), then at any point in time it could be rewritten from scratch. Evolution tree below is merely a visualisation of existing design approach inside a given team, serving as a way to make design options more explicit and allow better communication. Any change in team, business priorities or design methodologies could affect this evolution tree. Here is a bigger version of this image . Please note, that at any evolution along such design tree is a specific optimisation that comes at the cost of complexity. Sometimes it is better to delay paying that complexity cost and keep your options open .


Messages and Distributed Systems

Messages and Distributed Systems Messages are essential to building robust and distributed systems, so let’s talk a bit about them. Message is a named data structure, which can be sent from one distributed component to another. Components can be located on the same machine or on different sides of the Earth. The basic real-world example of message is an email. It has a sender, subject and one or more recipients. This email might take some time to reach the designation. Then it gets to the inbox, where it could spend even more time, before recipient finally gets time to read it and may be even reply. Messages, just like emails, might take some time to reach recipient (it could be really fast but it is not instantaneous), and they could spend some time in the message queues (analogue of inbox), before the receiving side finally manages to get to work on this message. The primary disadvantage of messages and emails is their asynchronous nature. When we send email or message, we expect the answer some time later, but we can never be sure that we will get it right away. Direct phone calls (or direct method calls) are much better here – once you get the person on the phone, you can talk to him in real time and get results back almost immediately. Despite all these disadvantages, messages could be better than calls for building distributed and scalable systems. With phone calls and method calls you: Can get response immediately, once your call is picked up. Must be calling, while the other side is available (greater distance you have, harder it is to negotiate the call). More stressed the other side is, more time it will take before your call will be picked up. And this does not guarantee, that you will get the answer (when the other side is really stressed you are likely to get: we are busy right now, call next millennia). With messages you: Can send a message and then get back to your work immediately. Must organize your work in such a way, that you will not just sit idle waiting for the response. Can send a message any time, the other side will receive and respond, as soon as it gets to the job. More stressed the other side is, more time it takes to receive the answer. No matter what the level of stress is, the other side will still be processing messages at its own pace without any real stress. Since we are mostly interested in building distributed and scalable systems (which can handle stress and delays) messages are a better fit for us, than the direct method calls in the majority of the cases. They allow decoupling systems and evenly distributing the load. Besides, it is easy to do with messages such things like: replaying failing messages, saving them for audit purposes, redirecting or balancing between multiple recipients. Note, that there are cases, where direct calls work better than messaging. For example, querying in-memory cache does not make sense with messaging. Cache is fast and you want to have the response immediately. For an overview of how messaging works together with a scalable distributed system, check out Decide-Act-Report model.


Microsoft: Lokad is Windows Azure Platform Partner of the Year I just wanted to share some exciting news with you. Microsoft has just issued a press release for its upcoming WPC 2010 . Lokad is referenced as the winner in Windows Azure Platform Partner of the Year category. “Lokad’s Lokad.Cloud solution earned them the Microsoft Windows Azure Platform Partner of the Year Award,” said Allison Watson, corporate vice president, Worldwide Partner Group, Microsoft Corp. “Lokad.Cloud facilitates the development of enterprise apps on Windows Azure to bring substantial benefits to the growing community of ISVs who target Windows Azure. Lokad’s early adoption of these technologies and impressive solution has made them a leader in the space.” I'd like to congratulate Lokad Team working together with Microsoft and Windows Azure teams to achieve this significant milestone. Stay tuned for more exciting news from Lokad.Cloud and Lokad.CQRS Open Source projects, as we continue to share our passion and experience with the community (while still focusing on delivering outstanding analytics to your business).


Migrating Legacy Systems to Event Sourcing These days I'm working on migrating really legacy system towards the simplified CQRS/DDD design with event sourcing for the cloud. As part of the migration process, I'm reverse engineering legacy SQL database into a stream of events. These events are not precise representation of what has happened in the past (this exact information is irreversibly lost, as in almost any data-driven system), but rather a pretty good estimate that could be used to prepopulate the new version. Essentially, reverse engineering events is about writing a throw-away utility that will scan database tables (MS Access files or punch-cards) and spit out events that could be used to reproduce that state. For instance, consider this customer record in DB table: Customer { Name : "GoDaddy", Id : SomeGuid, Created : 2008-13-12, Status : Deleted, Phone : "111-22-22", Reason : "Supporting SOPA was poor PR move" } This record could be reversed into the following events CustomerCreated!( Id: SomeGuid, Name: "GoDaddy", Created: 2008-13-12 ) CustomerPhoneSpecifid!( Id: SomeGuid, Phone: "111-22-22" ) CustomerDeleted!( Id: SomeGuid, Reason: "Supporting SOPA was poor PR move", Deleted: 2011-12-24 ) Note, that we actually had to improvise while coming up with this event stream: date of deletion was not stored in the original database (we were losing this information). So we are just substituting some predefined date here (i.e. date of upgrade to CQRS/DDD+ES). When you have a system with a few years of history, quite a few events are generated. The system that I'm currently migrating has data that dates back to the early dates of Lokad , hence 300-400 thousand events is something expected. As part of development process, these events are run through the aggregate state objects and also through the projections . The goal here is to pass all possible sanity checks and get read models that match exactly to the UI currently visible in the old system. If new system looks and behaves exactly like the old one (even if the guts are completely simplified), then we are moving in the right direction. Obviously, during this process, a lot of problems show up, especially with logically inconsistent or corrupt data (i.e. accounting inconsistencies caused by race conditions and dead locks in the legacy database). These things are generally to be resolved manually - there is no magical silver bullet.


Migrating to EventStore in Windows Azure We have a number of systems at Lokad, which use various versions of Azure EventStore from Lokad.CQRS. Our in-house store is nice, but has performance limitations and some administrative burden. Currently I'm working on gradual migration of our systems towards centralised EventStore on Windows Azure. Some community members expressed an interest in the process, so I'm sharing my current thoughts . Administration Here are a few considerations and assumptions that affect our requirements for Event Store server setup: Uptime with many nines is not critical for at the moment. Built-in Azure replication is more than enough for us at the moment. We'd prefer to use single-node deployment of Event Store at the time being (with built-in Azure replication). We're hosting event store in Widows Azure, exposing it to the entire world. There are two main deployment options that fit: Option 1: Install Event Store as Windows Azure Worker Role. This setup requires a little bit of coding but is more native to Azure. However, single-node configuration would require use of CloudDrive (XDrive) to mount Azure Blob storage as Windows Disk visible to Event Store. I've heard people were having problems with this setup in production (data corruption). If we were to switch to clustered version of Event Store, then worker role setup (with local transient disks) would be the preferred approach, without any doubt. Much better performance, too. Option 2: Install Event Store as windows service on Windows Azure Virtual Machine . This option uses host-managed disks to mount the same Azure Blob storage as Windows Disks. These are the same disks that are used to host operating system itself, so they are more reliable. Option 2 is our current path at Lokad right now. Additional setup tweaks include: Host Event Store db on a separate VHD drive , making sure that it has read-caching turned on. Logging is better to go to a separate transient disk (so that NLog would not compete for IO with event storage). Write-caching is better to be turned off. Expose port 1113 to internet as public endpoint . This port hosts TCP protocol of Event Store, which has just got per-stream security available. keep port 2113 as private . This port serves Web UI and projections via REST API. Migrating systems Theoretically, switching systems to new event storage would be rather simple: Replace current event store implementation in code with adapter that uses .NET client to Greg's Event Store; Migrate all data to a new server. In theory this should work. In practice, blindly jumping to a new setup would be reckless. I expect a number of edge cases that might surface simply from the fact that such Windows Azure setup is not native for EventStore : Event Store unavailability due to VM restarts by Azure. This is expected but has to be dealt with. Transient network issues while communicating with EventStore server via .NET client. Changes in network topology by Windows Azure (e.g. EventStore server is no longer located on the same IP). Technically, EventStore deals with many problems via graceful degradation (as opposed to simply giving up). E.g. if it can't connect to EventStore, then it will retry connecting internally. However, in practice, components using EventStore must be designed with this degradation kept in mind, they should degrade gracefully as well (as opposed to simply giving up). So the migration process would be: Keep existing production systems on Lokad's Event Store, which is battle-tested on Azure. Setup replication to Greg's Event Store and start moving non-critical projections and components there. Let them run there for a while. If the setup holds - gradually move existing productions systems to the new Event Store. Long-term Strategy If things hold well on Azure, I expect to have a number of design simplifications in Lokad systems. Some examples are: Use event store instead of Lokad.CQRS messaging adapter (Files / Azure queues) and avoid Azure message throttling. Switch to catch-up projections, gaining ability to store projected views on any storage (not just azure-hosted blobs). This would massively improve performance and simplify client applications. Interestingly enough the same design improvements allow to make system less coupled to constrains of Windows Azure. In my experience, this is a really good thing. Feedback? I know that some of my readers are already using EventStore in production on Windows Azure. Would you kindly share your setup and problems encountered? Update 1: using RAID for better speed This is an update by Ryan A. You might want to take a look at creating a RAID 0 of 4 disks for Medium (8 disks for Large, 16 for XL) within the VM. This will allow you to utilize all of the IOs allocated to your VM. There is a MAX of 500 IOPS per disk. Virtual Machine and Cloud Service Sizes for Windows Azure In some initial tests, I was able to get WRFL 10 1000000 --> 5576/s running the test client from an external machine in the same datacenter. Albeit, the host was a Extra Large (8 cores, 14 GB memory) with 16 5GB disks in RAID 0 Update 2: Doubtful performance of Cloud Drive This is an update by Alexey Zimarev CloudDrive is still based on SDK 1.8 and according to Msft people " the future of CloudDrive is now being discussed ". I was busy deploying RavenDb in a worker role and it works with the StorageClient assembly, which is now seems to be a wrapper around the new Storage 2.0 client. However we can never be sure what Msft will decide to do with CloudDrive. Also there were some reports about performance of the CloudDrive versus VHD although it must be the same, but still. One of the issues with data stored on a CloudDrive in worker roles is a scaling up problem when you increase the number of instances for your worker role and it is unable to get write access to the database file since it is on exclusive lock. In this fashion VMs are much more controllable since you instantiate each machine separately and have separate disks and you setup replication between the VMs in a way you want it.


Mindsets in development - Enterprise vs Startup A while ago we were recording a distributed podcast with Udi Dahan (not yet published, I believe). In this podcast I quite enthusiastically opposed everything Udi said. The entire beginning of the podcast (one or two dozens of minutes) had to be cut out and redone because of that. I personally believe in a good argument (and enjoy one), but we didn't want to make the podcast look non-constructive or scare all the other potential guests away :) This experience got me thinking for a while, making more aware of the arguments and collisions that happen in the software and business worlds these days; especially in parts of it, where new technologies and approaches meet something that has been working for decades. Opening dev center in Ufa provided some additional food for thoughts as well. As it turns out, I was not conflicting with Udi's approaches directly (this also applies to my other ongoing critics of Microsoft or any other enterprise-level entity or solution). I still extremely value their experience and ability to solve problems I would've given up on. Root of the conflict is a bit deeper - in the business environment and the way the problems are structured in it . Let us consider two extremes (exaggerating the slightly just for the sake of making the point). Corporate enterprise is on one of the extremes. It tends to have an extremely formal and regulated environment (it is the only way to keep big system from falling apart) with large budgets, slow processes and established teams. Such an organization usually wins its day if it keeps status quo. All software projects will be affected by such environment. Developers are generally restricted to time-proven technologies, formal architectures and long development cycles. That is exactly the environment, where complex N-Tier designs, GUI tools and multi-page specifications come from, backed up by established stacks like MS SQL + MSMQ + BizTalk/NSB and armies of paid consultants to help whenever there is a problem. Product managers in such conditions are restricted to time-proven technologies and approaches backed up by the books and solid authorities (nobody is fired for using Microsoft). Given the armies of stake holders and complex political games involved around any single project (they would make any Bene Gesserit jealous), it is safer just to do everything in a slow and controlled manner, making the efficient use of the resources available (and there are a lot of these). Small start-up environment is on the other extreme. Such companies do not have a long history, stability of established routines and big budgets; but they possess founders who were crazy enough to favor unpredictability of a new business to a soothing comfort of established company. Start-ups are usually small companies below the profitability state, so they must fight every day just to have resources to move on. Staying in the office at evenings or pulling weekends is not unusual either, just like nervous break-downs. Lack of resources and time are usually compensated by enthusiasm and a lot of personal collaboration. Such an environment has its own drastic effects on the IT projects that happen within. Business people tend to care less about all things that are "time-proven" and "as explained on the training by X". Keeping the company afloat and moving forward is what matters more. Hence IT departments (usually consisting of a single dev or even a founder himself) are allowed and encouraged to do whatever it takes to solve the problems at hand as efficiently as possible. Technology and methodology do not matter as long as they work. It is even considered to be acceptable to sign a contract with devil or dump the entire stack into the cloud, if it would help to cut down cash burn rate or achieve a milestone. We did that, by the way. Ok, this was just a brief overview of the differences in these extremes. There are much more complexities and details underneath the surface. Books are written about them. The point that I'm trying to make is: these environments have completely different rules of the play . Things that get projects delivered in one of these extremes, are not guaranteed to work in the other . For example, start-up might give up on using some complex clustering or replication, because it is too complex and expensive for the job at hand, while enterprise might take extreme care before rushing into all these cloud things or hand-made linear scalability. Same applies to all other things that are associated with a successful delivery of products, starting from management of development projects and up to selecting deployment and evolution strategies. So when we are talking about any complex buzz-word and practical approaches for it (CQRS and DDD being an example), it is recommended to keep in mind this point. It will help to both avoid the confusion while listening and help to deliver the message while explaining. If some explanation or approach does not highlight these specifics, then we can try to clarify the situation in our own minds by answering a few simple questions: what environment does the speaker come from, what are his financial interests, affiliations and expertise? what context does the methodology or approach target? What constraints does it have in mind (resources, budgets, time, risks, skills, regulations etc)? if some specific technologies are recommended, for which context were they developed? Which companies do have interest in them? What are the lock-down risks and costs? Just to make it clear, when I am personally talking about development efficiency and architectural approaches (for both cloud and on-premise environments), I'm doing that from the perspective of "start-up mindset" where: budgets and time are constrained; really constrained; business does not care about the specific tech or regulate development, as long as the job is done; it is not that hard to find and hire good developers (they tend to find you); motivation and reach inter-personal collaboration tend to replace formal planning and various management practices (as written in the books); there are not enough resources to allocate large teams to a project, one or two devs is all you can get (sometimes even less than one); development can affect the business in order to structure the problem in a way that it will be solved more efficiently (by breaking into smaller steps or realigning business priorities); there are no real requirements up-front, ever changing business environment dictates these, as the company moves forward; it is more efficient to "reinvent the wheel" internally, rather than ask for the paid support. I personally believe such an environment is much more efficient and capable of delivering interesting and exciting solutions, if played out properly. It has the advantage of using the full potential of recent massively enabling technologies. High pressure, scarce resources and lack of formal regulation are among the obvious downsides (if you look at them this way). A quick example might be helpful here to illustrate potential differences between the environments and mindsets. Corporate developer might consider data replication, backups and synchronization to be something really complex and extremely expensive (bringing thoughts here of SQL licenses, clusters and various sync frameworks). Another developer, that is used to event-centric architectures, would consider implementing data replication and streaming to be a boring task, that requires just a few lines of code and picking the properly certified cloud storage provider. Both viewpoints are valid, because they are based on the constraints and assumptions present in their respective environments. More than that, each of the developers has a decent chance of failing horribly in the other environment (unless they are flexible enough). My personal "environmental" beliefs, obviously, affect the way I structure problems or choose from various options and technology stacks, when given the luxury of choice. The same applies to all people that write, blog and preach these days. Please, keep contextual and mindset differences while listening and learning from anybody.


Move Forward by Discarding Complex Tech

Move Forward by Discarding Complex Tech Good things are either well-forgotten past or a complete rip-off from the nature. It seems that at Lokad we are going all the way back in time ourselves as well. Over the course of the last few days we had really interesting times at Ufa office, while migrating entire event replication infrastructure to a new model . If you wish, you can call this infrastructure as bounded context of digital nervous system that is represented by green arrows in our context maps . This is a really interesting place for us, since it "touches" multiple other bounded contexts and actually crosses 2 clouds and 1 additional datacenter deployment-wise. Change shocks are mesmerizing to observe. Now, instead of a mixture of Azure queue delivery and ZeroMQ streaming, our applications just push large event streams over hand-made HTTP replication protocol . This effectively uses HttpListener and WebRequests, which are: rather performant; dead-simple and well understood; have minimal friction of introducing replication to new projects (ZeroMQ is pretty invasive here, if you go for Azure); can be debugged with a lot of HTTP-based tools. The design is rather simple, practical and works well for streams of half a million of events (albeit performance could be improved a lot). This was really important, since we have now a number of bounded contexts to integrate together and the volume of event streams just keeps on growing. It is curious, how our movement forward towards better and simpler designs happens concurrently with stepping back from complex technologies to much simpler ones . In other words, we gain by discarding things . Another example of such behavior is related to our recent decision to discard ProtoBuf as the storage format for large data objects , while replacing ProtoBuf+Gzip with TSV+Gzip . This applies specifically to bounded contexts that deal with big data . Reasons for that being: ProtoBuf by default loads all objects directly into the memory at once (imagine a dataset of 1 GB), while the default behavior of text files is streaming; For numerical data TSV+Gzip compresses better than ProtoBuf+Gzip, since archivers were initially designed and optimized specifically for handling text data; You can read and parse TSV dataset with tools on any platform, including scripts and Excel. While with protobuf, some intermediate dancing would be required. So, if I can reduce a number of technologies in a given bounded context, while making it more practical and performant, then that's a clear choice. As you can see, in certain scenarios , we are stepping back from cool and smart tech towards something more practical and simple. This "stepping back" actually enables us to solve certain problems that exist in this specific scenario. Surprisingly enough, this brings us closer to the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface. I certainly didn't expect to see this happening before, not even theoretically. However, in practice, there is a big difference between theory and practice . Caveats Please, keep in mind, that: we are aware of ProtoBuf capability to read items sequentially. we still will be using ProtoBuf for serializing messages, including events that are used for our event sourcing scenarios (leveraging for .NET development a wonderful library by Marc Gravell ) these examples just serve the purpose of illustrating possibility of cases, where you can move forward by discarding a technology. Specific decisions might not be applicable directly to your case.


My Current Development Environment

My Current Development Environment A few years ago I wrote a blog post about the development environment for building xLim 2 applications. Ben recently uncovered it and asked a question if anything has changed since then. Yes, it did. 2 years are a huge time interval, especially when you try to learn as fast as you can. So let's talk about my current software environment around the PC workstation which is mainly used for the development (there also is a notebook for develop-while-you-travel and an iPad for learning and planning, but that's a different story which I'll tell later, if you're interested). Side story - what is this xLim? For those of you who are still confused with the whole xLim concept , here's how it showed up. 3.5 years ago, while working remotely as a Developer and Team Lead in an American company I got fed up with the absolute lack of progress in the project (it hit hard the complexity barrier) and decided that it should be possible to do better with much fewer resources, than 5-15 developers. Much fewer meant - just one developer or slightly more. When you have idea, passion and a bravery to follow it (which is sometimes referred as "childish stupidity" by more conservative part of the society), then two things are certain to follow: cool name (xLim standing for eXtensible Lightweight Information Management system); a bunch of hard lessons and disappointments. So they did show up on the horizon rather soon and were put to a good use. Practical experience was turned into the first "lessons learned" report, which actually opened me the way to the first remote freelance project and even more experience. You can still download the report and associated screenshots from the xLim page (just scroll to the very bottom). Lessons were continuously learned (or learnt in GB English) since then and recorded in form of the reports, articles and teaching materials for the Russian teams I happened to manage and train from time to time. Most part of them was republished as articles within the xLim series. Every time there was a decent paradigm shift in the perception of the problem at hand (and the simplest possible solutions that work in practice), xLim version number was incremented. Every next version incorporated the best of the experience from the previous steps, but reshuffled the whole vision and opened room for new paradigms and further movement forward. Current version is 4 - "xLim 4: CQRS in the Cloud" and I still maintain that you can build and run incredibly rich and beneficial applications without spending a lot of resources or time)) OK, let's get back to the point - how does my development environment look right now, in October 2010, while it matches xLim 4. IDE, Required Tools and Libraries Visual Studio 2010 (mine is Ultimate, but Professional is enough) Microsoft .NET Framework 4.0 (TPL and Rx for .NET are a must) ReSharper 5.1 (some things never change, just get better) NUnit for Unit Testing Mercurial (TortoiseHg) or Git (TortoiseGit) for version control (as long as this is not CVS, SVN, TFS or SourceSafe). Autofac IoC Container Protobuf-net for fast and efficient serialization, implementing custom storage formats. ASP.NET MVC with MvcContrib and a dash of jQuery (completely replaces DXperience in web). SQLite for file-based databases with incredible performance, SQL syntax and schema-tolerance (i.e.: you can put 4MB Blob into INT field). Reflector for .NET Putty SSH client . Gmail. RabbitMQ - preferred message server implementing AMQP (in cases when I don't have to use Azure Queues) As you can see SQLite is the only persistence-focused item in this list. There is no SQL Express, mySQL, Oracle or, NHibernate+FluentNHibernate+LINQ with all libraries it uses. I'm still using them on a daily basis because of the existing projects (or databases we need to integrate with). Plus NHibernate is featured in Lokad.CQRS (as a single library) because that's what people are mostly used to. However, in my personal mental model (and while evolving projects) ORMs are being rotated out along with the relational databases - with a great success, more flexibility and simplicity. Check out the article on CQRS, Reliability and Redundancy . It may not seem particularly simple in the theory, yet when it gets to the code (backed up by the reasoning), things do get more simple and efficient. In addition to a few things being removed or replaced, there is a new and critically important piece: cloud computing . Virtual Machines from Rackspace Cloud are used frequently. For example, when I need test against some version of DB engine, which I don't want to install on my development machine. That's 1-3 minutes to provision Ubuntu and SSH into it and a few more to apt-get and configure mySQL instance. That's a bit faster than Windows Azure (which takes just half an hour to update Worker Role) which I also happen to use at Lokad. Obviously, Cloud Computing, as perceived from the point of efficient development, is not just a quick way to get VMs to play with and pay pennies for that. Cloud storage and on-demand computing are actually affecting the whole concept of xLim 4 and allow an individual to build systems that could compete with corporations in scalability and flexibility (and still fit within a limited budget). The concept of Almost-infinitely scalable systems (as applied to CQRS) would be much more complex to achieve without cloud computing. Just check out the CQRS Roadmap for the better high-level picture. Recommended Tools Launchy (keystroke launcher that I also configured to open ticket shortcuts in various trackers). FSCapture - for making screenshots. Fiddler2 - the best tool to debug these AJAX Calls or REST APIs (esp. when you need to prove to Azure team that their APIs do not behave according to the documentation) JetBrains dotTrace profiler KeePass Password Safe - for keeping my passwords, logins and sensitive configuration settings. JungleDisk (Desktop Edition) - for backing up my files into the cloud (in encrypted way) and transparently syncing them between the machines. FoxitReader - because Adobe PDF reader is an overkill. Evernote - to gather small notes, articles and reference materials. 7-Zip - the Archive Manager Windows 7 64 bit - the best operating system that works with Visual Studio 2010. GhostDoc - VisualStudio Add-in for generating XML documentation stubs. Please note, that a few of these items create additional value outside my development machine. For example, Evernote and Jungle Disk frictionlessly sync important information between my development PC, travel notebook and an iPad. Development Environment Collaboration Strict development environment requirements from xLim 2 are getting obsolete. With the cloud and SaaS services available you can rent various wikis, issue trackers and version control repositories at rather competitive rates or even for free (for smaller projects). For everything else there is a cloud. Open source projects benefit significantly from offers like: Google Projects and Google Sites . At Lokad we are currently using CodebaseHQ (Mercurial hosting + issue tracker), Google Sites and self-hosted CruiseControl.NET (yes, this one is antique but it still does the job and even handles a bit of misuse by Lokad R&D) In short, you can get yourself rather flexible development environment for the distributed teams either for free or at the cost of a few dollars per month. Just add Skype to it. Reference Materials CQRS section aggregates various references specific to the architectural approaches of the current xLim version (and all the solutions I'm building). As for the pre-architecture requirements of xLim 4 (development practices, recommended reading, design guidelines, low-level building blocks and producing decent code etc) - that will be a rather long story to tell within this post (or probably even series). For references I just recommend to look through previous versions of xLim, discarding everything that does not fit into the CQRS approaches or does not scale in the cloud. NB: Composite application approach still holds, but all these Enterprise Application blogs or universal frameworks just do not work out. You can rapidly develop various Client applications on top of CQRS without any need for the complexity. Hardware Actual hardware is nothing special these days and hasn't been upgraded for a year. Yet it works for me (especially when compared with more painful configs). Let's start with the most important things: important! Microsoft Natural Ergonomic Keyboard 4000 (after more than 10 years of coding my wrists are quite touchy about keyboards and eventually start burning on any other keyboard) important! Microsoft Natural Wireless Laser Mouse 6000 (same reasons as the keyboard, although I would try to get one of Logitech's Trackballs these days). important! Ergonomic chair designed for workers-of-the-keyboard (I picked one at the local store) important! Two decent wide-screen LCD monitors. And they are better to be connected to DVI/HDMI ports (ATI Radeon HD 4800 in my case), unless you want your eyes to be hurting by the end of the day. Less important stuff: Intel Core 2 Quad CPU Q6700 @ 2.67 (would use i7 these days) 8 GB of RAM RAID 0 HD Setup for faster disk IO over the OS and project files (would use SSD these days) Decent power box to supply tower with all this hardware. Note, that I am not doing any over-clocking to "get the bang out of the buck". It is just not worth it. Tools For the Project Manager Microsoft Office 2010 (Word, Excel, Powerpoint) Microsoft Visio 2010 - for creating all these nice diagrams you see in my blog. As you can see we've lost MindManager, MS Project, NDepend and Source Monitor from our list. Complex management software is no longer needed (at least personally for me) due to work in environment that rapidly changes and sometimes requires new feature to be designed, implemented, deployed and stabilized within 1-5 days. At such pace well-thought projects become obsolete really fast (the next few days) and become just a waste of time. Instead, Amazon.com development style starts working better and better. Getting Things Done methodology, priority lists and OmniFocus (which unfortunately runs only on iPad or Mac) help to stay sane and keep all things under control even when there are multiple projects running concurrently with multiple tasks that have volatile priorities. Word, Visio and Moleskine help to do the actual planning, brainstorming and architecture design. Summary Now, locking back at changes within these couple of years I can clearly see reduced dependency on the libraries, frameworks or specific tools (aside from the Visual Studio + ReSharper). There's a clear tendency of getting rid of the relational databases and layers of persistence and using more capabilities offered by the cloud. Projects get smarter, more reliable and efficient not because of the building blocks used, but because of a lot of theory being put into them and simplicity enforced at all levels. Actually I believe simplicity to be the hardest target to arrive at. It easy to develop a complex solution. It's much harder to develop solution that achieves the same but in a more simple and straightforward way. This at least requires some experience and I feel like standing at the beginning of the road towards this experience. Despite these changes I still maintain that a single developer with limited resources can build and manage rather interesting solutions; especially if he uses these resources efficiently, keeps everything simple and thinks a lot before doing anything (obviously now the term "interesting solution" means a bit more, than 3 years ago).


New Case Study on CQRSGuide.com

New Case Study on CQRSGuide.com I've published final version of a new case study about one of latest Lokad's projects. This project had to go live on February 29th. This was caused by an unfortunate sequence of events with our cloud provider and immediate need to fail over to a different provider in order to keep some of the services running. Case study provides rough overview of the most important CQRS/DDD concepts that made their way into this project. Some of them are already presented in Lokad.CQRS Sample Project. Some other are yet to be ported into the open source. CQRSGuide.com Case Study: Lokad Hub These are kinds of things that enable us to solve everyday problems, help customers and try to push the state of the art. BTW, if you happen to be a fan of Lokad forecasting technologies - stay tuned. There is a really big thing being prepared by our awesome analytics team. Announcement will come soon.


One Example of Using Message-Driven Design at Lokad Lokad Salescast is an inventory optimisation platform for retail , capable of dealing with big datasets. It gets inventory and sales information and does some number crunching . Produced reports tell when you need to reorder your products (and how much) in order to serve forecasted demand and avoid overstocking. One of the objectives of Salescast is to make it available and affordable for small customers. Hence we introduced "Express Plan", which is free for small customers, but comes without any support. Making software free is easy. Making software usable without support is much harder. So Lokad developers had to create complicated heuristics to help customers deal with the problems. TSV parsing is one of problematic regions. Even though the major scenario for big data transfer at Lokad is "upload TSV-formatted text files to FTP", there are multiple things that can go wrong with this simple setup. No matter how precise is tech documentation, people can always miss seemingly unimportant things that are critical for computers. Here are some examples: text encoding of files; culture-specific format of dates; culture-specific format of numbers; optional columns in invalid format; required columns missing; missing files; non-standard separators. Yet, we are trying to provide the best experience out-of-the-box even with improperly formatted data. This would require doing a lot of smart TSV analysis in code. Here's how an output of one analysis process would look like (latest log entries at the top): Message-driven design patterns help to develop and maintain such logic . Public contract of it in the code might look like a simple function (with complicated heuristic inside): static IMessage[] AnalyseInput(SomeInput input) { .. } Here messages are strongly-typed classes that explain return results of that function (unlike event sourcing, they are not used for persistence). For example: public class UsedNonstandardExtension : ITsvFolderScanMessage { public readonly string Extension; public UsedNonstandardExtension(string extension) { Extension = extension; } public virtual AdapterTweet ToHumanReadableTweet() { return new AdapterTweet { Severity = AdapterTweetSeverity.Hint, Tweet = String.Format("Salescast found Lokad TSV files using" + " non-standard extension {0}.", Extension), }; } } Function would return one or more event messages . Various input scenarios might be unit-tested using given-when-expect approach , where we express test case as: given certain inputs ; when we invoke function; expect certain outcomes and assert them (e.g. verify that we get expected messages). Or in code: public sealed class given_compressed_files_in_txt_format : tsv_folder_analysis_fixture { public given_compressed_files_in_txt_format() { // setup all expectations in constructor, using helper methods // from the base class given_files( "Lokad_Items.txt.gzip", "Lokad_Orders.TXT.gzip" ); } [Test] public void expect_detection_with_extension_warning_and_compression_hint() { // assert expectations, using helper methods from the base class expect( new TsvFolderScanMessages.UsedNonstandardExtension("TXT"), new TsvFolderScanMessages.CompressedFilesDetected(), new TsvFolderScanMessages.StorageDetectionSucceeded( TsvInputFile.Item("Lokad_Items.txt.gzip").WithGzip(), TsvInputFile.Order("Lokad_Orders.TXT.gzip").WithGzip() )); } } This is an example of a single test scenario. There could be many others for a single function, reflecting complexity of heuristics in it: Each of these test scenarios shares same "when" method and helpers to setup "given" and "expect", so they are pushed to the base fixture class, which can be as simple as: public abstract class tsv_folder_analysis_fixture { readonly List<string> _folder = new List<string>(); ITsvFolderScanMessage[] _messages = new ITsvFolderScanMessage[0]; protected void given_files(params string[] files) { _folder.AddRange(files); } [TestFixtureSetUp] public void when_run_analysis() { // this is our "When" method. It will be executed once per scenario. _messages = TsvFolderScan.RunTestable(_folder); } static string TweetToString(ITsvFolderScanMessage message) { var tweet = message.ToHumanReadableTweet(); var builder = new StringBuilder(); builder.AppendFormat("{0} {1}", tweet.Tweet, tweet.Severity); if (!string.IsNullOrEmpty(tweet.OptionalDetails)) { builder.AppendLine().Append(tweet.OptionalDetails); } return builder.ToString(); } protected void expect(params ITsvFolderScanMessage[] msg) { CollectionAssert.AreEquivalent(msg .ToArray(TweetToString),_messages.ToArray(TweetToString)); } } If you look closely, then you'll find a lot of resemblance with specification testing for event sourcing . This is intentional. We already know that such tests based on event messages are non-fragile as long as events are designed properly. This additional design effort pays off itself really quickly when we deal with complicated heuristics. It makes development process incremental and iterative, without fear of breaking any existing logic. Step by step, one can walk around the world. In essence, we go through all the hoops of expressing behaviours via messages just to: express diverse outcomes of a single function; provide simple functional contract for this function; make this function easily testable in isolation ; ensure that tests are easily maintainable and atomic . Downstream code (code which will use components like this one) might need to transform a bunch of event messages into a some value object before further use, but that is a rather straight-forward operation. Interested to dive deeper into Lokad development approaches? We are looking for developers in Paris and Ufa. You can also learn some things by subscribing to BeingTheWorst podcast which explains development ways of Lokad.


Our Ultimate Technological Challenge

Our Ultimate Technological Challenge // beware: this article is really important for me, but might bore you to death. To make it worse, I will mention Justin Bieber down there. Our world seems to be in a complete mess these days . A few dozen of grown-up people chasing a ball on a grass field get much more attention than news from war zones on East. Students know more about personal life of Rihanna than about genocide in Rwanda in 1994. And amount of money wasted by Facebook on Instagram is equivalent to total funding of SpaceX company since it's creation till May 2012 (world's first privately held company to send a cargo to the International Space Station). Yes, (mass murder < pop singer) && (cute photos == space flight) . Given ever accelerating speed of changes, such disparities and problems are only to going to get stronger . Personally I don't give a damn about musical taste of rich countries, but when kids are dying from starvation and malaria in Africa - this is just not right. This is not the world I was promised back in soviet school and definitely not the world I would want to pass to my own children. Version of Matt Harding is better. When something does not go as it supposed, you give it your best shot to straighten things out. So, what can we change? Most importantly, how it can be changed? Let's start with the simple assumption. In order to make things right, we need to change how entire nations think and act : both rich nations with abundance of resources and poor ones. That's the very thing that Gustave Le Bon called "soul of the nation". It was supposed to be extremely hard to alter, nearly impossible. Fortunately, things improved a little bit since then. As it turns out, it can take as little as a clever excuse and a staged act to change one nation . For example, this was vividly demonstrated on Easter Sunday Parade of 1929 in New York. Edward Louis Bernays was doing a mission for tabacco companies which were upset by loosing huge potential profits to a superstition that women should not smoke cigarettes. Edward (who later became known as "father of public relations") simply combined ideas of Wilfred Trotter and Gustave Le Bon (with a few other sources of inspiration, including his famous uncle) and simply hired a few girls to start smoking torches of freedom as a protest against sex taboos. Of course, he also tipped local reporters about upcoming feminist protest and hired photographers to make sure, that good and plausible pictures are available. Faster than you know it, the entire USA fell for this act and started discussing. Needless to say, that tobacco companies were happy with the sales in the next years. "Tools" used in this change only improved since then: we've got internet, social networks and massive spread of smartphones and plain cellphones. These are the things that actually made it possible for the recent revolts in North Africa to take place (if you know Pieter Hintjens, the guy behind ZeroMQ project, check out his guide to Digital revolution ), and they keep improving over the exponential curve. If you should know one thing about exponential curves - that's the kind of thing that helped Chernobyl Disaster to happen (in addition to flaw in the design of control rods). So these days civilization has an enormous potential available at finger-tips of anybody who has access to the internet: starting from limitless power of cloud computing and up to world-wide penetration of mass media, cell phones (and their smarter brothers). Couple this with herd instincts baked into our DNA by thousands of years of evolution, and you have outstanding things taking place: the good, the bad and Justin Bieber. Combining digital and social resources can have an outstanding effect even in constrained situation. A couple of years ago, I played a small role in helping a few kids to start a social project to support orphans. It wasn't that easy, but MyDreamCity went worldwide and is still running. At the heart of the process was simple enthusiasm supported by some basic IT infrastructure. Take this to a higher scale, and you can probably have a shot in improving a nation. I'm thinking about two directions that can be exploited: education and efficient use of resources . The former - to make a positive change at the most vulnerable point of any society - kids; the latter - to actually provide some real foundation for these changes to stand upon (money is the blood of our society, as we've learned well by living through the crumbling Soviet Union). Fortunately, I'm working at a place, where we study optimization of resource consumption at all levels - from high-level organizations and going down to the level of individual households. Plus, there is some limited experience at teaching at my own university, coupled with a little bit of community work in the development field. This is not enough to make a real change right now, but enough to give unsettling feeling that something can really be done to improve the situation. So, imagine for a second that you have the ability to reach every human being in this world (including these kids in Somali and Zimbabwe), provide inspiration, support, answers to any common question and access to all knowledge of humanity. Don't worry about computing resources and don't worry that much about money. Technology can easily provide all that, especially today and tomorrow. The question and ultimate technological challenge is: What could be done with all this in order to improve our world within the next 10-20 years?


Parameter vs. Argument

Parameter vs. Argument Parameter is variable in the declaration of function. Argument is the actual value of this variable that gets passed to function.