Retrospective of 2015 Q1

Retrospective of 2015 Q1 Introduction First quarter of 2015 is behind. We managed to achieve some impressive results at SkuVault. This will allow us to offer new features and a better product to customers in the second quarter of 2015. Catalyst 2015 - ChannelAdvisor’s annual industry conference, one of the most important events for e-commerce. Last week Catalyst 2015 took place, where SkuVault was a bronze sponsor and had an exhibition booth. This conference was a big milestone for us: we had an opportunity to demo a completely new feature there - Interactive WavePicking . Wave picking in warehouse management systems is a process of grouping incoming orders into small batches (picklists), which are assigned to workers for assembly. This way a worker can assemble multiple orders in parallel, increasing the overall throughput. Interactive WavePicking in SkuVault provides ability to create these picklists from pending sales, using various criteria. When this picklist is assigned to a worker, the worker will get guidance from SkuVault, telling where to go, which item to pick and which compartment in the cart to put it into. This guidance is location-aware, thus it minimizes walking around the warehouse and increases the overall throughput of the assembly process. It was hard to implement this feature in SkuVault v1 design (based on CQRS/ES architecture), so a different approach to design and development was required. We aimed at two goals simultaneously: come up with a new software design, tailored for SkuVault project to use in further development and constructed to serve customers better in the upcoming years; implement a new feature using that design. Implementation of the feature was an indicator of success or failure of the new design . If development of the feature works out, then the new design is good enough to move forward with. If it doesn’t, then we fail early and save the effort of a major rewrite and a cost of a big failure. I think, we did well. SkuVault team demonstrated WavePicking at Catalyst 2015 , and soon this feature will be available to all customers. Benefits of the new design will come forth in the next quarters of 2015. Major thanks go to: Slav Ivanyuk - for managing the process and doing all super-helpful code reviews Chris Witt - for picking up ReactJS/Flux and doing all the work in UIv2 Jason Henson - for helping with the demo setup, testing and deployment Feature design was driven by SkuVault experts, who had to do a very complex job of identifying the feature essence and figuring out which parts would need to make it into MVP: Andy Eastes , Slav Ivanyuk and Danny Shaw . In order to prepare path for Interactive WavePicking we had to: Design and implement a new infrastructure for the next UI version and backend with APIs. Provide a way to integrate v2 with v1 in various environments: production, where we have more than 20GBs of events coming at high throughput, QA with auto-deployable backend and demo with in-memory backend hosted inside IIS process. Figure out an approach for gradual migration of all existing features from v1 to v2. Figure out a way to improve testability, scalability in new design, while making it easier to develop and learn. So here is what we achieved so far. New Infrastructure Caveat: this retrospective covers only a fraction of SkuVault's life that I was involved in. If we forget about integration with v1 code, the new infrastructure is quite simple. Backend is a collection of .NET modules with an API (JSON over HTTP, provided by NancyFx). Each module can subscribe to events and publish events (async, batched). Modules behaviour can be captured in use cases, which are then applied to verify the correctness of the implementation. This .NET implementation is very similar to reference implementation in golang , which is an open source reference implementation based on the design insights we came up at Happy Pancake. I did almost a 1:1 conversion of golang code to C#. Middleware to pass events between modules and also from v1 to v2 is MessageVault ( available on github ). It is a simple Kafka-inspired event bus, which uses Windows Azure to maintain a highly-available transaction log. Reads are served by Windows Azure storage (via a .NET client library), while writes are handled by a cluster of worker roles (master election is done via Azure blob locks). Front-end in v2 is an absolute pleasure to work with. Previous version is a single project based on ASP.NET MVC with AngularJS and Lokad.CQRS client libraries. New version is a collection of single-page web applications (one web app per major feature), which are statically compiled into JS and CSS bundles. These apps are stateless (web server simply serves the content) and get all data directly from backend API. Stateless web applications are very easy to scale, especially if they are represented by statically compiled content. Besides, any web server can serve them, making development and deployment process very flexible. Just at SkuVault, we can serve UIv2 via: webpack (dev environment), Azure Web Sites (QA and PROD environments), IIS (demo environment). Internally we chose to use ReactJS with Fluxible, since that was the stack we had arrived at during HappyPancake. This software design is backed up by the work done at Facebook and Instagram (it is always good to stand on the shoulders of giants). Chris Witt jumped right into the development process and tackled the UI side of feature development since then. He did a very good job. For example, the UI below allows warehouse managers to create picklists from pending sales. They can apply a dozen of filters to these sales, pick individual sales or batches, reorder sales in the picklist. With ReactJS, we decomposed a relatively complex UI into domain-specific reusable components; FLUX architecture pattern provided a consistent way to capture event-driven UI and client-server interactions in the code. It was easy for multiple developers to work in the resulting codebase, both to continue each other’s work and to develop UI elements in parallel. Software Design Feature decoupling is the most important aspect in our new design. Even though wave picking is quite complex to implement (that’s why it is rarely handled properly in warehouse management), in SkuVault this complexity is isolated from the rest of the system. WavePicking backend API is a separate event-driven module, which can be tested, deployed and scaled independently from the rest of the backend (it can also run in-process with all the rest, for the demo, development and on-premises deployments). Similarly, Wave-picking UI is a separate web application composed from reusable ReactJS components, Flux actions and stores. This application benefits from shared elements (e.g. styles, login/logoff, UI components, build process), but it can be developed and deployed separately. This gives us the benefit of controlled system evolution : we can take existing features and transition them to our new design one by one, minimising the risks and avoiding the total rewrite. In parallel, we can add completely new features. Splitting the system into modules and UI features with well-defined boundaries also simplifies team management and resource allocation . It is easier to manage development of several isolated features than to coordinate development within a single tightly coupled product. It is also simpler to scale the development process, too. Explicit separation between the UI and backend with API was an important design decision with a long-lasting impact on existing product. First, it allows us to divide (and conquer) development into two distinct contexts with very different specifics and challenges: backend and API development focuses on the core domain, scalability and making it very easy for UI folks to build various front-end features. That is pure .NET with low-level optimisations for performance and scalability. UI is going to be THE primary consumer of API, but not the only one: mobile clients and partners will be using it as well. It cares more about the User Experience, feedback, rapid development iterations and pure HTML/JS/CSS development (for the Web). Another interesting side-effect of the design is that we get cheap UI deployments and foundation for A/B testing . One can simply copy feature UI files to a new directory on a web server, getting a different version deployment. If you point different users to different versions of a feature, you get the ability to do gradual roll-outs, per-user customizations and the grounds for A/B testing. On the backend side, development process is enhanced by event-driven use-cases introduced to specify and verify API behavior scenarios. These use-cases improve upon existing specifications at SkuVault, making them less fragile and focused more on the public contract rather than internal implementation. Unlike traditional unit tests, API verification with use-cases can pinpoint the problem in case of failure. As a result, this process saves development time and lends itself to “Getting Things Done” mentality. Use-cases additionally grant us other benefits: API documentation can be generated automatically. It will always stay up-to-date and its quality will be better than what libraries like ServiceStack and NancyFx can auto-generate provide out-of-the-box. Use-cases align very well with the development process, making it easier to manage. Especially well it works with Domain-Driven Design, which SkuVault already employs. Sensible stress-tests can be auto-generated out of the use-cases. They allow running the system through all the scenarios from the specifications, but repeated 1000 times or more. This capability does not replace custom stress-testing scripts, but it comes for free. Build server can run stress tests on each commit, watching for performance regressions and correlating them with changes in the code. Development Process I think, we managed to reduce development friction in v2. UI features in v2 are incrementally recompiled on-the-fly whenever a file changes (thanks to the webpack). We also leverage webpack dev server to handle hot reload (when compilation happens in memory and changes are pushed to the browser). This speeds up web development dramatically. UIv2 features no longer require ASP.NET MVC, so it is not bound to Visual Studio (or to Microsoft Windows itself). Developers are free to choose an environment that fits their needs. It seems that Atom IDE and Sublime are the most frequent choices of people doing web development with ReactJS. Tools like WebPack, ESLint, ES transpilers work from the command line and are supported by all modern IDEs. We also observed that ReactJS simplifies UI development , especially for user interfaces with complex interactions, and when compared to the MV* designs (MVC, MVVM, MVP). That increases productivity and lowers development risks. There obviously is some learning barrier while switching from ASP.NET MVC + AngularJS + jQuery to ReactJS, however the effort pays for itself soon. To make UI development more productive, we use LESS preprocessor for styles (also managed by the webpack), lodash for functional helpers in JS and superagent for AJAX calls. User interactions are all captured in vanilla JavaScript at the moment (later we might enable some ES6 features, which can be transpiled down to ES5 by webpack). JavaScript can be tricky , to make development more reliable we run a linter (ESLint) with a rather strict set of rules. It forces all code to be written consistently, avoiding code smells and bad practices. These rules are enforced by the build server. Real-time statistics and logs are still evolving at SkuVault. Although there are a few glitches (e.g. Hekad integration is less than perfect on Windows), we are consistently improving the experience. At the moment we have more than 80GB of searchable logs handled by ElasticSearch and visualized by Kibana. Carbon and Graphite take care of capturing and reporting dozens of stats from various cloud services running on Azure (starting from RAM/CPU consumption and down to a specific web request latency). Statistics and logs aren’t a mission-critical piece of infrastructure, however they are extremely helpful in understanding software behaviour under production loads. SkuVault needs to scale a lot to serve new customers better, and this toolkit provides real-time insight for that (aside from helping us to debug any potential issues). Instrumenting the existing code to write to the distributed log or to report a new stat metric is easy. API v2 comes with these capabilities from the start, since we are planning to go for aggressive scalability targets with it. Learning process and complexity had to be factored in the new design as well. The simpler it is for developers to understand the design and become productive in it, the simpler it will be for the company to find new talent and to grow. Here is the list of technologies which v2 aims to discard (for good): ASP.NET MVC and all web development in .NET Angular.JS with jQuery ServiceStack API Lokad.CQRS Instead we introduce: ReactJS/Flux NancyFx So far knowledge transfer for UIv2 development had been rather smooth within the company (thanks to the talented developers of SkuVault). Long-term Impact It is always nice to consider possible long-term benefits that could come either cheap or for free. With the new design we potentially get: Well-used and tested API, which could support various clients. Ability to reuse the experience and product knowledge of our web developers to build native clients for the modern mobile platforms: iOS and Android (thanks to React Native). Path for scaling out the system (API is scaled by modules and then partitioned by tenants; UI is stateless and can scale infinitely). Support for on-premises deployments of SkuVault, along with geo-affinity around the world. Ability to deploy system to different clouds to provide higher availability guarantees to our customers. What's Next? For SkuVault, scaling and stability is the primary focus for Q2 2015 . We are going to take existing features one by one and migrate them to the new design, while improving test coverage and performance. New event-driven design gains solid APIs in UI (ReactJS) and on the backend (event-driven design), which reduces coupling and fragility, leading to a simpler and smaller codebase. In order to achieve these goals, continuous integration and build process will also have to be enhanced. We are interested in fast builds and development feedback. Time that a developer spends waiting for his computer to run MSBuild or Resharper to finish code indexing is wasted time. Visual Studio and MSBuild are notorious for being extremely slow on large solutions. Fast builds in v2 are already a part of the design: UIv2 doesn't have .NET in the pipeline at all (plus webpack provides continuous builds), while backend gets faster builds due to the smaller solution footprint. Build process itself could get feedback from: Static analysis: ESLint and Flow in JavaScript. Unit-Tests: testing a single component. Integration tests: use-case verification, derived sanity checks and tests for interactions between the components. Automatic UI/UX verification via scripted user interactions (e.g. WebDriver) Performance tests: scenarios derived from use-cases and custom ones. Codebase size and complexity tracking. Ideally, developers would get this feedback within 5-10 minutes after a commit - while they can fix issues most efficiently. As more features migrate from v1 to v2, we will pull more developers into the new environment. The knowledge transfer facilitation has already started, it will have to be managed more explicitly in Q2. All in all, 2015 Q2 is going to be a very interesting time at SkuVault :) Many thanks to Slav, Andy and Ksenia for reviews.


Emerging DSL

Emerging DSL This is a quick update on the progress of our migration from Cassandra to LMDB (initial plan outlined in previous blog post ). Fast Tests We started by rewriting one of secondary systems from event-driven Cassandra to event-driven LMDB design. We kept all the tests and API contracts, everything else was rewritten by a team member in a week or two. This system is read-heavy with a short burst of writes on event replay, when it builds local read models. At this point it needs to replay of 100M events taking 50GiB. Cassandra implementation took 10 hours, LMDB does it in 2-3 without any serialization optimizations. Here are immediate observations: Cassandra implementation used to run our tests (they verify behavior of the entire system at the API level) at the speed of 1 test per second . LMDB tests currently run at the speed of 950 per second on the same laptop. Since all ~700 tests now run in a second, we enabled test auto-generation by default . This added 2200 tests, and improved edge case coverage at the cost of 3 more seconds. Cassandra and LMDB data access patterns tend to be similar (despite the obvious differences between the databases). Structure of the keys and values stays roughly the same. This made migration straightforward most of the time. Screenshots are from my own development laptop, which runs Windows in a separate VM and gives it only two hardware threads out of 4 available. Benchmark laptop has 8 of them and runs Windows directly. We achieved high test throughput with two trivial things: switching LMDB to async flushing and creating a database per thread. High availability LMDB is an embedded transactional database (unlike Cassandra which is a replicated eventually consistent data storage). Transactions make life better, but in exchange we need to deal with high availability and fail-over . This will come via: Partitioning nodes by tenants. Running multiple replicas of a partition behind a load balancer (we could pick which partitions are replicated and how). Nginx can easily handle routing and failover in this scenario. Having secondary hot replicas in a different data center. If the primary datacenter goes down, we could switch to the secondary by updating DNS records (good enough for now). This is the hard part. Fortunately, we can make some business-specific trade-offs and simplify the overall design considerably: write-side could survive 1-5 seconds of downtime (especially, if API clients can queue up work) as long as it is consistent; read-side could lag behind up to 5 seconds as long it is always available, transactionally consistent with version numbers and fast. Emerging design While porting system to LMDB we agreed to start by writing everything by hand. FoundationDB client libraries made our life simpler, but still required some manual byte wrangling. public void When ( Tx tx , StatusRenamed e ) { var oldKey = FdbTuple . Create ( ModuleTables . StatusByName , e . RefInfo . TenantId . Id , GetCorrectKey ( e . OriginalName . Name )). ToSlice (). GetBytes (); try { tx . Delete ( oldKey ); } catch ( Exception ) {} var newKey = FdbTuple . Create ( ModuleTables . StatusByName , e . RefInfo . TenantId . Id , GetCorrectKey ( e . NewName . Name )). ToSlice (). GetBytes (); tx . Put ( newKey , BitConverter . GetBytes ( e . Id . Id )); } After a week of work we reviewed the system and identified emerging patterns. Instead of writing a data access library to made them simpler (that would've involved some reflection and complex performance optimizations), we just plugged in a simple codegenerator which replaced 30-40% of the code with the generated stuff. We wanted this new code to be boringly simple and things turned out well. Given these DSL definitions from the single schema file: ( space "StatusByName" [ tenant/id status/name ] [ status/id ] ( put "AddStatusName" ) ( del "DeleteStatusName" ) ( fetch "GetStatusIdByName" )) Developers would have boring LMDB access methods generated for them. public void When ( Tx tx , StatusRenamed e ) { var tenantId = e . RefInfo . TenantId . Id ; var statusId = e . Id . Id ; Lmdb . DeleteStatusName ( tx , tenantId , GetCorrectKey ( e . OriginalName . Name )); var dto = Lmdb . NewStatusByNameDto (). SetStatusId ( statusId ); Lmdb . AddStatusName ( tx , tenantId , GetCorrectKey ( e . NewName . Name ), dto ); } These generated methods simply spell out method parameters and pass them in a proper order to byte managing routines. public static void AddStatusName ( Tx tx , long tenantId , string statusName , StatusByNameDto dto ) { var key = DslLib . CreateKey ( Tables . StatusByName , tenantId , statusName ); DslLib . Put ( tx , key , dto ); } Dtos are also generated automatically in such a fashion that we could later swap out Protobuf serialization format for values to something more performant without requiring developers to rewrite business logic. Currently we are evaluating FlatBuffers / Cap'n Proto for storage of complex objects that rarely change. Native multi-value capability of LMDB would work well for frequently changing property groups. Custom bitstream compression is the ultimate goal. Edge cases While doing that we agreed to codegen only the cases which were repeating all over the place. Edge cases will have to be written by hand until we get enough of them to see emerging patterns. Resulting code feels trivial and fast, if compared to the other versions of the same logic. Like a breath of fresh air on a clear and chilly autumn morning in the Hudson valley. This is the goal we are striving for: boring code designed to run fast and scale well for new features. This should ultimately let us create more value for the customers, while making our own days more enjoyable and less stressful. Linguistic complexity Technically, the language of the system is more complex - we have two languages instead of one (with a codegen step). Practically, it still feels simpler. Probable reasons for that are: our handwritte code uses only a small (good) subset of C#/.NET features. It is just a bunch of static methods that transform data between different reprentations, while being conscious about performance. DSL doesn't hold much magic. It is just a bunch of Clojure files doing obvious transformations according to the simple rules. Anybody could rewrite it to another language (e.g. Racket or CommonLisp) in a couple of days. Side benefits DSL design forces us to write out type aliases before referencing them in LMDB layer. They spell out types of data (properties) within a linguistic (bounded) context, while aliasing them with short namespaced identifiers and descriptions. ( group "tenant" "tenant-related schemas" ( alias id "Numeric tenant id" positive-long ) ( alias name "Tenant name" string not-empty )) ( group "user" "User within a tenant" ( alias id "Unique user id" positive-long ) ( alias name "User name" string not-empty ) ( alias email "User email/login" string not-empty ) ( alias role "User role, affects permissions" string not-empty ) ( alias password-hash "Password hash" string not-empty ) ( alias enabled "Set to false to disable access" bool )) ( group "product" "product details" ( alias capped-key "Capped sku/code that fits 200 chars" string ( max-len 200 )) ( alias is-kit "Is this a kit or not" bool ) ( alias id "Product id" positive-long ) ( alias key-type "Product key type constant" byte )) These aliases can also be associated with data specs, which spell out how should the data look like. For example, if we know that product/capped-key will always comply with (max-len 200) , then we don't even need to define types or write argument checks. These aliases will help in defining consistent APIs with auto-generated documentation. We can also perform consistent argument validation not only in API implementations, but also in client libs in multiple languages. Generating Swagger definitions (to get beautiful API explorer) and JavaScipt client libraries are the next planned steps in this direction. Developer onboarding and knowledge transfer It takes a couple of days to get a first commit from a developer new to LMDB and this DSL (provided that API design and scenarios are already done). It would probably take him 3-4 weeks to get used to writing finely-tuned LMDB code in places, where this matters. We're planning to add continuous performance testing to shorten the feedback loop and improve productivity in this area. Rewriting existing DSL from scratch or porting it to a new language is probably a week of effort for a senior developer familiar with Lisps and AST manipulation. There are just 450 lines of code there (mostly with C# code fragments). Destination - golang In the long run we plan to migrate the core business logic to golang. The language itself is a good fit for the job: simple, fast and runs natively on Linux. We don't care much about the advanced features from C#, since they are handled by the DSL layer anyway. DSL design pursues the goal of streamlining migration from C# to golang: generated code doesn't have to be rewritten by hand, only the edge cases.


High Availability and Performance

High Availability and Performance This is a summary of interesting things we've achieved at SkuVault since the previous blog post on LMDB and emerging DSL . Current stats are: 1.43B events with the total size of 316.3GB . ~450 event types. System Architecture System architecture didn't change much since the last update. However, we improved our ability to reason and communicate about it. Let's go quickly through. The core building block of the system is something we call a service . It has a public API which can handle read and write requests according to the public contract of the service. We do an event-driven variation of service-oriented architecture . Whenever a service needs to persist a state change, instead of overwriting DB state with new values, we craft change events and append them to the distributed commit log . Apache Kafka could work as such distributed commit log. You can also try rolling your own commit log on top of a distributed transactional key-value store (to act as a buffer and a serialization point) and a replicated blob storage. Any interested service could subscribe to this commit log, replay interesting events and get its own version of the state. This state will stay in sync with the global commit log via the subscription mechanism. This means that you can roll out new service versions, features or even specialized services (e.g. analytics and reporting that have different storage requirements) side-by-side with the existing services. Further in this blog post you will discover the term module - it is a monolithic unit of deployment for a group of services that share similar functional, performance and availability characteristics. In other words, it is convenient to deploy, manage and version them together. You could scale by placing a load balancer in front of a cluster to route different requests to the service instances that are designed to handle it. The same load balancer would allow you to fail over to replica nodes in case of outage or a simple rolling upgrade in the cluster. This approach also allows multiple teams work together on a large system while building software on different platforms. In this environment, the service behavior can be expressed via its interactions with the outside world : events it subscribes to; API requests it handles; API responses it returns events it publishes to the commit log. If we somehow were able to capture all important combinations of these interactions of the service, then we'd probably specify its behavior well enough to prevent future regressions. This is what event-driven specifications are for. These specifications come with a few potential benefits: separate design phase from coding phase (which reduces cognitive burden); help you to decompose complex requirements into simple event-driven tests; break rarely (see aside); let you refactor services by moving around requirements, behaviors and functionality. In my experience these specifications almost never break. They just keep on being added to the codebase when new requirements and edge cases show on the horizon. This comes with a potential downside: code that never breaks - never gets refactored and cleaned up. So you might end up with specifications written in different flavours. Imagine, you have a service with some complex functionality inside. This functionality has evolved over time. Service accumulated specifications for the behavior, weird edge cases and strange regressions that slipped through. Over the time, it became apparent that this service implementation houses two distinct feature sets. They are currently intertwined but would live better if separated. What you can do in this situation: split specifications for these services into two different groups; discard the implementation completely (or keep it as a reference); iterate on new versions of the services until all tests pass. What if there are important behaviors that weren't covered by the scenarios? This is very good - thanks to this refactoring exercise you've just discovered gaps in your test suite. Throwing out the implementation logic and rewriting services from scratch might be scary. In practice it is a bit simpler and safer than it looks like: these event-driven specifications capture all important design decisions and service interactions (including edge cases and error responses); developers can focus on one specification at a time, iterating till all specs pass. These specifications are the reason why SkuVault V2 backend was able to evolve through multiple storage backends till it arrived to a place we are quite happy with: LMDB. Cassandra to LMDB At this moment we completely migrated away from Cassandra to an embedded LMDB as a data store. Outcomes: better (and more predictable) performance, reduced DevOps load and simpler scalability model. This also means that: We have to write our own application-specific data manipulation layer , which would've been difficult without lisp DSL and code generation. Our reads are served from memory (best case) or local SSD (worst case) . This simplifies scenarios like route generation for wavepicking algorythm or running complex search filters - you have to worry less about the performance. This also means that our operations are currently mostly IO-bound, while CPU is under-utilized. Were we to switch API from JSON to binary format, while also compressing event chunks, then CPU would become the next bottleneck. For now, event replay speeds on smallish nodes look like this: Replay is the process of downloading relevant event chunks from the commit log, getting interesting events and dispatching them to local handlers in order to rebuild node state. We tend to do that on every new deployment. This is the last step in our journey from one storage engine to another. It looks like we'll stay here for some time. Idempotency didn't work out Exactly-once delivery of messages or events in a distributed system is hard . Initially we've tried to work around this problem by enforcing idempotency in the code via auto-generated scenarios. You take an ordinary Given-When-Then scenario and mutate it by duplicating one event at a time. If the scenario produces the same result, then the affected code paths are idempotent. However, as it turned out, it can be hard or plain impossible to implement idempotent event handlers in some cases. Especially, when you have to deal with events coming from the legacy code. So we are planning to push this deduplication concern into the subscription infrastructure . It could handle deduplication automatically by checking up on known message IDs before passing messages to handlers. New IDs could then be pushed into LMDB within the same transaction in which handlers are executed. This would also solve an edge case when a node crashes after pushing event to the commit log but before writing down acknowledgement from it. Edge case looks like this. Imagine that node-1 produces an event-A and pushes it to the distributed commit log. However, before ACK comes back, the node crashes. System will fail over to the replica node-2 and continue working there. Later, when node-1 comes back online, it will receive the same event-A from the commit log, ignoring it - the event is stamped with node-1 identity (so it is supposed to already be in the local state). Cluster Simulation At the moment of writing, V2 part of SkuVault backend has 3 different types of modules: public (serving web UI), internal (serving legacy systems) and export (handles export jobs). Each module has a couple of replicas. Public module also has 4 partitions (each with a replica). These modules collaborate with each other via our cluster infrastructure: distributed key-value store acting as event buffer, tenant version dictionary and a queue; replicated commit log; forwarder process - single writer responsible for pushing events from the buffer into the replicated commit log. Options for such a distributed key-value store are: Google Spanner , VoltDB , CockroachDB and MarkLogic . Existing cluster infrastructure works fine for development, qa and production purposes. It requires some setup, though. We needed another way to run the entire cluster locally with zero deployment . That's when we finally started tapping into the simulation mode: replaced the distributed key-value store with local LMDB implementation, while keeping all cluster-enabling logic intact; replaced the azure-locked version of the commit log with the file-based simulation; launched all modules in one process in this simulated environment that works very similarly to the production setup (there are some caveats, of course). While it is possible to reason about behavior of the distributed system in some edge cases, seeing it happen under the debugger with your own eyes is much better. It helps to reproduce and fix infrastructure-related bugs, while building an intuitive understanding of the system behavior. A first edge case that was reproduced locally - behavior of the system, when cluster fails from a node-A to a node-B that isn't fully up-to-date yet (e.g. it could be in another datacenter). And this happens while a user edits a version-sensitive entity. In that case UI negotiated with the new node to postpone the edit until the node received all the relevant events. Besides, having a simulated local cluster works well for demo deployments , where you regularly wipe everything and restart from a preset state. It is also a prerequisite for on-premises deployments that can sync up with the remote data-center or fail over to it. Note, that this isn't yet a deterministic simulation that can be used to test distributed systems. It is merely the first step in this direction. Missing bits include: bringing all event handling loops to run on a single thread (for determinism and ability to rewind time forward); simulating transaction conflicts in the cluster infrastructure (current LMDB implementation never conflicts - it takes write lock); simulating load balancing, its error handling and fail overs; injecting faults and adding code buggification in the simulation layer and above; running this setup for a couple of months. If you are interested in this kind of setup, check out things like: Discrete event simulation , The Verification of a Distributed System and especially Testing Distributed Systems w/ Deterministic Simulation . Why Does Deterministic Simulation Matter? Let's imagine we are building some distributed system that deals with business workflows across multiple machines. It might deal with financial transfers or resource allocation. Let's also assert that bugs in this system are very expensive . In theory, implementing that distributed logic shouldn't be a challenging task . We either have a cluster with distributed transactions or implement a saga (I'm oversimplifying the situation, of course). We can even prepare for the problems by planning compensating actions at every stage (and covering this logic with unit and integration tests). To be even more rigorous, we can even have multiple teams implement the same system on different platforms. These implementations would then double-check each other. In practice, how do you know that this implementation would actually work out in real world? How would it deal with misconfigured routers, failing hardware or CPU bugs? What if network packets arrive out of order, double-bit flips happen or disk controller corrupts data? Remember, we've said that bugs in this distributed system are very expensive. Studies by IBM in the 1990s suggest that computers typically experience about one cosmic-ray-induced error per 256 megabytes of RAM per month . This is where deterministic simulation comes into play. If you design your system to run in a virtual environment that is deterministic, simulates all external communications and runs all concurrent processes on a single thread (pseudo-concurrency), then you can get a fair chance of: fuzz-testing random problems and reactions or your system to them; injecting additional faults and timeouts for extra robustness; being able to reproduce any discovered failure, debug and fix it. In essence, this approach is about upfront design of the distributed system for testability and determinism. It allows to throw more machines at the problem of exploring the problem space for costly edge cases , giving us a chance to build a system that is better prepared for random failures the universe might throw at us in real life. Zero-downtime Deployments No much to say here. You throw a couple of load-balancers in front of your nodes, telling them how to route traffic and fail over. With that, a deployment turns into a process: spin up new node instances and let them warm-up local state (if needed); switch LB forward to the new version for a portion of the users (ssh, swap config and do nginx reload); if all goes well - switch more users forward; if things go bad - switch users back. Switch takes only a few seconds, since we don't have long-running requests that need to be drained. We are currently using nginx as the load balancer. However I see lyft/envoy as a promising replacement with a better high-availability story. Single-Writer Caveat The system is designed to have a single writer per process . This design decision lets us benefit from LMDB and its ability to have unlimited isolated reads in parallel. Read throughput is usually limited by the network IO in our case - services process incoming requests faster than they can arrive to the network card. There are some exceptions where CPU, memory and disk IO are the bottleneck; for example: routing and complex filtering which require deep graph-like traversal of persisted state. The caveat is: write requests have to be optimized for low latency (taking more than 15-50ms per write is too much). At the same time, we have a lot to do within a write: check local state to ensure that the write request is valid; generate a new event (or a couple of them); apply this event to the local state; commit events, jobs and any other messages to the cluster generate a response to the user (e.g. include newly generated IDs or totals). As it turns out, it isn't very hard to profile performance and improve. For that we hijacked Google Chrome tracing tools. We simply collect method call timings in memory (it helps that most of the code is auto-generated) and render them in Trace event format which could be opened with chrome://tracing . We've also added passive sampling to the servers - they will try to capture and store traces of potentially interesting requests. Note : for some important requests we also check with the cluster to ensure that the node has the latest version of relevant event stream. This adds a network round-trip to the commit cost. If we find the need to increase write throughput of the system, there are a few options: split the system into more partitions (since single writer limitation is per partition); a single machine could handle multiple partitions; move heavy reads to read-only replicas; trade off some latency for throughput by batching multiple writes together; migrate from Windows to Linux/Unix with optimized network and IO; switch from virtual machines to the dedicated hardware; move from .NET C# to golang (or even lower). The only memorable lesson during these optimizations was: don't use services provided by some cloud vendors, if you value consistently low latency. These services tend to value overall throughput for all customers over consistently low latency for one customer (which makes sense). Next Problems Below is the list of the problems that lie ahead for the project. You might find these problems a little bit less exciting than simulating cluster or running hundreds of full-stack tests per second. This is probably due to the fact that the technical design is established for this project, important challenges are mapped. Further scaling would mostly take place in the realm of people, teams and organization. Development Processes and Training Initial development processes were quite flexible. This was possible thanks to the small size of the team: we just went to the office kitchen and drew on the whiteboard till consensus emerged. As new developers come to the project (including developers from the remote locations), it has to switch to a more formal development model. Training process, methodology and materials also have to be established. Client libraries and DSL Code generation of client libraries (along with the API) will speed up development inside the company, while simplifying integration story for the partners and customers. It will save API consumers from doing double work and re-implementing half of the domain validation logic in their own language. Besides, shipping client libraries with a nice interface would save us the effort of precisely documenting our API along with intricacies of the transport layer, fail-over, back-off, authorization and version management. A client library would let us switch from JSON API to something more performant (e.g. GRPC with HTTP/2) without customers investing any development effort (aside from updating dependencies). Cluster Management and Visualization Imagine a clustered system with 3-4 types of the modules, where: each module has a different deployment profile (e.g. one scales by partitioning, another - by running multiple replicas, third - by adding more queue consumers); each module can have multiple versions running on the cluster (some versions could be enabled, others disabled, some more - partially rolled out); some modules could be in warm-up state (replaying commit log to build their local state); there could be additional deployments of modules used to test or demo new features (currently incompatible with the rest of the cluster); in order to save resources, multiple module instances could be crammed into a single VM (as long as doing that doesn't hurt high availability). This situation might look like a mess. It definitely can confuse project managers, stakeholders, testers and even developers. Confusion can lead to mistakes, anger and the dark side. We don't want that. One way to remedy the situation is to visualize cluster state for people and provide tooling to manage this state, while enforcing domain-specific rules . Fortunately, all this is simple to achieve with some diagnostic endpoints, shell scripts and d3.js . For example: never remove the last active node from the load balancer; never add a stale node to the load balancer; keep partition replicas on different machines; ensure replication factor of 3 for VIP replicas. Summary This design isn't doing anything particularly novel. We are essentially building application-specific database engine tuned for the challenges and constraints of the domain. Event sourcing and Domain-Driven Design are used as higher-level design techniques that let as align "what people want to achieve" with "what computers are capable of" in an efficient way. In more concrete terms, the current solution: solves existing scalability problems and provides foundation for solving future challenges, should the business experience rapid growth; lays groundwork for bringing in new developers, training them and organizing multi-team collaboration around the core product, services and add-ons; maps a way to tackle complexity in the software, while adding new features (including some that are unique for this industry) and maintaining them through implementation rewrites.


Black Friday Boring

Black Friday Boring 3 years ago I was invited in SkuVault to help with a set of business and technical challenges: short-term : figure out logging and telemetry for a distributed system at scale; mid-term : scale out the software in response to the business growth, while also improving the performance; long-term : simplify the design of SkuVault, making it easier to maintain and add new features. For me it all boiled down to a single goal: ensure that SkuVault can handle Black Friday peak load without any major issues (meaning that an ordinary busy day would just be a breeze). The journey took 3 years. Finally this year SkuVault had a very boring and uneventful Black Friday . During the same period some of the competitors have experienced customer-impacting performance issues. Previous two peak seasons were successful but quite stressful for everybody (emergency calls in the middle of the night aren't what we usually sign up for as software engineers). We organized shifts to maintain round-the-clock watch during the peak season of 2017 as well. Fortunately, no intervention was required during these boring hours. This achievement was a result of work done by multiple teams: V2 team drove the migration , continuously learning and fearlessly adopting new technologies and development practices; V1 team covered the backs by supporting the existing product and switching parts of it to the new infrastructure; QA and Customer Support helped to ensure the best customer experience despite all the changes, they went great lengths at that, too; DevOps/SysAdmin team tirelessly tended to the ever-changing and growing garden of services as the infrastructure evolved to adapt to the new requirements. Let's see what was achieved in the past 3 years. SkuVault V2 3 years ago SkuVault was based on a Lokad.CQRS framework - designed to assist in building distributed message-based systems that could be developed locally and deployed to Azure. This implementation, later called "SkuVault V1", was good enough to build a product. However, it wouldn't scale behind a certain point . As the product grew in complexity, adding new features also became difficult . V2 implementation allowed to scale out critical parts of the system , taking the load off the existing V1 components during the last Black Friday. V2 is also the place where the new and complex features are being implemented . SkuVault V2 backend is based on the event-driven services design which was covered in a few conference talks and blog posts (e.g. see the design overview ). Frontend is a classical single page application in React.js and Redux. It would still take some time to migrate the remaining V1 parts to V2. However, the process is well-understood, supported by the experience and some tooling. Wave Picking Wave Picking is an example of a feature that was possible to develop only on V2 design (if we wanted to support big customers with huge warehouses and diverse inventories, that is). This feature initially evolved with the understanding of "V2 design", being used as a litmus test of its maturity and usability. It was later extended with the new functionality like "carts", "zone picking" and "partial orders" (which warranted a UI/UX overhaul). At this moment Wave Picking is a stable feature that has been in use for a long time. A dedicated team is currently working on enhancing it with the capabilities which would let small retailers to keep it up in the game with Amazon . Telemetry Extensive telemetry infrastructure , which was non-existent just 3 years ago, is an essential part of SkuVault system these days . There are multiple Elastic Search and InfluxDB clusters ingesting gigabytes of logs and millions of metrics on a continuous basis. Grafana and Kibana offer a real-time insight into this data , providing tools for building visualizations, exploring the data and building up the intuition about the system behavior. New Office SkuVault always had some remote developers. However, 3 years ago I was the only one working from Ufa (Russia). Nowadays, there is an office with 11 people - team with the experience of working together . The experience and the roles include: doing business analysis and writing specs for V2 systems; designing and developing V2 frontend; developing, optimizing and maintaining V2 backend systems; testing and ensuring quality of the software; managing and organizing the development process; doing systems administration for the entire SkuVault and managing V2 infrastructure. It was an interesting experience to design the software system for simplicity of use and training . It was even more captivating to facilitate the initial knowledge transfer, adjusting the model based on the learned lessons, and seeing how ideas and practices spread further across the team. Epilogue My personal goal at SkuVault is complete: the last Black Friday was boring. There also is a shared understanding about how the software system could evolve in the next years to handle more features and meet new scalability targets. For instance, for the next year the teams at SkuVault will be quite busy completing V1 migration. However, this completion would only signify the start of the next phase of research and development towards V3 . But that would be another story for another scribe to document. Some of the long-term challenges include : completely decouple V2 implementation from the Windows Azure, once the V1 migration is complete (this involves developing a new version of Message Vault); build up a vision and tooling to develop a consistent user experience on multiple form factors (tablet, desktop and mobile) while running on different platforms (Android, iOS, Web and Win CE) and continuously delivering new features; bring platform experience to the partners, allowing them to extend user experience in SkuVault or even deliver completely new features. If you have experience that could complement this work and are interested in relocating to Louisville, KY, then SkuVault might have a compelling job offer for you. This company is growing (which often happens when the software doesn't get in the way of the business or even empowers it a little), so they are on the lookout for the capable talent .


Verification of Event-Driven Systems

Verification of Event-Driven Systems I'd like to share an interesting experience with event-driven design from SkuVault (the company is a warehouse management system provider). We developed a new feature called wave picking while building it on a new infrastructure, better suited for high loads and scalability ( more about this feature ). Here is the story. Starting with a prototype We started working on the feature with a prototype : a stateless web application (HTML/ReactJS/Fluxible) that talks to a backend via an API (JSON over HTTP). The Web UI focused on convenient user experience, while the backend handled persistence, business rules and domain complexity. This entire system was an add-on to the existing software, communicating with it via an event bus (with the ability to replay all events from the very start). To speed up the initial development, we chose in-memory persistence for views in the backend . This meant that the system would maintain all views (read models) in process, while rebuilding them upon the restart. This also meant no failover or scale-out. This approach allowed us to quickly capture important domain concepts in code : starting from flexible sales search and up to route generation inside the warehouse. It was fairly simple, since we only had to deal with in-memory structures and a little bit of concurrency. Once the first features were implemented in the backend, UI developers could pull the backend to their machines, launch it (there were no dependencies, except for the .NET runtime) and just concentrate on the UI. Further development took place in parallel. Given user stories, interface (HTML/CSS) and proposed API implementation, backend and frontend work could start at the same time. Backend developers would translate their requirements into explicit scenarios and in-memory model. In the meantime frontend developers would start building ReactJS/Flux parts. A few days later (or hours, depending on the complexity) first drafts of new in-memory implementation would be passed to them to give a spin. Sometimes frontend developers would point fingers and say “This method doesn’t work as expected, here’s the curl expression” or “We could use HasMore field on the collection to simplify paging”. These requests would be incorporated into the scenarios and implemented in in-memory. Scenarios A scenario in this project is a data structure that nails down behavior that we want to last . It captures this behavior as a transition between two states, where states are expressed via domain events. Given Events - a sequence of events that setup preconditions and describe what happened in the past (e.g. Sale 12345 was created with 1 bike-blue-pro ) When Request - API request that we are testing (e.g. POST “/wavepicking/session” with sale id 1234 ) Then Response - API response that is expected from the server (e.g. HTTP 200 OK { “version”: 2 } ) Then Events - any events that might be published to event bus (e.g. SessionCreated with sale id 1234 ). We don't verify the internal state, since it could change between versions. We are interested only in public contracts of the system: events and API calls. Here is an example of how such use case could be expressed in C# code: public sealed class When_create_cart : UseCaseSyntax { public static UseCase give_good_lowercase_code () { var author = New . Author (); var newDate = DateTime . UtcNow ; var ri = New . RefInfo ( author , newDate ); var newId = Guid . NewGuid (); return new UseCase { When = POST ( "/cart/" , new { code = "myCart" }) . As ( author ), ThenResponse = OK ( new { cart = new { code = "MYCART" , id = newId }, version = new { inventory = 1 } }), ThenEvents = Events ( new CartCreated ( new CartId ( newId ), ri , "MYCART" , 0 )), ServerGenerated = new object [] { newId , newDate } }; } /* more scenarios for cart creation */ } Note that in scenarios we don’t test the entire API response, but rather specify only DTO paths that we are interested in. This way we could write a bunch of scenarios testing a single feature in a given API response (e.g. a specific denormalization), while leaving other parts of the API response to other scenarios. Once you have the scenario, development becomes a simple matter of monkey-coding: bang on the keyboard, until all scenarios pass (including the new one). In essence, these scenarios became our type system checks for the business domain . They nail down behaviors that we want to last. There is one cool aspect with this approach: scenarios are an essential part of the development process. They are written whenever there is need to capture a new requirement, fix a bug or debug a complex behavior. This is an example of test-driven development that actually reinforces itself. Tackling Complexity of the Scale At some point down the road, we had to make the backend production-ready: durable and scalable. We picked Apache Cassandra as the primary storage, since it is widely adopted, scales elastically and never loses data, if used properly. All the complex intricacies of the domain logic were already captured in the in-memory prototype and reinforced by the scenarios. This turned a complete backend rewrite into a rather simple task approachable by a single developer: just make the Cassandra implementation work like the in-memory prototype, while making sure that all scenarios pass . A few weeks down the road, this new implementation was ready in a separate git branch. It brought a lot of technical complexity linked to the way data modeling works in the Cassandra universe, but this complexity was be tackled separately from the intricacies of the domain model. It felt good. We swapped QA backend to a new version (this swap happens without any downtime if you have a load balancer) and asked people to use the software just like they used to, while paying special attention to edge cases, where the behavior changed. There were a few differences, of course . They pinpointed real cases which weren’t covered by specifications, letting two implementations diverge from each other. Fixing that was quite trivial (since the problematic areas were already identified). More often than not, the root cause was within the implementation of Cassandra’s data model, forcing us to rethink it. Being able to separate domain complexity (captured in scenarios and in-memory) from the technical details (cassandra) led to a enjoyable experience, fast debugging and development iterations . We loved it. Let's Keep In-Memory Version At that point were were able to get rid of the in-memory implementation. However, the in-memory version worked so well for us in explaining how things should work, that we decided to keep it in the codebase while making it a first-class citizen . The branches were merged, and both implementations of the backend were kept. Here is how one of the modules looks like: public sealed class ProductListModule : IModule { public ModuleInstance Memory ( IMemoryDependencies provider ) { var store = new Memory . ProductStore (); var reg = new ModuleInstance (); reg . AddProjection ( new ProductsProjection ( store )); reg . AddProjection ( new ProductStatusProjection ( store )); reg . AddDataStore ( store ); reg . AddApi (() => new Memory . ProductsService ( store )); return reg ; } public ModuleInstance Cloud ( ICloudDependencies provider ) { var session = provider . GetSession ( "v2_product" ); var store = new Cloud . ProductStore ( session ); var reg = new ModuleInstance (); reg . AddDataStore ( store ); reg . AddProjection ( new ProductDenormalizer ( session , store )); reg . AddApi (() => new Cloud . ProductsService ( store )); return reg ; } } Keeping in-memory implementation around raised a few eyebrows outside of the dev team. "Wouldn’t we waste time on maintaining two backend technologies at once?" Fortunately, at this point we already understood benefits well enough to articulate that in-memory implementation is our domain model . It provides us with a quiet happy place in our minds, where we could forget about Cassandra specifics while focusing on the essence of the business. Once is it captured, further implementation is just an application of skill and prowess of a developer to write heavily optimized code. Besides, there were a few additional and very specific benefits to keeping in-memory implementation around: In-memory implementation is used to run local development versions backend for UI developers (we don't want them to bother with installing and maintaining Cassandra nodes locally). In-memory version is also used as a demo (we could create short-lived transient accounts in memory for customers interested in trying out the product). Thus, we were able to prove the point and preserve that quiet happy place in our minds. Idempotency Testing In a distributed system it is very hard (or rather expensive) to guarantee that a given event will be delivered to subscribers only once. It is far “easier” to ensure that subscribers are idempotent instead - that sending a duplicate event wouldn’t change a state. We can guarantee this to a certain extent by taking all existing scenarios, duplicating events in them on-the-fly and then passing through the same scenario verification code. Adding an “idempotency tester” mode to verification code is a simple operation, but it ensures that any existing or new behaviors are idempotent (as long as they are captured by scenarios). Delta Tests Delta testing is another approach which reuses existing scenarios and makes them more useful. We take all existing scenarios and run them against memory and cloud implementations, comparing outputs (published events and API responses) between each other. This includes even fields that we never check for in actual scenarios. If outputs differ somehow, then we have a probable issue on our hands. There is a nice psychological benefit to idempotency and delta tests - they make original scenarios more useful and rewarding to write. It is always awesome when one of these tests discovers an peculiar edge case you never thought about. Random testing At some point we realised that the application of this in-memory model for testing resembled property-based testing like Quick-Check. In QuickCheck you test a software by creating a model for a function, then letting the test suite hammer it with various inputs, while comparing output of the model vs output of the real system. If these differ - then you have a problem (the real implementation is more complex than this explanation, since you have to implement test reduction logic, which flattens the test sequence to minimal number of inputs required to reproduce the issue. With this insight we were able to extend our test suite with random tests. Random testing takes delta testing approach one step further. Its purpose is to explore various state transitions and find spots where behavior of the memory model differs from the one of the cloud implementation. The process looks like this: We take some events from a real system (normally a QA which usually is filled with all sorts of crazy data). We feed event sequence to cloud/memory implementations to populate their state via replay. Then we pass these event sequences through request generators to create random API requests based on that data. Send API requests to both implementations to get outcomes: API responses and any published events. We compare these outcomes between each other, searching for any differences. If there is one - it is a bug. Here is a simplified example of a generator: public static Request gen_search_sales ( IList < IEvent > events ) { var last = events . OfType < SaleCreated > () . LastOrDefault (); if ( last == null ) { return null ; } var author = New . Author ( last . RefInfo ); return GET ( "/wavepicking/sales" ) . As ( author ); } Tests are deterministic - passing the same sequence of events to API request generators will always create the same requests and lead to the same test outcomes. We can improve search space (where we compare memory model to cloud implementation) by: Adding more specific generators Taking different event sequences from the real data. Random tests differ from delta tests: We use subsets of events from a real system (instead of well defined clean scenarios) We generate random API requests that could make sense for these events. Random testing becomes especially interesting to run, if we have different implementations written by different developers. In this case, there is a chance that some implicit behaviors (e.g. behaviors that are required but somehow missed specification phase) will be implemented differently. Random testing might uncover these behaviors that went “under the radar”, allowing us to design them properly and nail them down with scenarios. Keeping an Eye On Performance Decent performance comes naturally when you have to replay hundreds of millions of events upon each schema change. Any inefficiency in how you access data storage, deal with RAM, GC or CPU intensive operations - and your deployment would grind to death. Obviously, it would be great to catch the most glaring performance problems during the development. Fortunately, it is trivial to enhance scenario verification suite with basic sanity checks and performance profiling. You just inject proper heuristics in the scenario verification environment. For example, we could measure execution time of any API call or handler, spot the bottlenecks and print them out. Another approach is to bake in sanity checks into the code, like ensuring that all requests to Cassandra run as prepared statements. Graphs and Analysis Scenarios are just data constructs which nail down behaviors (state transitions) which we want to last. These could be easily parsed and analyzed. Models also capture behaviors, but in an implicit form. We could observe these behaviors by running data through (real-world or synthetic) and capturing various data points. Here are a few ideas that we tried: Detect scenarios or behaviors that are untested (e.g. an event is declared in module dependencies but not covered by any scenario). Detect unused code paths (e.g. an event is published by a module, but there is not a single subscriber in the system). Detect specified and tested code paths that aren’t used by the customers in real world. Besides, you could always render this data as graphs and let human perception spot problems and irregularities. Next Steps This journey has been extremely rewarding so far. It feels great to be able to evolve the domain model while knowing that all the captured domain knowledge would be losslessly translated into actual implementation. There are still many things to explore in this direction: Using Lisp to express and generate contracts and schemas for events and API in multiple languages (currently custom ANTLR-based DSL is used). Switching cloud implementation to Kafka/Cassandra/Mesos/Spark/Elastic (currently it is locked to Windows Azure) Adding higher-level model validation based on valid shapes of data and data flows (e.g. when you have multiple data centers in different regions with replication across via event bus)


FoundationDB Layers

FoundationDB Layers A few days ago Apple open-sourced FoundationDB - a transactional (ACID) key-value database with crazy testing and ability to scale to peta-byte clusters. I wrote more on why this is so exciting in FoundationDB is Back! . A few companies have been using FoundationDB in the past years: Apple - bought FoundationDB 3 years ago; WaveFront by VMWare - running petabyte clusters; Snowflake - cloud data warehouse. What I didn't tell at the time: SkuVault happily used FoundationDB in the past years as well. This is the story. Commit Log Layer Initially SkuVault V2 was designed to run on top of a MessageVault - a custom highly-available hybrid between Kafka and EventStore , designed to run on top of Windows Azure Storage. We referred to it as the Distributed Commit Log . Distributed Commit Log at SkuVault holds 1.5B events spanning 400GB of data and growing. It is split into 4 partitions and a few thousand event streams (stream per customer). Some streams have 10M events and more. That implementation was good enough to let the development move forward. Further on it became apparent that we wanted better throughput, predictable and low tail latencies . It was hard to achieve while writing to Azure Storage. Commits that took a few seconds were fairly common. We also needed some kind of stream version locking while doing commits. Otherwise nodes had a slim chance of violating aggregate invariants during fail-over. Version Locking is similar to how aggregate invariants are preserved in classical event sourcing with aggregates. When any node attempts to commit an event, we want to ensure that it used the latest version of the view state in decision making (view state is derived by subscribing to the events). FoundationDB felt like the only option that would let us implement these features fairly quickly . It worked as planned. Given the green light, I was able to implement the first version of the new Commit Log Layer for FDB a few days . In comparison, it took a few months to implement the initial Message Vault. Layer implementation was simple. We designated a subspace to act as a highly available buffer to which nodes would commit their events. Key tuples included a stream id and UUID (that was also a lexicographically ordered timestamp when persisted) which ensured that: there would be no transaction conflicts between the different nodes, allowing high commit throughput; events would be ordered within the stream (and somewhat ordered globally). Another subspace acted as a stream version map . Commit Log Layer took care of consulting with it while performing a commit. If node acted on stale data, there would be a conflict, forcing the application to back off, get the latest events and retry the operation again. Both the commit and version check were a part of the same transaction, ensuring the atomicity of the entire operation. There was also a background repartitioner process that was responsible for downloading messages from the buffer in bulk and pushing them to the blob storage. Performance picture improved noticeably as well: Were I challenged to improve this layer, I'd use newly introduced versionstamps instead of UUIDs (for the global order). Write contention within a stream could also be reduced by bringing the concept of conflict ranges into the application code. Queue Layer Given the new Commit Log implementation, server-side logic became fast. Commit latencies were low and predictable. However, occasionally, the server would just block for a long time doing nothing. This often happened when we had to dispatch a message to Azure Queues. The tracing picture looked like this: 20ms delays were common, however we saw occasional queuing operations that blocked everything for a second or two . Given an existing FoundationDB setup, it was trivial to add a Queue Layer that would allow us to push messages in the same transaction with the events. Tenant version checks were improved along the way (switched to async mode) leading to a prettier performance profile: Messages would still be forwarded to a Azure Queues, but that was a side process, operating in bulk. Miscellaneous Layers Eventually a few more application-specific layers were implemented: Cluster Jobs - to schedule jobs for the background workers. Settings and CRUD Storage - to persist metadata and documents that don't need event sourcing. Async Request Processing - for the cases when a client wants to process a lot of request in bulk, caring only about throughput. Application logic can chain changes to all these layers in the same transaction , making sure that all operations either succeed or fail together. In the end we had half a dozen of various application-specific data models living on the same FoundationDB cluster . All that - without introducing and maintaining a bunch of distinct servers. Cluster Simulation API of the FoundationDB is quite similar to the one of Lightning Memory-Mapped Database (LMDB) . Both operate with key-value ranges where keys are sorted lexicographically. Both support true ACID transactions. The difference: FoundationDB is a distributed database used to coordinate the cluster; LMDB is an in-process database with a single writer , used to persist node data. This synergy allowed us to leverage FoundationDB layer modeling principles and primitives while working to the LMDB databases . That was even before we started using FDB! Later all that experience made us feel right at home while working with FoundationDB itself. We were also able to use LMDB to simulate FoundationDB cluster for the purposes of local development and testing. The last part was particularly amazing. You could literally launch an entire SkuVault V2 cluster in a single process ! Nodes would still use all these FDB Layers to communicate with one another, background repartitioner would still run, jobs would still be dispatched. Except, in that case all the layers would work against LMDB. This meant that you could at bring up the debugger, freezing the entire "cluster" at any point, then start poking around the memory and stored state of any node or process. How amazing is that?! This capability was later reused in SkuVault demo mode, which is a special deployment type that creates new accounts on-demand, wiping the data after some time. We packed the entire V2 backend cluster into a single process and instructed it to restart and wipe all data when requested. By the way, if you are interested in this kind of engineering and live in Louisville, KY (or are willing to relocate), SkuVault is hiring. Done Anything Different? Would I have done anything differently at SkuVault? Yes, I'd make a better use of FoundationDB. Use FoundationDB to split event blobs into chunks and store the metadata, making it easier to operate event stores at the terabyte scale . This would also simplify event versioning and GDPR compliance. Switch event pub-sub mechanism from blob polling to watches, reducing event delivery latency. Switch aggregate version locking from a single version number to aggregate conflict ranges (just like FoundationDB does it internally), increasing write throughput to large aggregates. Embrace the synergy between LMDB and FoundationDB, making sure that internal tools (debugging, dumps, REPL, code generation DSL) can target both from the start. Double down onto deterministic simulation testing, making it easier to learn, debug and develop fault-tolerant application logic. Epilogue All the way through the development process at SkuVault, FoundationDB was an invaluable asset . SysOps team particularly liked it - they never had to touch the production cluster. The cluster required zero maintenance during the Black Friday load (barely breaking a sweat under the load) or when Microsoft started pushing Meltdown and Spectre patches across the cloud. I'm overjoyed that this outstanding database is now open-source and available to everybody.


Planning Event-Driven Simulation

Planning Event-Driven Simulation In my previous article I shared with you through some of growth pains of an event-driven business system. We also discussed some possible solutions: use API calls and event contracts as the language of the system; capture system behavior with event-driven scenarios; create a simple in-memory implementation with the purpose for explaining the system; make it pass the tests; create robust and scalable implementation that also passes the tests; keep both solutions around, use them to reason about the system and run comparative tests. This approach worked rather well in practice and the system was pushed to production. We kept on adding more features and inviting more users. 6 months later, it faced peak season of 2016 and survived . Grafana and Kibana were an invaluable source of insight into the system under the stress. Coupled with various canaries and real-time alerting, they saved the day more than once. Practice also showed us the pain points of this design : complexity, speed and scale limitations . This blog post will present a short retrospective over the last 6 months: what worked, what didn't and what we are going to change. The Good Parts Here are the things that worked as advertised: Event-driven approach in modeling, design and collaboration. Given-when-then scenarios (these event-driven specifications are the most valuable part of the codebase by now). Statically compiled web app as a front-end. As you can see, the core of our event-driven approach held strong. However, some other implementation parts didn't work out as well. The Fallacy of Two Implementations The idea of running two implementation side by side looked good in the beginning, but couldn't handle complexity growth in the long term: One implementation stores everything in memory, helps to validate API and design, runs demo and development modes. The other is obeys the same specifications but is for production. Code is a liability , maintaining two implementations of the back-end takes more effort. Fixing a bug in memory implementation doesn't necessarily translate to the corresponding fix in the production implementation; models started diverging in subtle ways not described by the scenarios. In the long run, having a separate memory version of the system felt like a dead-end for us. Cloud Scale Cassandra storage back-end didn't work that well for us for a number of project-specific reasons: Once you reach 500 event-driven scenarios, verifying production implementation starts taking 20-25 minutes and hence is bound to happen less frequently. If you think about it, running a Cassandra cluster in the cloud can get rather expensive for the work it actually does. You need to consider: performance overhead of running a replicated DB on top of replicated SAN, DevOps overhead even with Terraform, unpredictable tail latencies that are inherent to any SAN but somehow are worst on Azure Storage. Cassandra is an eventually consistent database (it picks AP from the CAP). Cassandra is very good at what it does, but can't guarantee strong consistency even if you are using something like RF >= 3 with QUORUM on both reads and writes. This is the dark side of deceptively simple CQL that we've been burnt more than once. Terraform and Packer are amazing tools from HashiCorp. With them we were able to benchmark different cluster configurations in the cloud, iterating through them quickly. ScyllaDB felt like a worthy contender. This C++ implementation of Cassandra promises 2-10 improvement, but currently lacks some features we use in Cassandra. In the long run, having a production system running on top of Cassandra felt like a dead-end for us. If I have enough money for the load and a single writer per entity, I'd pick Cassandra in a heart-beat. It makes a perfect view storage. Besides, we wanted transactions. This is similar to what the paper on Google Spanner talked about: We believe it is better to have application programmers deal with performance problems due to overuse of transactions as bottlenecks arise, rather than always coding around the lack of transactions. Smart Scenario Testing Advanced verification modes (idempotency, delta and random testing) didn't work out for us either. They take a lot of time to run and hence were bound to be used even less frequently than basic verification runs. To be precise, idempotency tests were used occasionally, delta tests stopped being used after models diverged a little, random tests were never used. This was a pity, since I liked the idea of catching more bugs simply by running a console in an advanced "hunter" mode. What Next? We've been thinking about possible solutions to our problems for quite a while. We needed a solution that would make the code simpler and faster: get rid of model duality, reduce the amount of code that we write and maintain; replace Cassandra with something that allows us to run tests much faster (at least 30-50 times faster on a development machine); be able to handle our ever-increasing amount of data (more than 200GB of events by now), provide good throughput with bounded latency; run the same code in the cloud, on dev machines and on-premises. As we dug deeper, we've realized that the only way to have really fast verification runs was to move data storage inside the application (to get rid of the inter-process communication and also emulate the storage reliably). However, that option would come at a cost: we need to implement our own replication and fail-over; fast low-level embedded databases usually come with a simple key-value interface, where both keys and values are byte arrays, so we'd need to implement our own storage layer on top. FoundationDB Legacy Fortunately, there once was a nice database called FoundationDB, that was acquired by Apple. The database itself was unique - distributed key-value storage with strong ACID transaction guarantees. It scaled very well and self-healed. Apple has a policy of wiping out every trace of existence of the companies they buy. This disappointed FDB customers and made everybody else wary of closed-source storage engines. The engineering team behind that database was even more impressive. They tested their database in a simulation, running entire clusters on a single thread using emulated networking and storage. This approach allowed them to: inject faults that would be hard to get in a real world (swapping IPs, flipping network packets, emulating network partitions); reproduce any faults or problems deterministically under the debugger; emulate days of cluster lifetime in minutes. @aphyr said : "haven't tested foundation in part because their testing appears to be waaaay more rigorous than mine". So if we managed to implement some fairly simple (but robust enough) replication and fail-over approach while supporting proper system simulation, then we could test the living daylights out of them under different failure scenarios. We already have the building blocks for simulating everything except the storage, anyway. Here are a few inspiring resources that we could steal ideas from: Consistent Transactions with Inconsistent Replication Cheap replication via ordered multicast on the network Simple Testing Can Prevent Most Critical Failures Use of Formal Methods at Amazon Web Services Combining Model Checking and Testing Automating Failure Testing Research at Internet Scale The Verification of a Distributed System Testing Distributed Systems w/ Deterministic Simulation Lightning Memory-mapped Database Roughly at the same time we discovered a wonderful piece of engineering called LMDB . It is a very opinionated embedded key-value database with predictable performance and strong ACID transactional guarantees. LMDB is based on B+Tree design, and works very well with read-heavy workloads. LMDB also leverages memory-mapped files. This means that you can read an object from the database with zero memory allocations. You just need to encode your data using FlatBuffers or Cap'n Proto serialization format. Early benchmarks for event-driven projection with indexes achieved on my laptop 7k/sec (multi-write ACID tx); in event replay mode - 150k/second (ACI). Luckily for us, FoundationDB also left a lot of legacy in tutorials on how to model various storage layers on top of a key-value database. If you look at things from their perspective, Cassandra storage can map rather cleanly on top of an ordered key-value database. Same for SQL databases as well. DSLs for Boring Code We are already using Lisp-based DSL to generate all message contracts with schema validation. It is a natural next step to move even more of the boring and verbose code into a higher-level DSL: API definitions and DTOs (with added benefit that we can generate swagger definitions, client libraries and documentation); storage layer for key-value database (hand-coding tuple and byte operations is boring); pull application-specific consistency controls into the code. Why bother with Lisp? Check out this great article on Lisp and Racket written by a newcomer. Next Steps These insights helped us to plan our next design iteration: keep event-driven design , all scenarios and APIs; discard in-memory implementation completely; replace Cassandra backend with LMDB implementation using FDB storage layer approach; use LMAX-style replication approach (master replicates events to followers, we fail over to them via the LB); take ideas from papers like TAPIR to make consistency decisions part of an application logic (which matches what business did for years anyway); simulate the system and beat it with failures , taking inspiration from giants like Netflix, Twitter, FoundationDB, Amazon; save money by replacing a rather expensive Cassandra cluster with smaller and cheaper application node replicas , partitions; also get a simulation cluster (running on buggy hardware); replace repetitive code (or code that tries to work around the limitations of OOP) with meta-programming . In theory all this should work well.


Getting started

Getting started In the beginning we discussed with Slav the most immediate challenges related to software, maintenance and ongoing business growth. Short-term challenges revolve around logging and statistics aggregation. Mid-term challenges are about scaling the software in response to business growth and improving performance . Long-term objectives: simplifying the design of front-end and back-end, making it easier to add some required features. Logging and Statistics SkuVault generates hundreds of megabytes of logs each day , mostly related to integrations with 3rd party systems. At the moment they are just saved to local storage on back-end worker roles (there are many instances running). Slav wants to bring these logs into one central place for searching and convenient audits. Given the sheer size of logs, he was interested in ElasticSearch and Kibana. They are already used in the industry with a great success. ElasticSearch is an open source, distributed, real-time search and analytics engine. Kibana is ElasticSearch’s data visualization engine. The only trouble would be about feeding logs from the numerous Azure Worker Roles to one central location. We want to have an easy interop with .NET code, ability to perform logging even if ElasticSearch server is not available. Ideally, everything should work out-of-the-box, reusing existing solutions. I performed a quick background check for the stack and interoperability with the existing software. For log ingestion we could use Mozilla Heka, launching a daemon on each worker role to ingest log files produced by NLog . Heka is a high-volume logging infrastructure developed by Mozilla. It is written in go and hence requires very little resources. It runs as a statically compiled native code on many Operating Systems and processors. Another alternative for log ingestion is Logstash (the most popular one, part of ELK stack). However Heka seems to be a better fit, because: It runs as a native code and doesn't require any runtime (Logstash is JRuby), making it easier and cheaper to launch inside Azure Worker Roles. Heka can accept StatsD protocol out-of-the box, forwarding performance metrics to something like Graphite. Heka is a drop-in replacement for Logstash. Drawback of running Heka: it is a newer project with less production exposure and fewer plug-ins. However, if Heka doesn't work for some reason, replacing it with logstash is straightforward. Performance Metrics Improving performance and scalability of SkuVault is an important concern. In order to do that in the most effective manner, we need to answer following questions: What are the biggest bottlenecks and how exactly do they affect software? How do our code changes change performance (if they do)? These answers have to be based on real data - performance timers and counters from the software running in production. Fortunately, Heka does support StatsD format over UDP and there is a .NET library for sending metrics this way. Reporting performance metrics via UDP is used frequently in software. It is fast and doesn't slow down the software much. Besides, UDP is fire-and-forget: if the statistics aggregating process goes down, this will not affect actual application. This week I'll spend a few hours to setup a dedicated Windows Azure VM with ElasticSearch, Kibana and Graphite. Then I'll produce a sample Azure Worker Role, configured to send logs and statistics to that server through Heka. Team at SkuVault will then take their time to reconfigure existing servers to publish logs and stats to this new infrastructure. Ideally, that would require minimal changes to the existing code. Then, we could use captured data to understand behavior of the system, reason about it and plan next steps .


Improving Azure Ecosystem

Improving Azure Ecosystem The objective of this week was to deal with short-term concerns of SkuVault in 10 hours: setting up infrastructure for log aggregation and performance analysis. I think, we did a good job at that. Log Aggregation and Analysis First of all, we wanted to aggregate existing logs from Windows Azure roles pushing them to a central location for the analysis in real-time. One of the best tools for the job is Elastic Search with Kibana. It took just an hour to have them installed and configured from scratch on Linux. That's how Kibana would look like, once filled with data and configured: It feels like Windows Azure Virtual Machines aren't friendly with Linux. Operations took noticeable more time, compared to the other providers. VM in question wasn't that small - 4 cores and 7GB of RAM. Currently SkuVault uses NLog for logging in .NET. I implemented a small console application demonstrating how to redirect that NLog output to our new log server. Slava's team will need to import these changes into Windows Azure Roles. Mozilla Heka daemon is bundled into the console to act as a gateway. It picks changes in file logs, forwarding them to a Heka daemon on logging server via TCP. Remote daemon pushes them into the Elastic Search immediately. In this setup, log shipping has little impact on existing .NET code: C# code uses same code to write things to NLog which outputs them to files. Heka daemon doesn't use a lot of CPU to parse logs and ship them downstream. If any part of logging infrastucture goes down, logging to files will continue, untill servers come back online. Working with Mozilla Heka was a pure pleasure . It is well documented, works out of the box, easily bridging together Windows and Linux machines and has different plugins available. Dedicated Server Later Slav suggested to move this to a dedicated server. He got one with 2TB of local storage, 2 Intel X5650 processors @ 2.67GHz and 24GB RAM. If memory serves me well, this dedicated server is 2 times less expensive than 7GB RAM Virtual Machine at Windows Azure with 4 virtual cores. While rebuilding Elastic Search + Kibana + Nginx setup on a dedicated server I recorded a screencast covering all steps. This should make process reproducible internally, just in case. Capturing Performance Metrics With Heka insfrastructure in place, adding detailed perofmance metrics capabilities to .NET code was just a matter of: installing Graphite and Carbon to store, aggregate and graph metrics in real-time; including .NET library capable of speaking StatsD into the console sample; telling Heka daemons to aggregate StatsD metrics and push them to our logging backend every 5 seconds. Once I discovered a docker image for Graphite + Carbon, installing (and reinstalling from scratch) turned out to be a 10 minute task. In this setup .NET Library ( StatsdClient , available on Nuget) talks to the local Heka daemon via UDP. This means: There is a little performance overhead from measuring performance, since UDP is "fire-and-forget" protocol. Unlike TCP, we don't wait for the recipient to ACK back. If Heka daemon ever slows down or dies, this will not have any impact on production code, it will just continue broadcasting into the void. Introducing detailed performance metrics will require some changes to the existing .NET codebase, since metrics aren't captured at the moment. Fortunately, we can start with a limited amount of metrics first: count number and frequency of each command passed (by type and total); measure execution time of each command handler; capture similar statistics for events and event handlers; record byte sizes and access times for different view types. measure performance of event storage. Carbon retention policies are : store time series at 10 second resolution for 8 days, 1 minute resolution for 31 day, 10 minutes for 1 year and 1 hour resolution for 5 years. That should be enough for SkuVault team to have more than 100000 different time series . They will not need to worry about the disk space either, since carbon will aggregate and cleanup whisper files automatically . Here's how Graphite dashboards can look like, once populated with data for engineers: Ideally, SkuVault will eventually get a dedicated monitor somewhere in the office, displaying the most important metrics and graphs in real-time. Summary Final setup is rather simple. We just use existing software that is known to work well out-of-the-box. Software runs on a single dedicated Ubuntu server and can be setup from scratch in hours. If any part of this devops infrastructure fails, it will have no impact on the production code. Existing .NET codebase doesn't change much. We only need to adjust the infrastructure to report data. Here's how the setup looks like from the high-level perspective: At the beginning of the next week I'll need to tighten the security of this new server and setup backups. Meanwhile SkuVault developers will probably start tweaking Windows Azure Roles to ship logs and statistics for analysis. Afterwards, we'll switch to longer-term perspective: improving the performance in hot spots (where it hurts the most), simplifying the design .


DevOps and Event-Driven Design

DevOps and Event-Driven Design My third week with SkuVault was 10 hours long. It involved both infrastructure concerns and software design. First, I had to secure our new devops server: encrypt TCP traffic with logs and performance stats; serve web dashboards via HTTPS; add authentication to the web UI. Afterwards we started planning evolution of software design at SkuVault. TLS for Heka Fortunately, encryption is easy to setup with Heka. It works same way on Windows and on Linux. All Heka communications now go through TLS connection, using self-signed client-server certificates. That secures both log messages and performance stats. Golang is picky about certificates. You need to generate certificates with TLS Client extension. Heka would refuse to connect otherwise. HTTPS SkuVault already had a wild-card certificate bought for their domain. Securing the web UI was just a metter of copying certificate chains and keys to devops PC, then telling Nginx to accept only HTTPS connections. Nginx is an open-source reverse proxy server with capabilities for load balancing and web caching. It is performant and battle-tested. Besides, it runs on a operating system that can install updates without rebooting. We also perform HTTP redirect to HTTPS for all insecure connections. Along with that I added basic HTTP authentication to devops server. It is good enough, since we encrypt all traffic anyway. Event-Driven Design During the week we started planning design improvements with Slav. We had in mind following requirements in mind: Better scalability to sustain business growth. Higher availability. Simplify existing code. Make all changes in small steps (no big rewrites). Leverage strong points of Windows Azure. Big software rewrite can be appealing, yet it is a risky option. We all know stories where a long-term rewrite project turned out to be a complete waste of time and money. The purpose of a software design process is to find ways to gradually improve existing software. At the current moment SkuVault consists from multiple modules representing bounded contexts in the domain and running as individual azure worker roles . Modules are event-sourced, their interchange contracts shaped by CQRS principles. Each module has its own private event store and a set of projected views, which are publicly accessible. Being the author of Lokad.CQRS library, I'm guilty of creating a .NET framework which introduces design concepts leading to complex code in the long term. They say, there is a special place in hell for people like me. We can make this design more steady and decoupled by shifting from integration by sending commands and querying view state to integrate modules via events and crafted APIs . This shift would mean that: We hide internal state of the servers from the outside world, making it easier to evolve them and change the implementation. We can replace the majority of the commands in existing design with synchronous API calls. This would also simplify client code. APIs would act out as a natural Anti-Corruption Layer, besides we need to start introducing public APIs to the sytem anyway. Events are less fragile than commands (if crafted properly), they work well in pub/sub integration. Asynchronous code can promise better scalability and performance, yet in practice it introduces complexity, unpredictable execution and weird bugs that can drive other developers crazy. This can be especially dangerous, if they own weapons and know home address of the author. It is better to play safe and limit asynchrony to well-defined places, while keeping the rest of the code simple and synchronous. Ideally, we would introduce all these changes in small steps. Longest iteration would be 2 weeks. One way to start this process is to introduce a small piece of software, responsible for gathering events from all the private event stores and publishing them to all interested subscribers. This software will need to be simple, fast and highly available. At the end of the week I had 3 possible implementation options for this piece in mind: Fourth option was Apache Kafka. This highly scalable distributed commit log would be a good fit for a project with Java stack. In a .NET company it would look like an Englishman in New York. FoundationDB + Golang event storage. Implementation on top of Azure Blob Storage (essentially, next version of Lokad.CQRS event store). Thin layer on top of Azure Table Storage. Each option has its own set of trade-offs. We would need to pick one that fits SkuVault project the best. That is what the next week going to be about.