Retrospective of 2015 Q1

Introduction

First quarter of 2015 is behind. We managed to achieve some impressive results at SkuVault. This will allow us to offer new features and a better product to customers in the second quarter of 2015.

Catalyst 2015 - ChannelAdvisor’s annual industry conference, one of the most important events for e-commerce.

Last week Catalyst 2015 took place, where SkuVault was a bronze sponsor and had an exhibition booth.

This conference was a big milestone for us: we had an opportunity to demo a completely new feature there - Interactive WavePicking.

Wave picking in warehouse management systems is a process of grouping incoming orders into small batches (picklists), which are assigned to workers for assembly. This way a worker can assemble multiple orders in parallel, increasing the overall throughput.

Interactive WavePicking in SkuVault provides ability to create these picklists from pending sales, using various criteria. When this picklist is assigned to a worker, the worker will get guidance from SkuVault, telling where to go, which item to pick and which compartment in the cart to put it into. This guidance is location-aware, thus it minimizes walking around the warehouse and increases the overall throughput of the assembly process.

It was hard to implement this feature in SkuVault v1 design (based on CQRS/ES architecture), so a different approach to design and development was required. We aimed at two goals simultaneously:

• come up with a new software design, tailored for SkuVault project to use in further development and constructed to serve customers better in the upcoming years;

• implement a new feature using that design.

Implementation of the feature was an indicator of success or failure of the new design. If development of the feature works out, then the new design is good enough to move forward with. If it doesn’t, then we fail early and save the effort of a major rewrite and a cost of a big failure.

I think, we did well. SkuVault team demonstrated WavePicking at Catalyst 2015, and soon this feature will be available to all customers. Benefits of the new design will come forth in the next quarters of 2015.

Major thanks go to:

• Slav Ivanyuk - for managing the process and doing all super-helpful code reviews
• Chris Witt - for picking up ReactJS/Flux and doing all the work in UIv2
• Jason Henson - for helping with the demo setup, testing and deployment

Feature design was driven by SkuVault experts, who had to do a very complex job of identifying the feature essence and figuring out which parts would need to make it into MVP: Andy Eastes, Slav Ivanyuk and Danny Shaw.

In order to prepare path for Interactive WavePicking we had to:

• Design and implement a new infrastructure for the next UI version and backend with APIs.
• Provide a way to integrate v2 with v1 in various environments: production, where we have more than 20GBs of events coming at high throughput, QA with auto-deployable backend and demo with in-memory backend hosted inside IIS process.
• Figure out an approach for gradual migration of all existing features from v1 to v2.
• Figure out a way to improve testability, scalability in new design, while making it easier to develop and learn.

So here is what we achieved so far.

New Infrastructure

Caveat: this retrospective covers only a fraction of SkuVault's life that I was involved in.

If we forget about integration with v1 code, the new infrastructure is quite simple.

Backend is a collection of .NET modules with an API (JSON over HTTP, provided by NancyFx). Each module can subscribe to events and publish events (async, batched). Modules behaviour can be captured in use cases, which are then applied to verify the correctness of the implementation.

This .NET implementation is very similar to reference implementation in golang, which is an open source reference implementation based on the design insights we came up at Happy Pancake. I did almost a 1:1 conversion of golang code to C#.

Middleware to pass events between modules and also from v1 to v2 is MessageVault (available on github). It is a simple Kafka-inspired event bus, which uses Windows Azure to maintain a highly-available transaction log. Reads are served by Windows Azure storage (via a .NET client library), while writes are handled by a cluster of worker roles (master election is done via Azure blob locks).

Front-end in v2 is an absolute pleasure to work with. Previous version is a single project based on ASP.NET MVC with AngularJS and Lokad.CQRS client libraries. New version is a collection of single-page web applications (one web app per major feature), which are statically compiled into JS and CSS bundles. These apps are stateless (web server simply serves the content) and get all data directly from backend API.

Stateless web applications are very easy to scale, especially if they are represented by statically compiled content. Besides, any web server can serve them, making development and deployment process very flexible. Just at SkuVault, we can serve UIv2 via: webpack (dev environment), Azure Web Sites (QA and PROD environments), IIS (demo environment).

Internally we chose to use ReactJS with Fluxible, since that was the stack we had arrived at during HappyPancake. This software design is backed up by the work done at Facebook and Instagram (it is always good to stand on the shoulders of giants).

Chris Witt jumped right into the development process and tackled the UI side of feature development since then. He did a very good job.

For example, the UI below allows warehouse managers to create picklists from pending sales. They can apply a dozen of filters to these sales, pick individual sales or batches, reorder sales in the picklist.

With ReactJS, we decomposed a relatively complex UI into domain-specific reusable components; FLUX architecture pattern provided a consistent way to capture event-driven UI and client-server interactions in the code. It was easy for multiple developers to work in the resulting codebase, both to continue each other’s work and to develop UI elements in parallel.

Software Design

Feature decoupling is the most important aspect in our new design. Even though wave picking is quite complex to implement (that’s why it is rarely handled properly in warehouse management), in SkuVault this complexity is isolated from the rest of the system.

WavePicking backend API is a separate event-driven module, which can be tested, deployed and scaled independently from the rest of the backend (it can also run in-process with all the rest, for the demo, development and on-premises deployments).

Similarly, Wave-picking UI is a separate web application composed from reusable ReactJS components, Flux actions and stores. This application benefits from shared elements (e.g. styles, login/logoff, UI components, build process), but it can be developed and deployed separately.

This gives us the benefit of controlled system evolution: we can take existing features and transition them to our new design one by one, minimising the risks and avoiding the total rewrite. In parallel, we can add completely new features.

Splitting the system into modules and UI features with well-defined boundaries also simplifies team management and resource allocation. It is easier to manage development of several isolated features than to coordinate development within a single tightly coupled product. It is also simpler to scale the development process, too.

Explicit separation between the UI and backend with API was an important design decision with a long-lasting impact on existing product. First, it allows us to divide (and conquer) development into two distinct contexts with very different specifics and challenges:

• backend and API development focuses on the core domain, scalability and making it very easy for UI folks to build various front-end features. That is pure .NET with low-level optimisations for performance and scalability.
• UI is going to be THE primary consumer of API, but not the only one: mobile clients and partners will be using it as well. It cares more about the User Experience, feedback, rapid development iterations and pure HTML/JS/CSS development (for the Web).

Another interesting side-effect of the design is that we get cheap UI deployments and foundation for A/B testing. One can simply copy feature UI files to a new directory on a web server, getting a different version deployment. If you point different users to different versions of a feature, you get the ability to do gradual roll-outs, per-user customizations and the grounds for A/B testing.

On the backend side, development process is enhanced by event-driven use-cases introduced to specify and verify API behavior scenarios. These use-cases improve upon existing specifications at SkuVault, making them less fragile and focused more on the public contract rather than internal implementation.

Unlike traditional unit tests, API verification with use-cases can pinpoint the problem in case of failure. As a result, this process saves development time and lends itself to “Getting Things Done” mentality.

Use-cases additionally grant us other benefits:

• API documentation can be generated automatically. It will always stay up-to-date and its quality will be better than what libraries like ServiceStack and NancyFx can auto-generate provide out-of-the-box.
• Use-cases align very well with the development process, making it easier to manage. Especially well it works with Domain-Driven Design, which SkuVault already employs.
• Sensible stress-tests can be auto-generated out of the use-cases. They allow running the system through all the scenarios from the specifications, but repeated 1000 times or more. This capability does not replace custom stress-testing scripts, but it comes for free. Build server can run stress tests on each commit, watching for performance regressions and correlating them with changes in the code.

Development Process

I think, we managed to reduce development friction in v2. UI features in v2 are incrementally recompiled on-the-fly whenever a file changes (thanks to the webpack). We also leverage webpack dev server to handle hot reload (when compilation happens in memory and changes are pushed to the browser). This speeds up web development dramatically.

UIv2 features no longer require ASP.NET MVC, so it is not bound to Visual Studio (or to Microsoft Windows itself). Developers are free to choose an environment that fits their needs.

It seems that Atom IDE and Sublime are the most frequent choices of people doing web development with ReactJS.

Tools like WebPack, ESLint, ES transpilers work from the command line and are supported by all modern IDEs.

We also observed that ReactJS simplifies UI development, especially for user interfaces with complex interactions, and when compared to the MV* designs (MVC, MVVM, MVP). That increases productivity and lowers development risks.

There obviously is some learning barrier while switching from ASP.NET MVC + AngularJS + jQuery to ReactJS, however the effort pays for itself soon.

To make UI development more productive, we use LESS preprocessor for styles (also managed by the webpack), lodash for functional helpers in JS and superagent for AJAX calls.

User interactions are all captured in vanilla JavaScript at the moment (later we might enable some ES6 features, which can be transpiled down to ES5 by webpack).

JavaScript can be tricky, to make development more reliable we run a linter (ESLint) with a rather strict set of rules. It forces all code to be written consistently, avoiding code smells and bad practices. These rules are enforced by the build server.

Real-time statistics and logs are still evolving at SkuVault. Although there are a few glitches (e.g. Hekad integration is less than perfect on Windows), we are consistently improving the experience. At the moment we have more than 80GB of searchable logs handled by ElasticSearch and visualized by Kibana. Carbon and Graphite take care of capturing and reporting dozens of stats from various cloud services running on Azure (starting from RAM/CPU consumption and down to a specific web request latency).

Statistics and logs aren’t a mission-critical piece of infrastructure, however they are extremely helpful in understanding software behaviour under production loads. SkuVault needs to scale a lot to serve new customers better, and this toolkit provides real-time insight for that (aside from helping us to debug any potential issues).

Instrumenting the existing code to write to the distributed log or to report a new stat metric is easy. API v2 comes with these capabilities from the start, since we are planning to go for aggressive scalability targets with it.

Learning process and complexity had to be factored in the new design as well. The simpler it is for developers to understand the design and become productive in it, the simpler it will be for the company to find new talent and to grow.

Here is the list of technologies which v2 aims to discard (for good):

• ASP.NET MVC and all web development in .NET
• Angular.JS with jQuery
• ServiceStack API
• Lokad.CQRS

Instead we introduce:

• ReactJS/Flux
• NancyFx

So far knowledge transfer for UIv2 development had been rather smooth within the company (thanks to the talented developers of SkuVault).

Long-term Impact

It is always nice to consider possible long-term benefits that could come either cheap or for free. With the new design we potentially get:

• Well-used and tested API, which could support various clients.
• Ability to reuse the experience and product knowledge of our web developers to build native clients for the modern mobile platforms: iOS and Android (thanks to React Native).
• Path for scaling out the system (API is scaled by modules and then partitioned by tenants; UI is stateless and can scale infinitely).
• Support for on-premises deployments of SkuVault, along with geo-affinity around the world.
• Ability to deploy system to different clouds to provide higher availability guarantees to our customers.

What's Next?

For SkuVault, scaling and stability is the primary focus for Q2 2015. We are going to take existing features one by one and migrate them to the new design, while improving test coverage and performance.

New event-driven design gains solid APIs in UI (ReactJS) and on the backend (event-driven design), which reduces coupling and fragility, leading to a simpler and smaller codebase.

In order to achieve these goals, continuous integration and build process will also have to be enhanced. We are interested in fast builds and development feedback.

Time that a developer spends waiting for his computer to run MSBuild or Resharper to finish code indexing is wasted time. Visual Studio and MSBuild are notorious for being extremely slow on large solutions.

Fast builds in v2 are already a part of the design: UIv2 doesn't have .NET in the pipeline at all (plus webpack provides continuous builds), while backend gets faster builds due to the smaller solution footprint.

Build process itself could get feedback from:

• Static analysis: ESLint and Flow in JavaScript.
• Unit-Tests: testing a single component.
• Integration tests: use-case verification, derived sanity checks and tests for interactions between the components.
• Automatic UI/UX verification via scripted user interactions (e.g. WebDriver)
• Performance tests: scenarios derived from use-cases and custom ones.
• Codebase size and complexity tracking.

Ideally, developers would get this feedback within 5-10 minutes after a commit - while they can fix issues most efficiently.

As more features migrate from v1 to v2, we will pull more developers into the new environment. The knowledge transfer facilitation has already started, it will have to be managed more explicitly in Q2.

All in all, 2015 Q2 is going to be a very interesting time at SkuVault :)

Many thanks to Slav, Andy and Ksenia for reviews.

Published: April 06, 2015.

Next post in SkuVault story: Verification of Event-Driven Systems

🤗 Check out my newsletter! It is about building products with ChatGPT and LLMs: latest news, technical insights and my journey. Check out it out