Evolving the Stack and learning Nanomsg

Last week with HappyPancake was my first full-time week with the team. Time flew fast and left me wishing for more.

I explored nanomsg (glorified sockets) and how it is used in golang. nanomsg is going to be our communication layer between components within the app, hence understanding its usage patterns was important. It was extremely exciting to pair with Pieter on go programming exercises (and also picking up some Sublime/Linux tricks along the way).

While developing my first golang+nanomsg+FDB prototype, I was quite proud of the first code that was written. It was relatively robust, simple and somewhat performant. During a few next days I realised that it was actually an overcomplicated and under performing piece of software. Pieter shared his process about structuring and expressing ideas in golang. Tomas explained how to make that code brutally simple and yet more performant. That was awesome! Previously it would take me years or months before I could arrive to that breath-taking understanding of how stupid I were. With this team everything happens so much faster. Love it.

During the week we set ourselves a goal of building a system which could return uncached reads from the front (reverse proxy) within 25ms under the load of 50000 HTTP requests per second , while degrading gracefully under increased load. This, assuming that we run 3 relatively small app servers, 2 reverse proxies and FDB cluster of 5 nodes. Obviously, throwing more hardware to the system, should scale it out. It is nice to be working on a system, where latency is one of the design constraints and having fun is another one.

When I had to leave on Friday evening, Tomas and Pieter were discussing process of getting rid of state in services by pushing it all the way to reverse-proxy (using Lua on nginx to take care of HTTP connections, while preserving true stateless asynchrony over nanomsg on the inside). This approach has a synergy with building resilient system of small systems (aka micro-services architecture) communicating over asynchronous events and continuously evolving (versioning, A/B testing and continuous delivery are among the goals of the goals).

Apparently, during the course of the evening, they refined the approach to make it even more simple and robust. I can't wait to see what this idea has turned into.

By the way, FoundationDB just hit 2.0 version. They published a golang client library within that release, making use of FDB – a breeze. By the way, as Pieter and Tomas reported, upgrading our test cluster to v2.0 took 4 minutes. ETCD was also bumped to 0.3.0.

Published: February 08, 2014.

Next post in HappyPancake story: Designing for Throughput and Low Latency

🤗 Check out my newsletter! It is about building products with ChatGPT and LLMs: latest news, technical insights and my journey. Check out it out