Home » HappyPancake 🌟 AI Research · Newsletter · ML Labs · About

Designing for Throughput and Low Latency

For the last week I spent most of the time pairing with Pieter, learning more about our the performance and behaviour of our anticipated stack (for the second version of HappyPancake.com). It was thoroughly interesting exercise in systems engineering.

Here is what our anticipated design looks like right now:

Design

With this design we want to have 25ms latency of HTTP reads (non-cached, 99% percentile) given the throughput of 50000 requests per second. A/B testing, feature toggling, continuous delivery and live upgrades (with ghost mode and ramp-up) included.

Here is a short summary of lessons learned within the last week:

  • Tomas is an absolute beast when it comes to crunching out small open source libraries
  • It is quite easy to publish statistics from an app and then gather them in a nice WebUI for crunching (using client library fsd to publish to local statsD app via UDP. StatsD currently pushes stats to Librato Metrics with delay of 10 seconds).
  • HTTP servers in Go are quite good, but can be a pain to extend or augment
  • Nanomsg is really nice and performant, however the documentation is lacking.
  • Profiling capabilities of Golang are absolutely stunning.
  • Spending a week developing and debugging golang apps, while benchmarking them on a Digital Ocean cluster - teaches you a thing or two about Linux environment. It is awesome.
  • Software engineering is about making theories about how your code will behave in production, then running experiments to validate these theories. You iterate and learn.
  • Pairing up with somebody is an amazing opportunity to transfer knowledge and produce better quality code (I lost track of the number of times I was stunned and humbled by the experience and insight of other team members - so much to learn). We currently use TeamViewer (best image and keyboard sharing experience) and Skype for the voice. Campfire is for group chats (and chat ops).

For the upcoming week I'll be working on pushing our stack closer to the desired performance numbers (we don't meat the goal, yet). It is an interesting exercise which forces you to learn a lot and go deep (to the point of tuning the OS).

Published: February 17, 2014.

Next post in HappyPancake story: Containers, virtualization and clusters

🤗 Check out my newsletter! It is about building products with ChatGPT and LLMs: latest news, technical insights and my journey. Check out it out