Home » HappyPancake 🌟 AI Research · Newsletter · ML Labs · About

Back to Basics

By the beginning of the last week I ported infrastructure for event-driven design (with aggregates, projections and in-memory views) from C# to golang.

However, later on it was agreed that going through the event-driven modeling is yet not the fastest and simplest route to the working code. So this code was completely discarded (we could get back to it later) and we started evaluating something even more simple - CRUD approach with CouchDB and MySQL.

FoundationDB, does not have any projection or querying capabilities at the moment. This means additional effort required to design and maintain those and might be a premature optimization at this point.

While thinking about storage constraints in our design, I've been focusing on messaging functionality for the HappyPancake. Currently we have 150000 messages going through per day with text size up to 2000 characters (served by a large MS SQL database). 20000 users are usually chatting at the same time.

Ideally, next version would make this experience more enjoyable and interactive. More messages sent == better for the business.

I focused on prototyping a simple chat, where messages and presence notifications are served to the client with long polling http requests. CouchDB and mySQL were evaluated as storage engines at this point.

Image

Pieter, at the same time, focused on the storage problem from the perspective of profiles, storing and updating them, serving through http as documents and search feeds. We discovered that our favorite http library in go "Revel" can barely serve 4k requests per second due to all the magic it provides (including templates). Bare http server of golang can serve up to uncached 17k requests (to resources with templates) per second on the same machine. So there are some trade-offs to be made.

I personally think we could stock to basic http library just fine, since Tomas is pushing extra effort to make our UX and UI extremely simple.

CouchDB is a really old document database that has nice master-master replication, support for map-reduce and query engine. It is even used to support some experiments on Large Hadron Collider. To make things even more nice, CouchDB exposes change streams per database, to which you could subscribe from the client code. API is served over HTTP, while the core is written in Erlang.

Unfortunately CouchDB didn't fit well to a simple CRUD scenario. Reason being - CouchDB is IO bound, all caching is delegated to the operating system.

mySQL was, surprisingly enough, another contender for our storage engine. It previously felt to me that this is a legacy database from the early days of Internet. However, after starting to read "High Performance mySQL", I quickly realized that this exactly is its strongest point. This database was optimized and polished by the biggest internet companies in the world. It is rock-solid for both SQL and noSQL. Performance is predictable and tooling is rich.

Yet, mySQL can't do miracles if your IO operations are limited by the virtual environments. We can have no more than ~400 operations per second on Glesys machines.

So all through the weekend I've been searching for articles on clustered messaging architectures at a large scale, trying to figure out the simplest approach that would fit two constraints:

  • Provide fast and responsive messaging UX implementation of which is capable of serving ~20000 new long polling requests per second;
  • Have clustering capabilities (multiple app servers handling the load);
  • Work with a relatively slow storage engine, using no more than 10-20 requests per second.

Fortunately for us, we can live with:

  • Occasional write failures are tolerable
  • Cached data is fine in a lot of cases
  • Systems are not mission critical

If you think about it (and sleep over it a few nights, too), these design "relaxations" allow to deal with our domain with quite some ease: we can store messages and presence notifications simply in memory (replicated for some degree of reliability) going to the disk only for batched writes and cache misses (usually loading conversations that happened quite a while ago). Amount of memory dedicated for message cache can be tuned to find the sweet spot here.

So, at this point, we don't really care about the choice of the storage engine for the purposes of messaging, presence and notifications: CouchDB, mySQL or FoundationDB. Each one would work just fine. However, I would personally prefer mySQL at this point, since it is easier to capture the domain.

Some reading

Besides that, I started reading "Programming Distributed Computing Systems" by Carlos A. Varela, which is a very foundational and intense book. Highly recommended.

Published: April 14, 2014.

Next post in HappyPancake story: Messaging - Heart of a Social Site

🤗 Check out my newsletter! It is about building products with ChatGPT and LLMs: latest news, technical insights and my journey. Check out it out