Back from the Vacation

It is good to be back from the vacation. Not only you feel rested, but you also get to see all the cool things done by the team.

Tomas and Pieter focused on pushing forward seeding utility that takes data dumps from the production systems and transforms it into the events for the second version of HappyPancake. They moved beyond Finland dataset (the smallest one) and started seeding Sweden, which yields more than 500.000.000 events. This allowed to polish code heavily (e.g. memory and connection leaks are detected early this days) with real-world data.

I focused on cleanups and cross-cutting concerns this week. Removing member module helped in making modules more decoupled (now almost all data is denormalized by modules in their own storages).

Then, to push this even further, I physically separated modules from each other, giving each module a separate database for production and tests. This is a big win for us, since this allows to replace one big and expensive SQL server (running it currently in production) with a bunch of smaller servers, that cost less and are easier to scale.

This improvement required introduction of module specification - a data structure describing some static qualities of the module, which are known even before it is registered in the infrastructure. Such specification currently includes:

• module name (also acting as DB name, if it has one);
• module DB schema (SQL scripts to create and reset DB contents);
• use cases that describe behaviors of the module.

With such information, we can create databases while initializing environment and pass connections to these databases to modules on registration. This also allows to run use case verification in separate temporary databases, which are destroyed at the end of test run.

With improvements from Pieter our auto-generated module diagrams became even more helpful. They give additional perspective into the code, allowing to see missing or odd pieces. For example, here is diagram of the chat module:

As you can see, POST /chat/poll is marked as read, since it is implemented but not covered with any single use case. This endpoint serves data to legacy UI served directly by the back-end (to be removed). It is to be removed, hence there was no point in testing it. Red marker serves as concise reminder of that.

Same goes for member-typing and member-read-thread domain events, which are subscribed to but are never really used (in a way that is verified by use cases). This is also something that we will need to cleanup, once focus shifts back to the UI.

Next week we plan to decide on the road map for implementing our UI. Currently it is served as HTML directly by our back-end, which is not a good thing (modules get complicated). Possible options include:

• move html generation with all the resources into a separate front-end module (or a collection of them);
• kill existing UI code and reimplement everything with node.js and rendr;
• find some other alternatives.

In the longer term we want to have rich single-page application that offers smooth experience to our users (feeling almost like a native application). However, implementing it right now might be a long and risky shot. It seems to be safer to capture existing requirements in a simple User Interface (building upon the code that we already have but with better application design) and deliver that to production. We could always improve later.

Besides user interface, there also is some more back end work to do, like:

• implementing albums module (public and private photo albums);
• implementing tags and categories for the profiles;
• improving performance of the system to let it process Sweden dataset faster;
• figure out profile properties.

I personally look forward to get back to work on the front end, since it is a part of critical path (in terms of project management) to our delivery. Earlier this week I started reading the book on Single Page Web Applications only to discover that web app development these days strongly resembles development of desktop applications. Aside from the specifics of HTML, CSS, JS and various browsers, design principles and patterns are quite similar. It should be fun.

Published: August 16, 2014.

Next post in HappyPancake story: Native Performance

🤗 Check out my newsletter! It is about building products with ChatGPT and LLMs: latest news, technical insights and my journey. Check out it out