Last two weeks went fast.
I mostly focused on the backend, rewriting some of our older modules
auth) to match JSON API
requirements and covering them with use-cases. Missing edge cases were
addressed as well.
There were also a few technical features, affecting the entire codebase.
First of all, I added role-based security to the system, extending authorization to support various levels of super-users, like admins and reviewers. This required a slight adjustment of use case infrastructure.
Second, Tomas asked to get rid of all stored procedures in the code. So I simply reconfigured use case verifier to throw errors on any module that declares stored procedures. A few more hours to clean up the code, and we are guaranteed not to have this problem any more.
Third, I introduced a quick way to stress test a single module in
isolation by running a special
stress utility. This is faster than
generating Finland/Sweden events and passing them through the
Stress utility works by taking all use cases of a module and
using them to generate events over and over again. It works
automatically and provides really fast feedback.
Fourth, I went through the entire codebase, replacing backend-side
upsert statements with PSQL upsert queries, while also switching
queries to use prepared statements. This helped to resolve a lot of
concurrency issues (detected by
stress tool) and speed up the
execution by 40% on local machine and by 5-7% at the glesys
Fifth, I made all event handlers idempotent. This is needed to handle message duplication in cases of network partitions (something that might happen with RabbitMQ). The solution was to simply extend our use case verifier. After performing a normal run, it goes through all the use cases, duplicating one event at a time and making sure that all expectations still match. Then I simply went through the solution, fixing all tests that appeared to be broken. Afterwards this logic was added to the build process, to ensure that new code will stay idempotent.
Pieter spent most of his time working on the performance and stability of the codebase. He used our Finland and Sweden datasets as a tool to make the system more robust. This also involved improving our RabbitMQ bus to make it reconnect on network failures (golang client doesn't do that out of the box).
Tomas was busy with maintenance of HPC1, tweaking it and moving the entire system to new servers on Glesys. This included migration of all 3 countries, SQL DBs, media files and app servers. Lot's of work to do.
Tomas has a vision of a Single-Page Application (SPA), where we would have rich experience on the clients with the ability to support older browsers. This SPA could even be shipped as a native app to desktops and mobile phones (via something like PhoneGap), offering really nice user experience.
At the end of the last week I finally started working in this
direction. Our primary choice of technology is
rendr from AirBNB. It
is a relatively small library that allows to run the same JS code on
the server and the client, rendering HTML at either one, depending on
rendr actually involves touching multiple technologies:
handlebars.js. So I
started by picking these technologies one at a time and learning more
about them. That's what I plan to focus on during the next week.