Emergent Design Faces Reality

Emergent Design Faces Reality Last two weeks were packed. We are working hard to have a limited prototype of the application ready and available for a demo in June. So far things look really good for the schedule! Collaborative design process We chat frequently with Pieter , discussing things small and big: starting from component design to a naming choice of some variable or just a weird gut feeling about some code. I found out that disagreements with Pieter are especially productive and exciting. I’m really glad that he has patience to put through with my stupid questions. Here is one example . A few days ago Pieter started working on profile functionality and began introducing there PhotoUrl fields. That immediately gave me the big shivers, since I considered this to be a misleading design. Profile service was responsible for managing and providing published user information like gender, birthday or name. Photo urls have got to be a different concern! Bleeding them into the component responsible for creating and providing profile info felt like an over-complication, compared to the other components (they are clean and focused). I tried to explain these reasons to Pieter , but that didn’t get us far. He replied that it was ok to denormalize and mock some data within the profile service, since it would help him to get the profile viewing page faster. In response I tried to suggest to create mock stubs for photo urls in a dedicated photo component. This went on for a while. Looking at the code together through ScreenHero didn’t help much either. Some progress started only when we started talking about things in terms “this gives me shivers”, while trying to understand why each other sees things differently. As it turned out, we had different perspectives on decomposition of the components. I had in mind purely vertical responsibility for the profile component, where it would have all layers of an N-layered app along with full responsibilities : creating data, persisting it locally, publishing events, providing HTTP handler for the UX. All that, while focusing on a small and coherent set of behaviors around public user profiles. At the same time Pieter was working with the UX. He was interested in a design decomposition which would give him the component that would focus only on maintaining a cache of all user-related information for the purpose of serving profile pages and providing that information to the other components. That component would have a lot of data, but it would not contain any complex business rules mostly event subscriptions and denormalized read models. Seeing this difference was a huge step. I also needed that component (e.g. when you have a news feed and need to enrich entries in it with beautiful profile photos along with name, gender and age for each user). However, since I wasn’t aware of such distinction in our domain, I actually misused a bunch of components for this purpose. While flushing out boundaries and contracts of this new profile component we also touched it’s interactions with the future components, which are not even available in the current code (e.g.: review and draft). We talked about naming, responsibilities, contracts all things except for the implementation (which would be trivial at that point). We even made explicit things like : Ok, so we don’t have draft and review components in our codebase this week, however we will pretend that they exist by manually publishing events from them in our ‘prepopulateDB’ script. Since the other components subscribe to these events, they will not even notice any change when we introduce actual implementations. And since we model events from the perspective of the consumers, they will be useful. A better and more clear design emerged through this process, things clicking into the place like pieces of a puzzle. I find this process truly astonishing : you use codebase to drive exploration of the domain and also capture a deeper insight that is obtained during that process. Emerging design is a beautiful side-effect of that process . Design constraints Such process would not be possible without the design constraints which fuel and direct creativity. Here are a few that are important in our case: Distributed development team of three people , working remotely on the same codebase in a single github repository; mentality of golang , which forces us to think in terms of tiny packages with short and clear names; requirement to have a demo version in June and a working Beta in September ; shared belief in the power of simplicity ; high performance and scalability requirements, which we must not optimize for right now (since that put us behind the schedule for the June demo). Optimize for future performance I find it particularly interesting to optimize design for future performance optimizations, while consciously writing code that is designed for short-term evolvability (and hence is hacky and slow). This forces you to think about isolating this hacky code, preparing it for future replacement and possible optimization strategies. It is almost as if that non-existent better code was written behind the lines and continuously evolved every time you touch the component or think about it. It is impossible to forget about that, since actual code is so inefficient, just like the caterpillar. After a few iterations you end up with the component that is designed: to have high evolvability in the short term to be optimized in the longer term, making a bunch of strategies available (starting from a denormalized read model up to a in-memory cache across all nodes in the cluster, invalidated by messages). Making it all real All this process is not only fun, but it also tightly tied to the real world. Tomas makes sure of that. First of all, he acts as the domain expert and the stakeholder in the product, setting constraints and priorities, sharing insight. He also works on the vision of the product from the user perspective, capturing concepts in a tangible form of HTML templates which we started merging into the codebase. These HTML templates started showing up a few ago. They made Pieter and me feel as if New Year came early this year: it is awesome to see a real product instead of hacky UI; UX easily communicates important requirements that could be missed otherwise (e.g. “gender” symbol and “is online” highlight for every author in the newsfeed entry). In the end We keep saying: "let's see how much our approach will hold before it becomes a problem", however so far it holds up pretty well. Architecture, technology and other irrelevant implementation details have changed more than once during this period (e.g.: during the last weeks we switched from FDB to CRUD with shared transactions to event-driven CRUD (no event-sourcing, though). Design still supports growth of understanding and product through these minor perturbations.


Minimal Viable Product

Minimal Viable Product This week we were pushing forward major features missing from Minimum Viable Product for the demo in June. The progress was quite good, even ahead of the schedule. I attribute that to the design we came up with for the project. Pieter focused on introducing infinite scrolling to our feeds: alerts , news and diary . These feeds are provided by the separate golang packages of the same names. They don't own any data, but rather project incoming events into a local storage (mySQL tables used as key-value storage with some indexing) and expose an HTTP handler to render all the HTML for browsing these feeds. When we ran out of the things to do for the MVP, Pieter switched to implementing draft , register and review packages. Previously we assigned future responsibilities to them and established their contracts in form of events that cross boundary of these packages. These events were mocked by populateDB script and consumed by the other packages. This allowed to refine the design multiple times even before we started coding in this version. Tomas continued acting as Santa this week, working hard on the new HTML templates for the project, while also refining some of the old ones. These templates feature responsive UI, making them ideal for devices with small screen sizes (half of our visitors use them). Later on we could adjust HTML to work nicely for the desktop apps as well. It felt really awesome for me to skin diary , alerts and news with these new templates along with my favorite chat package. This process actually granted additional design insights: we can't generate HTML of the feed items in advance, since we need to embed things like on-line status , current profile photo ; while rendering final HTML for the feeds, profile service is queried for enrichment information dozens of times per render - I had to implement a simple in-memory cache with cluster-wide eviction of invalid items (driven by the events); we no longer could use application-wide long polling feed for updating chat pages in real-time, since this feed had to contain specific HTML templates and behaviors. Long-polling buffers had to be moved to chat , rewritten and enhanced with events like user-read-message . There are still a few missing bits and pieces related to the UI of profile views and chat conversations, but these are going to be easy to implement once we have the HTML templates to fill them in. For the upcoming week I will probably be busy with implementing nav package (it serves navbar html which is reactively highlighted whenever there is some new content for the user to consume). Ideally, we'll also tackle rendering of the ads into the feeds, since this is the most valuable feature for the business. At some point next week we might start enhancing our solution with package-level event-driven tests expressed in the form of specifications. We currently have such tests implicitly (in the form of event streams generated by populateDB script), however there is value it making them explicit.


Almost Demo

Almost Demo The error of my ways We are getting closer to the demo date, which is scheduled to be next Monday, and I did a big mistake last week . Instead of thinking for the whole team I focused only on the design and backend development. It would've been better if instead I tried to go out and unload some burden from Tomas , who was swamped with the other things this week. This way we could've avoided over-delivering on the back-end while getting behind the schedule on the UI design (which is usually the most visible and tangible part of any product). I'll try to pick up more skills in HTML+CSS and UI design in the upcoming days to have skills spread more evenly across our small team. Features delivered During the last week I added continuous integration server (using Drone on a VM) and introduced a shared staging server which could be used for demo. That server also has infrastructure for capturing various metrics from our application code, aggregating them and pushing to a nice dashboard UI. I introduced nav package responsible for maintaining navigation menu at the top of the page. Some items on that menu are highlighted in real-time, as new things arrive for the user to check out. Newly registered users now have a newsfeed that is pre-filled with interesting things to check out (as determined by our matching logic). Plus, now it is possible to see photos in the profile pages, like them and send flirts. Ability to register is something Pieter delivered this week, along with draft implementation of review service. He also came up with a really nice implementation for our online service, responsible for maintaining a list of currently active users across the cluster. Retrospective At this point, we have a working pre-alpha version with core functionality either implemented or envisioned in detail. We didn't burden the code with any performance optimizations , keeping it simple and capable of fast evolution. Performance optimizations, if introduced to immature software design, could hinder or prevent its growth to maturity. Technically, the implementation is extremely simple: a single executable application written in golang with mySQL persistence and in-memory bus . It exposes HTTP endpoint serving web pages and could be switched to clustered mode (if put behind load balancer and plugged to a proper messaging platform). This implementation is more developed from the design perspective : it is decomposed into simple golang packages which are designed to work together to deliver use cases of a dating web-site . These packages tend to be small - merely ~300-400 lines of code on average, including HTML templates). Majority of these packages are full vertical slices, encapsulating implementation details starting from the storage model and up to HTML rendering details within the http handlers. Concepts within the code map to the domain model rather well . They are quite focused and simple, thanks to hours spent working over them with Pieter. However, vocabulary could benefit from a better alignment with the business model . As Tomas mentioned, we managed to drift from original domain model during the development process. That is something we could fix after the demo. These design concepts are very prominent in the contracts of packages: published events and exposed services . There are quite a few of DDD Value objects as well. Design approach still seems to hold pretty well, although we are getting close to the next strain point: some packages get too complicated and would benefit from better tests at the contract level. Something like event-driven given-when-then specifications could work really well here. Adding such tests is something I'm looking forward to after the demo as well.


Our First Demo

Our First Demo We finally had our demo last week. As it always happens in practice, nothing went according to the theory . Unexpected problems Two big problems surfaced right before the scheduled demo time. First of all, RAID on one of the production databases (HPC1) suddenly died. This required full attention of Tomas , taking him away from the demo preparations. Second, I discovered that JavaScript part of chat (which I implemented) gets horribly messed up by subsequent PJAX page jumps . Fortunately, disabling PJAX on chat navigation links solved the problem in the short term. In the longer term, I'll need to pick up more Javascript skills. Tomas already recommended me to check out Javascript: The Good Parts . Despite these issues, together with Pieter we cleaned up the HPC2 for the demo. Tomas did an awesome job presenting the product and the vision behind it, which bought us trust from the stake-holders for moving forward. They loved it. We plan to have demos on a monthly basis from this point. NoSQL in SQL During the week we decided to give a try to PostgreSQL , which seems to have a slightly better fit to our needs , than mySQL: great replication story (e.g. "HotStandby and repmgr"); mature drivers in golang (if compared to MySQL); binary protocol that does not suffer from legacy issues like MySQL API does; more polished usage experience (if compared to MySQL); there is a book on PostgreSQL High Performance , which looks as good as the one I read on MySQL. PostgreSQL also benefits from being one of the widely used databases (although it probably has fewer installs than mySQL). Replacing MySQL with PostgreSQL was a simple thing, since we use SQL storage mostly for NoSQL purposes anyway. Using SQL for NoSQL gives us the best of the two worlds: mature ecosystem, polished experience and transactions of SQL along with ease of schema-less development from NoSQL. By the end of the week I migrated almost the entire application to PostgreSQL. Design decomposition into small and focused packages (with logically isolated storage) really helped to move forward. Next week I plan to finish the migration and improve test coverage in scenarios that were proven to be tricky during this migration. So far, PostgreSQL feels more comfortable than MySQL. If this feeling proves to be wrong, we could always jump back or try something else. Being the worst on errors and panics Sometime during the week, Pieter brought up the question of using panic vs error in our code. In golang it is idiomatic when functions return a tuple of result and error: func Sqrt ( f float64 ) ( float64 , error ) { if f < 0 { return 0 , errors . New ( "math: square root of negative number" ) } You can also issue panic which would stop the ordinary low of control and start going back in the call chain until recover statement is expected or the program crashes. Since I was burned pretty badly with Exceptions in .NET while working with cloud environments at Lokad (everything is a subject to transient failure at some point, so you have to really design for failure), I tried to avoid 'panics' in golang all-together. Instead, almost every function was returning a tuple of result and an error , problems were explicitly bubbled up. This lead to a lot of unnecessary error checking and some meaningless errors that were pretty hard to trace (since errors in golang do not have a stack trace). Thankfully Tomas and Pieter patiently explained that it is OK to throw panics even in the scenarios which would later require a proper error handling with a flow control. Initially this felt like a huge meaningless hack, but eventually it all "clicked". Refactoring with this new design insight already makes the code more simple and fit the future evolution (which is required by the current stage in a life-cycle of the project). Pieter also started cleaning up the language in our codebase, making it more aligned with the reality. This is a big effort involving a lot of merge conflicts, but the results are definitely worth it. Becoming a better developer through your IDE During last weeks I invested bits of time to learn about Emacs and customize it to my needs . One of the awesome discussions with Pieter on this topic helped to realize the importance of such IDE tailoring for personal growth as a developer. As you probably know, Emacs is almost unusable for development out-of-the-box (vim, even more so). You need to tweak configuration files, pick plugins and wire them together. Most importantly, you need to make dozens of decisions on how you are going to use this contraption for the development . That's what I used to hate about Emacs before, thinking that Visual Studio with ReSharper gave me everything that a developer would ever need. I came to realize that setting up your integrated development environment from the scratch forces you to become more aware about the actual process of development . You start thinking even about such simple things as organization of files in a project and how you are going to navigate between them. Or, how you are going to refactor your project in the absence of solution-wide analysis and renaming provided by ReSharper. Such troubles affect your everyday coding process, pushing design towards greater decomposition and simplicity . Ultimately, this leads to better understanding . In the end, Pieter got so inspired by our insights that he also decided to ditch Sublime, giving a try to Vim. We are going to compare our setups and development experiences as we progress through the project. I believe, this is going to lead to even deeper insight for us.


Scala, Modular Design and RabbitMQ

Scala, Modular Design and RabbitMQ Our system is event-driven in nature . Almost everything that happens is an observation of some fact: message-sent , photo-liked , profile-visisted . These facts are generated in streams by users interacting with a system. Due to the nature of human interactions, there is little concurrency in these streams and it is ok for them to be eventually consistent. In other words: A user is likely to interact with the site through one device and a single browser page at a time. While communicating through the system, users don't see each other and don't know how fast the other party responds. If a system takes 1 second to process and deliver each interaction then probably nobody will notice. The system should feel responsive and immediately consistent (especially while viewing your writes on the profile page and chatting). These considerations are very aligned with designs based on reactive and event-driven approaches. During the last 2 weeks we played with multiple implementation ideas of that: Use replayable event streams for replicating state between modules. Use either FDB-based event storage (which we already have) or the one based on apache Kafka. Use a messaging middleware cabale of durable message delivery across the cluster with a decent failover (read as "RabbitMQ") Use a pub-sub system without a single point of failure and relaxed message delivery guarantees (read as NSQ or Nanomsg with ETCD). Each of these approaches has its own benefits and some related complexity: mental - how easy or hard is it to reason about the system; development - how much plumbing code we will have to write; operational - how easy or hard will it be to run it in production. Obviously, we are trying to find approaches which reduce complexity and allow us to focus on delivering business features . Scala Theorem While talking about Apache Kafka Tomas had an idea of switching the entire codebase to Scala and JVM . Java has a lot of big supporters and a large set of great solutions fit for us. A few days last week were dedicated to evaluation on how easy or hard would it be to drop all go code and switch to Scala / JVM. Here are the conclusions: Scala is a nice language , although builds are insanely long slow (if compared to sub-second builds in golang). Porting our core domain code to Scala is not going to be a problem, it could probably be done in a week ( code is by-product of our design process ). Devil is in the details, learning the rest of JVM stack is going to take a lot more time than that (e.g.: how do we setup zookeeper for Apache Kafka or what is the idiomatic approach to build modular web front-end with Java?). In the end, switching to Scala was ruled out of the question for now. Even though this switch has its long-term benefits, it would delay our short-term schedule too much. Not worth it. Besides, Java stack seems to introduce a lot of development friction hurting rapid development and code evolution. These are essential for us right now. RabbitMQ We also switched to RabbitMQ for our messaging needs - Pieter single-handedly coded bus implementation which plugged into our bus interface and worked out-of-the-box. Previous implementation used in-memory channels. So far RabbitMQ is used merely do push events reliably between the modules: all modules publish events to the same exchange; each module on startup can setup its own binding and routing rules to handle interesting events. Although we no longer consider using event streams for replaying events as part of the development process, we could still have a dedicated audit log. This can be done by setting up a dedicated module to persist all messages, partitioning them by user id. Modules We spent some time discussing our design with Pieter . One of the important discoveries was related to a deeper insight into Modules . Previously we talked about our system using components , services , packages interchangeably. This was partially influenced by the term micro-services which was one of the ideas behind our current design. Some confusion came from that. Instead of "micro-services architecture" at HPC we started talking about "decomposing system into focused modules which expose services" These weeks we were able to refine our terminology, starting to clarify the codebase as well: our application is composed from modules - tangible way to structure code and visual way to group design concepts into ; we align modules at the design level and modules in the code - they share the same boundaries and names; at design level modules are boxes that have associated behavior, we need them to contain complexity and decompose our design into small concepts that are easy to reason and talk about ; in the codebase our modules are represented by folders which also are treated as packages and namespaces in golang; we like to keep our modules small, focused and decoupled , this requires some discipline but speeds up development; each module has its own public contract by which it is known to the other modules; implementation details are private , they can't be coupled to and are treated as black-box; Public contract can include: published events (events are a part of domain language), public golang service interfaces and http endpoints; there also are behavioral contracts setting up expectations on how these work together; in the code each golang package is supposed to have an implementation of the following interface, that's how it is wired to the system; all module dependencies are passed into the constructor without any magic. type Module interface { // Register this module in execution context Register ( h Context ) } type Context interface { // AddAuthHttp wires a handler for authenticated context which // will be called when request is dispatched to the specified path AddAuthHttp ( path string , handler web . Handler ) // AddHttpHandler wires raw http.Handler to handle unauthenticated // requests AddHttpHandler ( path string , handler http . Handler ) // RegisterEventHandler links directly to the bus RegisterEventHandler ( h bus . NodeHandler ) // ResetData wipes and resets all storage for testing purposes ResetData ( reset func ()) } Getting the notion of modules right is extremely important for us, since it is one of the principles behind our design process. We think, structure our work and plan in terms of modules. For the upcoming week we plan to: Cleanup the codebase (one module at a time), finishing the alignment to RabbitMQ; Capture and discuss next HPC features to be implemented (summer vacations are coming and we want to prepare work so that we could continue moving forward even when the rest of the distributed team is offline, taking motorcycle classes or hiking to the top of Mount Elbrus); this will add more stand-alone modules; Start writing two-phase data transformation tooling to export data from the current version of HappyPancake and transfer it into the event-driven version; this would allow to validate the design of existing modules and stress-test the system. PS: Why Emacs is better than Vim? Over my entire life I've been searching for a sensible way to handle tasks and activities, both everyday and job-related. Tooling ranged from MS Project Server (for large projects and distributed teams) to OmniFocus (personal todo lists). Earlier this year I discovered org-mode - a set of Emacs extensions for managing notes and tasks in text files. That was the reason for switching to Emacs from sublime. Recently I caught myself managing some small tasks and notes of HPC project via org-mode as well. All hpc-related information is stored in a textual hpc.org file kept in the repository with the source code. Anybody could read or even edit this file. Emacs, of course, provides more benefits on top of that mode: ability to view and manage entries from all org-modes on my machine; capturing new tasks with a few key strokes; powerful agenda and scheduling capabilities; exports to a lot of formats; auto-completion, tags, categories, outlining, refiling, filtering etc. For example, here is overview of my agenda, filtered by hpc tasks: I think, I got Pieter thinking about giving a try to Emacs, since Vim does not have org-mode (or a decent port).


Distributing Work

Distributing Work A season of vacations starts . This week was the last time when our team was online at the same time. Tomas takes a vacation starting from the next week. Pieter is probably going to take his as soon as he gets through bike exams (wishing him the best of luck). I'll travel to Georgia next week, while working remotely and taking longer weekends. Obviously, we want to stay productive during this period and move forward on our project. There are things that usually require full consensus: important decisions about design, specific feature requirements, everything that involves multiple packages at once. Last week was spent going through these things in advance to make sure we have plenty of non-blocking work queued up for the next month. More Features We have some basic features implemented in the system so far. Software design evolved a bit to support them all while keeping things simple. At this point, if HappyPancake2 were a brand-new product, I'd recommend going live (e.g. in stealth mode) as soon as possible in order to start getting real-world feedback from the users. No amount of testers and visionaries can replace knowledge and insights coming from the real world feedback. Duty of software developers is to make his happen as soon as possible and then iterate, incorporating lessons learned. However, HappyPancake2 is special - it is already used by thousands of users , so there is already plenty of feedback. We know quite well which features are necessary, which could be discarded and which enhancements we could try next. Hence we can keep on working on this project without releasing it . Tomas has all the domain knowledge we need right now. We are planning to introduce these features next: Interests - tags that members can add to their profile, allowing other people to find them by interests (and potentially allowing us to provide better matching); blocking - allowing a member to ignore another one (removing him or her from all search results and blocking communications); online list ; abuse reports on content with admin review queues; favorite profiles . During the week Pieter focused his efforts on developing review functionality , which is one of the most important features in our system. Node.js We are planning to make a slight tech change in our stack by implementing front-end in node.js (which is something Tomas explored last week). This is a relatively small change to the existing system - http endpoints will need to return JSON instead of rendered HTML, so the cost is relatively low. Benefits are: better separation of concerns in our design; ability to use Rendr (render backbone.js apps on the client and the server). This would turn our existing code into back-end with an API, serving JSON requests and streams to node.js front-end. Such separation allows to have more flexibility in UI while introducing a much better testing to the back-end. Behavior Testing Thanks to the switch from HTML endpoints to JSON, I started introducing package behavior tests to our system last week. These tests set and verify expectations about public contracts exposed by packages. This is quite simple to do: Given a set of events and dependencies When we execute an action (usually calling a JSON endpoint) Expect certain assertions to be true. In the longer term I hope to convert these tests to self-documenting expectations (like I did in my previous .NET projects). Ability to have up-to-date documentation of the code that is expressed in human-readable language can be a powerful thing for keeping project stake-holders involved in the project. This means better feedback and faster iterations. Code looks like this in golang: func ( x * context ) Test_given_nancy_flirts_bob_when_GET_bobs_alerts ( c * C ) { s : = run_nancy_flirts_bob ( x ) r : = x . GetJson ( s . bobId , "/alerts" ) c . Assert ( r . Code , Equals , http . StatusOK ) var m model r . Unmarhal ( & m ) c . Check ( m . Title , Equals , "Alerts" ) c . Check ( m . HasMore , Equals , false ) c . Assert ( m . Items , HasLen , 1 ) i1 : = m . Items [ 0 ] c . Check ( i1 . Member . Nickname , Equals , "nancy" ) c . Check ( i1 . Unread , Equals , true ) c . Check ( i1 . Member . IsOnline , Equals , true ) // since we have allOnline c . Check ( x . Service . AnyUnread ( s . bobId ), Equals , false ) } where nancy flirts bob scenario is a simple code setting up preconditions on the system: func run_nancy_flirts_bob ( x * context ) ( info * nancy_flirts_bob ) { info = & nancy_flirts_bob { hpc . NewId (), hpc . NewId ()} x . Dispatch ( hpc . NewRegistrationApproved ( hpc . NewId (), info . bobId , "bob" , hpc . Male , hpc . NewBirthday ( time . Now () . AddDate ( - 23 , 0 , 0 )), "email" , hpc . NoPortraitMale )) x . Dispatch ( hpc . NewRegistrationApproved ( hpc . NewId (), info . nancyId , "nancy" , hpc . Female , hpc . NewBirthday ( time . Now () . AddDate ( - 22 , 0 , 0 )), "email" , hpc . NoPortraitFemale )) x . Dispatch ( & hpc . FlirtSent { hpc . NewId (), info . nancyId , info . bobId }) return } The Truth is Born in Argument I can't be grateful enough to Pieter who has enough patience to go with me through the design discussions in cases when we disagree about something. Talking things through with him is one of the reasons why our design stays simple, clear and capable of future evolution. Design Game A lot of our work resembles some sort of puzzle, where we have to do 3 things: find names and words that let us communicate better (we are a distributed team from different countries); discover ways to break down large problem into small coherent parts (team is too small to be able to tackle huge problems); decide on optimal order in which these parts could be handled (our time is limited and has to be applied to the areas where it will make the biggest impact for the project). The hardest part is deciding which things have to be done right now and which can be deferred till some point in the future. In some cases implementing a feature without all the necessary data at hand can be a waste of time, in other cases, this could lead to a deeper insight required to move forward. We try to optimize implementation chain a lot - bringing most rewarding and easy features ("low hanging fruites") and depreriotizing ones that are less beneficial for the project. That is an ongoing process required for applying our limited time most efficiently. For example, previously we pretended to store photos in our system. We simply passed around urls pointing to photos from the original version of HappyPancake. That was a good decision (defer functionality as long as possible), but time came to implement it. During last weeks Pieter pushed new media module and spent some time integrating it with the other our services. This brought new insights to how we are going pass around this information through events. We also know how we could host and scale such such module in production (deploy to multiple nodes and rsync between them). Anything related to performance is another example of things we deferred. "Big Data" So far our development intentionally focused on software design while deferring any potential performance optimizations. Now it is time to start learning about the actual numbers and real-world usage. At the end of the week I went back to Visual Studio to start writing an extractor utility. This tool merely connects to the original database and saves some data to a compact binary representation (compressed stream of protobuf entities). Then, I started working on the golang code which will scan through that data, producing a stream of events which could be passed to our development project. It is recommended to use such two-step data processing (dump data store to intermediary format and then iterate on data dumps) whenever you are working with large datasets coming from a live system. This decouples the process from production systems, reducing the impact and allowing to have faster iterations. We started working on Finland database, which is one of our smaller installations, yet the already is a bit of data to process. For example, there are more than 1200000 messages, taking 230MB in compressed binary form, 11MB of member data and 2MB of user flirts. Sweden is 100-50 times larger than that. This might seem like a lot of data, however it is not so. Our entire Sweden dataset, if encoded properly, could fit on a single smart-phone and be processed by it. It's just large. However, since we didn't introduce many performance considerations into our design yet (aside from keeping it scalable), some tuning will be necessary. I haven't worked with large datasets for more than half a year, so I'm really looking forward to get back in this field. Real-time reactive nature of the data makes this even more interesting and exciting.


Smarter Development

Smarter Development Shorter Feedback Loop We have a continuous integration server responsible for running tests on code pushed to the repository. It is more diligent than humans and always runs all tests. However, in order to see build failures one had to visit a build page (which didn't happen frequently). Our builds were broken most of the time. I tried to fix that by plugging build server directly to our main chat. All failures and successes are reported immediately. Build stays green most of the time. Working with Finland I spent time trying different strategies to populate our system with Finland dataset. This population happens by generating events from raw data dump and dispatching them to our system. Currently we generate only a subset of events, but that already is more than 1000000 of them, sent at once. If we can handle that, then we stand a chance against Sweden dataset. I focused on news module, which has one of the most complicated and time-consuming denormalization logic: each member has his own newsfeed; each member has an interest in some people (e.g. in females with age between 25 and 30 and living in city X); newsfeed is populated with events coming from other members which are interesting to this member; new members by default will have an empty newsfeed, we need to back-fill it with some recent events from interesting members; if member blocks another member, then events from the blocked member will no longer show up in a newsfeed, existing events have to be removed. My initial implementation of news module was handling events at an astonishing speed of 2-10 events per second. I spend multiple days learning inner workings of our stack and looking for simple ways to improve the performance. StatsD and EXPLAIN ANALYSE from PostgreSQL helped a lot to reach speed of 200-400 events per second. Solution was: push all event denormalization to PostgreSQL server (fewer roundtrips); handle each event within an explicit transaction (no need to commit between steps within the event handling); rewrite queries till they are fast. So far the performance is quite good so we don't need to bother too much about pushing it further so far. Adding more features is the most important thing now. Control is important It is really important for members of our dating web site to know that they are in the control. They should be easily able to block out any unwanted communications. That's why we have block feature - ability to put another member into an ignore list, effectively filtering him out from all news feeds, conversations and any other lists. I started working on that interesting feature only to realize that it has a lot of implications. Each other module would react differently to the fact that a user is being blocked. We need to somehow keep a track of all these requirements. Preferably it will be not in a form of the document, since documents get stale and outdated really fast (keeping them fresh requires time and concentration which could also be spent developing new features). Ideally, these requirements could also be verified automatically. Improving Use Cases I invested some time to improve our module BDD tests, transforming them into proper use-cases. These use-cases: are still written in golang; are executed with unit tests (and provide detailed explanation in case of failure); can be printed out as readable stories; can be printed as a high-level overview of the system. Of course, these stories aren't readable by absolutely everybody. That's not the point. Their purpose is: Give sense of accomplishment for developers encouraging them to write tested code (me and Pieter); align tests with expectations from the system (help to make sure that we are testing what is important); provide a quick up-to-date documentation of the API and scenarios for other developers who would be working with the system (Tomas); express behaviors of the system in a way that is not tied to any language (e.g.: Tomas will not need to dive into the golang code in order to consume API from node.js). The best part is that these nice stories are generated automatically from the code. No extra effort is required (apart from writing a small rendering logic in golang while riding on a bus in Adjaria). With this approach it becomes simpler to have high-level overview of what is already done in the system and what has to be done. Simply list names of all passing use cases per module and you have that kind of overview. Other interesting transformations are also possible (i.e.: dependency graphs between modules, event causality patterns etc). They all provide additional insight into the domain model, allowing to have greater insight into the code we write and maintain its integrity . It is quite possible that we could completely discard this code once we hit the production. Need to maintain the integrity of domain model will be replaced by different forces by then. My plans for the upcoming days are to keep covering our existing functionality with these use-cases and adding new functionality.


Delivering Features and Tests

Delivering Features and Tests Last week was quite productive and exciting. Introduction of use cases into our development cycle worked out really well , helping to deliver tangible features in the absence of tangible UI to target (node.js front-end development is paused till Tomas gets back from the vacation). These use cases so far: serve as high-level behavior tests aligned with the domain language (compact and non-fragile); drive development towards a better event-driven design; produce nice human-readable documentation, as a side-effect; provide really fast feedback cycle. Actually, these use cases are the design . We can probably take them and rewrite the entire system in a completely different language in 2 weeks. And we can do that without loosing any features or scalability options. However, these nice benefits are not as important as the fact the we spent last week developing new features and improving code coverage, while really enjoying the process . Pieter jumped right into the game, picking up on use case development and extending the framework to support edge-cases which I missed (e.g.: testing file uploads or using real HTTP server to allow inspecting raw HTTP requests with WireShark). He already covered drafts module with use cases. Pieter also invested time last week cleaning things across the code-base. I spent the last week both adding use cases (coverage of chat , alerts , news , poll ), fixing bugs revealed by them and adding proper handling of member-blocked and member-unblocked across the system. As of now, we have 33 use cases covering 15 API calls. We know this number exactly, because of a tiny little helper command summary which can print out information about all use cases. With that command (and the power of BASH), one can easily answer questions like: How many use cases are in the system? Which URIs are not covered by the tests? Which events are published or consumed by module X? What are the dependencies between the modules? Which events are not covered by any use case? This self-building knowledge about the system is another reason which makes writing use cases so rewarding. I also took a bite and tweaked our build server to include commit summaries in chat messages posted to Slack. This way, it is easier to observe team progress without going to git repository. This also encourages frequent pushes, since drone picks up only the latest commit in a push. This week I'm going to continue delivering features, covering them with more use cases and also working on the ETL code to extract data from HPCv1 into our new system.


Data, Use Cases And New Module

Data, Use Cases And New Module Last week I was simply developing in a pleasing and steady way: alerts - clean JSON API and more use cases to verify its behavior; diary - clean JSON API, more use cases and support of member blocking; chat - more use cases; like - clean JSON API and use cases; favorite - implemented full module , including JSON API, major use-cases, etl and seeding. Data Extraction from v1 I spent some more quality time with .NET/C# last week, adding more data types to our script responsible for graceful data extraction from HPCv1 databases into compact binary representation (GZIPped stream of length-prefixed binary-encoded objects). This representation works very well for further data processing even at the scale of terabytes. So far I extracted data from all of the largest tables in Finland database, writing event generation code for all matching modules and smoke-testing them on glesys. HPCv2 handles that data pretty well, although RabbitMQ gets a little strained while handling 1500000 messages and copying them into a dozen queues. We'll probably need to optimize our message routing strategy a little here. Fortunately, we can simply reuse data from our wonderful use case suite. I'll be on a vacation next week, so we tried to reproduce process of data retrieval (from binary dumps) and event seeding on the machine of Pieter . It worked without issues. Use Cases We are slowly falling in love with use case approach in the codebase of HPCv2. Writing them is a pleasure, and they actually speed up and simplify the development. At the moment of writing we have 50 of them, verifying different behaviors of JSON API for the frontend that Tomas will be working on when he gets back. I added ability to render use cases into a dependency graph, helping to see development results from a different perspective. Visual representation allows your brain to understand code from a different perspective, making it easier to spot new dependencies or gaps in the code. It is easier to communicate, too. For example, while developing favorite module from scratch, its graph looked like this: Later that day, when the module was complete and covered with 7 use cases, it looked like this: This graph is auto-generated from the code via this process: Load a specific module (or all of them), inspecting registrations in the process. Inspect all use cases for input events, HTTP requests and output events. We can do that because each use case is simply a data structure, describing: GIVEN previous events, WHEN we call API endpoint, THEN expect API result and 0 or more events. Print resulting data model in a dot file format for graphviz program to render. Of course, if some dependency is not covered by a use case, then it is not visible on such graph . I consider this to be a good feature, since it encourages me to cover all important cases. Of course, it is possible to graph all modules and their dependencies. That map would be useful for spotting some loose ends or old dependencies. Pieter was busy cleaning up the overall codebase, working on the implementation of draft , review and getting read of some obsolete logic. Next week I will be completely on a vacation, spending time around the Elbrus mountain. If there are any free periods of time (e.g. long bus rides), I'd love to clean up the profile module, adding a clean JSON API to it. Living Documentation Tomas is coming back from the vacation that week. He'll probably get back to the front-end development on top of our new JSON API. When he does, he can see living documentation for out current system. First, run ./r story to see all use cases. Second, run ./r summary to see grep-able metadata about the system (mostly derived from the use cases). Third, run ./r graph | dot -Tpng > graph.png to create dot file for the system and then feed it to grapviz . Of course, each output can be altered with other programs like grep to filter, group and aggregate information in interesting ways. This kind of documentation always stays up-to-date and does not need any effort to maintain.


Back from the Vacation

Back from the Vacation It is good to be back from the vacation. Not only you feel rested, but you also get to see all the cool things done by the team. Tomas and Pieter focused on pushing forward seeding utility that takes data dumps from the production systems and transforms it into the events for the second version of HappyPancake. They moved beyond Finland dataset (the smallest one) and started seeding Sweden, which yields more than 500.000.000 events. This allowed to polish code heavily (e.g. memory and connection leaks are detected early this days) with real-world data. I focused on cleanups and cross-cutting concerns this week. Removing member module helped in making modules more decoupled (now almost all data is denormalized by modules in their own storages). Then, to push this even further, I physically separated modules from each other, giving each module a separate database for production and tests. This is a big win for us, since this allows to replace one big and expensive SQL server (running it currently in production) with a bunch of smaller servers, that cost less and are easier to scale. This improvement required introduction of module specification - a data structure describing some static qualities of the module, which are known even before it is registered in the infrastructure. Such specification currently includes: module name (also acting as DB name, if it has one); module DB schema (SQL scripts to create and reset DB contents); use cases that describe behaviors of the module. With such information, we can create databases while initializing environment and pass connections to these databases to modules on registration. This also allows to run use case verification in separate temporary databases, which are destroyed at the end of test run. With improvements from Pieter our auto-generated module diagrams became even more helpful. They give additional perspective into the code, allowing to see missing or odd pieces. For example, here is diagram of the chat module: As you can see, POST /chat/poll is marked as read, since it is implemented but not covered with any single use case. This endpoint serves data to legacy UI served directly by the back-end (to be removed). It is to be removed, hence there was no point in testing it. Red marker serves as concise reminder of that. Same goes for member-typing and member-read-thread domain events, which are subscribed to but are never really used (in a way that is verified by use cases). This is also something that we will need to cleanup, once focus shifts back to the UI. Next week we plan to decide on the road map for implementing our UI. Currently it is served as HTML directly by our back-end, which is not a good thing (modules get complicated). Possible options include: move html generation with all the resources into a separate front-end module (or a collection of them); kill existing UI code and reimplement everything with node.js and rendr; find some other alternatives. In the longer term we want to have rich single-page application that offers smooth experience to our users (feeling almost like a native application). However, implementing it right now might be a long and risky shot. It seems to be safer to capture existing requirements in a simple User Interface (building upon the code that we already have but with better application design) and deliver that to production. We could always improve later. Besides user interface, there also is some more back end work to do, like: implementing albums module (public and private photo albums); implementing tags and categories for the profiles; improving performance of the system to let it process Sweden dataset faster; figure out profile properties. I personally look forward to get back to work on the front end, since it is a part of critical path (in terms of project management) to our delivery. Earlier this week I started reading the book on Single Page Web Applications only to discover that web app development these days strongly resembles development of desktop applications. Aside from the specifics of HTML, CSS, JS and various browsers, design principles and patterns are quite similar. It should be fun.


Native Performance

Native Performance The highlight of the last week was switch from pure Go driver for PostgreSQL to a wrapper around native library jgallagher/go-libpq . Pure Go driver seemed like a nice thing in the beginning (no need to bother with native dependencies or header files), however it turned out to bee too immature. Confusing error reporting for connection issues was one of the biggest problems for us. Thanks to database/sql abstraction in golang, I was able to switch to native driver in one day. Cleanups mostly involved our handling of uuid type. Wrapper around native PostgreSQL client library offers similar performance and wonderful error reporting . There were a few issues with our code while trying to run Finland, they were mostly solved by replacing client-side transactional upsert statements with PostgreSQL upsert statements using Common Table Expressions. One rare race condition in chat module was fixed with a tiny smoke tester and a fine-grained row lock. We were able to run entire Finland dataset (except visits) in 2m53s , which is comparable to pure go driver. Sweden takes more time, but it actually runs, without any big issues. As long as our system can handle burst of events coming from the entire dataset, running the usual production load should never be a problem. Aside from the peformance, I spent time implementing albums module (permissions on private photo album), adding proper implementation of the profile (public view of approved user information with any permissions applied). Then, I also went across the codebase cleaning up modules and adding missing use cases. Functionality of these modules is completely covered now: news , alerts , chat , block , likes , poll , flirt , profile , diary , albums , favorite . draft , auth , review lack use cases, so that is one thing we will try to fix this week. These use cases become even more important to use, since we check multiple things while verifying them: verify that all expectations are met; ensure that there are no connection leaks in code being tested; check that all events present in given are listed in module declaration (it is used to route only the needed events to module queues in RabbitMQ) I also removed all legacy HTML stuff from our back-end code. This separation improved it a lot. Hopefully, today we'll decide on our front-end strategy and start implementing this last missing bit. Use cases help to close feedback loop, but there is nothing like a tangible product. Pieter and Tomas were busy this week dealing with performance optimizations of the code and performing these time consuming runs on the production data. I think, they have quite a lot to share about the inner workings of RabbitMQ and PostgreSQL on Glesys datacenter (VMWare virtualization).


Features, Use Cases, Rendr

Features, Use Cases, Rendr Last two weeks went fast. I mostly focused on the backend, rewriting some of our older modules ( register , draft , review and auth ) to match JSON API requirements and covering them with use-cases. Missing edge cases were addressed as well. There were also a few technical features, affecting the entire codebase. First of all, I added role-based security to the system, extending authorization to support various levels of super-users, like admins and reviewers. This required a slight adjustment of use case infrastructure. Second, Tomas asked to get rid of all stored procedures in the code. So I simply reconfigured use case verifier to throw errors on any module that declares stored procedures . A few more hours to clean up the code, and we are guaranteed not to have this problem any more. Third, I introduced a quick way to stress test a single module in isolation by running a special stress utility. This is faster than generating Finland/Sweden events and passing them through the system. Stress utility works by taking all use cases of a module and using them to generate events over and over again. It works automatically and provides really fast feedback. Fourth, I went through the entire codebase, replacing backend-side upsert statements with PSQL upsert queries , while also switching queries to use prepared statements . This helped to resolve a lot of concurrency issues (detected by stress tool) and speed up the execution by 40% on local machine and by 5-7% at the glesys environment. Fifth, I made all event handlers idempotent . This is needed to handle message duplication in cases of network partitions (something that might happen with RabbitMQ). The solution was to simply extend our use case verifier. After performing a normal run, it goes through all the use cases, duplicating one event at a time and making sure that all expectations still match. Then I simply went through the solution, fixing all tests that appeared to be broken. Afterwards this logic was added to the build process, to ensure that new code will stay idempotent. In this case we handle only idempotency, since this was a "low hanging fruit". Out-of-order messages is something we don't deal with, yet. Pieter spent most of his time working on the performance and stability of the codebase . He used our Finland and Sweden datasets as a tool to make the system more robust. This also involved improving our RabbitMQ bus to make it reconnect on network failures (golang client doesn't do that out of the box). Tomas was busy with maintenance of HPC1 , tweaking it and moving the entire system to new servers on Glesys . This included migration of all 3 countries, SQL DBs, media files and app servers. Lot's of work to do. User Interface Tomas has a vision of a Single-Page Application (SPA), where we would have rich experience on the clients with the ability to support older browsers. This SPA could even be shipped as a native app to desktops and mobile phones (via something like PhoneGap), offering really nice user experience. At the end of the last week I finally started working in this direction. Our primary choice of technology is rendr from AirBNB. It is a relatively small library that allows to run the same JS code on the server and the client, rendering HTML at either one, depending on the state. Working with rendr actually involves touching multiple technologies: node.js , expresso stack , backbone.js , handlebars.js . So I started by picking these technologies one at a time and learning more about them. That's what I plan to focus on during the next week.


Getting started with Node.js, Lazojs

Getting started with Node.js, Lazojs I spent the last week learning Javascript, Node.js and some of the ecosystem . JavaScript : The Good Parts and Node.js: Up and Running were a good introduction into the field. It is surprising how diverse and rich node.js ecosystem is . I think, this could be attributed to the language itself. JavaScript is everywhere, and it is extremely flexible. This creates a lot of diversity for both good and evil. Besides, it is really easy for anybody to publish node.js packages. There are thousands of them, including some real gems. I'm gradually falling in love with node.js for front-end development purposes . There just is too much goodness already implemented for you (if compared to .NET or golang environments, for example). Golang is still quite good for backends , though. My impression of node.js is clearly biased , since I was focusing only on the technologies required for our front-end. Here the list of things I've been learning about: Backbone.js - lightweight model-view library ( great introduction ). Handlebars.js - nice templating library . Grunt - JavaScript task runner . Underscore.js - functional programming helpers . Express - popular web application framework . for node.js (used in rendr) with a lot of middleware available. hapi - another server framework for Node.js (powering Lazo.js). Require.js - an implementation of Asynchronous Module Definition (AMD) , used by Lazo.js to load modules on client and server. Rx.js - reactive extensions for JavaScript , coming from the Microsoft world. Rendrjs - framework to render Backbone.js apps on the client and the server , using Node.js. Lazojs - client-server web framework for building decomposed single-page applications with optimized first page load and SEO compliance. Rendr project turned out to be lacking for our needs. However, Tomas discovered more polished alternative to it: LazoJS from Walmart Labs . By the end of the week we switched focus to LazoJS and started evaluating it. Documentation is awesome in this project. I'll start this week by implementing an infinite scrolling for our news feed in lazo.js (with all the SPA benefits), then will continue adding other front-end features. Tomas was extremely busy with the migration work during the week . Moving the most popular dating web site in a country has a lot of challenges. This process is done for HPC1 for now. This week Tomas plans to enrich HPC1 with additional data capture capabilities . This way we'll be able to serve better news for our members in HPC2. Pieter was away on his well-deserved vacation, driving a new bike all around the Europe. He is with us this week, getting back to making our HPC middleware battle-ready . Welcome back, buddy!


Web development, the good parts

Web development, the good parts Last week was dedicated to front-end development. We made some nice progress there, thanks to LazoJS framework and Jason Strimpel himself. This framework, supported by the entire Node.JS ecosystem, makes development quite simple even for somebody so unexperienced in UI like me. Of course, Node.js has some drawbacks. For example, its performance is worse than the performance of golang code , pushed to the extreme. There also are a lot of bad things in JavaScript development itself : callback hell, memory leaks, relatively poor debugging experience, low quality of some public packages. However, if you consciously stick with the good parts and avoid the bad ones , the experience can be quite rewarding. First of all, since our front-end code is rather simple and stateless, we could always solve some performance problems by getting another machine . This would be much cheaper, than the time of a developer. Virtual Machines are available starting from a few dollars per month. That is probably cheaper than a few hours of development. Second, Node.js has diverse ecosystem of things that help you develop web applications. If you pick a few good ones and stick with them, your life becomes simpler, speeding up the development. Here are some that were adopted last week: ESLint - pluggable linting utility that checks for common styling and syntax issues on JavaScript code. We adopted the style of Felix Geisendörfer for our project. ESLint is a part of our build no w. If there is an extra comma at the end of array or some trailing space - it will fail the entire build. JSFmt - utility for formatting, searching, and rewriting JavaScript . Whenever a file is saved to disk, we automatically reformat it. Reflex - runs a command or restarts a server whenever sources are changed . This reduces development friction a lot. Kudos go to Pieter for finding this one. Flycheck - syntax checking extension for Emacs that runs ESLint in the background. node-inspector - Node.js debugger based on Blink Developer Tools. We use it via the node-debug wrapper. web-mode - HTML template editing mode for Emacs that understands embedded CSS, Handlebars and Javascript. Lazo.js - client-server web framework for building responsive applications. It deserves a separate podcast. BackboneJS, UnderscoreJS and jQuery come in bundled. Besides, there is PhantomJS (headless WebKit scriptable with JS) which could be used to verify UI use-cases. If we make these use-cases runnable against BrowserStack , then we have all operating systems and browsers covered for us. Feature-wise, we implemented infinite scrolling for our feeds, started working on real-time notifications (highlighting menu items which contain new content). Pieter was also implementing the profile UI. Tomas will join HPC2 development effort later this week. Last week he deployed a new like feature to HPC1, to start capturing information about the most popular content (we could use it to populate feeds in HPC2). It was used almost a hundred thousand times within the first 24 hours!


Reactive User Experience

Reactive User Experience Last week I focused on implementing some requirements related to reactive user experience. That is basically about things that make the users feel as if they were working with a native desktop/mobile application. First, menu items in the web interface should highlight as soon as a user gets new information that he hasn't viewed yet (news items, alerts, chat messages). That's much like Facebook experience. Implementing this relatively simple feature was hard in our previous iteration over the User Interface, since we had no notion of events and view life-cycle. Switching to components that communicate through events makes everything simpler in our current iteration. It is currently implemented via 2 components: Poller component, which runs on the client-side. While the user is logged-in, it publishes events about feed changes as soon as they are retrieved from the server (delivered via the long-polling, would probably switch to sockets.io later). Navbar component, which renders status in HTML during server-side rendering (happens on first page load or in case where we don't have push-state) and then subscribes to feed change events on the client-side. Whenever status changes, CSS class of a DOM element is changed accordingly. In case, when user has JavaScript and push-state enabled, there will be only one instance of navbar and poller running in his browser tab, no matter how many pages he visits. Second, menu highlights should persist when you travel between the pages . There shouldn't be even flickering. This is solved by client-side rendering over Backbone. Third, news feed and alerts feed should reactively display "You've got new messages" whenever there are new things to read. That is very similar to what Twitter does. Once, I had the poller component, it was trivial to implement this feature. Feed views simply subscribe to poller events on rendering. Whenever there are new items - we show message box. Whenever a view is removed from the DOM, we detach all event listeners and avoid memory leaks. After getting this simple reactivity straight, I started working on the chat. That's what I'll be working on this week. Pieter and Tomas were working mainly on the Profile view, focusing on component reusability (e.g. like and flirt buttons). In parallel, Tomas was working with like feature that was recently introduced into the existing version of HPC1 across all the countries.


Feeds, Chat, Online list and CSS

Feeds, Chat, Online list and CSS Last week I focused on the user interface , rebuilding features from the first UI demo. This time the code was much better, though: all benefits of LazoJS framework are built-in; less memory leaks and unexpected behavior (last UI prototype stuck there), for example all views explicitly unsubscribe event handlers before being unloaded from the DOM; more reactivity and interactivity across the system. Long story short, I implemented/polished: diary, news, alerts feeds are now more polished , have same endless scroll behavior; message inbox and conversation views are now implemented and are working reactively ; all feed-based views have consistent ad display (which works for items retrieved from the server on the first page load and items added reactively); there is a new online list view, which displays last active members on the site; it acts as a roulette, never showing the same member twice within the session; the list is updated with a new photo once every 10 seconds. During the week Tomas introduced me to the articles on CSS Architecture . They describe how to design CSS classes and align them with HTML code in order to create web applications that are scalable (people-wise) and maintainable. Suddenly, HTML side started making a lot more sense to me - there is a lot of design process involved there. At the end of the week I went through all of my client-side code, refactoring HTML, CSS and JavaScript to a better design (previously there was no design at all). This way, Tomas will have less work to do, when he starts applying nice UI look to our system. Pieter and Tomas focused on implementing proper component reuse at Lazo level in the profile page . For example, like button is a reusable component which could appear multiple times on a single page. Their plan is to try and apply this approach to a collection views this week (e.g. news or diary feed). This week I'm working from Italy (Bologna). I have a bunch of outstanding tickets regarding UI features I implemented , so that's what I'll focus on. We are getting close to another demo and a release in Norway.


To ReactJS and Facebook Flux

To ReactJS and Facebook Flux During the last 3 weeks we evolved our understanding of frontend stack . LazoJS worked quite well in the cases that I dealt with. These were mostly simple feeds like news , inbox or conversation . However, it created some complexity and overhead for Tomas and Pieter as they explored encapsulation of UI elements and their reuse. In short, in LazoJS it was quite complicated to create and apply a reusable component (e.g. avatar or like button). While I focused on finishing features in LazoJS, Tomas and Pieter explored something else - ReactJS framework and Facebook Flux architecture . In the end they discovered that Facebook Flux was a better fit for us in the long term (at least, better than the other previously explored options like custom PJAX, AirBnB Rendr and WalmartLabs LazoJS). So we made a switch and then spent the last 2 weeks porting our frontend to isomorphic flux components from Yahoo. The investment of time was very worth it, I think. What are ReactJS, Flux and Yahoo components? ReactJS is a battle-tested javascript library for building user interfaces. It allows you to decompose your UI into very simple reusable components, which can then be composed together, rendered using Virtual DOM and then synced back to the HTML. ReactJS is very fast and focuses on unidirectional data flow, which makes code much easier to reason about. ReactJS is maintained by Facebook and also used by Instagram, Yahoo and Khan Academy. Flux is a Facebook architecture for building User Interfaces out of predefined building blocks using one-way data flow. It looks very much like CQRS and is known to scale well in large organizations. Flux architecture works very well with ReactJS. Here is what a developer from Facebook says: From first-hand experience, I can say that React+Flux has scaled well to 8+ developers over 800+ JS files and ~60k lines of code in a large single page app here at Facebook. At HappyPancake, we really liked how Flux helps us to decompose UI into components and also solve event cascade hell which is often present in apps based on BackboneJS or AngularJS. Yahoo Flux components are a set of components open-sourced by Yahoo. These components implement various building blocks of Flux architecture for building isomorphic web applications: dispatcher, router and store. Usually, single-page web applications render all HTML on the client-side. This means that the very first page load can take a few seconds or more: we need to load HTML, parse it, load all required JavaScript libraries, fetch the data for the current route, render it into HTML and then update the DOM. This doesn't work well for older browsers as well. Isomorphic web applications work around that by rendering first page on the server, so that user gets a user interface immediately. Javascript will be loaded later, turning the application into a usual Single-page application, with all its smoothness and responsiveness. Why did we switch from to Flux? The goal of the project is long-term. We don't simply want to build a new version of HappyPancake. Instead, we need to build a software that can continue evolving and scaling since the moment we release it. To achieve that, we have to iterate quickly through the possible development options as early as possible, while the cost of error is minimal. HappyPancake is a unique project due to the number of factors, so we need to evaluate and pick the combination of options (both technical and design-related) which would be good enough for accomplishing long-term goals of the business. Switch from LazoJS to Flux is an investment of time. In my estimates, it would pay off in 2-4 weeks already because of: codebase is polished and supported by very large companies; superior reuse of components; code is simpler to reason about; CQRS architecture is something that we know very well how to test. There are a few technical challenges that we would need to solve in the upcoming days as well: existing flux demos don't pay very much attention to solution structure ; and we got very spoiled by the benefits provided by decomposition of the domain into focused modules. We already have some ideas, but would need to agree upon the conventions; tooling for Flux is great, however there is no equivalent of gofmt/jsfmt for JSX files. We'll either need to wait for one or tweak existing solutions; naming of some building blocks in Flux is very weird . For example, even Facebook developers frequently use interchangeably terms actions and action creators (equivalents of command handlers and command outcomes). We need to get used to that. Despite these small drawbacks, I think that Flux is so far the best way to organize development of web user interfaces, where you need to manage the complexity and long-term development effort. At very least, Flux works so in our specific case of rewriting an already existing (and most popular in Sweden) dating web site with a team of 3 developers distributed around the Europe. So far we migrated to Flux a profile view, news feed (which is being renamed to discover , to match the purpose) and the chat. We need to do a proper 3-way team sync on the work accomplished before fanning out and working on the rest of the features. That's our plan for today and the rest of the week.


Project Complete

Project Complete Over the last days we revisited the entire stack, cleaning up the code and improving the design until we were happy with it. We've polished these functionally distinct pieces: Registration flow. Login/logout and authentication details. Discovery feed - equivalent of a facebook newsfeed, with infinite scrolling. Chat , delivering messages and presence notifications reactively. Reactive navbar . Profile view with public and private photos, liking and ability to favorite. That covers only functionality polished from front to the end. We also have backend implemented for the rest of the system (drafts, favorites, online lists, reviewing, alerts, history, flirts etc). Facebook Flux Architecture was a pleasure to work . It is more simple and elegant than any other web UI stack I've worked with before. Composition and reuse of UI components is superb. Flux methodology would be quite familiar to anyone who has dealt with CQRS/ES concepts before. It is very easy to reason about and leads to clean and intuitive codebases. We used isomorphic Flux components from Yahoo to speed up our development (the ones from the next version of Yahoo Mail), basing the initial frontend layout on the chat sample from Yahoo. Bill Fisher (Facebook) and Michael Ridgway (Yahoo) were very kind and extremely helpful. They helped to speed up our development a lot. Design Summary To summarize. Our tech stack and software design went through many iterations. We optimized for: simplicity of the code and design; performance and scalability; ability to evolve, grow and perform A/B testing; ability to develop and maintain with very few people. Of course, there are multiple implementation details that are not present in this bigger picture, starting from design process for frontend components and up to setting up ctags support in IDE. But these are the details. Resulting design looks quite simple: This combination of technologies and tools is optimized for a growing social web site with 600000 members (50% of them - active) and planned expansion to multiple countries. All that - with a limited development effort and resources.


Epilogue

Epilogue It is a pleasure to work with such a codebase. Someone else will have to continue that, though. This project is complete : we've got the foundation up and running. It was a great experience to work and learn in a distirbuted HPC team: Tomas has a unique ability to explore new technologies, learn fast and adapt. His decisions are always based on vast experience and large number of facts, while ultimately driving towards simplicity. Pieter is an awesome team-mate to work with. He learns and accepts changes fast. It was a pleasure to discover and refine project domain towards a deeper insight and simplicity with him. The journey was fun and exciting. We should do it again sometime. I'm definitely interested to get my hands dirty with a similar project.


Retrospective

Retrospective Our project at HappyPancake completed this week. We delivered a simple and scalable foundation for the next version of largest free dating web site in Sweden (with presence in Norway and Finland). Journey Below is the short map of that journey. It lists technologies and approaches that we evaluated for this project. Yellow regions highlight items which made their way into the final design. Project Deliverables Project deliverables included: Deployable full-stack application with major features implemented. Domain model captured in software design (back-end and front-end) and a set of declarative use-cases (acting as living documentation, system outline and behavior test suite). Configured environments for development and continuous integration (docker container). Strategies for further evolution and scaling of the system. Code for migrating existing production deployments to a new version of software. Final high-level architecture is simple to reason about and scale . It was designed to be that way. Logically the entire solution consists from backend modules (represented by golang packages) and elements of Facebook Flux architecture (grouped together in namespaces by naming conventions). Such structure helps to maintain the project as it grows in size and complexity. This design also helps to scale the deployment to handle higher loads. We can scale backend by: moving individual modules to bigger servers; launching multiple instances of a single module; switching storage of an individual module to clustered solution, moving it to bigger servers or even pushing to the cloud. We can scale frontend by simply launching new instances behind the load balancer. Solution structure also provides a natural way to split the work between the developers. Given the established published language (contracts of API and events), we can also bring in more developers, assigning them to work on individual backend modules or frontend namespaces. Lessons Learned Lessons learned : Picking the right technology can reduce the development effort. In my next project I'll try to focus even more on divide and conquer approach - isolate a small part first and then evolve it, limiting the amount of work in progress. It is crucial to establish feedback loop as early as possible, involving all stake-holders. This builds trust and helps to avoid surprises.