Erlang Basics

Erlang Basics Erlang is a general-purpose programming language and a runtime environment. It has built-in support for concurrency, distribution and fault-tolerance. These are my notes taken while learning about its basics and reading through Learn you some Erlang for Great Good! by Fred Hebert . It is available online. The document covers only the very introduction to Erlang. OTP and application design are outside of the scope. Table of Contents 1. Overview 2. Erlang basics 2.1. Getting started 2.2. Immutable variables 2.3. Atoms 2.4. Boolean algebra 2.5. Comparison and Equality 2.6. Tuples 2.7. Lists 2.8. List comprehensions 2.9. Binary Data 2.10. Binary Operations 2.11. Binary strings 2.12. Binary comprehensions 3. Modules 3.1. Defining 3.2. Compiling 3.3. Referencing 3.4. Macros 3.5. Metadata 4. Functions 4.1. Pattern matching 4.2. Guards 4.3. Guard patterns (ifs) 4.4. Case … if 4.5. Choosing between guards, if and case 4.6. Bound and unbound variables 5. Types 5.1. Type conversions 5.2. Type-test BIFs 6. Recursion 6.1. Basics 6.2. Tail recursion 6.3. More recursive functions 7. Higher order functions 7.1. Basics 7.2. Anonymous functions 7.3. More function helpers 8. Exceptions 8.1. Error 8.2. Internal Exits 8.3. Throws 8.4. Catching exceptions 8.5. Catch construct 9. Common data structures 9.1. Records 9.2. Key/Value Stores 9.3. Other data structures 10. Concurrency 10.1. Spawning Processes 10.2. Message passing 10.3. Receiving messages 10.4. Better design 10.5. Timeouts 10.6. Selective receive 11. Errors and Processes 11.1. Terminating a process 11.2. Links 11.3. Traps 11.4. Monitors 11.5. Naming processes 1 Overview Whenever somebody starts talking about building robust distributed systems in .NET, Java, Go etc, then that is the response they can get ( from HN ): …basically re-creating Erlang from the ground up, except half broken and not learning from Erlang's 20-30 year history Joe Armstrong's thesis is said to capture these lessons, but it is 295 page doctorate thesis. A shorter version is in one of his talks . The other sources include Erlang Factory talks, books Learn You Some Erlang, Joe's, etc. They say, that these lessons can be summarized as: CPUs aren't getting faster these days, so we are stuck building distributed and highly concurrent systems. These systems are better to be fault-tolerant. Especially we don't want a single corner-case scenario (triggered by an unlucky user) to bring down system for millions of the other connected clients. This tolerance can be achieved by composing systems from components, isolated to prevent failure from spreading. Isolation can be achieved via: A runtime that prevents memory sharing (OS Processes or Erlang VM, except that Erlang allows to have millions of procs). Proving that memory can't be shared (Rust's compiler can do this, making sure that you will not have data races. This is really exciting). Running in a container, VM or different PC. Keep in mind: a service running on a single machine isn't fault-tolerant. Isolated components need to communicate (they can't share memory), this can be achieved by sending messages. Such approach is called "actor model". Besides Erlang, there are many libraries and frameworks supporting it: Orleans, Akka etc Erlang is special, since it has: very light-weight processes which could be restarted very quickly (300 words each and take microseconds to create); ability for processes to monitor the other processes; communication only via messages which are copied, passed asynchronously and could easily go to another node. In addition to processes and actors, Erlang also comes with: Functional programming approach: no side effects, immutable data, explicit state updates. This makes code more explicit. Open Telecom Platform is a collection of middleware, libraries and tools (including an application server). It could be used to build, monitor and distribute these components. Monitoring and debugging capabilities. You could connect to Erlang VM node to inspect, trace, debug or even update code. All that without stopping a system. Decades worth of experience of building such systems. For example, Erlang is used heavily by Motorola, Ericsson, T-Mobile, Amazon, Yahoo, Facebook etc. Elixir is Erlang-esque language with improved code organization capabilities ( source ): Compile-time macros (which work on AST) Pipeline operator Polymorphism via protocols Mix tool (to like npm for node.js) It still runs on Erlang VM How hard is it to start developing Erlang? 2 Erlang basics 2.1 Getting started Install On OSX brew install erlang . Launch REPL erl . Abort Ctrl+G then q or q(). 2.2 Immutable variables All variables are immutable and named PascalCase . They are assigned for the first time using the pattern matching operator = . > One = 1. > Two = One + 1. 2.3 Atoms Atoms are constant literals, with a backing 4 or 8 byte integer. They are written in lowercase or with_understore . They are used in pattern matching. 2.4 Boolean algebra Erlang has the usual boolean comparison operators and , or , xor , not . These operators always evaluate both sides of the equation, while andalso and orelse are shortcut operators. 2.5 Comparison and Equality Equality operators are freaky: =:= and =/= . They test for precise equality. Comparison operators are slightly lighter: == and /= , they are more relaxed. For example: > 1 =:= 1.0. False > 1 == 1.0. True The other comparison operators are just slightly weird: < , > , =< and >= . 2.6 Tuples Tuples group multiple items together: {point,10,5} . You can also destructure a tuple via pattern matching to an unbound variable: > Point = {point,10,5}. {point,10,5} > {point, X , Y }= Point . {point,10,5} > X . 10 Tuples could be nested easily: {point,{10,5}} . 2.7 Lists Lists in erlang are constructed like [E1,E2,...E3] and could have different elements inside: [1,2,{point,{3,4}},4.5] . There is a catch. Strings in erlang are also lists of numbers. If all numbers in a list could represent a letter, then erlang will print it as string. To glue lists together use ++ and to split -- : > [1,2,3] ++ [4,5]. [1,2,3,4,5]. > [1,2,3] -- [1,2]. [3] These operations are right-associative, meaning that they are done from right to left: > [1,2,3]--[1,2]--[3]. [3] You can use [Head|Tail] pattern to compose lists (adding head is fast in erlang) and also to descructure them, where | is called a cons. In fact, any list can be constructed with [E1|[E2|[E3]]] . Note, that we can construct an improper list [2|3] , which will throw errors. 2.8 List comprehensions List comprehensions allow to generate and modify lists in a conscise way: ~[ Expr || Gen1 , Gen2 ...., Cond1 , Cond2 ...]~. For example: > [{ X , Y }|| X <-[1,2], Y <-[1,2], X == Y ]. [{1,2},{2,2}] We can also use pattern matching and existing lists: > Weather = [{ufa,sun},{moscow,rain},{odessa,fog},{spb,rain}]. [{ufa,sun},{moscow,rain},{odessa,fog},{spb,rain}] > Rainy = [ City || { City ,rain} <- Weather ]. [moscow, spb] 2.9 Binary Data To express binary data we can use: hexadecimal notation: 16#AACCEE . bit syntax: <<16#AACCEE:24>> (put these bytes into 24 bit space). For example: % declare a bunch of bytes > Bytes = <<13,234,12,34,41,1,151>>. % Take first byte and treat the rest as binary > << First :8, Rest /binary>> = Bytes . In general, binary segment could be described as: Value Value:Size Value/TypeSpecifier Value:Size/TypeSpecifier Where Size is in bits, if TypeSpecifier is not provided. The latter could be a hyphen-separated list of: Type: integer (default), float, binary, bitstring, bits, utf8, utf16, utf32, bytes (synonym for binary), bits (synonym for bitstring). Sign: signed and unsigned (default), matters only for integer . Endianness: big (default), little and native (from the current CPU). This applies for integer, utf16, utf32 or float. Unit: this is written unit:integer , where the value must be between 1 and 256. It is used for field alignment. Given that, it is trivial to parse TCP segment: << SourcePort :16, DestinationPort :16, AckNumber :32, DataOffset :4, _Reserved :4, Flags :8, WindowSize :16, CheckSum : 16, UrgentPointer :16, Payload /binary >> = SomeBinary . 2.10 Binary Operations Erlang has a common set of operators: bsr and bsl - bit shift left/right. band , bor , bxor , bnot 2.11 Binary strings Binary strings are a bolted abstraction on top of lists: <<"Some string">> . It is hard to pattern-match them, so they are mainly used for text storage. 2.12 Binary comprehensions You can deal with binaries using a special form of comprehension syntax. % Let us define some bytes > Pixels = <<123,34,21,45,102,32,65,61,62>>. <<123,34,21,45,102,32,65,61,62>> % Read them as a sequence of RGBS > RGBs = [ { R , G , B } || << R :8, G :8, B :8>> <= Pixels ]. [{123,34,21},{45,102,32},{65,61,62}] % convert these RGBs back to bits > Bits = << << R :8, G :8, B :8>> || { R , G , B } <- RGBs >>. <<123,34,21,45,102,32,65,61,62>> 3 Modules 3.1 Defining Erlang modules are represented as files with a bunch of attributes in -name(Attribute) form. The only required attribute is name: -module(useless) , it should match to the file name without the extension. -module (useless). -export ([ add/2 , multiply/2 ]). add ( A , B ) -> A + B . multiply ( A , B ) -> A * B . 3.2 Compiling In order to compile a module: in command line: erlc flags module.erl ; in shell or a module: compile:file(module.erl) ; in shell: c(useless). ; in Emacs buffer: C-c C-k (it will also load the module in REPL ). 3.3 Referencing We call functions from the other modules as module:function(args) . In order to importn the namespace: -import ( Module , [ Function1 / Arity , ..., FunctionN / Arity ]). 3.4 Macros Macros in Erlang are similar to compiler directives. They are defined as -define(MACRO,SomeValue). (where SomeValue is True if skipped), and could be referenced in the code as ?MACRO . We also have some predefined macros: ?MODULE - current module as an atom ?FILE - current file name as a string ?LINE - current line as an integer Macros can be tested: ifdef , else and endif . For example, you could provide a debugging macro: -ifdef ( DEBUGMODULE ). -define ( DEBUG ( S ), io : format ( "dbg: " ++ S )). -else . -define ( DEBUG ( S ), ok). -endif . ? DEBUG ( "entering some function" ) If DEBUGMODULE macro is defined, then we will have debugging, otherwise we will have an ok atom. 3.5 Metadata Compiler will capture all module metadata and make them available in module_info/0 function, for example useless:module_info() . 4 Functions 4.1 Pattern matching Erlang provides pattern matching in the form of multiple function clauses separated by ; . function ( X ) -> Expression ; function ( Y ) -> Expression ; function ( _ ) -> Expression . We can match on lists in the patterns: head ([ Head | _ ]) -> Head ; head ([]) -> []. We can do even fancier stuff, by destructuring in function head: add ({ Date = { YYYY , MM , DD }, Increment = {days, Days }}) -> { YYYY , MM , DD + Days }. 4.2 Guards Guards are additional that can go in function head to make it more specific. The syntax is like this: % constrain by Expr1 (must be true) function (args) when Expr1 -> Expression . % Expr1 and Expr2 must be true function (args) when Expr1 , Expr2 -> Expression . % Expr1 or Expr2 must be true function (args) when Expr1 ; Expr2 -> Expression . You can also use andalso and orelse inside guards. They are similar, but can be nested, however shortcut operators can fail (normal guard can proceed even if the first argument fails with exception). Guards will not accept user-defined functions! This is done to guarantee that we don't get any side effects there. 4.3 Guard patterns (ifs) Guard patterns are defined with if inside functions, they share the syntax. They must have catch-all clause, though: some ( A ) -> if A =:= 1 -> one; true -> something_else end . Generally, if patterns were written to get benefit if case statement (below) but without the need to write full syntax. 4.4 Case … if Case allows to have normal pattern matching inside a function. case Arg of : Pattern [ Guards ] -> Expression ; Pattern [ Guards ] -> Expression ; _ -> Expression end . In essence, this expression could be rewritten as fun ( Pattern ) [ Guards ] -> Expression ; fun ( Pattern ) [ Guards ] -> Expression ; fun ( _ ) -> Expression . 4.5 Choosing between guards, if and case Just use whichever makes the code more simple. Performance differences are negligible. 4.6 Bound and unbound variables Bound and unbound variables behave differently when we do pattern matching: unbound - attaches value to them; bound - error, unless new value is the same as the old one. 5 Types Erlang has a dynamic type system . This is aligned with a general belief that a failure in one subsystem shouldn't bring the others down. Erlang worries more about keeping the systems running, that trying to have error-free code. It is easier to do hot code reloading in a system with dynamic typing. Erlang is also strongly typed, performing type checks at runtime. For example 1 + lama would throw an error. 5.1 Type conversions Type conversions in Erlang are implemented with BIFs in erlang module: > erlang : list_to_integer ( "54" ). 54 > erlang : integer_to_list (42). "42" > erlang : atom_to_list (atom). "atom" There are many more conversions of this type. 5.2 Type-test BIFs Type-test BIFs are special BIFs that can be used in guard clauses: is_binary , is_atom etc. They help to write declarative code: Func( Arg ) when is_atom ( Arg ) -> atom; Func ( Arg ) when is_binary ( Arg ) -> binary; ... 6 Recursion Erlang has functions that can call themselves - recursive functions. That is the looping construct (aside from list comprehensions) that exists in the language. 6.1 Basics It is usually advised to start writing a recursive function from a base case (a well known scenario, when recursion terminates). Here is an example of function that calculates length of the list: len ([]) -> 0; len ([ _ ]) -> 1; len ([ _ | Tail ]) -> 1+ len ( Tail ). 6.2 Tail recursion Tail recursion aims to eleminate stacking of recursive operations as they happen by reducing them as they happen. Erlang could optimize tail calls. Let's rewrite our function to transform it into a tail recursion: tail_len ( N ) -> tail_len (0, N ). tail_len ( Acc ,[]) -> Acc ; tail_len ( Acc ,[ _ ]) -> Acc +1; tail_len ( Acc ,[ _ , Tail ]) -> tail_len ( Acc +1, Tail ). If function is calling itself in a tail position (last expression to be evaluated is the function itelf), then Erlang VM could avoid storing current stack frame. This is called tail call optimization (a specific case of last call optimization ). Such optimizations make tail recursions useful and alvoid wasting a lot of memory. Even if tail call optimization doesn't work, large per-process stack of Erlang VM could help to handle some scenarios. 6.3 More recursive functions Just to get into the habit. repeat/2 function: repeat (0, _ ) -> []; repeat ( N , X ) when N > 0 -> [ X | repeat ( N -1, X )]. and tail-recursive function: tail_repeat ( N , X ) -> tail_repeat ([], N , X ). tail_repeat ( L , 0, _ ) -> L ; tail_repeat ( L , N , X ) when N > 0 -> tail_repeat ([ X | L ], N -1, X ). Tail-recursive reverse function (note, that there is a BIF lists:reverse/1 ): reverse ( L ) -> tail_reverse ([], L ). tail_reverse ( Acc , []) -> Acc ; tail_reverse ( Acc , [ H | T ]) -> tail_reverse ([ H | Acc ], T ). Tail-recursive sublist function: head ( L , N ) -> reverse ( head ([], L , N )). head ( Acc , _ , 0) -> Acc ; head ( Acc , [], _ ) -> Acc ; head ( Acc , [ H | T ], N ) -> head ([ H | Acc ], T , N -1). 7 Higher order functions Higher order function is a function which can accept reference to another function as an argument. 7.1 Basics Let's define a map/2 function: map ( _ , []) -> []; map ( F , [ H | T ]) -> [ F ( H )| map ( F , T )]. If we put it into a useless module and also add inc(X)->X+1 , then: > useless : map ( fun useless:/1, [1,2,3]). [2,3,4] 7.2 Anonymous functions Always declaring functions can be boring, hence the inline version. > Fn = fun ( A ) -> A end . #Fun<erl_eval.6.90072148> > Fn (1). 1 In general an anonymous function can be declared as: fun ( Arg1 ) -> Expr1 , Expr2 , ... Expr N ; ( Args2 ) -> Expr1 , Expr2 , ... Expr N ; ( Args3 ) -> Expr1 , Expr2 , ... Expr N end In erlang anonymous functions inherit the scope that they have been declared in. We can have closures (capturing some variables that were a part of the scope). In the shell: > Var = 1. 1. > Closure = fun () -> Var end . #Fun.... > Closure (). 1 Assigning function to a variable Fx = fun useless: is_odd/2 7.3 More function helpers Just a bunch of common helpers: lists:reverse/1 ; lists:map/2 ( Select in LINQ); lists:filter/2 ( Where in LINQ); lists:foldl/3 ( Aggregate , starting left); lists:foldr/3 ; all/2 and any/2 ; dropwhile/2 and takewhile/2 ; partition/2 - generates two lists, where predicate matches and not; flatten/1 - select many. 8 Exceptions There are 3 types of exceptions in erlang: errors , exits and throws . 8.1 Error Errors can be created by erlang:error(Reason) . They will abort execution of the function and return a stack trace with all the arguments. 8.2 Internal Exits These are called with exit/1 . They don't have a stack trace and are generally used to pass "last breath" information between the processes. 8.3 Throws Throws (created by throw/1 ) are used to control the excecution flow. They can also be used for non-local returns in deep recursion (e.g. deep function would throw exception for a top-level function to catch and return a default value to the user). 8.4 Catching exceptions Exceptions can be handled with try...of...catch block: try Expression of SuccessPattern -> Expression ; SuccessPattern -> Expression catch TypeOfError : ExceptionPattern -> Expression ; TypeOfError : ExceptionPattern -> Expression end . where TypeOfError can be: error , throw or exit , defaulting to throw if skipped. There also is a special catch-all pattern: _:_ , which will handle any exception type. Expression between try...of could be a function or just a whole bunch of expressions: Expr1, Expr2... ExprN . This section is called protected . It is also possible to have finally block, which can't retrun any value but would be used for its side effects (e.g. closing a file). It is called after in erlang. The protected section can't be tail-recursive, since the VM would keep a reference, in case an exception shows up. Code between catch..of isn't protected and could be tail-recursive, unless after block is specified. 8.5 Catch construct This is a weird one. Keyword catch could be used alone to capture all either exception or a good result out of an expression: catch Expression . 9 Common data structures 9.1 Records Records are a hack that were added to the language later. They provide a syntactic sugar on top of the ordinary typles. 9.1.1 Defining Records Records are defined as module attributes: -module (record). -compile (export_all). -record ( creature , { name, color, life=1, attack=0, abilities=[] }). This is a creature with some fields and default values. first_creature () -> # creature { name= "Zombie" , color=black, attack=1, abilities=[drain_life] }. If we try running the code, tuple would be visible: > record:first_creature(). {creature,"Zombie",black,1,1,[drain_life]} Ok, these are the raw tuples. Let's load record definitions into the shell by rr(Module) : > rr(record). [creature] > record:first_creature(). #creature{name = "Zombie",color = black,life = 1,attack = 1, abilities = [drain_life]} There are a few erlang shell helpers to deal with records: rd(Name, Definition) - define records in the shell, as if this were a module attribute; rf() , rf(Name) and rf([Names]) - flush all records, specific one or a list; rl() , rl(Name) and rl([Names]) - print shell records in a copy-pasteable way. 9.1.2 Reading values You can read values from a record either via pattern matching (as if they were a tuple) or via a helping syntax (which is like an object cast): > Zombie#creature.name. "Zombie" Things could get pretty messy, if we have nested records (each one has to be cast separately). On the bright side, we get a nice and short pattern matching (without the need to write full tuple declaration): damage (# creature {life= Life , name= Name }, Damage ) when Damage >= Life -> "Die " ++ Name ; damage (# creature {color=black,name= Name }, Damage ) -> "Suffer you, black " ++ Name . Then, in the console: > record:damage(Zombie,0). "Suffer you, black Zombie" > record:damage(Zombie,1). "Die, Zombie" 9.1.3 Updating Records Records would be useless, if we couldn't update them: real_damage ( Meat = # creature {life= Life }, Damage ) when Damage < Life -> Remaining = Life - Damage , Meat # creature {life= Remaining }. This syntax is just a short-cut for calling erlang:setelement/3 behind the scenes. Then, deal some damage in console manually and via the function: > ZombieKing = Zombie#creature{life=5}. #creature{name = "Zombie",color = black,life = 5,attack = 1, abilities = [drain_life]} > record:real_damage(ZombieKing, 2). #creature{name = "Zombie",color = black,life = 3,attack = 1, abilities = [drain_life]} 9.1.4 Sharing records Theoretically, it is possible to share records by declaring them inside a shared header file, which would then be included in the other erlang files: % just some header file -record ( damage , {amount = 1, color}). This block could be included into the original module, and then used normally: -include ( "header.hrl" ). black_damage ( Amount ) -> # damage {amount = Amount , color = black}. Now, reload the records and test the command: > rr(record). [creature,damage] > record:black_damage(1). #damage{amount = 1,color = black} However, sharing records between modules like this isn't a good idea. It exposes fragile data structure implementations (guts) to the outside world and is as bad as sharing database between multiple services. A better approach would be to expose getter and setter functions, which would allow manipulating records while hiding the implementation details. 9.2 Key/Value Stores For storing small amounts of data we have a property list ( proplist ) and an ordered dictionary ( orddict ). 9.2.1 Proplist Prolist is a loose bag of [{Key,Value}...] useful for storing small amounts of data. You create it by constructing manually, update in a light-weight manner by appending a new item NewList = [NewItem|OldList] . Then you can use methods in prolists module to lookup, replace, compact etc. Proplists are defined very loosely and are usually used for managing configuration settings. 9.2.2 Orddict orddict module provides a more formal key-value store that works well for up to 75 elements: unique keys that are also ordered; faster lookups; items must respect strict {Key,Value} structure. It is best to avoid manipulating the orddict instance directly and rely on the methods: new/0 and from_list/1 to create; store/3 to save; find/2 , fetch/2 to get; erase/2 to delete. 9.2.3 Dict vs gb-tree dict module exposes the same interface as orddict . It works well for larger datasets and also has fold/2 method. gb_tree is a rebalancing tree with slow insertions (updates and reads are rather fast). It provides more direct access to the data structure (e.g. allows to perform updates without checks) but doesn't have fold/2 (requires using iterators on a subtree instead). 9.3 Other data structures 9.3.1 Sets Erlang has for main modules to deal with sets: ordsets - slowest, but the most simple, implemented as ordered list; sets - same interface as ordsets but scales better and has faster reads, supports =:= operator; gb_sets - same interface, but provides more fine-grained control; sofs - sets of sets, for mathematical reasoning. 9.3.2 Graphs Erlang allows manipulating with graphs via: digraph - construction and manipulation of a directed graph; digraph-utils - navigation, testing of these graphs. Also sofs module allows converting families to the graphs and back. 9.3.3 Queues Queues in erlang are FIFO, provided by queue module. Internally they are implemented as two lists: add to first list; pop from the second list; when second list is empty - reverse the first and swap them. APIs could be split in 3 groups: basic: new/0 , in/2 and out/1 ; advanced: peek/1 , drop/1 , reverse/1 etc; Okasaki API: you probably don't need it. 10 Concurrency Concurrency is when two tasks can run at the same time, but don't necessarily do (e.g. multi-tasking on a single core). Parallelism is when these tasks do run at the same time (e.g. on multi-core machine). Erlang had concurrency from the very beginning, parallelism was possible by running erlang on multiple machines. Modern multi-core systems allow parallelism on a single machine. However, your parallel program goes only as fast as its slowest sequential part . Erlang wouldn't be a good fit for all problems. 10.1 Spawning Processes A process in erlang is just a function that can be scheduled to run via spawn/1 . This function returns pid in form of <0.160.0> which can be used to communicate with the process. The process terminates when the function returns. BIF self/0 returns the pid of the current process. 10.2 Message passing You send messages with ! (bang) operator, which works in form of Pid ! hello and returns a message (so that it can be passed to multiple processes). 10.3 Receiving messages You can dump all messages via flush/0 which would simply print them. For real work use receive expression, which looks similar to case pattern matching: receive Pattern1 when Guard -> Expr1 ; Pattern2 when Guard -> Expr2 ; Pattern3 -> Expr3 ; _ -> Expr4 end . Let's write us a small module: -module (spell). -compile (export_all). cast () -> receive {heal, Amount } -> io : format ( "Healing ~p~n" ,[ Amount ]); {damage, Amount } -> io : format ( "Ouch for ~p~n" , [ Amount ]); _ -> io : format ( "WTF?~n" ) end . This function could be launched either via spawn(fun spell:cast/0) or via a helper spawn(spell, cast, []) , which takes a module, function and arguments. Once launched, cast would sit around waiting for the receive to get a message. It will process it and terminate. > S1 = spawn(spell, cast, []). <0.181.0> > S1 ! {boom}. WTF? {boom} The only way to know if the recipient is alive and got message is by sending a reply. We can do that by packaging return address into a tuple. cast2 () -> receive { From , heal, Amount } -> io : format ( "Healing ~p~n" ,[ Amount ]), From ! "healing!" ; { From ,damage, Amount } -> io : format ( "Ouch for ~p~n" , [ Amount ]), From ! "damage" ; _ -> io : format ( "WTF?~n" ) end . Then run in a shell: > spawn(spell,cast2,[]) ! {self(), heal,2}. Healing 2 {<0.155.0>,heal,2} > flush(). Shell got "healing!" ok Now we just need to make sure that the process can process more than one message. We can do that by making it tail-recursive: summon ( Life ) -> receive { From ,damage, Amount } when Amount >= Life -> % no recursion here io : format ( "Dead! ~p~n" , [ Amount ]), From ! { self (), "dead" }; { From ,damage, Amount } -> io : format ( "Ouch! ~p~n" , [ Amount ]), From ! { self (), "this hurts!" }, summon ( Life - Amount ); { From , heal, Amount } -> io : format ( "Healing ~p~n" ,[ Amount ]), From ! { self (), "healing!" }, summon ( Life + Amount ); _ -> io : format ( "WTF?~n" ), summon ( Life ) end . The process will continue running till we damage it too much. Note that we also pass the state through that recursion. 10.4 Better design Message structure is a bit like internal implementation detail. Do we really need to expose it that much? Let's encapsulate the details. summon_wolf () -> spawn (? MODULE , summon, [3]). fireball ( Pid ) -> Pid ! { self (), damage, 2}, receive { _ , Message } -> Message end . ointment ( Pid ) -> Pid ! { self (), heal, 1}, receive { _ , Message } -> Message end . This hides all the dirty details and allows us to focus damage dealing: 0> Wolf = spell:summon_wolf(). <0.289.0> > spell:fireball(Wolf). Ouch! 2 "this hurts!" > spell:fireball(Wolf). Dead! 2 "dead" 10.5 Timeouts There is a problem. If we send spell:fireball/1 to a non-existent process (or the dead one) from our shell, then it will freeze. We are stuck in a receive deadlock. To work around the issue, receive has an after Timeout construct: fireball2 ( Pid ) -> Pid ! { self (), damage, 2}, receive { _ , Message } -> Message after 1000 -> timeout end . ointment2 ( Pid ) -> Pid ! { self (), heal, 1}, receive { _ , Message } -> Message after 1000 -> timeout end . After also can accept infinity atom (in case timeout is passed as an argument and we want to wait forever). There are two special cases - sleeping ( receive without any patterns) and trying to get messages without waiting ( receive with zero timeout). 10.6 Selective receive We can perform a selective receive by ignoring some messages (which puts them into a save queue for later processing). This is done via: important () -> receive { Priority , Message } when Priority > 10 -> [ Message | important ()] after 0 -> normal () end . normal () -> receive { _ , Message } -> [ Message , normal ()] after 0 -> [] end . The approach has a pitfall: selective receive puts non-matching messages into a special save queue, which is then traversed on each message. We can work around that by: provide a catch-all Unexpected variable which will log and discard; store messages in min-heap , gb_trees module or whichever structure is applicable. 11 Errors and Processes 11.1 Terminating a process When a process terminates, it always terminates with an exit reason , which can be of any term. If exit reason is atom normal then the process terminated normally. A process can terminate: when a run-time error occurs with {Reason,Stack} ; terminate by itself by calling exit(Reason) , error(Reason, [Args]) , fault(Reason, [Args]) ; when exit signal is received with a reason other than normal . Terminating processes emit exit signals to all linked processes. These kill a process, unless the reason is normal . A process can call exit(Pid, Reason) to emit an exit signal with Reason to the target process. Sender is unaffected. 11.2 Links You can link/1 current process to another by PID, making them emit exit signals to each other on termination. Linking is idempotent. Note, that in link(spawn(Function)) spawned process could finish before link is established. It is better to use atomic spawn_link/1 . 11.3 Traps A process can be configured to trap incoming exit signals and convert them into messages of {'Exit', FromPid, Reason} that are put into mailbox. One exception - if a reason is kill (e.g. from exit(Pid, kill) ), then this will bypass the trap and kill the process. Upon termination, an exit signal with killed is sent to all linked processed (to avoid killing them unconditionally). 11.4 Monitors Monitors are a unidirectional way to monitor if the process is alive and when it goes down. A process can setup a monitor via monitor(process, Pid) which would return Ref reference to the monitor. That reference could be used to demonitor/1 the process. When a monitored process dies, we get a message: { 'DOWN' , Ref , process, Pid , Reason } If the process didn't exist in the first place we would get the message immediately with Reason set to noproc . 11.5 Naming processes To give a process a name, use erlang:register(Name,Pid) . If the process dies, it will loose its name. Or you can use unregister/1 to do it manually. Then, use whereis(Name) to get Pid (or an undefined ).


My Essential Software

My Essential Software List of my essential software and services at the moment. On a laptop (MacBook Pro, pre-M generation): JetBrains IDEs: PyCharm and GoLand Firefox (because of Containers for work accounts) and Safari (energy efficient) MailMate - outstanding mail agent that keeps me sane MS Teams - team collaboration (quite buggy) MS Outlook - to accept office invites (mail is handled by MailMate) Vorta and Borg Backup - incremental backups of important files Typora - Markdown editor in which I'm typing this blog post. Apple Notes - gets the job done and surprisingly lean Monosnap - making screenshots and quickly annotating them Anki Desktop - to manage my collections of flash cards for learning iTerm2 - terminal Mobile (Pixel 3a, Android): Firefox (because runs uBlock) AnkiDroid - flash cards client for Anki. This where I learn German vocabulary CoolReader - for reading books in epub and fb2 Telegram and WhatsApp - primary chats CLI (shared across systems via dotfiles ): vim - lightweight editing Emacs - just for the org-mode (timesheets) zsh - just a shell that I'm used to tmux - to keep things running and have multiple tabs in the terminal On a server : Jupyter Notebook - to play with ideas, data and charts PyJournal - web app to extend my brain when working with multiple projects Wireguard - private VPN (mobile and laptop are in that network) Services : Hetzner - dedicated server for apps, VPN gateway and encrypted off-site backups Integromat - automation service used for tracking interesting (for me) tweets and posting them to my Telegram channel Spotify and Netflix DigiCal - because beautiful calendars AWS - S3, Route 53 and CloudFront to host this blog iPad : Adobe Draw - for sketches (discontinued) GoodReader - to read and annotate PDFs


♻ Event Sourcing

♻ Event Sourcing Event Sourcing is a software pattern : we record application state as a sequence of events throughout the time. Events could be then be projected to create application state at any moment in time. We could even derive new facts from the events.


KanDDDinsky 2025

KanDDDinsky 2025 KanDDDinsky 2025 Here are the links and references for the attendees of my talk at KanDDDinsky 2025: When DDD Met AI: Practical Stories from Enterprise Trenches Slides : PDF 18MB First of all, there is a lot of structured material on Schema-Guided Reasoning: Schema-Guided Reasoning (SGR) SGR Patterns SGR Examples SGR Demo SGR Adaptive Planning Demo of building reasoning SGR agent in 160 lines of Python code is particularly interesting, because it tends to "clicks" for some engineers. A team of enthusiasts took that SGR Demo and built an Open Source chat with deep research capabilities and tool use. It got so popular, that even banks started studying and taking it apart. Here is the Github repository . If you want to dive deeper into the core AI research and all insights from the past few years of community research, a lot of that is available in the public materials: Enterprise RAG Challenge (make sure to scroll to the second leaderboard and expand rows of individual teams - there are mini-reports for each one) LLM Benchmark Reports that also include product and architectural insights: latest report , full archive My personal low-frequency newsletter If you want to learn more about patterns and practices of building LLM-driven products in different industries - check out Courses . If you are an enterprise or large business that needs expertise get a project done - go to TIMETOACT Austria (I'm head of ML and Innovation there). Especially, if you are interested in software modernisation projects and enterprise AI projects.


FoundationDB is Back!

FoundationDB is Back! I'm very excited today - Apple has just open-sourced FoundationDB . What is FoundationDB? FoundationDB is a distributed scalable transactional (ACID) key-value database with crazy thorough testing . It was so good that three years ago the entire company was bought by the Apple and shut down. Despite that unfortunate fact, FDB was still used as an unsupported product in companies like: Snowflake - Data warehouse for the cloud. They say that FoundationDB "is a key part of our architecture and has allowed ... to build some truly amazing and differentiating features". Wavefront - Cloud monitoring and analytics by VMWare "has been using FoundationDB extensively with over 50 clusters spanning petabytes of data in production". Update: SkuVault - online warehouse management system, used FoundationDB as a distributed event store (commit log) holding billions of events and a coordination layer for nodes in the cluster. Obviously, Apple used FoundationDB internally as well, pushing a lot of improvements into it over the years. One of the improvements was the system keyspace sharding which, as Will Wilson mentioned , was the major blocker for the petabyte-scale databases . Another interesting improvement since 3.0 is the introduction of version stamp operations - atomic operations that put the commit version and batch number of a transaction into a key or value. That could help, for example, when you need to push a lot of writes in order to a cluster (e.g while building a consistent and distributed commit log). I used FoundationDB in my projects, too. It is just that good and reliable, even when the cluster runs in the cloud environment with flaky networks and rebooting machines. In fact, it is perfectly designed for that environment because of the rigorous testing. Data Model FDB is a key-value database with a "weird" design. You can think of it as a giant sorted dictionary, where both keys and values are byte arrays. You can do your normal operations with that dictionary (e.g. set, get, delete, get range) while joining multiple operations into a single ACID transaction. It is similar to the LMDB - an embedded B-Tree database which I also love and use. It works best, if you use FoundationDB data modeling primitives with it. This interface is very low-level, however it allows you to build up your own data layers on top, while hosting them on a single cluster: object storage; lists; tables; graphs; indexes; blob storage; high-contention queues; distributed commit-logs; pub/sub. In essence, FoundationDB is a database constructor . We've implemented all sorts of layers at SkuVault. Once you get used to the data modeling approach, using traditional specialized data storage systems feels very limiting. At some point in the past I scraped Web Archive copy of FoundationDB to preserve their data modeling recipes . Testing FoundationDB team developed the database inside a deterministic simulation . They abstracted away IO operations like network and disk, which allowed injecting all kinds of faults while running clusters under the load inside an accelerated time. Examples of faults that could be introduced in such environment: network outage; buggy router (dropping network packets, duplicating or delaying them); disk outages (corrupting data, running out of the disk space or slowing down); machine reboots and freezes; human errors (e.g. swapping IP addresses or disks). FoundationDB also had a real hardware cluster which had all kinds of faults thrown at it: power outages, network disruptions and disk filling. Deterministic simulation in this context means that you can simulate an entire cluster using a single thread . When a bug manifests itself, you could replay that simulation as many times as you want. It will always be the same. Just keep the initial rand-seed. Running the simulated system under a custom scheduler also enables you to accelerate the time , just like with any discrete-event simulation . If you know that for the next 10ms nothing interesting is going to happen, you could instantly fast-forward the world to that point in time. Time acceleration means that you can simulate years of cluster life in hours of real time. FoundationDB runtime is extremely rigorous and probably has an acceleration factor of x10. Simpler approaches can get acceleration of x100 to x1000. You can learn more about the approach from this brilliant talk by Will Wilson: Testing Distributed Systems w/ Deterministic Simulation . By the way, Kyle Kingsbury ("Aphyr") didn't bother running Jepsen against FDB because FoundationDB's internal testing was much more rigorous that Jepsen. Simulation Prototype I've been trying to reverse-engineer and replicate this approach for some time. The research started with a simple CPU job scheduler but quickly escalated to a distributed cluster simulation. You can check out the last prototype which introduces: Configurable system topology - machines, services and network connections. Simplified simulation of TCP/IP . This includes connection handshake, SEQ/ACK numbers and reorder buffers. There is now proper shutdown sequence and no packet re-transmissions. Durable node storage in form of per-machine folders used by the LMDB database. Simulation plans that specify how we want to run the simulated topology. This includes a graceful chaos monkey. Simulating power outages by erasing future for the affected systems. Network profiles - ability to configure latency, packet loss ratio and logging per network connection. The prototype builds upon the C# async/await to mimic the benefits of the FoundationFB Flow language and allow writing code that looks like normal parallel async but could be run sequentially in a simulation, rescheduled or erased. Unlike his naive simulation research, FoundationDB is the real thing . It has all the edge-cases worked out, internal transaction pipelines - tuned and the overall experience battle-tested with huge clusters. Summary I'm very excited that Apple has decided to open source FoundationDB. It is an outstanding piece of engineering that helps to build scalable and self-healing distributed systems. FoundationDB is free to use at any scale. Check it out on github . You can continue reading this story in a blog post about how SkuVault used FoundationDB all these years.


Go: The Good Parts

Go: The Good Parts Go is an opinionated programming language initially developed by Google in 2007. It is a statically typed language with a simple syntax , resembling C or JavaScript. It features garbage collection, type safety and large standard library. Go can be statically compiled into a single executable binary, which could target a large number of operating systems (from Linux and Windows to Plan 9) and processors (i386, amd64 and ARM). In HappyPancake project we found that Golang was a good fit for developing event-driven backend services. Other contenders included Scala, C#, Haskell and Erlang. Here is how code can look like in Golang: func ( m * ContactsModule ) handleList ( r * ApiRequest ) ApiResponse { return NewObjectResponse ( & ContactListModel {}) } type ContactsModule struct {} func ( m * ContactsModule ) Register ( r Runtime ) { r . HandleApi ( "GET" , "/contacts" , m . handleList ) } Good parts Concise and simple syntax, easy to get started with. Good facilities for writing concurrent programs that share state by communicating (goroutines and channels). Good ecosystem for developing backend servers, all major drivers and libraries are generally available. Decent integration story with native code. Lightweight development stack with major IDE-like features provided by command-line tools (and available in different editors). Unified formatting style for the language, provided by gofmt . Compilation is fast even with large projects. Go supports all major OSes and CPU architectures. Lightweight type system with interfaces "by example" and type embedding. As a new language, Go attracts a lot of talent and interested developers. There are some interesting projects taking place in this ecosystem. Opinionated parts Golang is an opinionated language which started as an experiment. Some of its design decisions can take time to get used to. This especially applies to developers with a strong background in Object-Oriented Programming. Yet, these opinionated decisions ultimately add to the strengths of the language. Focused Packages Golang ecosystem encourages separation of code into compact cohesive modules which do one thing and do it well: packages . It is common to have packages with only one or two files. This way of structuring applications follows the Unix way of building software. It also aligns well with micro-services architecture. This approach might feel foreign for people coming from ecosystems which favor larger modules (e.g. projects in .NET). Yet it leads to code that is more simple and reusable. Type System Type system in golang is an opinionated one. It includes following features: Data types in form of structs (plain DTOs). Embedding types in each other. Functions and methods (functions with a receiver for dot syntax: response.Render() ). Implicit interfaces "by example" (types implement interfaces by providing methods with matching signatures). Lack of generics. Go is unique in the way methods, interfaces, and embedded types work together. Not so good parts You must set and tweak GOMAXPROCS variable for each program which needs parallel execution. You can build web applications and native UIs with Golang, but ecosystem there is rather immature, if compared to the other platforms. Ecosystem can be lacking in unexpected places. For example we had to deal with shortcoming of default SQL layer and patch PostgreSQL drivers. Debugging golang code is neither easy nor reliable. Golang has prominent explicit error handling style; if followed blindly, it can turn code into a spaghetti of error checks. Golang stores all dependencies as folders in GOPATH , e.g. abdullin/chk . Having a single global workspace makes things tricky, when you need to work with multiple versions of a library. This is the state of Golang at the moment of writing. Things will improve over time. More Examples This is an example of a simple web server in go. It maintains a list of accepted jobs, which are served for GET requests. You can POST new jobs to the server. It will process them asynchronously, adding to the list. We coded this example with Tom Janssens while working on Happy Pancake project. Neither of us had any prior knowledge of go. package main import ( "fmt" "net/http" ) var queue chan string var joblist [] string func init () { queue = make ( chan string , 10 ) joblist = make ([] string , 0 ) } func handler ( w http . ResponseWriter , r * http . Request ) { switch r . Method { case "POST" : val := r . PostFormValue ( "job" ) fmt . Fprintln ( w , "VALUE: " , val ) queue <- val case "GET" : fmt . Fprintln ( w , joblist ) default : fmt . Fprintf ( w , "Not supported" ) } } func projection () { for req := range queue { joblist = append ( joblist , req ) } } func main () { go projection () http . HandleFunc ( "/" , handler ) http . ListenAndServe ( ":8080" , nil ) } Summary I think, Golang is a great language for building fast back-end servers or utilities that work across operating systems. Ecosystem and tooling around golang are young. This attracts some talent but might create additional problems for certain projects. References An introduction to programming in Go Effective Go A Tour of Go Docker and Go: why did we decide to write Docker in Go? (Slides) SO Post : Erlang vs Go Video on concurrency patterns / principles


👍 HappyPancake

👍 HappyPancake At the end of 2013 Tomas Roos invited me to join the team of HappyPancake, largest free dating site in Sweden. It was initially written in ASP.NET with MS SQL Database server, soon to become a rather complex solution that was expensive to scale. Together with Pieter Joost , our small distributed team of 3 people started redesigning this site towards a simpler design that would be easier to evolve in the future. This is the story of that project.


New Team

New Team My starting days in HappyPancake were quite intense and interesting, despite the fact that I could spend only 20 hours per week on the project. I learned a lot of things that would be be completely out of reach for a .NET developer within Microsoft OS. Here are some bullet-points: Google Hangouts work nicely for team collaboration and screen sharing. SyncSpace drawing app is probably the best collaborative white-board for a distributed team. Mindmups are great for collaborative and personal brain-storming. Erlang is a great language for building low-latency and CPU-efficient apps. It has some learning overhead for a .NET guy like me. Golang is a great language with good ecosystem and really good performance. If compared to erlang, golang has a lower learning overhead for a .NET guy. Within these days we invested time in establishing high-level design guidelines for the system to be implemented. High-level goals were: Iterative development with emergent design (we don't know all the answers) Micro-services with support for experimentation and A/B testing (to get these answers and discard wrong assumptions) Base design evolution on reality and measuring it to validate assumptions Ubiquitous language for communication between services : HTTP with JSON/HTML Any language within the service (as long as it runs on Linux) Designing for a distributed team that wants to spend years learning things, experimenting and playing with cool tech. Making this work requires a lot of responsibility and ownership, which have to be factored into the design, as well. We currently believe that Micro-Services approach with a bit of Programmer Anarchy might work well for our case, as a foundation for building things up. For the upcoming week I plan to continue catching up with Golang (currently reading through The Way To Go) and then start drafting a prototype of low-latency Message Bus with Event Store capabilities and FoundationDb backend (codename BroStore ).


Language is an Implementation Detail

Language is an Implementation Detail Although everything about working at HPC is interesting, last week was quite peculiar on its own. There was an interesting discussion about use of async pub-sub messaging for communications between micro-services. That's what Fred George does, for example, with event messages. However, command messages have their own value as well, due to the behaviour that we associate with them (only one message handler could deal with command messages, unlike event messages, where there could be 0 or more). Yet, after a bit of discussions with Tomas, we discovered that introduction of command messaging breaks our nice decoupling with regards to ease of upgrades, experimenting or continuous delivery. Besides, if needed, you can always implement command messing within the boundaries of the service. This is possible, since we place a clear separation between high-level design decisions (the ones which talk about how mServices should behave and communicate) and implementation details (which govern how mServices should be actually implemented). For example, here are some high-level design decisions: Protocol for communications - JSON/HTML over HTTP in our case Messaging semantics - async pub/sub in our case Approaches for deployments, versioning them and experimenting with them - rapid iterations, A/B testing, using business metrics as the driver Set of recommended languages and technologies - still evaluating (see below) Design and development priorities - creating fun environment to work in, keeping things real, small and simple Execution and hosting constraints - Linux, clustered in our own DC with geo-replication Additional constraints - low latency and failure-tolerant Curiously enough, we are still iterating through the suitable languages for implementing new version of HPC (while also addressing the design and domain questions). So for this week I'm going to spend more time learning Haskell (in addition to doing dives into Erlang and Golang during the previous weeks). At the same point, our rewrite will probably start in .NET with micro-services design. Reason for that being - .NET, despite it's shortcomings and costs is the language where we all would be most productive and could release initial versions fast. This is crucial for gaining real-world feedback for evolving the system. By the way, if you listen to Fred George, he mentions that at one point 150000 lines of Java code where rewritten in 4000 lines of Closure code (or so). Based on my exposure to Haskell so far, I'd say that C# is almost as verbose as Java in this sense. Then, as the need arises, micro-services will be rewritten in one of more Linux-friendly functional languages to: Save on licensing costs. Improve performance and reduce latency. Make code more simple and concise. Stand on the shoulders of giants, reusing ecosystem and communities of languages we choose. In other words, languages are treated just like implementation details of the system. Even though there are some recommendations and guidelines, developers should be able to choose the tool they want in order to get the job done in the most efficient way.


Moving Forward with Golang

Moving Forward with Golang After a couple of iterations we settled for the go language as the primary language for rewrite of happy pancake from C#. Ideally we'll converge on Haskell later (that's something I would really like, due to the powerful type system and high suitability for capturing domain models). However, for the time being the primary language will be go. Reasons for that being: Simplicity of the language and similarity to C# Excellent ecosystem for the development of backend servers Availability of go drivers for FoundationDB Linux development stack (Ubuntu + Sublime/Vim) without large license fees Language is expressive enough for our needs Excellent resources and help tools Why FoundationDB is so important to us would be another blog post later (long-story short: it is like fast Redis with proper clustering support and only one data structure sorted key-value ranges). There are a few downsides of golang that we are going to live with: Concept of workspaces is somewhat messed up (imagine, that you have to work with two versions of a library). However, this is not nearly as bad as dll and nuget hell in .NET world Absence of generics or type inference that would work as such Getting started with golang was rather simple. We went with Tomas through: An introduction to programming in Go Effective go A tour of go All of these resources are an easy read (mostly attributed to the simplicity of the language itself). While doing that I setup an Ubuntu (LTS) with Sublime Text 2 and GoSublime package. Given all that, it was relatively easy to start porting layer code for FoundationDB from python to golang. I'm still running my dev environment as VM on my MacBookAir, although Ubuntu can live fine with 1GB of RAM, unlike Windows VM that had to ask for 2GB. Plus, since Parallels does not work well with Linux VMs, VMWare Fusion is used. While working on layer code, I had also to encounter Python along with its REPL. Syntax was a bit odd in the beginning, but quite simple in the long run. No tutorials even needed. For the next week I plan to finish porting queue and pub/sub layers for FoundationDB from python to golang. We'll see how it goes from there.


Starting with FoundationDB

Starting with FoundationDB During the last week at HPC my focus has been on FoundationDB . FDB is a nice NoSQL database which has a bunch of great properties: It stores key-value pairs of bytes, where keys are always sorted. You have usual GET/SET/DELETE operations along with range operations that come from sorted key nature Multiple key operations can happen in a transaction Many advanced operations can be implemented as Layers on top of that storage abstraction. There is even a SQL layer for that. FDB scales nicely as you add new nodes to the cluster Cluster of up to 6 nodes can be used for free in production We (or Tomas, to be more precise :]) managed to get 75k write transactions out of a small cluster we setup at the digital Ocean Setting up a cluster is a no-brainer even for a Linux noob like me FDB handles load distribution automatically, it moves data as necessary, too FDB has client libraries for python, golang, erlang, Node.js and even .NET Their team is extremely helpful and humble You can configure level of replication (e.g.: single, double, triple) before write is ACKed FDB can be configured to store views in memory or on disk, transaction store is always durable I personally really like that FDB is extremely opinionated about what it does (just a replicated transactional key-value storage), but it does this extremely well so far. We are planning to use FDB as our event store and for persisting view models (which will be replicated across the cluster). I'm actually the one having fun with that implementation. Event Storage itself is a simple abstraction, however making implementation work properly with FDB key-value storage is something that requires better insight into inner workings of FDB. Plus, I get to do that in a go language. My next week will focus on getting our full planned stack to play together (in an extremely jacky way), so that we could start developing components for the HPC2. PS: my current development environment looks like this (Ubuntu LTS + "awesome" tiling manager + sublime + GoSublime):


Evolving the Stack and learning Nanomsg

Evolving the Stack and learning Nanomsg Last week with HappyPancake was my first full-time week with the team. Time flew fast and left me wishing for more . I explored nanomsg (glorified sockets) and how it is used in golang. nanomsg is going to be our communication layer between components within the app, hence understanding its usage patterns was important. It was extremely exciting to pair with Pieter on go programming exercises (and also picking up some Sublime/Linux tricks along the way). While developing my first golang+nanomsg+FDB prototype, I was quite proud of the first code that was written. It was relatively robust, simple and somewhat performant. During a few next days I realised that it was actually an overcomplicated and under performing piece of software. Pieter shared his process about structuring and expressing ideas in golang. Tomas explained how to make that code brutally simple and yet more performant. That was awesome! Previously it would take me years or months before I could arrive to that breath-taking understanding of how stupid I were. With this team everything happens so much faster . Love it. During the week we set ourselves a goal of building a system which could return uncached reads from the front (reverse proxy) within 25ms under the load of 50000 HTTP requests per second , while degrading gracefully under increased load. This, assuming that we run 3 relatively small app servers, 2 reverse proxies and FDB cluster of 5 nodes. Obviously, throwing more hardware to the system, should scale it out. It is nice to be working on a system, where latency is one of the design constraints and having fun is another one . When I had to leave on Friday evening, Tomas and Pieter were discussing process of getting rid of state in services by pushing it all the way to reverse-proxy (using Lua on nginx to take care of HTTP connections, while preserving true stateless asynchrony over nanomsg on the inside). This approach has a synergy with building resilient system of small systems (aka micro-services architecture) communicating over asynchronous events and continuously evolving (versioning, A/B testing and continuous delivery are among the goals of the goals). Apparently, during the course of the evening, they refined the approach to make it even more simple and robust. I can't wait to see what this idea has turned into. By the way, FoundationDB just hit 2.0 version . They published a golang client library within that release, making use of FDB – a breeze. By the way, as Pieter and Tomas reported, upgrading our test cluster to v2.0 took 4 minutes. ETCD was also bumped to 0.3.0 .


Designing for Throughput and Low Latency For the last week I spent most of the time pairing with Pieter, learning more about our the performance and behaviour of our anticipated stack (for the second version of HappyPancake.com). It was thoroughly interesting exercise in systems engineering. Here is what our anticipated design looks like right now: With this design we want to have 25ms latency of HTTP reads (non-cached, 99% percentile) given the throughput of 50000 requests per second. A/B testing, feature toggling, continuous delivery and live upgrades (with ghost mode and ramp-up) included. Here is a short summary of lessons learned within the last week: Tomas is an absolute beast when it comes to crunching out small open source libraries It is quite easy to publish statistics from an app and then gather them in a nice WebUI for crunching (using client library fsd to publish to local statsD app via UDP. StatsD currently pushes stats to Librato Metrics with delay of 10 seconds). HTTP servers in Go are quite good, but can be a pain to extend or augment Nanomsg is really nice and performant, however the documentation is lacking. Profiling capabilities of Golang are absolutely stunning. Spending a week developing and debugging golang apps, while benchmarking them on a Digital Ocean cluster - teaches you a thing or two about Linux environment. It is awesome. Software engineering is about making theories about how your code will behave in production, then running experiments to validate these theories. You iterate and learn. Pairing up with somebody is an amazing opportunity to transfer knowledge and produce better quality code (I lost track of the number of times I was stunned and humbled by the experience and insight of other team members - so much to learn). We currently use TeamViewer (best image and keyboard sharing experience) and Skype for the voice. Campfire is for group chats (and chat ops). For the upcoming week I'll be working on pushing our stack closer to the desired performance numbers (we don't meat the goal, yet). It is an interesting exercise which forces you to learn a lot and go deep (to the point of tuning the OS).


Containers, virtualization and clusters

Containers, virtualization and clusters Last week was a bit hectic. We are waiting for a bunch of datacenters to provide us with the test access to virtualised and hardware servers. These have to be benchmarked in order to see how they perform. Some time during the week we realised two things: We aren't going to get decent numbers out of the machines on DigitalOcean Apparently DigitalOcean is using some cheap virtualisation environment which massively underperforms compared to VMWare This realisation lead us to the point where we started evaluating dedicated hardware option instead of the VMs. We are going to run Docker containers in them anyway, so there is not going to be any vendor or hardware lock-in. Here are a few notes on that: Dedicated hardware is fast Good virtualisation software adds little overhead on top of HW; bad virtualisation - kills any performance Docker containers add very little overhead (should be way below 10% in our scenario) but help a lot with software compartmentalisation Within the last week I was following the footsteps of Pieter, learning from him and writing my first docker containers. There are a few gotchas, but the entire concept is amazing! It is probably the best IT thing that happened to me since beer. It solves "works on my machine" syndrome in most of the cases, making it extremely easy to work with software both locally and in remote environments. The experience is a lot better than Lokad.CQRS abstractions for file and Azure backends that I came up with earlier. Eventually, while setting up the containers over and over again, we came to the conclusion that we want to automate the entire thing. Running a script to deploy new versions requires context switching which Tomas and Pieter don't like (I never been as productive in Linux as these guys, but I start feeling this too). Hence, we are thinking about using either Drone or fleet to deploy and manage containers across the cluster. We will probably be using ubuntu 12.04 LTS for the containers (long-term support and stable code). Trying something like CoreOS for the host OS seems compelling because it is native to etcd (awesome) and fleet. We'll see how it goes. This week is going to be about strengthening our continuous delivery story and getting more numbers from the stack in different configurations. A few more other highlights from the previous week: We switched to Slack from campfire (it is used for persisted chats between the team). Native client is awesome, works so much better than campfire and Skype group chats wrk is an awesome tool for doing load testing while measuring throughput and latencies


Benchmarking and tuning the stack

Benchmarking and tuning the stack I focused on testing our current stack, understanding how it behaves under the load and trying to improve it. We are currently running everything in a cloud environment with VMWare virtualization, setting everything up from scratch at the beginning of the day and tearing everything down at the end of the day. This helps to focus on automation from the very start. Our testing setup is quite simple at the moment: Benchmark Box (2 cores with 4GB RAM) - we run weighttp and wrk load tests from this one. Proxy and Application Boxes (8 cores with 4 GB RAM) - proxy box hosts terminator and web aggregator services, while app box hosts specialized services. FoundationDB Box (2 cores with 5GB RAM) - a single FoundationDB node Each of the boxes is by default configured with: Ubuntu 12 LTS and upgraded to the latest kernel (docker and FDB need that); Docker is installed (with override to let us manage lifecycle of images); ETCD container is pulled from our repository and installed, using new discovery token for the cluster; Logsd and statsd containers (our logging/statistics daemons) are downloaded and installed on proxy and app . Appropriate services are downloaded and installed on proxy and app boxes as containers (our build script creates containers of our code, pushing them to a private docker repository) FoundationDB is installed on fdb box. All services and containers are wired into the Ubuntu upstart (equivalent of windows services management). Whenever a service starts, it interacts with ETCD cluster to publish its own endpoints or get the needed endpoints from it. So for the last week I was polishing these install scripts (refactoring BASH is a fun exercise, actually) and also performing some tuning and optimization of that code. Currently we are using plain bash scripts to set up our environment. However bash scripts are just like imperative languages: they tell exactly what you want to do in steps. I’d see that trying out more functional tools might be beneficial for us in the longer term (ansible, puppet, chef or something like that). We have following baseline scenario right now: We run weighttp load testing tool on bench with keep-alive, 256 concurrent clients, 2 threads (1 per core) and enough requests to keep everything busy for 10 minutes; Each http request goes to terminator service on proxy box. Terminator, running basic http server of golang, handles each http request in a new goroutine. It simply serializes request to a message and pushes it to nanobus (our own thin wrapper library around nanomsg for golang). This will create an http context, which consists of a single go channel. Then goroutine will sleep and wait for the response to arrive on that channel. Timeout is another alternative. Nanobus will add a correlationId to the message and publish it to TCP endpoint via BUS protocol of nanomsg. Semantically this message is event , telling that an http request has arrived. Any subscribed service can get this message and choose to handle it. In our case there currently is a web aggregator service running in a different container and showing interest in these messages. Nanobus in web will grab the message and dispatch it to associated method (while stripping correlationID). This method will normally deserialize the request and do something with it. Currently we simply call another downstream service through a nanobus using the same approach. That downstream service is located on another box (for a change) and actually calls FoundationDB node to retrieve stored value. When web service is done with the processing, it will publish response message back to the BUS socket of terminator . nanobus will make sure that the proper correlationID is associated with that message. Nanobus in terminator service will grab all incoming messages on BUS socket and match them against currently outstanding requests via correlationId. If match is found, then we dispatch the response body into the the associated go channel. http handler method in terminator will be brought back to life by incoming message in go channel. It will write its contents back to the http connection and complete the request. In case of timeout we simply write back Invalid Server Operation . When I started benchmarking and optimizing this stack we had the following numbers (as reported by our statsD daemon): 12.5k http requests per second handled; 99th percentile of latency: ~18ms (99% of requests take less than 18 ms, as measured from the terminator ); CPU load on the proxy box: 9 (1 min average as reported by htop). Here are some improvements (resulting from a few successful experiments out of dozens of failed ones): Replacing BSON serialization/deserialization in nanobus with simple byte manipulation : +1k requests per second, –1ms in latency (99th), CPU load is reduced by 1; Switching to new libcontainer execution driver in docker : +0.5k requests per second, –0.5ms in latency (99th), CPU load reduced by 0.5; Removing extra byte buffer allocation in nanobus (halfing the number of memory allocations per each nanobus message being sent): +1k requests per second, –1ms in latency (99th), CPU load reduced by 1; Tweaking our statistics capturing library to avoid doing string concatenation in cases where sample is discarded afterwards : +1.5k requests per second, –1ms latency (99th). Hence, the final results are: 18k http requests per second; ~12.5ms latency (99th percentile). Our next steps would be to add more realistic load to this stack (like dealing with profiles, news feeds and messages), while watching the numbers go down and trying to bring them back up.


Change of Plans

Change of Plans Monday came with the change of plans in our team. Tomas and Pieter realized that although our planned architecture looks really awesome (with all that messaging and dynamic component switching) it is too futuristic for our current goals . We want to migrate out of .NET+SQL, for a start. We also want to learn more about our domain before investing a lot of time to optimize it to perfection. We archived our existing prototype code and switched gears to: Single git repository for the entire project Single process application with in-memory components Event-driven domain Our short-term goal is to capture HappyPancake domain in golang in the simplest possible way. Then we'll improve design from there . I spent part of the week working on our EventStore (which is just a layer on top of FoundationDB). After benchmarking it we encountered a very foundational problem: it is hard to append sequentially to a global event stream (transaction log) which is replicated on a cluster . You either get a horrible throughput or you need to sacrifice consistency which affects reading speed. Another alternative is to find something else to sacrifice in a way that has the least possible effect on event-sourced application. This was an interesting tech challenge, neatly isolated and spinning in our minds. We agreed to take some time to think about it before moving forward with the implementation. Today is the day we share our ideas. I also spent some time drafting a simple prototype of basic HappyPancake functionality decomposed into a bunch of event-driven components. It was an extremely rewarding experience to see concepts from C# being expressed in go language. This weekend I went to Chelyabinsk to deliver a talk on software design (masked under the title of "micro-services in .NET") at dotnetconf . Tomas was mostly dealing with the UI and UX, while sharing in some papers on algorithms and maintaining the first version of HappyPancake (something we have been spared from). Pieter was reevaluating golang web frameworks while also trying to make them work with PJAX for fast updates in the web UI.


Back to Basics

Back to Basics By the beginning of the last week I ported infrastructure for event-driven design (with aggregates, projections and in-memory views) from C# to golang. However, later on it was agreed that going through the event-driven modeling is yet not the fastest and simplest route to the working code. So this code was completely discarded (we could get back to it later) and we started evaluating something even more simple - CRUD approach with CouchDB and MySQL. FoundationDB, does not have any projection or querying capabilities at the moment. This means additional effort required to design and maintain those and might be a premature optimization at this point. While thinking about storage constraints in our design, I've been focusing on messaging functionality for the HappyPancake. Currently we have 150000 messages going through per day with text size up to 2000 characters (served by a large MS SQL database). 20000 users are usually chatting at the same time. Ideally, next version would make this experience more enjoyable and interactive. More messages sent == better for the business. I focused on prototyping a simple chat, where messages and presence notifications are served to the client with long polling http requests. CouchDB and mySQL were evaluated as storage engines at this point. Pieter, at the same time, focused on the storage problem from the perspective of profiles, storing and updating them, serving through http as documents and search feeds. We discovered that our favorite http library in go "Revel" can barely serve 4k requests per second due to all the magic it provides (including templates). Bare http server of golang can serve up to uncached 17k requests (to resources with templates) per second on the same machine. So there are some trade-offs to be made. I personally think we could stock to basic http library just fine, since Tomas is pushing extra effort to make our UX and UI extremely simple. CouchDB is a really old document database that has nice master-master replication, support for map-reduce and query engine. It is even used to support some experiments on Large Hadron Collider. To make things even more nice, CouchDB exposes change streams per database, to which you could subscribe from the client code. API is served over HTTP, while the core is written in Erlang. Unfortunately CouchDB didn't fit well to a simple CRUD scenario. Reason being - CouchDB is IO bound, all caching is delegated to the operating system. mySQL was, surprisingly enough, another contender for our storage engine. It previously felt to me that this is a legacy database from the early days of Internet. However, after starting to read "High Performance mySQL", I quickly realized that this exactly is its strongest point. This database was optimized and polished by the biggest internet companies in the world. It is rock-solid for both SQL and noSQL. Performance is predictable and tooling is rich. Yet, mySQL can't do miracles if your IO operations are limited by the virtual environments. We can have no more than ~400 operations per second on Glesys machines . So all through the weekend I've been searching for articles on clustered messaging architectures at a large scale, trying to figure out the simplest approach that would fit two constraints: Provide fast and responsive messaging UX implementation of which is capable of serving ~20000 new long polling requests per second; Have clustering capabilities (multiple app servers handling the load); Work with a relatively slow storage engine, using no more than 10-20 requests per second. Fortunately for us, we can live with: Occasional write failures are tolerable Cached data is fine in a lot of cases Systems are not mission critical If you think about it (and sleep over it a few nights, too), these design "relaxations" allow to deal with our domain with quite some ease: we can store messages and presence notifications simply in memory (replicated for some degree of reliability) going to the disk only for batched writes and cache misses (usually loading conversations that happened quite a while ago). Amount of memory dedicated for message cache can be tuned to find the sweet spot here. So, at this point, we don't really care about the choice of the storage engine for the purposes of messaging, presence and notifications: CouchDB, mySQL or FoundationDB. Each one would work just fine. However, I would personally prefer mySQL at this point, since it is easier to capture the domain. Some reading Long polling does not totally suck (article) Facebook architecture for 600M users (slides) High performance mySQL, 3rd edition (book) Google groupcache (project) Getting your first 200MM page views (erlang, video) Goodbye, CouchDB (article) CouchDB, the definitive guide (online book) Besides that, I started reading "Programming Distributed Computing Systems" by Carlos A. Varela, which is a very foundational and intense book. Highly recommended.


Messaging - Heart of a Social Site

Messaging - Heart of a Social Site At the beginning of the last week I had a simple responsive prototype of a chat server. It was a simple in-memory implementation in go, delivering messages and "user is typing" updates instantly over long polling http requests. Obviously, a single server chat application wouldn't be enough for HappyPancake.com. We want to serve live updates to 20000 online visitors (numbers from the current version of the web site), while also adding in some headroom for the scalability and fault tolerance. So the last week was dedicated to search, trials and failures on the way to multi-node clustered chat server . I started by reading a lot about existing chat designs and approaches outside of golang. Erlang and Akka were recurring theme here. So I tried to move forward by implementing something like akka's actor model (Singleton actor patter) in golang while using Leader Election module of ETCD. What is ECTD? ETCD is a highly-available key-value storage for shared configuration and service discovery. It is written in GO and uses RAFT algorithm (simpler version of PAXOS) for maintaining consensus across the cluster. That was a dead-end: Re-implementing akka in golang is a huge effort (too many moving parts that are easy to get wrong) Leader Election module in ETCD is theoretically nice. Yet, in practice it is considered as experimental by CoreOS team. Besides, go-etcd client library does not support it, yet. At some point we even pondered if switching to akka was a viable strategy. However, NSQ messaging platform (along with the other projects from bitly) served as an inspiration of getting things done under the golang. A few more days of consuming information on the design and evolution of social networks and I had an extremely simple working design of a multi-node chat. There were 2 small "break-throughs" on the way: You can use basic functionality of ETCD keys (with TTLs and CompareAndSwap) to implement entity ownership across the cluster We don't really need a concept of actors to implement a scalable cluster. Dead-simple semantic of Http redirects would do the job just fine. All chat conversations are associated with one out of N (where N is an arbitrary number) virtual chat groups through consistent hash of the conversation ID. A chat group can either be owned by a node (as indicated by the renewed lease on ETCD key) and be available in its memory. All other nodes will know that because of the ETCD registry and will redirect requests to that node. Alternatively, a chat group can be owned by nobody (in case of cold cache or if the owning node is down). Then a random node (smarter algorithm could be plugged later) would take ownership of that chat group. Why bother with concept of chat groups? Querying ownership of 100000 of chat conversations can be pretty expensive, besides we would need to send heartbeats for each of this conversations. It is easier to allocate N chat groups, where N is a fixed number. This can be changed later, though. Result of all that : a dead-simple chat prototype that runs on 1-K nodes, where nodes are extremely simple and can discover each other dynamically, sharing the load. If a node dies - another one would take ownership of the chat conversation. All chats are reactive. "user is typing" notifications and actual messages are immediately pushed to the UX over http long-polling connections. More types of events will be pushed over these channels later. Part of that simplicity comes from the fact that golang simplifies work with concurrency and message passing . For example, the snippet below flushes all incoming messages to disk in batches of 100. If there were no messages for a second, it will also flush captured messages. var buffer [] * Record for { select { case r : = <- spill : buffer = append ( buffer , r ) if len ( buffer ) >= 100 { persistMessages ( buffer ) buffer = make ([] * Record , 0 ) } case <- time . After ( time . Second ): if len ( buffer ) > 0 { persistMessages ( buffer ) buffer = make ([] * Record , 0 ) } } } For this week I plan to move forward: Finish implementing a proper node failover (currently nodes don't load chat history from FoundationDB) Made nodes inter-connected between each other (we actually need to publish notifications for a user in real-time, if he gets a message, flirt or a visit from another user). NSQ (real-time distributed messaging platform in golang by bitty) seems like a really nice fit here. During the week I also did some benchmarking of ID generation algorithms in golang for our event store layer on FoundationDB (not a big difference, actually). Here is the speed of appends (1 event in a transaction, 200 bytes per event, ES running in 1 node on Glesys with 5GB RAM, 2 Cores and VMWare virtualization; client - 4 core VM with 8GB RAM): 10 go routines : 1 k per second , 10 ms latency ( 99 ) 50 go routines : 3.5 k per second , 12 ms latency ( 99 ) 100 go routines : 5 k per second , 20 ms latency ( 99 ) 250 go routines : 7 k per second , 35 ms latency ( 99 ) Meanwhile, Pieter was working on profiles, news feeds and registration flows. He was stressing out different database engines by uploading data from the existing user base of happypancake.com. There is a lot to learn about the behavior of different technologies in our scenarios. In the end we seem to be converging back on FoundationDB as the primary storage. Tomas was mostly busy with admin, UI design and maintenance of the first version (protecting Pieter and me from the boring stuff). Here is some reading of the week: NSQ Messaging platform (project) Lessons learned optimizing NSQ (slides) About NSQ (slides) About NSQ (video) Service discovery in the cloud (article) Distributed algorythms in noSQL databases (awesome article)


Event-driven week

Event-driven week The last week started as planned. First, I implemented a persistence for a simple chat service, then moved forward with multi-master design for application nodes. In this design each application node can handle any request the same way. This approach: simplifies the design; does not prevent us from partitioning work between nodes later (e.g.: based on a consistent hashing, keep user-X on nodes 4,5 and 6); forces to think about communication between the nodes. The most interesting part was about the UX flows. For example, in a newsfeed page we want to : Figure out the current version of the newsfeed for the user (say v256) Load X newsfeed records from the past, up to v256. Subscribe to the real-time updates feed for all new items starting from v256 There is an additional caveat. While loading history of activities, we merely display them on the screen (with the capability of going back). However, activities that come in real-time need more complicated dispatch: Incoming messages need to pop-up as notifications and update unread message count in the UI. Incoming flirts and profile visits have to go directly into the newsfeed. Modeling these behaviors lead to some deeper insights in the domain. By the beginning of the week I wasn’t even able to articulate them properly :] Caveats of Event Sourcing Tracking version numbers in a reliable way was also a big challenge initially. The problem originated in the fact that our events are generated on multiple nodes. We don’t have a single source of truth in our application, since achieving that would require either consensus in a cluster or using a single master to establish a strict order of events (like Greg’s EventStore does, for example). Both approaches are quite expensive for high throughput, since you can’t beat the laws of physics (unless you cheat with atomic clocks, like Google Spanner) Initially, I implemented a simple equivalent of vector clocks for tracking version numbers of a state (to handle situation of reliably comparing state versions in cases, where different nodes will get events in different order). However, after a discussion with Tomas we agreed to switch to simple timestamps, which sacrifice precision for simplicity. We are ok with loosing 1 message out of 10000 in newsfeed, as long as it always shows up in the chat window in the end. NSQ For communication tech I picked NSQ messaging platform, since it already has a lot of tooling that boosts productivity. NSQ is used only as glorified BUS sockets with buffering and nice UI. Hence, if Tomas later on manages to push towards nanomsg, we could do that with quite an ease. A nice benefit of using something like nanomsg with ETCD or NSQ is that this system does not have a single point of failure. All communications are peer-to-peer. This increases reliability of the overall system and eliminates some bottlenecks. Micro-services Understanding of micro-services keeps on evolving in an predictable direction. We outgrew approaches like “event-sourcing in every component” and “CRUD CQRS everywhere” to a more fine-grained and balanced point of view. A component can do whatever it wants with the storage, as long as it publishes events out and keeps its privates hidden. Even in a real-time domain (where everything screams “reactive” and “event-driven”), there are certain benefits in implementing certain components in a simple CRUD fashion. This is especially true in case where you can use a scalable multi-master database as your storage backend. Pieter was exactly working on the CRUD/CQRS part of our design, modeling basic interactions (registration, login, profile editing and newsfeed) on top of FoundationDB. This also involved getting used to the existing HPC data, different approaches in FoundationDB and go web frameworks. Tomas was mostly busy with the admin work, supporting our R&D and gaining more insights about existing version of HPC (with the purpose of simplifying or removing features that are not helpful or aren't used at all). Plans This week is going to be a bit shorter for me - we have May 1st and 2nd as holidays in Russia. Still, I will try to finish modeling event-driven interactions for the newsfeed and chat. This would involve UX side (I still didn’t fit transient events like user typing notification into the last prototype) plus implementing a decent event persistence strategy. The latter would probably involve further tweaking our event storage layer for FoundationDB, since I didn’t address scenario, where the same event can be appended to the event storage from multiple machines. We want to save events in batches, while avoiding any conflicts caused by appending the same event in different transactions.


Reactive Prototype

Reactive Prototype Last week, as planned, was quite short but very interesting. Development of the reactive prototype at some moment hit a complexity point where a dead-simple hacky approach could no longer work. Although go language (with its simple flavor of behavior composition) allowed to go pretty far on that route, in order to move forward, I had to bite the bullet and refactor things from a big ball of mud to a collection of components. That’s when I realized that I already enjoy coding in golang as much as I enjoyed working with C# in Visual Studio with ReSharper after 8 years of practice in it. After that refactoring I was able to move forward with the domain exploration (in case of HappyPancake domain includes both the social networking and technical peculiarities of developing reactive application at a social scale). One of the interesting aspects of the development was the interplay between: reactive nature of this prototype (new notifications are rendered on the server and pushed to the client through http polling feed); different ways of handling the same event from different contexts and screens (e.g.: a chat message would be appended to the conversation in a chat screen but it will show up as a notification box in another screen); different ways of persisting and delivering information to the users (e.g.: chat history is loaded via range read from FoundationDB, while all updates to this history are pushed to the client through the notification feed); focus on reducing CPU and latency for the mobile devices (e.g. last 75 messages in a chat come pre-rendered in the page HTML on the first page request, while new messages are pushed incrementally by appending server-generated HTML to the DOM); our desire to have graceful degradation of the user experience for some of the older mobile platforms (users could still get some chat experience even if javascript does not work at all). At this point, I think, we have a pretty good understanding of the domain around messaging and notification feeds. We have: a bunch of implementations and use cases captured in the tangible and working code; strategy for scaling the performance in a variety of scenarios (with known price to pay for that in terms of complexity or consistency); some understanding of how we would deal with devops later on. Meanwhile, Pieter was working on the other half of HappyPancake understanding and developing interactions around document based flows in the social network - registration, logins, profile editing and reviewing. All with PJAX and basic http handlers (we discarded Revel, since it does too much CPU-intensive magic). Tomas , as usually, focused on backing up our development. He took care of the v1 maintenance, campaigns and also invested in capturing use cases for us to move forward. It was extremely interesting to sync up with Tomas and Pieter occasionally, sharing concerns and discoveries along the road. It felt like getting an instantaneous deeper insight into the problem we are trying to tackle here. Another really awesome part of the last week was about gradual transition from purely technical use cases (consistency, availability and latency issues) to practical use cases that matter to our users (flirts, messages, visits etc). Although technology is an important part of HappyPancake, users are the domain that we ultimately trying to understand and master. The upcoming week will be a bit longer than the previous for me, but still only 4 days (May 9th is another holiday). We plan to start my prototype into Pieter’s prototype, while moving forward and adding more use cases. I hope to also move forward with newsfeeds. They require a balance between consistency and availability that is different from notifications and chat messages (more like the Instagram news feeds).


Tactical DDD

Tactical DDD I started merging bits of my reactive prototype into the document-driven prototype of HappyPancake that Pieter was working on. While at that, we spent a lot of time discussing the design and iterating over it. It was really cool to see how the structure of the solution shifted focus from technical model to functional model . Previously our golang packages (which roughly map to lightweight .NET projects) contained files grouped by their technical intent (e.g.: controllers, models, documents). This added friction to development: a lot of context switching was required in order to work on a single use case, touching multiple packages; solution structure enforced certain architecture style upon the codebase (when you have folders like models, controllers, views and documents, naturally you will be trying to fit your implementation into these); merge conflicts were unavoidable, since too much code was shared. Over the course of the week, we switched to a different design, aligning packages with use cases. You might consider this to be a tactical domain-driven design (we didn’t touch any of the strategic parts like Bounded Contexts or Ubiquitous language, since our core domain is extremely simple). Golang packages get tightly aligned with our use cases. They either implement cases directly (e.g.: by exposing http handlers to render the UI and process POST requests from the browser) or they help other packages to fulfill their role by providing supporting functionality or structures (e.g. authentication utils, http helper methods, core value objects). Of course, the road wasn’t all about roses and pretty ladies - you can’t just split codebase between a bunch of folders and hope that all will work and make sense . It is never that easy. We had a lot of discussions like : How do we decompose functionality into multiple packages which will work together to implement these use cases? This code does not make any sense, what are we doing wrong? How do we name this thingy? What is the simplest approach to implement these use cases? How can we work together on this functionality? I really enjoyed every minute of these discussions with Pieter, they were focused on the problem domain instead of fiddling around artificial architectural constraints imposed by the overall design. Besides, so far, we were able to resolve these questions and thread the thin line between over-engineered monolith and messy big ball of mud. We are not sure if we’ll be able to walk this path later, yet so far each step led to a deeper insight in the domain of HappyPancake (just like domain-driven design promises). There are a few really cool things about our current design: it is extremely easy to collaborate on the code : there are almost no merge conflicts; we are free to experiment with implementation styles within packages without turning solution into a mess; golang is designed to support small and focused packages, this shows up frequently as yet another tiny and deeply satisfying moment. The most important part is : our code is a reflection of domain knowledge captured in a tangible form. Codebase is structured around that knowledge and not vice versa. In the meanwhile Tomas was busy with administrative work and HPC1. Towards the end of the week he also got a chance to start working on the HTML design of HPC2 in a stealth mode. Pieter and me are both really anxious to see what comes out of this work. Also on Friday we were interviewed by a couple of students on the topic of CQRS. I’d think that our joint statement was something like “CQRS is new name for denormalization with a little recollection of what happened before 2010”.