Message Vault

Message Vault This week with SkuVault was 10 hours long, like the previous one. It was purely dedicated to long-term design evolution. After going through a series of scenarios we ended up with the following plan: Introduce Message Vault - a simple pub/sub message streaming systems for Windows Azure. Make all existing modules publish events to Message Vault . Extract some projection code from the existing SkuVault code and move them into simple modules with API on top: Search , Reports , ProductViews etc. Cover these modules with use-cases. One by one, start migrating aggregate logic from existing SkuVault codebase to the new design. Remove commands and simplify aggregates, where appropriate. Implementing Message Vault is the biggest step here. All the other steps are going to be smaller and more incremental. Message Vault Message vault provides publish-subscribe with replay and batching for Windows Azure . It is heavily inspired by Apache Kafka. However, it is going to be a lot more simple than that. We are going to simply add a thin wrapper around Azure Blob Storage, to act as an API and write synchronization point. Windows Azure is going to do all the heavy-lifting. We need to embrace Windows Azure in order to keep the implementation simple, yet scalable. At the moment of writing, SkuVault stores more than 100000000 events with total size above 25GB. Semantics are going to be similar to Apache Kafka (this way we could migrate to Apache Kafka or any other implementation of a distributed commit log, if needed ). Producers push messages to the Vault which serves them to consumers. Consumers can replay events from any point in time or chase the tail. We partition messages by streams. Each stream is an immutable and ordered sequence of messages. All messages in a stream get a unique offset and timestamp. Absolute order between messages in different streams is not guaranteed, but we can still sort by timestamps (within the time drift on Azure). Message Vault is not going to be an Event Store , it is not designed for event sourcing with aggregates" (you need NEventStore or EventStore for that). Design Trade-offs Message Vault makes following trade-offs: optimize for high throughput over low latency ; optimize for message streams which are gigabytes large; prefer code simplicity over complex performance optimizations ; http protocol instead of binary protocol; rely on Windows Azure to do all the heavy-lifting (this simplifies code, but couples implementation to Azure); high-availability via master-slave setup (uptime limited by Azure uptime, no writes during failover); no channel encryption (if needed, use SSL with Azure Load Balancer or your load balancer); no authorization schemes (if needed, configure your load balancer or add a proxy on top); implemented in imperative C# (.NET runtime is heavy, but Windows Azure optimizes for it); client library is intentionally simple (view projections and even checkpoints are outside the scope); each stream is a sepate page blob (they can grow to 1TB out-of-the-box, having thousands of streams isn't a good idea). Implementation Implementation should be rather straightforward, since we brutally optimize the implementation for the task at hand (SkuVault project itself). We allocated 20 hours for that. The most tricky part is going to be: write and test master-slave failover (we need high availability) and handling writes between the moment the master goes down and moment, when azure lock expires. By the way, awesome folks from AgileHarbor (company behind the SkuVault) agreed to make the project open-source under the New BSD license. The project will be hosted on github . New BSD License (or _3-clause BSD_) allows you almost unlimited freedom with the code as long as you include the copyright notice. You don't need to share your code. You cannot use the names of the original company or its members to endorse derived products. Evolution Once we have MessageVault in place, we could start extracting some view-based logic into new modules. This will help us: we reduce complexity of existing codebase; new modules are designed for new performance requirements, take off the load from the existing codebase; new modules (at least the ones with pure denormalization logic) are going to be brutally simple. Eventually, we are planning to have SkuVault composed from the modules like this one: In essence, this is an application of "event-driven micro-services" paradigm to .NET environment. SkuVault C# modules are probably going to be "larger" than equivalents in erlang or golang. That is because C# .NET ecosystem is shaped by enterprise mindset. As a side-effect, everything tends to be bigger on .NET: libraries, classes, variable names and build times.


Delivering First Bits to QA

Delivering First Bits to QA Deploying MessageVault This week we finally pushed our MessageVault project to QA deployment. It will live there for some time, before going to production. To remind you, MessageVault is a simple message storage server that runs on Windows Azure. It is inspired by Apache Kafka. This server allows saving messages as streams to page blobs on Windows Azure Storage. Streams can reach terabytes and go beyond. Once we persisted messages to Azure Storage, we could read them directly from there, MessageVault server is not needed for that. You can have any number of consumers reading them or even replaying from the start. All events that are recorded in QA deployment of SkuVault, get pushed to dedicated streams on MessageVault. It is easy to consume them from any code at this point. From now we could start migrating some existing logic to dedicated modules to do reporting, exports, serve as API etc. The most important benefit of MessageVault setup is that it would allow to: Implement some of the existing functionality as stand-alone modules with API. Scale out the solution by batch processing messages, and adjusting implementation of each module according to the use-case. Experiment with writing new modules without risking the integrity of the existing solution. Simplify existing solution, by replacing integration via commands and views with integration via crafted domain events . Introduce testability of API modules as one whole. Simplify front-end (no need to query multiple key-value documents in order to display one simple report or a view). What's more important - these changes could happen incrementally without big rewrites to the system . At least that's the plan. We'll see how the things would work out in the next weeks. You might get an impression that we are going to transform a large application into a crowd of small micro-services in .NET. This is not entirely true. In fact, the number of projects could even go down, as we reduce coupling and duplication of view logic. We had to learn limitations, week points and scaling possibilities of MessageVault. So it went through some intensive stress-testing before the QA. The best approach was to deploy the system and test it in different scenarios. For example, these are the throughput numbers (transactions per second and messages per second) and latencies while running it on Large, Medium and Small instances of Windows Azure Role. As you can see, MessageVault doesn't handle long-term load well, if deployed as Small Azure Service. It handles load better (and with lower latencies), if deployed as Large Azure Service. These performance numbers are preliminary, there are things that could see improvement (e.g. improving scheduler for high-throughput scenarios). However, we are going to do an internal code-review of the project before moving forward with it. This approach follows the mantra: first make it work, then make it beautiful; afterwards make it fast. You could check out the sources and performance for yourself. MessageVault is available on github as part of AgileHarbor repository. It includes libraries for the client and server along with examples and an Azure deployment you can use. Hekad Gateway Graphite statistics on the image above are a courtesy of our dedicated devops server. The same server also provides a nice UI to access and search logs in real-time. In order to benefit from these capabilities, a code has to know how to push data to this devops server. Integration between .NET and Linux proved to be the most tricky part. However today, we pushed to QA a first deployment of SkuVault with real-time statistics and logging enabled. SkuVault codebase itself isn't instrumented with performance counters yet. However, it became easy to do so. This will allow to learn behavior of the system in real-time, understand the bottlenecks and improve performance for the users. The integration project for pushing logs and statistics from .NET code to Hekad is available as open source . It is probably too specific for reuse, but it could serve as sample. Breaking the Build Fun fact: in the process of QA deployment I broke the build half a dozen times. The project didn't compile on my machine due to a private dependency I didn't have. So I had to be pretend to be a compiler and MSBuild, verifying my own changes in the head before pushing them to the build server. It all worked in the end, but I probably didn't leave a good impression of a first-time committer to the core codebase :) SkuVault Record Obviously, the development effort in SkuVault didn't focus just on DevOps and messaging middleware. The team was busy improving user experience and stability of the rapidly growing product. By the way, during last Black Friday, SkuVault storage broke the record of 100M events stored in the system. At this point in time, there were more than 4.5M sales created in SkuVault in total (throughout the history) and 2.3M sale items picked. Kudos to the team of SkuVault for reaching these impressive numbers and exceeding them. Next Steps In the next weeks we'll be working on migrating some of the existing logic to a simpler event-driven design, as well as planning and prototyping new features for the product.


Simplifying Web UI

Simplifying Web UI During my last few weeks at SkuVault I focused on three areas: Find a way to simplify existing application design, scale it and introduce an API. Work on a new SkuVault feature (time-bound) implemented with API and a new UI prototype to back it up. Plan and prepare infrastructure bits needed to start rolling things out to the Production. So far, as it looks like, all design changes seem to come together rather nicely. Most of the credit here goes to Slav and his team for making everything event-driven from the start (the story would’ve been much less interesting, if the product was centered around the database). Plan for the backend is quite simple: gradually replace existing modules with API modules, while covering API with specifications and simplifying the internal implementation. A lot of commands and views will become obsolete. Language and platform of the new code would stay the same - .NET. While working on the new feature, we came to the idea of how to deal with the Web UI as well. SkuVault UI already reflects a lot of complexity from the underlying domain, while ASP.NET MVC, Lokad.CQRS and Angular.JS don’t necessarily make things easier. Fun domain fact: you can’t really prohibit a warehouse worker from doing something in the application. If really needed (as in: “manager just told him so”), he will do whatever is needed. However without the ability to record that change in the app, discrepancies will start accumulating. Solution to that problem couldn’t involve the Big Rewrite , because of the time-frames and the risk. Eventually, we came up with a simple idea: Start implementing new features as tiny single-page Web applications that talk to the HTTP API. Make sure that if you are logged into one feature, you are logged into the entire application. Any JS framework could be used to implement a feature. It all works, as long as it renders into a static JS bundle. Since a feature is implemented as a static single-page application, it could be hosted on any platform that can serve static files (Azure Web Sites, CDN, Amazon S3). Existing app could serve these features as well, and that’s how we will migrate the UI - replace existing code with small features that are versioned and served separately. Although, if React.JS is used to implement a feature, visual widgets from the shared library could be reused. Folks at Facebook use that a lot. This approach actually has a few additional benefits : UI Features could be developed, deployed and tested in parallel, without stepping on the toes of each other. We could have a setup, where a git push of a new version of a feature product-search to a branch smart-auto-complete will actually deploy that feature to a website under the url /product-search-smart-auto-complete/ . That could speed up the development and QA cycle. We have a clean separation of concerns between the stateful API (handles scaling and domain complexity) and the stateless UI (focuses on user interactions). A/B testing and gradual feature roll-out to customers also become quite easy to implement. While working on new functionality, I started implementing UI this way because it was much faster to prototype. However, once we discovered additional benefits of that approach and a simplicity of integration with the existing UI, we might give it a try. These changes could take some time but should bring the entire product into a rather technically boring stack: HTML + JS frontend and .NET services in the backend. Boring, if you forget ReactJS, possibility of React Native, event-driven backend services and the scalability challenges :) Within the upcoming weeks we plan to roll out new MessageVault integration and API bits to work with the events coming from the production (got to test realistic workloads). I’ll be focusing on the new feature most of the time.


Solving the right problem

Solving the right problem In the Solving the wrong problem , we took a look at what happens when the project applies a very good engineering solution to the wrong problem. This blog was an example. Here is a story of another project that focused on the right problems from the start. Getting priorities straight A few years ago I wrote a small journaling web app to help me work on multiple projects at Trustbit. The purpose was to let me track any piece of information with as few distractions as possible. A lot of effort went into the user experience . I've been ruthlessly killing and rebuilding all features that somehow affected the writing flow. I tried to minimize the friction and make the writing process as rewarding as possible. A little "pat on the back" once in a while can do wonders. Throughout the development, I deferred any performance optimizations or tech improvements until it really hurt (it happened only once). Whenever the project could be dumbed down, it would be. I killed these awesome features that were a technical and cognitive burden: full-text search note tagging expandable topic views topic name auto-completion within the editor (to make it a wiki). Plus, I have replaced Ruby-on-rails with a boring Python Flask app, file-based storage and SQLite indexing. What is the measurable outcome? Metrics speak for themselves. I've been using Py-journal for ~70 weeks. It has more than 1700 posts. That is 3 notes on average per day! Granted, only a small fraction of these notes could be shared publicly, but there are quite a few interesting stories there. In the hindsight, what were the most important decisions leading to this outcome? First, it takes only a few steps to write down something that belongs to an existing topic. Everything that is not so relevant is either faded or removed. The writing flow looks like this: Open the web app (it lives in a persistent Safari tab). Type a few letters to find the matching topic via instant fuzzy search (or hit "add" to create a new one). Hit "Enter" to open the first match. Hit "A" to add a new note and start typing right away. Write down the text in a visual editor. Copy-paste or drag images to attach them to the note. Save when finished. Second, it is the job of the application to make content discoverable and rewarding. As soon as the note is saved, the application will parse the HTML and pull any optional headers. These are added to the global table of contents associated with the topic. It is so rewarding to add a new note and see the outline grow! This Topic view also helps to maintain a high-level perspective on the progress . Along the way, it subtly encourages to continue the work : "Look, you have been working on this thing for so many years, what about digging a little bit more, just to keep the momentum"? Weekly review shows the work done within a specific week (also rewarding): All topic images are included in the pinboard view (more fun than useful, but encourages to add illustrations): Filtered export creates a nice PDF for all work that was done within a specific time period in some topics. This is how I send work notes to my colleagues twice a week. Lessons learned Core lessons that I've learned while working on that project (and writing this retrospective): Good engineering solutions can be a liability when they solve the wrong problem. Pick the metrics to optimize and focus on them. Remove anything that doesn't improve the metrics, this will make room for helpful additions down the road. Given these lessons, I plan to fix the problems with this blog as well. It should be easier and more rewarding for me to publish here. As a first step, I moved the entire blog to a tiny cloud VM (Debian) with the Caddy server running in front. The website is still static, but the editing loop is faster. It is a small relief. It is just the first week of 2022, but I'm already finishing my 3rd blog post. Next possible solutions to explore: introduce comments; speed up publishing pipeline; add WYSIWYG editor with image copy-paste experience; make URL slugs and filenames - optional; drop GA and set up privacy-preserving analytics to track the numbers of visits in Grafana; set up target metrics that I want to achieve: number of weekly visits, number of blog posts, reactions etc; encourage friends and colleagues to write more frequently as well. We'll see how this works out 😄 Follow-ups I don't like Google Analytics. It is slow and invasive. So I'm removing it completely. Lean web analytics


Solving the wrong problem

Solving the wrong problem I've blogged about Problem-driven design , then followed up with Problem-driven vs Domain-driven . Let's take a look at a more concrete example. We'll use this blog as an example of a technically correct solution that solved the wrong problem and actually sabotaged my writing efforts. Then we'll check out another project that has achieved a different outcome by focusing on a different problem. The story is split into two blog posts. Good solution This blog has a long history. It started somewhere on Livejournal, Blogger, and WordPress. After getting tired of WordPress maintenance, I moved to Squarespace. Sometime later - to Hugo, then to a static website generator written in node.js, then in Python. The story is captured in Blog Migrated to Python . From the operational perspective, the current setup is perfect . Content and the code are versioned in git. Python code runs a few hundred content files through the templates to generate a static website. Files are uploaded to S3 and served via Cloudflare CDN with edge servers across all major countries. Amazon handles the DNS, domain, and certificate renewal. I'm a software engineer, so I optimized the solution to some degree of perfection. The maintenance overhead and the costs are negligible, the scalability - unlimited. It never goes down. Wrong problem This "perfect" solution has an issue - it solves the wrong problem . While chasing the surface problem "Make it scalable, cheap, and low-maintenance", I forgot about the original purpose of the blog: to publish stuff and learn through that process . When in doubt, consult the metrics : there were just 3 blog posts published in 2021 and 5 blog posts in 2020. Clearly, this setup doesn't help me to publish blog posts more frequently. This is an example of what happens when you solve the wrong problem: the solution is valid from the engineering perspective, but the overall product is useless . Ugly outcome Let's dig into the writing workflow for the potential clues. I have an idea for a blog post, something to share. I go to the work folder, navigate to content/ and notice that the git state is dirty from the last time. Ok, time to commit previous changes. This will require reviewing the changes and writing an appropriate commit message. Meanwhile, the original idea faints away. Blog posts are stored as markdown files with YAML front-matter. So I need to create a new file. How should it be called? What should be the title? When the editing time comes, I'll need to keep in mind the original idea, the Markdown syntax. Where should I put that image attachment? What was that markdown syntax for the images? Why did I pick the markdown syntax? Because it is popular and makes it easy to edit existing text. The truth is - I rarely edit something that was published. I published something, but the changes don't show up. Ah, I need to invalidate Cloudflare CDN caches to see the changes online and share the URL. Let me log into the AWS console and find my way around. Ouch, there is an embarrassing typo. I probably need to edit the file, publish and then invalidate caches again. Also, don't forget to commit changes to the git with a meaningful name. Little by little, these tiny distractions nibble away at the original blog post idea and inspiration . Until I throw up my hands in exasperation and switch to something else. This also leaves the git repository in a dirty state. As such, I have published just one blog post every 4 months (on average) in the last six years. There are quite a few unfinished posts as well. How could we solve the underlying problem? Let's recap. We want to encourage writing and publishing more frequently . To optimize the entire flow for my specific constraints: limited brain capacity, little time, plenty of distractions, and interruptions. Not a problem : ultra-low maintenance, high scalability, and low-effort editing of the existing blog posts. From this perspective, the software stack is less relevant than the user experience . Follow-ups I'm getting rid of Google analytics on this website: Lean web analytics We take a look at another writing project of mine that focused on different problems and achieved better results: Solving the right problem .


Stop user session tracking experiment

Stop user session tracking experiment For the past months I've been running an experiment on abdullin.com: Removed Google Analytics from this website Replaced analytics with cookie-less user flow tracking: (Over) Designing privacy-first analytics . Used Clickhouse and Python to analyze http access logs from Caddy: Analyze website logs with clickhouse-local . IP addresses were not used. Outcome: not worth it for me . The setup was clever, but the simplicity of "I don't track your session, even in anonymous and aggregated way" - wins. I could produce charts like that: or that: But in reality a chart like that is good enough for me: Update list for the website: Remove Hotwired Turbo and stop tracking user sessions. We are back to plain old server-side pages. Upgrade rendering pipeline to Python 3.9 (and a couple of other dependencies) Convert project repository to a proper Python project with setup.py , setup.cfg , pyproject.toml etc. This is consistent with Python standards and data science projects that I help to manage at day work. Less context switching is good.


Structured Output

Structured Output Summary : Structured Output (constrained decoding based on grammar) forces LLM to respond only according to a predefined schema. Structured Output was popularised by OpenAI, but since then found its way to multiple cloud providers and local inference engines. The best way to illustrate the concept is with a code snippet. Let's say, we want to parse chat messages and extract calendar event data out of them. One way to approach that is by prompting LLM to respond in a specific format. Then, parsing the response with regular expressions to extract required fields. Another approach is to use response schema which will ensure that the output will be structured in a certain way. Like this: Response follows CalendarEvent schema, so we can parse and manipulate is as a type object right away. This saves a lot of development time. The code above prints the list of parsed participants. It will print ['Alice', 'Bob'] When using Python, you can leverage different types of properties to constrain the response: from pydantic import BaseModel from typing import Literal , List class SqlResponse ( BaseModel ): sql_query : str query_type : Literal [ "read" , "write" , "delete" , "update" ] tables : List [ str ] class ComponentResponse ( BaseModel ): height_mm : float width_mm : float depth_mm : float number_of_pins : int component_type : Literal [ "AC/DC" , "DC/DC" ] Different languages will make use of various typing frameworks. Implementation-wise, under the hood everything will most likely be converted to JSON Schema before being passed to LLM inference engine. Structured output is an essential tool for improving LLM accuracy via Schema-Guided Reasoning (SGR) How does this work? Under the hood Structured Output works like a regex for the token generation. LLMs generate probabilities for all tokens at each single token, and constrained decoding simply prohibits certain tokens from happening. We can illustrate this with a simple snippet. The code below prompts a local model (Mistral 7B in this case): "Write me a mayonnaise recipe. Please answer in Georgian". By default Mistral 7B is a small model that will not be capable of answering in a lesser-known language, but this specific code will work: from transformers import AutoModelForCausalLM , AutoTokenizer , LogitsProcessor import torch model = AutoModelForCausalLM . from_pretrained ( "mistralai/Mistral-7B-Instruct-v0.2" , torch_dtype = torch . float16 , device_map = "auto" ) tokenizer = AutoTokenizer . from_pretrained ( "mistralai/Mistral-7B-Instruct-v0.2" ) # MAGIC HAPPENS HERE messages = [ { "role" : "user" , "content" : "Write me a mayonnaise recipe. Please answer in Georgian" }, ] tokens = tokenizer . apply_chat_template ( messages , return_tensors = "pt" ) . to ( device ) generated_ids = model . generate ( tokens , max_new_tokens = 1000 , do_sample = True , num_beams = 5 , renormalize_logits = True , logits_processor = [ Guidance ()]) decoded = tokenizer . batch_decode ( generated_ids ) print ( decoded [ 0 ]) The reason for that is a small class called Guidance which we pass to logit_processor field. This class makes it impossible for LLM to answer in anything but Georgian: import regex alphabet = re . compile ( r '[\u10A0-\u10FF]+' ) punctuation = regex . compile ( r '^\P {L} +$' ) drop_mask = torch . zeros ( 1 , tokenizer . vocab_size , dtype = torch . bool , device = "cuda:0" ) for k , v in tokenizer . get_vocab () . items (): s = k . lstrip ( '▁' ) if alphabet . match ( s ) or punctuation . match ( s ): continue drop_mask [ 0 ][ v ] = True drop_mask [ 0 ][ tokenizer . eos_token_id ] = False class Guidance ( LogitsProcessor ): def __call__ ( self , input_ids , scores ): return scores . masked_fill ( drop_mask , float ( '-inf' )) Code will work as expected, but will come with a caveat: Mistral 7B will indeed answer only with Georgian letters but would sometimes respond in a complete gibberish. This highlights the major caveat with Structured Output - it forces the model to respond only in a very predefined format, but: this doesn't magically distill the model with corresponding skills this can actually reduce model accuracy, because we constrain not only response but also thinking process. We can leverage Schema-Guided Reasoning (SGR) to use Structured Output while improving accuracy. Caveats Caveat: Description Fields Some LLM APIs will use the response schema twice: To compile and load into the inference engine To silently insert into a prompt Because of this, the following structured request will work as expected on OpenAI: class ResponseFormat ( BaseModel ): say_hi_like_a_royal_person_briefly : str = Field ( ... , description = "Respond in German!" ) it will respond in German: { "say_hi_like_a_royal_person_briefly" : "Guten Tag, ich grüße Sie hochachtungsvoll!" } However, not all APIs and inference engines do that. You can use the ResponseFormat above to test this assumption. Caveat: Accuracy It is very easy to reduce accuracy of a model by introducing constrained decoding. This happens, because we take the ability of a model to think before providing an answer. Schema-Guided Reasoning (SGR) could help to mitigate the problem. Implementations Structured Outputs by OpenAI Mistral Structured Outputs Structured Outputs with Google Gemini Local: XGrammar Local: Outlines


ReactJS and Facebook Flux at DDDBE in Belgium In January 2015 I was invited by awesome DDDBE community in Belgium to give a talk about building responsive web applications with Facebook Flux and React.js. This was a big honor, since European DDD community is the place where the most of recent insights in Domain-Driven Design originate from. Below you will find links to video, slides and the source code for this presentation. References Source Code ( version from the talk ) Video ~2 hours Slides Trappist Beer Belgiam DDD community kindly introduced me to the Trappist Beer , which is probably among the best beers in the world. Trappist beer is brewed by Trappist breweries. Only ten monasteries — six in Belgium, two in the Netherlands, one in Austria and one in the United States — currently brew beer and sell it as Authentic Trappist Product. Weeks later, these few treasured bottles reminded me about the given promise to commit something cool to the Open Source, marking it with #dddbe_beers tag. Thanks to _Orval Trappist Ale , seq` was delivered. It is a library in Golang to verify assertions against complex objects (e.g. events and API responses). We always wanted to have this library, while working on event-driven specifications at HappyPancake project. You can find it out on github . Feedback Many thanks to Mathias Verraes and Stijn Vannieuwenhuyse for organizing the event. Also my thanks go to the DDDBE community for having me and providing such an inspiring feedback. DDD Belgium Community We liked our last speaker @abdullin so much, that we've booked another Russian. Bram De Moor : Great talk yesterday! A nice mix between tech and conceptual overview, good pace, and comprehensive answers to the questions. Tom Soete : Note to self: Take day off after each @dddbe meeting! Can't wait to start playing with #ReactJS + #FluxJS after the great talk by @abdullin David Cumps : Had a very nice @DDDBE presentation by @abdullin, thanks. Enjoyed it, going to look into react and flux, really resonated. Steven Willems : Thx @abdullin for introducing me tonight to #flux and #reactjs. I really liked what I heard. Actually, it really is CQRS in the browser :) Tom Janssens : Thank you; we had a blast! Kees van den Berg : thanx for the inspiring presentation of Flux with the latest @DDDBE meetup Sven Schelfaut : Really nice @DDDBE session from @abdullin Looking forward to the next session @combell Dieter Provoost : Great talk on ReactJs/Flux by @abdullin for @DDDBE. Thanks!


Workshop on Event-Driven Design in Madrid A few days ago I ran a small workshop on software design for the awesome folks at h4ckademy. h4ckademy - 6 weeks of immersive code retreat in Madrid, inspired by HackerSchool. It's designed for programmers that want to become better professionals. Israel Gutiérrez is one of the organizers of the event. Audience was very diverse - heterogeneous backgrounds, different levels of experience (recent college graduates and professionals with 10+ years in software) and different minds (from designers to console hackers). We decided to run a crash course in software design, focusing on the practical ideas that could be explained quickly and in a coherent manner . The plan looked like this: Essence of software design - principles of Domain-Driven Design, fast iterations, importance of feedback, focus on contexts, boundaries and contracts. Divide and conquer. Domain events and API contracts as the core part of interchange contexts. Event Storming Session - collaborative exercise on analyzing a new business domain and building a useful model: contexts, domain events, API contracts. Practical application of event-driven model - split domain implementation into modules; capture behaviors with runnable specifications expressed via domain-events and API contracts; scale design to handle more features, team members and higher loads; high availability. CQRS Beers - an informal discussion, focused on QA, actual code and implementation details. We talked about ReactJS/Flux vs AngularJS and the other MV* frameworks, career paths of a developer, building reputation, working remotely and dealing with burn-outs. The workshop went well, I enjoyed presenting to such a diverse audience and doing exercises together. Many thanks to Israel Gutiérrez for inviting me over for such an event. What could be improved To make this workshop better: allocate more time for the same amount of material (we had to rush through some concepts) or reduce the amount of material for this time-frame; explore more than one problem domain in exercises, switching people between teams; dedicate a block of ~15-20 minutes to front-end problems and designs. Things that worked out very well and should be kept: mixing presentations with QA and collaborative exercises - this allows to keep people more involved in the process, cover more material; CQRS Beers - that format works very well for relaxed discussions (as proven by the years of experience :) ); event-storming - it is one of the best parts of the design process, thanks to the EU DDD community, there is a lot of fun and interactivity; following the reasoning sequence from "foundational design principles at bird-view" to "collaborative design exercise" and then to "practical applications"; maybe it could be extended with hand-on exercises for longer workshops. References These additional materials contain more information on the topics covered during the workshop. They can also provide answers to some questions we didn't have time to address. Download slides (PDF). Slide 3 is the most important one there. HappyPancake Story - story of a HappyPancake project, covering many aspects of event-driven design, "micro-services", specifications and event-driven UI. Check this story, the others and simply browse the site. Being The Worst - light-hearted podcast on domain-driven design, implementation patterns and learning how to build a task manager. We are still doing it. Domain-driven Design - the foundational book by Eric Evans on DDD methodology and related patterns. Start reading it with the chapter on context Mapping. Keep in mind that "Domain Events" weren't considered to be very important when the book was written (things have changed since then). Implementing DDD - newer book by Vaughn Vernon, addressing some concerns and questions which appeared since the "Blue Book" came out. Don't read the appendix on Event Sourcing, material there is very outdated. Ziobrando's Lair - blog by Alberto Brandolini, a great guy and an experienced DDD practitioner that coined term "event storming". Sample TODO app in ReactJS/Flux - front-end sample that I showed during the CQRS Beers. It is work in progress. Sample modular backend for TODO app - work in progress, but already includes these specification tests I showed during the workshop. References above are most closely aligned with the material given during the workshop. If you have more specific questions, please don't hesitate to get in touch. Responses h4ckademy : Very funny workshop about event-driven design with @abdullin Enjoying it a lot! h4ckademy : ... A lot of fun and learning, both at the workshop and the beers! Ruben H : Great fun yesterday learning EDD with @abdullin at @h4ckademy Feedback Anonymous: What did you like the most? When you explained how to implement a microservices architecture with a DDD approach. What else would you like to hear about? More in depth event treatment, techniques and tools. Anonymous: What did you like the most? I liked a lot a general map of how the architecture is made. It was very clear, and interesting. Also i enjoyed play with the architecture over the table splitted in teams of a company, it's great to have a hands-on game. What could make this workshop better for you? The workshop was excellent, but to make it even better, maybe could be possible to make a little application or at least the basic architecture of event driven design, just to have a feeling with the architecture. What else would you like to hear about? I would like to know more use cases and maybe a real stories of implementing this kind of architecture in real projects. Ruben H : What did you like the most? First part, where you explained how to design an app/software using DDD. What could make this workshop better for you? More time on event-storming session. What else would you like to hear about? Compare DDD to other methods that also allow to design and implement app/software. Víctor Pérez : What did you like the most? It gives you a general overview of how to join bussiness and development worlds in a verbose way. What could make this workshop better for you? Maybe longer, but it was already adjusted to our needs, so maybe there's no reason for changes. Rubén Antón : What did you like the most? The explanation was really good and easy to follow for everyone. But the part I liked the most was the event storming one. It showed off how valuable can be in order to understand the domain and the acceptance tests we could write later on. What could make this workshop better for you? I'd love to hear more about the architecture inside the microservices. What else would you like to hear about? If you were at the workshop and want to share your feedback, please don't hesitate to get in touch with me via twitter @abdullin or email:rinat@abdullin.com.


Software Design in Barcelona at eBay Enterprise A few days ago I had a chance to give a talk on software design in Barcelona. The event was organized by Barcelona Software Craftsmanship group and hosted by eBay Enterprise International. Photo by Angel Rojo . The audience was quite diverse: folks with PHP, JavaScript, Java, C# background, most of them are programmers, a few technical managers, DevOps. There were people from eBay teams as well as the other members of Barcelona software scene. Plan for the class looked like this: Principles of software design - principles of the Software Design Process, DDD and decomposition, importance of fast feedback loop and iterations, boundaries and contracts. Divide and conquer principle. Domain events and API contracts as the core part of interchange contexts and enforcing long-term stability. Event Storming Overview - audience was too large to run an event-storming session within the conference. Instead we went though the theoretical aspects of event storming and various uses. Practical application of event-driven model - split domain implementation into modules; capture behaviors with verifiable use cases expressed via domain-events and API contracts; scale design to handle more features, team members and higher loads; high availability. CQRS Beers - an informal discussion, focused on QA, actual code and implementation details. We talked about CQRS/DDD principles, command buses, asynchronous UI updates, handling high load and ReactJS/FLUX. I enjoyed the class and the beers afterwards. Many thanks to Alvaro García , Manuel Rivero , Cabre Barrera and Villazala Gordo for the invitation and organization of the event. Thanks to Angel Rojo for the photo. References These additional materials contain more information on the topics covered during the workshop. They can also provide answers to some questions we didn't have time to address. Download presentation (PDF). Slide 3 is the most important one there. Sample modular backend for TODO app - work in progress, but already includes these use cases I showed during the workshop. HappyPancake Story - story of a HappyPancake project, covering many aspects of event-driven design, "micro-services", specifications and event-driven UI. Check this story, the others and simply browse the site. Being The Worst - light-hearted podcast on domain-driven design, implementation patterns and learning how to build a task manager. We are still doing it. Domain-driven Design - the foundational book by Eric Evans on DDD methodology and related patterns. Start reading it with the chapter on context Mapping. Keep in mind that "Domain Events" weren't considered to be very important when the book was written (things have changed since then). Implementing DDD - newer book by Vaughn Vernon, addressing some concerns and questions which appeared since the "Blue Book" came out. Don't read the appendix on Event Sourcing, material there is very outdated. Ziobrando's Lair - blog by Alberto Brandolini, a great guy and an experienced DDD practitioner that coined term "event storming". References above are most closely aligned with the material given during the workshop. If you have more specific questions, please don't hesitate to get in touch. Responses Angel Rojo : I enjoyed a lot the talk by Rinat... Interesting round of Q&A after the session and thanks to eBay to provide the snacks and drinks! Nacho Cabre : Masterclass on event driven design by @abdullin thanks for all the food for thought! Félix Delval : ...really enjoyed the presentation of @abdullin about event driven software jhvaras : The important is in the white spaces. Great talk. Thanks @abdullin @eBayESP @bcnswcraft Francesc Gil : @abdullin really good talk today at meetup ;) Alvaro Garcia : @abdullin thanks for the very interesting and information-packed conference at @eBayESP about #DDD and #eventDriven Feedback If you were at the workshop and want to share your feedback, please don't hesitate to get in touch with me via twitter @abdullin or email:rinat@abdullin.com. Nacho Cabre : What did you like the most? I enjoyed a lot the explanation of emergent design from a DDD standpoint, event storming and all the discussion around domain events. What could make this class better? It would be very interesting to explain all these concepts just working on a simple problem and following all the steps you described on the emergent design with the same problem. For instance, the typical TODO list application how it would be implemented following the DDD approach. What else would you like to learn about? I think it would be very useful to do a practical workshop and work on a problem and practice event storming, context mapping, develop some component using tactical design patterns, measure and then, refactor it in a second sprint.


🤔 Opinionated Tech

🤔 Opinionated Tech This is a collection of my ever-green notes on software design, languages and technologies. "Ever-green" means that these articles are kept up-to-date with my learning and discoveries. Unlike articles from Archive .


Telemetry

Telemetry Telemetry originates from Greek roots meaning "remote measurements". When applied to software development, it gives a real-time insight into a running software. Metrics can include: gathering logs from different machines; capturing performance metrics in real-time; tracing individual requests running through different machines. Linux world has a few handy (and battle-tested) tools to help with these problems. Installing and configuring them can be a task ranging from "it takes just an apt-get" to "it requires a day of admin work". Logging My preferred approach to logging can be described as: write structured text logs and ship them to an ElasticSearch cluster with Kibana dashboards . Let's discuss that in more detail. Structured logging libraries so far worked the best (for me) for capturing logs in applications. They allow to: capture meta-data in the events; write these events to rolling text files; ship events in batches to ElasticSearch cluster (with retries and cluster fail-over). Serilog is a logging library by Nicholas Blumhardt (also famous for Autofac.NET). It preserves the structure of events it writes, making it well-fitted for feeding that data to ElasticSearch. Recommendations: Always enable self-diagnostics log of your logging library (e.g. write it to a file that is truncated every time it reaches 1MB). If you have legacy-style log lines (e.g. "System shuts down after {0} hours" ), replace them with the annotated metrics ( "System shuts down after {uptimeHours} hours" ), which will come in handy for log processing later. Enable rolling logs and constrain maximum size for each file, based on your known storage quota. This is especially important for deployments with limited disk space, like Windows Azure. Elastic Search is an open-source server for searching and analyzing data in real-time. There are a lot of ways to ship the data to it from different machines. For example, you can use Logstash in Java world or Serilog with ElasticSearch sink in .NET. There are a lot of materials written about ElasticSearch. Here is a feature list . Recommendations: Given the choice between a cloud and a dedicated machine to host ElasticSearch, go for the latter. Non-virtualized environments have more performance and are generally cheaper. Individual shards can fail. It is advised to run ElasticSearch in a small cluster. Don't do all the setup yourself, just find a sysadmin to spend a day or to and handle the task. It will be faster and cheaper. Kibana is a nice Web Interface for exploring and visualizing the data. It plugs into the ElasticSearch and just works. Recommendations: Give access to Kibana to the developers and just let them build dashboards for themselves. Setup an automated backup of Kibana dashboards (even if they are stored in ElasticSearch cluster). Metrics Application metrics are usually real-time events that fall into the following categories: timers express duration of a particular method call (e.g. API call or DB request). counters measure number of calls, amount of data that goes through or anything else that can be counted (e.g. a number of requests per second, bytes per minute or exceptions per day). gauges represent a single variable that can change over the time (e.g. amount of used disk space or RAM consumption). These metrics are frequently represented as tiny events with a complex name ( API_v1.PROD.Worker_IN_0.azure.queues.sales.count ) and a numeric value. StatsdClient is one of the libraries for capturing metrics in a .NET application. Metrics can be shipped from a running process via a UDP broadcast to a locally running aggregator daemon. This daemon will then send them to a metrics storage back-end. I like Mozilla Hekad for metrics aggregation, however it might encounter some issues under Windows. Carbon (storage), Graphite (robust but ancient Web UI) and Grafana (front-end for Graphite front-end) are frequently used to store these performance metrics, explore and build dashboards for them. Graphite can be a pain to setup, so reusing some existing docker container is recommended. InfluxDB is more robust alternative to Carbon/Graphite (and Grafana works with it), however sending data to it would require rethinking metrics and rewriting the dashboards. Automated alerts could be setup, once you have application metrics available. For example you could get emails: if there is no activity in queues during a certain peak interval; there is an unusually high number of registrations or web requests; server runs out of disk space or starts swapping. API takes unusually long time to respond.


Terms and Conditions

Terms and Conditions Terms and Conditions - ML Labs 1. Introduction Welcome to abdullin.com , including its subdomain labs.abdullin.com (referred to as the "Website"). These Terms and Conditions (T&Cs) govern your access and use of the Website, including any restricted areas that may require account creation. By accessing or using the Website, you agree to be bound by these T&Cs. If you do not agree with these terms, please do not use the Website. 2. Accounts and Access to Restricted Areas Account Creation: To access certain restricted areas of the Website, you may be required to create an account by providing an email address. The Website may send you an access link to verify your email and grant you access. You agree that the information you provide is accurate and up to date. Account Responsibility: You are responsible for maintaining control over access to your account (e.g., access to your email). Please notify us immediately if you suspect any unauthorized access to your account. Account Termination: We reserve the right to suspend or terminate your account at any time if you violate these T&Cs, misuse the Website, or engage in any unlawful or harmful behavior. 3. Use of the Website Informational Purpose: The content provided on this Website is for informational purposes only. While efforts are made to ensure that the content is accurate, the Website does not guarantee its completeness, reliability, or timeliness. Prohibited Activities: You agree not to: Use the Website for any illegal or unauthorized purpose. Interfere with the security of the Website or any restricted areas. Share or transfer your access to the Website to another person. Attempt to access areas of the Website you are not authorized to access. Distribute, reproduce, or share content from the Website without prior written consent and proper credit to the Author. 4. Intellectual Property and Content Ownership The content, design, and layout of the Website, including text, images, graphics, and logos, are protected by intellectual property rights, including copyright, and are owned or licensed by the Website owner. You may not reproduce, modify, distribute, or republish any part of the Website without prior written consent. For further details, please refer to the Copyright section of the Website. 5. Limitation of Liability Use at Your Own Risk: The Website is provided "as is" without any warranties, express or implied. The Website owner makes no guarantees as to the accuracy, completeness, or reliability of the content and disclaims any liability for damages arising from your use of the Website. No Liability for External Links: The Website may contain links to third-party websites, which are not under our control. We are not responsible for the content or practices of these external websites. 6. Governing Law These Terms and Conditions shall be governed by and construed in accordance with the laws of Austria. Any disputes arising from the use of the Website shall be subject to the exclusive jurisdiction of the Austrian courts. 7. Changes to the Terms and Conditions The Website reserves the right to update or modify these T&Cs at any time without prior notice. Any changes will be posted on this page, and it is your responsibility to review these T&Cs regularly. By continuing to use the Website after any modifications, you agree to the updated terms. 8. Contact Information For any questions or concerns regarding these Terms and Conditions, please contact the Website owner at biz -bot @ abdullin . com.


Privacy Policy for Online Courses

Privacy Policy for Online Courses Privacy Policy for Online Courses 1. Introduction This Privacy Policy applies to all personal information collected by Rinat Abdullin, e.U., through the online courses available at labs.abdullin.com. By registering and using the Courses, you consent to the procedures described in this policy. 2. Data Controller Company Name : Rinat Abdullin , e . U . Address : 1220 Vienna , Gisela - Legath - Gasse 5 / 1 , Austria Email : biz @abdullin . com is the data controller responsible for the processing of your personal data as described in this Privacy Policy. 3. Personal Data Collected and Stored The following personal data is collected when you use our services: Email addresses Information about access rights and purchased courses Answers to quiz questions, where applicable Records of watched materials 4. Purpose of Data Collection The data collected is used for the following purposes: To provide access to the digital content of the Course To manage and administer your access rights to the purchased materials To process transactions and maintain a purchase history To communicate with you regarding your account and any relevant product announcements or updates 5. Legal Basis for Processing The processing of your data is necessary for the performance of a contract to which you are a party (i.e., the provision of the Courses as per the Terms and Conditions agreed upon purchase of an individual course or a subscription). 6. Data Retention Your personal data will be retained for as long as necessary to fulfill the purposes outlined in this Privacy Policy unless a longer retention period is required by law. 7. Data Security We implement appropriate technical and organizational measures to ensure a level of security appropriate to the risk, including hashing of your passwords and securing your data against unauthorized access, destruction, or alteration. 8. Sharing and Disclosure The author does not share your personal data with third parties for their marketing purposes. We may disclose your personal data to third-party service providers, who perform services on our behalf under confidentiality agreements, ensuring they adhere to the privacy standards mentioned in this policy. No website analytics services track or collect data about your visit or usage patterns on our site. 9. Your Rights Under the GDPR, you have the following rights regarding your personal data: Right to Access (Art. 15 GDPR): You can request confirmation and details about the personal data we process about you. Right to Rectification (Art. 16 GDPR): You can correct inaccurate or incomplete personal data. Right to Erasure and Restriction (Art. 17 and 18 GDPR): You can request deletion of your data, or restrict processing under certain circumstances. Right to Data Portability (Art. 20 GDPR): You can receive your personal data in a structured format or transfer it to another party. Right to Object (Art. 21 GDPR): You may object to processing of your data based on our legitimate interests, especially for direct marketing. Right to Lodge a Complaint (Art. 77 GDPR): You can lodge a complaint with a supervisory authority if you believe your data is misused. We are also obligated to notify any recipients of your data about your requests for rectification, erasure, or restriction of processing. 10. Changes to the Privacy Policy We may update this Privacy Policy from time to time by posting a new version on our website or third party services. You are advised to review this Privacy Policy periodically for any changes. 11. Contact Us If you have any questions or concerns about our Privacy Policy or the data processing practices outlined herein, please contact us at biz -bot @ abdullin . com.


Terms and Conditions for Video Courses - Hosted Video Terms and Conditions for Video Courses - Hosted Video 1. Introduction These Terms and Conditions govern the sale and use of all online courses (hereafter referred to as "the Courses") Rinat Abdullin (hereafter referred to as “the Author”), a sole proprietor in Austria, available at labs.abdullin.com (hereafter, “the Website”). By purchasing access to any Course, you agree to be bound by these Terms and Conditions and our Privacy Policy. The contract between the Customer and the Author is concluded at the time of purchase when the Customer completes the payment process. 2. License and Access The Courses are offered as digital products available for viewing on the Website and are strictly for viewing at the Website and not for download. Upon purchase, customers will receive the access link via their registered email, which will serve as their login for accessing the Courses. This license is limited, revocable, non-exclusive, and non-transferable except as explicitly provided herein. 3. Pricing and Payment The pricing for the Courses varies and is displayed on the website. Payment is due at the time of purchase and is a one-time fee unless otherwise specified. Access to the Courses is granted immediately after the payment is processed. If you purchase a multi-seat or team access plan, you must provide the email addresses of the team members who need access. Access for these additional seats will be granted promptly after you provide the required details. The number of seats available for each pricing plan is specified at the Website. 4. Cancellation and Refund Policy In accordance with Austrian law regarding digital content and online services, consumers have the right to withdraw from the purchase contract within 14 days of the contract's conclusion without giving any reason. However, this right of withdrawal expires once the streaming of the digital content has begun with your prior express consent and acknowledgment that you thereby lose your right of withdrawal. Therefore, refunds are not supported once access to the Course has been initiated. 5. User Obligations You agree not to provide your login credentials to others or to allow others to use your account. You agree not to use the Course in any manner that violates applicable laws or regulations. 6. Intellectual Property All intellectual property rights in the Course, its content, and all materials distributed at or in connection with the Course are owned by the Author. Any unauthorized use, reproduction, or distribution is strictly prohibited. 7. Disclaimer of Warranties and Limitation of Liability The Course is provided 'as is' without any guarantees, conditions or warranties as to its accuracy, completeness, or reliability. We do not warrant that the website or the services that make the Course available will be continuous, uninterrupted, secure, or error-free. 8. Miscellaneous These Terms and Conditions constitute the entire agreement between you and the Author, regarding your purchase and use of the Course. 9. Governing Law and Jurisdiction These Terms and Conditions are governed by and construed in accordance with the laws of Austria. Any disputes arising out of or in connection with these Terms and Conditions shall be subject to the exclusive jurisdiction of the Austrian courts. 10. Contact Information For any inquiries or complaints, please contact biz -bot @ abdullin . com.


Event Sourcing Video

Event Sourcing Video This video explores relations between domain model, event sourcing and traditional fact-based state persistence. Video link DDD Transport Tycoon kata could be found on github . See also There is another video on Event Storming (12 min) (workshop format). You can also check out a few blog posts about a large event-driven project I participated in: Verification of Event-Driven Systems High Availability and Performance


Event Storming (12 min)

Event Storming (12 min) Video link Event Storming is a workshop where people use colored post-it notes to discover and understand together something complex: business, process or software. This video goes through: prerequisites for running an Event Storming workshop; workflow of a single workshop; what happens after the workshop. You can learn more about Event Storming in a book by Alberto Brandolini . See also Event Sourcing Video .


Webinar Terms & Conditions

Webinar Terms & Conditions


Write things down

Write things down If you have some things on your mind - write them down. Your brain has a limited capacity to keep things alive. Besides, it takes energy to keep things around. So your brain will start shedding away some "irrelevant" details anyway. By writing things down, we preserve them on a more durable medium. Paper or files are a cheaper form of storage than our brain (energy-wise). When we write things down, we permit ourselves to forget things. The unburdened mind could then give complete attention to the task at hand . I found out that I could work with 5-6 projects during the day without feeling too stressed (2-3 is the comfortable zone). The only trick was to treat me as a very dumb goldfish and write things down in a journal as soon as possible . This way, at any moment in time during the workday, I'll have most of the brain capacity free for focusing on the task at hand. Notes don't have to be verbose . A short sentence and a chat screenshot can be all it takes to get current context out of my mind and switch to a new task with full dedication.