Shipping Software Products

Shipping Software Products Hi! I'm Rinat Abdullin. Through my software career I have been involved in everything from warehouse management to social platforms and international logistics. These days customers come to me for technical advice on shipping LLM-driven products faster . See About me for more details or check out BitGN - a platform for agent benchmarks & challenges. I work worldwide from Vienna, Austria 🇦🇹. My newsletter " ML Under the Hood " talks about ChatGPT and building ML-driven software products. It covers important news, technical insights, and interesting product cases. Want to get started with ChatGPT/LLM? Check out ChatGPT quick-start guide for developers ! It is concise and beginner-friendly. Blog Sections 🔥 Schema-Guided Reasoning (SGR) - improve LLM accuracy by forcing it to think according to the predefined plan. Shipping products with LLMs and ChatGPT - answers to the frequently asked questions (also benchmarks) ♻ Event Sourcing - event sourcing and CQRS tutorials and explanations. 🤔 Opinionated Tech - thoughts about software design and tech. 💊 Simulation - dive into deterministic simulation of actors, clusters and logistic systems. It is awesome! 🦾 Robotics - story about trying to build a robot hand. 📦 SkuVault - evolving and scaling online warehouse management system (C# .NET and a lot of event sourcing at scale). 👍 HappyPancake - scaling the largest dating website in Sweden. 🎙 Being the Worst - adventures in practical software design with Kerry and Rinat. Blog Updates 🔥 New experiment in 2026 How to talk to your knowledge base? My team has no experience with ML/GPT. How do we proceed? How to get into ML for a developer? ChatGPT quickstart for developers How to segment texts for embeddings? Better performance with DOD Black Friday Experiment - Report 1 You can find my older posts in the archive .


2015 Retrospective

2015 Retrospective Our family almost doubled in size in 2015: we took a pug puppy in May and our daughter was born in July (events are ordered chronologically and not by importance). This was a wonderful experience which completely changed the way we perceive life, spend days and nights. As a side effect, time became a very limited commodity for us, continuous sleep - a fairy tale from a long ago. Time constraints influenced my approach to software development and continuous learning - there was much less coding in spare time, but more reading, learning and improving existing skills. SkuVault At SkuVault we are gradually maturing V2 project branch. It is in closed beta right now (full release of its features is expected in Q1'16). However, some ideas that proved themselves in V2 are already being taken back into V1. While working on V2 we also have to maintain V1 and all systems in between. So I had to learn bits of: InfluxDB time-series database (which felt like a more robust alternative to Carbon/Graphite for real-time metrics at a scale); R and plenty of libraries in the ecosystem - for performing data-driven research for devops, development and marketing; JavaScript with ES6, ReactJS, Webpack and all the joy of building interactive frontends without ASP.NET and IIS; even more Linux (mostly Ubuntu) for managing devops infrastructure. In the long term, I look forward to the slow (because we have to be extra careful) process of scaling SkuVault, while adding more features and unhooking it from Azure. Plus, there is a lot of work to be done with capturing data coming from SkuVault, analyzing it and leveraging to improve the product further. Cooking I think, I perfected our cheese cake recipe (using ingredients still available in Russian stores despite the sanctions) and got hang of making reasonable steaks in our electrical oven. I'd love to try making some cured meat in 2016. Erlang I invested time in learning me some more Erlang . It is a very nice language with a lot of value in OTP libraries. I don't think, I'll get to apply Erlang practically any time soon, however lessons already affected they way I think, reason and design distributed systems in the other languages. Emacs and Org-mode I believe, I became slightly better at managing everyday flows of tasks, problems and information in an unpredictable environment. All credit goes to Time Management for System Administrators by Limoncelli, Moleskine notebook and awesome org-mode in Emacs. Here's the layout: A bunch of org-mode files contains all my notes, tasks, projects and calendar entries (including recurring events and routines). I try to review and rectify them on a regular basis. All work hours are tracked in org-mode as well, they are used for invoicing. I use outstanding org-mode agenda to get a high-level overview of current projects and planned tasks, while planning my next day in Moleskine. Things are crossed out from the Moleskine list as I complete them. Unfinished tasks are moved to the next day or scheduled further down the road via org-mode. Any new information coming in (e.g. "buy diapers tomorrow" or "plan driving school next month") is logged immediately either to voice recorder app or in notebook, to get them out of my mind. As a part of my daily routine (I'm still trying to make it a routine), I incorporate them into my digital notes and agenda. Stand-alone projects get their own org files. They are also added to the global agenda list. This way, if any of the files contains tasks or scheduled entries, they would show up in the agenda views. org-mode has also a great support for literate programming, making it trivial to intermix executable code with prose. This affected the way I write about programming and learn. My Emacs config and Erlang Basics are both examples of the literate programming. I also use Emacs Speaks Statistics with org-mode and R to perform documented and reproducible data research. As you can, I'm spending a lot of time in Emacs. Almost all of the development (except for .NET) happens in there as well. Mastering this environment and tuning it, makes me a better developer, so I'm definitely going to continue this trend in 2016. JVM and Clojure .NET is a nice runtime with a decent ecosystem. It currently is the development tool I'm most familiar with. However, it misses some features and libraries which are available in Java world (e.g. Apache Kafka consumer library, proper bindings for LevelDB/RocksDB, libraries for sending real-time tagged metrics etc). Besides, managing servers at scale is more pleasant with Linux than with Windows (and Java is a first-class citizen there). My previous attempts to get into Java world were spoiled by C# and F#, which are nice .NET languages, one with strong support for object-oriented programming, the other with an emphasis on functional programming style. Compared to these primary .NET languages, it felt that Java 8 is too verbose, while Scala felt too complex and slow to compile (even with background compilation daemon). However, this year I finally paid attention to Clojure, which is a dialect of Lisp developed for JVM and JS by Rich Hickey. Clojure places a strong emphasis on simplicity, functional programming, immutability and interoperability with existing Java ecosystem. There is also a version of Clojure for CLR which isn't as popular as JVM or JS versions. In fact, if you look beyond the funny lisp syntax with tons of brackets, you could encounter simplicity of golang (with go-like channels and light-weight processes), backed up by JVM ecosystem and templating capabilities of LISP. ( defn old? [blob period] ( let [ date ( c/from-date ( . getLastModified ( . getProperties blob ))) limit ( t/minus ( t/now ) period ) ] ( t/before? date limit ))) Yes, the syntax is very funny and unusual. This is because code is data in Lisp (and I'm still grasping all implications of that). I'm not sure if I will end up using Clojure in my everyday work, however learning it will make me a better developer. I'm sure of it. It is a nice coincidence that Emacs itself is an interpreter for a different flavor of Lisp: ELisp. Mastering Clojure would make it easier to customize Emacs itself. There are a few tricks with Clojure macros and evolving neural networks that I want to give a shot to :) Internet of Things When Maya was born, we realized that we need to measure temperature and humidity around her crib. This became especially important during the winter period. "Thanks" to the central heating and cold winters, humidity could easily drop to 10% (very bad for the health of the family). I started playing with DHT22 temperature/humidity sensor and Arduino/ATTiny controllers to capture the data. However, it turned out that reading data from OLED display in an inconvenient location is no fun. A few iterations down the road I discovered EP8266 controller - a tiny chip smaller than a quarter. It has a Wi-Fi module and a programmable micro-controller. That baby has everything needed to grab temperature/humidity from the sensor and dump it to InfluxDB time-series database via our Wi-Fi network. Then it all gets charted on a Grafana dashboard. Obviously, this is just a start. There are a few other things I could do to improve our home IT infrastructure (e.g. a better control of the music player daemon, better tracking of sleep patterns of our baby etc). I believe, this year could see a few more hardware projects, more related to everyday programming. The End I didn't achieve much in the area of software development in 2015, rather just discovered things I want to learn more about in 2016. This already makes 2016 a more interesting year :) Unfortunately, some areas of my life were sacrificed in 2015 to free up room for work: writing, blogging and sports. I hope to bring them back to life in the upcoming year.


2024-webinar-ai-assistants-4

2024-webinar-ai-assistants-4 Вебинар “Проектируем и внедряем AI-ассистенты с LLM под капотом” Для кого: для инженеров, которые руководят созданием подобных ассистентов. А еще - для продактов и сейлзов, которым нужно понимать возможности и ограничения подобных продуктов. О чем: Мы посмотрим на работу и реализацию одного такого ассистента на конкретной задаче в отдельной предметной области. Это будет end-to-end кейс с реальными документами. Помимо кода, мы обсудим с вами принципы построения и внедрения подобных систем, от первого AI навыка до создания платформ. Поговорим про контроль качества, сбор данных и построение работы с пользователями. Расскажу про Knowledge Maps для адаптирования ассистентов с LLM под капотом к работе в конкретных доменных областях. Время проведения (московское): 7 марта, четверг, 13:00 GMT+3 (места закончились) 9 марта, суббота, 15:30 GMT+3 (места закончились) 5 апреля, пятница, 11:00 GMT+3 (места закончились) 25 апреля, четверг, 11:00 GMT+3 (места закончились) Update : места в последний поток закончились. Если вы не попали на вебинар - не расстраивайтесь! В дальнейшем я планирую записать лекционную часть вебинара, без обсуждения кейсов участников, специально для более широкой аудитории и на английском языке. Ее можно будет купить отдельно. Длительность - 2 часа. Язык - русский. Что нужно для участия? Для участия вам понадобится телеграм-аккаунт: вся основная информация, вопросы, обсуждения будут в телеграм-чате потока. После оплаты вам придет персональный инвайт в чат. Вебинар будет проводиться в Zoom. Будет ли запись? Запись будет доступна участникам вебинара только для персонального использования - мы обсуждаем рабочие кейсы, и не все участники хотели бы ими делиться на публику. Пожалуйста, не делитесь этой записью. Участники получат запись вебинара, презентацию, ноутбуки из демо-кейса и архив с документами для задачи, которую мы будем рассматривать. Можно ли купить запись вебинара отдельно? Конкретно эта запись продаваться не будет - чтобы можно было свободно обсуждать рабочие кейсы на вебинаре. В дальнейшем я сделаю отдельную запись лекционной части этого вебинара, специально для более широкой аудитории. Хочу, чтобы мой кейс разобрали на вебинаре, что нужно сделать? После оплаты вебинара вам на емейл придет приглашение в чат потока - можете прислать кейс туда, я прочитаю его заранее и постараюсь на вебинаре или в чате рассказать, как сам решал бы подобные задачи. Если кейсов будет много, то на вебинаре выберу несколько показательных или кластеризую похожие кейсы. Можно ли оплатить участие другого человека? Да, только обязательно укажите его емейл при оплате. После оплаты вам придет email со ссылкой на закрытый телеграм-чат потока. Если вдруг по какой-то причине емейл не пришел в течение 30 минут, пишите мне на rinat @ abdullin.com (но сначала, как водится, проверьте папку “спам”). Совершая оплату, вы соглашаетесь с условиями проведения вебинара (Terms & Conditions).


404 - Not Found

404 - Not Found I'm really sorry about that. You arrived at a page that was moved to another location or dropped completely for being no longer relevant. Ping me on twitter and I'll locate the article for you. Or just check out the archive .


Rinat Abdullin

Rinat Abdullin LinkedIn | Twitter | Github | rinat @ abdullin . com Hello! I'm a technical advisor. I help ship LLM-driven products faster. You might know me from LLM Product Benchmarks , Schema-Guided Reasoning or from the Enterprise RAG Challenge . These days I focus on developing BitGN - a Platform for Benchmarks and Challenges around Autonomous Agents. I specialize in identifying patterns for AI solutions in business and driving them to practical application. I work with teams in EU and the USA, helping them to build LLM-driven business products and ship them faster. This is the current snapshot of successful and proven AI Cases that I currently have in my portfolio. Darker colors mean more cases. High-level perspective across industries and AI Cases gives me systemic insights into the approaches that work. I guide companies to leverage these insights, and they save months of effort on the way to market. If you have an interesting AI Case to complement this matrix or would like to know more about proven applications - feel free to reach out for a chat via rinat @ abdullin . com . I don't take any development projects . However, for large-scale enterprise projects I can refer to TIMETOACT GROUP Austria (and the entire international GROUP). I'm Head of ML and Innovation there, working to establish an institute of building LLM-driven business solutions that are based on relevant real-world practices. Longer Story I have been working in commercial projects since 2003. Through my career, I have been involved in various domains from warehouse management to social platforms to international logistics. This exposure allows me to understand business problems across domains. Helping to set up IT development offices for 4 different companies gave me a deeper insight into the challenges of organisational dynamics, leadership and building healthy teams. I have worked with software startups for the most of my career. This created a strong connection with product development, a focus on value delivery and adaptation to changing market conditions. Along the way, I have been lucky to learn from outstanding people and work with them on the exciting projects: contributed to the book on Implementing Domain-Driven Design ; designed architecture of a social network ; helped to deliver big data analytics for retail in the cloud ; helped to scale multi-channel inventory management software past 1.5B events; delivered real-time analytics for a social network ; helped to build a data science infrastructure for an international transport organization for the whole past year, I've been working on my AI Research: a series of interviews with companies on adopting LLMs for solving their business challenges. I used to blog and occasionally talk at the conferences about CQRS, Event Sourcing, Domain-Driven Design and scaling event-driven systems . You can check out my newsletter ML Under the Hood and latest blog posts to find things I'm passionate about. These days, I'm focused on Machine Learning. I help companies to build and ship ML-driven software products faster. If you want to talk, ask a question or work with me, don't hesitate to write me an email at rinat @ abdullin . com . I'm learning a lot from the smart and kind people around the world. I'd be glad to help out someone as well.


About This Site

About This Site This website started around year 2007 as a simple blog. It went through multiple changes since then. This is the story behind it. History Initially I hosted blob posts on open platforms like LiveJournal and Blogger . They tended to be limiting, clunky and with advertisements. At some point I discovered Linux for the first time and decided to give a try to WordPress . It worked out well for a couple of years, till I got tired from maintaining it. Some time later, commercial Squarespace platform attracted me with good performance and sleek designs. It was hassle-free and enjoyable for some time. At some point, I got tired with limitations of the platform and their pricing. Around the year 2014 I imported my posts to Octopress . Octopress is a blogging framework for Jekyll - static site generator powering GiHub pages. I loved the idea of static websites: low maintenance and high availability; great performance; hosting prices are negligible. However Octopress was slow and it was running on Ruby. I experienced occasional frustrating issues with runtime versions and ruby gems. Hugo While working on HappyPancake project, I discovered the ecosystem of Go. Go (or golang) is an opinionated language, introduced by Google in year 2007. It is [[L2 golang|simple, fast and compiles to native code]]. It wasn't long before Hugo got my attention. This static website generator was really fast, building full site out of my 200+ posts in less than a second. This was worth migrating to. Roughly at the same time I moved my site to the Amazon cloud for static hosting. Static website hosting requires resources to be pre-generated, but provides high availability at low cost. This can apply to the entire web site or to specific parts. The combination worked great. Fast site regeneration allowed have smooth feedback loop. HTML templates were a bit uncomfortable and limiting, but that is the nature of html/template in go. Custom Node.JS + React.JS In 2014, while working on front-end logic for 👍 HappyPancake , I discovered Node.js and React.JS. Node.js ecosystem is totally different from ecosystems of golang, .NET, Erlang or Haskell. It is build upon JavaScript, which is probably the most widely used language out there. There are a lot of bad features in the language and poor quality projects written in it. Yet, if you stick with the good parts of the language, nice packages and align your design with the ecosystem, you would get a brilliant set of tools for front-end development. I also loved templating capabilities of React.JS - a JavaScript Library for building web interfaces. It has a convenient way of expressing composable User Interface components in a single file. Here is an example of a simple page listing all posts within a tag: render: function () { var site = this.props.site; var title = "Articles in " + this.props.tag; var list = []; var posts = site.tags[this.props.tag]; lists.sortLatestFirst(posts).forEach(function(p) { list.push( <li key= {p.url} > <Permalink post= {p} /> </li> ); }); return ( <PostLayout title= {title} site= {this.props.site} nav= "archive" > <article> <ul> {list} </ul> </article> </PostLayout> ); } So I decided to replace Hugo with a hand-written static HTML generator, using markdown for content and React.JS with JSX for HTML. I purposefully misused ReactJS in this project for the benefit of the syntax. Normally ReactJS shines, when used with Flux-based architecture for building fast and responsive web applications. December 3 - New Version Deployed This week I pushed a new version of the web-site to production. It took roughly 2 weeks to replace Hugo with my own version, working in spare time. I simply picked existing node.js packages, wired them together and then rewrote Hugo HTML templates as ReactJS components. This was also an opportunity to improve web design. React has renderToStaticMarkup method which produces raw HTML without the attributes needed for the DOM magic. Here are some of the features that came for free with this approach. Fast generation Hugo generated HTML for my website in less than a second. It used all 4 cores of my MacBookAir. Golang works good for building utilities and servers which use multiple CPU cores efficiently. Node.js processes, on the other side, run on a single core by default. Build takes just 4 seconds, which came as a surprise. I use some brutal HTML validation and regex replacements on old posts. Without this logic, build would be faster. In-memory processing HTML processing happens in memory. In the start node.js loads all my source files into the virtual file system. All further operations would happen there. mock-fs package provided virtual file system capabilities. At the end, I either dump /build folder into the real filesystem for sync with Amazon S3 or launch a simple HTTP server that behaves exactly like Amazon. Since regeneration is fast, and it doesn't write thousands of files to my SSD, I can watch the input directory for changes and rebuild on every save. Continuous compilation makes writing flow more convenient. No vendor lock-in If, at some point, I decide to go away from Amazon, it would require a little effort. Nginx with a simple configuration could host the entire website in exactly the same way. Code snippets are compiled I added syntax for extracting named code snippets from source files and injecting them into the HTML. They are auto-indented and then highlighted with highlight.js . Obviously, build script lints, compiles and tests source files, ensuring that they have consistent quality. This process is fast for golang, which is one of the reasons I started using it for new snippets. HTML-specific processing Node.js has an ecosystem for dealing with web resources. I use it for: HTML minification CSS linting and minification Atom Feed generation Markdown rendering Semantic HTML5 New layouts use semantic HTML5 elements like aside , header , footer or section . Not only this helps with the design, but also makes website friendly for search engines. Publishing Features There also are some miscellaneous features related to publishing. They were either easy to add or came for free. Build process extracts all TODO comments across all content, updating a simple org-mode outline file. I can easily see what needs to be done next . There are some broken links in my old content, build process finds them and lists by priority. Current UI layout forces new content to be book-ready. In fact, new content could be converted from the sources into O'Reilly AsciiDoctor format or transformed to Epub directly. Some HTML templates display the most relevant articles to read after the content. If relevant articles aren't provided manually, build process computes them. Org-mode is an Emacs extension for managing outlines, tasks, documents and agendas. It was the initial reason why I switched to Emacs for all my work. I also introduced additional constraints for the new prose. For example, all essays should have a central idea that they focus on (build will fail if that property is not provided). This idea is then used in description HTML header or in tweet summary, should you click Discuss button by the post header. if ( form === "essay" ) { if ( idea === undefined ) { throw new Error ( "Essays MUST specify an idea they focus on." ); } } else if ( form === "bio" ) { /* ... */ } Different forms of prose have different constraints. To be continued... The blog engine gets another rewrite to Python in 2020 .


2024-ai-assistants-course-landing

2024-ai-assistants-course-landing Курс в записи “LLM под капотом: выбираем эффективные технические решения для AI-ассистентов” О чем этот курс · Для кого? · Курс будет полезен · Кому курс не подойдет? · Структура · Комьюнити курса · Формат курса · Стоимость · Вопросы и ответы · Купить Update: a revised english version of this course was published in April 2026: building-ai-assistants-course . О чем этот курс Этот курс о том, как обоснованно выбирать эффективно реализуемые технические решения продуктовых задач на базе LLM в различных доменных областях так, чтобы они: давали стабильное качество; были контролируемыми и тестируемыми; развивались без деградации при расширении функциональности. Главная проблема большинства LLM-ассистентов в продакшене - качество часто получается нестабильным и плохо управляемым. Появляются галлюцинации, рассыпается формат, ответы становятся непредсказуемыми. В курсе мы разберем, почему это происходит, и как строить решения, которые работают надежно. Я покажу вам подход к проектированию и диагностике LLM-систем на практических кейсах внедрений. Отдельный акцент курса — SGR (Schema-Guided Reasoning) : подход, который помогает управлять рассуждением модели через структурированный вывод, чеклисты и этапы самопроверки. Это повышает предсказуемость, снижает «творческую самодеятельность» модели и упрощает автоматическую проверку результата. Методология обучения основана на моем подходе в консалтинге и кейсах успешных внедрений AI. Купить курс Для кого? Этот курс для тех, кто уже разрабатывает продукты с LLM самостоятельно или в составе команды, и сталкивался с типовыми вопросами: почему качество нестабильно и «то работает, то нет»; как перестать лечить всё промптами; какая модель лучше всего подойдет для конкретной задачи; как сохранить качество при добавлении новых возможностей. Мы будем учиться проектировать решения для продуктов с LLM под капотом так, чтобы качество было воспроизводимым: снижать галлюцинации через структуру и проверки, выбирать архитектуру под задачу и выстраивать процесс улучшений через набор паттернов, которые делают поведение модели предсказуемым и пригодным для продакшена. Вместо того чтобы переизобретать решения с нуля, вы получите библиотеку проверенных паттернов и подходов, которые можно сразу применять и адаптировать под свою задачу - экономя бюджет и время. Курс будет полезен: инженерам - увидите разбор диагностики качества на реальном сценарии, получите набор практик для управляемых пайплайнов, включая Schema-guided Reasoning; техлидам / CTO - получите набор паттернов, из которых складываются устойчивые архитектуры, разберетесь, как масштабировать решения без сюрпризов; продактам - чтобы выбирать жизнеспособные LLM-сценарии, задавать измеримые критерии качества и формулировать требования так, чтобы команда предсказуемо доводила фичи до продакшена; фаундерам - получите ориентиры по выбору решений на MVP и при развитии продукта: где обычно возникают проблемы качества, как закладывать устойчивую траекторию развития и какие паттерны дают быстрый прогресс. Непосредственно опыт программирования для курса не обязателен, но понимание принципов разработки важно. Кому курс не подойдет? Тем, кто только начинает и пока не строил никаких LLM-решений. Тем, кого интересуют ассистенты исключительно на локальных моделях (в курсе акцент на инженерных принципах и паттернах, применимых везде, но это не курс по локальной инфраструктуре). Тем, кто ожидает туториал «как подключить LangChain/LlamaIndex и индексировать документы». Курс не про фреймворки, а про архитектурные решения, качество и контроль информационного потока. Структура курса Модуль 1: Основы Разбираем типичный сценарий «ассистент по документам», воспроизводим проблемы и учимся находить основные причины ограничений. Описание структуры курса Путь к галлюцинациям - воспроизводим проблему RAG-ов на годовых отчетах компаний Как по годовым отчетам компаний отвечают лучшие RAG-и и в чем их ограничения? Минимальное воспроизведение проблемы в один промпт LogProbs - инструмент анализа галлюцинаций Паттерны галлюцинаций LLM Первопричины проблем в RAG-ах Как избежать тупика? Приходим к простому решению Модуль 2: Кейсы и паттерны Разбираем повторяющиеся архитектурные паттерны из кейсов успешных внедрений проектов c AI. Prompt и Query Expansion Prompt 2 Knowledge Base Learn from Feedback Searching for data Sandbox и практические задания Router - когда нужна специализация Instruction Distillation Dedicated Agents Workflow Human in the Loop Structured Data Extraction и визуализация ошибок REPL (Self-review) Checklists и Prompt Caching Structured Outputs & Custom CoT - Deep Dive Для каждого паттерна я показываю на кейсах, как формулируется задача, какие ограничения проявляются в реальной системе, где именно появляются ошибки качества, и какие решения дают устойчивый эффект в продакшене. Комьюнити курса После покупки вы получаете персональный инвайт в закрытое сообщество курса. Это рабочее пространство, где можно задавать вопросы по материалам курса, разбирать свои кейсы, обсуждать инженерные и продуктовые решения. За время жизни курса в сообществе собрались практики, которые строят и внедряют LLM-решения: инженеры и техлиды, CTO, фаундеры, руководители DS/ML-команд, директора Gen-AI направлений. Опыт - из разных контекстов: финтех и банки, HR-tech, B2B SaaS, контакт-центры и sales-аналитика, медтех, enterprise-автоматизация, документные пайплайны, knowledge-боты по внутренней документации, VC-аналитика. Формат курса Формат : видеолекции, разделенные по темам, с навигацией; дополнительные разборы, практические примеры и иллюстрации. Длительность видеолекций : около 4 часов. Время прохождения курса : self-paced; для вдумчивого изучения требуется в среднем неделя времени. Язык : русский. Доступ к курсу : насовсем, включая обновления материала. Оплата : только зарубежной картой. Доступ к комьюнити курса : персональный инвайт в Телеграм-чат, где можно будет задавать вопросы мне и общаться с другими специалистами, проходящими курс. Купить курс Стоимость Курс можно оплатить любой карточкой, которая работает зарубежом и принимается Stripe. Закрывающие документы генерируются при оплате и придут вам на указанную электронную почту. Компании при оплате могут указать свой billing address и EU VAT ID. Принимающее оплату юрлицо зарегистрировано в EU. Личный доступ: 1 место на курсе Курс в записи с обратной связью в чате поддержки: €258.00 (VAT included). Командный доступ: 5 мест на курсе для команды из одной организации Курс в записи с обратной связью в чате поддержки: EUR €1290.00 (VAT included). Вопросы и ответы Я работаю в Европе / США / РФ, подойдет ли мне этот курс? Да, принципы и паттерны решений основаны на статистике успешных внедрений кейсов AI в разных странах и отраслях. На курсе мы разберем как повторяющиеся архитектурные паттерны, так и особенности их применения в разных кейсах. Что такое паттерны Case/Solution? Я проводил AI Research, в ходе которого пообщался с десятками компаний из разных областей. Это позволило мне выявить паттерны проблем и решений, в которых можно применить LLM в разных доменах на различных задачах. Об этих паттернах я рассказываю на курсе, и показываю на реальных примерах, как пользоваться библиотекой паттернов применения LLM для поиска оптимально работающих решений. Вместо того, чтобы придумывать (и тестировать!) свои решения, можно применить Pattern Matching и переиспользовать проверенный опыт других. Нужен ли опыт программирования? Непосредственно писать код самому не обязательно. Курс ориентирован на проектирование и принятие решений. Инженеры смогут применять материал сразу в коде. Есть ли практическая часть? Курс лекционный: видео в записи и методические материалы. Я показываю практические кейсы, разбираю решения для них и даю инструментарий для выбора оптимального решения. Используя библиотеку паттернов, о которой я рассказываю на курсе, вы сможете научиться делать это самостоятельно. В курсе есть два практических задания - если вы не пишете код, можете смело пропускать их. Можно ли оплатить участие другого человека? Да, только обязательно укажите его емейл при оплате: доступ на курс и в чат поддержки будет персональным. Можно ли оплатить курс криптовалютой, PayPal или переводом на карточку? К сожалению, нет. Оплата возможна только международными картами и средствами платежа, которые принимает Stripe. Остались вопросы? Пишите мне в телеграм или на почту. Купить курс Купить можно здесь: https://labs.abdullin.com/courses/ai-assistants-ru?a=unlock Курс расположен на моей площадке Abdullin Labs, перед покупкой нужно будет завести там аккаунт для доступа к курсу.


AI+Coding Kata

AI+Coding Kata AI in Coding helps a lot with high-level tasks like prototyping, reasoning and finding bugs. Please use your favourite tools (no limitations) to implement as much of this spec as possible in a language of your choice. This should be a parser that can parse any document in this spec. Imagine that your team will have to support this code for a few years, so you want to do a thorough job here. There are no limits here. If you spot something unusual - use your best judgement. BizDocumentAI Spec Let's define a simple document format that could describe a contract, procedure, or any other business document in a structured way. It may be used to load this business data into AI Assistants (like in Enterprise RAG Challenge ). We’ll work with the documents. Our documents will consist of blocks . A block is a logical piece of text (like a paragraph). It can optionally have a head , number , and body . A block’s body can contain: Another block Text A list A dictionary Blocks can contain heterogeneous content—texts, other blocks, dictionaries, etc. Lists can contain only similar block items that also have a number . Document Layout The document below describes a simple text format that can be deterministically parsed into JSON objects. This document is also a test suite ! Code admonitions always come in pairs: first input and then json . When the parser is implemented, parsed input should always produce output that is structurally similar to the expected JSON. The headline before the code blocks is the name of the text. Python Data Structures Below is an example of how you might structure your data models in Python using Pydantic: from typing import List , Optional , Union , Dict , Literal from pydantic import BaseModel , Field # This type alias helps with readability and forward references. ContentNode = Union [ str , "Block" , "ListBlock" , "Dictionary" ] class Dictionary ( BaseModel ): """ A distinct dictionary structure for key-value pairs. """ kind : Literal [ "dict" ] items : Dict [ str , str ] = Field ( default_factory = dict ) class Block ( BaseModel ): """ A general-purpose container for a 'section' or item. - 'number' can store a section number (e.g., "5", "5.1") if applicable. - 'head' is an optional heading for the block. - 'body' can hold any mix of strings, sub-blocks, dictionaries, or lists. """ kind : Literal [ "block" ] number : Optional [ str ] = None head : Optional [ str ] = None body : List [ ContentNode ] = Field ( default_factory = list ) class ListBlock ( BaseModel ): """ A container for a list of items, each item being a 'Block'. """ kind : Literal [ "list" ] items : List [ Block ] = Field ( default_factory = list ) # Important for forward references within union types Block . model_rebuild () Specifications Empty text Empty text results in an empty document block. Input : (there is no content) JSON : { "kind" : "block" } Body Plain text goes into the block body straight away. Different paragraphs are separated by new lines. Input : First paragraph. Second paragraph. JSON : { "kind" : "block" , "body" : [ "First paragraph." , "Second paragraph." ] } Note that we strip and skip empty lines! Input : First paragraph. Second paragraph. (An empty line in between) JSON : { "kind" : "block" , "body" : [ "First paragraph." , "Second paragraph." ] } Head Text marked with <head> goes directly into the head of the current block. Input : <head> Test Document </head> Content JSON : { "kind" : "block" , "head" : "Test Document" , "body" : [ "Content" ] } Blocks You've seen that the document is parsed into a root block . But everything is a block, and blocks can be nested explicitly: Input : <head> AI Coding Kata </head> Let's get started with the kata <block> <head> Preface </head> Here is a little story </block> JSON : { "kind" : "block" , "head" : "AI Coding Kata" , "body" : [ "Let's get started with the kata" , { "kind" : "block" , "head" : "Preface" , "body" : [ "Here is a little story" ] } ] } Dictionaries Dictionaries are used to capture key-value pairs. By default, they are separated by : . Input : <dict sep= ":" > Key One: Value One Key Two: Value Two Key Three: Value Three </dict> JSON : { "kind" : "block" , "body" : [ { "kind" : "dict" , "items" : { "Key One" : "Value One" , "Key Two" : "Value Two" , "Key Three" : "Value Three" } } ] } We can also have a non-standard separator and empty values: Input : <dict sep= "-" > Title - AI Coding - for TAT Kata Number - </dict> JSON : { "kind" : "block" , "body" : [ { "kind" : "dict" , "items" : { "Title" : "AI Coding - for TAT" , "Kata Number" : "" } } ] } Lists Lists are very important! By default, each non-empty line is a list item. They go inside the root block. There are multiple kinds: . for ordered lists that are dot-separated * for bulleted lists Note that the list item’s text goes into head and the item number goes into number . Ordered Lists Input : <list kind= "." > 1. First 2. Second </list> JSON : { "kind" : "block" , "body" : [ { "kind" : "list" , "items" : [ { "kind" : "block" , "number" : "1." , "head" : "First" }, { "kind" : "block" , "number" : "2." , "head" : "Second" } ] } ] } As a convenience, nested lists are automatically detected: Input : <list kind= "." > 1. First 2. Second 2.1. Subitem 1 2.2. Subitem 2 </list> JSON : { "kind" : "block" , "body" : [ { "kind" : "list" , "items" : [ { "kind" : "block" , "number" : "1." , "head" : "First" }, { "kind" : "block" , "number" : "2." , "head" : "Second" , "body" : [ { "kind" : "list" , "items" : [ { "kind" : "block" , "number" : "2.1." , "head" : "Subitem 1" }, { "kind" : "block" , "number" : "2.2." , "head" : "Subitem 2" } ] } ] } ] } ] } Unordered lists We can have unordered lists too: Input : <list kind= "*" > • First • Second • Third </list> JSON : { "kind" : "block" , "body" : [ { "kind" : "list" , "items" : [ { "kind" : "block" , "number" : "•" , "head" : "First" }, { "kind" : "block" , "number" : "•" , "head" : "Second" }, { "kind" : "block" , "number" : "•" , "head" : "Third" } ] } ] } And nesting can be done with "o" : Input : <list kind= "*" > • First o Subitem • Second • Third </list> JSON : { "kind" : "block" , "body" : [ { "kind" : "list" , "items" : [ { "kind" : "block" , "number" : "•" , "head" : "First" , "body" : [ { "kind" : "list" , "items" : [ { "kind" : "block" , "number" : "o" , "head" : "Subitem" } ] } ] }, { "kind" : "block" , "number" : "•" , "head" : "Second" }, { "kind" : "block" , "number" : "•" , "head" : "Third" } ] } ] } Mixed lists We can mix lists, but we need to designate different types separately with tags. Input : <list kind= "." > 1. Beginning 2. Main 2.1. Subsection <list kind= "*" > * Bullet 1 * Bullet 2 </list> 3. Ending </list> JSON : { "kind" : "block" , "body" : [ { "kind" : "list" , "items" : [ { "kind" : "block" , "number" : "1." , "head" : "Beginning" }, { "kind" : "block" , "number" : "2." , "head" : "Main" , "body" : [ { "kind" : "list" , "items" : [ { "kind" : "block" , "number" : "*" , "head" : "Bullet 1" }, { "kind" : "block" , "number" : "*" , "head" : "Bullet 2" } ] } ] }, { "kind" : "block" , "number" : "3." , "head" : "Ending" } ] } ] } Lists with content Lists can also have additional content. If something in the current list doesn't match the prefix, then it is treated as a block body : Input : <list kind= "." > 1. First First body 2. Second Some more text <dict sep= ":" > Key: Value Another Key: Another Value </dict> </list> JSON : { "kind" : "block" , "body" : [ { "kind" : "list" , "items" : [ { "kind" : "block" , "number" : "1." , "head" : "First" , "body" : [ "First body" ] }, { "kind" : "block" , "number" : "2." , "head" : "Second" , "body" : [ "Some more text" , { "kind" : "dict" , "items" : { "Key" : "Value" , "Another Key" : "Another Value" } } ] } ] } ] }


ai-research

ai-research Call for Research: Identifying patterns for AI solutions in business Duration : 45-minute focused session Your input : highlight your key business challenges and review potential AI applications together My role : provide expert AI insights and potential solutions Outcome : collaborative exploration to identify practical AI applications for business Let's Explore AI Innovations Together Throughout my career, I have been involved in various domains from warehouse management to social platforms to international logistics. This exposure allows me to understand business problems across domains. In the past year, while working as a consultant, I’ve been gaining more and more understanding on how to power up businesses, solving their key challenges with the benefits that AI has to offer. If you want to read more about me and my expertise: this is my website and my newsletter where I from time to time publish my AI-driven cases and solutions. The Core of Our Research My current research revolves around the process of AI adoption in various business sectors. The goal is to identify patterns of opportunities for custom AI solutions. This involves understanding specific industry challenges, potential AI applications, and the impact of these technologies on existing workflows. Is This Research Right for You? This initiative is designed for decision-makers who are actively shaping the IT and AI landscape of their organizations. Ideal collaborators are those in roles such as Senior Product Managers, CTOs, CEOs, Founders, or other high-level positions where strategic decisions about technology are made. If you're someone who can drive change and implement innovative IT solutions in your business, your insights and participation would be highly valuable in this research. I'm looking for thought leaders eager to explore AI's potential and impact at the top levels of business decision-making. I focus on the cases of AI adoption in EU and USA businesses. Let's Collaborate on AI Research I'm looking for businesses interested in exploring AI possibilities within their domains. Our collaboration will include a focused 45-minute call to exchange thoughts on potential AI applications in your industry. I'll share insights from similar cases, discuss common pitfalls, and explore potential shortcuts that can potentially be applicable as your solutions. My approach is about partnership and mutual learning – I believe that the best way to understand and leverage AI is by sharing experiences and ideas. Let's explore the possibilities of AI together. Your Unique Perspective In return for sharing my insights, I'm eager to hear about the specific challenges and AI adoption hurdles your business might be facing. Your input is invaluable in shaping the direction of this research. If your case is part of a recurring pattern observed at different companies, I might use it for a generalized analysis in my posts or newsletter articles. Client names and sensitive details are always kept confidential, ensuring that your proprietary information remains secure. Ready to Dive into AI Exploration? If you believe we could work well together, please complete the form below. I will review each submission, then reach out to discuss more and plan a joint session. Expect a reply from me within a few days. Form is loading…


Analyze website logs with clickhouse-local Lean web analytics (see Lean web analytics ) has been in place on my website for a couple of days. It doesn't share data with third parties, doesn't track individual users, and is treated nicely by all adblockers. It tries to get away by: running the website as a single-page application annotating each normal request with page number and page time. Let's see if we can get any useful insights out of the limited dataset. It would be a waste of time to move forward without proving that the approach works. Fail fast or succeed fast. Given the output log in Caddy structured log format, what is the shortest path to experiment with the data? We need to: convert JSON to some table format; use a fast database engine capable of SQL, because nothing beats SQL; plot results on a chart. Let's skip the " production-worthy " setup for the time being. No streaming and real-time updates. Just batch processing and a few utilities: jq , Clickhouse, and Jupyter notebooks to connect these two. Tools Here is a list of tools that I always have installed on my dev machine for cases like that: jq - command-line utility for JSON data Jupyter - interactive development environment for code and data Clickhouse - fast column-oriented database. It also has a clickhouse-local utility for manipulating data. Along the way, we'll use a few "must-have" Python libraries: pandas - data analysis library matplotlib - visualization with Python plotly - another graphing library for Python Workflow First, use jq to convert a structured log.json to a clean.tsv : FORMAT = """[ .ts, .request.method, .request.uri, .request.headers.Referer[0?], .request.headers[ \\ "X-Page-Num \\ "][0?], .request.headers[ \\ "X-Page-Sec \\ "][0?] ] | @tsv """ ! jq -- raw - output - r - c "$FORMAT" log . json > clean . tsv Note: this is a python code running in Jupyter. Lines that start with ! are executed as shell commands by the environment. Then, use Clickhouse in local mode to load clean.tsv , run a query, and print the results. For example, to display TOP 10 visited pages: SCHEMA = """ CREATE TABLE table ( ts Float32, method Enum8('GET' = 1, 'HEAD' = 2, 'POST'=3), uri String, ref String, num UInt8, sec UInt8 ) ENGINE = File(TSV, 'clean.tsv'); """ QUERY = """ SELECT uri, COUNT(uri) AS visits FROM table WHERE endsWith(uri,'/') GROUP BY uri ORDER BY visits DESC LIMIT 10 FORMAT Pretty """ ! clickhouse - local -- query "$SCHEMA;$QUERY" The output would be like that (thanks to FORMAT Pretty ): ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┓ ┃ uri ┃ visits ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━┩ │ /beautiful-tech-debt/ │ 1337 │ ├───────────────────────────────────────────┼────────┤ │ / │ 650 │ ├───────────────────────────────────────────┼────────┤ │ /lean-web-analytics/ │ 342 │ ├───────────────────────────────────────────┼────────┤ │ /designing-privacy-first-analytics/ │ 147 │ ├───────────────────────────────────────────┼────────┤ │ /post/dddd-cqrs-and-other-enterprise-... │ 46 │ ├───────────────────────────────────────────┼────────┤ │ /about-me/ │ 46 │ ├───────────────────────────────────────────┼────────┤ │ /my-productivity-system/ │ 42 │ ├───────────────────────────────────────────┼────────┤ │ /post/when-not-to-use-cqrs/ │ 41 │ ├───────────────────────────────────────────┼────────┤ │ /cqrs/ │ 40 │ ├───────────────────────────────────────────┼────────┤ │ /archive/ │ 37 │ └───────────────────────────────────────────┴────────┘ To render a chart, save query results as a file that matplotlib will render: QUERY = """ SELECT uri, COUNT(uri) AS visits FROM table WHERE endsWith(uri,'/') GROUP BY uri ORDER BY visits DESC LIMIT 10 FORMAT CSVWithNames; DROP TABLE table """ data = ! clickhouse - local -- query "$SCHEMA;$QUERY" > tmp . csv df = pd . read_csv ( 'tmp.csv' ) plt . pie ( df [ 'visits' ], labels = df [ 'uri' ]) plt . title ( "TOP 10 pages" ) plt . show () This should render something like that: And here is how it looks all together in Jupyter interface: The amazing part of clickhouse-local - it runs a full column database engine in a single command-line request . You could use Clickhouse queries to build advanced transformations on top of the plain-text files. For example, to display daily visits to the most popular pages: CREATE TEMPORARY TABLE top_pages AS SELECT uri FROM table WHERE endsWith ( uri , '/' ) GROUP BY uri ORDER BY COUNT ( uri ) DESC LIMIT 9 ; SELECT date_trunc ( 'day' , toDateTime ( ts )) as date , uri as page , COUNT ( * ) as visits FROM table WHERE uri in ( SELECT uri from top_pages ) GROUP BY date , uri FORMAT CSVWithNames Then pivot the results with pandas and display as area chart: df = pd . read_csv ( 'tmp.csv' , parse_dates = [ 'date' ]) . set_index ( 'date' ) dfp = df . pivot_table ( index = df . index , columns = 'page' , values = 'visits' , aggfunc = 'sum' ) dfp . plot . area ( figsize = ( 12 , 6 )) plt . title ( "Daily visits to most popular pages" ) plt . show () To create a sankey diagram, use plotly library: QUERY = """ SELECT ref as src, uri as trg, num, COUNT(*) as count FROM table WHERE num > 0 GROUP BY ref, uri, num FORMAT CSVWithNames """ ! clickhouse - local -- query "$SCHEMA;$QUERY" > tmp . csv df = pd . read_csv ( 'tmp.csv' ) labels , sources , targets , values = {},[],[],[] # brutally simple processing below, will not scale def index ( s ): if s not in labels : labels [ s ] = len ( labels ) return labels [ s ] for i , row in df . iterrows (): count = row [ 'count' ] if count == 1 : continue num = row [ 'num' ] src = f " { row [ 'src' ] } : { num - 1 } " trg = f "https://abdullin.com { row [ 'trg' ] } : { num } " sources . append ( index ( src )) targets . append ( index ( trg )) values . append ( count ) fig = go . Figure ( data = [ go . Sankey ( node = dict ( pad = 15 , label = list ( labels . keys ()), color = "#cd2e29" ), link = dict ( source = sources , target = targets , value = values ) )]) fig . update_layout ( title_text = "User flow" , font_size = 12 ) fig . show () I don't have enough visits to render a nice chart (most visitors drop off after the first page), so here is what we have: Summary The approach looks good enough for my purposes. Small gotchas: some requests come without a referrer field, which messes up the sankey diagram; not much multi-step data on my website, so it is hard to tell where this stops scaling; it is hard to tell bots from the real people; perhaps should try using User-Agent field. All in all, the experience is quite nice. More Reading Retrospective on a project in building a lean platform for real-time analytics: Real-Time Analytics with Go and LMDB . Retrospective about a high-load project at SkuVault (inventory management SaaS): High Availability and Performance .


Archive

Archive Here is a list of older blog posts that I wrote. 2016 2015 Retrospective 2014 Data Anonymization Behavioral Analytics Decomposing HappyPancake into components How do you decompose complicated system into components? How Micro-Services approach worked out in production 2013 From .NET to Erlang - Starting Work with Large Free Dating Website Stop overdoing things Separation of contract and implementation in software Many ways for an Emergent Design in a component How to produce a superb software design? How sticking to HTTP and REST makes your life simpler How I Stopped Worrying and Learned to Love the WWW and UNIX Way Implementing C# projections for Event Store Essay on Messaging and Design Patterns Replicating events to remote EventStore One Example of Using Message-Driven Design at Lokad Migrating to EventStore in Windows Azure Helping non technical people understand the data Greg's Event Store and Windows Azure Making Sense of Insane Software Requirements Use Software Design to Aid Your Development Process Domain-Driven Design and Organizational Politics Does your event store only serve one node? Aggregate Design and Security Utility Classes are Lesser Evils Essence of Domain-Driven Design - Re-Explained 2012 Being the Worst Updates Rule of Time Limiting Recommended Reading on Big Systems Management Lifehacks Analysis Paralysis Design Obsessions Design Obsessions Specification Testing For Event Sourcing Using Redis for Event Sourcing and much more Business Amnesia Lokad Topologies Updates on Lokad.CQRS - IDDD Branch Don't Be Depressed, Be The Worst Behind the Scenes of DDD/ES Project - Salescast2 Structure of DDD+ES Solution in Visual Studio Bounded Context is a Team Working Together DDD/ES - Missing Guidance Recent Lessons Learned in Lokad.CQRS Our Ultimate Technological Challenge Technology Demons Essential Reading on Big Data and Persistence Design Observations on Big Data for Retail DDD/CQRS Challenge - Integrating Distributed Systems DDD Summit 2012 Summary - Move Forward by Discarding Complex Tech Processing Big Data in Cloud à la Lokad DDD - Evolving Business Processes a la Lokad DDD From Reality to Implementation Software War Starts with a Map, Context Map Bird's-eye view of a Distributed System - Context Map Anatomy of Distributed System à la Lokad Get Real and Move Forward Building Blocks in CQRS world à la Lokad New Case Study on CQRSGuide.com Less is More Get Better Understanding Importance of Execution How to Deal with Unexpected Problems When NOT to use CQRS? Reading List on Big Data Getting Rid of CQRS View Rebuilds Make Code Explicit and Stupid People Don't Think in Tables Handling Big Data in Cloud 2011 Example of Self-documenting Unit Test with Event Sourcing Migrating Legacy Systems to Event Sourcing Migrating Legacy Systems to Event Sourcing Handling Multiple Event Streams in Event Store Tech Layer of CQRS Systems: Pushing it Further Store and forward for Event Streams Importance of Good Domain Models Query Multiple Aggregates from Event Stream Roadtrip-2011 Lessons Learned at CQRS/ES Road-trip lessons learned at Experience with CQRS+ES/DDD in Current Project Golden Rule of Technology Vienna CQRS Notes Mindsets in development - Enterprise vs Startup Update on CQRS Tour Around xUSSR - Ukraine Self-Improvement Process Continuous Learning at Lokad Shelfcheck - Cloud Scalability Challenge for CQRS from Lokad Messages and Distributed Systems Distributed Decide-Act-Report Model Use CQRS When Constrained on Developers and Time Why Even Bother Trying Linux Keeping Mind of .NET Developer Open Handling Concurrency in Cloud Views Scalable and Simple CQRS Views in the Cloud 2010 Cloud CQRS Lifehacks From Lokad - Part 2 Troubleshooting Azure Deployments Contributing To Open Source Is So Much Better Than Taking A Class 10 Steps To Become Better .NET Developer Key CQRS Ingredient CQRS Architecture and Definitions Was Your Software Career Like This? My Current Development Environment Top 10 Reasons to do CQRS (in a PDF) What is your project like - Titanic or a street dog? Practical CQRS/Cloud Samples and an eBook: Does This Interest You? CQRS Lifehacks From Lokad Production Command Handlers without 2PC and with Various Levels of Reliability Types of Messages and Events in Reactive CQRS World Scenario-based Unit Tests for DDD with Event Sourcing The Best Way to Learn CQRS, DDD and Event Sourcing Redirect Tcp Connections in Windows Azure Space Travel and Infinitely Scalable Solutions Rx works nicely with DDD and Event Sourcing Windows Azure Storage Can Be Confusing Challenges of the Code Documentation Importance of Tooling and Statistics in CQRS World Microsoft: Lokad is Windows Azure Platform Partner of the Year Salescast - Scalable Business Intelligence on Windows Azure DDDD, CQRS and Other Enterprise Development Buzz-words 2009 Efficient Development, Deployment and Customer Feedback at Lokad Concurrency Programming is Like Thinking in 5 Dimensions Legacy Code should still be Agile Maintenance and Monitoring Infrastructure for the Enterprise Software Systems Why is Cloud Computing important for us? 2008 Parameter vs. Argument Const vs. readonly vs. static readonly in C# .NET Hunger for the integration builds Some tips on writing event handling code in C# .NET Stress testing the stability The power of simplicity You can build a house with big LEGO blocks. Can you add a nice porch? Some side thoughts on the development Long story made short - Project Report for the xLim 1 2007 Lessons Learned - Diagnostics Lessons Learned - Development Methodology


Beautiful technical debt

Beautiful technical debt What looks like a technical debt in a software solution, might be the most efficient way for people to deliver business value in a given situation. There is a beauty in that, if observed from a distance. Don't make paying off technical debt your mission. Use it as a guide to understand the landscape and build solutions. Take a look at that beautiful painting "Pine Tree at Lake Side" by Imoto Tekiho. The pine tree in front looks nothing like the straight pine trees on the right side of the picture. Yet, this lone pine tree fits the landscape - the barren rock above the water. This crooked shape is the most efficient way for nature to grow on tough terrain. It clings to a rock while withstanding the storms, getting water and nutrients. It is perfect for this environment. Software solutions are like that. They take time to grow, from an idea to a prototype to a lifetime in production. While this happens, the solutions are affected by various forces of nature: team structures, personal incentives, pandemics, or bottlenecks in company processes. These forces affect the shape. Available resources also matter. We all want to build understandable and efficient solutions. Creating a mess isn't in our nature. We prefer straight lines and angles. Nature didn't invent measuring tape - humans did. If there is some "ugliness" in a solution - there usually is a cause. Maybe a data scientist could not get proper access to the database, not without a temporary workaround. Maybe a software engineer had to comply with a short-lived regulation. Maybe the lead architect was poached by a competitor before he could finish converging everything towards desired architecture. All these nuances will leave marks on the final product. It is sometimes called technical debt. It is a mark of external forces that had to be dealt with for the solution to survive . There is immense value and beauty in that. Don't just try to pay the technical debt off, but embrace and dig beneath. Dig into the problems , conduct customer interviews with stakeholders, get more data, and validate solutions early. Use existing technical debt to guide your questions and product design. Read More Problem-driven design Solving the wrong problem


Real-Time Analytics with Go and LMDB

Real-Time Analytics with Go and LMDB I'd like to share a few details about a project I've been working this year: a lean platform for real-time analytics and batch reporting . Project Inception The project started as an internal tool for running ad-hoc reports on business event streams. Existing tools took 5-8 hours to download a billion of events, replay them and produce useful reports. Event replay within 5-8 hours is good enough for warming up local caches on production nodes. However, it doesn't fit well with data analysis which has to iterate through multiple theories and experiments. You will probably forget 80% of the experiment by the time the results are ready. In my spare time I started working on experimental tooling to speed up the process and figure approaches that combine iterative and incremental event processing with occasional full replays. Constraints This project had a few constraints from the start: No multi-node data processing frameworks . Although Kafka and Spark could do the job, they come with dependencies like ZooKeeper and DevOps related overhead. I needed something simple that could efficiently utilize hardware on a single machine. Encryption and compression had to be supported by the infrastructure from the start. While modern file systems could handle both aspects, in some specific cases it would be more efficient to let the application layer handle these concerns. Throughput over latency is a trade-off that allows to avoid a lot of complexity in the code. Analytical pipelines are likely to spend seconds and minutes digging through the numbers anyway, so there is no need to optimize for sub-millisecond latency for the individual events. Simplicity and reuse - an inherent constraint, since the project was initially running on a limited time budget (roughly 15 hours per month). First Version in .NET The initial implementation was mostly done in .NET, since that is the platform I was most familiar with. I evaluated .NET Core, but at the moment tooling was too fragile and important libraries were still missing from the ecosystem. Despite the promise of Linux-friendliness (which halves hosting costs), actually developing in .NET core would've blown my time budget out of the water. Implementation consisted of: Analytical event storage , optimized for throughput and supporting a single writer with multiple readers. Data processing stages responsible for incremental event import, compression, filtering and aggregation; they operated on event storage streams (similar to how Kafka is used in multi-node systems), using durable in-process database for indexes, metadata and non-stream artifacts. Full replay stages which operated on compressed event streams, feeding them to in-memory actors for anomaly detection and fine-grained analysis. Visualization and report generation was done with python (charting) and lisp (html dahboard generation) stages. Due to the nature of Lisp, I inevitably ended up with a DSL for defining tiles and dashboards. That is something worth exploring in more detail later. This overall setup was similar to what you'd do these days with Apache Kafka, Spark and Spark Streaming, but it lived on a single machine. It was able to provide up-to-date visual and numeric answers to complex questions in matter of minutes instead of the hours. This worked because "heaviest" data transformations were incremental, while full replays operated on optimized datasets. LMDB - The Transactional Heart Almost all of the storage needs were handled with LMDB , which saved a ton of time in development without sacrificing performance. Databases like SQLite, LevelDB, RocksDB were evaluated, but discarded for performance or complexity issues (out-of-process databases were ignored). Besides, designing data models in terms of lexicographically sorted byte ranges is something I'm very comfortable with (provided the store provides true ACID guarantees). LMDB responsibilities: meta-data of the analytical event store: event chunk headers, event processing checkpoints; indexes, lookup tables, aggregated counters and mappings used by the data processing stages; final reporting artifacts: tables, lookups, time-series and meta-data. Switching to golang Some time along the way, I started porting core parts of the platform to golang for following reasons: Golang offers lightweight concurrency primitives that work well for real-time data ingestion, aggregation, incremental and batch processing. I wanted to migrate deployments to Linux/Unix to reduce hosting costs and gain access to its ecosystem: tools and knowledge on performance optimization. Golang also works well for fast and efficient API gateways that ingest data into the system or expose resulting data to the other systems. Rewrite took some time, but it was quite straightforward. There were some nice performance gains along the way. For instance, the new version of event storage layer can read event partitions at the speed of ~600 MB per second on a modest Intel Xeon server with NVMe SSD. This includes decryption and decompression of event data. My Windows Azure deployments were ~10x times slower and way more expensive for obvious reasons. First Linux deployment First deployment of the new version was rolled out recently. It is an analytical platform that captures events from real-time bidding auctions (header bidding and google AdX) coming from thousands of sessions running in parallel. This data can then used for real-time insight, experiments and long-term reporting driven by the financial indicators. This deployment handles 10M impressions per day across 4 different shards . A few interesting features: real-time dashboards with statistics for CPM, revenue (including Google AdX estimate), Ad size performance, bidder performance and responses, placements, user segments and sessions; aggregating and slicing data across placements, bidders, shards and experiments; running experiments (fine-grained A/B testing) and tracking their financial impact in real-time; long-term data retention and analysis which enable tracking revenue indicators per session and user lifetime, bidder behavior changes and the long-term impact of experiments. Storage and analysis layer is designed to support machine-assisted user segmentation and tuning of real-time bidding parameters within these segments. Experiments and revenue indicators provide a tight feedback loop for this process. All this is essential for publishers that want to increase their revenue in the world driven by players with large investments in automation and machine learning. Implementation Details Complete tech stack of this solution is rather simple: Ubuntu 16 LTS running on dedicated server (e.g.: Intel Xeon with DDR4 ECC RAM and SSD RAID on NVMe); Nginx for API intake and SSL termination; Golang and LMDB as the data processing heart of the system; Partitioned analytical event storage with AES encryption of data-at-rest (high-throughput, single writer and multiple readers); Telegraf with InfluxDB and Grafana for real-time display; python for custom reports that don't fit real-time UI. I find python invaluable for some data manipulation and analysis tasks these days. Its rich ecosystem (e.g. starting from matplotlib , pandas and up to keras and tensorflow ) can save a lot of development effort. Obvious fragility and complexity side-effects could be mitigated by freezing dependencies and exposed abstractions. What Next? It will be interesting to push this platform to its limits either with data loads or challenging ML tasks. So far, event streams running at rate of 10-30M events per day from thousands of connected devices aren't a big challenge for a single small-ish server with Linux, golang and LMDB. Bear in mind that hardware is evolving rapidly these days. Catching up with these limits can be a challenge.


Blog Migrated to Python

Blog Migrated to Python Once every few years this blog is ported to a new platform. Each migration attempts to simplify writing while preserving the existing content. Here is a brief history of my blogging platforms: LiveJournal - that was my first blogging platform; WordPress - I discovered Linux and wanted to play with WordPress, like other cool kids; Squarespace - I didn't want to maintain the WordPress installation anymore; Octopress - I wanted something less expensive and more flexible than Squarespace; Hugo - I discovered go and wanted to have something faster and less fragile than Ruby site generator; Static site generator using React components - I fell in love with Node.js and React, wanted to have flexible UI components in my blog. It isn't that hard to notice the emerging pattern here: each time a new tool shows up, promising to solve some problems. The platform ultimately gets migrated to new pastures with a greener grass. All the content, including legacy urls and asset locations, is dragged along. A few years ago, I started spending more time with Python, getting to love its opinionated simplicity and poetic elegance. You can guess what happened next. Current Implementation The current iteration ( source code ) uses Python3 with Flask to render all the pages. Flask is an extremely lightweight HTTP server in Python that also implements a few building blocks like url routing, request handling and template rendering. It isn't as heavy and opinionated as Django or Ruby on Rails, but is perfect for a quick API or a simple web site. The structure of the website is determined by the content : blog posts and post series, all spread between the different folders. When the application starts, we enumerate and load all the available content files. Flask application has a generic url handler that renders the appropriate template based on content type: @app . route ( '/<path:path>' ) def url ( path : str ) : value = s . urls . get ( '/' + path , None ) if not value : return "Not found" , 404 if isinstance ( value , blog . Story ) : return render_template ( "story_cover_page.html" , site = s , story = value ) if isinstance ( value , blog . Post ) : return render_template ( "post_page.html" , site = s , post = value ) if isinstance ( value , str ) : return render_template ( "redirect.html" , url = value ) HTML layout is done with Jinja templates . I grew to like them over the past years, just like Python itself. Here is, for example, a template for the story page. It lists all posts that make up a story. { % extends "layout.html" % } { % from "macros.html" import link % } { % block title % }{{ story . title }}{ % endblock % } { % block content % } < section > < p > {{ story . description }} </ p > { % for year , group in story . _items | groupby ( 'date.year' ) | reverse % } < div > < h3 > {{ year }} </ h3 > < ul > { % for post in group | reverse % } < li > {{ link ( post , date = "" ) }} </ li > { % endfor % } </ ul > </ div > { % endfor % } </ section > { % endblock % } Jinja markup is more constrained than React components: you can't mix HTML markup and code with the same ease. This made some features slightly more difficult to implement (e.g. "article tags" and "recommended articles"). So I just threw that functionality away. Maintaining tag lists added mental overhead anyway. The website is static in nature . It could be rendered once and pushed to any cloud provider. This eliminates almost all of the maintenance pains and effort. There are no servers to maintain and keep updated. Static html generation is performed by spinning up a test Flask client, requesting all of the existing urls and saving them into files. Resulting folder will then be synced to S3 bucket that is served via CloudFlare CDN. cli = app.test_client() urls = list(s.urls.keys()) urls.extend(['/', '/archive/', '/atom.xml', '/about-me/', '/404.html']) for u in urls: save_url(cli, u) for k, v in assets.items(): shutil.copy2(v, 'build/images/' + k) The website differs from the previous iteration by dropping a bunch of features: Article tags and recommendations. Syntax highlighting on the snippets. Complex and fine-tuned CSS/HTML layout. React.js templates (with complex inheritance). Re-implementation of the literate programming model that allowed to inject snippets from the code files. Async content processing pipelines (to speed up things). Drafts - articles that are rendered but don't show up in the index or home page. Instead of that, all that we have is a few functions that load content files into a big in-memory dictionary, plus a Flask site that renders that content with Jinja templates. Writing Flow The writing flow itself is evolving as well. I used to write blog posts by editing markdown files in Emacs or Vim, while using the website to preview the results. This introduced unnecessary ceremony and added friction. Git folder was always dirty. Fewer posts got published. I'm currently trying a simpler approach that is just easier: write the blog posts via a dedicated markdown writer (e.g. 1Writer on an iPad). The polished article could be then added to the site codebase and published.


🎙 Being the Worst

🎙 Being the Worst In a summer of 2012 Kerry Street offered to start a small podcast about learning software development. It wasn't supposed to be fancy or professionally edited, focusing instead on discovery, coding and mistakes that we were going to make. Years later, we are still pushing, thanks to Kerry and our awesome community. Later, the podcast turned out to be one of the longest and most successful projects I ever participated in. For that I will be gratefull to Kerry forever. In that podcast we started exploring things like Domain-Driven Design, Event Sourcing and CQRS. Somewhere along the way we decided to take on the domain of Getting Things Done , writing yet another task manager. Our GTD implementation was inspired by OmniFocus - an awesome task manager for OSX and iOS. This is the story of that journey.


E01 - The Worst Welcome

E01 - The Worst Welcome Audio apprenticeships for the aspiring software craftsman. Kerry and Rinat introduce the show and describe the plan for the road ahead. They are still figuring out how to produce and host a podcast so feel free to cut them a little slack on any episode quality issues. References: Study plan Feedback on current episode - in the comments to this episode post Discuss upcoming episodes - in trello General feedback - in UserVoice Being The Worst on twitter - @beingtheworst Kerry on twitter - @kcstreet Rinat on twitter - @abdullin Also mentioned in this episode: Be The Worst - pattern from the book, Apprenticeship Patterns Lokad.CQRS - in GitHub Book - Implementing Domain-Driven Design by Vaughn Vernon - in Safari Books IDDD book's code sample - in GitHub Greg Young twitter profile


E02 - Messaging Basics

E02 - Messaging Basics Kerry and Rinat start with the foundational topics of messages and messaging. What is a message? Why do we use them? How do you create them in code? Their conversation references our sample code that is available on GitHub.com . References: Sample Code on GitHub ( E002 folder ) Kerry's Homework Code on GitHub ( E002 homework folder ) Feedback on current episode - in the comments to this episode post Discuss upcoming episodes - in trello General feedback - in UserVoice Being The Worst on twitter - @beingtheworst Kerry on twitter - @kcstreet Rinat on twitter - @abdullin What do you think?


E03 - Commanding Your Words

E03 - Commanding Your Words Kerry and Rinat discuss the importance of the spoken words we use and how they impact the messages we create in code. From there, they introduce two critical message types: commands and events. Here's the factory plan, referenced in this episode: Homework: Write a short dictionary of words that reflect the names of the commands and events which could be used to describe a day at this car factory. This is how we define the vocabulary and ubiquitous language of the context being modeled. What are some of the parameters of these words that would make up their definition? Try to write a coherent story about situations that can happen at the factory. Use the names of these commands and events that are in our domain dictionary to ensure that the spoken language has been captured properly within the story. Normal sentences that would be used as part of normal spoken language. Explain how the factory workers would react to these commands and how they would carry them out. References: Endymion poem by John Keats Sample Code and Documentation on GitHub (E003 folder) Kerry's Homework Code on GitHub (E003 folder) Feedback on current episode - in the comments to this episode post Discuss upcoming episodes - in trello General feedback - in UserVoice Being The Worst on twitter - @beingtheworst Kerry on twitter - @kcstreet Rinat on twitter - @abdullin What do you think?


E04 - Event Sourcing Basics

E04 - Event Sourcing Basics Event Sourcing Basics (a day in the car factory). Kerry and Rinat review questions about designing command and event messages and then introduce the new topic of Event Sourcing. We model one day at the Factory using event sourcing, domain language and concepts discussed in earlier episodes. Here's the car factory implementation diagram, referenced in this episode: Homework: CSharp developers: implement 2 remaining methods of Factory using event sourcing other languages: reimplement Sample E004 in your own language References: Sample Code and Documentation on GitHub (E004 folder) Kerry's Homework Code on GitHub (E004 folder) Domain-Driven Design: Tackling Complexity in the Heart of Software - book by Eric Evans on Amazon Implementing Domain-Driven Design - book by Vaughn Vernon on Amazon - with an Appendix by our very own Rinat Abdullin! Feedback on current episode - in the comments to this episode post Discuss upcoming episodes - in trello General feedback - in UserVoice Being The Worst on twitter - @beingtheworst Kerry on twitter - @kcstreet Rinat on twitter - @abdullin What do you think?


E05 - Aggregating Stories

E05 - Aggregating Stories Kerry and Rinat introduce the concept of aggregates and their importance to the design of distributed systems. Then, they describe another way to express the story of an aggregate, through the use of specifications and the testing of use cases. Homework: CSharp developers: Several factory behaviors are missing from sample E005 that were in E004. Implement a few of these missing behaviors (methods) and create the specifications/unit tests for them. Other languages: First, take the pseudo test/specification framework in E005's framework.cs file and reimplement it in your language. Then do the same assignment above but in your own language. Episode References: Lokad User group DDD/CQRS Group Pat Helland's - Life beyond Distributed Transactions (PDF) Behavior-driven development (BDD) - Given, When, Then Sample Source Code and Documentation: Being The Worst code samples on GitHub (E005 folder) Kerry's Homework code on GitHub (E005 folder) Feedback and Contacts: Feedback on current episode - in the comments to this episode post Discuss upcoming episodes - in trello General feedback - in UserVoice Being The Worst on twitter - @beingtheworst Kerry on twitter - @kcstreet Rinat on twitter - @abdullin What do you think?


E06 - Community Code and Questions 1

E06 - Community Code and Questions 1 Kerry and Rinat discuss community questions about naming messages, aggregate state, and event implementation. Sample Source Code and Documentation: Being The Worst code samples on GitHub (E006 folder) Kerry's Homework code on GitHub Feedback and Contacts: Feedback on current episode - in the comments to this episode post Discuss upcoming episodes - in trello General feedback - in UserVoice Being The Worst on twitter - @beingtheworst Kerry on twitter - @kcstreet Rinat on twitter - @abdullin Other Community Links: Lokad User group DDD/CQRS Group What do you think?


E07 - Re-Factory

E07 - Re-Factory Kerry reflects on his homework experiences with Rinat. Then, they discuss ways to refactor the code to accurately capture the car factory's requirements. A hint of how this ongoing refactoring will evolve us towards a more production-worthy implementation is also mentioned. Episode References: Episode 7 code samples on GitHub Kerry's Episode 7 homework code on GitHub Feedback and Finding Us What do you think?


E08 - Frameworks Over Forms Is A Design Obsession Rinat covers some of his previous obsessions with certain tools and technologies [like composite UIs, Object-Relational Mappers (ORMs), and Inversion of Control (IoC) Containers] and reminds us that even the approach we have been learning is also, JUST A TOOL . The focus of your design should be on the essence of the domain that you are capturing and the problems you are trying to solve, not on specific tools or technologies. Homework: Have a look at the source code inside of the Open Source Event Store project Notice the use of various programming languages (C#, C++, JavaScript) as well as how the code reads like a story given the context of the event message persistence domain that it was designed for. Episode References: Book - Implementing Domain-Driven Design by Vaughn Vernon - in Safari Books Plato's theory of Forms Event Store by Gregory Young Feedback and Finding Us What do you think?


E09 - Greg Young Has A Major Event In Store Kerry and Rinat welcome their first guest, Gregory Young. You may know Greg from his blog, training, or CQRS fame but in this episode we discuss software design, distributed systems, and the new open source Event Store product that Greg recently released. Episode References: Event Store by Gregory Young Event Store Architectural Overview and source code on GitHub Staged event-driven architecture (SEDA) Cohesion in Computer Science Byzantine fault tolerance ØMQ - The Guide Feedback and Finding Us What do you think?


E10 - Don't Break The Chain

E10 - Don't Break The Chain Kerry and Rinat discuss when they will wrap up the car factory domain from previous episodes and then introduce the "Don't Break The Chain" domain. That domain and sample project will be used for the next set of episodes and code samples. Episode References: Source Code for Productivity Project on GitHub Don't Break The Chain Topic in Lokad Discussion Group on Google Groups Don't Break The Chain article on lifehacker.com Implementation of Don't Break The Chain article on lifehacker.com Feedback and Finding Us What do you think?


E11 - Specs Of The Living Dead

E11 - Specs Of The Living Dead Kerry and Rinat discuss the code updates that were made to correct the car factory sample. After that, they introduce the concepts of Value Objects and Strategic Modeling. Episode References: Episode 11 code sample on GitHub Zombies from the movie Night of the Living Dead Feedback and Finding Us What do you think?


E12 - Now Serving DSL

E12 - Now Serving DSL Kerry and Rinat introduce Application Services that host our Aggregates, the Domain Services that they use, and a code-generating DSL tool to help along the way. Episode References: Episode 12 code sample on GitHub Lokad-codedsl utility on GitHub Feedback and Finding Us What do you think?


E13 - So, Uh, We Need Boundaries

E13 - So, Uh, We Need Boundaries A homework question takes the conversation to unplanned places. Kerry and Rinat start with clarifying Application and Domain Services, but end up discussing many things. From system design, tactical options like CQRS (Command Query Responsibility Segregation) and SOA (Service-Oriented Architecture), to an attempt to influence a presidential election. Episode References: Architectural Patterns and Styles Service-Oriented Architecture What is CQRS? Feedback and Finding Us What do you think?


E14 - Back In The U.S.S. caR Factory Tests Kerry and Rinat continue discussing the Car Factory sample domain from Episode 12. Then, they cover its Domain.Test project and the updated specifications and testing approach within. Episode References: Episode 14 code sample on GitHub Lokad-codedsl utility on GitHub Feedback and Finding Us What do you think?


E15 - The Bits That Keep On Giving

E15 - The Bits That Keep On Giving Kerry and Rinat discuss new ways to reuse and benefit from our modeling efforts. This includes using our messages to test any serializer, auto-generating diagrams of our current design, and giving our domain its own command prompt. Episode References: Episode 15 code sample on GitHub Graphviz - Graph Visualization Software The Gift That Keeps On Giving clip from the movie Christmas Vacation Feedback and Finding Us What do you think?


E16 - Adding Value

E16 - Adding Value A listener's questions lead Kerry and Rinat into the details of Value Objects. They review several examples of Value Objects and cover some pros and cons of their usage. This very important concept is commonly used to represent the nouns in the sentences of our ubiquitous language. Episode References: Episode 16 code sample on GitHub Matthew's Questions That We Discuss Feedback and Finding Us What do you think?


E17 - View Projections

E17 - View Projections Kerry and Rinat wrap-up the Car Factory sample domain with an introduction to View Projections. The Event messages that we generate represent the state of our Aggregates. Projection classes can also listen for these Event messages and generate Views to provide query results to client applications. Episode References: Episode 17 code sample on GitHub Feedback and Finding Us What do you think?


E18 - Git Army Knife

E18 - Git Army Knife Kerry and Rinat dig into some of the development tools and practices used at Lokad. They specifically discuss Distributed Version Control Systems (DVCS) with a focus on Git's familiar design and our usage of it. Homework: Create a free GitHub account if you don't have one Fork the official Being The Worst sample repository or Lokad.CQRS Add or clarify some comments to anything that may need clarification in the code/comments of the repository Commit your changes to your own local repo and then submit a pull request to us on GitHub to update the master repository with your changes Bonus: Port BTW samples to your own programming language and ask us to add your repo to our GitHub account to share with the rest of our community Episode References: GitHub for Windows or Mac GUI Client Git site with x-plat installers Analyzing Git history blog post by Yves Reynhout and his C# code to do it Graph Theory at Think Like (a) Git Codebase code hosting services Feedback and Finding Us What do you think?


E19 - The Hermit: An Unexpected Journeyman Kerry and Rinat introduce the topics to be covered and the general direction for the next set of episodes. They discuss the new domain to be implemented, as they officially launch the next expedition. Homework: Get familiar with the "Getting Things Done(R)" Domain and learn the most important concepts Capture behaviors and concepts that describe the actual human practices involved Layout the vocabulary and start writing stories (ex: 10 terms/4 paragraphs) Episode References: General Itinerary for Expedition 2 (i.e. what we plan to discuss and learn) Getting Things Done book by David Allen Getting started with "Getting Things Done" post by Merlin Mann OmniFocus, GTD, and You (pdf) - The Omni Group Feedback and Finding Us What do you think?


E20 - Staring At A Blank Page

E20 - Staring At A Blank Page Kerry and Rinat discuss approaches you can use to start the domain modeling process in various situations. This helps them determine the first step to take to start designing their new domain. Episode References: Getting Things Done book by David Allen OmniFocus, GTD, and You (pdf) - The Omni Group Feedback and Finding Us What do you think?


E21 - Linguistic Cartographers

E21 - Linguistic Cartographers Kerry and Rinat start the discussion with a Context Map that has several Bounded Contexts. They use their conversation to refine the Context Map and carve out the initial key words of the Core Domain's Ubiquitous Language. Subscribe via RSS | Subscribe for free in iTunes Episode References: Context Maps and Initial Ubiquitous Language Published Language and Shared Kernel definitions We used Twiddla in this episode to collaborate on the updated Context Map diagram Feedback and Finding Us What do you think?


E22 - Biohazard Blah Blah

E22 - Biohazard Blah Blah Kerry and Rinat discuss the problem space, Bounded Contexts within it, Subdomains, and the Core Domain of "Getting Things Done"(TM). After the DDD discussion, they dive into the initial approach to structuring the code, exploring the Published Language, and making sense of information flows. Homework: Fork our GTD repository and explore branch for Episode E22 Add a local branch to your fork of the repo (use YourName_E22 as the name of the branch) Try adding more methods, Commands, and Events according to the GTD Methodology . Can you think of a better name for "Tenant" as discussed in the episode? Episode References: Codebase for Episode 22 Getting Things Done Domain-Driven Design: Tackling Complexity in the Heart of Software Implementing Domain-Driven Design Feedback and Finding Us What do you think?


E23 - Spaghetti-Free Terminology

E23 - Spaghetti-Free Terminology Kerry and Rinat revisit and correct some DDD terminology they used based on new learning and listener feedback. After they readdress Domain, Domain Model, Subdomains, Bounded Contexts, Published Language, Shared Kernel, and Anti-Corruption Layer, they try to relate these terms to real examples and code. Episode References: Rinat's 'Essence of Domain-Driven Design - Re-Explained' blog post GTD Context Map from E21 that is still valid Codebase for Episode 22 we looked back at Domain-Driven Design: Tackling Complexity in the Heart of Software Feedback and Finding Us What do you think?


E24 - Grambo's Got A Shotgun

E24 - Grambo's Got A Shotgun Kerry and Rinat dive into the GTD® code and discuss ways to choose words in the ubiquitous language. The new interactive shell allows them to use and shape the domain model as it evolves. In addition, an Event store is introduced to persist messages to disk, and they discuss ways to replay and view these state changes in the console. Homework: Go into the project, run the Shell, and try adding more Commands/Events and Console commands to make it work. Episode References: Codebase for Episode 24 Riak bitcask PDF Redis persistence demystified Feedback and Finding Us What do you think?


E25 - These Aren't The Projects You're Looking For Kerry and Rinat discuss changes to the ubiquitous language, the characteristics of lists and projects, and lessons learned from using a similar productivity system. In between, they probably talk too much about some variable names and system implementation details. Homework: Try adding a DefineAction command to the solution, this should include: Adding the console "shell" command to enter it in the shell Adding a View that allows you to list all Actions currently inside of a Project Episode References: Codebase for Episode 25 Trusted System: David Allen on twit's Triangulation show #22 Trusted Place: David Allen at Google Star Wars: These aren't the droids you're looking for Feedback and Finding Us What do you think?


E26 - Rinat Gits Committed

E26 - Rinat Gits Committed In this episode, there is a lot of new code to dig into. Kerry and Rinat cover the newly added features, discuss how the console relates to future UIs and assists with ongoing maintenance, dabble in Event message design, and get into the CQS pattern that was used to implement some of the Entity classes. They wrap-up with how the current message-based approach can be applied synchronously or asynchronously depending on needs, and explain why there are hints of Redis in the code repository. Episode References: btw-gtd Codebase branch for Episode 26 DDD Community site Command–query separation (CQS) Community's Java port of the BTW Expedition 1 (original) Samples Redis Event Store Feedback and Finding Us What do you think?


E27 - Evolving Event Centric Infrastructure Kerry and Rinat review some of the lessons learned from the deployment of Lokad.CQRS. Then, they discuss Rinat's blog post about those experiences, and some of his ideas for future improvement. This results in a sneak peek of the new event centric hosting infrastructure that will be used in their GTD sample project. Episode References: Best Infrastructure for Event Sourcing and CQRS Event Store Lucene.net Service Stack Feedback and Finding Us What do you think?


E28 - Learning Lokad CQRS

E28 - Learning Lokad CQRS Kerry and Rinat dig into the Lokad CQRS project that laid the code foundation for their current projects. This open source sample provides a working slice of a production system's registration and user management subdomains that were implemented with Aggregates and Event Sourcing. While they get into the details of its multi-Aggregate implementation, they cover the communication and workflow between these aggregates, how data storage and querying is achieved, and even get into some authentication and authorization options. Episode References: Lokad CQRS ( codebase version at the moment of recording) Stact - An actor library and framework for .NET Feedback and Finding Us What do you think?


E29 - Acting Like We Get The Message

E29 - Acting Like We Get The Message Kerry and Rinat introduce the Actor in the Actor Model of Computation. They wonder if the Actor's embodiment of communication (via messaging) may simplify the way that they reason about and implement their solutions. They discuss this potential use of the Actor Model in the context of their current usage of Domain-Driven Design (DDD), Application Services, Aggregates, and Event Sourcing. Episode References: The Actor Model (everything you wanted to know, but were afraid to ask) The Actor Model (Wikipedia) Carl Hewitt's homepage Scala programming language Akka toolkit Erlang programming language Stact - An Actor library and framework for .NET Red programming language Feedback and Finding Us What do you think?


E30 - The 30 Thirty Experience

E30 - The 30 Thirty Experience Rinat returns home from his trip to the United States which was filled with milestones, memories, and marriage. The guys briefly catch-up on some of the insights that were gained from looking over the fence at the Java ecosystem, and how it has made more progress than .NET in specific areas. They wrap-up with an intro to the DDD Summit, and how the things learned there and along the way will shape the upcoming episodes and code. Episode References: The Scala programming Language Akka toolkit and runtime Play Web Framework for Java and Scala built on Akka Erlang programming language MassTransit - Service Bus and Stact - Actor framework helped by Chris Patterson . Paxos protocols Greg Young's open-sourced Event Store Slav - contributor to Lokad CQRS 'The Blue Book' and DDD Design Pattern summaries by Eric Evans . Feedback and Finding Us What do you think?


E31 - Cloud Clearing Events

E31 - Cloud Clearing Events Kerry and Rinat choose the event storage to use for their project. They're using Windows Azure, so should they stick with Lokad CQRS event storage, or switch to Event Store from Event Store LLP? They discuss some of the pros and cons of their options, review some scaling approaches, and guess at what a transition may look like. Subscribe via RSS | Subscribe for free in iTunes Episode References: Event Store LLP's open-sourced Event Store product Lokad CQRS sample project Paxos protocols Raft: In Search of an Understandable Consensus Algorithm Basho Riak storage SEDA - Staged Event Driven Architecture C# infrastructure for doing eventsourcing using aggregates from Yves Reynhout Jonathan Oliver's Event storage (NEventStore) Feedback and Finding Us What do you think?


E32 - Questionable Approach

E32 - Questionable Approach Kerry and Rinat answer listener questions about code syntax, differences between event sourcing and relational storage, and concrete examples of Domain-Driven Design (DDD) concepts. Along the way, your questions lead them to questioning themselves and to consider an alternate approach. Episode References: Question #1 - C# dynamic syntax in sample code BTW sample code related to question #1 Question #2 - Event sourcing and relational databases Rinat's post about helping non-tech people to understand the data Rinat's post about event versioning that was mentioned during question #2 Question #3 - Concrete DDD examples Feedback and Finding Us What do you think?


E33 - Client-side Components

E33 - Client-side Components Kerry and Rinat return from their summer break to discuss their initial explorations into client-side development. This episode, and the two episodes after it, cover a two month period in which Rinat and Kerry went off to write some new client code for their Getting Things Done(tm) sample project. They start this three-part spike with a simple WinForms client, and end it with the beginnings of their cross-platform mobile application. Episode References: BTW Sample Code for E33 on GitHub Staged event-driven architecture Model–view–controller (MVC) pattern Model View ViewModel (MVVM) pattern Event Store Feedback and Finding Us What do you think?


E34 - Model View Who?

E34 - Model View Who? This episode is the second of a three part series on client-side development. Kerry and Rinat discuss the changes that were made to the sample's WinForms client to make it easier to add new features. They review the three main contexts in their GTD sample, the benefits of passing around immutable state, and dig into graphical user interface architectures like MVC and MVVM. This sets them up for the next episode where they apply MVVM to the sample's cross-platform mobile application. Episode References: BTW Sample Code for E34 on GitHub Martin Fowler's article on GUI Architectures Free MvvmCross Training Videos compliments of @slodge - Stuart Lodge MvvmCross code on GitHub @mvvmcross on Twitter Micorsoft's Roslyn compilers Model–view–controller (MVC) pattern Model View ViewModel (MVVM) pattern Feedback and Finding Us What do you think?


E35 - Totally Crossed Out

E35 - Totally Crossed Out This episode is the third of a three part series on client-side development. Kerry and Rinat discuss their initial experiment with applying the MvvmCross framework to the GTD sample's cross-platform mobile application. They review why they selected this approach, the initial code in the solution, and plans to revise their Core Domain and Published Language into reusable Portable Class Libraries (PCLs). Episode References: BTW Sample Code for E35 on GitHub Free MvvmCross Training Videos compliments of @slodge - Stuart Lodge The Tablet Show (podcast) - Stuart Lodge Builds Cross Platform Apps with MvvmCross MvvmCross code on GitHub @mvvmcross on Twitter JabbR chat room for MvvmCross Microsoft and Xamarin Partner Globally to Help You Build Great Apps PCL and .NET NuGet Libraries are now enabled for Xamarin Greg Shackles on Twitter Jon Dick on Twitter Greg and Jon's 'Gone Mobile' podcast Tomasz Cielecki 'Cheesebaron' on Twitter AngularJS Knockout with Durandal Model View ViewModel (MVVM) pattern Feedback and Finding Us What do you think?


E36 - Break It Down Again

E36 - Break It Down Again In this episode Kerry and Rinat discuss some of the benefits of decomposing your software into even smaller components. Some may refer to this technique as "micro services" or "system-of-systems". They end their conversation with a look at how this approach was applied to the Lokad codebase to result in a more evolvable design. Episode References: Design, Composition and Performance Implementing Micro Service Architectures - Video Breaking the monolith: Towards a system-of-systems architecture - Video Model storming - a different approach to collaborative model discovery Feedback and Finding Us What do you think?


E37 - Variety Is The Spice of Pancakes

E37 - Variety Is The Spice of Pancakes Kerry and Rinat discuss some of the learning that Rinat and the HappyPancake team did to evaluate their language and tool options for their new project. Rinat makes some observations about his long history with C#, and the benefits of looking at other languages and approaches. In Episode 38, they may even discover new ways that C# can do some things that Rinat says can be hard to do using traditional C# and .NET techniques. They finish up by getting into more of the "Microservices" or maybe "component-based" approaches that Rinat has experienced recently. Episode References: Dr. Erik Meijer Videos on Functional Programming Fred George on implementing Micro-Services Micro-Services - Java, the UNIX way HappyPancake R&D (Rinat's new gig) Docker Syncspace (collaborative drawing) Java Metrics and a .NET port ServiceStack Event Store Google+ Hangouts SimCity Feedback and Finding Us What do you think?