E38 - Glenn Block Evolves APIs

E38 - Glenn Block Evolves APIs This is an archive copy of BeingTheWorst podcast Episode 38 by Kerry Street and Rinat Abdullin. Check out the original site for comments, RSS and iTunes feeds. Kerry and Rinat welcome guest Glenn Block to discuss hypermedia Web APIs, and the new book he co-authored on the subject. They get into some of the differences between systems that are designed to take advantage of the constraints of the REST architecture style, with those that are not. What is hypermedia? Should you always adhere to REST constraints and use hypermedia when building a Web API? What are some ways to implement these concepts on the client and server and how do they interact? What are some common implementation mistakes? API security? What can your book help me with? UCWA? And much more. Episode References Book: Designing Evolvable Web APIs with ASP.NET – Harnessing the power of the web GitHub source code for the Web API book Roy Fielding: Architectural Styles and the Design of Network-based Software Architectures Glenn Block on twitter Hypermedia Representational state transfer (REST) Hypertext Transfer Protocol (HTTP) Mike Amundsen apiary Mark Seemann’s Hyperlinkr library on GitHub OpenRasta Lync – Microsoft’s Unified Communications Web API (UCWA) HAL – Hypertext Application Language Microsoft Azure Simple Object Access Protocol (SOAP) Feedback and Finding Us


Event-driven specs

Event-driven specs Last week I continued working on the back-end API server for Getting Things Done task-manager. GTD domain is something we explored for some time with Kerry Street in BeingTheWorst Podcast . This project is a continuation of that journey but with the things learned recently from the 👍 HappyPancake project and DDD community. Use-Cases Specification micro-framework seems to be stable now. I wrote it from scratch, while bearing in mind the limitations and issues we discovered in previous event-driven project. That is how single API use-case looks like: func given_unchecked_task_when_check_then_event () * env . UseCase { taskId := lang . NewTaskId () return & env . UseCase { Name : "Given new task, when PUT /task with check, then event" , Given : spec . Events ( lang . NewTaskAdded ( newEventId (), taskId , "ho-ho" ), ), When : spec . PutJSON ( "/task" , seq . Map { "checked" : true , "taskId" : taskId , }), ThenResponse : spec . ReturnJSON ( seq . Map { "taskId" : taskId , "name" : "ho-ho" , "checked" : true , }), ThenEvents : spec . Events ( lang . NewTaskChecked ( IgnoreEventId , taskId ), ), Where : spec . Where { IgnoreEventId : "ignore" }, } } Such use-cases focus on a single specific task: they describe behavior of a module in a declarative way via its public contract : consumed and published events; RPC API requests and responses. As such, these are not exactly unit-tests or integration tests. They have some nice properties when applied with the event-driven design process. I'll talk about that later. Other applications for these use-cases are: printing human-readable API usage stories to impress managers, generating detailed up-to-date API documentation to impress developers, generating system dependency diagrams to scare managers and generating realistic load-tests to stress the heck out of hardware and impress DevOps people. Partial Responses While implementing use-case library, the trickiest part was to figure out how to test only partial responses. For example, an API call might be returning a large task object, however in a specific use-case only three specific fields would be relevant. We would want to test them and ignore the rest of the object. At this point I solved the problem by constructing response objects from maps by hand and comparing them with the actual response. ThenResponse : spec . ReturnJSON ( seq . Map { "taskId" : taskId , "name" : "ho-ho" , "checked" : true , }), Of course, one could pass directly a strongly-typed response object instead of a weakly-typed map. That would defeat the whole purpose of API tests, though. We want to test actual responses. The same approach applies to events that we expect and actually get. Verifying Use-Cases Verifying behavior of an application in a clear and non-fragile way - is the primary field of application for these use-cases. I implemented a simple verification utility, which plugs into the unit-testing pipeline of Go. It runs all tests, printing a detailed information for each failure. Behind the scenes, verification runner loads a module into memory, wiring it to the real http server and in-memory event bus. After dispatching events and performing http requests, outputs are compared with expectations. ✗ Given new task, when PUT /task with check, then event Given_events: 1. TaskAdded { "eventId": "13bcdee63b9888cb0000000300886534", "taskId": "13bcdee63b9887dc0000000200886534", "name": "ho-ho" } When_request: PUT /task Expect_HTTP: 200 { "checked": true, "name": "ho-ho", "starred": false, "taskId": "13bcdee63b9887dc0000000200886534" } Actual_HTTP: 500 { "error": "Not implemented" } Expect_Events: 1 0. TaskChecked { "eventId": "", "taskId": "13bcdee63b9887dc0000000200886534" } Manually comparing expected and actual responses can be counter-productive. It might be better to work with a clear list of issues, fixing them one by one. Use-case verifier now does exactly that: Issues_to_fix: 1. Expected 'Body.checked' to be 'true' but got 'nothing' 2. Expected 'Body.name' to be 'ho-ho' but got 'nothing' 3. Expected 'Body.starred' to be 'false' but got 'nothing' 4. Expected 'Body.taskId' to be '13bcdee63b9887dc0000000200886534' but got 'nothing' 5. Expected 'Status' to be '200' but got '500' 6. Expected 'Events.length' to be '1' but got '0' 7. Expected 'Events[0].$contract' to be 'TaskChecked' but got 'nothing' 8. Expected 'Events[0].taskId' to be '13bcdee63b9887dc0000000200886534' but got 'nothing' Productivity Golang compiler can build sources and run tests in milliseconds. The rest of tooling is comparably fast as well. We could even perform all these operations almost instantaneously: reformat code save file perform static code analysis for common issues build run unit tests and verify use-cases In fact, I configured F2 to perform all these operations in sequence on single a key-press. This makes the process of implementing API logic rather simple: Pick the next issue in the Issues_to_fix list. Type a few lines of code. Hit F2 . Verify that the issue is gone, then go to 1. If I were doing that with .NET and MSBuild, then I'd need to insert "Take a break" between 3 and 4. Next Steps This API back-end is going to provide persistence and scalability to Flux/React.JS front-end, which I'm building in parallel (currently front-end uses in-memory storage). So my next steps would be: Implement all API features needed to wire front-end to back-end. Implement Projects and Contexts . Add some persistence to the back-end.


E39 - In Flux and Back to React

E39 - In Flux and Back to React This is an archive copy of BeingTheWorst podcast Episode 39 by Kerry Street and Rinat Abdullin. Check out the original site for comments, RSS and iTunes feeds. Kerry and Rinat return. The guys catch-up on some of the development that Rinat’s been doing while they were away and set their recording timer to 30 minutes. They discuss that their sample GTD domain, from past episodes, is being implemented with Facebook Flux and React.JS as they set the stage for future episodes. Episode References Rinat’s Talk on ReactJS and Facebook Flux at DDDBE in Belgium GitHub source code GTD Domain in Flux and React.JS React Developer Tools from Facebook for Google’s Chrome browser Feedback and Finding Us


building-ai-assistants-course

building-ai-assistants-course Building Reliable AI Assistants: Patterns and Practices About · Who It’s For · What You’ll Get · What’s Inside · Pattern Library · Format · Access · Pricing · FAQ · Buy A self-paced course for people who design, scope, ship or oversee LLM-powered assistants. This course is about making sound technical decisions for LLM-powered products across domains, so the systems you build: deliver stable quality stay controllable and testable keep their quality as functionality expands. We’ll look at where systems fail and how reliable solutions are designed. I’ll show you a practical approach to predictable behavior and continuous improvement. That approach comes from my hands-on work shipping AI systems and advising teams on architecture and quality. Buy Course in AI Labs Why I made this course Teams keep running into the same problems with LLM systems: unstable quality, unclear architecture, weak control over output, and too much time spent rediscovering lessons that already exist. I built this course to shorten that path. Inside, I walk through how AI assistants fail, how to diagnose the real causes, and how to use proven patterns to build systems that are more reliable, more testable, and easier to extend. Who this course is for This course is for people who design, scope, ship, or oversee LLM-powered assistants — whether you write the code yourself or guide the team making the build decisions. It is a strong fit once you already have some exposure to LLM-driven products and want a more systematic way to make them production-reliable. Engineers Trace failures to the information flow, reproduce them quickly, and resolve them with proven patterns. Tech Leads and CTOs Design architectures that remain structured as scope grows, with control points for quality, testing, and evolution. Product Leaders Choose feasible AI use cases, define measurable quality targets, and turn vague ideas into scope teams can ship quickly. Founders Pick MVP-friendly approaches, avoid slow dead ends, and set a quality trajectory that supports the next stage of the product. When it’s a mismatch You’re just getting started and haven’t built any LLM-based systems yet. Your focus is exclusively local-model infrastructure (this course focuses on patterns and engineering principles that apply broadly). You want a framework tutorial (LangChain/LlamaIndex setup, indexing walkthroughs). The course focuses on architecture and is framework-agnostic. This course is also a mismatch, if you want to learn more about deploying cutting-edge agents like OpenClaw or building something similar. The reason for that - the course is based on the statistics of successful cases of AI adoption in the business. While AI Agents are hot in theory, in practice they lack a history of successful adoptions. It will take some time to establish proven patterns and practices of shipping trustworthy and reliable AI Agents. The course on AI Agents will have to wait until then. In the meanwhile, if you are really interested in leading R&D around AI Agents - check out BitGN - my platform for autonomous agents and amazing teams building them. What you’ll walk away with Practical methods to diagnose and reduce hallucinations in real workflows. A clearer sense of what makes an AI assistant behave reliably in production. The ability to trace failures to the information flow and fix them with repeatable approaches. A working mental model for choosing architectures that hold up as scope grows. More control over outputs through structured responses and guided reasoning. A way to evaluate quality with concrete checks, not intuition. The Pattern Library proven in real-world success cases, helping you choose approaches that hold up in production. What’s inside Module 1 — Foundations for reliable AI assistants We start from a familiar document-assistant scenario, reproduce the failure modes, and work downward until the behavior becomes clear. You will build intuition for how LLMs behave, how context engineering shapes quality, why retrieval quality matters, how hallucinations get triggered, and how structured outputs and custom chain of thought improve control. From there, we move back up to testing, evaluation, trust, and AI case mapping. Module 2 — Pattern Library The second module turns those foundations into repeatable implementation patterns drawn from real successful AI cases. It moves from prompt patterns and knowledge base design to search, workflows, routing, structured data extraction, feedback loops, Schema-Guided Reasoning, and LLM + Domain-Driven Design. For each pattern, I show the task framing, the real constraints, where quality breaks, and what produces stable results in production. Together, these patterns help you recognize recurring problem shapes and reach workable solutions faster. Why the Pattern Library matters Most teams spend too much time rediscovering the same failure modes. The Pattern Library is based on 40+ real AI success cases across 20+ companies that I helped teams ship. It gives you a reusable set of design moves drawn from successful AI implementations. You can adapt them to your domain and move with more confidence when a case grows in complexity. Format Recorded video lessons with chapter navigation and supporting materials. 4+ hours of course video. Self-paced, so you can move through the material on your own schedule. Two practical exercises are included and can be skipped if you are not coding. Access The course is delivered through my platform, AI Labs , and purchase happens there as well. Authentication at the AI Labs is done via Gmail. Personal purchases unlock the course in your AI Labs account. You can buy access for yourself, buy an activation code for someone else, or purchase seats for a team. Purchases for someone else and team purchases are delivered as activation codes, so seats can be assigned later without immediate activation. Pricing Personal access — 1 seat EUR 280.00 Team access 5-seat pack: EUR 1400.00 10-seat pack: EUR 2800.00 Tax calculated at checkout where applicable. Companies can add a billing address and EU VAT ID during purchase. Billing documents are generated automatically and sent by email. Payments are handled via Stripe. Buy in AI Labs FAQ Will this course fit me if I work in Europe, the US, or elsewhere? Yes. The course is built for an English-speaking audience worldwide, and the patterns come from real AI implementations across industries and countries. Do I need programming experience? You do not need to write code yourself to benefit from the course. The material focuses on design, scoping, architecture, and quality decisions. Engineers can apply it directly in code. Is there a practical part? The course is lecture-led and comes with supporting materials. Two practical exercises are included and can be skipped if you are not coding. Can I buy access for another person? Yes. You can buy an activation code and give it to someone else. Can I buy access for a team? Yes. Team purchases are available through AI Labs. Team seats are delivered as activation codes, so they can be assigned later without immediate activation. How do I get access? Purchase happens in AI Labs. Authentication there is done via Gmail. Personal purchases unlock the course in your AI Labs account. Still have a question? Write to biz -bot @ abdullin . com This course gives you the foundations, the Pattern Library, and the decision framework for building LLM-powered assistants with more control over behavior and quality. Buy Course in AI Labs


Let's build robotic arm

Let's build robotic arm This blog post is time-boxed by my sleeping 4mo old son. I most likely have 12 more minutes of uninterrupted time left before he wakes up, so I’ll keep it short. Spoiler alert, Erik slept for 1 hour and 51 minutes! :) Work Situation Since last year, I’ve been working with SoftwarePark - an Austrian software company focusing on turn-key solutions for its clients. At work, I’ve been mostly involved with infrastructure for data engineering and machine learning. Simply put, we are creating an environment for the data scientists to design, push to production and improve their models . Technology revolves around Kubernetes, Helm, Python, Jupyter, Apache Airflow, Dask Distributed and Ceph over S3 with a pinch of Google Cloud. If compared to my previous projects, this work is less pure R&D and more plain engineering. This puts me in a little happy place — with two kids, there isn’t enough time and focus left to, say, work on a consistent distributed application-specific database (like 📦 SkuVault . Domain design in our current project focuses on creating a comfortable and coherent R&D environment for data scientists. In a sense, it feels like OOP at the level of software systems - we are still removing excessive concepts, refactoring and renaming things. New Hobby Along the way, I’ve accidentally found a new hobby to replace software development for fun. It is less demanding for undivided attention, long chunks of undisturbed time and availability of both hands. There is an interesting company called Haddington Dynamics that is building a low-cost high-precision robotic arm . They focus on bringing the cost down, while going for really high precision (measured in 10-50nm). Most of their work is done open-source , except for the FPGA code and the latest designs. Low cost in this case means: from 6800 to 9800 USD. My two other colleagues and I are trying to design and build such an arm from scratch , while achieving similar levels of precision but using less expensive parts and tech. This constraint is where all the fun comes from. Current Status We have built a first tech prototype out of two 3D printed parts (from the Haddington Dynamics STLs), Lego Technic blocks, Arduino and a cheap DC motor. It has an optical encoder that helps to know and maintain its position more or less precisely. However, enormous backlash is killing any hope of establishing even crude haptic feedback. To move forward, we need to get our hands on a speed reducer with smaller backlash (speed reducer is needed, since electric motors operate best when spinning fast). It seems that our best bet is to build a 2-stage cycloid drive with 56:1 gear reduction or so. There is a simple single-stage prototype being printed by our friend right now (and our own 3D printer is en-route from China). In parallel, we are investigating how to improve speed and precision of an encoder. Current designs will get the most out of ATtiny micro-controller. In the future we might try using LVDS IO on an FPGA to work as an Analog-to-Digital Converter (ADC) for an encoder disk. Stellamar did it, Haddington Dynamics did it, so it could probably be reproduced. Similar to how stack flattening works in software design, bringing ADC into the digital logic might allow to make a relatively simple high-frequency converter. After all, if we cache all signals produced by the encoder disk, this could be used as a lookup table to speed up ADC sampling (we’ll check the most probable signal levels first). I’m probably being the worst person in the world to work on a nanometer-precision robotic hand with the haptic feedback, however: this is mostly an excuse to learn new skills; we still have a couple of years filled with chunks of time when we have at least one free hand and partially uninterrupted focus; it is an opportunity to do something engaging with the older kids.


Command-Query Separation

Command-Query Separation Command-query separation (or CQS for short) is a design principle devised by Bertrand Meyer. CQS states that every method should either be a command that performs an action (changes state), or a query that returns answer to the caller (without changing the state or causing side-effects), but not both. In other words, functions should return a value only if they are pure (don't make any visible state changes). This convention, if followed consistently, can simplify programs, making them easier to understand and reason about. Following method violates CQS principle, it does too much: func SaveUser ( name string , age int ) * Result { if name == "" { return NewError ( "Name is empty" ) } if age <= 0 || age >= 100 { return NewError ( "Age is out of range" ) } users = append ( users , NewUser ( name , age )) return nil } We can simplify this code by breaking it into two methods. Command : func CreateUser ( name string , age int ) { users = append ( users , NewUser ( name , age )) } and query : func ValidateUser ( name string , age int ) * Result { if name == "" { return NewError ( "Name is empty" ) } if age <= 0 || age >= 100 { return NewError ( "Age is out of range" ) } return nil } We can reuse and test these methods separately or compose them. For example, to print validation errors: func InputChanged ( name string , age int ) { var result = ValidateUser ( name , age ) if ! result . Valid { RenderErrors ( result ) } else { RenderSaveButton () } } Saving user while safe-guarding against invalid input: func SaveButtonClicked ( name string , age int ) { var result = ValidateUser ( name , age ) if ! result . Valid { // protect from programmer error panic ( result ) } SaveUser ( name , age ) } Following CQS principle is a good guideline for a project coding style. However, it might be unfeasible to follow it everywhere. Instead, we could be consistent within a context. For example: accept that UI is messy but keep backend clean. Exceptions CQS works best when it we treat it as a design principle. If used consistently, it helps people understand code better and faster. Yet, there always are exceptional edge cases, where it could be wise to step away from such principle: Operations related to concurrency, where it is important to both mutate state and get the result back: Interlocked.Increment or sync.Add . Well-established data structures, where functions with result and side-effect are common: queue.Dequeue or Stack.Pop It is possible to modify all of these methods to follow CQS principle. Yet, that would introduce additional complexity and deviate from the expected behavior. It might be better to accept the deviation and note it explicitly (e.g. in documentation, inline comments or method names). CQS in Languages CQS is prominent in Functional Programming . Languages supporting it often have this principle baked right into the language design itself. Haskell and F# emphasize functional programming. Learning them would be a good exercise and can make you a better developer. Languages following Object-Oriented Programming approach don't enforce CQS by default. Yet, following these principles can lead to simpler code that is more predictable and easy to reason about. Some ecosystems offer additional tooling to make CQS principle more explicit to the developers. For example, Microsoft Code Contracts introduces Pure Attribute to indicate methods that don't have visible side effects. References Command-query separation - Wikipedia Pure Attribute


My Courses

My Courses Building AI Assistants: Patterns and Practices This is a recorded online course on building efficiently LLM-driven products in different industries. The course is based on practical experience and AI research with teams in EU and USA. It covers both the fundamentals, as well as concrete patterns and case studies. The course includes ~5 hours of densely packed video materials, private community and a set of practical exercises English version · Русская версия AI+Coding This will be a recorded online course for teams about patterns and practices of software development with modern AI tools. Vibe coding is mentioned there as well, however the content mostly focuses on systematic development approach for the large projects and teams. English version - ETA Q3 2026. Join waitlist to be notified.


(Over) Designing privacy-first analytics While getting rid of Google Analytics and setting up Lean web analytics for my website, I hit a problem. It is a balance between privacy and gathering feedback. My current setup uses standard web access logs. These include IP addresses and timestamps. That is enough to track individual user actions on the website, regardless of the ad blocker. However, I don't want to record IP addresses or use session cookies , since this involves personal information (IP) and cookie consent. At the same time, I want to see: which pages are interesting for people - to get the motivation to write more. how do people arrive at a specific page and where do they go next - to improve information structure. I don't want to spy on specific people, just to have aggregate feedback. So what if we change common web analytics this way? don't persist IP addresses , not even after hashing; don't use any cookies, fingerprints, or identifiers ; On every page request from the website, send two additional headers: X-Page-Num : is it the first page visited on this site, 2nd or Nth? X-Page-Sec : how much time was spent on the previous page (rounded to seconds)? Server-side, append this information to a daily table. For example: { "date" : "2022-06-13" , // daily bucket "referer" : "/lean-web-analytics" , // HTTP Referer header "url" : "/privacy-first-analytics" , // Requested url "visits" : { 0 : 12 , 1 : 2 , 2 : 1 }, // number of visits, by page num "time" : [ 1 : 2 , 2 : 10 , 3 : 1 , 6 : 1 , 12 : 1 ] // seconds spent on a page ] Page visits from multiple users (and bots) traveling through the website get aggregated. Without a way to link visits to a session or IP address (because we aren't storing them), it is impossible to figure out who did what. How can we deduce any statistics from that? The number of browser sessions per day? Total count of visits with page num 0 Average time spent on a page per day? Group all rows by URL, then average time . Where do people go from page X ? Find all rows, where referer==X and weight by the number of visits How do people come to page X ? Find all rows where url==X How many people do leave the site after page X ? The number of visits to page X minus number of visits from X is the number of leaves. Gotchas What we can't deduce from the statistics? The number of unique visitors. We can only count unique sessions. Which sequence of pages leads to page X . We know only pages that lead directly to page X , but we can't reconstruct the entire flow. So we can't exactly track the sales funnel. Can ad blockers prevent this setup from working? Only by disabling JavaScript on the website or messing up with the headers. Can this setup be misused? Yes, by recording IP addresses and timestamps, like it is done in web server access logs. How to implement this setup? Here are just a few ideas. First, add Hotwire Turbo (or equivalent) to the website, then add an event listener to include page number and time: var page_num = 1 ; var page_time = Date . now (); ( function () { addEventListener ( 'turbo:before-fetch-request' , async ( event ) => { const headers = event . detail . fetchOptions . headers ; headers [ 'X-Page-Num' ] = page_num ; headers [ 'X-Page-Sec' ] = Math . round ( Date . now () - page_time ); page_time = Date . now (); page_num ++ ; }) })() Server-side, drop standard web access logs . Replace them with custom logs that don't persist IP addresses or timestamps. Push these as updates to a database (e.g. Clickhouse). Don't record individual events - aggregate them immediately. If using a message bus between the web server and Clickhouse , set a short retention period (e.g. a day), so that individual analytics messages don't live long. RedPanda might be a good fit (resource-efficient, Kafka-compatible and passes Jepsen). Or even use NSQ without spill-over to the disk for small cases. NSQ used to handle 10-30M events per day from thousands of connected devices on a single box (read on [[2017-09-12 real-time analytics|real-time analytics with LMDB, Go and NSQ]]). Next Let's validate if this approach actually works. We'll take a few shortcuts and use clickhouse-local to build a PoC: Analyze website logs with clickhouse-local .


Dive into FPGA, PCB and 3D printing

Dive into FPGA, PCB and 3D printing In the previous post I mentioned that my colleague Aigiz and I had started a hobby that was particularly friendly to parents of young kids. We are trying to replicate the technology used by Haddington Dynamics in their high-precision robotic arm but to use less expensive parts and processes, where possible. "Kids-friendly" comes from the fact that we have years filled with small chunks of time during which our hands might be busy, but cognitive skills and brains are free. I've started typing this blog post while helping my son to hold his bottle, only to finish the post 2 months later. As it turns out, there is a lot to learn, so this hobby is taking us down the rabbit hole (in a good way). We've split our roles: Aigiz focuses on the mechanical design (3D design and printing, milling, power drivers, interop with the stepper motors); I focus on the electrical design and the software (FPGA, PCBs, interop between the high-level code and the HDL programs). Our next milestone is to have a simple high-precision robotic joint with a haptic feedback. This includes designing out and building: speed reducer gear with ratio 1:50 to 1:100; FPGA pipeline to translate analogue signals from the encoder wheel to absolute position and to control the stepper motor with low latency; interop between the FPGA and the computer (could be an ARM running on the FPGA dev board). Mechanical Design Aigiz took the ownership of the mechanical design : everything necessary for a NEMA 17 motor with a speed reducer gear. The progress so far: We have Ender-3 3D Printer that is almost working (one motor driver on the board was dead on the delivery, there is a replacement on the way); we've learned that it is better to pick 3D printers that have CPU and the motor control separately (especially the ones that allow you swapping individual drivers); we read everything available on the internet about the cycloid drives; they are a pain to build without the high-precision equipment and processes (which we don't have at this point); lowered expectations for the next milestone to use the planetary gear or whatever that would work (since size isn't the issue at this point); learned a great deal about the control drivers for the stepper motors and interacting with them; started working with the Fusion 360 (free CAD/CAM software). Next steps: to get the 3D Printer working and print out a planetary gear for the NEMA 17. FPGA and Verilog We started going through the FPGA basics together. Aigiz picked the proper IDE route with his Altera Cyclone IV, while I've started with the Verilog via the Verilator route (free Verilog HDL simulator). It was amazing to discover that simulation (something that doesn't happen frequently in the software design) is an expected norm in the electrical engineering. There is a lot to learn in this field! We didn't get that far along the route, however the journey has already helped to connect a lot of dots in the way modern computers work (and start appreciating all the work needed to produce even a simplest micro-controller). In addition to the Cyclone IV, we have a MiniZed (a single-core Zynq 7Z007S development board) en route to Russia. It has enough computing power and XADC IO in order to drive a single robotic joint or two. This development branch is currently paused while we are waiting for the parts and the speed reduction gear. PCBs The original Haddington Dynamics design has little optical boards close to the LED/Phototransistor pairs (for removing the noise and calibrating the IR LED intensity). Although we don't need anything like that for the first iteration, PCB design might be helpful to move forward later, so we've taken up the task of rebuilding these PCBs. Current progress: learned about reflow soldering; ordered enough SMT components for 5 optical boards; started designing a PCB in order to have it printed out (finding and/or designing footprint models for the components is the most time-consuming part here). The next step is to finish the PCB design and order a batch of boards. Current Status This is what this milestone looks like at the moment. We aren't even connecting optical encoder, since the cycloidal speed reducer just keeps on jamming. Hence the need for another version of the speed reducer in order to move forward.


DataFest-2024

DataFest-2024 Data Fest 2024 Это список ссылок и материалов к моим докладам Data Fest 2024 Knowledge Maps: Как бороться с галлюцинациями в RAG-системах Три паттерна успешных внедрений проектов с LLM под капотом Больше информации про кейсы продуктов с LLM под капотом можно найти в телеграм-канале LLM под капотом . Например: Кейс про захват рынка с LLM Кейс с ChatGPT про 10.000 € в месяц (бизнес-переводчик) Персональный ассистент в компании Автоматическое исправление ошибок в каталогах продуктов Ассистент маркетолога в компании Переводчик со знанием специализированной терминологии Нишевый lead generator для B2B продаж Платформа для автоматизации бизнес-процессов в компании Про свой ChatGPT для муниципалитетов Слайды с презентации я выложу в канале в течение суток. Бенчмарки LLM в первую очередь я публикую тоже в нашем канале.


Enterprise RAG Challenge

Enterprise RAG Challenge Enterprise RAG Challenge is a friendly competition that compares different RAG architectures. The goal is to build an AI-driven system that will be able to answer questions about annual reports of the companies. You can find more technical details in this github repository . Round 1 Round 1 was organised by TimeToAct Austria ( read more ). Solution using Checklist pattern with Structured Outputs took the first place. Second place used a classical vector database with LangChain. AIR - teams leveraged my AI Research TTA - teams were a part of TimeToAct community Round 2 Round 2 was organised by TimeToAct Austria ( read more ) and sponsored by IBM WatsonX AI. Teams had to build a solution that would automatically answer 100 randomly generated questions about 100 Annual Reports. Largest PDF was 1047 pages. Some questions required looking up multiple PDFs and comparing companies. Read more: ERC Submission Server (still running) Video: Keynote by Stefan Gillich: The Power of Context - RAG enhanced AI Video: Winner announcement and recap Deep dive from the winner : How I Won the Enterprise RAG Challenge and source code . Below you will find the top leaderboard for teams (regardless of their prize nomination status). This leaderboard focuses on the R&D process and will also be updated with late submissions. You can also jump straight to the deep dive from the winner of the competition : Ilya Rice: How I Won the Enterprise RAG Challenge If you want a canonical competition leaderboard - go to the TAT ERC page . Time - time it took the team to produce the results R - Retrieval Score. Max: 100 G - Generation Score. Max: 100 Score - Final score (R/3+G). Max: 133 🤗 - Team participates in AI R&D community. 🔒 - this is a fully local solution. Click on the table row to read more about the architecture and lessons learned. Show Local Models Only 🏠 Show submissions within hours Team / Experiment Time R/G Score 1. ▶ Ilia Ris 🤗 49 min 83/81 123.7 Ilya Rice Best experiment : Dense Retrieval; Router; LLM reranking; o3-mini Signature : f1d79f Summary : Dense retrieval combined with LLM reranking and SO CoT. Article : How I Won the Enterprise RAG Challenge Source code : Github Models used : o3-mini-2025-01-31 Architecture Ilya Rice solved the problem by making it easy to run numerous experiments before the competition has even started. He created an evaluation pipeline that let him quickly evaluate different architectural solutions. The best solution was also among the fastest ones. The winning experiment had this configuration: PDF Analysis: Documents are processed using a highly modified Docling Library from IBM. Modifications were needed to preserve page references. Router Pattern: First step in question answering flow picks the most suitable agent. Dense Retrieval: The system searches for relevant information based on semantic similarity ( FAISS library and OpenAI vector embeddings ). Parent Document Retrieval: Instead of retrieving only the chunk, full page is loaded to preserve relevant context. LLM Reranking: Retrieved information is re-evaluated and reordered by the LLM. Reasoning Patterns: Improve LLM accuracy within a single prompt by controlling its thinking process with Custom Chain-of-Thought and Structured Outputs. Final Answer generation: The optimized result is generated using o3-mini. Self-Consistency with Majority Vote: Multiple answer variations are generated, compared, and the most consistent one is selected. R&D Experiments Total experiments submitted : 11 Other approaches : Dense Retrieval; LLM Reranking; Router; SO CoT; o3-mini Dense Retrieval; Router; SO CoT; llama3.3-70b Dense Retrieval; Tables serialization; Router; LLM reranking; o3-mini Dense Retrieval; llama-3.3 70b Dense Retrieval; llama-3.1 8b Full Context; gemini-2.0 thinking Dense Retrieval; Router; LLM reranking; Self-Consistency; o3-mini Dense Retrieval; Router; LLM reranking; Self-Consistency; llama-3.3 70b What didn't work? Using llama-3.1 8b for reranking Incorporating Full Context with gemini-2.0 thinking Future experiments: Evaluating various local embedding models for fully offline solutions Experiment journal: 16 min → R: 83.9 , G: 72.8 , Score: 114.8 ▲ - Dense Retrieval; LLM Reranking; Router; SO CoT; o3-mini 23 min → R: 81.4 , G: 74.7 , Score: 115.4 ▲ - Dense Retrieval; llama-3.3 70b 49 min → R: 83.8 , G: 81.8 , Score: 123.7 ▲ - Dense Retrieval; Router; LLM reranking; o3-mini 50 min → R: 81.1 , G: 68.7 , Score: 109.3 - Dense Retrieval; llama-3.1 8b 51 min → R: 75.5 , G: 75.0 , Score: 112.8 - Full Context; gemini-2.0 thinking 66 min → R: 83.0 , G: 78.8 , Score: 120.3 - Dense Retrieval; Tables serialization; Router; LLM reranking; o3-mini 22 hours → R: 83.5 , G: 81.8 , Score: 123.6 - Dense Retrieval; Router; LLM reranking; o3-mini 22 hours → R: 80.8 , G: 75.7 , Score: 116.1 - Dense Retrieval; llama-3.3 70b 33 hours → R: 83.4 , G: 79.8 , Score: 121.6 - Dense Retrieval; Router; LLM reranking; Self-Consistency; o3-mini 33 hours → R: 81.3 , G: 79.7 , Score: 120.3 - Dense Retrieval; Router; LLM reranking; Self-Consistency; llama-3.3 70b 2. ▶ Emil Shagiev 🤗 55 min 86/78 121.6 Emil Shagiev Best experiment : LLM_Search Signature : 0a8782 Summary : A multi-step process involving query expansion, efficient search, question answering, and answer finalization. Models used : gpt-4o-mini-2024-07-18 gpt-4o-2024-08-06 o3-mini-2025-01-31 Architecture The best solution didn't use vector embeddings, it leveraged a structured approach: the input query is expanded to enhance search coverage and enable semantic search; relevant pages are retrieved using a cost-effective and rapid LLM; retrieved information is then passed to powerful LLM to generate answers; answers are refined and finalized for presentation. R&D Experiments Total experiments submitted : 3 Other approaches : LLL_Search_2: Similar architecture with added capability for mathematical operations. Experiment journal: 55 min → R: 86.3 , G: 78.5 , Score: 121.6 ▲ - LLM_Search 21 hours → R: 86.1 , G: 77.5 , Score: 120.5 - LLL_Search_2 3. ▶ Dmitry Buykin 🤗 8 hours 81/76 117.5 Dmitry Buykin Best experiment : Dynamic Structured Output with SEC EDGAR Ontologies Signature : 6b0d78 Summary : Dynamic structured output with query expansion and page-focused chunking. Models used : gpt-4o-2024-08-06 Architecture Used SO/CoT approach with ontologies to retrieve relevant information. Key highlights: embeddings and vector databases were not used; dynamic structured output approach combined with SEC EDGAR ontologies for query expansion (SO CoT); utilized CBOW similarity for majority selection across multiple runs, focusing on balancing pages versus tokens during chunking significant effort was dedicated to evaluating PDF quality heuristics to optimize OCR input synthetic tags were implemented to stabilize page detection and assess model quality. 4. ▶ Sergey Nikonov 🤗 30 hours 85/73 116.4 Sergey Nikonov Best experiment : main v2 Signature : 00c0e1 Summary : For every question, all pages are processed using gpt-4o. Models used : gpt-4o o1-mini Architecture Solution involves feeding all pages of the provided documents into the gpt-4o model for each question. This simple but practical approach ensures comprehensive coverage of the content to extract accurate answers. R&D Experiments Total experiments submitted : 2 Other approaches : Finding the PDFs that correspond to questions, cutting the PDFs by page, running the question against each PDF page by loading the PDF directly into gpt-4o (through the assistant API), scanning all PDF pages for the answer, and combining the answers by simple logic. What didn't work? Using the o3-mini model instead of o1-mini in the architecture. Experiment journal: 5 hours → R: 85.3 , G: 69.0 , Score: 111.6 ▲ - Main 30 hours → R: 85.1 , G: 73.9 , Score: 116.4 ▲ - main v2 5. ▶ ScrapeNinja.net 🤗 23 hours 82/71 112.5 ScrapeNinja.net Best experiment : fixed multiple companies search Signature : 417bbf Summary : Node.js-based architecture utilizing pgvector for efficient data handling. Models used : Gemini Flash 2.0 Gemini Flash Lite 2.0 Flash Thinking Exp Architecture The solution used Node.js for backend operations and pgvector for vectorized data processing. It focused on efficient handling of complex queries and data retrieval tasks. The team utilized: Gemini Flash 2.0 Gemini Flash Lite 2.0 Flash Thinking Exp. R&D Experiments Total experiments submitted : 2 Other approaches : OCR and PG Experiment journal: 20 hours → R: 82.6 , G: 64.2 , Score: 105.5 ▲ - OCR and PG 23 hours → R: 82.6 , G: 71.2 , Score: 112.5 ▲ - fixed multiple companies search 6. ▶ xsl777 🤗 16 hours 79/71 110.9 xsl777 Best experiment : multi-query, gpt-4o Signature : 66ab5c Summary : Structured PDF parsing, metadata extraction, query expansion, hybrid search, reranking, and CoT. Models used : gpt-4o gpt-4o-mini Architecture The architecture integrates following patterns: structured PDF parsing and chunking; metadata extraction; query expansion; hybrid search mechanisms; reranking strategies. It synthesizes document metadata and chunks while utilizing Chain-of-Thought (CoT) reasoning to enhance response accuracy and relevance. gpt-4o and gpt-4o-mini help with high-quality language understanding and generation capabilities. R&D Experiments Total experiments submitted : 2 Experiment journal: 16 hours → R: 79.4 , G: 71.2 , Score: 110.9 ▲ - multi-query, gpt-4o 3 days → R: 80.1 , G: 70.7 , Score: 110.7 - Open source, Advanced RAG 7. ▶ nikolay_sheyko(grably.tech) 🤗 25 hours 81/69 110.4 nikolay_sheyko(grably.tech) Best experiment : nikolay_sheyko(grably.tech)_with_o3_mini Signature : db8938 Summary : Relevant pages are identified and processed to generate answers. Models used : gpt-4o-mini o3-mini Architecture The solution employs a two-step process: first, it identifies relevant reports for a given question and evaluates the relevance of each page asynchronously using the gpt-4o-mini model; then , all relevant pages are compiled into a prompt, and the o3-mini model is utilized to generate the final answer. R&D Experiments Total experiments submitted : 7 Other approaches : Dynamic data extraction with pydantic classes Binary checks per page Parallel question splitting Subquestion generation for multi-entity queries Single-page reference experiments What didn't work? Binary checks per page Single-page reference experiments Experiment journal: 55 min → R: 77.2 , G: 51.2 , Score: 89.9 ▲ - grably.tech/with_extra_reasoning_from_different_pages_hacked96160725 25 hours → R: 81.1 , G: 69.8 , Score: 110.4 ▲ - nikolay_sheyko(grably.tech)_with_o3_mini 25 hours → R: 79.7 , G: 60.2 , Score: 100.1 - nikolay_sheyko(grably.tech)_dummy 8 days → R: 80.5 , G: 64.3 , Score: 104.6 - o3-mini-no-restrictions 8 days → R: 80.5 , G: 66.3 , Score: 106.6 - o3-mini-no-restrictions-fixed-names 12 days → R: 81.2 , G: 67.1 , Score: 107.7 - o3-mini-no-restrictions-single-reference 12 days → R: 80.5 , G: 67.3 , Score: 107.6 - o3-mini-no-restrictions-fixed-names-and-boolean 8. ▶ Felix-TAT 🤗 7 days 80/69 109.4 Felix-TAT Best experiment : Gemini-4o Multiagent RAG Signature : a2faff Summary : Multiagent, mixed-model approach with delegation and execution agents. Models used : gemini-2.0-flash gpt-4o-2024-08-06 Architecture The solution uses a multiagent architecture where a delegation manager (OpenAI) splits the user query into company-specific subqueries. These subqueries are processed by expert agents using Google's Gemini flash model, which has access to the entire company PDF in context. The responses are then aggregated and synthesized by an execution agent (OpenAI) to produce the final answer. R&D Experiments Total experiments submitted : 4 Other approaches : Gemini Naive IBM-4o-based Multiagent RAG OpenAI Multiagent RAG What didn't work? Using a single model without multiagent delegation Relying solely on vector database retrieval without full PDF context Experiment journal: 6 days → R: 79.0 , G: 60.3 , Score: 99.8 ▲ - Gemini Naive 7 days → R: 81.7 , G: 47.3 , Score: 88.2 - IBM-4o-based Multiagent RAG 7 days → R: 82.2 , G: 66.0 , Score: 107.1 ▲ - OpenAI Multiagent RAG 7 days → R: 80.2 , G: 69.3 , Score: 109.4 ▲ - Gemini-4o Multiagent RAG 9. ▶ A.Rasskazov/V.Kalesnikau 30 hours 84/67 109.3 A.Rasskazov/V.Kalesnikau Best experiment : multi_agent_ibm_openai Signature : efabd4 Summary : A multi-agent system leveraging LLMs for question answering using similarity-based retrieval. Models used : meta-llama/llama-3-405b-instruct ibm/granite-embedding-107m-multilingual text-embedding-3-small gpt-4o-mini Architecture The solution employs a multi-agent architecture to address the challenge. Initially, it generates a database for the Retrieval-Augmented Generation (RAG) model. Upon receiving a query, the system extracts key metrics such as company, industry, and currency. These metrics are then used to identify the most similar question in the database. The answer associated with this similar question is retrieved and refined using a Large Language Model (LLM). Finally, the system consolidates and presents the answer to the user. R&D Experiments Total experiments submitted : 2 Other approaches : pjatk_team_002: A system that preprocesses questions, retrieves relevant PDF pages using a vector database, and extracts answers with page references using LLMs. What didn't work? Alternative embedding models for retrieval. Different strategies for key metric extraction. Experiment journal: 30 hours → R: 84.0 , G: 67.2 , Score: 109.3 ▲ - multi_agent_ibm_openai 7 days → R: 82.5 , G: 64.0 , Score: 105.2 - pjatk_team_002 10. ▶ Dany the creator 🤗 3 hours 82/67 108.4 Dany the creator Best experiment : gpt-4o-mini + pgvector Signature : ee29ae Summary : Utilized a structured approach to parse and analyze text chunks, creating embeddings and generating questions. Models used : gpt-4o-mini Architecture The solution preprocesses text by chunking, generating embeddings with pgvector library, and formulating questions that could be answered by the respective chunks. 11. ▶ SergC 🤗 7 days 77/69 108.1 SergC Best experiment : submission_1 Signature : c0d776 Summary : QE + SO + CoT Models used : gemini 2.0 Architecture The solution uses a combination of: Query Expansion (QE) Semantic Optimization (SO) Chain of Thought (CoT) reasoning to enhance the performance of the Gemini 2.0 model. 12. ▶ Swisscom Innovation Lab 🔒 21 hours 83/66 107.8 Swisscom Innovation Lab Best experiment : Multi-Agent Langgraph-Llamaindex-MarkerPDF-Llama3.3 Signature : debcf6 Summary : A multi-agent system leveraging LangGraph, LlamaIndex, MarkerPDF, and Llama 3.3 for accurate and contextual multi-company query processing. Models used : llama-3.3-70b-instruct Architecture This offline solution uses a multi-agent architecture with: LangGraph for workflow orchestration LlamaIndex for data indexing MarkerPDF for document parsing Llama 3.3 for natural language processing. Solution supports multi-company queries by: extracting relevant entities validating inputs processing each entity individually retrieving and evaluating documents aggregating results for numeric-based comparisons. R&D Experiments Total experiments submitted : 3 Other approaches : Iterative refinement of query processing pipeline Enhanced document retrieval mechanisms What didn't work? Simplified single-agent architecture Direct query-to-response mapping without intermediate validation Experiment journal: 80 min → R: 83.3 , G: 65.2 , Score: 106.8 ▲ - Multi-Agent Langgraph-Llamaindex-MarkerPDF-Llama3.3 21 hours → R: 83.3 , G: 66.2 , Score: 107.8 ▲ - Multi-Agent Langgraph-Llamaindex-MarkerPDF-Llama3.3 13. ▶ fomih 🤗 10 days 83/65 107.4 fomih Best experiment : gemini-flash CoT with question type detection fixes Signature : 60bc28 Summary : Enhanced question type detection for improved accuracy. Models used : gemini-flash 2.0 Architecture The solution utilized the gemini-flash 2.0 model, incorporating a refined approach to question type detection. This enhancement aimed to improve the accuracy and relevance of the responses generated by the system. The architecture involved preprocessing input documents into structured formats, creating knowledge bases tailored to specific question types, and leveraging these resources during the question-answering phase. The system identified the question type and relevant entities, retrieved pertinent knowledge base entries, and generated answers by combining the question with the retrieved data. R&D Experiments Total experiments submitted : 4 Other approaches : gemini-flash CoT with structured output gemini-flash CoT with structured output and small fixes gemini CoT with structured output final What didn't work? Initial handling of 'n/a' cases Fallback processing without structured knowledge bases Experiment journal: 10 days → R: 83.2 , G: 59.9 , Score: 101.5 ▲ - _gemini-flash CoT + structured output _ 10 days → R: 82.9 , G: 62.8 , Score: 104.3 ▲ - gemini-flash CoT + structured output small n/a handling fixex 10 days → R: 83.0 , G: 65.9 , Score: 107.4 ▲ - gemini-flash CoT + so small fixes in question type detection 12 days → R: 83.3 , G: 64.4 , Score: 106.1 - gemini CoT + SO final 14. ▶ Al Bo 12 days 81/65 105.9 Al Bo Best experiment : albo Signature : 1e89b6 Summary : Docling, Vector, Agent with search tool into documents Models used : gpt-4o Architecture The solution utilized a sophisticated architecture combining document processing (Docling), vector-based representation, and an agent equipped with a search tool for document retrieval. 15. ▶ NumericalArt 8 days 70/70 105.3 NumericalArt Best experiment : Vhck-R0-002 Signature : 32aae7 Summary : Preprocessing questions, raw retrieval, filtering, retrieval, detailed page analysis, and answer generation. Models used : 4o-mini 4o 3o-mini Architecture The best employs a structured approach to information retrieval and answer generation. The process begins with preprocessing the input questions to enhance clarity and relevance. This is followed by an initial raw retrieval phase to gather potential information sources. Subsequently, a filtering mechanism is applied to refine the retrieved data. The refined data undergoes a detailed page analysis to extract precise and contextually relevant information. Finally, the system generates answers based on the analyzed data, leveraging the capabilities of the LLM models 4o-mini, 4o, and 3o-mini. R&D Experiments Total experiments submitted : 2 Other approaches : Parsing text from PDFs only, separate VDB for each document, one chunk equals one page, extract four pages by entity value from question (excluding company name), detailed parsing of extracted pages, asking LLM question with detailed information in context. Experiment journal: 7 days → R: 75.9 , G: 63.3 , Score: 101.3 ▲ - Vhck-R0 8 days → R: 70.0 , G: 70.3 , Score: 105.3 ▲ - Vhck-R0-002 16. ▶ Pedro Ananias 🤗 4 hours 80/64 104.9 Pedro Ananias Best experiment : rag-3w-cot-gpt-4o-mini Signature : d44b72 Summary : A 3-way FAISS MMR Search & Stepped Chain Of Thought RAG Models used : openai/gpt-4o-mini Architecture The solution uses a 3-way FAISS MMR Search mechanism combined with a Chain Of Thought (CoT) approach. FAISS MMR Search involves query expansion, file selection based on exact matches and cosine similarity, and database searching using maximum marginal relevance. CoT pipeline consists of three sequential model calls with specific prompts for reasoning, formatting, and parsing. This architecture leverages the openai/gpt-4o-mini model for processing. R&D Experiments Total experiments submitted : 5 Other approaches : rag-3w-cot-gpt-4o-mini-hi-res rag-3w-cot-deepseek-r1-distill-llama-8B-fast-fp16 rag-3w-cot-deepseek-r1-distill-llama-8B-hi-res-fp16 rag-3w-cot-microsoft-phi4-14B-hi-res-int8 What didn't work? Using lower resolution PDF extraction for certain tasks Employing fully local processing without cloud integration in some scenarios Experiment journal: 4 hours → R: 80.4 , G: 64.7 , Score: 104.9 ▲ - rag-3w-cot-gpt-4o-mini 9 hours → R: 70.6 , G: 56.0 , Score: 91.3 - rag-3w-cot-deepseek-r1-distill-llama-8B-fast-fp16 9 hours → R: 77.0 , G: 64.6 , Score: 103.1 - rag-3w-cot-gpt-4o-mini-hi-res 11 hours → R: 72.3 , G: 58.0 , Score: 94.2 - rag-3w-cot-deepseek-r1-distill-llama-8B-hi-res-fp16 31 hours → R: 78.1 , G: 59.7 , Score: 98.7 - rag-3w-cot-microsoft-phi4-14B-hi-res-int8 17. ▶ Daniyar 3 days 62/72 104.1 Daniyar Best experiment : Fixed reference page indices Signature : 8bb723 Summary : The architecture utilizes fixed reference page indices for efficient information retrieval. Models used : gpt-4o Architecture Solution uses a strategy of fixed reference page indices to enhance the accuracy and efficiency of document parsing and question answering. This approach ensures that the model can quickly locate and utilize relevant information from the provided documents, leveraging the capabilities of the GPT-4o model. R&D Experiments Total experiments submitted : 2 Other approaches : Sliding window PDF page reading with checklists over questions addressed to files. What didn't work? Alternative indexing methods or dynamic page referencing strategies. Experiment journal: 3 days → R: 62.2 , G: 72.9 , Score: 104.0 ▲ - First draft 3 days → R: 62.4 , G: 72.9 , Score: 104.1 ▲ - Fixed reference page indices 18. ▶ RubberduckLabs 🔒 2 days 74/66 103.3 RubberduckLabs Best experiment : RubberduckLabs - RAG experiment attempt 001 Signature : ee7519 Summary : A multi-step LLM processing pipeline for document question-answering. Models used : deepseek-r1-distill-llama-70b:bf16 llama-3.1-70b-instruct:bf16 Architecture The architecture preprocesses documents to generate detailed page-level summaries and extracting structured metadata, particularly focusing on financial data. The retrieval process employs a two-stage approach: document selection based on metadata matching; precise page identification using semantic relevance and explicit reasoning. Answer generation utilizes 'Context-Guided Response Generation' combining retrieved contexts with structured reasoning to ensure factual accuracy and traceability. The system maintains explicit reasoning trails and incorporates robust error handling for production stability. R&D Experiments Total experiments submitted : 2 19. ▶ Machine Learning Reply 28 hours 74/66 103.2 Machine Learning Reply Best experiment : ML Reply - Submission 1 Signature : fa34f3 Summary : Integration of Azure Document Intelligence and Azure AI Search. Models used : GPT-4o Architecture This solution utilized a combination of Azure Document Intelligence for document processing and Azure AI Search for efficient information retrieval. R&D Experiments Total experiments submitted : 2 Other approaches : ML Reply - Submission 2 Experiment journal: 28 hours → R: 74.5 , G: 66.0 , Score: 103.2 ▲ - ML Reply - Submission 1 29 hours → R: 74.0 , G: 63.5 , Score: 100.5 - ML Reply - Submission 2 20. ▶ Aleksandr Podgaiko 🤗 3 days 81/62 103.0 Aleksandr Podgaiko Best experiment : smolagent_simple_v1 Signature : 6afedb Summary : Utilized smolagents library with basic PDF extraction and a coding agent. Models used : openrouter/google/gemini-2.0-flash-001 Architecture The solution employed the HuggingFace smolagents library for agent-based interactions, integrating basic PDF extraction using PyPDF2 . The architecture featured a default coding agent equipped with two tools: pdf_search for keyword-based search with contextual display and pdf_content for full-page content retrieval upon request. Additionally, the final_answer tool was customized to adhere to the submission format. 21. ▶ Vlad Drobotukhin (@mrvladd) 🤗 🔒 6 days 68/68 102.3 Vlad Drobotukhin (@mrvladd) Best experiment : Qwen 2.5-72b + Multi-Query BM25 + Domain-Specific Information Extraction + Router Signature : fa77e2 Summary : System combining LLM-based reasoning with optimized retrieval techniques. Models used : Qwen-2.5-72b-INT4 Architecture This offline solution employs a multi-step process: start with question analysis to determine the type and domain; generate multiple search queries to maximize recall; relevant pages are retrieved using OpenSearch and processed with domain-specific LLM extractors to build structured knowledge; final answers are synthesized with reasoning and confidence scores. R&D Experiments Total experiments submitted : 10 Other approaches : Qwen2.5 72b + FTS (rephrase query) +SO + CheckList's Qwen2.5 72b + FTS +SO + CheckList's Qwen2.5 + FTS (rephrase query) + SO + CheckList's Qwen 2.5-72b + Multi-Query BM25 + Domain-Specific Information Extraction Qwen 2.5-72b + Multi-Query BM25 (top 15 pages) + Domain-Specific Information Extraction + Router Qwen 2.5-72b + Multi-Query BM25+ Domain-Specific Information Extraction + Router Qwen 2.5-72b-4bit + BM25 + Domain-Specific Information Extraction + Router MagicQwen-4bit + BM25 + Domain-Specific Information Extraction + Router Qwen 72b-4bit + FTS + Domain-Specific Information Extraction 0803 What didn't work? Simplified query generation without diversification Lack of domain-specific term boosting Absence of structured output validation Experiment journal: 3 days → R: 74.7 , G: 59.2 , Score: 96.5 ▲ - Qwen2.5 72b + FTS (rephrase query) +SO + CheckList's 3 days → R: 71.8 , G: 62.3 , Score: 98.2 ▲ - Qwen2.5 72b + FTS +SO + CheckList's 4 days → R: 74.7 , G: 59.2 , Score: 96.5 - Qwen2.5 + FTS (rephrase query) + SO + CheckList's 5 days → R: 69.1 , G: 65.7 , Score: 100.2 ▲ - Qwen 2.5-72b + Multi-Query BM25 + Domain-Specific Information Extraction 6 days → R: 68.3 , G: 68.2 , Score: 102.3 ▲ - Qwen 2.5-72b + Multi-Query BM25 + Domain-Specific Information Extraction + Router 7 days → R: 67.6 , G: 67.4 , Score: 101.2 - Qwen 2.5-72b + Multi-Query BM25 (top 15 pages) + Domain-Specific Information Extraction + Router 8 days → R: 64.6 , G: 62.0 , Score: 94.3 - Qwen 2.5-72b + Multi-Query BM25+ Domain-Specific Information Extraction + Router 9 days → R: 61.9 , G: 63.0 , Score: 93.9 - Qwen 2.5-72b-4bit + BM25 + Domain-Specific Information Extraction + Router 9 days → R: 69.2 , G: 63.2 , Score: 97.8 - MagicQwen-4bit + BM25 + Domain-Specific Information Extraction + Router 10 days → R: 78.4 , G: 63.0 , Score: 102.2 - Qwen 72b-4bit + FTS + Domain-Specific Information Extraction 0803 22. ▶ Ivan R. 🤗 71 min 79/62 101.9 Ivan R. Best experiment : Round 2 submission Signature : b29973 Summary : A multi-step approach leveraging LLMs for question decomposition, search, and validation. Models used : gpt-4o gpt-4o-mini Architecture The solution employs a structured pipeline: document loading using PyPDFDirectoryLoader from LangChain; question decomposition with GPT-4o; multiple OpenAI assistants, each dedicated to a specific company, perform targeted searches using GPT-4o-mini; results undergo answer validation with GPT-4o local FAISS vector store is used for similarity search to collect reference pages. 23. ▶ PENZA_AI_CREW 🤗 7 days 72/65 101.3 PENZA_AI_CREW Best experiment : gpt-4_claude3.5_unstructured Signature : 67ee86 Summary : A multi-step pipeline leveraging OCR, table/image analysis, and knowledge mapping for accurate question answering. Models used : gpt-4-mini claude 3.5 gpt-4o Architecture This RAG pipeline was composed of the following steps: PDF text is parsed using Unstructured library with OCR Tables and images are analyzed using Claude 3.5 Knowledge map is constructed using gpt-4-mini, utilizing Structured Outputs. Questions are analyzed in conjunction with the knowledge map using gpt-4-mini with Pydantic schema. Answers are generated by gpt-4o, employing chain-of-thought reasoning and Pydantic schema (SO CoT). R&D Experiments Total experiments submitted : 2 Other approaches : RAG_PNZ_PAYPLINE: OCR with Unstructured, table/image analysis with Claude 3.5, metadata extraction with gpt-4-mini, and final reasoning with gpt-4o. What didn't work? Alternative OCR methods not utilizing Unstructured. Direct question answering without intermediate knowledge mapping. Experiment journal: 7 days → R: 12.2 , G: 11.0 , Score: 17.1 ▲ - RAG_PNZ_PAYPLINE 7 days → R: 72.5 , G: 65.0 , Score: 101.3 ▲ - gpt-4_claude3.5_unstructured 24. ▶ Yolo leveling 25 hours 82/59 101.0 Yolo leveling Best experiment : Marker + Gemini Signature : 31b473 Summary : Convert PDFs to markdown, extract company names, and generate JSON representations. Models used : Surya (OCR) Flash 2.0 Architecture The solution starts converting each PDF document into markdown format using the Marker tool with OCR capabilities. Afterward, the system identifies the company name within the content. In cases where multiple companies are mentioned in the query, the system employs a hallucination control mechanism to determine the most relevant company. The markdown content is then incorporated into the context for the LLM, which extracts and generates a structured JSON representation of the required information. R&D Experiments Total experiments submitted : 2 Other approaches : Gemini 1M pdf "thinking" + 4o parser What didn't work? Queries involving multiple companies were marked as N/A in alternative approaches. Experiment journal: 25 hours → R: 76.0 , G: 60.0 , Score: 98.0 ▲ - Gemini 1M pdf "thinking" + 4o parser 25 hours → R: 82.2 , G: 59.9 , Score: 101.0 ▲ - Marker + Gemini 25. ▶ ArtemNurm 🤗 7 days 77/61 99.9 ArtemNurm Best experiment : brute_flash2.0&brute_flash2.0 Signature : 46e0e0 Summary : PDF2MD with Flash, relevant data extraction with Flash, the data is sent to LLM with questions using SO (no CoT). All steps include generator-critic workflow. Models used : Gemini Flash 2.0 OpenAI o3-mini Architecture The winning experiment employs a robust architecture leveraging the Gemini Flash 2.0 and OpenAI o3-mini models. The process involves converting PDF documents to Markdown format using Flash, extracting relevant data, and querying the LLM with specific questions using a straightforward approach without chain-of-thought reasoning. A generator-critic workflow is integrated into all steps to ensure high-quality outputs. R&D Experiments Total experiments submitted : 8 Other approaches : brute_flash2.0&CoT_flash2.0 index_flash2.0&brute_flash2.0 index_flash2.0&CoT_4o-2024-11-20 index_flash2.0&CoT_flash2.0 index_flash2.0&CoT_o3-mini-high index_flash2.0&CoT_o3-mini flash2.0_sees_all_content What didn't work? Using chain-of-thought reasoning in 'brute_flash2.0&CoT_flash2.0' did not outperform the winning approach. Concatenating all Markdown files into a single string in 'flash2.0_sees_all_content' was less effective. Experiment journal: 7 days → R: 77.8 , G: 61.0 , Score: 99.9 ▲ - brute_flash2.0&brute_flash2.0 7 days → R: 77.7 , G: 61.0 , Score: 99.8 - brute_flash2.0&CoT_flash2.0 7 days → R: 68.5 , G: 57.6 , Score: 91.8 - index_flash2.0&brute_flash2.0 7 days → R: 66.4 , G: 56.8 , Score: 90.0 - index_flash2.0&CoT_4o-2024-11-20 7 days → R: 66.3 , G: 57.6 , Score: 90.7 - index_flash2.0&CoT_flash2.0 7 days → R: 65.6 , G: 58.8 , Score: 91.6 - index_flash2.0&CoT_o3-mini-high 7 days → R: 65.9 , G: 59.3 , Score: 92.2 - index_flash2.0&CoT_o3-mini 7 days → R: 71.8 , G: 55.6 , Score: 91.4 - flash2.0_sees_all_content 26. ▶ ndt by red_mad_robot 🤗 🔒 9 days 72/63 99.7 ndt by red_mad_robot Best experiment : qwen32b+bge_m3 Signature : 30f0d1 Summary : PDFs were converted to markdown, vectorized using bge m3, and queried with Qwen 32B. Models used : Qwen 32B instruct BGE-M3 Architecture This offline solution involved processing PDF documents by converting them into markdown format using the Pymupdf library. These markdown representations were then vectorized using the popular BGE-M3 model. Qwen 32B instruct model was used to answer user queries by leveraging the vectorized data for relevant context retrieval. R&D Experiments Total experiments submitted : 5 Other approaches : full open-source + roter agent qwen7b-router-agent What didn't work? Directly querying without vectorization Using alternative LLMs for vectorization Experiment journal: 23 hours → R: 27.2 , G: 54.0 , Score: 67.6 ▲ - full open-source + roter agent 7 days → R: 73.2 , G: 51.0 , Score: 87.6 ▲ - qwen7b-router-agent 9 days → R: 73.2 , G: 59.0 , Score: 95.6 ▲ - ndt by red_mad_robot 9 days → R: 72.9 , G: 63.2 , Score: 99.7 ▲ - qwen32b+bge_m3 27. ▶ Neoflex DreamTeam 🤗 🔒 30 hours 77/58 96.9 Neoflex DreamTeam Best experiment : Simple LLM Brute Force Signature : 34a266 Summary : Utilized a straightforward LLM brute force approach for each page with predefined questions and example answers. Models used : Qwen 2.5 Architecture Solution used Qwen 2.5 model to process each page individually, applying a brute force methodology with a set of predefined questions and corresponding example answers to extract relevant information effectively. R&D Experiments Total experiments submitted : 2 Other approaches : Checklist based RAG What didn't work? Alternative configurations of the Checklist based RAG approach Experiment journal: 30 hours → R: 77.8 , G: 58.0 , Score: 96.9 ▲ - Best run 7 days → R: 67.3 , G: 51.7 , Score: 85.4 - neon_team 28. ▶ nightwalkers 🔒 6 hours 72/60 96.7 nightwalkers Best experiment : nightwalkers-baseline Signature : 356ef4 Summary : Utilized a vector database for efficient document retrieval and LLM for response generation. Models used : deepseek-r1-distill-llama-70b Architecture The team implemented vector database search using embeddings from all-MiniLM-L6-v2 and ibm/granite-embedding-107m-multilingual models. This facilitated the retrieval of the most relevant page and document based on the query. The retrieved information was then processed by the deepseek-r1-distill-llama-70b LLM to generate relevant answers. 29. ▶ Gleb Kozhaev 🤗 32 hours 79/56 95.5 Gleb Kozhaev Best experiment : pymupdf4llm + Structured Output Signature : 1442cb Summary : Utilized pymupdf4llm with structured output and three distinct system prompts/roles. Models used : gpt-4o-mini Architecture RAG solution employed the pymupdf4llm framework, leveraging Structured Outputs to enhance data processing and comprehension. Three distinct system prompts/roles were utilized to optimize the model's performance and ensure accurate and efficient results. 30. ▶ AndreiKopysov 🤗 33 hours 76/57 95.3 AndreiKopysov Best experiment : Gemini2.0 and DeepSeek R1 Integration Signature : 574182 Summary : The architecture processes PDF pages using Gemini2.0 and refines responses with DeepSeek R1. Models used : Gemini2.0 DeepSeek R1 Architecture This RAG solution used a two-step pipeline: each page of the PDF document is processed using the Gemini2.0 model to extract relevant information; extracted responses are refined and analyzed using the DeepSeek R1 model to ensure accuracy and relevance. R&D Experiments Total experiments submitted : 2 Other approaches : Reused the same architecture in different configurations. Experiment journal: 33 hours → R: 76.2 , G: 57.2 , Score: 95.3 ▲ - AndreiKopysov 33 hours → R: 76.2 , G: 57.2 , Score: 95.3 - AndreyKopysov 31. ▶ Serj Tarasenko 3 days 82/54 95.0 Serj Tarasenko Best experiment : complicated second Signature : a5cf25 Summary : RAG pipeline with query enhancement and re-ranking. Models used : gpt-4o-mini text-embedding-3-small Architecture The winning solution implemented a Retrieval-Augmented Generation (RAG) pipeline. The process involved extracting content from PDFs, segmenting it into manageable chunks, and indexing these chunks using FAISS for efficient vector-based retrieval. Queries were enhanced with financial terms to improve relevance, followed by a retrieval step that included re-ranking to prioritize the most pertinent information. Finally, an LLM was employed to generate comprehensive answers based on the retrieved data. The source code for this implementation is publicly available. 32. ▶ AAV 7 days 62/62 93.9 AAV Best experiment : Agent+Router Signature : 5e0479 Summary : The architecture employs an agent-based approach with a routing mechanism. Models used : gpt-4o-mini Architecture The solution uses the 'gpt-4o-mini' model in an architecture combining an agent with a router. This design enables efficient task delegation and processing, optimizing performance for the challenge requirements. R&D Experiments Total experiments submitted : 6 Other approaches : Agent Agent + sim search + tfidf What didn't work? Using 'private model' instead of 'gpt-4o-mini' Excluding the router component Experiment journal: 7 days → R: 60.7 , G: 62.8 , Score: 93.1 ▲ - llm1-sim-preselected 7 days → R: 62.9 , G: 62.5 , Score: 93.9 ▲ - llm2-sim-preselected 7 days → R: 62.7 , G: 57.3 , Score: 88.7 - llm2-sim-not-preselected 7 days → R: 61.0 , G: 60.8 , Score: 91.3 - llm1-sim-not-preselected 7 days → R: 25.1 , G: 60.9 , Score: 73.5 - llm1-sim-ifidf-not-preselected 7 days → R: 27.2 , G: 62.8 , Score: 76.4 - llm2-sim-tfidf-not-preselected 33. ▶ AI Slop 🤗 3 hours 80/53 93.5 AI Slop Best experiment : AI Slop Cursor+Sonnet 3.7 Signature : fc3dc9 Summary : Utilized a streamlined approach leveraging LLMs for direct question answering. Models used : gpt-4o-mini Architecture The team employed the gpt-4o-mini model to process and answer questions directly from the provided PDF documents. By utilizing metadata and targeted queries, they efficiently narrowed down relevant information, ensuring accurate and concise responses. The approach avoided complex retrieval-augmented generation (RAG) or OCR techniques, focusing on the inherent capabilities of the LLM. 34. ▶ RAG challenge Orphist 🔒 63 min 78/53 92.4 RAG challenge Orphist Best experiment : Iterative LLM Prompting with BM25 Signature : e98c1b Summary : The solution employs BM25 for document retrieval and iterative LLM prompting for query expansion and summarization. Models used : gemma-2-9b-it Architecture The solution utilized an architecture combining BM25plus for document retrieval and iterative prompting of the gemma-2-9b-it LLM. The process involved chunking PDF documents for ingestion, storing them in an in-memory local storage, and applying BM25plus for query matching with meta-filters. Due to a last-minute issue with embedding models, the team opted for a non-hybrid pipeline. The iterative prompting expanded the initial query and used a scratchpad for summary collection, culminating in a final prompt to extract the requested information. 35. ▶ Dennis S. 🤗 7 days 81/50 91.0 Dennis S. Best experiment : Deepseek naive questionfilter Signature : 53630f Summary : A question-centered approach leveraging document parsing and heuristic-based analysis. Models used : Deepseek V3 Architecture The solution employs a question-centered methodology to efficiently extract relevant information from documents. Initially, PDFs are parsed using PyMuPDF and Tesseract for OCR when necessary. The system analyzes provided metadata and questions to identify relevant companies and metrics, classifying questions into single_fact or aggregate types. It processes documents in parallel, extracting answers based on the question type, and aggregates results accordingly. This approach prioritizes speed and cost-efficiency. R&D Experiments Total experiments submitted : 2 Other approaches : Deepseek v3 - bruteforce questionfilter What didn't work? Using regex-based logic for question classification Dividing questions into first occurrence and aggregated types without clear pipeline integration Experiment journal: 7 days → R: 79.8 , G: 50.0 , Score: 89.9 ▲ - Deepseek v3 - bruteforce questionfilter 7 days → R: 81.9 , G: 50.0 , Score: 91.0 ▲ - Deepseek naive questionfilter 36. ▶ Slava RAG 🤗 7 hours 65/57 90.7 Slava RAG Best experiment : Slava RAG Signature : 282787 Summary : Embedding: OpenAI text-embedding-3-small, LLM: GPT-4o, Vector Database: Pinecone, PDF Processing: PyMuPDF, Chunk Processing: Custom algorithm Models used : gpt-4o Architecture This architecture combined: OpenAI's text-embedding-3-small for embedding generation; GPT-4o as the primary LLM; Pinecone for vector database management; PyMuPDF for efficient PDF processing; a custom algorithm for chunk processing. 37. ▶ Alex_dao 95 min 68/56 90.7 Alex_dao Best experiment : Alex_Dao_v1_final Signature : 93c0ef Summary : Utilized a kv-index architecture. Models used : gpt4o Architecture The winning solution implemented a key-value index (kv-index) architecture, leveraging the capabilities of the GPT-4 model (gpt4o) to efficiently retrieve and process information. This approach ensured high performance and accuracy in the challenge tasks. 38. ▶ Mykyta Skrypchenko 🤗 31 hours 42/64 85.3 Mykyta Skrypchenko Best experiment : Kyiv-bge1.5 Signature : d5fb15 Summary : Integration of advanced text retrieval and vector database with LLM for question answering. Models used : gpt-4o-2024-08-06 Architecture The solution is a multi-component architecture: Fitz for efficient text retrieval BAAI/bge-base-en-v1.5 Sentence Transformer for embedding generation ChromaDB as the vector database for storage and retrieval OpenAI API for question answering 39. ▶ F-anonymous 🤗 🔒 5 days 73/47 83.8 F-anonymous Best experiment : Fully local, own DeepThinking Signature : 2a2a1b Summary : Fully local graphRAG with hybrid search and custom-tuned LLM. Models used : Qwen2.5 14b Architecture The solution by F-anonymous a fully local graph-based Retrieval-Augmented Generation (RAG) architecture. They utilized their proprietary DeepThinking framework in conjunction with a custom-tuned Qwen2.5 14b model. The system integrated a hybrid search mechanism combining vector-based and BM25 methodologies to enhance retrieval accuracy and relevance. 40. ▶ DataNXT 🔒 5 days 54/55 82.6 DataNXT Best experiment : Prototype-RAG-Challenge Signature : 0e942a Summary : Pipeline with specialised prompted LLM Calls Models used : OpenAi-4o-mini Architecture The solution utilized a pipeline architecture with specialized prompted calls to the OpenAi-4o-mini model. This approach allowed for efficient and accurate information retrieval and generation. 41. ▶ AValiev 🔒 4 hours 43/60 81.8 AValiev Best experiment : IBM-deepseek-agentic-rag Signature : 493744 Summary : Agentic RAG with type validation, Pydantic typing, Qdrant vector store querying. Models used : deepseek/deepseek-r1-distill-llama-70b Architecture This RAG solution was based on an Agentic Retrieval-Augmented Generation (RAG) architecture. It utilized type validation and Pydantic typing for robust data handling, and Qdrant vector store querying for efficient information retrieval. PDF documents were processed using PyPDF and Docling for accurate text extraction. R&D Experiments Total experiments submitted : 5 Other approaches : openai-agentic-rag IBM-mixtral-agentic-rag granite-3-8b-instruct_rag_agentic deepseek/deepseek-r1-distill-llama-70b_sophisticated_chunking_rag_agentic What didn't work? Alternative LLM models such as OpenAI-gpt-4o-mini and mistralai/mixtral-8x7b-instruct-v01 were explored but did not achieve the same performance as the winning model. Experiment journal: 54 min → R: 43.5 , G: 60.0 , Score: 81.8 ▲ - openai-agentic-rag 3 hours → R: 43.5 , G: 33.0 , Score: 54.8 - IBM-mixtral-agentic-rag 4 hours → R: 43.5 , G: 60.0 , Score: 81.8 - IBM-deepseek-agentic-rag 4 hours → R: 43.5 , G: 48.5 , Score: 70.2 - granite-3-8b-instruct_rag_agentic 34 hours → R: 35.8 , G: 53.0 , Score: 70.9 - deepseek/deepseek-r1-distill-llama-70b_sophisticated_chunking_rag_agentic 42. ▶ bimurat_mukhtar 🤗 🔒 32 hours 36/31 49.4 bimurat_mukhtar Best experiment : bm_v1 Signature : c25e30 Summary : Multi-agent architecture with specialized branches for diverse answer generation. Models used : deepseek-r1 gemini Architecture The solution is a multi-agent architecture inspired by Self RAG, where input PDFs are converted to text, preprocessed, and filtered to extract relevant information. Different branches are utilized to handle specific types of queries, leveraging the strengths of the LLMs deepseek-r1 and gemini. 43. ▶ ragtastic 7 days 4/3 5.4 ragtastic Best experiment : ragtastic Signature : 43d4fd Summary : The architecture leverages the Mistral-large model for its implementation. Models used : mistral-large Architecture The solution used Mistral-large model to achieve its objectives. The architecture is designed to optimize performance and accuracy, ensuring robust results. Round 3 Round 3 is in the planning. We are going to make R&D process more focused and rewarding for the entire participants. Second round was won by a team that took time in advance to prepare a proper evaluation and experimentation framework. They simply iterated on various architectures and took a few best ones into the round 2. The goal of round 3 is to give such capabilities to everybody in advance. We are planning to prepare a proper evaluation and experimentation framework upfront. We also want to ground challenge deeper into the business, making insights more valuable and applicable to all the participants. Let's see how this turns out. ETA for the next round - May-June 2025.