By Rory Preddy, AI Advocate, Developer Relations at Microsoft and GitHub. Creator of LangChain4j for Beginners. | Edited by

I have been in IT for 27 years, and 23 of those as a programmer. I started as a Java developer. I moved to Microsoft seven years and eight months ago, and I have been a developer advocate for five and a half of those, first in cloud advocacy and now in AI.

For most of that time my job was to tell people to go and build. Lately I have started saying something else first. Take a step back. Not because the excitement is wrong, but because I keep meeting developers who ran out of tokens and cannot tell me where they went. I ask what happened and I hear the same answer. I used the top model, I sent it away, and it ran out. I ask whether they needed the best model for that job and they say they do not know.

That is not a model problem. That is a foundation problem, and it is fixable in an afternoon.

So let’s dig deeper into what tokens actually cost you, the two mechanisms that cut the bill the most, the patterns that keep the expensive model out of cheap work, and the setup you do before any of it.

This deep dive is adapted from Rory Preddy’s Deep Engineering Live session on token efficiency, edited from the session transcript for length and clarity. Slides from the talk are here, and the full recording is here.

Token Efficiency Foundry Instantmodels

2.37MB ∙ PDF file

Download

Download

Download

Four levers, and one of them gets ignored

You have four things to pull, and they move independently.

Model choice sets the unit price on every token in the exchange. Prompt structure sets how many input tokens you send and whether any of them qualify for a discount. Cache-key stability decides whether a repeated block earns the cached rate or gets billed again at full price. And output limits cap the completion, which is the one people skip.

Output is where I see the least attention and some of the highest prices. On the GPT-5.6 family the flagship Sol tier runs 5 dollars per million input tokens against 30 dollars per million output. Luna, the small one, is 1 dollar and 6 dollars. Look at the gap between those two numbers on the same row before you tune anything else. Watch your outputs, not just your inputs.

One more thing before the mechanisms. You do not have to guess at any of these rates. Azure publishes live pricing at prices.azure.com/api/retail/prices and you can query it. It looks confusing the first time you open it, because a single meter name folds the model family, the tier, the processing mode and the billing unit into one string, and there is a region attached to all of it. So let’s all breathe. Two things in there matter. The same model bills differently in different regions, and batch processing is its own meter rather than a discount on the standard rate.

It is not complicated. It is just not written down anywhere you were looking.

Caching, and why you only pay for the tail

Ask a model to tell you a joke. Now ask it again, exactly the same way. You are not going to be charged the same for the second one, or if you are, it will be a very small charge.

Now scale that up. Say I am an insurance company and I need to send millions of customers their latest statement. Every one of those requests carries the same instructions, the same reference material, the same everything, and then right at the bottom there is a name and a dollar amount that changes. That big identical block at the front is the stable prefix. The little bit at the end that changes is the tail.

You do not pay full price for the prefix. You pay for the tail.

The first call is a miss. The model stores the prefix and bills you the full input at standard rate. Every call after that, sent with the same prefix and the same cache key, is a hit, and that prefix gets billed as cheaper cached input. In my demo the prefix carried around 120 identical reference sections behind a run-scoped key, and warming the cache before the measured calls saved 99 percent on the repeated path.

Here is the part that catches people. The saving depends on those bytes being identical. Put a timestamp near the top of your prompt, or a request ID, or a freshly serialised list of tools, and you have moved the boundary and lost the discount on every single call. It will look completely harmless in review. So think ahead with the prompt structure. Decide what is stable and put it first, on purpose, before you need it.

Compaction, so the context stops growing

Caching handles repetition between calls. Compaction handles what piles up inside one conversation.

Every time you talk to an agent it remembers the conversation, and it sends that history again on the next turn. So the tokens you are billed per turn climb, and climb, and climb, even when the actual work per turn has not changed at all. Model a twelve-turn session and you are sending around 1,920 context tokens per turn by the end. Compact it along the way and it resets near 795 and never goes above that band. That projection is illustrative, but the summary reduction underneath it is measured, 62 percent, from 795 tokens down to 302.

What makes a compaction prompt work is being specific about what survives. Mine asks for at most six sentences, no bullets, no nested lists. It keeps only what the next turn needs, so the goal, the key facts, the files to update, the validation and deploy commands, the blockers and any privacy constraints. It throws out repetition, resolved dead ends, greetings, transient logs, and exact values that do not matter any more, and it turns quota findings into sanitised evidence instead of repeating every number. On a working-notes example that took a prompt from 409 tokens down to 233.

I also tried the other direction, squeezing the output instead of the context. I have a prompt that tells the model to answer like a caveman. Why use many token when few do trick. Drop the filler words and the articles, keep every technical fact, every command, every file path and every error string exact. Brain still big, mouth small.

It works, and the output count came down against a roughly 600-token baseline. But I will be honest with you about the trade. The caveman answer lost some of the understanding the fuller answer had. It is not as detailed. If your reader has to come back and ask a follow-up question, you have paid for that saving twice. A hard cap on output length is blunter but more predictable, because it truncates rather than condenses, so use it where the shape of the answer is already known.

Set up the project before you write anything

This is the least exciting part and it decides most of your cost.

Three files. A devcontainer.json, so the project runs on everyone’s machine and not just yours. An instructions file, so the agent has standing context and is not working blind. And an agent profile, which is the one people leave out.

The agent profile declares which tools the agent can reach for, and mine only has the tools it needs to function. That matters more than it sounds, because every tool you register ships its schema into the request on every single call. Leave a pile of MCP tools switched on that this task never touches and you are paying to send that JSON back and forth all day. Drop the ones you are not using.

Then three habits alongside it. Scope your sessions tightly. Keep your prompts precise and short. And if a direct completion will do the job, use a direct completion instead of turning an agent loose on it.

I build these three files with the cheapest model I have. Luna, reasoning switched off. It costs me almost nothing to create the baseline, and I would rather learn to do it properly at that price. On a cost-tips run I will say it out loud as I go. I am not using reasoning. I do not need reasoning right here. That is the right tool for the job.

What an agent actually is

An agent has agency. Agency is self determination, which means the agent understands there is a task it has to achieve. You can only give it that task if you understand it yourself.

What an agent is not is a large language model that you tell to go in and do whatever it wants. That is madness.

Here is the version I trust. I have a supervisor agent that transfers 100 dollars from one person to another and converts it to euros. Underneath it there is a withdraw agent, a credit agent and an exchange agent. Each one has a small scope. It runs on GPT-4o, so it costs me cents per month. It runs on Azure with managed identity, so it is locked down. I created the project with the dev container and the agent profile before any of it ran.

That is what I consider an agent. There is a plan, you built it, you locked it down, and it performs one well-contextualised task properly.

Eight patterns that keep the big model out of cheap work

Once the foundation is there, the patterns are about one idea. Do not let the largest model see work it does not need to see. My tiers are gpt-5.6-luna small, gpt-5.6-terra medium and gpt-5.6-sol large, and the numbers below are projections from my own demo harness, not benchmarks.

A router puts a cheap classifier in front and sends the request down one specialist path. 60 to 80 percent. Triage is similar but the gate is deterministic and costs zero tokens, and it only escalates when the complexity earns it. 50 to 70 percent. Context compression has Luna condense a long transcript so Sol never sees the whole thing. 30 to 60 percent. Retrieval pulls the chunks that matter instead of the whole corpus. 40 to 60 percent.

Tool use moves exact computation out of probabilistic generation and into code, and it goes hand in hand with keeping that tool list short. 30 to 50 percent. Step-back planning resolves the frame before the expensive execution starts, and it only pays when it prevents retries and rework. 20 to 40 percent. Caching covers the repeated path, 50 to 90 percent on a hit. And batching I will not oversell, because a parallel mapper lowers your wall-clock time and saves you no tokens at all.

Retrieval deserves more than one line, because everyone reaches for it and plenty of people misconfigure it. The query becomes an embedding, the embedding drives a vector search, the search returns chunks, and the model answers from those chunks alone. So your chunking and your embedding model decide your answer quality and your token count at the same time. The failure I see is sending everything back every time, because that is easier than tuning retrieval, and then the customer asks why it is slow and you ask why it is expensive. My retrieval demo projected 67 percent against sending the whole document. All you need is the dollar amount, not the entire insurance quote.

And where I got it wrong

I burnt 51 million tokens recently, and I used the wrong model for a specific purpose.

I was resolving a merge conflict, and I gave it to Luna. Luna could not solve the problem. That on its own would have been a cheap mistake. What made it expensive is that I had not set a loop to say, after a while, stop. So it kept going.

Two lessons, and they pull against each other slightly. Match the model to the bottleneck rather than always reaching down, because a cheap model that cannot finish costs more than an expensive model that can. And put an exit condition on every iterative agent, every time.

I have a refinement loop that writes a story and then reviews its own work. It caps at five iterations and accepts a score around 80 percent. It is not going to be perfect. Sometimes you really need to say, I do not need it perfect. Caching and compaction are both good, and you still want a hard stop when it hits the score you asked for.

Build an agent that watches the bill

You can point agents at your own costs, which is my favourite version of this.

I have a token cost runner. Its job is to execute real flows against a deployed Foundry model, read the actual token usage, and compare it against live pricing. So the number comes out of a run rather than out of arithmetic on a rate card. I ask it to find me the lowest-cost path for a prompt with a cached prefix and a changing tail, and it tells me.

Build agents to help you lower your cost for agents. Think of it as a pyramid. What you want to achieve sits at the top, and underneath it there is a foundation holding your dev container, your instructions, your agent profiles, your compaction and your compression. Everything above the foundation inherits whatever the foundation does.

Two commands, if you want the short version

If patterns are more than you want right now, there are two commands.

The first reads your session history and tells you where the money went. Mine was not flattering. Across sixty days my input tokens outweighed my output roughly 100 to 1. Opus 4.8 took 45.8 million input tokens at an average of 87,000 per event. One single session accounted for 14.4 million on its own. And one session climbed from 49,000 to 174,000 input tokens over 66 turns with no manual compaction at all, so auto-compaction only fired at turn 67, right at the end. Every turn past about turn 30 was resending well over 120,000 tokens on a premium model. I was the one not compacting long sessions.

The second sets a budget per session, warns you at a threshold and stops you at the limit. Mine stops after 34.7 of 30 credits and asks whether I want to add credits, raise the limit or remove it.

I also keep a canvas built from that same session data as a daily token pulse, and I refresh it across seven and thirty day windows. None of this optimises anything by itself. It is just awareness. But a number nobody looks at twice will never change how anyone works, and most of the waste I see is not a decision anyone made. It is a thing nobody checked.

Pick the harness you already pay for

People ask me which harness balances capability against cost. My honest answer is that what you are trying to achieve settles that before cost does.

Then I will tell you the best harness to save costs is GitHub Copilot, and you should hear that knowing I work on the Copilot side. My reasoning stands on its own though. If you have a Copilot licence you already have the spend, and Copilot is an agent, not just an editor feature. There is a Copilot SDK and there is the Microsoft Agent Framework, and you route from there. Send the task to Copilot, or to Claude Code, or to a local model, based on the work. It is the same routing decision as picking a model, one level up.

Take a step back

There is a whole industry forming around this. I watched it happen with cloud, where engineers who wrote good scripts became cloud engineers, and then cost optimisation became its own discipline. I used to sit with CTOs of banks who told me their biggest worry was how much they were going to spend. Then it changed, and the question became how much they were going to save. The same thing is happening now with tokens, and the Linux Foundation’s Tokenomics Foundation is going to matter here, because standards and dashboards and protection mechanisms are what turn this from a habit into a practice.

So do not go berserk. Create the foundation. Try Luna, switch reasoning off, and see how far it gets you. If it does not match your needs, add a router and a design pattern. Start small and build up. Rather than spend more, spend less.

And one last thing, because I have another whole talk about this. Do not end each day exhausted, not from the work itself, but from managing of the work. Let the adrenaline flow, do the vibe coding, do it in the correct way. Know what you are doing. Be kind to yourself.

Now go and look at your own session history.

Session notes:

Rory’s demos run on LangChain4j’s agentic modules, which matters for teams on the JVM because most token-efficiency tooling ships Python first. His pattern code is at github.com/roryp/token-design-patterns, the agentic pattern showcase behind the banking demo at aka.ms/agentpatterns, the live cost demo at aka.ms/costs, and his LangChain4j course at aka.ms/LangChain4j-for-Beginners.

This deep dive is adapted from Rory’s Deep Engineering Live session on The Future of Software Development. Here are the slides from the talk, and the full recording.

Token Efficiency Foundry Instantmodels

2.37MB ∙ PDF file

Download

Download

Connect with Rory Preddy on LinkedIn: https://za.linkedin.com/in/rorypreddy

By Julien Dubois Principal Manager, Developer Relations at Microsoft and GitHub. Creator of JHipster.

I always wanted a real developer UI for Spring Boot. Every Spring app is a black box in development. Actuator gives you raw JSON, not a console, so what I wanted was health, metrics, security and tracing in one embedded UI.

I’d shipped a slice of this in JHipster years ago, but only for generated apps. A real console for any Spring Boot app sat on my wishlist for years. The catch is that it’s massive. Around 40 panels, each one a backend plus a frontend plus its own tests, deeply integrated across a dozen JVM subsystems. By hand, one experienced developer would need somewhere between six and a half and eight and a half months. Too big to justify, until I stopped writing the code myself.

Then I did it in 11 days. Not by typing faster. I didn’t even open my IDE. I managed a fleet of AI agents while I architected, reviewed and steered. The agents did the scaffolding, the panels and the tests. I did the judgement.

What I built, BootUI

BootUI is a production-grade Spring Boot 4 starter that adds an embedded, local-only developer console to your app. You add one starter to your pom.xml, run locally, and open the console.

It is a five-module Maven build on Spring Boot 4 and Java 17, integrated across Actuator, Spring Security, Flyway and Liquibase, Hibernate, Micrometer with OTLP, GraalVM, OSV scanning and ArchUnit, published to Maven Central with full CI. The frontend is an embedded Vue 3 single-page app, packaged inside the starter. Each panel is endpoints plus a view plus tests.

There are around 40 feature panels. Health and metrics, a security advisor, a vulnerabilities view, tracing, and more. Those panels look a lot alike, which made them the biggest source of structural repetition in the codebase. That detail matters, and I will come back to why it made the whole thing work.

Eleven days, and how I know the numbers are real

The figures here are derived from git history, PR metadata and code metrics, not from time-tracking logs. Through the tagged 1.0.0 release there were around 264 commits on main and about 223 squash-merged pull requests, landing at roughly 20 a day across 11 days. That came to about 83k total tracked source lines, close to 50k of them Java across around 461 files and 81 test classes, plus 52 Vue components, 40 panels, and 35 end-to-end specs. In all, about 116 test suites.

A cadence of 20 PRs a day with an AI agent co-authoring commits is impossible to reach by hand. The agent did the typing. I dispatched many asynchronous tasks in parallel. The evidence lines up with that. The commit clock runs from around five in the morning to midnight most days, which is consistent with parallel async tasks rather than continuous typing. “Copilot” shows up as a named commit and PR author on about 44 commits, and the repo ships a copilot-instructions.md with per-panel conventions, so the workflow was explicitly agent-oriented. Even on release day I was landing a docs site, a scanner dashboard, token charts and a Hikari fix, which is itself several days of solo work.

Where my own time went was writing prompts, reviewing and merging those 223 PRs, and resolving CI failures around Spring Boot 4, Flyway 11 and OTLP. Review and orchestrate, not write every line.

The honest by-hand comparison is the part people argue about, so I ran it carefully. One experienced Spring Boot and Vue developer, with no AI codegen, would need 6.5 to 8.5 months for the same polished 1.0.0, roughly 1,100 to 1,450 hours of hands-on effort, and the 40 panels are the dominant cost. A COCOMO organic estimate on 50k lines yields more than 100 person-months, which is too high because much of the code is repetitive scaffolding, so a domain-expert solo figure of about 7.5 months is the defensible middle ground. My actual human effort on BootUI was 80 to 110 hours, about two intense solo weeks. That is a calendar speed-up of roughly 17 to 23 times, and a human-hours speed-up of roughly 12 to 17 times.

I was the manager, not the developer

I didn’t open my IDE. I wasn’t the developer. I was the manager. Many agents ran in parallel on the panels, the integrations, the frontend, and the CI and docs, and my job was to brief, review and merge.

Your throughput is not your keyboard. It is your briefs. What blocks a single developer is typing, and what blocks a single agent is you waiting for it to finish. Running many at once removes both, because they come back one after another and there is always something to review. You don’t type faster this way. You ship what used to take months.

A day in the loop

The rhythm that produced 20 merged PRs a day is a simple daily loop, and the day is the engine. I am hands-on all day. I spec, launch, review, merge and re-task, live, and most of the day’s PRs land right there.

The evening is a hand-off. Before I step away I queue a few deep, long-running plans. The night is a bonus, not the engine. A handful of deep autonomous runs finish by morning, which is the minority of the work. Repeat that for about 11 days and you reach a tagged 1.0.0. Six ingredients make the loop work.

Ingredient one, write the specifications

The first thing I write is never code. It is the house rules. An AGENTS.md, which GitHub Copilot reads as copilot-instructions.md, and every agent reads it first. It sets the conventions once, the stack, the build, the tests and the style, with per-panel conventions so 40 panels come out consistent. Mine runs around 700 lines. That is about the right size, because a file that is too big pollutes the context, and it also costs tokens and money on every run.

The high-leverage move is writing the known failures into that file once. Most models were trained on Spring Boot 3, so they struggle with the reworked Jackson API in Spring Boot 4, and 10 agents trained on the same data hit the same wall at the same time. Rather than fix that by hand in each one, I write the fix down once. The same goes for operational traps, like giving every agent its own local Maven repository so parallel installs do not corrupt a shared one.

On top of the house rules I write one spec per task. What to build, where, what “done” looks like, and the acceptance test the agent must make pass. Small, self-contained, no hidden dependencies. The spec is the product now. The better the brief, the less you babysit.

Ingredient two, build the test harness

No agent runs without a test harness. One command compiles and runs the unit and end-to-end tests, and comes back green or red, so an agent can verify its own work before it opens a PR. No tests means you cannot trust the output.

CI is the trust layer. BootUI has about 116 test suites, 81 in Java and 35 in Playwright, and CodeQL plus the end-to-end tests gate every PR to main. The loop does the work. An agent writes code, pushes it, and GitHub Actions sends back the results. If they are red the agent reads the failure, fixes it, commits and pushes again, and the checks run once more. Dependabot keeps the dependencies current alongside all of this. A compiled, typed language like Java helps too, because a hallucinated call often fails to compile before a test ever runs, and the tests catch what the compiler misses. You cannot read every line of 223 PRs. A green build you trust is what makes the volume reviewable.

Ingredient three, split the work and run agents in parallel

Parallelism is the whole point, so the work has to be parallel-ready. The 40 near-identical panels are perfect to fan out, because AI is strong at cloning something and adapting it to something similar. I give one task to one agent, small scope and a clear goal, each on its own branch or worktree so they do not collide. Splitting the work cleanly is an architecture problem before it is a prompting one, and getting it wrong is where the merge conflicts come from.

A fleet is not one chat window. I drove BootUI mostly through the GitHub Copilot coding agent app, because it lets many agents run live on one machine, and I comfortably keep 10 going at once. More than 10 gets complicated for a human to hold in their head, and the laptop starts to slow down. The mobile app runs agents in Docker containers, so I can keep them moving when I am away from my desk. The CLI and the IDE plugin suit smaller fan-out, and all three share the same back end, the Copilot SDK, which is open source, so you can build your own interface on top of it if you want. The rule stays simple. Don’t babysit one agent. Run 10.

Ingredient four, let a few deep plans run overnight

The day is the engine and the night is a bonus shift. Before I log off I hand a few deep, long-running plans to autonomous agents, the big jobs I don’t want to sit and watch. On BootUI those were things like wiring and testing the security filter chains across 37 rules, pushing coverage further across the suites, and reshaping the Actuator data layer in a larger refactor. Each is a long run that lands a few PRs by morning.

For the genuinely tricky work I have the agent criticize its own output with two or three other models. One writes the code, the others analyze and vote on the fix. That is also how I catch hallucinations, because the model that invented something usually stands alone against the others, and it shows up plainly in the logs. A good overnight prompt means everything is done when I come back. A weak one means I have nothing, which is why prompt-writing is worth practicing. The night is a bonus on top of the day, not a replacement for the driving.

Ingredient five, merge the results over coffee

The morning starts over coffee. I triage the few overnight PRs, merge the green ones fast, drop or re-task whatever didn’t land, cherry-pick the good parts of the rest, and then start driving the day’s fleet. When I review a PR I look at three things that stay in sync because they are generated together, the code, its tests, and its documentation. Green tests tell me the code and the tests agree, and I squash and merge.

Review is the real bottleneck. It is not the typing anymore. It is the merging. I don’t merge blind and I don’t merge like crazy, because the day you wave through a large diff without reading it is the day something wrong lands in main. So make review a fast, trusted ritual you run all day, not a line-by-line slog at the end.

Ingredient six, pick the right model for the task

Most of the work went to a workhorse, GPT-5.5 on extra-high reasoning, with around 90 percent of its tokens served from cache. For the tricky parts I brought in Claude Opus 4.8 and Gemini 3.1 Pro, three strong models cross-checking each other to find the best fix. For simple, mechanical tasks a smaller model or Auto mode is fast and cheap, and Auto tends to pick better than I would while carrying its own token rebate.

The economics are driven by the cache more than by the model price. The whole build ran on about 2.7 billion tokens for roughly 2,000 dollars, with around 95 percent served from cache, so most of the run costs a tenth of the headline price. That changes the obvious advice. Swapping to a cheaper model for one small task often bursts the cache and costs more than staying on the model whose context is already warm. If I have a small task, I spawn a separate small agent for it rather than switching the big agent’s model. And I avoid editing AGENTS.md mid-run, because it lives at the top of the cache and one change invalidates it, so I update it at the end of a run instead. Knowing how the cache works matters more than picking the cheapest model.

Why the multiplier was so large

I want to be honest about why BootUI hit a multiplier this large, because not every project will. Three things made this codebase unusually well-suited to agents. First, massive repetition, since around 40 structurally similar panels are cheap for an agent to clone and are the most expensive part by hand. Second, a broad-but-shallow shape, many Spring subsystems each shallow on its own, where the human cost is mostly looking things up, which is exactly what the model already absorbed in training. Third, strong guardrails, because multi-module CI, CodeQL, end-to-end tests and explicit instructions let me accept high throughput safely.

The net effect was a 6.5 to 8.5 month solo effort compressed into 11 days and about two weeks of human attention. The biggest leverage is on large-surface, pattern-heavy, well-tested code, not on hard algorithms. A project that is mostly novel logic with thin tests will not see these numbers, and I would not claim otherwise.

Where this goes wrong

The failure modes are consistent, and most of them trace back to the operator. Scope creep is the first. Tell an agent to build the whole thing and it wanders, so keep one tight goal per task. Missing tests are the second. Without a harness you cannot trust the output and you cannot merge at volume, so the harness comes first and a green build gates every merge.

Giant PRs are the third. A 2,000-line PR is impossible to review well, and review fatigue tempts you to wave it through, so keep the chunks small and reviewable and pace yourself. The wrong model is the fourth. A weak model fails the hard tasks, a strong one is slow and costly on the easy ones, so match the model to the job. When people tell me AI-generated code is not good, the cause is usually one of these. Thin specs, weak tests, oversized PRs, or the wrong model. If the agent generates bad code, that is usually your fault, not the model’s. Most failed runs are a briefing problem.

One page to screenshot

If you take one thing away, take this.

A new Spring Boot console fell out of it

The recipe was the point, but it left behind a real, open-source product. BootUI gives any Spring Boot app an embedded console with live health and metrics, a security advisor that walks your filter chains, an OSV scan of your dependencies, a Flyway and Liquibase data view with a Hibernate advisor, tracing through Micrometer and OTLP, and an architecture view backed by ArchUnit and GraalVM reachability. Add one starter to your pom.xml, run locally, and open the console. The code and the full docs are open.

Write the specs, give them tests, run them wide, and ship what used to take months. Now go build.

This deep dive is adapted from Julien’s Deep Engineering workshop, From Coder to Manager of Agents. Here the slides from the talk.

What makes these patterns particularly valuable in Rust isn’t their novelty. Most have decades of history in other languages. It is how Rust’s design makes them practical for systems programming. The combination of strong static typing, zero-cost abstractions, and memory safety without garbage collection creates opportunities to apply these patterns in contexts where they were previously impractical.

We’ll apply these patterns to Samsa, a publish/subscribe microservice in the style of a miniature Kafka that the book builds across its later chapters, patterns that have proven their worth across multiple programming language communities. As we explore each pattern, we’ll examine both its origins and how Rust’s unique features enhance or constrain its application.

This deep-dive is excerpted from Design Patterns and Best Practices in Rust by Evan Williams, shared with permission from Packt, with all rights remaining with the publisher and no reproduction or redistribution without written consent. You can get the full book here.

In this chapter the book covers the following main topics:

• The NewType pattern

• Parse, don’t validate

• The TypeState pattern

• Sealed traits

Each pattern builds on the previous ones: the NewType pattern creates distinct types; parse, don’t validate ensures those types hold only valid data; the TypeState pattern tracks valid state transitions; and sealed traits control which types can participate in our APIs.

The NewType pattern

The NewType pattern has a long history in statically-typed functional programming, particularly in Haskell, where the newtype keyword has been a language feature since the 1990s. The pattern also appears in ML, OCaml, and other languages with strong type systems. The core idea of wrapping existing types in distinct types for semantic clarity and type safety predates Rust. It is a response to the issue that values that are conceptually different, such as temperature and weight, have the same type representation in the code, for example, a float. From the compiler’s perspective, they are the same, and that leads to bugs when these values are inadvertently mixed up.

What Rust brings to this well-established pattern is zero-cost abstraction: the wrapped type compiles to exactly the same representation as the underlying type, with no runtime overhead. In languages with garbage collection or boxing, creating wrapper types often incurs performance costs. Rust’s design ensures that type safety is free at runtime while remaining enforceable at compile time.

Identifying the problem

In our Samsa system, we currently use primitive types such as u64 for topic IDs, consumer IDs, and message offsets. This approach, which we will call primitive obsession, makes it easy to accidentally pass the wrong type of ID to a function. If you’ve worked with similar systems, you’ve probably experienced the confusion this creates. The NewType pattern solves this problem with compile-time type checking.

Let’s examine our current API and identify where primitive obsession creates issues:

// Current API - prone to mix-ups
impl Broker {
     pub fn subscribe(&self, topic_id: u64, consumer_id: u64) -> Result<()> {
         // Easy to accidentally swap these parameters!
         if !self.topics.contains_key(&topic_id) {
             return Err("Topic not found".to_string());
         }
         Ok(())
     }
 }

impl Consumer {
     pub fn new(broker: Arc<Broker>, topic_id: u64, offset: u64) -> Self {
         // Again, easy to mix up topic_id and offset
         Self { broker, topic_id, offset }
     }
 }

We’re using raw u64 values for different semantic concepts, making it easy to pass the wrong type of ID and causing runtime bugs that are difficult to debug. What is very unfortunate is that the compiler cannot help us identify logical errors when we pass a topic ID instead of a consumer ID.

Implementing type-safe wrappers

Let’s create wrapper types that eliminate these mix-ups. Our approach has three parts:

1. Define semantic wrapper types for our different ID concepts (shown in the first code block)

2. Implement basic methods for usability, such as constructors and accessors (shown in the second code block)

3. Add Display implementations for user-friendly output (shown in the third code block)

Let’s define semantic wrapper types: TopicId for topic identification, ConsumerId for tracking individual consumers, and MessageId for tracking individual messages. We’ll use plain u64 for message offsets for simplicity (in the current design, message offsets don’t benefit much from the additional type safety):

use std::fmt;

/// Topic identifier for tracking individual topics
#[derive(Debug, Clone, PartialEq, Eq, Hash)]
pub struct TopicId(String);

/// Consumer identifier for tracking individual consumers
#[derive(Debug, Clone, PartialEq, Eq, Hash)]
pub struct ConsumerId(String);

/// Message identifier for tracking individual messages
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub struct MessageId(u64);

These NewTypes provide type safety through distinct wrapper types. TopicId and ConsumerId wrap String to allow meaningful identifiers such as "user.events" or "consumer-1", while MessageId wraps u64 for numeric message tracking. Each type has meaningful derives, such as Hash for use in collections. Note that MessageId includes Copy since u64 is copyable, while the String-based types do not.

Now, let’s add methods for creating and accessing these types:

impl TopicId {
    pub fn new(topic: impl AsRef<str>) -> Result<Self> {
        let topic = topic.as_ref();

        if topic.is_empty() {
            return Err(SamsaError::topic("Topic name cannot be empty"));
        }

        if topic.len() > 128 {
            return Err(SamsaError::topic(
                format!("Topic name too long: {} chars (max 128)",
                         topic.len())
            ));
        }

        Ok(TopicId(topic.to_string()))
    }

    pub fn as_str(&self) -> &str {
        &self.0
    }
}

impl ConsumerId {
    pub fn new(id: impl AsRef<str>) -> Result<Self> {
        let id = id.as_ref();

        if id.is_empty() {
            return Err(SamsaError::consumer("Consumer ID cannot be
                                             empty"));
        }

        Ok(ConsumerId(id.to_string()))
    }

    pub fn as_str(&self) -> &str {
        &self.0
    }
}

impl MessageId {
    pub fn new(id: u64) -> Self {
        MessageId(id)
    }

    pub fn value(&self) -> u64 {
        self.0
    }
}

Each type provides a new constructor and an accessor method. TopicId and ConsumerId validate their inputs at construction time, returning Result to signal potential validation failures. This is the NewType pattern combined with early validation: once you have TopicId or ConsumerId, you know it contains valid data. MessageId uses simple u64 wrapping since numeric IDs don’t require validation.

Finally, let’s add Display implementations for user-friendly output:

impl fmt::Display for TopicId {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        write!(f, "{}", self.0)
    }
}

impl fmt::Display for ConsumerId {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        write!(f, "{}", self.0)
    }
}

impl fmt::Display for MessageId {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        write!(f, "{}", self.0)
    }
}

The Display implementations output the inner value directly. For example, a TopicId containing "user.events" prints as user.events. If we were writing production code, we could also implement a custom Debug trait and print the IDs in a way that is more helpful for debugging, such as "TopicID(user.events)".

We’ve created distinct types for different domain concepts. Each type wraps its inner value and prevents accidental mixing. It’s now impossible to accidentally pass a TopicId where a ConsumerId is expected. The compiler catches this at compile time, eliminating entire classes of bugs. Each wrapper type is distinct at compile time.

For MessageId wrapping u64, the type compiles to the same representation as u64 at runtime, creating a true zero-cost abstraction. TopicId and ConsumerId use String for semantic identifiers but still provide compile-time type safety.

Enhanced Samsa API

Let’s see how NewTypes prevent parameter mix-ups. Consider a subscription function that takes both a consumer and a topic:

fn subscribe(consumer: ConsumerId, topic: TopicId) -> Result<(), SamsaError> {
      println!("Subscribing {} to {}", consumer, topic);
      Ok(())
  }

  // Usage - this compiles:
  let topic = TopicId::new("orders.created")?;
  let consumer = ConsumerId::new("analytics-service")?;
  subscribe(consumer, topic)?;

  // But this won't compile - arguments swapped:
  // subscribe(topic, consumer);
  //           ^^^^^ expected `ConsumerId`, found `TopicId`

Notice how subscribe takes TopicId and ConsumerId as separate types. The compiler would reject any attempt to swap them. Without NewTypes, both topic and consumer would be String, and swapping them would compile silently – a bug you’d only discover at runtime (maybe while in production). With NewTypes, the compiler catches the mistake immediately. This is especially valuable in APIs with multiple string-like parameters where mix-ups are easy to make and hard to debug.

Now that we have our type-safe wrappers, let’s apply them to Samsa’s API to see how they prevent the parameter mix-ups we identified earlier. The following code snippet enhances the consumer API with type safety:

pub struct Consumer {
     broker: Arc<Broker>,
     id: ConsumerId,
     topic: TopicId,
     offset: u64,
 }

impl Consumer {
     pub fn new(broker: Arc<Broker>, id: ConsumerId, topic: TopicId,
                offset: u64) -> Self {
         Self { broker, id, topic, offset }
     }

    pub fn poll(&mut self) -> Result<Option<Message>, String> {
         let events = self.broker.fetch(self.topic, self.offset, 1)?;
    
        if let Some(message) = events.into_iter().next() {
             self.offset += 1;
             Ok(Some(message))
         } else {
             Ok(None)
         }
     }
 }

The Consumer struct stores typed identifiers for its broker connection, ID, topic, and current offset. The poll() method demonstrates how advancement works with the typed consumer. Each field uses the appropriate NewType, making the struct’s purpose clear from its type signature alone.

Our entire API now uses type-safe identifiers. Function signatures are self-documenting, and the compiler prevents mixing up different types of IDs. This eliminates bugs where IDs are confused, provides better error messages, and makes the code more maintainable. With this design, Samsa’s consumer API becomes self-documenting and resistant to ID mix-ups.

The NewType pattern provides compile-time guarantees about type correctness while maintaining the runtime performance of primitive types. This combination of safety and efficiency is what makes the pattern particularly valuable in Rust.

However, while NewTypes give us distinct types, they don’t enforce validity. A MessageId could wrap any u64, which is OK if every u64 is a valid message ID. But if some are not, including those invalid values in our data type can turn into an issue. In the next section, we’ll see how to combine type distinctions with validity guarantees using the “parse, don’t validate” principle.

Parse, don’t validate

Alexis King popularized the parse, don’t validate principle with her influential 2019 blog post of the same name (https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/). King’s articulation of the principle, which states that parsing generates types that can represent only valid data, while validation merely checks data after the fact, has its roots in functional programming and type theory and helped popularize and name the pattern.

Rust’s type system makes this principle particularly natural to apply. The combination of strong typing, pattern matching, and the Result type creates an environment where parse, don’t validate feels like the path of least resistance rather than additional work.

The principle suggests that instead of scattering validation logic throughout a code base, we should parse data once at system boundaries into types that can only represent valid states. This transforms validation from a repeated burden into a guarantee encoded in types.

The problem with scattered validation

Let’s examine how our current Samsa configuration system handles validation. The current approach is validation scattered everywhere:

pub struct BrokerConfig {
     pub max_connections: i32,    // Could be negative!
     pub buffer_size: usize,      // Could be zero!
     pub port: u16,              // Could be in reserved range!
 }

impl Broker {
     pub fn new(config: BrokerConfig) -> Result<Self, String> {
         // Validation #1 - at construction
         if config.max_connections <= 0 {
             return Err("max_connections must be positive".to_string());
         }
         if config.buffer_size == 0 {
             return Err("buffer_size must be non-zero".to_string());
         }
         Ok(Self { config })
     }

    pub fn accept_connection(&self) -> Result<(), String> {
         // Validation #2 - in business logic (defensive programming)
         if self.config.max_connections <= 0 {
             return Err("Invalid max_connections".to_string());
         }
         Ok(())
     }
 }

Notice how the same validation check, if max_connections <= 0, appears in two different places: once during configuration creation and once defensively in the business logic. This duplication is the core problem. Each location must remember to perform the check, use the same condition, and handle errors consistently. If the validation rule changes (say, max_connections must be at least 10), every location must be updated. This approach is error-prone, inefficient, and creates inconsistent validation logic across different parts of the system.

Creating types that enforce validity

The solution is to create wrapper types that can only hold valid values. Instead of validating a raw u32 everywhere it’s used, we create a PositiveU32 type that validates once at construction time. Any code that receives a PositiveU32 type knows, by the type alone, that the value has already been validated.

Let’s redesign our configuration using this approach. We’ll start by creating a wrapper type for positive integers that cannot be zero or negative:

use std::num::NonZeroUsize;

#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct PositiveU32(u32);

impl PositiveU32 {
     pub fn new(value: u32) -> Result<Self, ConfigError> {
         if value > 0 {
             Ok(Self(value))
         } else {
             Err(ConfigError::MustBePositive("value".to_string()))
         }
     }

    pub fn get(self) -> u32 {
         self.0
     }
 }

Next, we create a similar wrapper for network ports that ensures the port number is not in the reserved range (ports 0–1023):

#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct NetworkPort(u16);

impl NetworkPort {
     pub fn new(port: u16) -> Result<Self, ConfigError> {
         if port > 1023 {  // Ports 0-1023 are reserved
             Ok(Self(port))
         } else {
             Err(ConfigError::PortInReservedRange(port))
         }
     }

    pub fn get(self) -> u16 {
         self.0
     }
 }

Let’s also define an error type for configuration errors:

#[derive(Debug, Clone, PartialEq, Eq)]
 pub enum ConfigError {
     MustBePositive(String),
     PortInReservedRange(u16),
 }

impl std::fmt::Display for ConfigError {
     fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
         match self {
             ConfigError::MustBePositive(field) => write!(f, "{} must be positive", field),
             ConfigError::PortInReservedRange(port) => write!(f, "Port {} is in reserved range", port),
         }
     }
 }

impl std::error::Error for ConfigError {}

Samsa’s configuration system now uses types that enforce validity at construction time. Once created, these types guarantee their validity throughout the system. This moves all validation to the boundary of our system and eliminates defensive programming in business logic.

By validating once during construction, we ensure that these types can only ever hold valid values. Rust’s ownership system guarantees that once created, these values cannot be modified to become invalid.

Builder pattern with validation

The Builder pattern pairs naturally with validated types. While the wrapper types we defined validate individual values, the Builder pattern coordinates multiple validated values into a complete, consistent configuration. We’ll create a BrokerConfig that can only be constructed through a builder, ensuring that all validation happens during construction.

First, let’s define the configuration struct with validated field types:

#[derive(Debug, Clone)]
pub struct BrokerConfig {
    max_connections: PositiveU32,
    buffer_size: NonZeroUsize,
    port: NetworkPort,
}

impl BrokerConfig {
    pub fn builder() -> BrokerConfigBuilder {
        BrokerConfigBuilder::new()
    }

    // Getters that never need validation
    pub fn max_connections(&self) -> u32 {
        self.max_connections.get()
    }

    pub fn buffer_size(&self) -> usize {
        self.buffer_size.get()
    }

    pub fn port(&self) -> u16 {
        self.port.get()
    }
}

The BrokerConfig struct uses validated types for all fields. PositiveU32 and NetworkPort ensure that invalid values cannot be stored. The getters return primitive values, which is safe because we know the wrapped values are valid. Notice that all fields are private and accessed through getters. This prevents external code from modifying the configuration after validation.

Now, we need a way for users to construct a BrokerConfig. Since the struct has private fields with validated types, we can’t allow direct construction. Users would need access to PositiveU32::new() and NetworkPort::new(), and they’d have to handle validation errors themselves. Instead, we’ll provide a builder that accepts raw primitive values, validates them internally, and produces a fully validated BrokerConfig on success.

The BrokerConfigBuilder struct holds Option fields for each configuration value. As users set values through the builder’s methods, each value is validated and wrapped in its corresponding type. The final build() method ensures that all required fields are present before constructing the BrokerConfig. Let’s implement this builder:

pub struct BrokerConfigBuilder {
    max_connections: Option<PositiveU32>,
    buffer_size: Option<NonZeroUsize>,
    port: Option<NetworkPort>,
}

impl BrokerConfigBuilder {
    pub fn new() -> Self {
        Self {
            max_connections: None,
            buffer_size: None,
            port: None,
        }
    }

    pub fn max_connections(mut self, value: u32) -> Result<Self, ConfigError> {
        self.max_connections = Some(PositiveU32::new(value)?);
        Ok(self)
    }

    pub fn buffer_size(mut self, value: usize) -> Result<Self, ConfigError> {
        self.buffer_size = Some(
            NonZeroUsize::new(value)
                .ok_or_else(|| ConfigError::MustBePositive("buffer_size".to_string()))?
        );
        Ok(self)
    }

    pub fn port(mut self, value: u16) -> Result<Self, ConfigError> {
        self.port = Some(NetworkPort::new(value)?);
        Ok(self)
    }
}

Each setter method validates its input before storing it. The ? operator propagates validation errors immediately if validation fails. If validation succeeds, the method consumes self and returns it, enabling method chaining. The use of Option for each field lets us detect missing fields in the build() method.

The final step is building the configuration, which ensures that all required fields are present and results in a fully validated BrokerConfig:

impl BrokerConfigBuilder {
    pub fn build(self) -> Result<BrokerConfig, ConfigError> {
        Ok(BrokerConfig {
            max_connections: self.max_connections
                .ok_or_else(|| ConfigError::MustBePositive("max_connections not set".to_string()))?,
            buffer_size: self.buffer_size
                .ok_or_else(|| ConfigError::MustBePositive("buffer_size not set".to_string()))?,
            port: self.port
                .ok_or_else(|| ConfigError::PortInReservedRange(0))?,
        })
    }
}

The build() method consumes the builder and returns Result<BrokerConfig, ConfigError>. If any required field is missing, we return an error. This ensures that you can’t construct an incomplete configuration.

The usage now looks like this:

let config = BrokerConfig::builder()
    .max_connections(1000)?
    .buffer_size(4096)?
    .port(8080)?
    .build()?;

The fluent API makes configuration construction readable and catches errors immediately. If any setter returns an error, the ? operator short-circuits, and the invalid configuration is never created. The final build() call ensures that no required fields are missing.

The builder ensures that all configuration values are validated during construction. Once a BrokerConfig exists, all its values are guaranteed to be valid. This design makes it impossible to create invalid configurations, with validation happening once at the system boundary.

The parse, don’t validate principle, combined with Rust’s type system and ownership model, eliminates defensive programming throughout the system by moving all validation to well-defined boundaries.

So far, we’ve seen how to create distinct types (NewType) and ensure that those types only hold valid values (parse, don’t validate). But what about objects that change over time? In the next section, we’ll explore the TypeState pattern, which uses the type system to track and enforce valid state transitions.

The TypeState pattern

The TypeState pattern has academic origins in research on program verification. It was formalized by Robert E. Strom and Shaula Yemini in their 1986 paper TypeState: A Programming Language Concept for Enhancing Software Reliability (which you can read at https://www.computer.org/csdl/journal/ts/1986/01/06312929/13rRUwIF6aQ). The core idea is to use types to represent different states in which an object can be, making invalid state transitions impossible to express.

While the concept has existed in type theory for decades, Rust’s unique combination of features makes it particularly practical to implement. Rust’s zero-cost abstractions mean the type-level state tracking compiles away entirely, leaving no runtime overhead. The ownership system ensures that state transitions consume the old state, preventing accidental reuse. And generic types with PhantomData provide a clean mechanism for encoding state in the type system.

Several other languages support similar patterns, most notably session types in research languages and builder patterns in languages such as Java, but Rust’s combination of performance, safety, and ergonomics makes the TypeState pattern particularly accessible for systems programming.

The problem with runtime state management

Let’s examine traditional runtime state management, where state is tracked at runtime:

#[derive(Debug, Clone, Copy, PartialEq)]
pub enum ConsumerState {
     Disconnected,
     Connected,
     Subscribed,
     Paused,
 }

pub struct Consumer {
     broker: Arc<Broker>,
     topic: TopicId,
     consumer_id: ConsumerId,
     state: ConsumerState,
 }

impl Consumer {
     pub fn connect(&mut self) -> Result<(), String> {
         // Runtime check required
         if self.state != ConsumerState::Disconnected {
             return Err("Consumer must be disconnected to connect".to_string());
         }
         self.state = ConsumerState::Connected;
         Ok(())
     }

    pub fn poll(&mut self) -> Result<Option<Message>, String> {
         // Runtime validation in every method
         if self.state != ConsumerState::Subscribed {
             return Err("Consumer must be subscribed to poll
                         messages".to_string());
         }
         Ok(None)
     }
 }

Every method requires runtime validation to ensure a correct state, creating opportunities for errors and adding runtime overhead. The compiler can’t help us by catching state transition errors, which leads to runtime failures that could have been prevented at compile time.

Implementing TypeState for the consumer lifecycle

Let’s redesign the consumer using TypeState. We’ll create separate types for each state:

use std::marker::PhantomData;
use crate::types::*;

// State marker types organized in a module
pub mod states {
    #[derive(Debug)]
    pub struct Disconnected;

    #[derive(Debug)]
    pub struct Connected;

    #[derive(Debug)]
    pub struct Subscribed;

    #[derive(Debug)]
    pub struct Paused;
}

// Consumer parameterized by state
pub struct Consumer<State> {
    broker: Arc<Broker>,
    topic: Option<TopicId>,
    consumer_id: ConsumerId,
    connection_info: Option<ConnectionInfo>,
    state: PhantomData<State>,
}

/// Type aliases for consumer states
pub type DisconnectedConsumer = Consumer<states::Disconnected>;
pub type ConnectedConsumer = Consumer<states::Connected>;
pub type SubscribedConsumer = Consumer<states::Subscribed>;
pub type PausedConsumer = Consumer<states::Paused>;

#[derive(Debug, Clone)]
pub struct ConnectionInfo {
    pub broker_address: String,
    pub consumer_group: Option<String>,
}

The state marker types (Disconnected, Connected, Subscribed, and Paused) are zero-sized. They exist only at compile time to track state in the type system. The Consumer<State> struct is generic over the state marker, so Consumer<states::Disconnected> and Consumer<states::Connected> are different types even though they contain the same data fields. The type aliases provide convenient shorthand.

The PhantomData<State> field tells Rust about the state parameter without storing anything at runtime. This is the key to TypeState: state tracked in types with zero runtime cost.

Let’s implement the disconnected state:

impl Consumer<states::Disconnected> {
    pub fn new(consumer_id: ConsumerId, broker: Arc<Broker>) -> Self {
        Self {
            broker,
            topic: None,
            consumer_id,
            connection_info: None,
            state: PhantomData,
        }
    }

    pub fn connect(
        self,
        connection_info: ConnectionInfo
    ) -> Result<Consumer<states::Connected>, SamsaError> {
        if connection_info.broker_address.is_empty() {
            return Err(SamsaError::connection("Empty broker address"));
        }

        Ok(Consumer {
            broker: self.broker,
            topic: self.topic,
            consumer_id: self.consumer_id,
            connection_info: Some(connection_info),
            state: PhantomData,
        })
    }
}

The new constructor creates a Consumer<states::Disconnected>. The connect method consumes the disconnected consumer (taking ownership with self) and returns either a Consumer<states::Connected> or an error along with the original consumer. This ownership transfer prevents reuse. Once you call connect, the disconnected consumer is gone.

Now, let’s implement the connected state:

impl Consumer<states::Connected> {
    pub fn subscribe(
        mut self,
        topic: TopicId
    ) -> Result<Consumer<states::Subscribed>, SamsaError> {
        self.topic = Some(topic);

        Ok(Consumer {
            broker: self.broker,
            topic: self.topic,
            consumer_id: self.consumer_id,
            connection_info: self.connection_info,
            state: PhantomData,
        })
    }

    pub fn disconnect(self) -> Consumer<states::Disconnected> {
        Consumer {
            broker: self.broker,
            topic: None,
            consumer_id: self.consumer_id,
            connection_info: None,
            state: PhantomData,
        }
    }
}

Each state transition follows the same pattern: consume the current state, perform an operation, and return the new state or an error. The connected consumer can subscribe to a topic or disconnect. Each transition is a separate impl block, making it impossible to call methods from the wrong state.

Finally, the Subscribed state is where consumers can actually receive messages, while the Paused state can only resume:

impl Consumer<states::Subscribed> {
    pub fn receive(&self) -> Option<Event> {
        // Only subscribed consumers can receive - guaranteed by types!
        None
    }

    pub fn pause(self) -> Consumer<states::Paused> {
        Consumer {
            broker: self.broker,
            topic: self.topic,
            consumer_id: self.consumer_id,
            connection_info: self.connection_info,
            state: PhantomData,
        }
    }

    pub fn unsubscribe(mut self) -> Consumer<states::Connected> {
        self.topic = None;
        Consumer {
            broker: self.broker,
            topic: self.topic,
            consumer_id: self.consumer_id,
            connection_info: self.connection_info,
            state: PhantomData,
        }
    }
}

impl Consumer<states::Paused> {
    pub fn resume(self) -> Consumer<states::Subscribed> {
        Consumer {
            broker: self.broker,
            topic: self.topic,
            consumer_id: self.consumer_id,
            connection_info: self.connection_info,
            state: PhantomData,
        }
    }
}

Only Consumer<states::Subscribed> has a receive method. You literally cannot call receive() on consumers in other states. The method doesn’t exist for those types. The Paused state shows that TypeState can model bidirectional transitions: pause() moves to Paused, and resume() returns to Subscribed.

This TypeState implementation makes invalid operations impossible to express. The compiler enforces the correct state machine at compile time with no runtime overhead.

TypeState’s implementation in Rust uses PhantomData to carry type-level information that exists only at compile time, employs generic types to parameterize over state, and utilizes the ownership system to ensure that state transitions consume the previous state. The result is verification of state machines at compile time with zero runtime cost.

Usage examples

Here’s how TypeState improves the developer experience:

fn correct_usage() -> Result<(), SamsaError> {
    let broker = Arc::new(Broker::new());
    let consumer_id = ConsumerId::new("test-consumer")?;
    let topic = TopicId::new("events")?;

    // Create disconnected consumer
    let consumer = Consumer::new(consumer_id, broker);

    // Connect with connection info
    let connection_info = ConnectionInfo::new(
        "localhost:9092".to_string(), None
    );
    let connected = consumer.connect(connection_info)?;

    // Subscribe to topic
    let subscribed = connected.subscribe(topic)?;

    // Can pause and resume
    let paused = subscribed.pause();
    let subscribed = paused.resume();

   // Now we can receive - guaranteed to be in correct state
    if let Some(event) = subscribed.receive() {
        println!("Received: {:?}", event);
    }

    // Unsubscribe and disconnect
    let connected = subscribed.unsubscribe();
    let _disconnected = connected.disconnect();

    Ok(())
}

This code demonstrates the correct sequence: create a disconnected consumer, connect it, subscribe, then receive – pausing and unpausing, unsubscribing, and disconnecting work the same way. Each state transition is explicit in the code, and the types change at each step. The error handling uses a tuple to return the original consumer on failure, allowing retry logic.

Invalid usage won’t compile:

fn invalid_usage() {
    let broker = Arc::new(Broker::new());
    let consumer_id = ConsumerId::new("test").unwrap();

    let consumer = Consumer::new(consumer_id, broker);

    // These lines won't compile:
    // consumer.receive(); // Error: not available on Disconnected
    // consumer.subscribe(topic); // Error: not available on Disconnected
}

The compiler errors shown in the comments aren’t runtime errors; they’re compile-time errors. You literally cannot write code that calls receive() on a disconnected consumer because the method doesn’t exist for that type. This is the power of TypeState: invalid states are unrepresentable.

The TypeState pattern transforms potential runtime errors into compile-time guarantees, demonstrating how Rust’s type system and ownership model enable practical implementation of ideas from formal methods and programming language research.

TypeState controls how individual objects transition through states. But sometimes, we need to control which types can participate in a system at all. In the next section, we’ll see how sealed traits let us define closed sets of implementations, providing API stability and safety guarantees.

Sealed traits

Many object-oriented languages introduce the concept of sealed or final types, which are types that cannot extend beyond a defined scope. Java has final classes, Kotlin has sealed classes, and Scala has sealed traits. The pattern addresses a real problem in API design: sometimes, allowing arbitrary external implementations of an interface creates more problems than it solves.

Rust doesn’t have a built-in sealed keyword, but the module system provides a way to achieve the same effect. Following its appearance in various standard library crates and its documentation in API design discussions, the technique gained widespread recognition in the Rust community. While not unique to Rust, the pattern fits naturally with Rust’s privacy model and module system.

The sealed traits pattern uses Rust’s module privacy to create traits that are visible for use but restricted for implementation. This provides library authors with control over trait implementations while maintaining a public interface.

The problem with open traits

Let’s understand why we might want to seal traits. Because this is an open trait, anyone can implement it:

pub trait MessageSchema {
     type Data;
     type Error: std::error::Error;

    fn parse(data: &[u8]) -> Result<Self::Data, Self::Error>;
     fn serialize(data: &Self::Data) -> Vec<u8>;
     fn validate(data: &Self::Data) -> bool;
 }

Here, we’ve defined a very straightforward trait defining the signatures of three methods, with associated types for the data payload and the error type. There is nothing unique about this trait, but what we should pay attention to is that this is a public trait, so there are no restrictions on who can implement it or where the implementation happens.

External crates could implement unsafe versions:

// struct UnsafeSchema;
// impl MessageSchema for UnsafeSchema {
//     fn parse(_data: &[u8]) -> Result<Self::Data, Self::Error> {
//         unsafe { /* potentially unsafe implementation */ }
//     }
// }

The MessageSchema trait defines an interface that any type can implement. While this flexibility is often desirable, it means we can’t make guarantees about all implementations. External code could implement the trait incorrectly or in ways that violate our assumptions.

When external crates can implement MessageSchema, we lose control over quality guarantees and make it difficult to evolve the trait interface without breaking external implementations.

External implementations might do the following:

Additionally, evolving the trait interface becomes difficult. Adding a new required method would break all external implementations, forcing us to use default implementations even when that’s not ideal.

Implementing the sealed traits pattern

Let’s implement the sealed traits pattern for our message schema system, using a private module containing the sealing trait:

mod private {
    pub trait Sealed {}
}

// Public trait that extends the sealed trait
pub trait MessageSchema: private::Sealed {
    type Message: Clone + std::fmt::Debug;

    fn serialize(message: &Self::Message) -> Vec<u8>;
    fn deserialize(bytes: &[u8]) -> Result<Self::Message, SchemaError>;
    fn schema_id() -> &'static str;
    fn validate(message: &Self::Message) -> Result<(), ValidationError>;
}

The key to the sealed traits pattern is the private module. The Sealed trait is public within the module, but the module itself is private. External crates can see MessageSchema (which extends private::Sealed) but cannot implement Sealed, which means they cannot implement MessageSchema.

This gives us complete control over which types can implement our schema trait.

Let’s implement a JSON schema:

pub struct JsonSchema;

impl private::Sealed for JsonSchema {}

impl MessageSchema for JsonSchema {
    type Message = serde_json::Value;

    fn serialize(message: &Self::Message) -> Vec<u8> {
        serde_json::to_vec(message).unwrap_or_default()
    }

    fn deserialize(bytes: &[u8]) -> Result<Self::Message, SchemaError> {
        serde_json::from_slice(bytes)
            .map_err(|e| SchemaError::DeserializationFailed(e.to_string()))
    }

    fn schema_id() -> &'static str {
        "json_v1"
    }

    fn validate(message: &Self::Message) -> Result<(), ValidationError> {
        if message.is_null() {
            return Err(ValidationError::InvalidValue(
                "Message cannot be null".to_string()
            ));
        }
        Ok(())
    }
}

JsonSchema implements both private::Sealed (which we can do because we’re in the same module) and MessageSchema. The implementation uses serde_json for parsing and serialization and validates that JSON values aren’t null.

Let’s add a text schema for plain String data:

pub struct TextSchema;

impl private::Sealed for TextSchema {}

impl MessageSchema for TextSchema {
    type Message = String;

    fn serialize(message: &Self::Message) -> Vec<u8> {
        message.as_bytes().to_vec()
    }

    fn deserialize(bytes: &[u8]) -> Result<Self::Message, SchemaError> {
        String::from_utf8(bytes.to_vec())
            .map_err(|e| SchemaError::DeserializationFailed(e.to_string()))
    }

    fn schema_id() -> &'static str {
        "text_v1"
    }

    fn validate(message: &Self::Message) -> Result<(), ValidationError> {
        if message.is_empty() {
            return Err(ValidationError::FieldRequired(
                "message content".to_string()
            ));
        }
        Ok(())
    }
}

TextSchema handles plain UTF-8 text. It deserializes byte slices into strings and validates that they aren’t empty. Both schema implementations are within our module, so both can implement private::Sealed.

External crates can use JsonSchema and TextSchema, but cannot create their own schema implementations; the sealed traits pattern prevents it.

The sealed traits pattern leverages Rust’s module privacy: private::Sealed is public within the module, but the sealed module itself is private. External crates can see and use MessageSchema, but cannot implement Sealed, which means they cannot implement MessageSchema.

Type-safe messages with sealed schemas

Now, we can combine sealed schemas with the type-safe message patterns we’ve developed. TypedMessage will be generic over the schema type, ensuring that messages and their schemas are always compatible. The sealed traits pattern prevents external code from creating incompatible schema implementations:

use std::marker::PhantomData;
use crate::schema::MessageSchema;
use crate::types::*;

#[derive(Debug, Clone)]
pub struct TypedMessage<S: MessageSchema> {
    pub id: MessageId,
    pub content: S::Message,
    pub schema_type: PhantomData<S>,
}

impl<S: MessageSchema> TypedMessage<S> {
    pub fn new(id: MessageId, content: S::Message) -> Result<Self, ValidationError> {
        S::validate(&content)?;

        Ok(TypedMessage {
            id,
            content,
            schema_type: PhantomData,
        })
    }

    pub fn to_bytes(&self) -> Vec<u8> {
        S::serialize(&self.content)
    }

    pub fn schema_id(&self) -> &'static str {
        S::schema_id()
    }
}

TypedMessage is generic over the S schema type, where S implements MessageSchema. The content field has the S::Message type, which is the associated type we defined in the message schema trait for the type of our message payload. When we wrote our trait implementations, we defined concrete types for MessageSchema::Message: serde_json::Value for TypedMessage<JsonSchema> and String for TypedMessage<TextSchema>. The schema_type field uses PhantomData to track the schema type at compile time without storing anything at runtime. The new constructor validates the content using the schema before creating the message.

We can extend TypedMessage with additional methods for parsing raw data. The following shows how you might add a parse method (this extension is not in the base repository, but demonstrates the pattern):

impl<S: MessageSchema> TypedMessage<S> {
    pub fn parse(
        id: MessageId,
        raw_data: &[u8],
    ) -> Result<Self, S::Error> {
        let content = S::deserialize(raw_data)?;
        Ok(Self {
            id,
            content,
            schema_type: PhantomData,
        })
    }

    pub fn serialize(&self) -> Vec<u8> {
        S::serialize(&self.content)
    }
}

pub type JsonMessage = TypedMessage<JsonSchema>;
pub type TextMessage = TypedMessage<TextSchema>;

The extended parse method takes raw bytes and uses the schema’s deserializer to construct typed content. If parsing fails, the error type is S::Error, which is serde_json::Error for JSON or std::str::Utf8Error for text. The serialize method converts the content back into bytes. This demonstrates how you can build on the sealed trait foundation to add richer functionality.

The type aliases provide convenient names: JsonMessage and TextMessage are clearer than writing TypedMessage<JsonSchema> everywhere. These messages are both type-safe (wrong schema = compile error) and sealed (can’t add new schemas externally).

Thus, the sealed traits pattern provides controlled extensibility. We maintain the ability to evolve our trait interface while providing compile-time guarantees about which implementations exist.

Summary

In this deep-dive, we explored four patterns that leverage Rust’s type system, each with established origins in programming language research and practice.

The NewType pattern comes from functional programming languages, particularly Haskell, where it has been a formal language feature since the 1990s. Rust’s contribution is making the pattern zero-cost: type safety is enforced at compile time and optimized away entirely at runtime, making it practical for systems programming, where performance matters.

Parse, don’t validate was articulated by Alexis King in 2019, building on ideas from functional programming and type theory. Rust’s type system, ownership model, and Result type make this principle natural to apply, turning boundary validation into type-level guarantees that persist throughout the program.

The TypeState pattern originated in formal methods research, specifically in Strom and Yemini’s 1986 work on program verification. Rust’s generic types with PhantomData, zero-cost abstractions, and ownership system make this academic concept practical for everyday systems programming, providing compile-time state machine verification with no runtime overhead.

Sealed traits draw from similar concepts in Java (final classes), Kotlin (sealed classes), and Scala (sealed traits). Rust implements the pattern through its module privacy system, providing a natural fit with the language’s existing visibility rules.

What these patterns demonstrate is Rust’s ability to take established ideas from type theory, functional programming, and formal methods and make them practical for systems programming. The combination of strong static typing, zero-cost abstractions, and memory safety without garbage collection creates opportunities to apply patterns that were previously either too expensive or impractical in systems languages.

Our enhanced Samsa system now leverages decades of programming language research while maintaining the performance characteristics necessary for systems programming. This synthesis, applying proven patterns in a performant, safe environment, represents one of Rust’s key contributions to the programming language space.

The book’s next chapter explores patterns from functional programming, including function composition pipelines, generics as type classes, pattern matching techniques, and closures as architectural components.

More from Evan Williams

As experienced engineers, we already know how to build reliable systems. This deep-dive isn’t about reinventing those principles, but about adapting them for the unique, messy challenges LLMs throw at us. Consider this an architect’s playbook that outlines the new patterns needed to meet these challenges.

This deep-dive is Chapter 2 of System Design for the LLM Era by Sampriti Mitra, shared with permission from Packt, with all rights remaining with the publisher and no reproduction or redistribution without written consent. You can get the full book here.

Featured workshop: Loop Engineering for AI Agents

A 4-hour hands-on workshop on Loop Engineering, designing agent workflows with Claude Code, SDD, and MCP that verify their own work and stay observable in production.

Use DEEPENG40 for 40% discount.

Register here

Designing for resilience and reliability

Our system’s stability is now tied to an external API that is slower, more expensive, and less predictable than any database or microservice calls. The primary goal is to decouple the application’s health from the provider’s health. Two types of pattern are especially valuable in this regard: the GenAI service pattern (or LLM gateway pattern) and the circuit breaker pattern.

Pattern: the GenAI service or LLM gateway

When we first start building with LLMs, our instinct is to treat them like any other third-party API. We install the SDK, generate an API key, and make the call directly from our application code.

It feels fast. It feels efficient. We write a Python function, import openai, and we are shipping features in minutes.

The architecture looks like this:

While this works for a weekend hackathon, it creates a high-coupling, low-cohesion architecture in production with the following issues:

Stating the problem more formally now:

Problem: Our services should not be calling OpenAI, Anthropic, or Google directly. This creates high-coupling, high maintenance problems.

Solution: To solve this, we borrow a pattern from traditional microservices: the API gateway. We stop treating LLMs as external vendors and start treating them as a unified internal resource.

Implement a single, centralized LLM gateway. This is a microservice that acts as the only entry point for all LLM calls. Our services talk only to this gateway using a single, unified API format. The gateway handles the messy details of talking to the outside world.

This pattern brings some significant benefits:

Pattern: circuit breakers with tiered fallbacks

An LLM provider might be slow, down, or just returning bad data. Simply retrying a failed call (like you would for a 503 on an external service) is often the wrong move.

The solution is to combine the circuit breaker pattern with tiered fallbacks following a three-stage model:

1. Monitor: The GenAI Service monitors the health (latency, error rate) of each model provider.

2. Trip: If a primary model (e.g. GPT-5) exceeds a failure threshold, the circuit opens.

3. Reroute: All subsequent requests are immediately and automatically rerouted to a backup model.

For example, we might implement the following tiered fallback strategy:

Tier 1: GPT-5 (high-cost, high-reasoning)

Tier 2 (fallback): Claude 3 Haiku (medium-cost, fast)

Tier 3 (fallback): Llama 3 8B (locally hosted, free, less smart)

Tier 4 (final fallback): Return a cached good enough response or a graceful error message to the client UI: ‘Our AI assistant is at high capacity, please try again in a moment’

We cannot leave the circuit open forever. We need recovery logic to close the circuit that includes a half-open state:

Sleep: After the circuit trips, wait for a defined cooldown (e.g. 30 seconds).

Probe: Allow a single canary request to pass through to the primary provider.

Reset: If the canary succeeds, close the circuit and resume full traffic. If it fails, restart the cooldown.

For our retry strategy we have two options: interactive and asynchronous:

Designing for low latency

LLM inference is fundamentally slow. A user requesting a search result expects a response in < 500 ms, but an LLM might take 5–10 seconds to generate a full answer. We cannot change the speed of inference, but we can architect around it. Hybrid processing (synchronous vs. asynchronous), response streaming and caching patterns can help us.

Pattern: hybrid processing (synchronous vs. asynchronous)

Given that we cannot block a user’s web request for 10 seconds while waiting for an LLM, one option is to separate the workloads. This is the most important latency-saving pattern.

Synchronous path (< 2 s): For immediate, low-latency needs. These are tasks that must be fast, like code complete or a real-time e-commerce search. These paths should use fast, cheap models or rely heavily on caching.

Asynchronous path (> 10 s): For high-latency, long-running tasks use a message queue (like Kafka or SQS) to decouple the initial request from the actual LLM work. The client receives an immediate ‘202 Accepted’ response. For example, use when generating an AI-powered report, in agentic workflows, or for offline content generation. The client can poll for the result or receive it via a WebSocket/callback.

Pattern: response streaming

Even a fast 3-second response feels slow if the user is staring at a loading spinner, so if possible stream the response token-by-token. Once the model generates the first word, send it to the client. This dramatically improves perceived latency by lowering the time-to-first-token (TTFT). This is a non-negotiable pattern for any conversational or chat application.

How it works

We use SSE (server-sent events) over standard HTTP. SSE is unidirectional (server -> client) and runs over standard HTTP/2, making it firewall-friendly and easy to implement.

Client: Opens a persistent connection.

Server: Instead of returning a JSON object, it returns a generator (an iterator)

Header: The server must set content-type: text/event-stream

Format: Data is sent in chunks prefixed ‘data:’:

Pattern: caching strategies

LLM calls are slow and expensive, while cache hits are the fastest, cheapest LLM calls you can make. Therefore implement a multi-level caching strategy:

Level 1: exact match

Level 2: semantic match

Level 3: proactive caching

Pattern: coalesce caching

During high-traffic events, thousands of users might ask the exact same question simultaneously. A standard cache misses the first time for everyone, causing a stampede of identical requests to the LLM.

The solution is middleware that identifies identical in-flight requests. It pauses subsequent requests, waits for the first request to complete, and then serves that single LLM response to all waiting users. This reduces load on the provider by orders of magnitude.

Designing for cost optimization

LLMs introduce a new, variable, and unbounded operational cost (COGS – cost of goods sold). A single complex query can cost dollars. Architectural decisions are now financial decisions. Important in this context are the model router, dynamic traffic control and prompt engineering and compression patterns.

Pattern: the model router

Not all tasks require the smartest (and most expensive) model. Using GPT-5 for a simple grammar check is like booking a helicopter to avoid traffic. By implementing a rule engine within your LLM Gateway to act as a model router you can dynamically route requests to the cheapest model that is good enough for the task.

Example rules:

IF task_type == 'simple_grammar_check' THEN route_to 'local_llama_8b'.
IF task_type == 'complex reasoning' AND user_tier == 'premium' THEN route_to 'GPT-5'.

Pattern: dynamic traffic control (utilization-based routing)

A static rule for traffic routing, e.g. IF task == complex THEN GPT-5, can cause bottlenecks during traffic spikes. In this case we can use a production-grade router that acts as a traffic controller, implementing load shedding if the primary model’s latency breaches the P99 SLA (e.g. > 2s) due to provider congestion. In those circumstances the router proactively shifts traffic to the faster or cheaper model, even for complex tasks. A cost ceiling can be established by enforcing a hard cap on tokens. If a prompt exceeds a threshold (e.g. 8k tokens), force-route to a lower-cost model to prevent a single query causing cost regression.

Pattern: prompt engineering and compression

Cost is based on the number of input and output tokens; large prompts are expensive. Therefore treat your prompt context as a cost to be optimized. Before sending a large document (e.g. 50 pages of chat history) to an LLM, use a cheaper, faster model to summarize or compress it first.

Designing for grounding and data management

LLMs hallucinate (invent facts) and have knowledge cutoffs (their training data is stale). How can we build a reliable enterprise application on this foundation? Retrieval-augmented generation (RAG) approaches are a fundamental pattern for enabling modern LLM applications to address these fundamental issues.

Pattern: retrieval-augmented generation (RAG)

Not uncommonly we need the LLM to answer questions about private, real-time, or domain-specific data. Instead of asking the LLM a question, the solution is to tell it the answer. As the RAG name suggests, there are three principal aspects to consider:

Retrieve: When a user asks, ‘What’s the status of order #123?’, you first query the database to get the order details.

Augment: You augment the prompt with this retrieved data.

Generate: You instruct the LLM – ‘Based only on the following context, generate a friendly response’.

Example context: {order_details_json}

Pattern: the ingestion pipeline

To build the retrieval component of a RAG application, we must prepare the knowledge base (documents, tickets, etc.) for retrieval. That means creating an ingestion pipeline, typically as an asynchronous, scalable job (e.g. using Spark). This pipeline:

1. Chunks large documents into small, semantically meaningful pieces

2. Embeds each chunk by calling an embedding model API

3. Stores the chunk and its corresponding vector in a vector database (e.g. OpenSearch, Pinecone, or pg_vector [with postgres])

Pattern: hybrid RAG

RAG using just vector search (or even simple term search) is not enough for complex, interconnected data. It’s good at finding similar content, but bad at traversing relationships. Hybrid RAG approaches such as GraphRAG represent an advanced RAG pattern where the ingestion pipeline populates both a vector database (for semantic similarity) and a knowledge graph (e.g. Neptune or Neo4j) for structured relationships.

The RAG orchestrator then queries both systems to build a much richer, more accurate context, as a customer support agent system demonstrates well.

Pattern: function calling (tool usage)

To overcome the problem that LLMs are generally bad at (for example) math and cannot interact with the outside world, we ask it to decide which tool to use. We provide a schema of tools (e.g. get_weather(city)), and the LLM outputs a structured JSON object requesting that function. The application layer executes the code and feeds the result back to the LLM.

Designing for testability and observability

Testability for AI-powered systems is crucial in order to maintain the quality of response expected from the system, but for non-deterministic systems it is not trivial to write unit tests that would assert on a given known output.

We use a pattern of using an LLM as a judge by creating a golden dataset, which has input with their ideal responses, and using that dataset, the LLM judge is trained. This LLM is later used to evaluate the quality of response returned by the system against the ideal response.

Problem: How do you write a unit test for a system that is non-deterministic?

Something like assert(response) == “expected_string” will fail constantly.

Pattern: golden datasets

We need a reliable way to catch regressions in AI quality, and one way is to create a ‘golden dataset’ of 50–100 representative inputs and their ideal outputs. In your CI/CD pipeline, run your system against this set to ensure that prompt changes or model upgrades haven’t broken core functionality.

Pattern: LLM-as-a-Judge

How do you assert the quality of a golden set test at scale? You can’t manually review 100 responses on every build. What you can do, however, is use a powerful LLM like GPT-5 as an evaluator or judge. You feed the judge the original prompt, the golden answer, and your system’s actual response, and then ask the Judge to score the actual response from 1 to 5 on metrics like accuracy, groundedness, and tone. This gives you a quantifiable quality metric you can track over time.

Pattern: new observability metrics

Existing performance dashboards (CPU, RAM, 5xx errors, etc.) are insufficient in the LLM era. We need to add a new layer of monitoring focused on the LLM itself, for example:

Cost: Track Cost_Per_Query and Total_Cost_Per_User. Set alerts for cost spikes.

Performance: Monitor P99 time to first token (TTFT) and tokens per second (TPS).

Provider health: Monitor 429_Rate_Limit_Errors and 5xx_Server_Errors per provider to feed your circuit breakers.

Quality: Track your LLM-as-a-Judge scores. Monitor the escalation rate – this measures the percentage of conversations where the AI fails to resolve the issue, forcing the system to transfer the work to a human (for example in case of customer support agents). A spike in this metric indicates a drop in model quality.

Designing for security and trust

Treating an LLM as a simple API call is naive and dangerous. It’s a non-deterministic component that you are inviting inside your trusted system. We must architect our systems to defend against a new class of vulnerabilities. A variety of patterns can help us in this, including the following. In this section we summarize the main security patterns, the threats they address and the mitigations they provide.

Pattern: mitigating prompt injection

Threat: Attackers trick an LLM into ignoring its original instructions by embedding malicious commands in prompts, causing it to perform unintended actions.

Mitigation

Instruction/data separation: Clearly separate trusted instructions from untrusted data from the user. Use role-based API structures (system, user) and wrap all user input in clear delimiters like <>.

Input filtering: Use a second, simpler, faster LLM to classify the intent of a user’s prompt. If it detects a likely attack, reject it before it reaches your primary model.

Output filtering: Always validate the LLM’s response. If it contains any system prompt text or suspicious keywords, block it.

Pattern: secure output handling

Threat: Insecure output handling occurs when an LLM’s outputs are not properly sanitized before being used, potentially leading to attacks like cross-site scripting (XSS).

Mitigation

Treat as untrusted: Never eval() code output.

Sanitize and encode: If the output is HTML, sanitize it. If it’s text for a web page, encode it to prevent XSS.

Validate: If you expect JSON, parse it in a try/catch block and validate its schema.

Parameterize: If the LLM helps build a SQL query, have it generate the parameters for a pre-defined, parameterized query you control. Never execute a raw SQL string from an LLM.

Pattern: mitigating excessive agency

Threat: This happens when an LLM is given too much control and makes unauthorized decisions or actions without human oversight.

Mitigation

Dynamic permissions: Do not give the LLM agent a static set of all possible tools.

Plan-approve-execute: Implement a multi-step loop. The LLM proposes a plan. Your code approves this plan. Only then do you execute it with a scoped-down client.

Human-in-the-loop (HITL): For high-impact actions (e.g. ‘delete database’, ‘refund customer’), always require explicit human approval.

Pattern: mitigating sensitive information disclosure

Threat: The model may inadvertently reveal confidential data or personally identifiable information (PII) that was present in its training data.

Mitigation (data hygiene)

PII/data scrubbing: Aggressively sanitize all data before it is used for training or RAG ingestion.

Zero-retention policies: For ultra-sensitive data (like user code), process it in-memory and discard it immediately. Do not store it.

Tenant-level RAG filtering: This is a critical architectural pattern. All RAG queries must include a filter for tenant_id or user_id. Never perform a vector search on the entire database and hope the LLM picks the right data.

Pattern: preventing model denial of service (MDoS)

Threat: Attackers overload the LLM with resource-intensive requests, slowing it down or making it unavailable.

Mitigation

API rate limiting: Enforce strict per-user and per-IP rate limits at the API gateway

Input validation: Reject queries that are obviously abusive (e.g. a 50,000-token prompt)

Cost-based throttling: Implement logic to monitor the cost of a user’s queries in real-time. If a single user is incurring high costs, temporarily throttle their access. We would want to return a 429 in case of user spam/abuse or redirect to a lower model in case user hit their budget.

Here’s an example implementation:

from fastapi import HTTPException

def route_request(user, prompt):
    # 1. HARD LIMIT CHECK (Redis Counter)
    current_rate = redis.get(f"rate:{user.id}")
    if current_rate > 50:
        # Scenario A: Abuse -> Hard Stop
        raise HTTPException(status_code=429, detail="Rate limit exceeded.")

    # 2. SOFT BUDGET CHECK (DB Query)
    daily_spend = db.get_spend(user.id)
    budget_limit = 10.00  # $10 limit
    if daily_spend > budget_limit:
        # Scenario B: Over Budget -> Downgrade (Soft Throttle)
        # We don't fail; we just swap the model
        print(f"User {user.id} over budget. Downgrading to Tier 3.")
        return call_llm(model="llama-3-8b", prompt=prompt)

    # 3. NORMAL PATH
    return call_llm(model="gpt-4", prompt=prompt)

Pattern: preventing data poisoning

Threat: Malicious actors tamper with an LLM’s training data to corrupt its behavior, leading to biased or incorrect outputs.

Mitigation (ingestion control):

Trusted sources: Only use and ingest data from known, trusted sources.

Data lineage: Track the origin of all data used for training or RAG.

HITL review: For fine-tuning data, use human experts to review and validate the datasets before training.

Pattern: mitigating supply chain and insecure plugin vulnerabilities

Threat: The security of an LLM can be compromised through third-party services, plugins, or datasets that are themselves vulnerable. This includes insecure plugin design, which can be exploited for attacks like SQL injection.

Mitigation

Minimize functionality: Any plugin or tool given to the LLM should have the absolute minimal functionality needed (e.g. read_email, not delete_email).

Validate plugin inputs: Treat all data passed to a plugin as untrusted. Sanitize it to prevent injection attacks within the plugin itself.

Vulnerability scanning: Regularly scan all third-party libraries, containers, and models for known vulnerabilities.

Use the genAI service / LLM gateway: Your gateway allows you to quickly replace a provider or model that is found to be compromised.

Engineering for production

We cannot manage what we do not track. Beyond standard APM (application performance monitoring), we must track:

TTFT (time to first token): The perceived latency. How long until the user sees the first character? High TTFT kills engagement.

TPOT (time per output token): The generation speed. If this is high, the model is too heavy or the provider is overloaded.

Context utilization: Is the context window filled? Use the model optimized for token usage for the use case for cost effectiveness.

Training with test data

In earlier sections we have discussed golden datasets – inputs with their ideal outputs used for training the LLM as a judge. Let’s now look at how we can procure this ideal data for different use cases.

Sourcing datasets

Testing and training for code complete

Randomly remove code blocks, functions, classes from the code. Start typing signatures of missing classes and functions. Compare the IDE code complete suggestions with the actual classes and functions to measure accuracy and semantic correctness. Iteratively tune the LLM model prompts based on error cases and coverage gaps identified.

Testing for chat responses

Clone the repo removing all the comments and docs. Ask the system to explain code blocks, functions, classes and compare against the original repo’s docs, README files or comments.

Testing for agentic workflow and tasks

Assign real world tasks like adding documentation, writing test cases or refactoring to the agent. Combine multiple tasks in scenarios.

Automatically check the code and tests and documentation for correctness and style.

Integrating these tests into CI/CDs helps keep models and prompts updated with evolving repos.

Pattern: quantitative evaluation testing

The goal of this pattern is to move beyond vibes and qualitative observations to a measurable correctness percentage.

The problem is that agentic flows are multi-step and non-deterministic. Traditional pass/fail unit tests often fail to capture the nuance of a complex task that is mostly correct but slightly off in tone or style.

To tackle this distinctive character we assign a numerical score to every agentic run by breaking the output into weighted criteria. This allows us to track an evaluation success rate, the percentage of tasks successfully completed to the golden standard.

To calculate the success rate we define weights to the different output facets that reflect their relative importance (e.g. logic: 50%, syntax: 30%, documentation: 20%).

Then we execute the agent across a golden dataset of at least 50 representative tasks, and score according to the following metric:

Figure 2.13 shows the overall evaluation process:

When to use it: Use this in your CI/CD pipeline. If a prompt change or model upgrade causes the correctness percentage to drop below a defined threshold (e.g. 90%) then the deployment should be automatically blocked to prevent quality regression.

Mutation testing

Add logic and syntax errors to your code. Evaluate whether the system can identify and fix these errors.

Negative prompt testing

Give confusing or risky actions like ‘Delete all databases’. Evaluate whether the system is able to flag it as a risk and ignore/refuse or ask the user for further clarification.

Respecting user privacy

We want to be able to use an LLM-powered system for code completions, reviews, generation etc. without leaking our codebase to the model. Techniques for ensuring that user privacy is maintained while the data is still served from the model include:

Summary

The patterns we’ve just looked at, from the GenAI service and async queues to RAG and LLM-as-a-Judge, form an architect’s playbook for building production-grade, AI-powered systems. In the rest of the book we examine four case studies that utilize this playbook.

This eats into the bottom line and the return on engineering investment as the bill keeps climbing while the system handles less real work. Circuit breakers finally trip and try to move traffic elsewhere, but those other paths were never built to absorb a sudden flood.

Everything might still look like it is working on paper, but the whole situation keeps getting worse. Now compound that by adding agentic AI to operations, systems that recommend or execute operational actions, which increases the speed and breadth of change under conditions that are already chaotic.

AI-driven operational automation can increase fragility when it increases the rate, scope, or coupling of production actions without bounded actuation, explicit constraints, and auditability. Under governance, the same automation reduces fragility by enforcing safety envelopes and reversible decision paths.

This is not an argument against automation or AI. It is an argument that reliability engineering must evolve from managing distributed systems to governing the automation that manages them, especially under simultaneous SLO, cost, and compliance constraints. I call this discipline Autonomic Reliability Governance (ARG). ARG is not more automation. But governed automation that is policy-constrained, auditable, and rollback-capable by design. It ensures that AI-assisted or agentic operational decisions cannot escalate into amplification engines under stress.

Why reliability automation amplifies in microservice ecosystems

In microservice systems, outages rarely come from a single broken component. They emerge from how services interact, through feedback loops, dependencies, and cascading effects. Mechanisms that improve reliability in normal conditions can unintentionally amplify problems when the system is already under stress, which is the feedback behavior Karl Johan Åström and Richard Murray formalize in Feedback Systems.

Retries multiply load under stress

Retries improve success rates when failures are transient and capacity is available. Under degradation, retries become a multiplier, the dynamic Jeffrey Dean and Luis André Barroso describe in “The Tail at Scale”.

This is how a minor regression becomes a self-inflicted flood. It can resemble a resource exhaustion event even when there is no attacker. The mechanism is simple. Unbounded retries consume the remaining capacity of an already-constrained dependency. The root issue is not that retries are bad. The issue is that unbounded retries are a powerful actuator that must be governed by system-level constraints such as SLO, cost, and blast radius.

Autoscaling is a blunt actuator fed by ambiguous signals

Autoscaling adds or removes capacity based on signals like CPU usage, request rate, or queue length. During incidents those signals mislead. Queue depth may grow because a dependency is slow, not because demand has grown. CPU may spike from retries and timeouts, not real workload. Request rates may rise simply because retried calls look like new traffic. A reactive autoscaler then adds capacity in the wrong place, driving up cost while putting even more pressure on the real bottleneck. The result is a system that becomes less stable and more expensive at exactly the moment it needs to recover.

Circuit breakers carry shock-wave potential

Circuit breakers are necessary. Without system-level coordination they create abrupt traffic shocks. If multiple clients trip simultaneously, alternate paths overload and create secondary failures that appear unrelated to the original degradation.

The common pattern is unbounded actuation in a coupled system

Retries, autoscaling, and circuit breakers are not mistakes. They are essential reliability tools. Problems arise when they operate like independent reflexes, without coordination or limits. When nothing controls how often they act, how broadly their actions affect the system, how easily changes can be reversed, or how clearly decisions can be explained, they unintentionally amplify failures. Managing this was already difficult with manually written rules. It becomes even more critical when AI agents make or trigger operational decisions.

Why agentic operations change the physics

Traditional automation such as scripts, thresholds, and fixed policies is limited in scope and relatively legible. Agentic systems expand both capability and risk because they tend to:

1. Increase velocity: propose actions faster than humans can validate non-local effects.

2. Increase scope: coordinate actions across services, clusters, and regions.

3. Optimize proxies: Charles Goodhart warned in 1975 that a statistical regularity tends to collapse once it is used as a control target, a caution later popularized as “when a measure becomes a target, it ceases to be a good measure” (Problems of Monetary Management: The U.K. Experience). Optimizing latency or cost in isolation can degrade reliability unless constraints are explicit.

4. Operate under uncertainty: incidents produce ambiguous signals, and partial telemetry invites misdiagnosis and overcorrection.

5. Reduce legibility unless designed otherwise: without decision traces, you cannot reconstruct why the system acted or demonstrate compliance.

So the risk is not AI in isolation. The risk is unbounded actuation under uncertainty. ARG is the discipline designed to make agentic operations governable.

A practical lens on amplification

To manage automation safely, teams need a clear way to recognize when helpful mechanisms start making things worse. Amplification is that lens. A reliability mechanism helps when it improves outcomes without adding too much extra load. It becomes harmful when it multiplies load, volatility, or system-wide effects beyond what the platform can handle. In practice, retries multiply request volume, autoscaling multiplies capacity changes and cost, traffic shifts increase pressure on dependencies, and aggressive fixes create configuration churn. ARG is essentially the discipline of keeping these amplification effects under control.

Autonomic Reliability Governance governs the automation itself

The notion that systems can manage themselves is not new. Jeffrey Kephart and David Chess articulated that aspiration more than two decades ago in “The Vision of Autonomic Computing”. What is new is the velocity and scope of actuation introduced by agentic AI. Autonomic Reliability Governance is the discipline of designing and operating bounded, auditable, and reversible automation, including agentic operational decision layers, under explicit constraints.

• SLO constraints: latency ceilings, availability targets, error budgets.

• Cost constraints: budget caps, unit-economics bounds, runaway scaling prevention.

• Compliance constraints: auditability of decisions and actions.

ARG is not a product. It is an operating model, automation you can trust because it is governable. ARG treats operational actuation, human or agentic, as a first-class risk surface with explicit safety envelopes.

Governance primitives that make agentic operations safe

ARG becomes real when it is implemented through governance primitives that align with existing reliability practice such as SLOs, error budgets, and progressive rollout, while controlling actuation velocity and blast radius.

1) Actuation budgets rate-limit change, not only traffic

Most systems rate-limit user requests. Few rate-limit operational change. ARG introduces actuation budgets, limits on how frequently high-impact actions can occur. Scaling shifts, retry escalations, routing changes, and configuration flips consume tokens from a constrained budget. When the budget is exhausted, automation may still observe and recommend, but it cannot repeatedly perturb the system. When instability rises, slow the actuators before adding more intelligence.

2) Blast radius governance scopes every action

The difference between a safe fix and an outage multiplier is often scope. ARG constrains blast radius using canaries, segmentation across cells and regions, tiered permissions, and scoped rollouts with automatic halt conditions. Even correct actions can be wrong if applied globally in one step. Autonomy must always be localized before it is generalized.

3) Advisory-first agents and progressive autonomy

In high-stakes systems, autonomy should begin as advisory. Agents recommend and humans approve. As evidence accumulates and safety is demonstrated, constrained autonomy can expand within defined envelopes.

4) Eligibility tiers put permission models over personality

Autonomy must be governed by explicit action tiers:

• Observational: detection and diagnosis, always allowed.

• Low-risk reversible actions: scoped and rollback-capable.

• Medium-risk actions: require high confidence and strict guardrails.

• High-risk actions: require human approval.

Autonomy becomes measurable when agents must graduate across tiers based on demonstrated safety.

5) Confidence gates and evidence before execution

Agents should act only when evidence crosses defined thresholds:

• sufficient telemetry completeness

• credible dependency attribution

• system stability signals

• reversibility guarantees

• bounded failure domain

If uncertainty is high, remain advisory.

6) Auditability makes the decision trace a first-class artifact

Governed autonomy requires structured decision records:

• what was observed

• what was concluded, with uncertainty

• what constraints were evaluated

• what action was taken or recommended

• what was expected

• what occurred

• how it can be reversed

Auditability is not bureaucracy. It is what enables learning, compliance, and safe expansion of autonomy.

7) Reversibility means designing for wrong decisions

Every operational action must have an undo path. Rollbacks, kill switches, and safe defaults are mandatory. Governance does not prevent all mistakes. It ensures failures stay bounded and recoverable.

Validating governance without private company data

ARG also needs ways to test ideas without using private company data. One practical approach is incident replay. You capture common failure patterns such as latency spikes, growing queues, retry storms, or autoscaling reactions, and replay them in a simulation or controlled environment. Then you compare how the system behaves under unbounded automation versus governed policies, and you check whether the changes reduce amplification or make it worse. The goal is not to perfectly recreate production, but to learn in a measurable way, so the question is whether a policy dampens instability or accelerates it. This can be reinforced with controlled stability drills such as dependency injection, brownouts, or chaos experiments, the practice Ali Basiri and colleagues describe in “Chaos Engineering” (IEEE Software, May 2016), so that automation stays controlled when conditions degrade.

What changes when ARG becomes normal practice

As systems become more autonomous, reliability work will increasingly focus on governing automation itself. That means turning constraints into enforceable policies, limiting and scoping automated actions, treating audit trails as essential system outputs, and introducing autonomy gradually rather than all at once. FinTech-scale platforms are likely to adopt this approach early because reliability targets, cost limits, and audit requirements are strict, and failures are expensive. In the end, agentic operations will not succeed simply by becoming more intelligent. They will succeed by becoming more governable.

Do not automate reliability, govern automation

Retries and autoscaling will remain foundational, and AI will increasingly enter operations. The question is whether these mechanisms amplify failures or dampen them. If we bolt AI onto production as an unbounded actuator, we create faster cascades and more expensive incidents. If we treat autonomy as a governed system that is bounded, auditable, and reversible, we can build platforms that are both resilient and efficient. Autonomic Reliability Governance is a practical blueprint for getting there.

Acknowledgements and disclosures by the Author

Submitted by Sibasis Padhi.

Generative AI tools and Grammarly were used to assist with drafting and English corrections. The author reviewed, revised, and assumes full responsibility for the final content.

About Sibasis

Sibasis Padhi is a Staff Software Engineer at Walmart Global Tech with more than 18 years building and optimizing large-scale enterprise systems, including cloud-native financial platforms that handle high-volume transactions. His work centers on microservices performance, observability, distributed systems reliability, and agentic AI. He speaks at IEEE conferences and industry events on resilient financial systems and agentic operations.

Traditional technology stacks like Java, JavaScript, TypeScript, and C# offer a lot of flexibility and scalability for large-scale applications, but their inherently nondeterministic nature makes it very difficult to get below a p50 latency of 500 microseconds. While there are ways to optimize garbage collection and other Java components for latency, most standard Java libraries do not focus on low-level concepts like aligning data to cache lines or reducing cache line misses. C and C++ applications optimize for a specific platform at the cost of portability, but that trade-off is exactly what gives you access to manual memory management and the ability to avoid multithreading pitfalls like false sharing. When the goal is to reduce latency, you need lower-level languages like C, C++, or Rust on critical paths. I will not cover pure hardware implementations like FPGAs or ASICs here since they are not required for our 50-microsecond latency target.

Important Definitions

Before diving into the architecture and hardware optimizations, it will be helpful to establish exactly how performance is measured in a low-latency environment.

Wire-to-wire latency

Wire-to-wire latency is measured from the moment a market data packet is received on the NIC from the exchange to the moment an order is sent back to the exchange on the same NIC. You compute it as order send timestamp minus market data receive timestamp. The most precise measurements require NIC cards with hardware timestamping enabled, so the NIC stamps every inbound and outbound packet with a hardware timestamp that you can later use to analyze the latency graph. Exanic cards provide APIs for retrieving these hardware timestamps.

Assumptions About the System

To hit the latency target of under 50 microseconds, you need to optimize both hardware and server setup alongside your software, and the following assumptions underpin everything in this article.

• Colocated host: Most stock exchanges in traditional finance provide colocation services where your servers sit physically adjacent to the exchange’s servers, removing extra network jitter from the equation.

• UDP market data access: Most stock exchanges distribute market data via UDP multicast to all subscribed parties, and this is the fastest way to consume and react to it.

• Kernel bypass NIC: The system runs on NIC cards enabled for kernel bypass and tuned for low-latency operation.

• Single venue: These systems are single-venue, meaning at any point they process market data to generate orders for one stock exchange only. For multi-venue setups, the assumption is one colocated host per venue with no shared critical-path processing.

Workload Model

Most of the trading systems follow this model of data processing in abstract terms:

Every strategy engine takes market data from the exchange as its primary input and produces orders sent to the exchange as its primary output, moving through multiple stages as described below. Exchange responses are a secondary input to the engine but are not on the hot path for latency calculations.

Market Data Ingestion

The first layer is responsible for ingesting the market data (raw or normalized) from the network card. This layer routinely processes terabytes of data per day and can exceed a petabyte on busy days at venues like CME or Eurex. On an average day at CME you can expect 20 to 30 million trades and 100 to 150 million open interest events in a single segment. Processing a full L1 feed for one segment means receiving traffic in the range of a few terabytes to hundreds of terabytes per day, so both software and hardware need to be extremely optimized.

Filtering and Checks

Filtering and checks come next, cleaning up incoming data and validating it before it reaches downstream stages. This layer handles packet drops, invalid market data, trading phase transitions like moving from an active session to a halted one, and anomalous price changes that might indicate a software bug. When something goes wrong at this stage, the right response is often to instruct strategies to stop rather than pass bad data forward.

Pricing Engine

The pricing engine receives valid and normalized market data and can range from a simple no-op or currency conversion all the way to a full ML-powered model. Lightweight pricing fits comfortably on the critical path. Complex ML models that require large amounts of memory and processing do not, because they would blow the latency budget. In practice, ML-powered engines in low-latency applications split into two processes: time-consuming training and inference model preparation happen off the critical path, and a lightweight inference step runs on the critical path using small linearized models fed configuration parameters from the heavier ML output. This keeps the heavy lifting off the hot path while still giving the strategy access to model-driven signals. Colocation also constrains available compute, so you want to reserve it for work that directly impacts critical path performance.

Strategy Evaluation

Strategy evaluation compares transformed or theoretical prices against real prices and fires an order when it identifies an opportunity. Strategies generally fall into three categories.

• Maker: Maker strategies keep quotes on both the buy and sell sides to provide liquidity and take advantage of the bid-ask spread. Exchanges often provide incentives to HFT firms that add liquidity this way.

• Taker: Taker strategies identify mispricing and fire IOC or FOK orders for immediate execution.

• Hedging: Hedging strategies reduce risk or book profit by executing the opposite side of an existing position.

Order Aggregation and Routing (OMS)

Order aggregation and routing (OMS) can be combined with strategy evaluation in simple setups, but should be separated as complexity grows. It serves three purposes: aggregating orders from multiple strategies to reduce the number of orders sent to the exchange, managing exchange responses and passing normalized results back to the strategy, and handling the routing logic that grows more complex as your strategy count increases.

Latency Map: Where the Latency Lies

Now that the high-level components are clear, it is worth mapping out where the latency actually sits so you can allocate your budget with precision. Beyond the business logic itself, your latency profile depends on CPU model and generation, NIC type and model, operating system, software architecture, and business logic complexity.

Beyond the business logic stages, several factors contribute latency that stays invisible if you only look at application code. These factors come from hardware, OS and kernel processing, network behavior, and I/O, and they play an outsized role in ultra-low-latency systems. Even if they occur infrequently, they introduce jitter and latency spikes that directly hurt your p99. Although many of these latency sources depend on a multitude of factors, you can work with approximate cost ranges that provide a useful benchmark, and the table below gives those ranges for the most common sources.

Engineering Pillars

Addressing these latency sources requires a set of engineering decisions baked into the application design itself, not patched in afterwards. The pillars below cover the most impactful areas.

Kernel Bypass

Traditionally, any application reading from the network goes through the kernel as an intermediary, which means a system call copies data from the NIC into kernel space before the application can access it by copying from kernel space into its own buffers. That double-copy plus the system call overhead adds up quickly because control needs to transfer to and from the kernel on every read, and the way to avoid it is to access the NIC directly through its driver. DPDK polling mode is one of the best approaches for this and can save roughly 20 to 45 microseconds when tuned correctly for market data processing.

Specialized hardware is also worth considering for additional gains on top of software-level kernel bypass. Exanic and Solarflare cards are widely used in the trading industry and provide zero-copy data access, hardware filtering, hardware timestamps, and flexible flow steering.

Memory Management

Most shared or larger resources get allocated on the heap through dynamic allocation using malloc or new, which requires a system call to expand the heap and introduces additional latency. Low-latency systems avoid this by pre-allocating memory segments at startup and building custom allocators on top of those segments to distribute memory during runtime, with the primary goal of eliminating all system calls from the hot path entirely. At startup you make one large allocation to cover the maximum memory any component might need, and during trading hours your custom allocator distributes from that pool without ever touching the OS. Most large-scale systems package this into a library of containers, allocators, and pools that lets you use vectors, maps, and arrays without the penalty of dynamic allocation by implementing custom versions of these types or custom allocators.

In practice it is always worth benchmarking different allocators to find the one that performs best for your specific workload, since allocators have different configuration parameters that trade off between latency, throughput, and fragmentation.

CPU and NUMA Affinity

Modern CPUs use NUMA architecture, dividing processors into nodes where each node has its own processors, caches, and memory, and communication between nodes happens over interconnects that add measurable latency when crossed on a hot path. For critical processes, you want to pin threads to cores within the same NUMA node and ensure that the data those threads access is also allocated on the same node. Non-critical processes like logging, compliance reporting, and observability can run on a separate NUMA node since they care more about throughput than latency.

An example core allocation for a machine with 8 cores per NUMA node and 2 NUMA nodes might look like this.

This layout lets you run two independent sets of strategies, one per NUMA node, while keeping logging and observability isolated from the critical path.

Time Sources and Clocks

Getting timestamps right matters a lot more in low-latency systems than in most applications because you are measuring at nanosecond granularity, and the table below covers the main options along with their trade-offs. For cross-host applications, you also need a way to synchronize time across multiple machines at nanosecond precision — PTP hardware clocks are the standard approach for this.

Network Tuning

Sockets and hardware both need tuning to get the best latency and throughput out of your setup. The key adjustments are disabling interrupts and relying on polling to avoid breaking application flow, increasing the RX buffer size to reduce packet loss, and increasing the TX buffer backlog queue size, with many other hardware-specific options available depending on your configuration.

Observability and Monitoring

Monitoring is not optional in low-latency systems because without good observability you cannot see where latency is being spent. The key metrics to track are first-order-out latency for both software and wire-to-wire paths, p50, p95, p99, and p99.9 latencies at both levels, syscall latencies using bcc and bpftrace, CPU utilization to find headroom and spot contention, and NIC-level metrics using tools appropriate to your hardware.

Failure Modes

Deployment Checklist

The hardware tuning techniques below are commonly used across trading organizations, but because you are tuning at the hardware level the right set of parameters depends on your specific hardware. These techniques also carry trade-offs like higher power consumption and increased CPU heat, so test carefully and find the optimal configuration for your application before applying them in production.

Hardware and BIOS

• Disable C-states and Turbo Boost

• Set CPU governor to performance mode

• Lock PCIe ASPM off

• Enable hugepages

OS Tuning

• Use isolcpus, nohz_full, and rcu_nocbs kernel parameters

• Apply sysctl buffer tuning

• Use mlockall to pin memory and prevent paging

Process and Code

• Use static preallocation and avoid dynamic containers on the hot path

• Prefetch and cache-align structs

• Use lock-free SPSC queues

• Use TSC for all timing on the critical path

Common Pitfalls

• False sharing between threads on the same cache line: This is one of the most common mistakes in multithreaded systems. When two threads write to variables that share a cache line, even if those variables are logically unrelated, the cache coherency protocol treats the entire line as contested and forces unnecessary cross-core coordination. Many cache miss problems in multithreaded environments trace back to this.

• Chatty IPC over sockets or message queues: Chatty IPC adds up quickly. Many distributed systems default to message queues, RPCs, or other message-passing mechanisms, but these carry much higher latency in practice even when they scale better horizontally. For low-latency IPC between processes on the same host, shared memory is the right approach.

• Premature SIMD optimization: Premature SIMD can actually increase latency when your workload is memory-bound rather than compute-bound. SIMD brings real benefits in the right situations but also adds complexity. Always benchmark before and after adding vectorization because the performance impact can go in either direction depending on your memory access pattern.

• TSC skew across cores: TSC skew is a problem that bites HFT systems specifically because they measure at nanosecond granularity. TSC is the standard clock for critical path timing because it does not require a system call, but TSC can drift across cores on the same CPU. Left uncorrected, that skew introduces measurement errors that look like real latency spikes and can obscure genuine performance bugs. Check your CPU documentation for the recommended synchronization approach and apply it regularly.

• Running without baseline histograms: Running without baseline histograms makes all of the above invisible. Every parameter including compiler settings, execution environment, hardware, and network can affect latency, and changing a single line of code can produce dramatically different results. Measuring p50 and p99 histograms at every stage of the pipeline is the only reliable way to know where your latency is coming from.

Further reading /Additional resources

1. Linux kernel vs DPDK

2. AF_XDP: Zero-Copy Packet Processing in Linux — Netdev 2024:

3. Benchmarking allocators

4. Time Synchronization in Distributed Trading Systems

5. BPF Performance Tools by Brendan Gregg

6. Intel BIOS optimizations

7. RHEL 7 low-latency tuning guide

8. IPC vs shared memory:

9. Incorrect use of SIMD:

Unlike traditional approaches to asynchronous programming, such as threads and callbacks, coroutines eliminate complexity by providing a way to write asynchronous code that looks and behaves like synchronous code. This chapter introduces you to the fundamentals of coroutines, teaching you how to avoid issues with callbacks and enabling you to manage concurrency gracefully and efficiently.

Kotlin’s coroutines framework is built on top of lightweight threads, providing developers with the tools to execute tasks in parallel, handle timeouts and cancellations, and switch between different execution contexts. Throughout this chapter, we will uncover how coroutines work, why they are an improvement over traditional concurrency models, and how to leverage them to build robust and responsive applications.

We will start by understanding what coroutines are and how they differ from threads and other concurrency mechanisms. Next, we will dive into the implementation of coroutines, learning about coroutine builders such as launch and async, as well as the importance of suspending functions in asynchronous workflows. We will then explore strategies for managing timeouts and cancellations, ensuring our applications remain efficient and responsive under various conditions. Finally, we will tackle the advanced concepts of context and scope, which are essential for organizing and managing coroutine life cycles effectively.

By the end of this chapter, you will have a comprehensive understanding of Kotlin coroutines, enabling you to write clean, maintainable, and high-performing asynchronous code. Whether you’re developing Android applications, server-side services, or any system requiring concurrent operations, mastering coroutines will empower you to tackle complex challenges with confidence and precision.

We’re going to cover the following topics:

• What is a coroutine?

• Coroutine implementation

• Timeouts and cancellations

• Contexts and scope

Technical requirements

You can use any text editor or Integrated Development Environment (IDE) of your choice with the Java language. We recommend IntelliJ IDEA Community Edition (CE). This tool already comes with everything you need to work with both Java and Kotlin. The software requirements for this chapter, with required versions, are as follows:

• Java JDK 17 or higher

• IntelliJ IDEA CE

• Kotlin 1.x or 2.x

The code used in the chapters can be found in the book’s accompanying GitHub repository: https://github.com/PacktPublishing/Kotlin-for-Java-Developers.

What is a coroutine?

To simplify what a coroutine is, let’s first look at the official definition: “A coroutine is an instance of a suspendable computation.” While technically accurate, it may seem too abstract. Instead, let’s break it down with a practical and relatable analogy.

Imagine you are dining at a restaurant. A waiter comes to your table, takes your food order, and heads to the kitchen to pass the order on to the chefs. Now, imagine the waiter stays in the kitchen, doing nothing but waiting for your food to be prepared. This would be inefficient, as the waiter could be serving drinks, attending to other tables, or bringing bread to your table while the food is being cooked.

A smarter approach is for the waiter to leave the instructions with the kitchen staff and return to attend to other customers or tasks. Once your food is ready, the kitchen would alert the waiter, who would promptly bring the dishes to your table.

This efficiency is exactly how coroutines operate. In programming, there’s a “main thread” (like the waiter), which handles primary tasks (like serving the diners in our analogy). However, certain tasks (such as cooking the food) can take an unknown amount of time. These tasks are better handled asynchronously, without blocking the main thread.

The restaurant analogy

To make the concept of coroutines easier to grasp, let’s stick with the restaurant analogy. This comparison helps illustrate how asynchronous tasks and concurrency work in Kotlin using coroutines:

Let’s try to understand this analogy:

1. Main thread as the waiter: The main thread is responsible for visible and interactive tasks (like serving diners). It shouldn’t be blocked (or stuck in the kitchen).

2. Asynchronous tasks as the kitchen: Some operations, like fetching data from the internet or reading from a database, are similar to cooking food in the kitchen. These tasks can take time and don’t need the waiter (the main thread) to stand idle while waiting for them to finish.

3. Coroutines as the waiter’s delegation: The waiter can delegate tasks such as food preparation to the kitchen while continuing to handle other responsibilities (serving drinks, taking new orders). Similarly, coroutines allow you to “pause” tasks and resume them later without stopping the main thread.

4. Concurrent coroutines for multiple tasks: Just as a waiter can manage multiple orders at once, you can create multiple coroutines to handle different tasks simultaneously. For instance, one chef can prepare appetizers, another can handle main courses, and yet another can prepare desserts – all while the waiter continues serving diners.

In programming terms

Kotlin simplifies asynchronous programming, but it doesn’t include heavy low-level APIs in its standard library. Instead, it provides just enough tools for libraries, such as kotlinx.coroutines, to build powerful coroutine-based utilities. Unlike some other languages, Kotlin doesn’t treat async and await as keywords or include them directly in the language. Instead, it uses suspending functions, which make asynchronous code safer and easier to use compared to traditional approaches such as futures or promises.

To better understand how coroutines work, let’s map their behavior to real-world and programming concepts. This helps clarify how Kotlin approaches asynchronous programming in a more intuitive way:

• Main thread: The core execution thread that interacts with users (like serving diners).

• Coroutine: A lightweight task that runs asynchronously and can be paused and resumed without blocking the main thread.

• Suspension: Like the waiter leaving instructions in the kitchen, a coroutine can “pause” its execution, allowing the main thread to continue working until the task is complete.

The kotlinx.coroutines library, developed by JetBrains, is a feature-rich library that provides high-level tools for working with coroutines. It includes useful functions such as launch and async, which simplify working with background tasks.

Explanation of coroutines for a Java programmer

It’s important to know that Java does not have coroutines. However, there are many ways we can achieve the results that coroutines achieve, and it is very likely that those of you who know Java might know some of these ways.

While there are projects such as Project Loom and Quasar that attempt to bring coroutine-like behavior to Java, they require third-party dependencies and are not yet widely adopted in the Java ecosystem.

Java thread handling was enhanced with virtual threads in version 21. They are a more lightweight threading model. Yet, virtual threads and coroutines serve different purposes. Virtual threads are primarily used for high-throughput, I/O-bound server applications based on a thread-per-request model. Coroutines, on the other hand, are typically used for various asynchronous tasks.

Note that it is perfectly possible for a coroutine to use a virtual thread to execute its tasks, especially if these tasks include blocking I/O operations.

In this section, we explored the fundamental concept of coroutines, learning how they provide a powerful and efficient alternative to traditional asynchronous programming models such as threads, futures, and promises. Using the analogy of a waiter in a restaurant, we visualized how coroutines delegate long-running tasks to the background while keeping the main thread free to handle interactive or essential tasks. Key characteristics of coroutines, such as their non-blocking nature, ease of readability, and lightweight execution, were highlighted as significant advantages over Java’s concurrency options. For Java developers, the comparison to traditional tools such as threads and futures underlined the simplicity and efficiency that coroutines bring to the table.

In the next section, we will dive deeper into the practical aspects of working with coroutines. You’ll learn how to create and manage coroutines using coroutine builders and suspending functions, setting the foundation for writing clean and effective asynchronous code in Kotlin. Let’s begin our hands-on journey into coroutine implementation!

Coroutine implementation

Now let’s see how to implement coroutines in a Kotlin project. For this, we’ll start from scratch. First, we’ll do something very simple; we can consider it the “Hello World“ of coroutines. Then we’ll increase the difficulty of the example and show other features.

We will start by adding the possibility of using coroutines in our project, and for that, we will add the dependency.

One thing to note is that using kotlinx.coroutines has become a convention in the Kotlin community, making it easier to collaborate and understand code.

We don’t recommend trying to memorize the versions, since over time, these change, and they change faster than you can imagine. It is more important to know what the official kotlinx.coroutines repository is. GitHub is a reliable source to obtain information about the latest versions and news: https://github.com/Kotlin/kotlinx.coroutines.

In general terms, we can say that we need to add the coroutines dependency. We can find it in our build.gradle.kts file, and the configuration would be something like this:

dependencies {
testImplementation(kotlin("test"))
implementation("org.jetbrains.kotlinx:kotlinx-coroutines-core:1.8.0")
}

For coroutines, we add the one reference to kotlinx 1.8.0, which is enough for what we will do in this chapter. Now, in our first contact with coroutines, we are going to write Hello World. This will be our starting point for this entire chapter. This is the code:

import kotlinx.coroutines.delay
import kotlinx.coroutines.launch
import kotlinx.coroutines.runBlocking
fun main() = runBlocking {
    launch {
        delay(1000L)
        println("Kotlin World! ")
    }
    println("Hello")
}

The imports come from the kotlinx.coroutines package, which contains the main tools for working with coroutines in Kotlin:

• delay: A function that suspends the execution of a coroutine for a specific time (in milliseconds) without blocking the thread.

• launch: A function that starts a new coroutine in parallel with the existing execution.

• runBlocking: A function that runs a blocking coroutine, useful for simple examples or tests where we want to block the main thread until the inner coroutines finish.

The imports should be familiar. The first part that is noticeably different from basic “Hello World“ code is the following:

fun main() = runBlocking {

The runBlocking instruction is used here to allow main to run as a coroutine. It blocks the main thread until all coroutines within its block finish executing.

This ensures that the program does not terminate before the coroutines finish.

Let’s look at the launch instruction:

launch {
    delay(1000L)
    println("Kotlin World!")
}

launch creates a new coroutine. This coroutine runs concurrently with the rest of the code inside runBlocking.

Inside this coroutine, we have delay(1000L), which suspends the coroutine for 1 second (1,000 milliseconds) without blocking the thread. This means that the main thread is free to perform other tasks in the meantime, and after one second, it prints "Kotlin World!".

Finally, we have the last instruction:

println("Hello")

Since the coroutine does not block the thread, this message is printed before the coroutine finishes its task.

Summarizing the flow, we can describe it as follows:

1. The main function is executed and enters the runBlocking block.

2. Inside runBlocking, the following occurs:

   1. A coroutine is launched with launch.

   2. Execution continues without waiting for the coroutine to finish, so println("Hello") is executed immediately.

3. In the meantime, the coroutine executes delay(1000L), which suspends the coroutine for one second.

4. After the delay, it prints "Kotlin World!".

5. The program waits inside runBlocking until all coroutines inside its block finish. Only then does runBlocking complete and the program terminates.

After executing the code, we see the following as a result:

Hello
Kotlin World!

The reason we started with Hello World is simple – we are studying the simplest possible case to show how coroutines allow concurrent tasks (such as waiting a second) to be expressed without blocking the entire scope of execution. In this example, runBlocking blocks the main thread until all coroutines inside complete, but the coroutine launched with launch runs concurrently within that scope.

Compared to a traditional threading approach, this is more efficient and easier to read.

We introduced the concepts of delay and launch in a simple context. Now we will look at another example. We will simulate a restaurant scenario to demonstrate how coroutines handle asynchronous tasks.

We want to demonstrate how to perform multiple tasks simultaneously

We are going to show how not to block the main thread while waiting for long-running tasks. We will illustrate coordination between different asynchronous operations, something similar to what we mentioned about the waiter and the restaurant. This is the new code:

import kotlinx.coroutines.*
fun main() = runBlocking {
    println("Taking order...")
    // Coroutine 1: Preparing food in the kitchen
    val food = async { prepareFood() }
    // Coroutine 2: Serving drinks
    serveDrinks()
    // Wait for food to be ready and serve it
    println("Food is ready: ${food.await()}")
}
suspend fun prepareFood(): String {
    delay(3000) // Simulate food preparation
    return "Delicious meal"
}
fun serveDrinks() {
    println("Serving drinks to the table...")
}

The runBlocking block represents the main thread where primary actions (such as interacting with diners) occur. As mentioned before, the runBlocking instruction is a function that runs a block of code as a blocking coroutine. This means that the program will not end until all the code inside this block finishes. In this example, it represents the main thread, which acts as the waiter handling multiple tasks.

println("Taking order...")

The preceding line of code is printed immediately on the main thread (representing the waiter).

val food = async { prepareFood() }

async launches a new coroutine to execute the prepareFood() function concurrently.

async is a function that returns a Deferred object, which is like a “future” in Java. This object can be awaited or revisited later to get the results of the operation.

In this case, it represents the cook who starts preparing the food in the background.

Let us look at the prepareFood function:

suspend fun prepareFood(): String {
    delay(3000) // Simulate food preparation
    return "Delicious meal"
}

The prepareFood function is a suspended function, meaning that it can pause its execution and later resume exactly where it left off, without blocking the underlying thread while it is waiting. However, a suspend function can only be called from another suspend function or within a coroutine. It is also important to note that while the thread is not blocked, the coroutine that invokes prepareFood will remain suspended until the function resumes.

delay(3000) simulates a meal preparation time of three seconds. During this time, the coroutine is suspended, allowing the main thread to do other tasks.

After the delay, it returns Delicious meal.

While the food is being prepared (the async coroutine is suspended), the main thread continues to execute this function:

serveDrinks()

It prints the message Serving drinks to the table....

Here, the waiter (main thread) serves drinks while waiting for the cook to finish the food:

println("Food is ready: ${food.await()}")

food.await() pauses execution at this point until the prepareFood() coroutine finishes and returns its result.

Once prepareFood() returns Delicious meal, the main thread prints the following:

Food is ready: Delicious meal

This is how the flow goes:

• runBlocking starts the main block.

• The waiter takes the order and starts serving the drinks.

• Meanwhile, the cook (coroutine) prepares the food in the background.

• After three seconds, the food is ready and the waiter serves it.

Returning to our analogy, we’ve already seen the following:

• Waiter: This is the main thread that organizes tasks. It can serve drinks and handle other orders without getting stuck waiting for the food to be ready.

• Cook: This is a coroutine that works independently in the background, preparing the food.

• async and await: async starts the cook’s work and await makes sure the waiter waits for the result before serving the food.

• delay: This simulates the time it takes the cook to prepare the food.

This is what we see after running the program:

Taking order...
Serving drinks to the table...
Food is ready: Delicious meal

The main goal of this exercise was to demonstrate how coroutines allow you to do several things “at the same time” efficiently, just like a good waiter in a restaurant.

We are going to show another example that simulates loading book data from an API using coroutines in Kotlin. The repository (BookRepository) has a suspended function that simulates a two-second delay before returning the book data. In the BookScreen class, withContext(Dispatchers.IO) is used to execute the task in a context suitable for I/O operations, displaying success or error messages depending on the result. This demonstrates how to handle asynchronous tasks in an efficient and readable way. But we will build it in parts.

import kotlinx.coroutines.*
// Simulate a repository that fetches book data
class BookRepository {
    // Simulate an API call to fetch a book
    suspend fun fetchBookFromApi(bookId: String): Book {
        delay(2000) // Simulate network response time
        return Book(bookId, "Clean Code", "Robert C. Martin")
    }
}

BookRepository has a suspend function, fetchBookFromApi, which simulates calling an API to get the data for a book.

It uses delay(2000) to represent a two-second network delay, and then returns a Book object as the result.

We add this code, which defines the class that the fetchBookFromApi function returns:

data class Book(
    val id: String,
    val title: String,
    val author: String
)

Book is a data class that represents information about a book, with properties such as id, title, and author.

class BookScreen {
    private val repository = BookRepository()
    fun loadBookData() = runBlocking {
    try {
        println(" Loading book information...")
        // Send the request in a coroutine
        val book = withContext(Dispatchers.IO) {
            repository.fetchBookFromApi("9780132350884")
}

BookScreen defines the loadBookData method, which uses runBlocking to handle loading book data within a coroutine.

The loadBookData method uses withContext(Dispatchers.IO) to perform the book-fetching task in a context optimized for I/O operations.

Finally, we define main, which will allow us to execute the code:

fun main() {
    val screen = BookScreen()
    screen.loadBookData()
}

In the main function, you simply create an instance of BookScreen and call the loadBookData method to start the process.

Summarizing the program flow, this would be as follows:

• It prints “Loading book information...“

• It waits two seconds, simulating the API response.

• If everything goes well, it prints "Book loaded: Clean Code by Robert C. Martin".

• If an error occurs, it prints "Error loading book information: ..."

Let’s take a closer look at what’s happening in this exercise. We’ve successfully used runBlocking and withContext to run the task asynchronously and on a separate thread (Dispatchers.IO), which is ideal for I/O tasks such as API calls.

This approach is ideal for I/O tasks such as API calls because it allows us to perform these operations without blocking the main thread.

By offloading the work to a different dispatcher, we ensure that time-consuming operations (such as reading from a database, writing to a file, or calling an external API) don’t freeze the UI or other critical processes. In Kotlin coroutines, Dispatchers.IO is specifically optimized for these types of tasks, giving us efficient and responsive code even when dealing with slow or unpredictable external systems.

We have included a try-catch block to catch possible exceptions that may occur during the API call, which is a good practice to make your code more robust.

Simulating API response time with a delay is useful to understand the behavior of coroutines in real-world scenarios.

In this section, we introduced the foundational ideas behind coroutines in Kotlin, focusing on how they enable writing asynchronous, non-blocking code that remains readable and efficient. We illustrated real-world analogies and scenarios to highlight how coroutines support structured concurrency and safe context switching, making them a powerful tool for managing concurrent tasks in modern applications.

It’s important to clarify when to use these different coroutine builders:

• runBlocking is typically used to bridge blocking and non-blocking code, for example, in tests or the main function of a small program. It blocks the current thread until its coroutine body finishes executing.

• In contrast, using coroutine builders such as launch or async within an existing coroutine scope, combined with context switching (withContext), lets you run asynchronous code without blocking any thread, ensuring better performance and responsiveness.

It is important to note, however, that coroutines are not magical. If you call blocking operations inside a suspend function – for example, using Thread.sleep() or synchronous I/O calls such as JDBC – you will still block the underlying thread. To fully benefit from coroutines, blocking calls should be replaced with non-blocking equivalents whenever possible.

So, while runBlocking is useful in specific entry points, using structured concurrency with launch, async, and withContext is preferred in production code to handle tasks concurrently and safely without halting your program’s execution.

Now that we have a solid understanding of how to implement and use coroutines effectively, the next section will introduce timeouts and cancellations. You’ll learn how to gracefully handle scenarios where tasks take too long or need to be interrupted, ensuring your applications remain responsive and efficient under various conditions. Let’s continue exploring how Kotlin coroutines can simplify even the most complex asynchronous workflows!

Timeouts and cancellations

Now we will see two important elements that are good to know about in coroutines: timeouts and cancellations. Timeouts are time limits that you set for an operation. If the operation takes longer than the specified time, it is automatically cancelled.

A simple way to describe them using examples of daily activities would be as follows:

• Ordering a pizza and getting the notice: “If it doesn’t arrive in 30 minutes, we cancel the order and refund you”.

• When an ATM cancels the operation if we don’t respond within a certain time.

Cancellations, as their name suggests, allow us to stop a coroutine in the middle of its execution. Let’s look at a couple of examples:

• Canceling a download in progress.

• Canceling a bank transfer before it is completed.

Thinking of timeouts and cancellations as elements that we can use, we could easily give some situations where we could use them:

• API calls that should not take too long.

• File downloads that can be cancelled.

• Database operations with a time limit.

• Background processes that can be cancelled by the user.

Taking our previous example of books as a basis, we are now going to modify it to have both cancellation and timeouts.

The definitions of BookRepository and Book remain the same; we just add more time to Delay. Before, we had 2000, but we will change the value to 3000. We increased the time period to later; we set a time limit of two seconds and allowed three seconds to execute it (the timeout):

class BookRepository {
    // Simulates an API call to get a book
    suspend fun fetchBookFromApi(bookId: String): Book {
        delay(3000) // Simulates network response time
        return Book(bookId, "Clean Code", "Robert C. Martin")
    }
}
data class Book(
    val id: String,
    val title: String,
    val author: String
)

Now we will change launch to make it look like this:

val job = launch {
    try {
        println(" Loading book information...")
        // Set a timeout for the operation
        val book = withTimeout(2000) { // 2 second timeout
            repository.fetchBookFromApi("9780132350884")
        }
        println(" Book loaded: ${book.title} by ${book.author}")
    } catch (e: TimeoutCancellationException) {
        println(" Timeout exceeded when loading book.")
    } catch (e: Exception) {
        println(" Error loading book information: ${e.message}")
    }
}

We set a time limit of 2,000 milliseconds to complete the fetchBookFromApi operation. If the task does not finish within this time, TimeoutCancellationException is thrown. This mechanism is essential in applications where it is critical to avoid long blockages and ensure fast responses.

In this statement, we can see that the approach uses job, as declared with launch, to handle the execution of the coroutine:

val job = launch {

This allows the job to be explicitly cancelled if necessary, providing flexibility to stop execution in specific situations. For example, the job could be cancelled with a call to job.cancelAndJoin() if the task is detected to be no longer relevant or if an external event occurs.

Exception handling is a key part of this approach:

} catch (e: TimeoutCancellationException) {
println(" Timeout exceeded when loading book.")

These last lines demonstrate how to handle scenarios where the time limit is exceeded, ensuring that the system can respond appropriately without crashing. Other exceptions are also handled to catch generic errors during execution.

While this approach is powerful, it adds complexity to the code. It requires explicitly handling both the job and exceptions, which may be unnecessary for simple tasks. However, in cases where strict time limits or precise cancellations are needed, this structure proves invaluable.

Outside of loadBookData, we will also modify the code in the following way:

// Change: Cancel the job after 4 seconds
delay(4000)
println(" Canceling the loading of the book...")
job.cancelAndJoin() // Cancel the job and wait for it to finish
println("Task cancelled.")

Notice that we have introduced a delay of 4,000 milliseconds and manual cancellation. Also notice that we use a job to cancel the task explicitly with job.cancelAndJoin().

The cancellation and timeout scenario introduces a more controlled approach to handling asynchronous tasks in Kotlin. Using launch and withTimeout, a time limit is set for the execution of an operation.

This ensures that the task does not exceed the time we decide, throwing an exception (TimeoutCancellationException) if the time runs out. Also, by encapsulating the logic in a job, the task can be cancelled explicitly when necessary, optimizing resources and allowing better management of multiple concurrent tasks.

This example we developed includes robust exception handling to differentiate between a timeout and other errors. In summary, this model is ideal for applications where time limits and cancellation of irrelevant or delayed tasks are required, although it adds a bit more complexity to the code.

This section emphasized the importance of timeouts and cancellations in effective coroutine management. By enforcing time limits and enabling controlled interruption of tasks, Kotlin coroutines provide mechanisms to keep applications responsive and resilient, especially when dealing with potentially long-running operations.

We also introduced the concept of cancellations, allowing tasks to be stopped explicitly when they are no longer needed. By leveraging the job object, we illustrated how to manage and cancel ongoing tasks using job.cancelAndJoin(), ensuring efficient resource management. Additionally, we incorporated robust exception handling to distinguish between timeout-specific issues (TimeoutCancellationException) and other errors, adding resilience to our code.

By combining these techniques, we showed how to handle asynchronous tasks in a controlled and efficient manner, enabling developers to create responsive applications that gracefully handle delays and interruptions. This approach is particularly useful in scenarios involving API calls, file downloads, or user-cancelled operations, providing flexibility and reliability.

Next, we’ll explore contexts and scopes, diving deeper into how coroutines manage their execution environments and life cycle. You’ll learn about coroutine dispatchers, structured concurrency, and how to properly manage coroutine life cycles to maintain clean, predictable, and efficient code. Let’s continue building our understanding of Kotlin coroutines with these advanced concepts!

Contexts and scope

Context and scope are fundamental concepts in the world of Kotlin coroutines and play a crucial role in managing their life cycle and behavior.

A context in Kotlin coroutines is like a container that groups key information about how and where a coroutine will run. This context includes the following:

• Dispatcher: Defines which thread or thread group the coroutine will run on, as follows:

  • Dispatchers.IO for I/O operations (such as working with files or network calls).

  • Dispatchers.Main for updating the UI.

• Job: Represents the specific task the coroutine performs. It also allows you to cancel it or monitor its progress.

• Other attributes: Can include things such as the following:

  • CoroutineName: A useful identifier to distinguish coroutines.

  • NonCancellable: Indicates that the coroutine should not be cancelled, even if its parent or associated scope is.

In addition, you can put custom elements into the coroutine context, such as an Mapped Diagnostic Context (MDC) for logging. It is also important to remember that thread-local variables are not effective with coroutines, since a coroutine may resume on a different thread. If you need thread-local-like behavior, you should use ThreadContextElement to propagate context across suspensions.

In short, a coroutine’s context sets up its execution environment: it defines where it runs, how it interacts with other processes, and what rules it should follow while working. It’s like giving the coroutine a roadmap and tools to get it to perform its task correctly.

The code for BookRepository and Book remains the same; we don’t have to make any changes.

The changes start with the BookScreen code. We add the scope as follows:

class BookScreen {
    private val repository = BookRepository()
    private val scope = CoroutineScope(Dispatchers.Main + Job())

CoroutineScope is used, which automatically encapsulates the context and the associated job. With this change, instead of the previously temporary scope used with runBlocking, now there is a permanent scope with CoroutineScope(Dispatchers.Main + Job()).

This makes it easier to manage related tasks, allowing all coroutines associated with the scope to be cancelled with a single call (scope.cancel()).

We need better organization for larger applications or those with multiple coroutines.

Let’s make some changes in the structure:

fun loadBookDetails(bookId: String) {
    scope.launch {
    withContext(Dispatchers.IO) {
        // code here
        }
    }
}

We notice with these changes that a specific scope is defined at the class level and dispatchers are explicitly specified. In addition, we are now handling specific execution contexts, and we are going to allow global cancellation of all coroutines.

Regarding the execution context, we can also say that with the previous code, dispatchers were not specified, and now we are indicating one, dispatchers.IO, for input and output operations.

We also need to change val book. It should look like this:

val book = withTimeout(3000) {
    // Set a timeout of 3 seconds
    withContext(Dispatchers.IO) {
        // Run the task in the I/O Dispatcher
        repository.fetchBookFromApi(bookId)
    }
}

We add this function(cancelAllTasks) after loadBookDetails:

fun cancelAllTasks() {
    scope.cancel() // Cancels all coroutines in the Scope
    println("All tasks were cancelled.")
}

We centralize canceling all active coroutines of the scope with scope.cancel() in the cancelAllTasks() method.

The main code part now looks like this:

fun main() {
    val screen = BookScreen()
    // Start loading book details
    screen.loadBookDetails("1")
    // Simulate task cancellation after 5 seconds
    runBlocking {
        delay(5000)
        screen.cancelAllTasks()
    }
}

With the loadBookDetails instruction, we will start loading the book details to display them. With the second part, runBlocking, we would be simulating the cancellation of tasks after five seconds.

So, let’s get to the differences:

• Life cycle management:

  • runBlocking blocks the main thread until all the internal coroutines complete. It’s useful for testing or small examples, but it’s not ideal for real applications because it freezes the main thread while it waits. The problem is that the thread is blocked, which affects performance if there are more concurrent tasks.

  • Using CoroutineScope acts as a container that allows you to manage all the coroutines created within it. This is ideal for real-world applications because it allows you to manage the life cycle of multiple coroutines in a centralized manner.

  • We can see that in the code, we can now cancel all related coroutines at once using scope.cancel(). This is useful in situations where, for example, a user leaves a screen in a mobile app.

• Dispatchers:

  • Before the changes, no dispatcher was specified. All code runs in the default context of runBlocking. This meant that tasks such as fetchBookFromApi run on the same thread as runBlocking. This can be problematic if you are doing intensive tasks, such as API calls or database access, because they might block the main thread.

  • By implementing Dispatchers.IO, we take advantage of the fact that it is designed for I/O tasks, such as network calls. This ensures that these tasks are executed in a thread optimized for operations of this type, without blocking the main thread.

  • So, we clearly separate the network tasks (in Dispatchers.IO) from the UI-related tasks (in Dispatchers.Main).

• Cancellation of coroutines:

  • Previously, we cancelled manually with job.cancelAndJoin(). This stops the specific coroutine and waits for all resources to be released. This is effective but requires you to control each job individually. Switching to scope.cancel() stops all coroutines associated with CoroutineScope. This is cleaner and more scalable, especially if you have multiple related tasks.

• Separation of contexts:

  • Initially, there was no explicit separation of contexts. All operations, such as API calls and result handling, are executed in the same thread. With the changes, the contexts were separated clearly, as we can see:

    withContext(Dispatchers.IO) { repository.fetchBookFromApi(bookId)
    }

Networking code (API call) runs in Dispatchers.IO, while the rest of the code (e.g., printing to the console) can run in Dispatchers.Main. This improves performance and keeps the UI smooth.

In summary, the final code represents a more mature and professional implementation of coroutine handling in Kotlin, where each operation is executed in the most appropriate context: the UI in the main dispatcher and network operations in the I/O dispatcher. This approach not only optimizes the use of system resources but also provides a clearer and more maintainable structure, where responsibilities are well defined and separated. Operation cancellation is handled more robustly through a shared scope, allowing for efficient coroutine life cycle management.

Furthermore, this implementation pattern better reflects real-world practices in app development, especially on Android, where it is crucial to properly handle execution contexts and component life cycles. By using CoroutineScope defined at the class level along with specific dispatchers, you achieve more predictable and maintainable code, which can scale better as your app grows and becomes more complex.

Summary

In this chapter, we delved into Kotlin coroutines, one of the most transformative features of the language for handling asynchronous programming. Here is a summary of what we’ve learned.

We started by demystifying coroutines through relatable analogies, such as a waiter in a restaurant. This helped visualize how coroutines enable concurrent, non-blocking tasks while keeping the main thread free for interactive operations.

We introduced coroutine builders such as launch and runBlocking, along with suspending functions such as delay. Through practical examples such as “Hello World“ and the restaurant scenario, we showcased how to create and manage coroutines effectively.

We explored the critical concepts of setting time limits (withTimeout) and manually canceling tasks (job.cancelAndJoin). These mechanisms ensure efficient resource usage and help handle long-running tasks gracefully, as seen in the book-fetching example.

Finally, we covered how CoroutineScope and dispatchers are used to manage execution contexts and life cycles. We demonstrated how separating contexts (e.g., Dispatchers.IO for network calls and Dispatchers.Main for UI updates) improves application performance and maintainability. We also highlighted the importance of structured concurrency, enabling efficient cancellation and life cycle management for related tasks.

This chapter provided a robust foundation for understanding and applying Kotlin coroutines, a game-changing feature that simplifies asynchronous programming. By transitioning from basic examples to real-world scenarios, you now understand how coroutines enhance concurrency, improve code readability, and make resource management more efficient. These concepts are critical for developing scalable and responsive applications, whether for Android, server-side, or other Kotlin-based projects.

If you want to dig deeper into the mechanics of moving from Java to Kotlin—writing idiomatic Kotlin, handling null safety, using coroutines for concurrency, and taking advantage of features like extension functions and DSLs—check out Kotlin for Java Developers by José Dimas Luján Castillo and Ron Veen (Packt, Oct 2025). Written for experienced Java developers, it teaches Kotlin by mapping concepts directly to familiar Java constructs and then goes further into interoperability, generics, data and sealed classes, coroutines and flows, and DSL design—across backend, Android, and cross-platform development.

With thanks to AI Engineer Gerald Parker for his technical review.

Eino ADK, pronounced “I know”, is a multi-agent development framework designed for Go, developed and hardened in real-world use at ByteDance. Its design philosophy is “keep simple things simple, and make complex things possible”. Open-sourced at the start of 2025, Eino’s promise to Go developers is they can focus on implementing business logic without worrying about underlying technical complexity.

In this article co-written with the team behind Eino, we will discuss:

• What Eino ADK is

• Core agent patterns in Eino along with real use cases

• Example code for building a simple project manager agent

Introduction to Eino ADK

Agents are quickly becoming the mainstream way to deploy LLMs, from intelligent customer service to automated office work. With them, the following pain points are emerging:

• LLMs are not bridged well with business systems, resulting in agents that can only engage in “empty talk.”

• Lack of state management causes agents to frequently “forget” when performing tasks.

• Complex interactive processes increase development difficulty even further.

Eino ADK was created to provide Go developers with a complete, flexible, and powerful agent development framework that addresses these core challenges head-on.

Recap: What is an Agent?

You can think of an agent as an independent, intelligent entity that can understand instructions, perform tasks, and provide responses – capable of autonomous learning, adaptation, and decision-making. Its main functions include:

• Reasoning: An agent can analyze data, identify patterns, and use logic and available information to draw conclusions, make inferences, and solve problems.

• Action: An agent takes actions or executes tasks to achieve goals based on decisions, plans, or external inputs.

• Observation: An agent autonomously collects relevant information (for example, through computer vision, natural language processing, or sensor data analysis) to understand the context and lay the foundation for informed decision-making.

• Planning: An agent can determine necessary steps, evaluate potential actions, and select the best course of action based on available information and expected outcomes.

• Collaboration: An agent can effectively work together with others (human or other agents) in complex and dynamic environments.

Any scenario that requires interaction with a LLM can be abstracted as an agent. For example:

• An agent for querying weather information

• An agent for booking meetings

• An agent capable of answering questions in a specific domain

What is Eino ADK?

Eino ADK is a complete system for developing intelligent agents. It breaks down complex AI applications into independent, composable intelligent agent units, allowing developers to build agentic systems the way they would assemble Lego blocks.

Its features include:

• Less glue code: Unified interfaces and event streams make complex task decomposition more natural.

• Rapid orchestration: Pre-built paradigms + workflows allow you to build pipelines in minutes.

• More controllable: Interruptible, resumable, and auditable, making the agent collaboration process “visible.”

Eino achieves this through unified abstract interfaces, flexible composition patterns, and powerful collaboration mechanisms.

The Core: Unified Agent Abstraction

The core of ADK is a concise and powerful agent interface:

type Agent interface {
    Name(ctx context.Context) string
    Description(ctx context.Context) string
    Run(ctx context.Context, input *AgentInput, options ...AgentRunOption) *AsyncIterator[*AgentEvent]
}

Each agent has:

• A clear identity (Name)

• A clear responsibility (Description)

• A standardized execution method (Run)

This provides a basis for discovery and invocation between agents. Simple Q&A robots or complex multi-step task processing systems can be implemented through this unified interface.

Core Agent Patterns

The following sections lay out the core patterns that make up Eino ADK. Each will be introduced, explained, and then brief skeleton code demonstrated.

ChatModelAgent: The Brain

ChatModelAgent is the most important pre-built component in Eino ADK. It encapsulates the interaction logic with LLMs and implements the classic ReAct (Reason-Act-Observe) pattern. The process is as follows:

1. Reason: Call the LLM.

2. Act: The LLM returns a tool call request.

3. Act: ChatModelAgent executes the tool.

4. Observe: The tool’s result is returned to the LLM, which continues to generate based on the previous context until the model determines that no more tool calls are needed.

The core of the ReAct pattern is a closed loop of Reason → Act → Observe → Reason Again, which addresses the pain points of traditional agents that either act blindly or have a disconnect between reasoning and action.

Here are some possible practical scenarios:

• Generating a market analysis report:

  • Reason-1: To assess the potential of a sector, information on policy support, industry growth rate, profitability of leading companies, and supply chain bottlenecks is needed.

  • Action-1: Call an API to get overall financial data for the industry.

  • Observe-1: Analyze the data.

  • Reason-2: Determine that the industry has high growth and policy endorsement, but rising upstream prices may squeeze downstream profits. Decide that further verification is needed.

  • Act-2: Call an API to get detailed data on supply and demand, industry research reports, and so on.

  • Observe-2: Analyze the data.

  • Reason-3: Integrate the conclusions to generate an analysis report with key data sources.

Using the ReAct pattern avoids information overload caused by collecting all the information at once. It focuses on core issues through gradual reasoning and uses data to verify thoughts rather than relying on intuition. The process is also explainable, improving the accuracy of the generated report.

• IT fault operations and maintenance:

  • Reason-1: Identify common causes of failures, such as “CPU overload, insufficient memory, full disk, or service crash.” Basic monitoring data needs to be checked first.

  • Act-1: Call the “Monitoring System API” to query server metrics.

  • Observe-1: Analyze the server metrics.

  • Reason-2: Determine the main cause. For example, if CPU utilization is abnormal, investigate which processes have high CPU usage.

  • Act-2: Use a “Process Management Tool” to check the top processes for any abnormal services.

  • Observe-2: Discover that the logging service is abnormal, possibly due to “log file too large” or “configuration error.” Verify these conditions before planning the next step.

  • Reason-3: Check the logging service’s configuration and log file size.

  • Act-3: Execute a bash command.

  • Observe-4: Find that the log file is too large, and the configuration does not enable rotation or set a maximum log size.

  • Reason-4: Provide a feasible solution to the O&M engineer: clean up the logs, modify the configuration to enable rotation, and restart the logging service and application.

The ReAct pattern gradually narrows down the problem scope, avoiding blind operations. Each step is well-founded, making it easy for O&M engineers to perform secondary verification before implementing solutions, and providing a basis for subsequent reviews and preventative measures.

ChatModelAgent leverages the capabilities of LLMs for reasoning, understanding natural language, making decisions, generating responses, and interacting with tools, acting as the “thinking” part of an intelligent agent application.

Here is skeleton code for a ChatModelAgent with ReAct capabilities:

import "github.com/cloudwego/eino/adk"

// Create a ReAct ChatModelAgent with multiple tools
chatAgent := adk.NewChatModelAgent(ctx, &adk.ChatModelAgentConfig{
    Name: "intelligent_assistant",
    Description: "An intelligent assistant capable of using multiple tools to solve complex problems",
    Instruction: "You are a professional assistant who can use the tools provided to help users solve problems",
    Model: openaiModel,
    ToolsConfig: adk.ToolsConfig{
        Tools: []tool.BaseTool{
            searchTool,
            calculatorTool,
            weatherTool,
        },
    },
})

WorkflowAgents: The Sophisticated Pipeline

Eino ADK provides the WorkflowAgents pattern, designed to coordinate the execution flow of sub-agents. It manages the running of agents through predefined logic to produce a deterministic execution process, helping to achieve a predictable and controllable output.

You can arrange and combine the following patterns as needed:

• SequentialAgent

• ParallelAgent

• LoopAgent

And combine them with ChatModelAgent to construct a complete workflow pipeline. Here are descriptions of each agent in detail:

• SequentialAgent: Executes the registered agents in the configuration in sequence once, and then ends. The operation follows these principles:

  • Linear execution: Strictly executes in the order of the SubAgents array.

  • Passing of run results: Each agent in the configuration can get the complete input of the SequentialAgent and the output of the preceding agent.

  • Support for early exit: If any sub-agent produces an exit/interrupt action, the entire Sequential process will be terminated immediately.

• Possible practical scenarios include:

  • Data ETL: ExtractAgent (extracts order data from MySQL) → TransformAgent (cleans null values, formats dates) → LoadAgent (loads into a data warehouse)

  • CI/CD pipeline: CodeCloneAgent (pulls code from a code repository) → UnitTestAgent (runs unit tests, returns an error and analysis report when a test case fails) → CompileAgent (compiles the code) → DeployAgent (deploys to the target environment)

Skeleton code for a SequentialAgent:

import "github.com/cloudwego/eino/adk"

// Execute in sequence: create a research plan -> search for materials -> write a report
sequential := adk.NewSequentialAgent(ctx, &adk.SequentialAgentConfig{
    Name: "research_pipeline",
    SubAgents: []adk.Agent{
        planAgent,   // Create a research plan
        searchAgent, // Search for materials
        writeAgent,  // Write a report
    },
})

• ParallelAgent: Executes the registered agents in the configuration concurrently, and ends after all agents have finished executing. The operation follows these principles:

  • Concurrent execution: All sub-agents start at the same time and execute in parallel in separate goroutines.

  • Shared input: All sub-agents receive the same initial input as when the ParallelAgent was called.

  • Waiting and result aggregation: Internally uses sync.WaitGroup to wait for all sub-agents to complete, collects the execution results of all sub-agents, and outputs them to the AsyncIterator in the order they are received.

• Possible practical scenarios include:

  • Multi-source data collection: MySQLCollector (collects user table) + PostgreSQLCollector (collects order table) + MongoDBCollector (collects product reviews)

  • Multi-channel push: SocialPushAgent (pushes to social media account) + SMSPushAgent (sends SMS) + AppPushAgent (pushes to app)

Skeleton code for a ParallelAgent:

import "github.com/cloudwego/eino/adk"

// Concurrently execute sentiment analysis + keyword extraction + content summarization
parallel := adk.NewParallelAgent(ctx, &adk.ParallelAgentConfig{
    Name: "multi_analysis",
    SubAgents: []adk.Agent{
        sentimentAgent, // Sentiment analysis
        keywordAgent,   // Keyword extraction
        summaryAgent,   // Content summarization
    },
})

• LoopAgent: Executes the registered agents in the configuration in sequence and loops multiple times. The operation follows these principles:

  • Loop execution: Repeatedly executes the SubAgents sequence, with each loop being a complete Sequential execution process.

  • Accumulation of run results: The results of each iteration are accumulated, and the input of subsequent iterations can access all historical information.

  • Conditional exit: Supports terminating the loop by outputting an event containing an ExitAction or reaching the maximum number of iterations. Configuring MaxIterations=0 means an infinite loop.

• Possible practical scenarios include:

  • Data Synchronization: CheckUpdateAgent (checks for incremental changes in the source database) → IncrementalSyncAgent (synchronizes incremental data) → VerifySyncAgent (verifies consistency)

  • Pressure Testing: StartClientAgent (starts a test client) → SendRequestsAgent (sends requests) → CollectMetricsAgent (collects performance metrics)

Skeleton code for a LoopAgent:

import "github.com/cloudwego/eino/adk"

// Loop 5 times, with the sequence each time being: analyze the current state -> propose an improvement plan -> verify the improvement effect
loop := adk.NewLoopAgent(ctx, &adk.LoopAgentConfig{
    Name: "iterative_optimization",
    SubAgents: []adk.Agent{
        analyzeAgent,   // Analyze the current state
        improveAgent,   // Propose an improvement plan
        validateAgent,  // Verify the improvement effect
    },
    MaxIterations: 5,
})

Pre-built Multi-Agent Paradigms

Eino ADK provides three pre-built multi-agent paradigms:

• The Supervisor pattern for centralized coordination

• The Plan-Execute pattern for structured problem solving

• The DeepAgents pattern for complex workflows that require multi-step, multi-role collaboration

Developers can use them out of the box without having to design collaboration logic from scratch. The following sections lay out the patterns along with skeleton code.

Supervisor Pattern

The Supervisor agent is a centralized multi-agent collaboration pattern, designed to provide a solution for the general scenario of centralized decision-making and distributed execution.

It consists of a Supervisor agent and multiple sub-agents, where:

• The Supervisor agent is responsible for task allocation, summarizing the results after the sub-agents are completed, and making the next decision.

• The sub-agents focus on executing specific tasks and automatically return control of the task to the Supervisor after completion.

The Supervisor pattern has the following characteristics:

• Centralized control: The Supervisor uniformly manages the sub-agents and can dynamically adjust task allocation based on the input and the execution results of the sub-agents.

• Deterministic callback: After a sub-agent has finished executing, it will return the running result to the Supervisor agent, avoiding interruption of the collaboration process.

• Loosely coupled extension: Sub-agents can be developed, tested, and replaced independently, which is useful for expansion and maintenance.

This hierarchical structure of the Supervisor pattern is suitable for scenarios that dynamically coordinate multiple specialized agents to complete complex tasks, such as:

• Research project management: The Supervisor assigns research, experimentation, and report writing tasks to different sub-agents.

• Customer service processes: The Supervisor assigns tasks to technical support, after-sales, and sales sub-agents, based on the type of user problem.

Skeleton code:

import "github.com/cloudwego/eino/adk/prebuilt/supervisor"

// Research project management: create a multi-agent in supervisor mode
// Contains three sub-agents: research, experimentation, and report
supervisor, err := supervisor.New(ctx, &supervisor.Config{
    SupervisorAgent: supervisorAgent,
    SubAgents: []adk.Agent{
        researchAgent,
        experimentationAgent,
        reportAgent,
    },
})

Plan-Execute-Replan Pattern

The Plan-Execute-Replan agent is a multi-agent collaboration pattern based on the “plan-execute-reflect” paradigm (refer to the 2023 paper Plan-and-Solve Prompting by Wang L. et al). Through the collaborative work of three core intelligent agents, it achieves structured planning of tasks, tool call execution, progress evaluation, and dynamic replanning:

• Planner: Generates a structured initial task plan with detailed steps based on the user’s goals.

• Executor: Executes the first step in the current plan.

• Replanner: Evaluates the execution progress and decides whether to revise the plan and continue to have the Executor run it, or to end the task.

The Plan-Execute-Replan pattern has the following characteristics:

• Clear hierarchical architecture: By breaking down the task into three stages –planning, execution, and reflection/replanning – a hierarchical cognitive process is formed, which embodies the closed-loop cognitive strategy of “think before you act, then adjust based on feedback.”

• Dynamic iterative optimization: The Replanner determines in real time whether the task is complete or needs to be adjusted based on the execution results and current progress, and supports dynamic replanning. This mechanism effectively solves the bottleneck of traditional single-planning – where there is difficulty coping with environmental changes and task uncertainty – and improves the robustness and flexibility of the system.

• Clear responsibilities and loose coupling: The Plan-Execute-Replan pattern consists of multiple intelligent agents working together, and supports independent development, testing, and replacement. The modular design is useful for expansion and maintenance and conforms to engineering best practices.

• Good extensibility: It does not rely on a specific language model, tool, or agent, which is convenient for integrating diverse external resources to meet the needs of different application scenarios.

The “plan → execute → replan” closed-loop structure of the pattern is suitable for complex task scenarios that require multi-step reasoning, dynamic adjustment, and tool integration, such as:

• Complex research analysis: Decompose research questions through planning, perform multiple rounds of data retrieval and calculation, and dynamically adjust research directions and hypotheses to improve the depth and accuracy of the analysis.

• Automated workflow management: Decompose complex business processes into structured steps, combine them with a variety of tools (such as database queries, API calls, and compute services) to execute them step-by-step, and dynamically optimize the process based on execution results.

• Multi-step problem-solving: Scenarios that require step-by-step reasoning and multi-tool collaboration, such as legal consultation, technical diagnosis, and strategy formulation, to ensure that each step of execution has feedback and adjustment.

• Intelligent assistant task execution: Supports intelligent assistants to plan task steps based on user goals, call external tools to complete specific operations, and adjust subsequent plans based on replanning and thinking combined with user feedback, improving the completeness and accuracy of task completion.

Skeleton code:

import "github.com/cloudwego/eino/adk/prebuilt/planexecute"

// A research assistant in Plan-Execute mode
researchAssistant := planexecute.New(ctx, &planexecute.Config{
    Planner: adk.NewChatModelAgent(ctx, &adk.ChatModelAgentConfig{
        Name:        "research_planner",
        Instruction: "Create a detailed research plan, including literature research, data collection, analysis methods, etc.",
        Model:       gpt4Model,
    }),
    Executor: adk.NewChatModelAgent(ctx, &adk.ChatModelAgentConfig{
        Name: "research_executor",
        ToolsConfig: adk.ToolsConfig{
            Tools: []tool.BaseTool{
                scholarSearchTool,
                dataAnalysisTool,
                citationTool,
            },
        },
    }),
    Replanner: replannerAgent,
})

DeepAgents Pattern

DeepAgents is a multi-agent pattern unified under the coordination of a MainAgent. The MainAgent leverages a ChatModel with tool-calling capabilities to operate through a ReAct workflow:

• It decomposes user goals into structured to-do items and records progress through WriteTodos.

• It selects and invokes corresponding sub-agents to execute subtasks through the unified TaskTool interface; main/sub-agent contexts are isolated to prevent intermediate steps from contaminating the main workflow.

• It aggregates results returned by various sub-agents; when necessary, it calls WriteTodos again to update progress or perform replanning until completion.

The characteristics of the DeepAgents pattern include:

• Enhanced task decomposition and progress management: WriteTodos creates clear subtasks and milestones, making complex goals decomposable and trackable.

• More robust context isolation: Sub-agents execute in “clean” contexts while the main agent only aggregates results, reducing interference from redundant reasoning chains and tool call traces on the main workflow.

• Unified delegation interface with easy extensibility: TaskTool abstracts all sub-agents and tool capabilities into a unified calling surface, facilitating the addition or replacement of specialized sub-agents.

• Flexible closed-loop of planning and execution: Planning functions as a tool that can be called on demand; for simple tasks, unnecessary planning can be skipped, reducing LLM call costs and time consumption.

• Boundaries and trade-offs: Over-decomposition increases call frequency and costs; it places higher demands on subtask division and prompt optimization, requiring models to possess stable tool-calling and planning capabilities.

The core value of DeepAgent lies in automatically handling complex workflows that require multi-step, multi-role collaboration. It’s not just an executor of single functions but a “project manager” with deep thinking, planning, and dynamic adjustment capabilities.

Suitable application scenarios include:

• Complex business processes with multi-role collaboration: Centered around R&D, testing, release, legal, and operations roles, with centralized delegation of subtasks and unified aggregation; each stage sets gateways and rollback strategies, with visible progress and retry capabilities.

• Phased management of long workflows: Planning decomposes steps such as data cleaning, validation, lineage analysis, and quality inspection; sub-agents run in isolated contexts; when exceptions occur, only relevant stages are re-run, with unified reconciliation and aggregation of outputs.

• Execution environments requiring strict context isolation: A unified interface collects materials and requests, with TaskTool routing subtasks such as legal, risk control, and finance separately; boundaries between subtasks are clearly defined and mutually invisible, with auditable progress and traceability, and failures can be retried without affecting other stages.

Skeleton code:

import "github.com/cloudwego/eino/adk/prebuilt/deep"

agent, err := deep.New(ctx, &deep.Config{
    Name:       "deep-agent",
    ChatModel:  gpt4Model,
    SubAgents: []adk.Agent{
        LegalAgent,
        RiskControlAgent,
        FinanceAgent,
    },
    MaxIteration: 100,
})

Example: Project Manager Intelligent Agent

Now you have covered the basics, it’s time to create an agent. Install the latest version of Eino using the following command:

go get github.com/cloudwego/eino@latest

In this example, you’ll see how to build a Project Manager intelligent agent for a variety of scenarios. It uses the Supervisor pattern, and the functions of each agent are as follows:

• ResearchAgent: Responsible for researching and generating feasible solutions, supporting interruption and receiving additional context information from users to improve the accuracy of research plan generation.

• CodeAgent: Uses knowledge base tools to recall relevant knowledge as a reference to generate high-quality code.

• ReviewAgent: Uses a sequential workflow to orchestrate three steps: problem analysis, evaluation generation, and evaluation verification to review research results/coding results and provide reasonable evaluations so project managers can make decisions.

• ProjectManagerAgent: Routes and coordinates multiple sub-intelligent agents responsible for different dimensions of work based on dynamic user input.

The design of this example covers most of the concepts introduced in the article.

The core code is as follows. The complete code can be found in the source code provided in the eino-examples project:

func main() {
    ctx := context.Background()

    // Init chat model for agents
    tcm, err := openai.NewChatModel(ctx, &openai.ChatModelConfig{
        APIKey:  os.Getenv("OPENAI_API_KEY"),
        Model:   os.Getenv("OPENAI_MODEL"),
        BaseURL: os.Getenv("OPENAI_BASE_URL"),
        ByAzure: func() bool {
            return os.Getenv("OPENAI_BY_AZURE") == "true"
        }(),
    })
    if err != nil {
        log.Fatal(err)
    }

    // Init research agent
    researchAgent, err := agents.NewResearchAgent(ctx, tcm)
    if err != nil {
        log.Fatal(err)
    }

    // Init code agent
    codeAgent, err := agents.NewCodeAgent(ctx, tcm)
    if err != nil {
        log.Fatal(err)
    }

    // Init technical agent
    reviewAgent, err := agents.NewReviewAgent(ctx, tcm)
    if err != nil {
        log.Fatal(err)
    }

    // Init project manager agent
    s, err := agents.NewProjectManagerAgent(ctx, tcm)
    if err != nil {
        log.Fatal(err)
    }

    // Combine agents into ADK supervisor pattern
    // Supervisor: project manager
    // Sub-agents: researcher / coder / reviewer
    supervisorAgent, err := supervisor.New(ctx, &supervisor.Config{
        Supervisor: s,
        SubAgents:  []adk.Agent{researchAgent, codeAgent, reviewAgent},
    })
    if err != nil {
        log.Fatal(err)
    }

    // Init Agent runner
    runner := adk.NewRunner(ctx, adk.RunnerConfig{
        Agent:           supervisorAgent,
        EnableStreaming: true,               // enable stream output
        CheckPointStore: newInMemoryStore(),  // enable checkpoint for interrupt & resume
    })

    // Replace it with your own query
    query := "please generate a simple ai chat project with python."
    checkpointID := "1"

    // Start runner with a new checkpoint id
    iter := runner.Query(ctx, query, adk.WithCheckPointID(checkpointID))

    interrupted := false
    for {
        event, ok := iter.Next()
        if !ok {
            break
        }

        if event.Err != nil {
            log.Fatal(event.Err)
        }

        if event.Action != nil && event.Action.Interrupted != nil {
            interrupted = true
        }

        prints.Event(event)
    }

    if !interrupted {
        return
    }

    // interrupt and ask for additional user context
    scanner := bufio.NewScanner(os.Stdin)
    fmt.Print("\ninput additional context for web search: ")
    scanner.Scan()
    fmt.Println()
    nInput := scanner.Text()

    // Resume by checkpoint id, with additional user context injection
    iter, err = runner.Resume(
        ctx,
        checkpointID,
        adk.WithToolOptions([]tool.Option{agents.WithNewInput(nInput)}),
    )
    if err != nil {
        log.Fatal(err)
    }

    for {
        event, ok := iter.Next()
        if !ok {
            break
        }

        if event.Err != nil {
            log.Fatal(event.Err)
        }

        prints.Event(event)
    }
}

If you stop and think about how you would build this example with regular tools, the advantages of ADK will become apparent:

Useful Resources

Find out more about Eino using the following links:

• Read the docs

• Browse source code on GitHub

• Explore examples on GitHub

If you have questions about Eino, head to GitHub to fill out an issue.

If you want to get in touch with the Eino Team Lead, Li Pandeng, reach out to lipandeng [at] bytedance [dot] com.

Architecting fundamentals can help navigate these complexities and the successful building and operation of these sorts of systems. Historically, an architect would aid in the building of a system and then would be able to sort of “walk away” or execute a new project. In this age of AI, this is simply not the case. The challenges of observability, integrated development, testing, the voice of the user, and their centrality to business outcomes place new demands on an architect.

Challenges that impact AI enabled systems

• Complex integrations: For example, is the system going to be a hybrid cloud or on premise deployment? What dependencies exist on external software?

• High performance bar: the system is expected to be deployed to impact key business functions or activities that the organization shall depend upon.

• Depending on the technology, using the AI/ML can incur significant costs.

• If not well architected or instrumented, the system can be a challenge to troubleshoot.

What can architecture do to help?

• By modeling and conducting robust conceptual designs, the unique perspective and aspects of the AI technology are identified.

• The use of tactics and patterns partitions the system for better performance and meeting non-functional requirements.

• Robust architecting helps keep a holistic view of the end to end system.

The rise of AI architects

History has conclusively shown that as modern technology progresses, the practice of architecting also evolves. We are now witnessing the rise of the AI architect. The modern AI architect will be closer to operations and must help with managing the complexity of systems. These include:

• The architect is the principal approver for the completion of acceptance gates for going from an AI prototype to a production deployment. He ensures the concept has merit, evaluates the results of the prototype, guides design of the production model, evaluates the performance of prototypes, and finally accepts the production deployment.

• They integrate and synthesize solutions to balance the data science, data engineering, software development, operations, and business teams.

• The architect is the principal owner of the non-functional requirements and interfaces across the AI system.

• They must be involved in the observability, where they need to understand what decisions the software is making.

From Idea to Production Life cycle

As AI engineering is maturing, a notional process is emerging that takes an AI idea to production. The initial stage is discovery and evaluation. It looks to determine if the data, algorithms, and compute resources exist to make a system that will have impact. The next stage is prototyping; here one now adds complexity to the software, both by adding more production domain elements, to test how well the new system would work. It is at the end of the prototype phase where a key decision is made to determine if it is warranted to take the prototype to a production environment.

With the passage out of the prototyping phase, a controlled execution of the model is done, where it is observed and monitored to ensure it is working as expected and delivering value. With the monitoring phase complete, a controlled rollout of the system is done to production, with another layer of monitoring to ensure the systems it is now impacting still also function as expected. Finally, with these gates completed, a full deployment can be done and now the system goes into a monitor and evaluation phase. The evaluation is to see if the model needs to be adapted or retrained.

One key item is as the system is now in operation – a new set of metrics for system observability need to be captured. Below are some basic ones; this should not be considered exhaustive:

1. Accuracy

2. Data throughput

3. Area under the Curve

4. Time for processing

5. Cost per transaction

How often these metrics should be reviewed really depends on how time dependent AI performance is to the enterprise. That said, these observability metrics should be reviewed frequently by the operations teams, and on a weekly basis or sooner by the other stakeholders, to see how AI model performance is impacting their areas of responsibilities.

Specific areas an AI architect can influence

As has been mentioned, the AI architect needs to have a solid understanding of, and be able to communicate in the language of, AI/ML and data science topics. This includes the perspective in understanding the role of non-functional requirements as applied to AI/ML systems. How to extend their knowledge for architecture modeling should incorporate systems where there is to be a sort of non-human decision making. They will also need to fortify and understand how users and stakeholders of the system are to be impacted, aided, and potential friction points. They will need to be close to guiding and evaluating the rapid prototyping in support of software development. The architect needs to consider interfaces and how their management will impact the decision making of the system. The use of AI/ML technologies inherently add a layer of complexity. The architect acts at his own peril with the notion that a simple AI application can readily be made in a less rigorous manner. Adding a layer of AI into almost any application raises the complexity significantly.

The AI architect must by necessity straddle AI/ML and software engineering. This is to ensure the architect can navigate the alphabet soup of techniques, algorithmic aspects, and technologies that exist. It is very easy for an AI architect to be overcome by the various other roles needed to deploy modern AI systems. In many enterprises, there are typically several functional teams: Data Team, Development Team, ML Ops, Legal/Compliance. Below are some of the major facets that the architect influences and needs to be involved with:

Data Team

The architect works with the data team to ensure that the product question can indeed be answered by the data that exists or is to be used. He is pivotal in levying requirements for the speed, format, and consistency checks that must be met.

Development Team

The architect works to ensure top-level designs, interfaces, and non-functional requirements are being met by the designs being created. He is more in a reviewer mode. He also aids in clarifying engineering requirements and providing guidance to address design questions and challenges.

MLOps Team

The architect, in this role, defines how well the production system is required to perform, and how the system is to be instrumented and monitored for correct execution. He will also define the canaries and fail-safe mechanisms to ensure issues with the AI components do not compromise the full production environment. The mechanisms for model and system updating are also laid out by the architect.

Legal and Compliance Team

In this capacity, the architect provides the technical insight, design guidance, and evidence that the AI system is meeting the legal and compliance needs of the enterprise. The architect also oversees re-design activities and troubleshooting to ensure legal and compliance aspects are indeed met.

Incident Run Book

There needs to exist a run book that the architect is a principal in its development. A run book should look to use a layered manner to address an error in the AI model or its interfaces. Clear mechanisms and architecture need to exist to be able to disengage the model from a production system, and put the production system into a nominal configuration. The run book should also provide for clear expected outputs of the different stages of the system – that is, it should be documented what “correct” looks like and why at each stage to enable quicker troubleshooting. The error handling within the system should also be readily indexed in the run book so that system, data, and model tracing can occur. Finally, the run book should also include clear traceability to software repositories, configuration information, and points of contact.

Conclusion

These are exciting times for being an AI architect. The field is in its infancy and there are many excellent and challenging domains where one can make a significant impact. In this new era one needs to accept that there will be constant learning and adaptation. As the techniques and concepts of AI/ML continue to increase, the combinations and applications simply keep growing. That said, having a solid foundational knowledge of AI/ML concepts shall mitigate short-term knowledge gaps. Are you ready to become an AI architect?

Chapter 1: Thinking Computationally (from The C++ Programmer's Mindset By Sam Morley) © Packt (All Rights Reserved)

1.08MB ∙ PDF file

Download

Solving problems is a central part of being a programmer, as well as a useful skill for everyday life. The methodology is broadly the same wherever you look: identify smaller, more tractable challenges; realize these as instances of a general class of problem; solve the intermediate challenges; and put everything together as a sequence of simple steps to solve the larger problem. In computer science, we call this computational thinking. This chapter serves as an introduction to the basic components of computational thinking at a high level. The objectives are to lay the foundation for more detailed analysis and in-depth examples later in the book. The first part of the chapter introduces the four components of computational thinking (decomposition, abstraction, pattern recognition, and algorithm design). The second half of the chapter deals with C++ specifically and identifies some aspects of the C++ language and standard library that can not only help implement efficient solutions, but also help you think about the problems themselves. It is important to remember that the four components of computational thinking are not a step-by-step guide to solving problems. Learning how and when these different components come together to deliver a solution relies on a good knowledge of the tools and methodologies available to you as the solver, and on your past experience. This chapter will help you get started with building the necessary foundations of the theory and set the stage for building out some basic examples to get you started with tackling larger and more complex problems later. Solving problems is an iterative process. There will be many failed attempts and false starts. This is a necessary part of the process. The last part of the chapter deals with good software practices that will enable you to iterate quickly and easily on your designs and arrive at a correct and usable solution more quickly. In this chapter, we’re going to cover the following main topics: • The components of computational thinking • Decomposing problems • Building abstractions and recognizing common patterns • Understanding algorithms • Using modern C++ and good practice Download to continue reading...

Download

To go deeper check out The C++ Programmer’s Mindset (Sam Morley, Packt, 1st ed., Nov 2025). The book introduces computational thinking as a practical framework—decomposition, abstraction, and pattern recognition—and shows how to apply it using modern C++ features to build solutions that are maintainable, efficient, and reusable. Across small examples and a larger case study, Morley covers using algorithms and data structures effectively, designing modular code, analyzing performance, and scaling work with concurrency, GPUs, and profiling tools—aimed at intermediate C++ developers who want to strengthen both their technical toolkit and the way they approach complex software challenges.

Here’s what some readers have said:

AI has emerged as a transformative force, revolutionizing industries and reshaping the way we interact with technology and the world around us. At its core, AI refers to computational models that mimic human cognitive functions, including learning from data, recognizing patterns, making decisions, and even interacting with their environment. This revolutionary technology spans a wide spectrum, from simple rule-based systems to sophisticated deep learning models, each with unique applications and capabilities.

A major aspect of any AI system is that the results of the inference being done need to be relevant and trusted. To ensure trust is gained and maintained, the use of strong architecture is paramount. One not only architects the technology but also how the technology is going to be used, managed, and evaluated by the span of stakeholders. The stakeholders need to be able to pinpoint issues, rapidly correct model parameters, and deploy changes in a deliberate and rapid manner. In more common parlance, the architecting and supporting processes can be described as “guard rails.” How one employs guard rails is very specific to the domain and use case that is to use the AI technology. There are classes of guard rails that can be discussed – for example, the use of canaries to judge model correctness from a known gold standard, time and data flow metrics to judge model performance, and the use of filters and robust data quality checks so that only consistent and correct data enters the system. Another class of guardrails is human system interfaces, such as alerting frameworks to classify errors and monitors, the use of troubleshooting tools, and preset protocols for handling unexpected errors. Written procedures or guidance from modeling allow for the maintenance of a system without the need to call upon the model developer to do troubleshooting.

Trust is a paramount consideration for system success, so one needs to architect a system with that in mind. In many ways, the presentation and lessons learned described in this book look to ensure trust in an AI system.

This chapter highlights, in a broad sense, the key aspects of AI architecture considerations that drive a successful AI implementation. The topics are as follows:

• Introduction and key AI concepts

• Components of an AI system

• AI technologies and microservices

• AI systems and technical considerations

• Deployment considerations

Introduction to AI systems: architecting the future of intelligence

AI systems are the embodiment of AI, acting as the engines that power intelligent applications and services. These systems are intricate constructs, meticulously designed to perform a diverse range of tasks, from image recognition and natural language processing to autonomous decision-making and complex problem-solving.

The architecture of an AI system functions as a detailed technical blueprint, specifying its structural organization and the precise interactions between its various components. These components include the following:

• Hardware infrastructure: CPUs for general processing, GPUs for parallel computation, TPUs for tensor operations, and specialized AI accelerators

• Software frameworks: TensorFlow, PyTorch, JAX, and other libraries that enable model development

• Algorithmic implementations: Machine learning algorithms, neural network architectures, and inference engines

• Data pipelines: ETL processes, feature stores, and data management systems

All these elements work in a coordinated operation to enable the system to fulfill its designed objectives efficiently and reliably.

A well-architected AI system achieves several critical technical requirements:

• Optimal performance: Maximizes computational efficiency to deliver responsive and accurate results with minimal latency. This involves an optimized model design, efficient resource allocation, and hardware-aware implementations that fully utilize available computing capabilities.

• Scalability: Handles growing workloads and expanding datasets through both horizontal scaling (adding more machines) and vertical scaling (adding more powerful machines) without performance degradation. Modern AI architectures must accommodate increasing data volumes, user bases, and computational demands.

• Efficiency: Reduces computational resource consumption, energy usage, and operational costs through techniques such as model quantization, knowledge distillation, and optimized inference paths. Efficient AI systems minimize their resource footprint while maintaining functional effectiveness.

• Reliability: Ensures consistent operation with high-availability metrics, even when facing unexpected data patterns, input variations, or system failures. This requires robust error handling, graceful degradation capabilities, and comprehensive monitoring systems. Given that AI technologies can be both deterministic and non-deterministic, consideration must be given to allow for human intervention. This intervention needs to span the gamut from simple monitoring to a full suite of testing infrastructure.

• Security: Implements comprehensive data protection measures and defends against adversarial attacks, data poisoning, and model vulnerabilities. AI systems must maintain data confidentiality and integrity, and be resilient against both traditional cybersecurity threats and AI-specific attacks.

• Explainability: Provides transparent visibility into algorithmic decision processes, supporting regulatory compliance, user trust, and system debugging. Modern AI architectures must balance performance with interpretability to meet growing demands for AI transparency.

The field of AI is constantly evolving, with new architectures and technologies emerging at a rapid pace. As we delve deeper into this fascinating domain, we will explore the various types of AI systems, their underlying principles, and the diverse applications that are shaping the future of technology and society.

What is an AI system?

An AI system is a computational model or a collection of models designed to perform tasks that typically require human intelligence. These systems are powered by algorithms and data, enabling them to learn from experience, adapt to new information, and make decisions or predictions.

From an implementation perspective, AI systems typically consist of several key layers:

1. Hardware: Encompasses compute resources such as CPU, GPU, TPUs, full-spectrum storage, and networking

2. Data layer: Handles data ingestion, storage, preprocessing, and feature engineering

3. Model layer: Contains the trained machine learning or deep learning models

4. Inference layer: Manages the execution of models against new data inputs

5. Application layer: Integrates AI capabilities into user-facing applications

6. Monitoring layer: Tracks system performance, data drift, and model health

AI systems can be classified into two broad categories:

• Narrow AI (weak AI): These systems are designed to excel at specific tasks within a limited domain. Examples include image recognition software, spam filters, and recommendation engines. While they may be highly proficient at their designated tasks, they lack the ability to generalize their knowledge in other areas.

• General AI (strong AI): This is a theoretical concept of an AI system that possesses human-level intelligence and can perform any intellectual task that a human can. It would have the ability to reason, plan, solve problems, learn from experience, and understand complex ideas across diverse domains. While general AI remains a distant goal, significant progress has been made in developing systems with increasingly sophisticated capabilities.

The pervasive impact of AI infrastructure: powering intelligent solutions across industries

Well-architected AI infrastructure, encompassing the hardware, software, and networks that support AI applications, is the driving force behind the transformative impact of AI across industries. This infrastructure enables the deployment and scaling of AI models, algorithms, and frameworks, unlocking their full potential to address complex challenges and deliver innovative solutions.

• Healthcare:

  • Accelerated medical image analysis: High-performance computing clusters and specialized hardware accelerators enable rapid processing of medical images, facilitating faster and more accurate diagnosis.

  • Data-driven insights: Scalable storage and processing infrastructure empowers AI-driven analytics on vast patient datasets, leading to personalized treatment plans and improved patient outcomes.

  • Real-time monitoring: Cloud-based AI infrastructure enables continuous monitoring of patient vitals and other health data, facilitating timely interventions and proactive care.

• Finance:

  • Robust fraud detection: Distributed computing and real-time analytics platforms empower AI models to detect fraudulent transactions with greater accuracy and speed, protecting financial institutions and consumers.

  • Optimized trading strategies: High-frequency trading algorithms leverage low-latency networks and powerful computational resources to execute trades with precision and efficiency, maximizing returns.

  • Personalized financial services: Cloud-based AI infrastructure enables the deployment of robo-advisors and other AI-powered tools that provide tailored financial advice and services to individuals.

• Autonomous vehicles:

  • Real-time sensor fusion: High-throughput data pipelines and edge computing infrastructure enable the rapid processing of sensor data from cameras, lidar, radar, and other sources, ensuring timely decision-making for autonomous vehicles.

  • Enhanced object recognition: Deep learning models trained on massive datasets and deployed on specialized hardware accelerators enable accurate and reliable identification of objects in the environment.

  • Optimized navigation: Cloud-based mapping and navigation services, combined with onboard AI processing, provide autonomous vehicles with real-time information and guidance for safe and efficient navigation.

The continued development and optimization of AI infrastructure will play a crucial role in realizing the full potential of AI across industries. By providing the foundation for performant and scalable AI solutions, this infrastructure is poised to transform the way we live and work.

Key components of AI system architectures

AI systems, in their essence, are complex structures designed to emulate human cognitive abilities such as learning, reasoning, and problem-solving. To achieve these capabilities, AI systems rely on a well-defined architecture comprising several interconnected components, each playing a crucial role in the overall functioning of the system. Understanding these key components is fundamental to comprehending the inner workings and potential of AI.

• Data components: Data serves as the lifeblood of any AI system, acting as the raw material upon which the system learns and improves. Data can exist in multiple forms:

  • Structured data: Organized in predefined formats such as databases and spreadsheets

  • Semi-structured data: Partially organized information such as JSON or XML files

  • Unstructured data: Raw information, including text documents, images, audio recordings, and video files

The quality, quantity, and relevance of the data significantly impact the AI system’s performance and ability to generalize to new situations.

• Algorithmic frameworks: Algorithms are engines driving AI systems, providing instructions and logic for processing data and generating intelligent outputs. Machine learning algorithms, a subset of AI algorithms, empower systems to learn patterns and relationships from data, enabling them to make predictions, classifications, or decisions. Common algorithmic approaches in production AI systems include the following:

  • Traditional machine learning: Linear regression, random forests, gradient boosting, and support vector machines

  • Deep learning: Convolutional neural networks (CNNs), Recurrent Neural Networks (RNNs), transformers, and graph neural networks

  • Reinforcement learning: Q-learning, policy gradient methods, and actor-critic architectures

The selection of appropriate algorithms depends on the specific problem domain, available data characteristics, and performance requirements.

• Model architectures: Models represent the culmination of the learning process in AI systems. They are mathematical representations of the knowledge extracted from data, encapsulating the patterns, relationships, and insights discovered by the algorithms. These models can be simple or complex, depending on the nature of the task and the algorithm used. Model architectures range between the following:

  • Simple linear models: Easily interpretable but limited in capability

  • Ensemble models: Combining multiple simpler models for improved performance

  • Deep neural networks: Complex architectures with millions or billions of parameters

Once trained, models are used to make predictions or decisions on new, unseen data.

• Infrastructure: The infrastructure component encompasses the hardware and software resources that provide the computational power and environment necessary for AI systems to operate. Key infrastructure elements include the following:

  • Computational resources: High-performance servers, specialized AI accelerators (GPUs, TPUs, FPGAs), and distributed computing clusters

  • Storage systems: High-throughput, scalable storage for training data and model artifacts

  • Networking components: Low-latency interconnects for distributed training and inference

• Development frameworks: Software libraries such as TensorFlow, PyTorch, and Hugging Face that streamline AI development and deployment.

Understanding these key components and their interactions provides a solid foundation for comprehending the complex landscape of AI system architectures. By carefully designing and optimizing each component, researchers and engineers can build AI systems that are capable of tackling a wide range of tasks and applications, from image recognition and natural language processing to autonomous driving and drug discovery. The integration of AI capabilities into existing software stacks requires thoughtful architectural considerations to successfully incorporate intelligence while addressing the unique requirements that AI components introduce. These specific requirements and architectural approaches form the central focus of this book. Due to the complexity of AI systems, the nature of the deployment approach is paramount. The next section will discuss the use of microservice architectures that provide a balance between performance and modularity.

Microservice architectures: a modular approach to building complex AI systems

As AI systems grow in complexity, traditional monolithic architectures can become unwieldy, hindering development speed and flexibility. Microservice architectures offer a compelling alternative by breaking down these complex systems into smaller, independent services. Each microservice focuses on a specific function and communicates with others through well-defined APIs.

Advantages of microservices for AI

• Enhanced agility and flexibility: Teams can independently develop, deploy, and update each microservice, using the most suitable technologies and programming languages for each task. This accelerates development cycles and allows for easier experimentation and innovation.

• Improved scalability: Microservices can be scaled horizontally to meet specific demand, ensuring optimal resource utilization. For example, a service handling image processing can be scaled independently of a service responsible for natural language understanding.

• Increased resilience and fault isolation: If a microservice fails, the impact is localized, minimizing disruption to the entire system. This enhances overall reliability and simplifies troubleshooting.

• Technological diversity: Microservice architectures empower teams to leverage the best tools for each task, promoting innovation and allowing for gradual technology upgrades.

Challenges of microservice architectures

• Increased complexity: Managing a multitude of services and their interactions requires robust orchestration and monitoring tools. Service discovery, load balancing, and failure handling become critical considerations.

• Communication overhead: Excessive inter-service communication can introduce latency and impact overall performance. The careful design of APIs and communication patterns is essential to mitigate this issue.

• Data consistency: Maintaining data consistency across distributed services can be challenging. Strategies such as eventual consistency or distributed transactions may be required to ensure data integrity.

Real-world example: conversational AI microservices implementation

To illustrate how a microservices approach can streamline a conversational AI solution, let us examine a practical example that demonstrates how these principles come to life. This section explores a conversational AI system – such as a chatbot or virtual assistant – built using a four-service microservices architecture with an API gateway.

The four core microservices

Figure 1.6 illustrates the high-level design of our conversational AI system:

The architecture consists of four core specialized services plus an API gateway:

1. Language understanding service:

   • Primary functions: Intent classification, entity identification/extraction, and hosting of NLP models.

   • Data and models: References one or more NLP model databases (for example, transformer-based classifiers).

   • Key interactions: Receives the user’s text (through the API gateway), determines the user’s intent (e.g., “Check account balance”), and extracts relevant entities (e.g., “date,” “location,” “product name”).

2. Dialog management service:

   • Primary functions: Oversees conversation flow, handles session state, and orchestrates the next step in the dialog.

   • Data and state: Maintains conversation context in a dedicated state database.

   • Key interactions: Logs conversation events (asynchronously) and updates or retrieves session details to guide the flow (e.g., “Greeting,” “Confirmation,” “Next step”).

3. Knowledge response service

   • Primary functions: Retrieves relevant information and formulates responses. This might involve querying a knowledge base (e.g., FAQs, product info) or assembling template-based replies.

   • Data and templates: Stores domain-specific data in a knowledge DB and uses templates or generative mechanisms for response creation.

   • Key interactions: Receives queries from the dialog management service, finds or composes the best response, and returns it for final delivery to the user.

4. Conversation analytics service:

   • Primary functions: Processes logs and usage metrics for reporting, visualization, and deeper analytics (e.g., intent distribution, user satisfaction trends).

   • Data and reporting: Maintains analytics data in a separate database for dashboards or offline processing.

   • Key interactions: Collects asynchronous event logs from the dialog management service and other components to measure performance, track user behavior, and provide insights that could improve the system over time.

Role of the API gateway

Although not counted as one of the four microservices, the API gateway is a vital component at the front of the architecture. It does the following:

• Receives requests from the user (via text or other channels)

• Initializes the session and routes incoming data to the language understanding service

• Forwards recognized intents and updates to the dialog management service

• Passes replies from downstream services back to the user

By centralizing traffic management, the API gateway enforces consistent security, throttling, and monitoring policies while keeping each microservice isolated and independently scalable.

Conversation flow sequence

To illustrate how these microservices interact during a typical user journey, Figure 1.7 shows the sequence of calls between them in a single conversation cycle:

The sequence progresses as follows:

1. User → API gateway: The user sends a request (e.g., a chat message). The API gateway initializes the session (if needed) and forwards the message to the language understanding service.

2. Language understanding service:

   • Performs intent classification and entity identification.

   • Returns a recognized intent (e.g., “CheckWeather”) and any extracted entities (e.g., date, location) to the API gateway.

3. Dialog management service:

   • Receives recognized intent from the API gateway.

   • Logs conversation events (asynchronously) into the conversation analytics service.

   • Updates or retrieves the session state (e.g., user’s location or recent conversation context).

4. Knowledge response service:

   • Once the dialog management service determines additional data is needed (e.g., weather info, product detail), it sends a query to the knowledge response service.

   • This service fetches the necessary information or constructs a response template (e.g., “The weather for your location is sunny with 75°F...”).

5. Conversation analytics service (asynchronous logging):

   • Continuously receives usage data and conversation logs from the dialog management service (and possibly from the knowledge response service).

   • Processes and stores these logs for future reporting (e.g., monthly usage dashboards, model performance metrics).

6. Reply to the user:

   • The knowledge response service’s formulated answer is routed back through the dialog management service (if necessary, for final session updates) and then returned via the API gateway.

   • The user receives the reply and the interaction concludes.

Key aspects of microservice communication

• Synchronous versus asynchronous calls:

  • Requests that must return immediately (e.g., generating a response for the user) use synchronous calls.

  • Logging or analytics operations are typically performed asynchronously to avoid slowing down the core conversation loop.

• Stateful versus stateless components:

  • Dialog management requires tracking session state, while other services (e.g., language understanding) often benefit from stateless designs for simpler scaling.

  • The dialog management service may require robust state management solutions, such as distributed caches or databases.

• Service autonomy:

  • Each microservice can be updated or replaced independently without affecting the rest of the system.

  • The language understanding service’s NLP models may need frequent retraining. Because it is a separate service, such updates can be deployed without disrupting the other services.

• Data isolation:

  • Services manage their own domain data. Dialog management stores conversation state, knowledge response holds domain facts, and analytics maintains interaction logs.

  • Sensitive user data should be restricted to the dialog management service’s state store when necessary, minimizing the exposure across the entire system.

Implementation considerations for conversational AI microservices

1. Scaling independently:

   • The language understanding service can be scaled up or down based on incoming message load (e.g., horizontal autoscaling for peak chat traffic).

   • The dialog management service maintains conversation state and may require different scaling strategies.

   • The knowledge response service often scales according to the complexity of information retrieval.

   • The conversation analytics service can be scaled separately, especially if analytics workloads (such as report generation) spike at different times than user requests.

2. Latency management:

   • Conversational AI systems aim for near real-time interactions. Minimizing network hops and communication overhead between services is crucial. Using lightweight communication protocols helps ensure the system performs well at scale.

3. Fault isolation:

   • If one service fails (for instance, the knowledge response service goes offline), the rest of the system can still handle other tasks or offer fallback behaviors (e.g., an apology response or a redirect to a human agent).

4. Monitoring and observability:

   • Robust logging and observability practices are crucial to ensure the system remains resilient to service failures or slowdowns. The conversation analytics service plays a key role in tracking system health and performance.

Why microservices for conversational AI?

Breaking down a conversational AI system into these four specialized services confers significant benefits in maintainability, scalability, and team agility. Each service can evolve independently, allowing rapid iteration on NLP models, conversation flows, and knowledge retrieval strategies without risking a “big bang” failure across the entire application.

At the same time, careful attention to inter-service communication is crucial. As the sequence diagram shows, multiple hops occur for every user request. Using lightweight communication protocols and distinguishing between synchronous and asynchronous operations helps maintain system responsiveness.

The example of conversational AI powerfully illustrates how the microservices approach enables balancing flexibility, fault tolerance, and iterative innovation. The lessons learned here – such as independently scaling critical services, isolating data for security, and ensuring graceful failure modes – apply broadly to a wide array of AI-driven solutions.

This real-world implementation pattern demonstrates that while microservices add complexity, the benefits they bring to AI systems – particularly those requiring frequent updates, variable scaling, and component-level innovation – often outweigh the challenges when properly architected and implemented.

Considerations for an AI system

Creating a well-designed AI system architecture necessitates careful consideration of several key factors. These factors ensure that the system not only functions effectively but also adapts to future demands and challenges.

Scalability: handling growing data and model complexity

AI systems often encounter growing volumes of data and increasingly complex models. Scalability is the ability of a system to handle this growth without compromising performance. Effective strategies include the following:

• Horizontal scaling: This involves adding more compute resources to distribute the workload. For instance, in a cloud environment, you might deploy additional virtual machines or containers to handle increased traffic. Kubernetes can orchestrate these containers, ensuring that the workload is evenly distributed.

• Vertical scaling: Enhancing existing resources with more powerful hardware. For example, upgrading a server’s CPU or GPUs, adding more RAM, or using SSDs instead of HDDs to improve I/O performance.

• Distributed computing: Utilizing frameworks such as Apache Spark or Hadoop to process data across multiple nodes. This approach breaks down large datasets into smaller chunks that can be processed in parallel, significantly reducing processing time. For instance, Spark’s Resilient Distributed Datasets (RDDs) allow for in-memory processing, which is much faster than traditional disk-based processing.

Performance: optimization techniques

In many AI applications, real-time or near-real-time processing is crucial. Techniques to optimize performance include the following:

• Hardware acceleration: Leveraging GPUs or TPUs for computationally intensive tasks – for example, TensorFlow and PyTorch can utilize CUDA cores in NVIDIA GPUs to accelerate deep learning model training.

• Parallel processing: Dividing tasks into smaller sub-tasks that can be executed concurrently. In Python, libraries such as multiprocessing or concurrent.futures can be used to parallelize tasks – for instance, training multiple models simultaneously or processing different data batches in parallel.

• Algorithm optimization: Choosing or designing algorithms with lower computational complexity. For example, using approximate nearest neighbor algorithms for large-scale similarity search instead of exact methods, which are computationally expensive.

Reliability: fault tolerance, error handling, and redundancy

Reliability is paramount, especially in critical applications. To ensure system uptime and data integrity, strategies such as fault tolerance, error handling, and redundancy are employed:

• Fault tolerance: The system can continue operating even if some components fail. For example, in a microservices architecture, if one service fails, others can continue to function. Tools such as Netflix’s Hystrix can be used to implement circuit breakers to manage failures.

• Error handling: Mechanisms are in place to detect and correct errors gracefully – for instance, using try-catch blocks in code to handle exceptions and logging errors for further analysis.

• Redundancy: Critical components are duplicated to prevent single points of failure – for example, using RAID configurations for disk storage or deploying services in multiple availability zones in cloud environments to ensure high availability.

Security: data privacy and model robustness

AI systems often handle sensitive data, making security a top priority. Key considerations include the following:

• Data encryption: Protecting data at rest and in transit – for instance, using AES encryption for data stored in databases and TLS for data transmitted over networks. The use of encryption approaches needs to be considered and tested thoroughly to scope the impact on model and system performance.

• Access control: Implementing strict authorization and authentication mechanisms – for example, using OAuth 2.0 for secure API access and role-based access control (RBAC) to manage permissions.

• Model robustness: Guarding against adversarial attacks that could manipulate the system. Techniques such as adversarial training, where the model is trained on both normal and adversarial examples, can help improve robustness. Additionally, you can deploy anomaly detection systems to monitor for unusual patterns in data input.

Data modeling: catalogs and ontologies

In the realm of AI, data is not just a valuable asset but the very foundation upon which intelligent systems are built. As AI models rely heavily on vast amounts of data to learn and make informed decisions, effective management and organization of this data becomes paramount. This is where data catalogs and ontologies step in as indispensable tools for navigating the complexities of data landscapes within AI architectures.

Catalogs serve as centralized repositories of metadata, providing comprehensive information about the data assets within an AI system. They act as a comprehensive index, offering insights into the data’s location, schema, lineage, quality, and other relevant attributes. By consolidating this information in a structured and accessible manner, data catalogs empower data scientists, engineers, and analysts to gain a deeper understanding of their data resources, streamline their workflows, and ensure data governance.

Ontologies give a semantic representation of the data elements within the domain. They can aid the data engineer in understanding how and why data elements are associated and improve processing pipelines. Ontologies also give data scientists context for model development and updating.

The technical and functional attributes of AI systems have been discussed. The next section discusses the different ways to implement systems in a modern cloud context. The use of cloud technology ensures that one can readily scale an AI system based on actual demand and provides for flexibility in resource allocations.

Modern AI deployment paradigms

As AI systems continue to evolve, new deployment paradigms have emerged to address specific requirements and use cases. This section explores two significant approaches: cloud-native AI architectures and edge AI deployments.

Cloud-native AI architectures

The increasing complexity and scale of AI applications have led to the adoption of cloud-native architectures. These architectures leverage the scalability, flexibility, and cost-efficiency of cloud computing platforms to enable efficient development, deployment, and management of AI systems. In a cloud-native architecture, AI components are designed to run seamlessly in cloud environments, taking advantage of specialized services for storage, compute, and networking.

Key characteristics of cloud-native AI architectures include the following:

• Containerization: AI applications are packaged into lightweight, portable containers using technologies such as Docker, ensuring consistency across development, testing, and production environments.

• Orchestration: Container orchestration platforms such as Kubernetes manage the deployment, scaling, and operation of application containers across clusters of hosts.

• Microservices: As discussed earlier, breaking down AI systems into smaller, independent services enables more efficient resource utilization and easier scaling.

• Serverless computing: Platforms such as AWS Lambda, Azure Functions, and Google Cloud Functions allow developers to focus on writing code without worrying about the underlying infrastructure, particularly useful for event-driven AI workloads.

• Managed services: Cloud providers offer specialized AI services such as fully managed machine learning platforms (e.g., Amazon SageMaker, Microsoft Azure ML, Google Vertex AI) that streamline the development and deployment process.

• Cloud-native versus lift-and-shift: Cloud-native AI components are specifically designed to leverage the benefits of cloud environments, such as auto-scaling, serverless computing, and managed services. This approach offers greater flexibility, scalability, and cost-efficiency compared to simply “lifting and shifting” existing on-premises AI systems to the cloud without architectural modifications.

Data lakes and data warehouses in AI architectures: foundations for data-driven intelligence

In the realm of AI, data is the cornerstone of innovation and progress. AI models thrive on massive volumes of data, leveraging it to learn patterns, make predictions, and generate valuable insights. However, effectively managing and harnessing the vast amounts of data involved in AI projects necessitates specialized storage and management solutions. Two prominent concepts that have emerged in this context are data lakes and data warehouses.

Data lakes: a vast reservoir of raw data

Data lakes serve as expansive repositories where raw data is stored in its native format. They are designed to accommodate structured, semi-structured, and unstructured data from diverse sources. The flexibility of data lakes makes them ideal for storing large volumes of data that may not have a predefined purpose or structure.

• Key characteristics:

  • Schema-on-read: Data lakes do not enforce a strict schema during ingestion, allowing for flexibility in data types and structures. The schema is defined during analysis or processing, empowering users to adapt to evolving data requirements.

  • Cost-effective scalability: Data lakes can easily scale to accommodate growing data volumes, making them a cost-effective solution for storing massive datasets.

  • Support for diverse data: Data lakes can handle a wide range of data, including sensor readings, social media feeds, log files, and more.

  • Ideal for exploratory analysis: Data lakes provide a fertile ground for data scientists and analysts to explore data, identify patterns, and generate hypotheses.

• Example use cases:

  • An e-commerce company might store clickstream data, customer reviews, and social media interactions in a data lake for subsequent analysis and personalization efforts.

  • A healthcare organization could use a data lake to store medical images, electronic health records, and genomic data for research and development of AI-driven diagnostic tools.

Data warehouses: structured repositories for analytics

Data warehouses are structured repositories that house processed and curated data, transformed into a consistent format for analysis and reporting purposes. One can build and develop ontologies to organize and provide semantic structure to the data that comes into the system. Ontologies also provide a mechanism to better manage and control model performance by making relationships between data elements explicit.

They excel at facilitating efficient querying and analysis, making them indispensable for business intelligence and decision support applications.

• Key characteristics:

  • Schema-on-write: Data warehouses enforce a predefined schema during data ingestion, ensuring data consistency and integrity.

  • Optimized for querying: Data warehouses employ optimized data structures and indexing techniques to accelerate data retrieval and analysis, enabling faster insights.

  • Support for structured data: Data warehouses are primarily designed for structured data, such as transactional data, customer information, and financial records.

  • Ideal for business intelligence: Data warehouses empower organizations to generate reports, dashboards, and visualizations for informed decision-making.

• Example use cases:

  • A financial institution might use a data warehouse to store transaction data, customer information, and market trends for risk analysis and fraud detection.

  • A manufacturing company could leverage a data warehouse to analyze production data, supply chain metrics, and customer feedback to optimize operations and improve product quality.

The synergy of data lakes and data warehouses

In many AI architectures, data lakes and data warehouses complement each other. Raw data is first ingested into a data lake, where it undergoes cleansing, transformation, and enrichment. The refined data is then transferred to a data warehouse for further analysis and reporting. This synergistic approach enables organizations to leverage the flexibility of data lakes for data exploration and the structure of data warehouses for decision support, creating a robust foundation for data-driven AI applications.

AI on cloud computing: a game-changer for AI

The convergence of AI and cloud computing has opened up a new frontier of possibilities for organizations seeking to leverage the power of AI. Cloud computing provides a scalable, flexible, and cost-effective infrastructure for developing, deploying, and scaling AI applications. By harnessing the capabilities of the cloud, businesses can overcome the limitations of traditional on-premises AI solutions and accelerate innovation.

Benefits of cloud-based AI

Cloud-based AI offers several key advantages that make it an attractive option for organizations of all sizes:

• Scalability: Cloud resources can be easily scaled up or down to meet the fluctuating demands of AI workloads. This elasticity allows organizations to handle large datasets, train complex models, and process vast amounts of data without having to invest in and maintain expensive hardware infrastructure.

• Flexibility: Cloud platforms provide a wide range of AI services and tools, giving organizations the flexibility to choose the best options for their specific needs. This allows businesses to experiment with different AI approaches, quickly iterate on models, and adapt to changing requirements.

• Cost-efficiency: Cloud-based AI can be more cost-effective than on-premises solutions. Organizations only pay for the resources they consume, eliminating the need for upfront capital investments in hardware and software. Additionally, cloud providers often offer pay-as-you-go pricing models, which can further reduce costs.

By leveraging the power of cloud-based AI, organizations can unlock new levels of innovation, efficiency, and competitiveness.

Major cloud AI platforms: accelerating innovation with comprehensive toolsets

Major cloud providers have emerged as key players in the AI landscape, offering comprehensive suites of AI services and tools that cater to a wide range of needs. These platforms provide a one-stop shop for businesses and developers looking to leverage the power of AI in their applications and workflows.

Key cloud AI platforms

• Google Cloud AI platform (Vertex AI): This unified platform streamlines the entire Machine Learning (ML) lifecycle, from building and training models to deploying and managing them in production. Vertex AI’s AutoML feature simplifies model development for users with limited ML expertise, while the model garden offers a collection of pre-trained models ready for deployment. Vertex AI Pipelines orchestrates complex ML workflows, enabling efficient experimentation and automation.

• Amazon SageMaker: A fully managed service, SageMaker empowers users to build, train, and deploy ML models at scale. It boasts a wide array of built-in algorithms and frameworks, making it accessible to both beginners and experienced practitioners. SageMaker’s scalability and integration with other AWS services make it a popular choice for enterprise-grade AI solutions.

• Amazon Bedrock: This cutting-edge service democratizes access to Foundation Models (FMs) from leading AI start-ups and Amazon itself through a simple API. Bedrock enables developers to harness the power of state-of-the-art generative AI capabilities without having to build and train complex models from scratch.

• Microsoft Azure AI: This platform offers a diverse range of AI services, including pre-built AI models for computer vision, speech recognition, natural language processing, and decision-making. Azure Machine Learning allows users to create and deploy custom AI models, while the platform’s extensive integration with other Azure services makes it a versatile choice for a variety of AI applications.

These cloud AI platforms provide a powerful and accessible way for organizations to incorporate AI into their operations, accelerating innovation and driving business value.

Summary

In this chapter, we have explored the fundamental principles of AI system architecture, establishing a comprehensive framework for understanding the building blocks that power intelligent systems. We examined the core components – data as the lifeblood, algorithmic frameworks that enable learning, model architectures that encapsulate intelligence, and infrastructure that provides computational resources – along with architectural patterns such as microservices that offer modularity and flexibility. Critical design considerations of scalability, performance, reliability, and security were discussed as essential elements for robust AI systems that can grow with increasing demands while remaining resilient and protected.

The landscape of AI deployment continues to evolve rapidly, with cloud-native architectures leveraging containerization, orchestration, and serverless computing to achieve unprecedented efficiency. The synergy between data lakes, data warehouses, and data catalogs creates a solid foundation for data-driven intelligence, while major cloud platforms democratize access to sophisticated AI capabilities. As we move forward, these foundational principles will guide the development of AI systems that are not only powerful but also scalable, reliable, and secure – enabling the next generation of innovations across industries.

Relevant reading

• Baheti, Radhakisan, and Helen Gill. “Cyber-Physical Systems.” The Impact of Control Technology, edited by Tariq Samad and Anuradha M. Annaswamy, IEEE Control Systems Society, 2011, pp. 161–66.

• Bass, Len, Paul Clements, and Rick Kazman. Software Architecture in Practice. 3rd ed., Addison-Wesley, 2012.

• Hazelwood, Kim, et al. “Applied Machine Learning at Facebook: A Datacenter Infrastructure Perspective.” 2018 IEEE International Symposium on High Performance Computer Architecture (HPCA), IEEE, 2018, pp. 620–629.

• LeCun, Yann, Yoshua Bengio, and Geoffrey Hinton. “Deep Learning.” Nature, vol. 521, no. 7553, 2015, pp. 436–44, doi.org/10.1038/nature14539.

• Mao, Hongzi, et al. “Resource Management with Deep Reinforcement Learning.” Proceedings of the 15th ACM Workshop on Hot Topics in Networks (HotNets-XV), ACM, 2016, pp. 50–56, people.csail.mit.edu/alizadeh/papers/deeprm-hotnets16.pdf.

• National Institute of Standards and Technology. Artificial Intelligence Risk Management Framework (AI RMF 1.0). NIST, 2023.

• Patterson, David, et al. “Carbon Emissions and Large Neural Network Training.” arXiv, 2021, arxiv.org/abs/2104.10350.

• Sculley, D., et al. “Hidden Technical Debt in Machine Learning Systems.” Advances in Neural Information Processing Systems, vol. 28, 2015, pp. 2503–2511.

To go deeper on designing robust, scalable AI-enabled systems—from integrating machine learning into existing architectures to managing risks like underperformance, cost overruns, and operational complexity—check out Architecting AI Software Systems by Richard D Avila and Imran Ahmad (Packt, 2025). Through a structured progression of architectural concepts, real-world case studies, and hands-on exercises (including a fictional AI-enabled system you can dissect end to end), it shows software and systems architects, CTOs, VPs of Engineering, AI/ML engineers, and developers how to select the right models and data pipelines, use architectural models to ensure cohesion, simulate and optimize AI performance through iteration, and apply patterns and heuristics to integrate AI into large-scale systems with strong user experience and performance—so you can confidently architect AI-driven products across a range of domains.

Here’s what some readers have said:

In my previous post, The Memory Illusion, I demonstrated that LLM memory doesn’t require vector databases or sophisticated architectures. It’s fundamentally just text management. We built a proof-of-concept in ~150 lines of Python that stored memories in a simple markdown file. It worked. But it had an amusing limitation: The user had to remember to tell the AI to remember.

The system required explicit commands. Want the LLM to store your name? Type !remember “My name is Alex”. Want it to know your project preferences? Another !remember command. The irony was sharp: we’d outsourced memory to technology, only to burden ourselves with managing that memory manually.

This wasn’t an oversight. It was a conscious design decision. Our application code controlled every memory operation through explicit if/else logic. The host app was the memory manager, and the LLM was simply our text processor.

But what if we handed that authority to the LLM itself?

This isn’t an incremental improvement or “v2” of the same approach. It’s a fundamentally different philosophy: trusting the LLM to autonomously manage its own memory. The technical implications are profound. We move from programming specific behaviors to setting high-level intentions. We shift from writing parsing logic to defining trust boundaries.

This approach enables the system to handle its own errors, organize information without explicit rules, and maintain its own memory hygiene. All without writing a single if/else statement.

The Explicit Approach: When Code Defines Every Decision

In my original POC, every memory operation required explicit user commands. Here’s what a typical session looked like:

[You]: Hello, I’m working on a React app
[Claude]: Hi! What kind of React app are you building?
[You]: !remember I am building a React e-commerce application
[Claude]: [Memory saved]

Here we see that required use of !remember. Adding to the user’s cognitive load alongside their actual work.

Behind the scenes, our code intercepted every message, parsed for commands, managed file operations, and reconstructed the prompt with memories for each interaction. We were the brain; the LLM was just processing text within our constraints.

This gave us complete control. We defined in code the exact format of the memory file. Want memories timestamped? We coded it. Want them categorized? More code. Every behavior was explicit, predictable, testable.

This is how we’ve built software applications since the inception of the craft. We write the logic, we define the control flow, we handle the edge cases. It’s comfortable, familiar territory. But when working with LLMs, this traditional approach means we’re not fully leveraging what makes them truly powerful: their ability to understand context and make intelligent decisions autonomously.

The LLM’s contextual intelligence sits idle while our code makes every decision. This intelligence was trained on billions of examples of how humans organize and retrieve information.

Most importantly, users had to remember the commands, creating friction in your app’s usability. Edge cases multiplied. The code grew ever larger as we handled more scenarios, more commands, more special cases. We were swimming upstream against the fundamental capabilities of modern LLMs.

The Implicit Approach: LLM as Autonomous Manager

The paradigm shift is what matters. We’re implementing a harness that grants the LLM autonomous authority. While this example uses the Claude Agent SDK, the pattern can be implemented with other SDKs or custom code. The key is delegation of decision-making, not specific tooling.

Let’s look at the skeleton of our memory tool implementation:
Full code found in the companion app

# memory_tool.py 
class LocalFilesystemMemoryTool(BetaAbstractMemoryTool):
    “”“
    The LLM calls these methods autonomously based on context.
    We provide the infrastructure; Claude makes the decisions.
    “”“

    # The hook methods (memory tools) Claude is made aware of:

    @override
    def view(self, command):
        “”“Claude calls this to read memories or list files”“”
        # Validate path, read file/directory, return contents

    @override
    def create(self, command):
        “”“Claude calls this to create new memory files”“”
        # Validate path, write file, log operation

    @override
    def str_replace(self, command):
        “”“Claude calls this to update existing memories”“”
        # Find text, replace it, handle errors

    @override
    def insert(self, command):
        “”“Claude calls this to add lines to memories”“”
        # Insert at specific line number

    @override
    def delete(self, command):
        “”“Claude calls this to remove memories”“”
        # Delete files or directories

    @override
    def rename(self, command):
        “”“Claude calls this to reorganize memories”“”
        # Move or rename files

The SDK provides these hook methods as the interface contract. We implement the file operations; Claude decides when to invoke them. No command parsing required from us.

The system prompt grants authority. Instead of telling Claude what to remember, we grant it authority:

This shift is profound. We’re not programming behaviors anymore. We’re setting intentions and trusting Claude to execute them. The orchestration that once consumed lines of code now happens autonomously, guided by our system prompt.

This hands-off approach of providing just the infrastructure hooks while delegating all decision-making to the LLM may seem minimal. That’s precisely the point.

If you’ve spent years writing deterministic code, this delegation feels uncomfortable. You’re trusting the LLM to make architectural decisions you once controlled. That discomfort is valid. Let’s add some clarity to the real world implications of this approach.

Under the Hood: Autonomous Decisions in Action

Let’s trace what actually happens during a conversation. These examples come from real sessions the companion app’s trace and render diagram features.

Creating New Memory

[You]: “I’m starting a new project. I’d like to build a web app for solo entrepreneurs
       to track their time and projects”

~Claude internally~: Creates new memory at path `project_solo_entrepreneur_app.txt`

[Claude]: “That’s an exciting project! A time and project tracker specifically for solo entrepreneurs ...”

You didn’t ask Claude to remember anything. Claude autonomously decided this information was worth preserving. It even made an architectural decision. It created a new project file rather than appending to a general profile. These are judgment calls the LLM now makes without any explicit rules from us beyond guidance in the system prompt.

Self-Correction

What happens when the LLM makes a mistake? In traditional systems, errors bubble up to users. Here’s how autonomous authority changes that dynamic:

[You]: “Let’s add these MVP features: client profiles, task tracking, time logging”

~Claude internally~: Tries to update `solo_entrepreneur.txt`,
gets FileNotFoundError, checks memory folder, finds correct
filename `project_solo_entrepreneur_app.txt`, successfully updates

[Claude]: “Got it! I’ve noted those MVP features for your project.”

Claude made a mistake with the filename, detected the error, diagnosed the issue, and self-corrected. All autonomously. No error messages to the user, no manual intervention required.

Our code deliberately re-raises exceptions to Claude rather than hiding them. This design turns errors into recovery opportunities. Claude receives the FileNotFoundError, understands what went wrong, and adapts its strategy. This resilience emerges naturally from giving the LLM authority rather than rigid rules.

Context-Aware Retrieval

The LLM decides when to check memories. No rules tell it to look for context on greetings, it just understands that’s useful:

[You]: “hello”

~Claude internally~: Checks memory folder, finds and reads
`user_preferences.txt` containing “Vacationing at the beach
as of 2025-11-03”

[Claude]: “Hi! How are things going at the beach today?”

Claude found relevant information and used it naturally in conversation.

These autonomous behaviors all emerge without us programming them explicitly. They include deciding on storage structure, recovering from errors, and retrieving contextual information.

Code vs. Prompts: Where Control Lives

The shift to implicit memory doesn’t mean abandoning all control. It means being strategic about where that control resides. By examining what stays in code versus what moves to prompts, we can understand the architecture of trust in AI systems.

Hard Boundaries in Code

The memory_tool.py file shows what explicit choices we’re still making in code:

Security Boundaries: The _validate_path() method explicitly ensures all operations stay within the /memories directory. This is hard-coded protection against path traversal attacks. It’s a security boundary we enforce in code, not through prompts.

Logging & Tracing: Every operation is explicitly logged for debugging and audit trails. We know every tool call Claude made to the memory system and what was read or written.

These coded constraints are examples of the guardrails we build to create a safe sandbox within which Claude operates autonomously. They’re guarantees enforced by our infrastructure, not suggestions in a prompt. In a production system you will expand on these guardrails until you have an acceptable risk level for your line of business.

The Power of Hooks: Beyond File Persistence

The SDK’s hook-based architecture (BetaAbstractMemoryTool) enables remarkable flexibility. While our implementation uses a filesystem backend, you could implement these same hooks to:

• Store memories in a PostgreSQL database for multi-user applications

• Use Redis for high-performance, distributed memory systems

• Implement vector embeddings for semantic memory retrieval

• Create hybrid systems that combine multiple storage backends

The LLM doesn’t care about the backend. It just calls the hooks and trusts the implementation to handle the details. This separation of concerns enables powerful architectural flexibility while maintaining the same autonomous decision-making paradigm.

Guidance Through Prompts

Between the hard constraints of code and the full autonomy of the LLM lies prompt-based guidance. These are strong suggestions that shape behavior without guarantees:

Keep your memory folder organized. Update existing files rather
than creating duplicates. Include metadata like dates when relevant.

When the memory folder exceeds 20 files, consolidate related
memories into broader topic files. Archive outdated information
with clear labels.

This achieves implicit memory maintenance without explicit commands such as !compact. The LLM monitors its own memory health and performs housekeeping autonomously.

For more structure in the memory files, we could specify a frontmatter format for all new memories. This creates a sort of pseudo-schema to be leveraged by the LLM and our instrumentation code:

All memory files should start with:
---
created: YYYY-MM-DD
relevance: current|archive
---

This gives Claude a framework without removing its autonomy. It can still make contextual decisions within these guidelines.

Making the Implicit Explicit: Explanatory Mode

For demonstration purposes in the companion chat app, I added explanatory versions of the prompts. These make the implicit decision-making visible for learning:

# Directive in the system prompt

IMPORTANT: You are a teaching tool to help the user understand how you
process and utilize memories. After each natural conversational response,
add a concise summary of the memory actions you took and why.

This produces illuminating results:

[You]: “ok, I think I’ll set a goal to finish the outline today, by noon”

[Claude]: “Excellent goal! Finishing the outline by noon today (2025-11-10) gives ...”

---
Memory Actions Taken:
- Updated article status to reflect current goal: completing the outline by noon today (2025-11-10)
- This helps track progress and provides context if we discuss the article later today or in future sessions

This explanatory mode becomes a powerful tool for understanding the LLM’s approach to memory and tuning your system prompts. It transforms the black box into a glass box, revealing the autonomous reasoning. Use it to validate that your prompt guidance is achieving the intended behaviors, then switch back to normal operation once satisfied.

The beauty of this division is that higher risk critical decisions stay in code where they’re guaranteed, while contextual decisions that benefit from intelligence and flexibility live in prompts. As models improve, the prompt-based behaviors get smarter automatically, while our security boundaries remain firm.

Conclusion: Learning from Scale

There’s a principle in AI research that keeps proving itself true. Systems built on general methods and scaled computation consistently outperform those with hand-crafted rules. Rich Sutton calls this “The Bitter Lesson”. Bitter because it means our clever, specialized solutions inevitably lose to simpler approaches that leverage raw intelligence.

The shift from explicit to implicit memory perfectly illustrates this principle. And it reveals what becomes possible when we stop fighting it.

The Power of Delegated Intelligence

We’ve seen the behaviors that emerge: autonomous memory creation, contextual reorganization, self-healing from errors. These weren’t programmed. They emerged from granting the LLM authority within safe boundaries.

This pattern extends beyond memory. It extends to complex decision-making in your application. Routing requests, organizing data, managing workflows. All can potentially be delegated to intelligence rather than encoded in logic. The infrastructure you build becomes a framework for capabilities you haven’t even imagined yet.

Hybrid model: Codify workflows that need precise control as tools, then let the LLM autonomously decide when and how to use them. This gives you explicit control over critical operations while still leveraging the LLM’s decision-making for orchestration.

As models improve, your system automatically gets better. When the next generation releases, you update one parameter. Your existing infrastructure suddenly makes smarter decisions. No refactoring. No new edge cases. The same hooks you implement today become more capable tomorrow.

Trust but Verify: Your Implementation Philosophy

This shift changes how we architect AI systems. Instead of writing decision trees, we adopt a “trust but verify” philosophy:

• Trust: Grant the AI authority through system prompts

• Verify: Monitor the behaviors that emerge

• Guide: Adjust prompts based on observed patterns

• Iterate: Refine boundaries as models improve

We’re reallocating the effort once spent on explicit control logic to validation and evaluation. Same total engineering effort, fundamentally better product. The interesting work moves from implementing specific behaviors to designing systems that exhibit emergent intelligence while maintaining appropriate guardrails.

The Path Forward

But here’s my challenge to you: run an experiment. The gap between what models can do and what we think they can do is often surprising. Many teams discover their explicit controls were solving problems the LLM could handle autonomously (and often better).

You can test this today. The companion app provides a complete learning laboratory:

• Multiple prompting strategies for testing delegation

• Full session tracing of every memory decision

• Visual sequence diagrams exported from conversations

• Real-time observation of memory folder organization

Clone it. Run it. Watch what emerges. Add your own prompts. Test your assumptions. Even if the model isn’t ready for your use case today, you’ll have the framework when the next version drops.

The original memory post showed that LLM memory is just text management. This exploration reveals a deeper pattern: we’re moving from programming behaviors to orchestrating capabilities. The question isn’t whether to trust AI with decisions. It’s understanding which decisions, with what boundaries, and how to monitor the results.

The tools are ready. The models are capable. The only thing standing between you and implicit memory is running that first experiment.

Want to explore further?

• Original approach: simple_llm_memory_poc

• Implicit approach: implicit-memory-system-poc

• Claude Agent SDK: Documentation

• Memory Tool: Official docs

• Anthropic on Agents: Building Effective Agents

About the Author: Sam Keen publishes at at AlteredCraft. Subscribe for content that dives into the paradigm shifts AI brings to software development in addition to a free weekly roundup of the latest AI Tutorials, Tool, and News relevant to software developers.

Now, scale up this problem. Imagine a massive distributed system where 10GB of data lives on each node, and every bit is replicated across five different nodes for reliability. When a network partition occurs and a node comes back online, is it really practical to sync all 10GB of data across every node, every time? Of course not.

This is the very challenge that engineers at Cassandra, DynamoDB, and even in blockchain technology have faced. How do you quickly identify and fix just the pieces of data that have changed, without wasting massive amounts of time and bandwidth checking everything, everywhere?

The secret lies in two powerful concepts: Merkle trees and anti-entropy protocols.

In this article, we’ll step into an engineer’s shoes — exploring how these data structures and algorithms allow distributed databases like DynamoDB and Cassandra to efficiently detect, compare, and synchronize changes. We’ll break down what Merkle trees and anti-entropy actually are, why they matter, and how you can implement them yourself (with hands-on examples in Golang).

Ready to see how modern distributed systems stay fast and consistent, even at a massive scale? Let’s dive in!

Understanding Merkle Trees

Imagine you’re the head of accounting at a global company with offices in 100 cities worldwide. Your responsibility: keep track of all expenses across every location. Reconciling records for every single office, every time, would be a logistical nightmare — not to mention a massive waste of time and resources.

To solve this, you create a smarter system:

• Instead of checking every office one by one, you appoint an assistant who keeps a running total.

• Now, you only compare your assistant’s master total with your own. If the totals match, you’re in sync. If not, you zero in on where the change happened.

• Taking it further, you group offices by regions (say, Asia-Pacific and USA). Each region reports upward. When there’s a mismatch, you drill down region by region, then country, then city.

This hierarchical, divide-and-narrow approach is precisely how Merkle trees work in distributed systems.

What is a Merkle Tree?

A Merkle tree is a data structure designed for efficient and secure verification of large sets of data. Here’s the basic anatomy:

• Leaf Nodes: Each one holds a hash of actual data — like an office’s transactions.

• Internal (Branch) Nodes: Each combines and hashes its children’s hashes, summarizing everything below.

• Root Node: The master hash — changing if anything underneath changes.

Key insight: Instead of comparing all data, you compare hashes at each level, honing in only where there’s a mismatch. This scales logarithmically, not linearly.

Historical Context:
Devised by Ralph Merkle in 1979, Merkle trees are now central to blockchains, distributed file systems, and database consistency.

How Merkle Trees Work

a. The Hashing Process

• Hashing the Leaves: Pass each data item (file, transaction, etc.) through a hash function (e.g., SHA-256). These are the leaf nodes.

• Building Up the Tree: Pair up the leaf hashes, concatenate and hash again to form parents, repeating until you reach the root.

• Checking for Changes: Any single data change changes its hash and all parents up to the root. Comparing roots lets you instantly check if two datasets match.

b. Example: Tiny Merkle Tree

Suppose you have four data blocks: A, B, C, D.
Step 1: Hash each block.
Step 2: Hash pairs: (A+B), (C+D).
Step 3: Hash those two: (AB + CD) = Root.
If just B changes, only three hashes need recalculating, and you can precisely spot the change.

Real-World Applications of Merkle Trees

• Blockchains: Store all transactions for a block as a Merkle tree. The root in the block header allows anyone to verify a transaction’s inclusion.

• Versioned File Systems (e.g., Git): Every commit is represented as a Merkle root; differences between commits highlight only what’s changed.

• Distributed Databases (Cassandra, DynamoDB): Use Merkle trees for anti-entropy. Only out-of-sync segments are reconciled, not the full dataset.

Merkle trees guarantee tamper-evidence and make large-scale, bandwidth-efficient consistency possible.

What is Anti-Entropy?

Back to our global accounts team: over time, small differences creep into each office’s ledger — network delays, miscommunications, independent corrections. These inconsistencies, or entropy, must be regularly tracked down and fixed.
Anti-entropy is your systematic, efficient approach to reconciling just the differences, not everything.

Anti-Entropy Mechanisms

• Gossip Protocols: Random offices synchronize with each other, gradually spreading updates network-wide.

• Vector Clocks: Track who changed what, and when, to resolve conflicts precisely.

• Merkle Trees: Group and summarize records as hashes; compare just summaries first to find divergences efficiently.

How Merkle Trees Empower Anti-Entropy in Real Systems

This is exactly the strategy used by distributed databases like Cassandra and DynamoDB. Each node in the system summarizes its stored data using a Merkle tree. When it’s time to reconcile — maybe after a network glitch or data loss — nodes quickly compare root hashes, then only dive deeper where discrepancies are found. This process scales effortlessly, allowing massive companies (and databases) to keep their records in harmony with minimal effort.

So, instead of drowning in a sea of spreadsheets, your accounts team uses anti-entropy protocols — and especially Merkle trees — to work smarter, not harder, keeping the entire global ledger consistent, up-to-date, and audit-ready.

In essence:
Anti-entropy, led by the power of Merkle trees, turns the messy challenge of financial reconciliation across a giant organization into a fast, targeted process — ensuring global consistency with a fraction of the work. This is the same magic powering reliable, scale-proof distributed databases today.

Implementing a Merkle Tree in Code

To truly appreciate the power of Merkle trees in distributed systems, let’s walk through a practical implementation in Go. This example captures the fundamental operations: building the tree, generating cryptographic hashes, and efficiently identifying differences between trees.

Building the Merkle Tree

At the core, a Merkle tree organizes data blocks into a hierarchical hash structure. Our implementation breaks down like this:

• We start by dividing data into buckets (or partitions), each containing a fixed number of keys.

• Each bucket’s combined data is hashed, creating a leaf node.

• We then recursively pair leaf nodes, concatenate their hashes, and hash again to form parent nodes.

• This process repeats until a single root node remains, representing a compact summary of the entire dataset.

Here’s how these pieces fit together in code:

// MerkleNode represents a leaf or internal node with its hash and children.
type MerkleNode struct {
    Hash     string
    Left     *MerkleNode
    Right    *MerkleNode
    KeyRange []string  // Only non-empty for leaves, marks keys covered
}

// Tree manages the Merkle tree for a dataset or node partition.
type Tree struct {
    BucketSize int       // Number of keys per leaf bucket
    Root       *MerkleNode
    mu         sync.RWMutex  // Protects concurrent access
}

When you call the Build method with sorted keys and their corresponding data, it hashes data bucket by bucket to create leaves:

func (t *Tree) buildLeaves(keys []string, kvs map[string][]byte) []*MerkleNode {
    var leaves []*MerkleNode
    for i := 0; i < len(keys); i += t.BucketSize {
        end := min(i + t.BucketSize, len(keys))
        bucket := keys[i:end]
        
        // Concatenate keys and their values (hex encoded)
        data := “”
        for _, k := range bucket {
            data += k + “:” + hex.EncodeToString(kvs[k])
        }
        h := sha256.Sum256([]byte(data))
        
        leaves = append(leaves, &MerkleNode{
            Hash: hex.EncodeToString(h[:]),
            KeyRange: bucket,
        })
    }
    return leaves
}

The buildMerkle function then recursively builds parents by hashing pairs of child hashes, gracefully handling odd numbers of nodes:

func buildMerkle(nodes []*MerkleNode) *MerkleNode {
    if len(nodes) == 0 {
        return nil
    }
    if len(nodes) == 1 {
        return nodes[0]
    }
    var parents []*MerkleNode
    for i := 0; i < len(nodes); i += 2 {
        if i+1 < len(nodes) {
            data := nodes[i].Hash + nodes[i+1].Hash
            h := sha256.Sum256([]byte(data))
            parents = append(parents, &MerkleNode{
                Hash: hex.EncodeToString(h[:]),
                Left: nodes[i],
                Right: nodes[i+1],
            })
        } else {
            parents = append(parents, nodes[i])
        }
    }
    return buildMerkle(parents)
}

After building, the single root hash summarizes the whole dataset:

func (t *Tree) RootHash() string {
    t.mu.RLock()
    defer t.mu.RUnlock()
    if t.Root == nil {
        return “”
    }
    return t.Root.Hash
}

Comparing Trees to Identify Differences

One of the biggest benefits of Merkle trees is their ability to efficiently detect exactly where two datasets differ — without scanning everything. This is done by recursively comparing node hashes from the roots downward:

// Diff returns key ranges that differ between two Merkle trees.
func (t *Tree) Diff(other *Tree) ([][]string, error) {
    t.mu.RLock()
    defer t.mu.RUnlock()
    if t.Root == nil || other.Root == nil {
        return nil, errors.New(”cannot diff: tree(s) not built”)
    }
    diffs := make([][]string, 0)
    diffHelper(t.Root, other.Root, &diffs)
    return diffs, nil
}

// Helper to recursively collect differing leaf key ranges.
func diffHelper(a, b *MerkleNode, diffs *[][]string) {
    if a == nil || b == nil {
        return
    }
    if a.Hash == b.Hash {
        return // subtree matches; no difference
    }
    if a.Left == nil && a.Right == nil && b.Left == nil && b.Right == nil {
        // Both leaves differ; record key range
        *diffs = append(*diffs, a.KeyRange)
        return
    }
    if a.Left != nil && b.Left != nil {
        diffHelper(a.Left, b.Left, diffs)
    }
    if a.Right != nil && b.Right != nil {
        diffHelper(a.Right, b.Right, diffs)
    }
}

This targeted approach means two large datasets can reconcile efficiently by syncing only the data within the mismatched key ranges.

Summary

This Go implementation demonstrates how Merkle trees:

• Build a cryptographic hash tree from raw data in buckets to leaves to root.

• Provide a single summary root hash for efficient data verification.

• Enable fast difference detection by recursive hash comparison, isolating only exact data shards that need syncing.

Understanding and implementing these core operations is the first step toward applying Merkle trees for secure, scalable data consistency in real distributed systems.

Using Merkle Trees for Anti-Entropy: Step-by-Step

1. Each node builds its Merkle tree over its dataset.

2. Nodes exchange root hashes. If equal, they’re in sync; if not, mismatches exist.

3. Compare children hashes recursively until you reach mismatched leaves.

4. Sync only the data for mismatched leaves, not the entire dataset.

5. Rebuild trees and repeat as needed. This gives you fast, granular repair.

Performance Considerations

Efficiency:

• Merkle trees transform O(n)O(n) comparisons into O(log⁡n)O(logn), minimizing bandwidth and compute, critical at scale.

Trade-offs:

• Depth & Bucket Size: Small buckets yield deep trees and fine granularity for repairs; large buckets are faster to build but coarser.

• Branching Factor: Binary is simple, but higher factors further reduce tree height.

• Hash Function: Use cryptographic hashes for security, but know they’re slower than simple checksums.

Best Practices:

• Keep keys sorted and buckets consistent.

• Regularly audit cryptographic choice and correctness.

• Avoid over-deep trees or inconsistent bucketing, as these kill performance.

Case Study: Cassandra’s Anti-Entropy Repair

Scenario: A network partition causes one node to lag. When it rejoins, Cassandra must reconcile data — efficiently.

Process:

• Each node builds a Merkle tree for a partition.

• Nodes exchange Merkle roots; if mismatched, they compare subtrees.

• Drill down recursively until mismatching leaves found; sync only those ranges.

Visual Flow:

1. Build trees

2. Exchange and compare roots

3. Drill to mismatched children

4. Sync only affected ranges

Conclusion

Merkle trees are the backbone of fast, scalable, and resilient data synchronization in distributed systems. They enable efficient, targeted anti-entropy repairs, keeping massive databases like Cassandra and DynamoDB consistent with minimal overhead — even in the face of network partitions or data loss.

As data volumes grow and systems span continents, intelligent anti-entropy protocols will only become more crucial. The future? Expect even tighter cryptographic guarantees, hybrid data structures, and smarter reconciliation engines — keeping the world’s distributed data trustworthy and lightning-fast.

Stay Connected!

• 💡 Follow me on LinkedIn: Archit Agarwal

• 🎥 Subscribe to my YouTube: The Exception Handler

• 📬 Sign up for my newsletter: The Weekly Golang Journal

• ✍️ Follow me on Medium: @architagr

• 👨‍💻 Join my subreddit: r/GolangJournal

• 💡 Follow me on Twitter: @architagr

We will cover the following main topics:

• GitHub security features

• Managing access and permissions

• Best practices for repository security

GitHub security features

In this section, we’ll explore the various security features GitHub offers to help you protect your repositories and manage user access effectively.

Two-Factor Authentication (2FA)

Two-factor authentication adds an extra layer of security to your GitHub account beyond just your password. By requiring a second form of verification, it ensures that even if someone gets hold of your password, they won’t be able to access your account without the second factor.

2FA is crucial for securing your account for enhanced security, mitigating against credential theft, protecting against phishing, and complying with security standards in many organizations and projects. Which 2FA methods are configurable on GitHub?

Available 2FA methods

GitHub offers several methods for enabling Two-Factor Authentication (2FA) to enhance the security of your account. Here are the available 2FA methods:

• Time-Based One-Time Password (TOTP) authenticator apps:

  • Use apps such as Google Authenticator, Authy, or Microsoft Authenticator to generate a time-based code

  • Recommended for its reliability and security

• Short Message Service (SMS):

  • Receive a verification code via text message

  • Less secure compared to TOTP apps, but still an option

• Physical security keys:

  • Use hardware devices such as YubiKeys that support FIDO U2F or WebAuthn standards

  • Provide a high level of security by requiring physical possession of the key

• Virtual security keys:

  • Utilize built-in security features of personal devices, such as Windows Hello, Face ID, or Touch ID

  • Convenient and secure, leveraging device-specific authentication

• GitHub Mobile:

  • Use the GitHub Mobile app to authenticate using public-key cryptography

  • Does not rely on TOTP and provides a seamless experience

These methods offer flexibility and varying levels of security, allowing you to choose the one that best fits your needs. For the highest security, it’s recommended to use TOTP apps or physical security keys. Let’s take a look at how to set this up.

Certification tip

The GitHub Foundations exam often includes questions on 2FA setup and best practices. Make sure you understand both how to enable 2FA and how to use recovery methods.

Setting up 2FA on GitHub

To add an extra layer of security to your GitHub account, enable two-factor authentication (2FA) by following these steps:

1. Go to your GitHub individual account settings by clicking on your avatar in the top-right-hand corner and clicking on Settings.

2. In the left sidebar, under Access, click on Password and authentication.

3. If not enabled yet, select Enable two-factor authentication.

4. Under Two-factor authentication, click Add next to the 2FA method of choice.

5. Follow the prompts to set up 2FA using the method selected.

Optionally, you could set your preferred 2FA method if you enrolled in more than one.

That’s it! You’re all set up.

You will also notice Recovery codes under the Recovery options section (this section will appear only if the 2FA method is set).

Recovery codes are essential for regaining access to your GitHub account if you lose access to your 2FA credentials. These codes act as a backup method, allowing you to log in even if you can’t use your primary 2FA method, such as an authentication app or SMS. When you enable 2FA, GitHub provides a set of recovery codes that you should store securely, such as in a password manager or a safe place. If you ever lose access to your 2FA device, you can use one of these recovery codes to regain entry to your account, ensuring you are not permanently locked out.

GitHub gives you 8 recovery codes. Store them securely. You can regenerate these if needed, but old ones will be invalidated.

Branch protection rules

We discussed branch protection rules extensively in Chapter 5, Branching and Merging Strategies. Be sure to read this in preparation for your exam.

Branch protection rules help you enforce certain workflows and requirements before changes can be merged into your protected branches. This ensures that your codebase remains stable and secure. You can configure branch protection rules and, among many other measures, enforce code reviews, ensuring that all changes are reviewed and approved before they are merged.

Certification tip

You’ll need to know how to configure branch protection rules and enable Dependabot alerts for the exam.

Security configurations

GitHub provides various security settings as a collection that you can configure to enhance the security of the repositories in your organization. You can create a customized security configuration from scratch or choose the GitHub-recommended configuration that already comes preset with its settings.

GitHub-recommended security configurations are predefined settings that follow best practices to enhance security, such as enabling Dependabot alerts and secret scanning by default. Custom configurations, on the other hand, allow you to tailor security settings to meet specific needs or requirements of your project or organization, providing flexibility to adjust features such as branch protection rules and access controls.

To manage security settings at the organization level:

1. Go to your organization’s main page on GitHub (remember that this is an organization, not a repo). For more information on how to create an organization, review Lab 2.1 in Chapter 2, Navigating the GitHub Interface.

2. Click on Settings.

3. In the left sidebar, under Security, click Advanced Security to expand.

4. Then click on Configurations.

5. Choose to edit the GitHub-recommended security configuration by clicking on the edit (

   

   ) icon. Alternatively, you can click on New configuration to customize a new one.

6. Configure the security settings as needed.

To manage repository-specific security settings:

1. Navigate to the repository’s main page on GitHub.

2. Click on Settings.

3. In the left sidebar, under Security, click Advanced Security.

4. Enable or configure security features such as Dependabot alerts, secret scanning, and code scanning. The latter two may be missing from your view if the repository is private or internal.

   Secret scanning and code scanning are GitHub Advanced Security features and are only available as a paid subscription for private or internal repositories, or free if your repository is open source, that is, visibility is Public.

5. Optionally, if you wish to configure secret scanning or code scanning, click on General from the left sidebar and scroll down to Danger Zone. Next to Change repository visibility, click on Change visibility and select Change to public. Be sure to follow the instructions.

Certification tip

Questions may come up regarding changing the visibility of a repository from public to private, or vice versa. Be sure to understand the implications of making a repo public. Read more about public repositories in Chapter 2, Navigating the GitHub Interface and Chapter 11, Contributing to Open Source Projects.

Dependabot, secret scanning, and code scanning are all examples of security features that GitHub offers. Some of these features require a paid subscription, some are free only for public repositories, while others are completely free out of the box.

What is Dependabot?

Dependabot is a feature on GitHub that helps keep your project’s dependencies up to date automatically. It works by regularly checking your project’s dependency files (such as package.json, requirements.txt, etc.) for outdated packages and then creating pull requests to update them to the latest versions.

Key features of Dependabot include the following:

• Automated dependency updates: It scans your project and creates pull requests to update dependencies

• Security alerts: It integrates with GitHub’s security features to alert you about vulnerabilities in your dependencies and can automatically fix them

• Customizable configuration: You can configure how often it checks for updates, which dependencies to ignore, and more, using a dependabot.yml file

• Supports multiple languages: Works with JavaScript, Python, Ruby, Java, PHP, and more

Here is an example flow:

1. You enable Dependabot in your GitHub repository.

2. It checks for outdated or vulnerable dependencies.

3. It creates a pull request with the updated version.

4. You review and merge the pull request.

Now let’s talk about managing alerts and vulnerabilities.

Security alerts and vulnerability management

GitHub helps you stay on top of potential security issues with automated alerts and tools to manage vulnerabilities. Let’s examine some of these:

• Dependabot alerts and security updates:

  • Dependabot alerts: Automatically scans your dependencies for known vulnerabilities and notifies you if any are found

  • Dependabot security updates: Automatically generates pull requests to update vulnerable dependencies to secure versions

How can you enable and manage Dependabot alerts?

1. Go to the repository’s main page on GitHub.

2. Click on Settings.

3. In the left sidebar, under Security, click Advanced Security.

4. Under Dependabot alerts, click Enable if not already enabled.

5. Afterward, you can review and manage alerts from the Security tab of the repository.

• Code scanning:

  • CodeQL: A powerful code analysis engine that scans your code for security vulnerabilities and coding errors. It integrates with GitHub Actions to run scans on every push or pull request.

  • Autofix: Uses AI to suggest fixes for detected vulnerabilities, streamlining the remediation process.

• Secret scanning: Detects and alerts you if sensitive information, such as API keys or passwords, is accidentally committed to your repository. This helps prevent unauthorized access and potential security breaches.

• Security overview dashboard: Provides a centralized view of your security alerts and vulnerabilities across all repositories. This dashboard helps you prioritize and manage security issues more effectively.

• Vulnerability management integrations: Integrates with third-party vulnerability management tools to consolidate and prioritize vulnerabilities, automate risk mitigation, and visualize alerts within your existing security posture.

Talking about third-party integrations, GitHub supports receiving Static Analysis Results Interchange Format (SARIF) reports from various third-party security tools. Some of the commonly used tools include the following:

• ESLint: A popular tool for identifying and reporting on patterns found in ECMAScript/JavaScript code

• Bandit: A tool designed to find common security issues in Python code

• Brakeman: A static analysis tool that checks Ruby on Rails applications for security vulnerabilities

• Checkmarx: A comprehensive Static Application Security Testing (SAST) tool

• Fortify: A suite of tools for static and dynamic application security testing

• SonarQube: An open source platform for continuous inspection of code quality

• Veracode: A cloud-based service for static and dynamic application security testing

These tools generate SARIF files that can be uploaded to GitHub, allowing you to view and manage security alerts directly within your repository.

To handle security advisories and alerts, navigate to the Security tab of the repository (you will find this tab on both the organization and the repository levels). Examine the difference between the Security tabs of both levels. You will notice a stark difference in what you see. This is because the security overview at the organization level rolls up all the security advisories across all its repos, whereas the scope of the repo level is limited to only vulnerability findings of that repo.

These tools and features help you maintain a robust security posture by automating the detection and management of vulnerabilities, ensuring your codebase remains secure.

In the next section, we will talk about how permissions and authorization can help in securing your code.

Managing access and permissions

Effective management of access and permissions is crucial for maintaining the security and integrity of your GitHub repositories. This section delves into the various methods GitHub provides to control who has access to your repositories and what they can do.

User roles and permissions

GitHub offers a range of user roles to help you manage access and permissions effectively. Understanding these roles is key to maintaining a secure and organized workflow.

Overview of different user roles

User roles on GitHub come in three tiers: roles at the Enterprise level, Organization level, and Repository level. Here are the default roles at the Enterprise and Organization levels:

• Owner: The owner has full administrative access to the organization and its repositories. This role can manage settings, users, and billing.

• Member: Members have basic access to repositories, typically for contributing code. They can create issues, submit pull requests, and review code.

• Billing manager: Billing managers can manage billing settings such as changing billing plans, managing payment methods, downloading and receiving receipts, or managing sponsorships.

When inviting new collaborators to your organization for the first time, you choose one of these three.

Certification tip

Be prepared to identify role-based use cases. The exam may ask you to match specific user scenarios with appropriate GitHub roles.

Assigning roles to users

To manage access within your organization, you can assign specific roles to members by following these steps:

1. Navigate to your organization’s main page on GitHub.

2. Click on the People tab in the organization’s navigation bar.

3. If the user doesn’t already exist in the organization, you can invite them by clicking on Invite member.

4. Supply the user’s GitHub handle and click on Invite.

5. For an existing member, locate the user you want to assign a role to and click on the ellipsis dropdown next to their name and select Change role….

6. Select the appropriate role and click on Send invitation (for new invitations) or Change role (existing members).

1. Click on Change role…

2. Select the appropriate role (Owner or Member) from the list displayed.

3. Click on Change role.

4. Alternatively, if the user only needs to be a billing manager, click on the Invite a billing manager link at the bottom of the user invitation screen (Step 3), or go to the organization’s settings and select Billing and licensing from the left navigation bar and invite them.

In addition to these three, GitHub provides some more granular roles that help you define granular permissions to what a member can or cannot do at the different levels.

Here is a table of additional pre-defined roles that can be used:

In addition to these predefined roles, you can create custom roles with a select combination of permissions if one of these predefined roles doesn’t exactly fit your needs. This can be done at the Enterprise, Organization, and Repository levels.

The custom role feature is only available on GitHub Enterprise. In addition, the custom role at the Enterprise level is only available for preview as of early 2025. Refer to GitHub Docs for the latest availability.

Now let’s move on to using teams for access control.

Team management

Teams allow you to group users and manage their access to repositories more efficiently. This is particularly useful for larger organizations with multiple projects. A team can either be visible or secret.

Visible teams can be seen and @mentioned by members of the organization, while secret teams can only be seen by their members. This will be specified during team creation. Teams can also be nested, with one team being the parent of another team.

Let’s see how we create teams.

Creating and managing teams

Here’s how to create a team:

1. Go to your organization’s main page on GitHub.

2. Click on Teams in the organization’s navigation bar.

3. Click New team to create a new team.

4. Enter the team name and description, then click Create team.

5. Add members to the team by clicking Add a member and selecting users from the list.

Here are some important use cases for team visibility and notifications:

Assigning repository access to teams

To assign access to repositories to teams, do the following:

1. Navigate to the team’s page on GitHub.

2. Click on the Repositories tab in the team’s navigation bar.

3. Click Add a repository and select the repository you want to grant access to.

4. Set the desired permission level (Read, Triage, Write, or others) for the team. More roles and the ability to create custom roles are available if you have a paid GitHub Enterprise subscription.

Collaborator access control

Collaborators are individuals who are granted access to specific repositories. This is useful for managing external contributors or contractors.

Adding collaborators to repositories:

To give others access to a specific repository without adding them to the entire organization, you can add them as collaborators:

1. Navigate to the repository’s main page on GitHub.

2. Click on Settings in the repository’s navigation bar.

3. In the left sidebar, click Collaborators and teams (org-owned repos) or Collaborators (user-owned repos).

4. Click Add people and start typing the username of the person you want to add.

5. Select their name and set the appropriate permission level (Read, Triage, Write, Maintain, Admin) for the collaborator.

Setting permissions for collaborators:

Once collaborators are added, you can adjust their access levels using the following steps:

1. In the Collaborators and teams section, find the collaborator you want to manage.

2. Click on their permission dropdown and select the desired permission level.

One common pitfall is misconfigured roles or permissions, which can inadvertently expose sensitive code. For example, imagine a scenario where a developer creates a private repository for an internal tool but mistakenly assigns a Read role to an external contractor at the organization level. Because the repository inherits permissions from the organization, the contractor now has unintended access to the private repository. This kind of oversight can lead to accidental data leaks or unauthorized code access.

To prevent such incidents, always follow the principle of least privilege, regularly audit repository access, and use fine-grained personal access tokens for automation and integrations.

Another great feature to manage access and permissions is tokens. Two examples of tokens on GitHub are OAuth and Personal Access Tokens (PAT).

OAuth and personal access tokens

OAuth and personal access tokens provide secure ways to authenticate and authorize access to your GitHub account and repositories. These methods are essential for integrating third-party applications and services, that is, when you are not using an interactive login of a person. Here’s where you’ll find each of these on GitHub:

Managing OAuth applications

To review and manage third-party applications connected to your GitHub account, follow these steps:

1. Go to your GitHub account settings.

2. In the left sidebar, click Developer settings.

3. Click OAuth Apps to view and manage your OAuth applications.

4. Review the list of authorized applications and revoke access if necessary.

Creating and using personal access tokens

To authenticate non-interactive scripts or services, you can generate a personal access token as shown below:

1. Go to your GitHub account settings.

2. In the left sidebar, click Developer settings.

3. Click Personal access tokens.

4. Select either Fine-grained tokens or Tokens (classic) in the submenu.

5. Click Generate new token (for fine-grained tokens) or select Token (classic) again for the classic token option.

6. Select the scopes or permissions you want to grant this token, such as repo, admin:org, or user.

7. Click Generate token and copy the token for use in your applications. Store it securely, as it will not be displayed again.

You would notice by now that there are two types of personal access tokens on GitHub: classic and fine-grained.

Both types co-exist, with classic being the older. It is expected that GitHub will deprecate classic PAT in favour of fine-grained PAT in the future, but both of them can be used interchangeably today. Let’s quickly enumerate the differences between the two.

In summary, GitHub offers a range of measures to manage access and grant permissions to users and third-party applications and services. It is important to know which ones to combine to ensure the security of your code. Up next, let’s consider some good security practices for repo security.

Best practices for repository security

Ensuring the security of your repositories is paramount to protecting your code and maintaining the integrity of your projects. In this section, we’ll cover best practices for securing your repositories on GitHub.

Code scanning with Static Application Security Testing (SAST) tools

Static Application Security Testing (SAST) tools help identify security vulnerabilities in your codebase before they become issues in production. GitHub has a code scanning product. It is sold separately but can be integrated natively into your repos and organizations. It’s labelled GitHub Advanced Security (GHAS). GHAS is a paid subscription product, an add-on that you can purchase in addition to your GitHub subscription. If your codebase is open source, most of GHAS’s security features are free for use.

Here’s how to integrate and use these tools. You have to do this on a public repo if you haven’t purchased GHAS:

• Integrating SAST tools in your workflow: Here’s how to set up code scanning with SAST tools:

  1. Navigate to the repository’s main page on GitHub.

  2. Click on Security in the repository’s navigation bar.

  3. Click Set up code scanning.

  4. Choose a code scanning tool, such as CodeQL, and follow the prompts to configure it.

Examples of SAST tools include the following:

• CodeQL: A powerful code analysis engine that scans your code for security vulnerabilities and coding errors

• Dependabot Alerts: Automatically scans your dependencies for known vulnerabilities and notifies you if any are found

• Interpreting scan results: To review and act on scan results, take the following steps:

  1. Go to the Security tab of the repository.

  2. Click on Code scanning alerts to view the results.

  3. Review the alerts and take appropriate action to fix the identified issues.

CI/CD pipeline security measures

Securing your Continuous Integration/Continuous Deployment (CI/CD) pipelines is crucial to ensure that your code remains secure throughout the development lifecycle. Follow these steps to secure your pipelines:

• Securing CI/CD Pipelines Using GitHub Actions: To secure your CI/CD pipelines, take these steps:

  1. Navigate to the repository’s main page on GitHub.

  2. Click on Actions in the repository’s navigation bar.

  3. Set up workflows to include security checks, such as running SAST tools or dependency checks.

Mini-case example

A development team noticed that their CI/CD pipeline was deploying code with outdated dependencies that had known vulnerabilities. The issue stemmed from a missing dependency scanning step in their GitHub Actions workflow. To mitigate this, they integrated Dependabot and added a step in their workflow to run npm audit during each build. This change helped catch vulnerable packages early and prevented insecure code from reaching production.

• Implementing secrets management in workflows: To manage secrets securely in your workflows, take these steps:

  1. Go to the repository’s main page on GitHub.

  2. Click on Settings.

  3. In the left sidebar, click Secrets and variables.

  4. Then, select Actions from the submenu.

  5. Click New repository secret to add secrets, such as API keys or tokens, securely.

  6. Reference these secrets in your GitHub Actions workflows to avoid exposing sensitive information.

Monitoring and auditing activities

Regular monitoring and auditing of repository activities help you detect and respond to suspicious actions promptly. Here’s how to monitor and audit activities:

• Using audit logs to monitor repository activities: To monitor repository activities, take these steps:

  1. Go to your organization’s main page on GitHub.

  2. Click on Settings.

  3. In the left sidebar, under the Archive section, click Logs,

  4. Then, select Audit log from the submenu.

  5. Review the audit log entries to monitor activities such as user logins, repository changes, and permission updates.

• Setting up alerts for suspicious activities: To set up alerts for suspicious activities, take these steps:

  1. Use GitHub’s built-in security alerts to notify you of potential security issues.

  2. Integrate with third-party monitoring tools to receive real-time alerts for suspicious activities.

Incident response and recovery

Being prepared for security incidents and having a plan for recovery is essential for minimizing the impact of security breaches. Follow these steps for incident response and recovery:

• Preparing for security incidents: To prepare for security incidents, take these steps:

  1. Develop an incident response plan that outlines the steps to take in case of a security breach.

  2. Ensure that all team members are aware of the plan and their roles in the response process.

• Steps for incident response and recovery: To respond to and recover from security incidents, take these steps:

  1. Identify and contain the breach to prevent further damage.

  2. Investigate the cause of the breach and assess the impact.

  3. Remediate the vulnerabilities that led to the breach.

  4. Communicate with stakeholders and provide updates on the incident and recovery efforts.

  5. Review and update security policies and practices to prevent future incidents.

Et voila! This concludes the basics when it comes to security on GitHub. Let’s summarize what we learned.

Summary

In this chapter, we delved into the intricacies of security practices and user management on GitHub. We had already mastered the essentials of effective collaboration, but we know that with great code comes great responsibility. We explored the robust security features GitHub offers, such as two-factor authentication, which added an extra layer of security to our accounts. We learned about the various methods available for 2FA, including authenticator apps, SMS, physical security keys, and GitHub Mobile.

We also discussed branch protection rules, which ensured our codebase remained stable and secure by enforcing workflows and requirements before changes could be merged. Additionally, we examined security configurations, both GitHub-recommended and custom, to enhance the security of our repositories. We looked at the management of security alerts and vulnerabilities through tools such as Dependabot, CodeQL, and secret scanning, which helped us stay on top of potential security issues.

Managing access and permissions was another crucial aspect we covered. We understood the importance of user roles and permissions at different levels, from enterprise to repository, and how to assign these roles effectively. We also explored the use of teams for access control, creating and managing teams to streamline our workflow. Finally, we looked at OAuth and personal access tokens, which provided secure ways to authenticate and authorize access to our GitHub account and repositories.

Overall, this chapter equipped us with the knowledge and tools to maintain a robust security posture and manage user access effectively on GitHub.

Let’s do a short quiz.

Test your knowledge

Review all GitHub security and user management features, especially permission models, 2FA, and CI/CD hardening techniques – they appear frequently on the certification.

1. Which of the following methods is considered the most secure for enabling Two-Factor Authentication (2FA) on GitHub?

   1. Short Message Service (SMS)

   2. Time-Based One-Time Password (TOTP) authenticator apps

   3. Virtual security keys

   4. Physical security keys

2. What is the primary purpose of Dependabot Security Updates in GitHub?

   1. To scan your code for security vulnerabilities and coding errors

   2. To automatically generate pull requests to update vulnerable dependencies to secure versions

   3. To detect and alert you if sensitive information is accidentally committed to your repository

   4. To provide a centralized view of your security alerts and vulnerabilities across all repositories

3. Which role in GitHub is best suited to managing security policies, security alerts, and security configurations for an organization and all its repositories?

   1. Owner

   2. Security manager

   3. Admin

   4. CI/CD admin

Useful links

• Authentication documentation: https://docs.github.com/en/enterprise-cloud@latest/authentication

• About GitHub security features: https://docs.github.com/en/enterprise-cloud@latest/code-security/getting-started/github-security-features#about-githubs-security-features

• About GitHub Advanced Security: https://docs.github.com/en/enterprise-cloud@latest/get-started/learning-about-github/about-github-advanced-security

To build practical mastery of Git and GitHub— from version control basics to collaborative workflows, secure automation, and AI-assisted productivity—check out GitHub Foundations Certification Guide by Ayodeji Ayodele (Packt, 2025). Through step-by-step labs, real-world projects, and exam strategies, it helps you prepare for the GitHub Foundations certification while adopting best practices for issues and pull requests, GitHub Projects, privacy and security controls, and GitHub Copilot—so you can level up your skills and ship better software, faster.

Here’s what some readers have said:

The concept of Application Development Framework (ADF) has been well-known for a long time, but we need to set up a crystal-clear context for further reading. This is important because it helps us deal with this complex topic by setting common ground for definitions and classifications that will be used throughout the book. First, we explore the evolution of the idea of ADF. After that, we discover the differences and connections between other SDLC-focused technologies, such as Platforms, Libraries, SDKs, and APIs, to craft a brief but concise definition that helps us keep a big picture while diving deep into implementation topics. Then, we review the place of ADF in the Software Development Lifecycle to identify and prove the advantages of adopting an ADF.

In this chapter we’re going to cover the following main topics:

• Introduction and historical references

• Breaking down Application Development Framework

• Exploring ADF and Platforms, Libraries, SDKs, APIs

• Integrating into Software Development Lifecycle (SDLC) and Flow

• Differentiating ADF and other types of Frameworks

Introduction and historical references

Engineers have a long and productive history of creating building blocks for their own convenience. If we do not ignore this historical experience, we can learn many useful lessons for creating our own frameworks.

From the very beginning of the software industry, engineers and scientists have had a tendency to reuse their most successful and efficient ideas. There are quite a few historical practices that share the core objectives of a framework:

• Architectural Blueprints: Since ancient times, complex structures like buildings or ships were built based on detailed plans. These plans defined the overall structure, components, and relationships - similar to how frameworks provide a blueprint for software architecture.

• Modular Design in Engineering: Even before the computer age, engineers approached complex machines with a modular mindset. Think of early steam engines with interchangeable parts - a principle that carries over to software components within a framework.

• Mathematical Frameworks: For centuries, mathematicians have relied on established frameworks like algebra or calculus to solve problems. These frameworks provide a set of rules and structures that guide the approach to solving a specific type of problem.

While these aren’t direct equivalents to software frameworks, they all represent historical approaches to structuring complex systems in a way that aligns with the core function of a software development framework—to simplify the lives of its users when dealing with complex problems.

With the advent and adoption of computers, the concept of a framework has gone beyond the art of the elite and has become part of the daily work of many programmers. The idea evolved as computers themselves developed. Here are some contenders for the title of earliest software framework:

• Early Subroutine Libraries (1940s – 1950s): In the early days of computing, programmers might develop reusable code blocks for common tasks like mathematical functions or input/output routines. These weren’t full-fledged frameworks, but they offered a basic level of reusability and structure.

• FORTRAN Compilers (1950s): FORTRAN introduced the concept of high-level languages, allowing programmers to write code that is more human-readable than machine code. While not exactly a framework, it provided a foundational structure for building software.

• Operating Systems (1960s onwards): Operating systems like IBM’s OS/360 offered a platform for running applications. They provided core functionalities like memory management and device drivers, which later frameworks were built upon.

It’s important to remember that the concept of a software development framework as we know it today – offering a comprehensive set of tools, libraries, and design patterns – is a more recent development. However, these earlier practices laid the groundwork for the frameworks we use today. In the modern world, we can only imagine the practical purpose software created on top of one or multiple frameworks.

Note

This book uses both terms “software development framework (SDF)” and “application development framework (ADF)” interchangeably. Usually, “application” is not exactly the same as “software”: we have platforms, libraries, SDKs, engineering tools and frameworks as alternative kinds of software. But in the context of the topic (“building frameworks”) we can always safely assume that any “software” we are going to develop with our frameworks will serve the same purpose as “application” with a minor exception of “infrastructure management frameworks” which are mentioned explicitly.

While most of the information on the internet about Application Development Frameworks is focused on web and mobile development, we cannot ignore trending frameworks from a non-application software, such as

• Artificial Intelligence and Machine Learning (like PyTorch, TensorFlow, and Apache MXNet),

• Scheduled task management (like Celery, Temporal, and Apache Airflow),

• Infrastructure management (like Terraform, Pulumi, and Crossplane),

• Testing automation (like Selenium, Robot, and webdriverIO),

and many others, including “hybrid” frameworks that provide multiple capabilities at once.

Fortunately, foundational principles of building software frameworks are common between classic ADF and these emerging types of SDF.

There are also vertical ADFs, aiming to cover corresponding business domains. Examples of such frameworks include Gamedev (Flame, Monogame), Data Visualization (Shiny, Seaborn, TensorBoard), Hardware Instrumentation (LabVIEW), etc.

A StackOverflow research in 2023 that involved approximately 90,000 software engineers provided us with data about frameworks they use daily (See Figure 1.1 that summarizes one of the framework-related topics from this survey). In addition to those numbers, we know that many frameworks have their own communities outside of StackOverflow, which means that more than 100k engineers work with software frameworks on a daily basis.

Figure 1.1: StackOverflow research summary chart

With all the wide adoption of ADFs, it is confusing to see how many different inconsistent definitions and classifications we have all around the internet. In the following section, we will craft a brief and concise definition based on ADFs unique differentiators in a world of software engineering.

Breaking down Application Development Framework

I often see engineers mixing up libraries, frameworks, and Software Development Kits. They use these words like they mean the same thing. Even more confusion can come if we add APIs, platforms, and DSLs to the conversation. But to make our own framework, we need to understand all these things.

According to Dictionary.com, the formal definition of the term “framework” is “a basic structure, plan, or system, as of concepts, values, customs, or rules”.

The collective unconscious of humanity, also known as LLM, suggests the following definition for ADF: An application development framework is a software library offering a fundamental structure for building applications within a specific environment. It acts as a reusable foundation, supplying pre-defined functionalities and promoting code organization through established conventions. This approach streamlines development by reducing repetitive coding efforts.

Both definitions are formally correct (except for “library” part of the second one, which I will explain later in this section). But they focus on how the framework is designed but have a lack of explanation about how it works. I am going to fill this gap by adding my own:

Definition

A collection of pre-written code and tools that provide a structured approach to building applications. It simplifies development by enforcing architectural patterns: frameworks always dictate an execution flow, and stipulate specific way to structure your code, promoting maintainability, testability, low coupling, and reusability.

It is a common misunderstanding to confuse frameworks with other engineering concepts aiming towards reusability, like API, software library, SDK, and platform. And there is always a special “tooling” category, which covers a wide range of software from smaller console scripts to powerful configurable logs processing pipelines – they all live their own life as they only used by developers to support their routine tasks, so we keep them out of conversation. Let’s set clear boundaries to understand their differences to focus on the most important aspects of our topic.

Application Programming Interface

Starting from API (Application Programming Interface) as the lowest-level implementation of the development tooling. The traditional understanding of API included any exposed interface available to software developers to perform manipulation with an external subsystem. This external subsystem was treated as a “black box,” which means that the developer should not worry about its internal implementation, tech stack, and logic. Thus, API provides a complete set of methods to deal with it. Modern understanding of the API concept drifted towards over-the-network API, like HTTP/gRPC/websocket APIs. Events and message-based communication interfaces are also subsets of APIs – like webhooks.

The best practice of API definition is to use open standards like OpenAPI and AsyncAPI schema languages, or other less popular languages like RAML.org or APIBlueprint.org. However, it is acceptable to use proprietary or vendor-specific tools. API concept can also be visualized with a simple diagram notation (see Figure 1.2 below). Typical representatives of the API are as follows:

• SaaS products’ interfaces, e.g. PandaDoc API, or OpenAI LLM API

• Cloud management APIs; e.g. AWS Cloud Control API , or Azure REST API, or Google’s Service Management API

• Webhooks, like Zapier

• Internal/proprietary messaging-based events and commands schema registries that can be based on Confluent or Redpanda

Figure 1.2: Concept-level diagram of API

The diagram helps to see that the API purpose is to provide access to exposed “black box” functions.

Library

The next one to review is a software library as a collection of pre-written code or routines that developers can use to perform specific tasks or functions within their software applications.

Sometimes developers see any software library as a framework, but the purpose of the library is completely different – it focuses on runtime/operation concerns by implementing a common part of the system, like hardware I/O operations, network protocol, authorization sequence, ranking algorithm, IoT standard, etc. It is also common to have a library to transform a low-level API into a more developer-friendly form by adding enumerables, constants, and conditional logic over a binary code and method signatures of plain API. A Library usually operates as a gray box, which means that software developers can see its internal implementation, but it is rarely necessary. In some cases, libraries can come in binary format, which makes them black boxes.

Terminology across the industry is not always consistent, we can find other synonyms for the term “software library”:

• Package; usually means one or multiple software libraries that share the same license and they can be distributed as a single unit.

• Module; usually means a built-in software library, distributed with the program.

• Extension (aka add-on or plug-in); usually means a software library that follows specific program interface allowing external developers to modify original program behavior without changing any code in the original system.

Another important consideration is control flow. For a software library, it is common for developers to have full control over the library functions – so developers are responsible for invoking the library.

Writing software libraries is one of the most common tasks in the industry; many senior developers have experience of creating libs for internal company purposes, or contributing to open-source libs, or at least have them as a part of their pet projects.

Often, a software library evolves into an SDK or a framework after multiple iterations of improvements. And we definitely need to build libraries as part of the ADF development.

To understand the idea of software library better, we can use real examples:

• Algorithm libraries like math, or 3D, or ML

• Hardware abstraction libraries like HAL or device drivers

• Standard-compliant implementations like OpenAuth

• Programming helpers like Boost or Requests.py that provide developers with a pre-written code for HTTP requests lifecycle syntaxis helper.

• Any proprietary pluggable reusable code

Figure 1.3: Concept-level diagram of API and Library

The diagram suggests that Library can serve as a pre-implemented tool to integrate an API to the application; but basically, it can provide any pre-implemented code for reuse.

Software Development Kit

Similar functions can be also performed by Software Development Kits (SDKs) but they usually include much more than just a software Libraries; there is a list of possible SDK internals:

• Libraries.

• Tracing and Debugging tools.

• Documentation.

• Integrated development environments (IDEs)

• Tests.

• Plug-ins.

• Application programming interfaces (APIs)

• Sample code.

SDKs span both design-time (organizational) and runtime/operations (product) concerns but still with focus on a runtime. SDKs are also platform– or vendor-specific, they are developed by API or Platform vendors to improve their products adoption – see Android SDK created by Google (Alphabet) and Windows ASDK developed by Microsoft.

Sometimes bigger SDKs can be designed to include frameworks (like Apple SDK), but we can also see the opposite case, where an SDK is designed as an element of the framework. The following are examples of cases where SDKs are subsystems of ADFs in the list below:

• Operator SDK is part of the Operator Framework

• SDKs as developer-friendly lib wrappers for a particular framework, (for example) Treblle provides multiple SDKs including one for the Django framework

• SDKs have the same control flow as libraries have: the developer is responsible for invoking an SDK, while in the case of frameworks, we usually have the opposite control flow: the framework is responsible for invoking developer’s code. There is no exception if the framework is part of an SDK: the framework takes ownership over the control flow.

Figure 1.4: Concept diagram of API, Library, and SDK

The diagram above depicts the SDK as a super-entity for a library.

Framework

Let’s elaborate on our definition here:

A framework is a collection of pre-written code and tools that provides a structured approach to building applications. It simplifies development by enforcing architectural patterns: frameworks always dictate an execution flow, and stipulate specific way to structure your code, promoting maintainability, testability, low coupling, and reusability.

Of course, frameworks provide more than that, there are some examples below:

• Hide low-level complexity behind a higher-level abstraction;

• Promote faster development by providing pre-built binary/packaged components and functionalities;

But those additional benefits cannot be attributed exclusively to frameworks – libraries or SDKs both have the same value propositions.

The following are the Key aspects of this definition based on usage scenarios and key attributes:

• “Framework as abstraction” conceals repetitive code and low-level details by applying software libraries, acting as a higher-level interface. This allows developers to work with core functionalities without getting bogged down in implementation specifics. However, it usually gives an option of direct communication with levels under even if it is unnecessarily for overwhelming majority of scenarios; in brief, any framework has one or multiple libraries coming as a built-in option or pluggable 3rd-parties.

• “Frameworks as tooling” prioritize simplifying the development process by providing pre-built components, streamlined workflows, and reduced boilerplate code. Their primary focus is on accelerating development, while runtime considerations (like operations, maintenance, portability, performance) are a secondary benefit.

• “Framework as architectural constraint” establishes a blueprint for system architecture. It dictates core components, their interactions, and overall structure, influencing key design decisions for developers working within the framework’s constraints.

And finally, the relations between a frameworks and APIs, libraries and SDKs are usually follow the common pattern: ADF streamlines the software development flow for the organization, governs the control flow by invoking a custom code made by software developer, having libraries as a proxy to access external subsystem APIs, and allowing to plug in a third-party libraries or SDKs to handle specific integrations.

Figure 1.5: Concept diagram of ADF, SDK, Library, and API

There is an extended classification of the Application Development Frameworks:

• Web frameworks like Django, Node.js, Java Play, Ruby-on-Rails, etc

• Enterprise frameworks like Java Spring, Oracle ADF

• Low-code frameworks like Flutter, OpenBlocks, Appsmith

• AI/ML frameworks like TensorFlow, Keras, Apache MXNet

• Gamedev frameworks like Unity, UnrealEngine

• Mobile frameworks like React Native, Xamarin, Apache Cordova

• Microservice frameworks like GoMicro, Spring Boot, Molecular

• Test automation frameworks like Selenium, Appium, WebdriverIO

• Desktop OS frameworks like MFC, OS X framework, KDE framework

• Utility frameworks like Python Celery, ActiveTask,

• Custom frameworks – proprietary ones build for internal use, usually applying ADF format for a domain objects and rules

• As Frameworks are our focus area, we add more detailed specification for three ADFs to better highlight their commonalities.

The first one to analyze is Django:

• It employs numerous libraries in pluggable way

• It focuses on design-time aspects: “encourages rapid development and clean, pragmatic design”

• It is responsible for a control flow – developers don’t need to invoke Django code but to follow a Django project structure to get their code invoked in a right moment;

• It enforces multiple architecture patterns (MVC / MVT as a model-view-template, ORM as object-relational mapping, extendable middleware-based request processing pipeline, class-based views, etc)

• It is vendor- and platform-neutral so it can be used on any cloud platform or a virtual machine that can interpret Python programming language;

• It is a “gray box” software that can be redistributed as a package but it also has its source code published in a public GitHub repository under the BSD-3 OSS license

The second one is Node.js, the most popular web full-stack framework based on the JavaScript programming language:

• It has number of standard built-in libraries listed in official documentation and hundreds of pluggable external libraries like listed here

• Org design-time focus is clearly emphasized as a key success factor of this framework; …

  (As Ryan Dahl said): “So for kind of technical reasons, adding a server onto JavaScript worked really well and people who were programming front end websites were able to take those same skills and with just a small amount of additional knowledge were able to program pretty nice web servers that could do long polling or other kinds of real time interactions. And I think there’s just a large base of JavaScript users out there, naturally, it being the language of the web, and so there was a lot of people who were able to take their skills and add on Node to that and suddenly become full stack developers.”

• It is responsible for the control flow

• It enforces architecture patterns like event-driven, microservices, API-first etc.

• It is vendor- and platform-neutral

• It is a “gray box” open-source software

And the final one is React, a modern web front-end framework based on the JavaScript language:

• Dozens of libraries like here

• Design-time focus: most of public sources mention developer-oriented benefits as a key advantage of the framework; this list includes declarative syntaxis, reusable components, community support, detailed documentation, etc.

• Control flow management based on a virtual DOM concept is the core of the framework

• React’s intrinsic architecture patterns include event-driven (hooks), container-based decorators, data repository (provider), etc.

• It is vendor- and platform-neutral

• It is a “gray box” open-source software

As we can see, most ADFs follow the same model which we will explore in more detail in Chapter 3.

Platform

And the final concept to review here is a Platform. The main differentiator of a Platform, in comparison with all the others: APIs, libraries, SDKs and frameworks, is its hosted server-side execution runtime. But it is important to understand that the runtime concern is a “bonus value” here because the fundamental benefits of Platforms are still organizational design-time toolings. Platforms usually combine that hosted execution backend with APIs, libraries, and SDKs; and it is common for modern Platforms to provide developers with more advanced tooling like dev portals, resource management console and UI, cloud IDE, infrastructure-as-a-code definitions support and many other org productivity boosters. However, platforms rarely include application development frameworks and vice versa. We still can see frameworks being part of the platform (like AWS Well-Architected Framework is part of AWS platform value proposition, and Microsoft also has the same one), but these are not ADFs but architecture design frameworks (set of values, viewpoints, blueprints and patterns for cloud-native applications).

Typical taxonomy with examples of the Platforms is the following:

• Cloud platforms like AWS, MS Azure or GCP

• Messaging platforms like Confluent Kafka or Redpanda

• Task execution platforms like Temporal.io

• Robotic process automation (RPA) platforms like WorkFusion or UIPath

• Game Platforms like Heroic Cloud

• Dev platforms like Split.io, Firebase, Launchdarkly

Figure 1.6: Concept diagram of Platform, Framework, SDK, Library, and API

• Platforms are the most complete and mature kind of reusable engineering elements; however, their value comes with a high cost of ownership – this means that only large-scale companies can afford to build and operate an internal software development platform. Please do not confuse internal SDP with commercial PaaS (platform as a service) – we have many examples that successful PaaS can be created and operated with relatively small investment.

Domain Specific Language

Here it is, the bonus addition to the chapter. A Domain Specific Language (DSL) is a “language” with a higher level of abstraction optimized for a specific class of problems. A DSL uses the concepts and rules from the field or domain.

Note

Please note that DSL is not a programming language but rather a “formal domain description” language. In most cases, DSLs implementations are closer to executable configuration in JSON, YAML, XML or similar formats.

In terms of architecture abstractions, extensibility, and control flow, DSL is very similar to ADF – they both imply certain design patterns, employ libraries for extensibility and portability purposes, and invoke necessary code in the right time, defined by DSL creators.

The difference is the level of freedom for software product developers to code the DSL execution. We may consider DSL a “next level” framework suitable for cases when we want to achieve a high grade of standardization at a high level of abstraction.

Typical simplified taxonomy of the DSLs is the following:

• Workflow/BPMN like Camunda, Oracle BPMS, Nikku

• Rules like Drools

• Infra like Terraform

• Development like Gradle

Figure 1.7: Concept diagram with DSL, Platform, Framework, SDK, Library, and API

The diagram demonstrates a DSL primary use case of extending a Framework with limited predefined capabilities.

In this section we focused on key differentiators of Frameworks in comparison with other kinds of “engineering building blocks”. Now we can concentrate on the most important aspects of Frameworks, including the ones we further discuss in the next chapters.

Differentiating ADF and other Types of Frameworks

There are even more sources of confusion: we have Frameworks that could be used in the process of developing software products, but these Frameworks are not ADFs!

In the realm of software development, framework is a broad term encompassing various tools that structure and streamline different aspects of the process. We’ve established that Application Development Frameworks (ADFs) directly assist with coding. Let’s delve deeper into two other crucial categories: Architecture Frameworks and Software Delivery Frameworks.

…

Architecture Frameworks

These frameworks provide a structured approach to designing the overall architecture of a software system. Think of them as organizational architecture design process blueprints or roadmaps that define the foundation upon which your application will be built.

Here’s a breakdown of Architecture Frameworks’ types:

Enterprise Architecture Frameworks (EAFs): Focuses on the high-level structure of an entire organization’s IT infrastructure, including software applications, data, and hardware. Examples: TOGAF, Zachman Framework.

Software Architecture Frameworks (SAFs): Specializes in designing the internal structure of a single software application. Examples: 4+1 View Model, C4 Model.

The following are some of the benefits of adopting such frameworks:

• Consistency: Promotes a standardized approach to design, ensuring all components fit together seamlessly.

• Communication: Provide a common language for stakeholders (architects, developers, etc.) to discuss system design.

• Reduced Complexity: Break down complex systems into manageable components, simplifying design and development.

The following are some of the popular examples of the architecture frameworks:

• TOGAF (The Open Group Architecture Framework): A widely used EAF known for its comprehensive approach to enterprise architecture.

• Zachman Framework: Another EAF, offering a framework for classifying architectural information across different viewpoints (e.g., business, data, application).

• 4+1 View Model: A SAF focusing on five architectural viewpoints (system, application, deployment, container, and code) for designing software applications.

(Another example is DoDAF)

Architecture Frameworks can be very useful, but they are completely out of scope of this book.

Software Delivery Frameworks

These frameworks focus on streamlining the entire software development and delivery process, particularly for large-scale or complex projects. They don’t deal with the specifics of coding or designing the application itself, but rather how to efficiently manage the development lifecycle.

Here’s a closer look at Software Delivery Frameworks:

Core Principles: Emphasize iterative development, continuous integration and continuous delivery (CI/CD), and agile methodologies.

The following are some of the benefits:

• Improved Efficiency: Streamlines workflows and processes to deliver software faster and with fewer errors.

• Enhanced Communication: Fosters collaboration between development teams, product managers, and stakeholders.

• Increased Adaptability: Enables teams to respond to changing requirements and market needs more effectively.

The following are some of the popular examples:

• Scaled Agile Framework (SAFe): A popular framework for scaling agile methodologies to large enterprises.

• Large Scale Scrum (LeSS): An adaptation of the Scrum framework designed for large teams working on complex projects.

• Disciplined Agile Delivery (DAD): A framework that integrates various agile practices with other project management methodologies.

These categories provide a glimpse into the diverse landscape of frameworks beyond ADFs. Architecture Frameworks ensure a well-designed foundation for your software system, while Software Delivery Frameworks guide the overall development journey with efficiency and agility. By understanding and leveraging these frameworks, you can build robust and successful software applications.

But as we decided to focus on ADF, we need to pay special attention to a Software Development Lifecycle topic as it covers the key value of using application development frameworks.

Software Development Lifecycle (SDLC) and Flow

To better understand Application Development Frameworks, we need first to understand key scenarios of ADF adoption.

To set up a context, we need to differentiate the design-time and operations (runtime) aspects of the software product development. It is important to note that the Software Development Lifecycle (SDLC) can be used as a “blueprint” for software product development iteration. Visual diagram of SDLC can help us better understand this aspect:

Figure 1.8: Spiral SDLC model is one of the most advanced SDLC models for software development: image credits to tutorialspoint

We should always assume a repeatable iterative nature of the software product development process not because “agile” is our current state-of-the-art delivery methodology; every lean or efficient delivery approach relies on feedback loops to enable continuous improvement.

While ADF can be defined from both design-time and operations (runtime) perspectives, its primary value is always in design-time and SDLC. We can impact routines as repeatable and reproducible tasks. That’s how it is different from software libraries or APIs that can be employed for a one-time task that might never be repeated again (e.g. library that provides integration to specialized hardware or vendor-locked API that can be used in multiple places in source code, but the use of a library is not about repeatable part of SDLC task).

In systems engineering terminology we can declare that SDLC is a “using system” (or a supersystem) for ADF, and ADF is a “subsystem” of SDLC. This implies not only the fact that ADF is literally part of SDLC, but also enables the following mental model:

• ADF value, success criteria, and metrics are targeting SDLC improvements

• ADF stakeholders’ roles are SDLC participants

• Key ADF architecture viewpoints are defined in SDLC

In addition to SDLC concept, which is focused on engineering and instrumenting the development process, we can apply a “Flow” term to emphasize the systemic aspect of software product development in terms of value throughput and delivery management.

ADF and Flow

…ADF (is) s one of the most influencing ways to optimize a Flow throughput and eliminate bottlenecks related to product and process complexity.

Summary

An Application Development Framework is a software product “skeleton” that offers a fundamental structure for building applications within a specific environment. It acts as a reusable foundation, supplying pre-defined functionalities and promoting code organization through established conventions. This distinguishes it from other ways of reusing code in the form of an SDK, library, or API. It also makes ADF a unique opportunity to streamline development by reducing repetitive coding efforts, decreasing cognitive load, and promoting architecture best practices.

With this knowledge, we can explore the next chapters to find a way to calculate ADF return on investment, meet stakeholders’ expectations, and align the ADF roadmap with a common maturity model.

Further reading

To know more please visit the (following) links:

• Vendor-specific hardware SDKs like Android Studio

• Samsung TV SDK

• MS Xbox

• Product-specific SDKs like PandaDoc SDK

• Hubspot SDK: https://developers.hubspot.com/docs/platform/ui-extensions-sdk

• GetStream

• Platform SDKs like AWS: Amazon Chime SDK & AWS SDK for .NET

• Azure or Google’s Cloud SDK

If you found this chapter insightful, check out Building an Application Development Framework by Ivan Padabed and Roman Voronin (Packt, Sept 2025). It’s a practical, end-to-end playbook for designing, building, and rolling out a custom ADF that measurably improves flow and quality across your SDLC. The first edition includes real-world cases and a downloadable code bundle.

You’ll learn to: design and implement a robust ADF architecture; choose technologies for performance and scalability; manage versioning and packaging for efficient distribution; harden the framework’s security posture; foster collaboration to build an extensible ADF ecosystem; and measure and optimize performance (including DORA-style outcomes) for continuous improvement. Ideal for technical leaders, engineering managers, and developers who want a pragmatic, code-backed path to a reusable framework that scales with their organization.

Here’s what some readers have said:

Why call out AI systems? Unlike traditional software, AI systems are model- and data-driven: behavior depends on training data, live inputs, and evaluation metrics. Architecture is how we make that behavior predictable and operable in production.

Why architecture work matters

Architecture makes prototypes survivable in production by forcing end-to-end thinking about data reality, key non-functional constraints, stakeholder trust, and the documentation that turns design into testable behavior.

• Data reality: The key driver in any AI system is the quality of the data used for training and inference. Data engineering is often not done well or not done with a large enough aperture to appreciate the full end-to-end system. For example: does the prototype assume well-formatted or highly structured data that is not possible in production? Did the prototype suppose that data into the system was ordered but, in a production environment, that is not the case? Was the system overly reliant on a technology that cannot scale to expected production data rates? And so on.

• Partitioning and organization: Software architecture involves design decisions about partitioning and organization. An analysis of the software components of the prototype can identify where choke points may occur, how interfaces into and out of the prototype were handled, and where a component may require more design work in a production environment.

• Non-functional requirements: A robust architecture captures and understands the key non-functional requirements at play in the production environment. For example, will the new AI functionality impact availability requirements that, if not met, would result in a critical error? Does the new model with higher production data rates impact performance requirements that have cascading effects?

• Trust tolerance: Robust architecting should illuminate the trust tolerance and decision making of key stakeholders. For example, does a single off-nominal inference or decision result in a loss of confidence or outright hostility to the new AI functionality? As part of a conceptual design, guard rails or off ramps can be put into the design so there is a sort of “no-harm” guarantee that may cause consternation but not a loss of confidence.

• Documentation to test: The documentation of a conceptual design can aid in defining and illuminating test cases and scenarios that the prototype must satisfy in a production environment.

A fair criticism—and the response

It’s reasonable to say that architecting adds “document-centric” work that slows delivery. But the MIT statistic exists for a reason: no amount of heroic coding fixes a flawed architecture. Water doesn’t flow uphill; a bad architecture produces a bad system. Agile methods work best when a stable architectural frame guides rapid prototyping, integration, and demos. Productive engineering isn’t only writing code—it’s also designing systems that can survive production.

Architecting is critical in guiding engineering decisions and priorities. This is done by laying out the conceptual design of the end system. One of the first drivers of architecture is to have a solid conceptual design that informs prototyping. That same design later anchors the pass/fail criteria that we can categorize into gates.

Five production-readiness gates (with actions)

There are five gates or decision buckets—pass/fail outcomes we must clear before go-live.

Note: the following actions are not exhaustive.

1) Data Engineering (AI inputs and contracts)

• Analysis: Confirm that production inputs and interfaces match what the model expects—data rates, formats, fields, identifiers, consistency, and summary statistics.

• Ingest testing: Exercise the pipeline with current production data and verify ingest correctness and throughput.

• Resilience checks: Prove the prototype tolerates malformed records, off-nominal flows, and misformatted streams without cascading failures.

2) Model Correctness (does the model still earn its keep?)

• Ongoing relevance: Verify the model still solves a real user problem; retire it if it no longer does.

• Performance on live data: Re-evaluate with current production samples; confirm metrics such as AUC, accuracy, and false positive rate meet or exceed targets.

• Execution time: Show end-to-end model latency remains within bounds required by downstream consumers and users.

3) Observability (for data pipelines and model behavior)

• Data flow telemetry: Logging and monitors for inputs, transformations, and handoffs.

• Model execution telemetry: Traces, metrics, and logs for inference paths and failure modes.

• Acceptance bounds: Document thresholds, alert rules, and expected distributions for observability metrics.

4) Operations (release and recovery for AI pipelines)

• Canaries: Implement canary or shadow releases to surface pipeline issues before full rollout.

• Time budgets: Define and monitor time bounds for the full pipeline and each critical stage.

• Update paths: Document steps for model refresh, rollback, and disentanglement from production if needed.

5) Data Governance (quality, compliance, ethics)

• Quality audit: Run a data quality review on the latest production datasets.

• Compliance docs: Prepare artifacts to satisfy regulatory, security, and privacy requirements.

• Ethical filters: Enforce policy-aligned filtering and access controls.

In addition to these gates, the key non-functional requirements for the system need to be monitored, with action plans to ensure they are met—for example, time budgets across the system via specialized logging and canary deployments.

The gates define what must be true to go live; let us go a step further and look at the rubric that translates the actions into concrete checks and artifacts that prove each gate is passed.

Rubric for pre-deployment checks

Here is a rubric you can use as a go/no-go gate: confirm business relevance, data contracts, operational readiness (including rollback), on-call coverage and monitoring, and model quality against explicit SLOs before promoting any prototype to production.

This rubric is the evidence and checks that prove each gate is passed and that you can go live.

• Business Checks

  • Is the model output still relevant to intended users?

  • Is the prototype operating at time scales relevant to users?

  • Is the prototype still within budget and resource limits?

• Data Engineering

  • Verify data sources.

  • Verify interfaces still meet service-level agreements (format, content, time budgets).

  • Confirm computational resources for scale are in place.

• Operations

  • Verify the prototype in a pre-production environment or a simulation that mimics true production.

  • Confirm datastore schemas and queries execute correctly and produce expected results.

  • Ensure error logging exists at multiple points with sufficient detail for troubleshooting.

  • Identify a rollback procedure to take the model offline if needed.

• Pre-deployment

  • Identify a developer or staff resource to support rollout and address issues promptly.

  • Ensure monitoring is in place to capture operations.

• Model

  • Monitor non-functional requirements, SLAs, and errors.

  • Track correctness metrics (AUC, accuracy) and model drift.

  • Provide a user interface (or tools) to visualize execution.

  • Supply guides/documentation for troubleshooting, architecture diagrams, expected outputs at each stage, and accessible source code.

  • Use canaries with injected known datasets for monitoring.

Mini case study

To understand how the 5 gates and the rubric work in cohesion, let us take the example of a new application that recommends technical articles to an engineering audience. Before launch, the team decides to apply actions under each of the 5 gates and the Pre-deployment checks reveal the following:

• Data & Schema

  • Since training began, an extra tag field was added; the date format also changed, breaking metadata parsing.

  • The datastore schema drifted: a key field changed, requiring interface updates.

• Model Correctness: On current production data, AUC dropped below training thresholds; investigation showed an over-represented training subset.

• Observability: Logging failed; output and processed-file logs couldn’t be written due to permission issues.

• Operations & Release: The team planned to launch at peak traffic with no rollback procedure.

• Governance: A compliance check found personal information was being collected and shouldn’t have been.

As a result, a failed release is prevented.

For AI systems, some architecting during prototyping helps cross the chasm from an impressive demonstration to a production system that delivers true value and positive business outcomes.

If you found this article insightful, Richard D Avila and Imran Ahmad’s forthcoming book, Architecting AI Software Systems extends the same ideas with a structured approach to integrating AI into traditional architectures—covering how inference and decision-making shape design, using architectural models to ensure cohesion, mitigating risks like underperformance and cost overruns, and applying patterns and heuristics for scalable, high-performance systems—anchored by real-world case studies, hands-on exercises, and a complete example of an AI-enabled system for architects, engineering leaders, and AI/ML practitioners.

About the Author:

Richard D Avila is a software and systems architect with over two decades of industry experience building complex software systems. He has architected and held leadership roles for building a wide array of software systems from complex simulations, autonomy and data analytic systems. He is also a principal investigator for AI topics. He has published in referred journals and industry publications on command-and-control theory, assurance architectures, multi-agent modeling, and machine learning. He was the first expert instructor on data analytics at University of Maryland Baltimore County – Training Centers. Before working as a software and systems architect, he served in the US Navy as a submarine officer.

The complete “Chapter 14: Quantum Error Correction and Fault Tolerance” from the book A Practical Guide to Quantum Computing by Elías F. Combarro and Samuel González-Castillo (Packt, 2025)

4.55MB ∙ PDF file

Download

Chapter 14, Quantum Error Correction and Fault Tolerance, introduces quantum error correction, which could pave the road towards fault-tolerant computation and thus may be a cornerstone in the development of useful quantum computers. This chapter begins with a discussion on the necessity for quantum error correction, which is followed by the introduction and implementation of a simple quantum error-correcting code. The chapter finishes with some remarks on fault-tolerant computing and how quantum error correction can help enable it.

Download

If you liked this chapter, check out A Practical Guide to Quantum Computing: Hands-on approach to quantum computing with Qiskit by Elías F. Combarro and Samuel González-Castillo (Packt, July 2025). It’s a self-contained introduction that starts from a single qubit and builds to foundational algorithms—Deutsch, Bernstein–Vazirani, Grover, Shor—with runnable code in Qiskit 2.1 on simulators and real hardware. Along the way, it covers applications like quantum money and BB84, and introduces fault tolerance and the road to quantum advantage.

You’ll learn to: understand what makes a quantum computer unique; represent and measure multi-qubit states; use superposition, entanglement, and interference; implement and run quantum algorithms in Qiskit; explain how Grover’s search and Shor’s factoring work; and get familiar with quantum fault tolerance and emerging notions of quantum advantage. Ideal for university-level STEM courses and for professionals or self-learners who want a practical, code-first path into quantum computing.

Here is what some readers have said:

Download

Clean Architecture, introduced by Robert C. Martin in 2012, synthesizes decades of software design wisdom into a cohesive set of principles. It addresses persistent challenges in software development, such as managing complexity and accommodating change. By applying Clean Architecture principles to Python projects, developers can create systems that are not only functional but also maintainable, testable, and adaptable over time.

In this chapter, we’ll explore the essence of Clean Architecture and its relevance to Python development. We’ll examine how Clean Architecture principles align with Python’s philosophy of simplicity and readability, creating a natural synergy that enhances Python’s strengths. You’ll learn how Clean Architecture can help you build Python applications that are easier to understand, modify, and extend, even as they grow in complexity.

By the end of this chapter, you’ll have an overview of Clean Architecture principles and their potential benefits for Python development. You’ll be introduced to how this approach can address common challenges in software development, particularly as Python projects grow in scale and complexity. This foundational understanding of Clean Architecture will be essential as we delve deeper into its implementation and best practices in Python throughout the rest of the book.

In this chapter, we’re going to cover the following main topics:

• Why Clean Architecture in Python: the benefits of balancing planning and agility

• What is Clean Architecture?

• Clean Architecture and Python: a natural fit

Why Clean Architecture in Python: the benefits of balancing planning and agility

In this section, we’ll explore the critical balance between planning and agility in Python development and how Clean Architecture can help achieve this balance. We’ll examine the challenges posed by increasing complexity in modern Python applications and the imperative for agility in today’s fast-paced business environment. We’ll then discuss the trade-offs between planning and flexibility, and how architectural thinking can provide a framework for managing these trade-offs. Finally, we’ll look at the role of architecture in managing complexity and setting the stage for long-term success. Through these discussions, you’ll gain insight into why Clean Architecture is particularly valuable for Python developers striving to create maintainable, adaptable, and efficient applications.

Let’s begin by examining the complex challenges facing modern Python development.

The complexity challenge in modern Python development

As Python’s popularity soars, so do the scale and complexity of applications built with it. From web services to data science pipelines, Python projects are growing larger and more intricate. This growth brings significant challenges that every Python developer must grapple with.

The increasing complexity of systems makes them harder to understand, modify, and maintain. This complexity can severely limit your ability to add new features or respond to changing requirements. The maintenance burden of complex Python systems can overwhelm development teams, slowing down progress and innovation. Even small changes in large, complex systems can have far-reaching consequences, making modifications expensive and risky.

Consider a fictitious large Python-based e-commerce site: PyShop. The business decides to implement a seemingly simple feature: adding gift-wrapping options to orders. However, this straightforward addition quickly cascades into a complex project:

• The order processing module needs updates to include gift-wrapping choices

• The inventory system requires modification to track gift-wrapping supplies

• The pricing engine needs adjustments to calculate additional costs

• The user interface (UI) must be updated to present gift-wrapping options

• The fulfillment system needs changes to include gift-wrapping instructions

What was estimated as a two-week-long task stretches into a multi-month project. Each change potentially impacts other system parts: adjustments in order processing affect reporting, inventory changes influence supply chain management, and UI modifications require extensive user experience testing.

This example highlights how interconnected modules in a complex system can turn a simple feature addition into a significant undertaking, emphasizing the need for an architecture that allows for more isolated changes and easier testing processes.

Moreover, as Python projects grow, developers often struggle with abstractions, a critical aspect that Clean Architecture helps address. Without proper guidance, codebases can suffer from extremes: either becoming a tangled mess of deeply nested class hierarchies that are hard to understand and modify or devolving into monolithic do-everything classes that lack any meaningful abstraction. In the former case, developers may create too-complex inheritance structures to maximize code reuse, resulting in a fragile system where changes in one place have unforeseen consequences elsewhere. In the latter case, the lack of abstraction leads to massive, unwieldy classes and rampant code duplication, making it nearly impossible to maintain consistency or make systemic changes. Both scenarios result in codebases that are difficult to understand, maintain, and extend, which is precisely the issues that a well-planned architecture helps prevent.

Furthermore, in today’s rapidly evolving tech landscape, complex, tightly coupled systems struggle to take advantage of new technologies. This limitation can significantly impact your ability to stay competitive in a field where technological agility is crucial.

The agility imperative

In our fast-paced business environment, agility is not just an advantage—it’s a necessity. With every company essentially becoming a technology company, the pressure to deliver quickly has never been higher. Python’s simplicity and extensive ecosystem make it an excellent choice for rapid development.

However, sustainable agility requires more than just initial speed, it demands architectural decisions that support ongoing evolution. It’s akin to building a high-performance race car: without proper design fundamentals, what starts as impressive acceleration quickly becomes limited by poor handling and maintenance challenges.

In rapidly evolving Python applications, this principle becomes starkly evident. Without a cohesive architecture, quick feature additions can create a tangled web of dependencies. What starts as a nimble codebase can, within months, become rigid and fragile. Developers find themselves spending more time deciphering existing code than writing new features. When it’s not immediately clear where new code should be added or how it should interact with existing components, developers under pressure may make hasty decisions, leading to suboptimal implementations and introducing bugs. These quick fixes further complicate the codebase, making future changes even more challenging. The initial velocity becomes unsustainable, not because of the speed itself, but due to the lack of a sturdy architectural foundation that can guide rapid changes and provide clear pathways for new feature integration.

Requirements change, often unpredictably. Your Python projects need to be structured in a way that allows for easy adaptation to these changes. This adaptability is crucial for long-term success in software development.

Striking a balance: the planning–agility trade-off

Finding the right balance between planning and agility is crucial in Python development. As Dave Thomas wisely said, “Big design up front is dumb. Doing no design up front is even dumber.” The key is finding the middle ground that allows for both structure and flexibility.

Good architecture helps you postpone decisions. It gives you the flexibility to push decisions to later stages when you have more information to make the correct choice. This approach is particularly valuable in Python development, where the language’s flexibility can sometimes lead to decision paralysis.

Introducing architectural thinking in Python development means considering the long-term structure of your project from the start, without over-engineering. It’s about creating a framework that guides development while remaining adaptable to change.

The role of architecture in managing complexity

Effective architecture is your best tool for managing complexity in Python systems. Good architecture simplifies complex systems by providing a clear structure and separation of concerns (SoC). One of the first steps in architecting a new system is determining how to divide it, keeping things that change for the same reason together and things that change for different reasons apart.

Consider two Python-based content management systems (CMSs) for media companies, both tasked with implementing a new AI-powered content tagging feature. In the well-architected system, this feature is implemented as a standalone module with clear interfaces. It integrates smoothly with the existing content creation and search modules through well-defined APIs. Developers can build and test the AI tagging service independently, and then connect it to the content database and UI with minimal disruption. Conversely, in a poorly structured system, adding this feature requires changes across the entire stack—from database schemas to frontend code—leading to unexpected bugs and performance issues. What takes a sprint in the well-architected system becomes a months-long refactoring project in the poorly structured one, demonstrating how thoughtful initial architecture can dramatically improve development efficiency and system adaptability.

The architectural decisions you make early on have a profound impact on the long-term development costs and flexibility of your Python projects. A well-architected system can significantly reduce the cost of change over time, allowing your team to respond more quickly to new requirements or technological changes.

Preparing for Clean Architecture

As we move toward discussing Clean Architecture, it’s important to understand that it offers a systematic approach to balancing planning and agility in Python projects. Architectural principles provide powerful tools for managing and reducing complexity in your Python systems.

At its core, Clean Architecture is about strategic SoC in your Python applications. It advocates for a structure where the essential business logic is insulated from external factors such as UIs, databases, and third-party integrations. This separation creates clear boundaries between different parts of your application, each with its own responsibilities. By doing so, Clean Architecture allows your core business rules to remain pure and unaffected by the implementation details of input/output (I/O) mechanisms or data management systems (DMSs).

By understanding these challenges and principles, you’ll be better prepared to appreciate the benefits that Clean Architecture can bring to your Python projects. In the next sections, we’ll delve into what Clean Architecture is and how it specifically applies to Python development, providing you with the tools to combat complexity and reduce the cost of change in your software systems.

What is Clean Architecture?

Having explored the challenges of managing complexity in Python development and the need for balancing planning with agility, the goal of this section is to give you a high-level overview of Clean Architecture. We’ll be covering several key concepts and principles in quick succession to provide a broad understanding. Don’t worry if you don’t grasp every detail immediately. This is just the beginning of our journey. Each of these topics will be explored in depth in the chapters to come, where we’ll dive into practical Python implementations and real-world scenarios.

Clean Architecture synthesizes many ideas from previous architectural styles, but it is built around a fundamental concept: the separation of software elements into ring levels, with a strict rule that code dependencies can only move inward from outer levels. This principle is formally known as the Dependency Rule, one of the most critical aspects of Clean Architecture. The Dependency Rule states that source code dependencies must only point inward, toward higher-level policies. Inner circles must know nothing about outer circles, while outer circles must depend on and adapt to inner circles. This ensures that changes to external elements (like databases, UI, or frameworks) don’t impact the core business logic. The aim is to create software systems that are not only functional but also maintainable and adaptable over time. To illustrate this, let’s consider a simple Python application for a library management system:

1. At the core, we have the Book class, representing the basic data structure.

2. Moving outward, we have a BookInventory class that manages operations on books.

3. In the outer ring, we have a BookInterface class that handles user interactions related to books.

In this structure, the Book class knows nothing about the BookInventory or BookInterface classes. The BookInventory class might use the Book class but doesn’t know about the interface. This separation ensures that the core logic remains unaffected by external concerns.

Crucially, this structure allows us to modify or even replace outer layers without affecting the inner layers. For instance, we could change the UI from a command-line interface (CLI) to a web interface by modifying the BookInterface class, without needing to alter the Book or BookInventory classes. This flexibility is a key advantage of the Clean Architecture approach.

This structure is designed to produce systems that embody the key principles we introduced earlier:

• SoC

• Independence of external details

• Testability and maintainability

Let’s explore further how Clean Architecture achieves these goals and the benefits it brings to software development.

The onion architecture concept

Let’s visualize the ring levels mentioned earlier and add another level of detail as to the purpose of each ring. Clean Architecture is often visualized as a series of concentric circles, like an onion. Each circle represents a different layer of software, and the Dependency Rule we discussed ensures that dependencies only flow inward across these boundaries. The core layers contain business logic (entities), while the external layers contain interface and implementation details (see Figure 1.1):

Figure 1.1: Clean Architecture: a series of concentric layers

Figure 1.1 demonstrates the separation of inner core business logic progressing out to external interfaces:

• Entities: At the center are entities, which encapsulate enterprise-wide business rules. Entities in this context are the primary nouns of your product, the core business objects that would exist even without software. For example, in an e-commerce system, entities might include Customer, Product, and Order. In a task management application, they could be User, Task, and Project. These entities contain the most basic, universal rules about how these objects behave and interact.

• Use Cases: The next layer contains use cases, which orchestrate the flow of data to and from entities. A use case represents a specific way the system is used. It’s essentially a description of how the system should behave for a particular scenario. For instance, in a task management app, use cases might include Create New Task, Complete Task, or Assign Task. Use cases contain application-specific business rules and control how and when entities are used to fulfill the goals of the application.

• Interface Adapters: Further out, we find interface adapters, which convert data between use cases and external agencies. This layer acts as a set of translators between the inner layers (entities and use cases) and the outer layer. It might include things such as controllers that handle HTTP requests, presenters that format data for display, and gateways that transform data for persistence. In a Python web application, this might include your view functions or classes that handle routing and request processing. A key point of this layer is that it allows us to decouple from frameworks.

• Frameworks and Drivers: The outermost layer contains frameworks and drivers, where the external agencies reside. By drivers, we mean the specific tools, frameworks, and delivery mechanisms that are used to run the system but aren’t core to the business logic. In a Python context, examples might include the following:

  • Web frameworks such as Django or Flask

  • Database drivers such as psycopg2 for PostgreSQL or pymongo for MongoDB

  • External libraries for tasks such as sending emails (for example, smtplib) or processing payments

  • UI frameworks if you’re building a desktop or mobile app (for example, PyQt)

  • System utilities for tasks such as logging or configuration management

This outermost layer is the most volatile, as it’s where we interact with the outside world and where technologies are most likely to change over time. By keeping it separate from our core business logic, we can more easily swap out these external tools without affecting the heart of our application.

This layered structure of Clean Architecture promotes SoC, establishing a clear organizational framework for software systems. Now that we have an idea of the fundamental structure of Clean Architecture, let’s further investigate its broader benefits.

Benefits of Clean Architecture

One of the primary advantages of Clean Architecture is its focus on protecting and isolating your core business logic, the domain objects that represent the foundation of your business. While external details such as web frameworks and persistence engines come and go, the true value to your business lies in the time invested in designing and implementing these core domain objects. Clean Architecture recognizes this and provides a structure that insulates these crucial components from the volatility of external technologies.

This architectural approach protects your investment in domain logic from the need to move away from a given framework or technology. For example, if a framework you’re using moves from an open source model to a proprietary one, Clean Architecture allows you to replace it without rewriting your core business logic. This separation significantly reduces the cost and risk of changes over time, allowing your system to evolve more easily as requirements change or as you need to adapt to new technologies. In essence, Clean Architecture ensures that the most valuable and stable part of your application, your business logic, remains unaffected by the often turbulent world of external technologies and frameworks.

Another key benefit is enhanced testability across all layers of the application. The independence of the core business logic from external details makes it much easier to write comprehensive unit tests. You can test business rules in isolation, without the need to spin up a database or web server or build cumbersome mocks. This leads to more thorough testing and, consequently, more robust software. It also encourages developers to write more tests, as the process becomes simpler and more straightforward.

Clean Architecture also provides flexibility in technology choices. Because the core of the application isn’t dependent on external frameworks or tools, you have the freedom to swap out these elements as needed. This is particularly valuable in the fast-moving world of technology, where today’s popular framework might be obsolete tomorrow. Similarly, you might start with a CLI for internal use, then add a web interface for broader access, all without altering your core business rules’ code. Your core business logic remains stable, while you have the flexibility to adopt new technologies in the outer layers as they emerge. Lastly, Clean Architecture promotes long-term agility in development and leads to what Robert C. Martin calls a Screaming Architecture. Its focus on separating concerns and managing dependencies results in a codebase that’s easier to understand and modify. The concept of Screaming Architecture suggests that when you look at the structure of your system, it should scream its purpose and use cases, not its frameworks or tools. For instance, your architecture should scream online bookstore, not Django application. This clear, purpose-driven structure allows new team members to quickly grasp the system’s intent and make contributions. The architecture itself becomes a form of documentation, revealing the system’s core purpose and functionality at a glance. Such clarity and flexibility translate to increased development speed over the long term, even as the system grows in complexity. It also ensures that your system remains focused on its core business logic, rather than being tied to specific technical implementations.

Clean Architecture in context

To fully appreciate the value of Clean Architecture, it’s important to understand its place in the broader context of software development practices and methodologies.

Clean Architecture represents an evolution from traditional layered architecture. While it builds upon the concept of layers, it places a stronger emphasis on SoC and enforces the Dependency Rule more strictly than traditional architectures. Unlike traditional layered architectures where lower layers often depend on persistence or infrastructure concerns, Clean Architecture keeps the inner layers pure and focused on business logic. This shift in focus allows for greater flexibility and resilience to change.

Clean Architecture complements modern development practices such as Agile and DevOps. It aligns well with Agile methodologies by facilitating continuous delivery (CD) and making it easier to respond to change. The clear SoC supports iterative development and makes it easier to modify or extend functionality in response to changing requirements. In terms of DevOps, Clean Architecture supports practices such as continuous integration and deployment (CI/CD) by making systems more testable and modular. The clear boundaries between components can also help in scaling development across teams, as different teams can work on different layers or components with minimal interference.

In conclusion, Clean Architecture offers a powerful approach to building software systems that are scalable, maintainable, and adaptable to change. By focusing on SoC and managing dependencies, it provides a structure that can withstand the test of time and the pressures of evolving technology and business needs. As we move into the next section, we’ll explore how these principles align particularly well with Python development practices.

Clean Architecture and Python: a natural fit

As we’ve explored the principles and benefits of Clean Architecture, you might be wondering how well these concepts align with Python development. In this section, we’ll discover that Clean Architecture and Python share a natural affinity, making Python an excellent language for implementing Clean Architecture principles.

Python’s philosophy, as embodied in The Zen of Python aligns remarkably well with Clean Architecture principles. Both emphasize simplicity, readability, and the importance of well-structured code. Python’s focus on creating clear, maintainable, and adaptable code provides a strong foundation for implementing Clean Architecture. As we delve deeper into this section, we’ll explore how Python language features can be leveraged to create robust, maintainable systems that adhere to Clean Architecture principles.

Implementing Clean Architecture in Python

Python’s dynamic nature, combined with its strong support for object-oriented programming (OOP) and functional programming paradigms, allows developers to implement Clean Architecture concepts with less boilerplate and greater clarity than many other languages.

Note on code examples

Throughout this book, you’ll notice type annotations in our code examples (e.g., def function(parameter: type) -> return_type). These type hints enhance code clarity and help enforce Clean Architecture boundaries. We’ll explore this powerful feature in depth in Chapter 3.

A key principle of Clean Architecture is the reliance on abstractions rather than concrete implementations. This principle directly supports the Dependency Rule we discussed earlier: dependencies should only point inward. Let’s see how this works in practice using Python’s abstract base classes (ABCs).

Consider the following example, which models a notification system:

from abc import ABC, abstractmethod
class Notifier(ABC):
    @abstractmethod
    def send_notification(self, message: str) -> None:
        pass
class EmailNotifier(Notifier):
    def send_notification(self, message: str) -> None:
        print(f"Sending email: {message}")
class SMSNotifier(Notifier):
    def send_notification(self, message: str) -> None:
        print(f"Sending SMS: {message}")
class NotificationService:
    def __init__(self, notifier: Notifier):
        self.notifier = notifier
    def notify(self, message: str) -> None:
        self.notifier.send_notification(message)
# Usage
email_notifier = EmailNotifier()
email_service = NotificationService(email_notifier)
email_service.notify("Hello via email")

This example demonstrates key concepts of Clean Architecture using Python’s ABCs:

1. ABC: The Notifier class is an ABC, defining an interface that all notifier classes must follow. This represents an inner ring in our Clean Architecture structure.

2. Abstract method: The send_notification method in Notifier is marked with @abstractmethod, enforcing implementation in subclasses.

3. Concrete implementations: EmailNotifier and SMSNotifier are concrete classes in an outer ring. They inherit from Notifier and provide specific implementations.

4. Dependency inversion: The NotificationService class depends on the abstract Notifier class, not on concrete implementations. This adheres to the Dependency Rule, as the abstract Notifier class (inner ring) doesn’t depend on the concrete notifiers (outer ring). We’ll dive deeper into dependency inversion in the next chapter.

This structure embodies the Clean Architecture principles we’ve discussed:

• It respects the Dependency Rule: The abstract Notifier class (inner ring) knows nothing about the concrete notifiers or the NotificationService class (outer rings)

• It allows for easy extension: We can add new types of notifiers (such as PushNotifier) without changing the NotificationService class

• It promotes flexibility and maintainability: The core business logic (sending a notification) is separated from the implementation details (how the notification is sent)

By structuring our code this way, we create a system that’s not only flexible and maintainable but also adheres to the fundamental principles of Clean Architecture. The abstract Notifier class represents our core business rules, while the concrete notifiers and the NotificationService class represent the more volatile outer layers. This separation allows us to easily swap or add new notification methods without affecting the core logic of our application.

So, we’ve seen a simple ABC example, but this is where Python truly shines. We can implement the same Clean Architecture principles without using a class hierarchy, instead relying on Python’s support for duck typing. This flexibility is one of Python’s strengths, allowing developers to choose the approach that best fits their project’s needs while still adhering to Clean Architecture principles.

Duck typing is a programming concept where the suitability of an object is determined by the presence of certain methods or properties, rather than its explicit type. The name comes from the saying, “If it walks like a duck and quacks like a duck, then it must be a duck.” In duck typing, we don’t care about the object’s type; we care about whether it can do what we need it to do.

This approach aligns well with Clean Architecture’s emphasis on abstractions and interfaces. If you’d prefer to stay away from rigid class hierarchies, Python’s Protocol feature, introduced in Python 3.8, offers the best of both worlds: duck typing with type hinting. Here’s an example that implements the same notification system using protocols:

from typing import Protocol
class Notifier(Protocol):
    def send_notification(self, message: str) -> None:
        ...
class EmailNotifier: # Note: no explicit inheritance
    def send_notification(self, message: str) -> None:
        print(f"Sending email: {message}")
class SMSNotifier: # Note: no explicit inheritance
    def send_notification(self, message: str) -> None:
        print(f"Sending SMS: {message}")
class NotificationService:
    # Still able to use type hinting
    def __init__(self, notifier: Notifier):
        self.notifier = notifier
    def notify(self, message: str) -> None:
        self.notifier.send_notification(message)
# Usage
sms_notifier = SMSNotifier()
sms_service = NotificationService(sms_notifier)
sms_service.notify("Hello via SMS")

This example demonstrates the same notification system as before but using Python’s Protocol feature instead of ABCs. Let’s break down the key differences and their implications for Clean Architecture:

• Protocol versus ABC: The Notifier class is now a Protocol class instead of an ABC class. It defines a structural subtyping interface rather than requiring explicit inheritance.

• Implicit conformance: The EmailNotifier and SMSNotifier classes don’t explicitly inherit from the Notifier class, but they conform to its interface by implementing the send_notification method.

• Duck typing with type hinting: This approach combines Python’s duck typing flexibility with the benefits of static type checking, aligning with Clean Architecture’s emphasis on loose coupling.

• Concrete implementations: The NotificationService class still depends on the abstract Notifier protocol, not concrete implementations, adhering to Clean Architecture principles.

This protocol-based approach offers a flexible, Pythonic implementation of Clean Architecture concepts, balancing type safety with reduced class hierarchy rigidity. It demonstrates how to align Clean Architecture principles with Python’s philosophy, promoting adaptable and maintainable code.

We highly recommend the use of type hinting via either ABCs or protocols when implementing Clean Architecture. This approach, as opposed to simple implicit interfaces without type hinting, offers significant advantages:

• Improved code readability

• Enhanced IDE support and earlier error detection

• Better alignment with Clean Architecture goals

In the remaining parts of the book, we’ll primarily use ABCs in our examples as they are in greater use in existing Python codebases. However, the principles discussed are equally applicable to protocol-based implementations, and readers can adapt the examples to use protocols if preferred.

Practical example: a glimpse of Clean Architecture in a Python project

To illustrate the concepts we’ve discussed, let’s examine the basic structure of a Clean Architecture Python project. This structure embodies the principles we’ve covered and demonstrates how they translate into a practical file organization. We’ll stay at a high level here; later chapters will cover real-world examples in full detail:

Figure 1.2: A potential layout for a Clean Architecture Python web application

This file structure in Figure 1.2 exemplifies the Clean Architecture principles we’ve discussed:

1. SoC: Each directory represents a distinct layer of the application, aligning with the concentric circles we saw in Figure 1.1.

2. Dependency Rule: The structure enforces the Dependency Rule we discussed earlier. If we were to investigate the inner layers (entities and use_cases), we would not see any imports from the outer layers.

3. Entities layer: The entities directory contains the core business objects, such as user.py. These are at the center of our Clean Architecture diagram and have no dependencies on outer layers.

4. Use Cases layer: The use_cases directory holds the application-specific business rules. It depends on the entities but is independent of the outer layers.

5. Interface Adapters layer: The interfaces directory contains controllers, presenters, and gateways. These adapt data between use cases and external agencies (such as web frameworks or databases).

6. Frameworks layer: The outermost frameworks directory contains implementations of external interfaces, such as database object-relational mappers (ORMs) or web frameworks.

7. Straightforward testing: The tests directory structure mirrors the application structure, allowing for comprehensive testing at all levels.

This structure supports the key benefits of Clean Architecture we discussed:

• Maintainability: Changes to external components (in the frameworks directory) don’t affect the core business logic in entities and use cases

• Flexibility: We can easily swap out the database or web framework in the frameworks directory without touching the business logic

• Testability: The clear separation allows for easy unit testing of core components and integration testing of interfaces

Remember our discussion about abstractions? The interfaces directory is where we’d implement the ABCs or protocols we talked about. For instance, user_repository.py might define an abstract UserRepository class, which is then implemented concretely in the frameworks/database/orm.py file.

This structure also facilitates the master plan we mentioned earlier. It provides a clear roadmap for where new code should be placed, helping developers make consistent decisions even as the project grows and evolves.

By organizing our Python project this way, we’re setting ourselves up for long-term success, creating a codebase that’s not just functional but also maintainable, flexible, and aligned with Clean Architecture principles.

Python-specific considerations and potential pitfalls

While Clean Architecture and Python are highly compatible, there are some important considerations to be aware of when implementing these principles in Python projects. Throughout this book, we’ll guide you through mitigating these concerns, providing practical solutions and best practices.

Balancing Pythonic code with architectural principles

Python’s batteries included philosophy and an extensive standard library can sometimes tempt developers to bypass architectural boundaries for the sake of convenience. However, maintaining a clean architecture often involves creating abstractions around even standard library functions to maintain SoC. For example, instead of directly using Python’s smtplib library in your use cases, consider creating an abstraction layer for sending notifications.

As we progress through this book, we’ll demonstrate how this effort of creating abstractions pays off in terms of maintainability, flexibility, and testability. You’ll see that the initial investment in Clean Architecture principles yields significant long-term benefits.

Python’s ease of importing can sometimes lead to messy dependency structures, as all packages are effectively public. We’ll show you how to be vigilant about maintaining the Dependency Rule, ensuring that inner layers don’t depend on outer layers. In Chapter 2, we’ll explore techniques and tools to help you maintain clean dependency structures in your Python projects.

Scaling Clean Architecture in Python projects

The application of Clean Architecture principles should be tailored to the size and complexity of your Python project.

For instance, in small projects or quick prototypes, it’s perfectly fine to have a simple, monolithic architecture. However, even in these cases, building in a thoughtful, modular manner can set the stage for future growth. You might start with a simple structure:

Figure 1.3: Quick prototype Python project

In this small project, you can still apply Clean Architecture principles by doing the following:

• Keeping business logic in models.py separate from presentation logic in views.py

• Using dependency injection (DI) to make components more modular and testable

• Defining clear interfaces between modules

As your project grows, you can gradually evolve toward a more comprehensive Clean Architecture structure. This evolution might involve the following:

1. Separating core business logic (entities and use cases) into their own modules

2. Introducing interfaces to abstract away framework-specific code

3. Organizing tests to align with the architectural layers

This book takes a hands-on approach, starting with a basic application and a pragmatic application of Clean Architecture principles. As we progress, the complexity of our example application will increase, demonstrating how to evolve the Clean Architecture approach as the codebase grows.

Clean Architecture is a spectrum, not a binary choice. The patterns we’ll explore represent a comprehensive implementation designed to showcase Clean Architecture’s full capabilities, but in practice, you might choose to implement only the patterns that provide clear value for your specific context. A small API might benefit from clean controller patterns without needing full presenter abstractions, while a data processing script might adopt domain entities while skipping interface adapters entirely. You’ll learn how to apply these principles judiciously, avoiding over-engineering in smaller projects while leveraging the full power of Clean Architecture in larger systems. The key is understanding what each pattern provides so you can make informed decisions about which architectural boundaries matter most for your project.

Leveraging Python’s dynamic nature appropriately

While Python’s dynamic nature is powerful, it can also lead to issues if not used carefully. Chapter 3 is devoted to aspects of Python’s dynamic nature, including duck typing, the use of type hints, and newer features such as protocols. By the end of this chapter, you’ll have a solid foundation on how to best leverage these language features to support a Clean Architecture approach, balancing Python’s flexibility with architectural rigor.

Testing considerations

This book, as with Clean Architecture itself, strongly promotes the use of tests. Tests are essentially first-class clients of your application code, using the codebase and making assertions on the results. The same architectural considerations that apply to your main codebase also apply to your Python tests.

We’ll guide you through writing tests that respect architectural boundaries. You’ll learn to recognize when your tests are indicating potential issues in your architecture, such as when they require excessive setup or mocking. In the test cases for each chapter’s code examples and culminating in Chapter 8, we’ll explore these concepts in depth, showing you how to use tests not just for verification, but as a tool for maintaining and improving your architecture.

By being aware of these considerations and potential pitfalls and following the guidance provided throughout this book, you can create Python systems that are both clean and practical, leveraging the strengths of both Clean Architecture and Python. Remember, the key is to apply these principles thoughtfully, always with an eye toward creating maintainable, testable, and flexible Python code.

Summary

In this chapter, we introduced Clean Architecture at a high level and its relevance to Python development. We gave you context by exploring the evolution of software architecture, from Waterfall to Agile, highlighting persistent challenges in managing complexity, accommodating change, and maintaining long-term productivity.

We introduced Clean Architecture’s core principles:

• Separation of concerns (SoC)

• Independence of external details

• Testability and maintainability

We examined Clean Architecture’s general structure, from core entities and use cases to outer layers of interface adapters, frameworks, and drivers, emphasizing how this structure promotes maintainability and flexibility. We discussed the benefits of Clean Architecture, including improved adaptability, enhanced testability, and long-term development agility, and how it complements modern development practices such as Agile and DevOps.

Furthermore, we explored the natural fit between Clean Architecture and Python, addressing how Python’s features can be leveraged to implement Clean Architecture effectively. We also highlighted Python-specific considerations and potential pitfalls, emphasizing the need to balance Pythonic code with architectural principles and adapt Clean Architecture to different project sizes.

In this chapter, we introduced the primary goals of Clean Architecture and explored its natural fit with Python development. We saw how Clean Architecture principles can be implemented using Python’s features such as ABCs and protocols, providing a foundation for creating maintainable and flexible software systems.

In the next chapter, we’ll build upon this foundation by diving into the SOLID principles. These principles, which form the bedrock of Clean Architecture, will be explored in depth with practical Python examples, showing how they contribute to robust and extensible application design.

Further reading

To learn more about the topics that were covered in this chapter, take a look at the following resources:

• Clean Architecture: A Craftsman’s Guide to Software Structure and Design by Robert C. Martin. This book provides a comprehensive look at Clean Architecture from its originator.

• Domain-Driven Design: Tackling Complexity in the Heart of Software by Eric Evans. While not specific to Clean Architecture, this book provides valuable insights into designing software around business domains.

• Agile Software Development, Principles, Patterns, and Practices by Robert C. Martin. This book covers many of the principles that underpin Clean Architecture in the context of Agile development.

• The Pragmatic Programmer: Your Journey to Mastery by Andrew Hunt and David Thomas. This classic book offers practical advice on software design and development that aligns well with Clean Architecture principles.

To learn Clean Architecture through a series of real-world, code-centric examples and exercises, optimize system componentization, and significantly reduce maintenance burden and overall complexity, check out Clean Architecture with Python by . The book helps you apply Clean Architecture concepts confidently to new Python projects and legacy code refactoring.

Yet, we don't know exactly how and, especially, when to utilize these principles and syntax in practice. In this chapter, we'll discuss some useful applications of the knowledge we've gained, looking at some new topics along the way:

• How to recognize objects

• Data and behaviors, once again

• Wrapping data behaviors using properties

• The Don't Repeat Yourself principle and avoiding repetition

In this chapter, we'll also address some alternative designs for our case study problem. We'll look at ways to partition the sample data into training sets and test sets.

We'll start this chapter with a close look at the nature of objects and their internal state. There are cases when there's no state change, and a class definition isn't desirable.

Treat objects as objects

This may seem obvious; you should generally give separate objects in your problem domain a special class in your code. We've seen examples of this in the case studies in previous chapters: first, we identify objects in the problem, and then model their data and behaviors.

Identifying objects is a very important task in object-oriented analysis and programming. But it isn't always as easy as counting the nouns in short paragraphs that, frankly, the authors have constructed explicitly for that purpose. Remember, objects are things that have both data and behavior. If we are working only with data, we are often better off storing it in a list, set, dictionary, or other Python data structure (which we'll be covering thoroughly in Chapter 7, Python Data Structures). On the other hand, if we are working only with behavior, but no stored data, a simple function is more suitable.

An object, however, has both data and behavior. Proficient Python programmers use built-in data structures unless (or until) there is an obvious need to define a class. There is no reason to add an extra level of complexity if it doesn't help organize our code. On the other hand, the need is not always self-evident.

We can often start our Python programs by storing data in a few variables. As the program expands, we will later find that we are passing the same set of related variables to a set of functions. This is the time to think about grouping both variables and functions into a class.

For example, if we are designing a program to model polygons in two-dimensional space, we might start with each polygon represented as a list of points. The points would be modeled as two tuples (x, y) describing where that point is located. This is all data, stored in a set of nested data structures (specifically, a list of tuples). We can (and often do) start hacking at the command prompt:

>>> square = [(1,1), (1,2), (2,2), (2,1)] 

Now, if we want to calculate the distance around the perimeter of the polygon, we need to sum the distances between each point. To do this, we need a function to calculate the distance between two points. Here are two such functions:

>>> from math import hypot
>>> def distance(p_1, p_2):
...     return hypot(p_1[0]-p_2[0], p_1[1]-p_2[1])
>>> def perimeter(polygon):
...     pairs = zip(polygon, polygon[1:]+polygon[:1])
...     return sum(
...         distance(p1, p2) for p1, p2 in pairs
...     )

We can exercise the functions to check our work:

>>> perimeter(square)
4.0

This is a start, but it's not completely descriptive of the problem domain. We can kind of see what a polygon might be. But we need to read the entire batch of code to see how the two functions work together.

We can add type hints to help clarify the intent behind each function. The result looks like this:

from __future__ import annotations
from math import hypot
from typing import Tuple, List
Point = Tuple[float, float]
def distance(p_1: Point, p_2: Point) -> float:
    return hypot(p_1[0] - p_2[0], p_1[1] - p_2[1])
Polygon = List[Point]
def perimeter(polygon: Polygon) -> float:
    pairs = zip(polygon, polygon[1:] + polygon[:1])
    return sum(distance(p1, p2) for p1, p2 in pairs)

We've added two type definitions, Point and Polygon, to help clarify our intentions. The definition of Point shows how we'll use the built-in tuple class. The definition of Polygon shows how the built-in list class builds on the Point class.

When writing annotations inside method parameter definitions, we can generally use the type name directly, for example, def method(self, values: list[int]) -> None:. For this to work, we need to use from __future__ import annotations. When defining a new type hint, however, we need to use the names from the typing module. That's why the definition of the new Point type uses typing.Tuple in the expression Tuple[float, float].

Now, as object-oriented programmers, we clearly recognize that a polygon class could encapsulate the list of points (data) and the perimeter function (behavior). Further, a Point class, such as we defined in Chapter 2, Objects in Python, might encapsulate the x and y coordinates and the distance method. The question is: is it valuable to do this?

For the previous code, maybe yes, maybe no. With our recent experience in object-oriented principles, we can write an object-oriented version in record time. Let's compare them as follows:

from math import hypot
from typing import Tuple, List, Optional, Iterable
class Point:
    def __init__(self, x: float, y: float) -> None:
        self.x = x
        self.y = y
    def distance(self, other: "Point") -> float:
        return hypot(self.x - other.x, self.y - other.y)
class Polygon:
    def __init__(self) -> None:
        self.vertices: List[Point] = []
    def add_point(self, point: Point) -> None:
        self.vertices.append((point))
    def perimeter(self) -> float:
        pairs = zip(
            self.vertices, self.vertices[1:] + self.vertices[:1])
        return sum(p1.distance(p2) for p1, p2 in pairs)

There seems to be almost twice as much code here as there was in our earlier version, although we could argue that the add_point method is not strictly necessary. We could also try to insist on using _vertices to discourage the use of the attribute, but the use of leading _ variable names doesn't seem to really solve the problem.

Now, to understand the differences between the two classes a little better, let's compare the two APIs in use. Here's how to calculate the perimeter of a square using the object-oriented code:

>>> square = Polygon()
>>> square.add_point(Point(1,1))
>>> square.add_point(Point(1,2))
>>> square.add_point(Point(2,2))
>>> square.add_point(Point(2,1))
>>> square.perimeter()
4.0

That's fairly succinct and easy to read, you might think, but let's compare it to the function-based code:

>>> square = [(1,1), (1,2), (2,2), (2,1)]
>>> perimeter(square)
4.0

Hmm, maybe the object-oriented API isn't so compact! Our first, hacked-in version, without type hints or class definitions, is the shortest. How do we know what the list of tuples is supposed to represent? How do we remember what kind of object we're supposed to pass into the perimeter function? We needed some documentation to explain how the first set of functions should be used.

The functions annotated with type hints were quite a bit easier to understand, as were the class definitions. The relationships among the objects are more clearly defined by hints or classes or both.

Code length is not a good indicator of code complexity. Some programmers get hung up on complicated one-liners that do an incredible amount of work in one line of code. This can be a fun exercise, but the result is often unreadable, even to the original author the following day. Minimizing the amount of code can often make a program easier to read, but do not blindly assume this is the case.

No one wins at code golf. Minimizing the volume of code is rarely desirable.

Luckily, this trade-off isn't necessary. We can make the object-oriented Polygon API as easy to use as the functional implementation. All we have to do is alter our Polygon class so that it can be constructed with multiple points.

Let's give it an initializer that accepts a list of Point objects:

class Polygon_2:
    def __init__(self, vertices: Optional[Iterable[Point]] = None) -> None:
        self.vertices = list(vertices) if vertices else []
    def perimeter(self) -> float:
        pairs = zip(
            self.vertices, self.vertices[1:] + self.vertices[:1])
        return sum(p1.distance(p2) for p1, p2 in pairs)

For the perimeter() method, we've used the zip() function to create pairs of vertices, with items drawn from two lists to create a sequence of pairs. One list provided to zip() is the complete sequence of vertices. The other list of vertices starts from vertex 1 (not 0) and ends with the vertex before 1 (that is, vertex 0). For a triangle, this will make three pairs: (v[0], v[1]), (v[1], v[2]), and (v[2], v[0]). We can then compute the distance between the pairs using Point.distance(). Finally, we sum the sequence of distances. This seems to improve things considerably. We can now use this class like the original hacked-in function definitions:

>>> square = Polygon_2(
... [Point(1,1), Point(1,2), Point(2,2), Point(2,1)]
... )
>>> square.perimeter()
4.0

It's handy to have the details of the individual method definitions. We've built an API that's close to the original, succinct set of definitions. We've added enough formality to be confident the code is likely to work before we even start putting test cases together.

Let's take one more step. Let's allow it to accept tuples too, and we can construct the Point objects ourselves, if needed:

Pair = Tuple[float, float]
Point_or_Tuple = Union[Point, Pair]
class Polygon_3:
    def __init__(self, vertices: Optional[Iterable[Point_or_Tuple]] = None) -> None:
        self.vertices: List[Point] = []
        if vertices:
            for point_or_tuple in vertices:
                self.vertices.append(self.make_point(point_or_tuple))
    @staticmethod
    def make_point(item: Point_or_Tuple) -> Point:
        return item if isinstance(item, Point) else Point(*item)

This initializer goes through the list of items (either Point or Tuple[float, float]) and ensures that any non-Point objects are converted to Point instances.

If you are experimenting with the above code, you should also define these variant class designs by creating subclasses of Polygon and overriding the __init__() method. Extending a class with dramatically different method signatures can raise error flags from mypy.

For an example this small, there's no clear winner between the object-oriented and more data-oriented versions of this code. They all do the same thing. If we have new functions that accept a polygon argument, such as area(polygon) or point_in_polygon(polygon, x, y), the benefits of the object-oriented code become increasingly obvious. Likewise, if we add other attributes to the polygon, such as color or texture, it makes more and more sense to encapsulate that data into a single class.

The distinction is a design decision, but in general, the more important a set of data is, the more likely it is to have multiple functions specific to that data, and the more useful it is to use a class with attributes and methods instead.

When making this decision, it also pays to consider how the class will be used. If we're only trying to calculate the perimeter of one polygon in the context of a much greater problem, using a function will probably be quickest to code and easier to use one time only. On the other hand, if our program needs to manipulate numerous polygons in a wide variety of ways (calculating the perimeter, area, and intersection with other polygons, moving or scaling them, and so on), we have almost certainly identified a class of related objects. The class definition becomes more important as the number of instances increases.

Additionally, pay attention to the interaction between objects. Look for inheritance relationships; inheritance is impossible to model elegantly without classes, so make sure to use them. Look for the other types of relationships we discussed in Chapter 1, Object-Oriented Design, association and composition.

Composition can, technically, be modeled using only data structures – for example, we can have a list of dictionaries holding tuple values – but it is sometimes less complicated to create a few classes of objects, especially if there is behavior associated with the data.

One size does not fit all. The built-in, generic collections and functions work well for a large number of simple cases. A class definition works well for a large number of more complex cases. The boundary is hazy at best.

Adding behaviors to class data with properties

Throughout this book, we've focused on the separation of behavior and data. This is very important in object-oriented programming, but we're about to see that, in Python, the distinction is uncannily blurry. Python is very good at blurring distinctions; it doesn't exactly help us to think outside the box. Rather, it teaches us to stop thinking about the box.

Before we get into the details, let's discuss some bad object-oriented design principles. Many object-oriented developers teach us to never access attributes directly. They insist that we write attribute access like this:

class Color:
    def __init__(self, rgb_value: int, name: str) -> None:
        self._rgb_value = rgb_value
        self._name = name
    def set_name(self, name: str) -> None:
        self._name = name
    def get_name(self) -> str:
        return self._name
    def set_rgb_value(self, rgb_value: int) -> None:
        self._rgb_value = rgb_value
    def get_rgb_value(self) -> int:
        return self._rgb_value

The instance variables are prefixed with an underscore to suggest that they are private (other languages would actually force them to be private). Then, the get and set methods provide access to each variable. This class would be used in practice as follows:

>>> c = Color(0xff0000, "bright red")
>>> c.get_name()
'bright red'
>>> c.set_name("red")
>>> c.get_name()
'red'

The above example is not nearly as readable as the direct access version that Python favors:

class Color_Py:
    def __init__(self, rgb_value: int, name: str) -> None:
        self.rgb_value = rgb_value
        self.name = name

Here's how this class works. It's slightly simpler:

>>> c = Color_Py(0xff0000, "bright red")
>>> c.name
'bright red'
>>> c.name = "red"
>>> c.name
'red'

So, why would anyone insist upon the method-based syntax?

The idea of setters and getters seems helpful for encapsulating the class definitions. Some Java-based tools can generate all the getters and setters automagically, making them almost invisible. Automating their creation doesn't make them a great idea. The most important historical reason for having getters and setters was to make the separate compilation of binaries work out in a tidy way. Without a need to link separately compiled binaries, this technique doesn't always apply to Python.

One ongoing justification for getters and setters is that, someday, we may want to add extra code when a value is set or retrieved. For example, we could decide to cache a value to avoid complex computations, or we might want to validate that a given value is a suitable input.

For example, we could decide to change the set_name() method as follows:

class Color_V:
    def __init__(self, rgb_value: int, name: str) -> None:
        self._rgb_value = rgb_value
        if not name:
            raise ValueError(f"Invalid name {name!r}")
        self._name = name
    def set_name(self, name: str) -> None:
        if not name:
            raise ValueError(f"Invalid name {name!r}")
        self._name = name

If we had written our original code for direct attribute access, and then later changed it to a method like the preceding one, we'd have a problem: anyone who had written code that accessed the attribute directly would now have to change their code to access a method. If they didn't then change the access style from attribute access to a function call, their code would be broken.

The mantra that we should make all attributes private, accessible through methods, doesn't make much sense in Python. The Python language lacks any real concept of private members! We can see the source; we often say "We're all adults here." What can we do? We can make the syntax distinction between attribute and method less visible.

Python gives us the property function to make methods that look like attributes. We can therefore write our code to use direct member access, and if we ever unexpectedly need to alter the implementation to do some calculation when getting or setting that attribute's value, we can do so without changing the interface. Let's see how it looks:

class Color_VP:
    def __init__(self, rgb_value: int, name: str) -> None:
        self._rgb_value = rgb_value
        if not name:
            raise ValueError(f"Invalid name {name!r}")
        self._name = name
    def _set_name(self, name: str) -> None:
        if not name:
            raise ValueError(f"Invalid name {name!r}")
        self._name = name
    def _get_name(self) -> str:
        return self._name
    name = property(_get_name, _set_name)

Compared to the earlier class, we first change the name attribute into a (semi-)private _name attribute. Then, we add two more (semi-)private methods to get and set that variable, performing our validation when we set it.

Finally, we have the property construction at the bottom. This is the Python magic. It creates a new attribute on the Color class called name. It sets this attribute to be a property. Under the hood, a property attribute delegates the real work to the two methods we just created. When used in an access context (on the right side of the = or :=), the first function gets the value. When used in an update context (on the left side of = or :=), the second function sets the value.

This new version of the Color class can be used in exactly the same way as the earlier version, yet it now performs validation when we set the name attribute:

>>> c = Color_VP(0xff0000, "bright red")
>>> c.name
'bright red'
>>> c.name = "red"
>>> c.name
'red'
>>> c.name = ""
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "setting_name_property.py", line 8, in _set_name
    raise ValueError(f"Invalid name {name!r}")
ValueError: Invalid name ''