This hands-on masterclass takes you from LangGraph fundamentals to supervisor and hierarchical multi-agent systems, with live debugging in LangSmith throughout.

Deep Engineering readers save 50% with code - DEEPENG50.

Register here →

✍️ From the editor’s desk,

Welcome to the 58th issue of Deep Engineering!

On August 3, NTT announced that it has signed a capital and business alliance with OptQC, the University of Tokyo spinout building optical quantum processors, with both companies aiming at a fault-tolerant machine of one million qubits. The announcement from NTT lays out a phased roadmap. The companies aim to complete the system architecture and key component technologies by fiscal 2027 alongside a practical 10,000 qubit system, then begin verification work in fiscal 2028 and deliver a platform for running optical and classical machines together the year after.

One million is the largest number the field has yet attached to a headline, and it arrives in a year when the reported metric already shifted once. Through the first half of 2026 vendors moved from physical qubit counts to logical qubit counts as the figure worth announcing. Both are real results, and both leave out the properties that decide what a machine can actually compute, which is where this issue picks up.

Sebastian Hassinger has read claims like these from inside the companies making them, first on the IBM Quantum team and later leading go to market for AWS Quantum Technologies. He wrote The New Quantum Era for readers without a physics background, and today he walks us through which numbers carry the information and which ones do not.

You can watch the full session or read the transcript here.

Let’s get started.

Agent-written TLA+

An agent wrote our TLA+ spec. The model checker explored 14.3M states and caught a real race.

Read the write-up

Expert Insight

Qubit Count Measures Register Size, Not Capability

by with

Three logical qubit results landed in the first half of 2026, and in each one the number worth reading is not the number in the headline. QuEra published 96 logical qubits encoded across 448 neutral atoms, a ratio near five to one. Quantinuum reported 48 logical qubits drawn from 98 trapped ions, closer to two to one. IBM’s published fault tolerance roadmap targets 200 logical qubits from roughly 10,000 physical ones by 2029, a ratio near fifty to one.

Those ratios differ by an order of magnitude because the underlying codes differ, and the choice of code decides whether a logical qubit corrects errors or only detects them. A count reported on its own collapses all of that into a single integer, which is why the integer tells you very little about what the machine computes.

Sebastian Hassinger worked on the IBM Quantum team and later led go to market for AWS Quantum Technologies, and he wrote The New Quantum Era to give engineers without a physics background enough grounding to read results like these directly. During our interview when I asked him what actually carries information in a milestone result, he began by taking apart the metric the field has reported for a decade. “Qubit count is effectively the register size of that computer,” he said. Then came the harder line, aimed at a claim the field now repeats freely. “Anytime you hear somebody saying quantum computing is just a matter of engineering now, be suspicious of that person’s claims.”

Register size bounds information, not computation

The clearest demonstration that a raw count says little about capability comes from IBM’s own hardware history. And Hassinger was there for it. The technical roadmap produced a chip called Condor at just over a thousand qubits, which he describes as genuinely valuable for the research and fabrication effort it took to build. But it saw little use. Connectivity between qubits on the chip was low and the noise proved very difficult to manage, so researchers went back to the smaller machines in the 127 to 133 qubit range, which were more capable in practice.

The reason is architectural rather than numerical. Register size sets an upper bound on how much information you can load, and nothing beyond that. Hassinger points out that QuEra’s 256 qubit Aquila holds 256 bits at a time, which sounds unremarkable until you entangle those qubits and produce a state vector of two to the 256, a computational space you cannot physically recreate on classical hardware. The capability lives in the entanglement structure and in how well the problem maps onto it, so a machine with more qubits and worse connectivity computes less than a smaller machine with better ones.

That same logic now applies one level up. A logical qubit is an encoding, not a unit, and its value depends on the code family, the physical to logical ratio, and the error model the code assumes. Codes at distance two detect errors without correcting them, which is a different guarantee from correction, and the difference produced considerable argument when Microsoft and Quantinuum reported reliable logical qubits in 2024 using error detection with post-selection. Two systems reporting the same logical qubit count can therefore be doing categorically different things.

Code distance carries the information a count discards

Hassinger’s proposed substitute is quite specific, and it happens to be exactly the property that separates those cases. Fidelity and noise are what matter, he reasons, particularly the fidelity of one and two qubit gates, where two qubit operations mean entanglement. Those figures are hard to extract from a published result, so he offers a proxy that survives summarization.

Read the resilience of the error correction code, expressed as a distance or a d value. That is roughly how many errors the system absorbs before the encoded information collapses and the computation is lost, so a higher distance means a more resilient machine. He points to the Willow experiment as the useful reference, roughly a hundred physical qubits arranged in a surface code presenting as one logical qubit at distance seven. That snapshot carries what you need without the underlying gate fidelities, because the only two questions that determine what you can run are “how many logical qubits do I get, and how resilient is that error correction.”

Pair a logical qubit count with its code, its distance, and its encoding ratio and you have something you can reason about. Take the count alone and you have an integer that happens to increase.

Speculation hardens into certainty before it reaches you

There is a structural reason the public record runs ahead of the results, and it operates on the way from the lab to the summary rather than inside the science. Hassinger named the pressure that drives it, and he was unusually direct about where the gap opens.

“Since at least the beginning of the Q2B conferences put on by QCWare, there has been a recurring chorus demanding to know what are quantum computing’s use cases, how will it be useful for enterprises,” he told us. “Marketing can be tempted to take speculative ideas and present them as certainties, stretching the truth about a scientist’s speculation to reframe it as definitive. The other question is always when, so timelines are also something that marketing can take liberties with.”

Both distortions are directional, which makes them correctable. A researcher’s conditional loses its condition, and a scientific dependency acquires a date. Reading a result back through those two transformations usually recovers something close to the original claim.

Roadmaps model engineering determinism onto unsolved physics

The deeper issue Hassinger identifies is a category problem. A roadmap projects milestones one year out, three years, five years, and he is blunt about what kind of document that is. “That’s an engineering document,” he says, “and engineering is much more deterministic than the underlying scientific breakthroughs that are required to enable the engineering to deliver those milestones.”

Transduction is the concrete case. Superconducting qubits operate inside a dilution refrigerator near absolute zero, and a refrigerator has finite volume, so scaling past one fridge means entangling qubits across separate cryostats. That requires converting the quantum state to a photonic frequency used in telecom, carrying it over fiber as what the field calls flying qubits, then converting back at the far end. None of the known conversion methods delivers the fidelity a reliable device needs, and nobody yet knows what closing that gap requires. “It’s not just hard work,” he says of that class of problem. “It’s a lot of hard work, but it’s also luck, because we don’t know what we don’t know.”

This is the reason he treats specifications and milestone dates as the least informative part of any hardware program, and the unsolved science underneath as the part that determines whether the dates mean anything. It is also why his sharpest formulation of the field’s position lands where it does. “A qubit is a very interesting device with no intrinsic commercial value,” he says, and converting it into something useful still depends on physics nobody has finished.

Classical simulability is the only threshold that changes anything

Hassinger’s position does not end in skepticism, because the field is converging on one measurable target regardless of which architecture reaches it. “The consensus is we need to deliver fault tolerant logical qubits at a scale that is not simulatable by a classical computer,” he says. “That’s the North Star we’re all sailing towards.” Once a system passes the point where your laptop or your GPU cluster can reproduce its output, running it on quantum hardware becomes necessary rather than interesting, and nothing before that crossing changes what you can compute.

That threshold also tells you where the physics pays off first, and his answer is narrower than the general coverage implies. Materials science arrives first because condensed matter behaviour maps naturally onto these systems, with small molecule chemistry close behind, while optimization, cryptography, and machine learning all wait on thousands of logical qubits.

So the technical reading is straightforward. When a new result publishes, work out its encoding ratio and its code distance before you compare it to anything, since those two numbers determine what the machine tolerates and the logical qubit count does not. And when a roadmap updates, separate the engineering milestones from the scientific dependencies underneath them and check which unsolved physics the far dates rest on. Then put the effort into building quantum intuition inside your own team, because recognizing the high dimensional structure in your own problems transfers whichever architecture crosses the threshold first.

In case you missed

🛠️ Tool of the Week

Stim is an open source stabilizer circuit simulator maintained under Google’s quantumlib organisation, built for analysing quantum error correction circuits at speed.

Highlights

• Derives a circuit’s actual code distance instead of relying on the number a vendor publishes.

• Turns a noisy circuit into a detector error model ready to configure matching-based decoders.

• Samples circuits with thousands of qubits and millions of operations at kilohertz rates.

• Installs as a Python package and also runs as a C++ library or a command line tool.

Learn more about Stim

📎 Tech Briefs

• Anthropic ships inference hooks for Claude Enterprise - Governed prompts now route to customer security servers before inference, centralizing DLP across Claude Enterprise surfaces.

• OpenAI cuts GPT-5.6 prices and adds Fast mode - Luna and Terra get lower API pricing, while Sol Fast mode offers 2.5× speed at 2× cost.

• D-Wave and Nasdaq Verafin agree a quantum proof of concept for financial crime detection - Nasdaq Verafin will test D-Wave quantum-hybrid workflows on hundreds of financial-crime signals and network relationships.

• Cloud SQL makes PSC reconciliation default - New or newly enabled PSC instances now close existing connections after project removal from allowed lists.

• Palo Alto Networks integrates Prisma AIRS with Claude Enterprise - Prisma AIRS can inspect Claude prompts before inference, applying existing DLP policies across Claude Enterprise surfaces.

That’s all for today. Thank you for reading this issue of Deep Engineering.

We’ll be back next week with more expert-led content.

Keep building,

Saqib Jan

Editor-in-Chief, Deep Engineering

If your company wants to reach senior developers, software engineers, and technical decision-makers, speak to us about partnering with Deep Engineering.

By Sebastian Hassinger, author of The New Quantum Era and former quantum lead at AWS and IBM | This piece is adapted from his live Deep Engineering session, Quantum Computing Beyond the Hype. Edited by Saqib Jan

People list the same five domains whenever they ask where quantum computing will actually pay off, so chemistry, materials, optimization, cryptography, and machine learning. Of those, materials science is closest to real value, with small molecule chemistry a close second, and small molecule chemistry is sometimes treated as a type of materials science anyway. Optimization, cryptography, and machine learning all need thousands of logical qubits, so those are considerably further off.

The reason materials leads is that materials science is condensed matter physics. It is about how atoms pack together into a lattice, into a crystalline structure, and how the material behaves as a result of that packing. All of that comes down to physics calculations, which is precisely the kind of work a quantum computer should do well. There is already interesting research going on around battery technology, using quantum information and quantum computing to help with designing and inventing new battery chemistries.

What sits behind that work is more ambitious. The aspiration is that if we can simulate materials precisely enough at sufficient scale, we can start creating designer materials. Maybe something much lighter for the same strength, or much stronger for the same weight. Maybe photosynthesis built into the material itself, so you coat your car in something that generates electricity without any cells on the outside. The imagination gets stimulated by the idea of engineering the attributes of a new material at atomic scale, and that is one of the genuinely underestimated parts of this whole story.

Chemistry follows for the same underlying reason, because chemistry is quantum mechanical at its core. It is the way atoms interact with one another, and that interaction is a quantum mechanical phenomenon happening at scale, which is what makes it so difficult to simulate precisely. People talk about small molecule chemistry as the next frontier beyond materials, and there is a natural leap from small molecule chemistry to pharmaceuticals, which tend to be larger and more complex molecules. You need a bigger machine to simulate them.

Provided we can build one large enough, you can easily imagine drug discovery and research happening inside a very high fidelity simulation, with all the acceleration we are used to getting from computer simulation. The pipeline could compress quite a bit, because you might screen a whole set of drug candidates in simulation before you ever formulate anything, and by the time you are making the drug in the real world you already know how it will interact with the human system.

The word doing the work in all of this is fidelity. Classical simulation of chemistry is approximate by necessity. Methods like DMRG throw away a great deal of information about the reaction because there is simply too much to calculate, so we have devised ways to focus algorithmically on the small slice of the problem we hope matters most to the answer. The promise of quantum computing is not throwing away as much, and getting a simulation that captures more of the real dynamics of the system accurately.

So if you are working out where to point attention, watch the physical sciences rather than your own industry for the first real result. And treat any near term claim about quantum optimization, quantum machine learning, or breaking encryption with the logical qubit count in mind, because those applications are waiting on hardware that does not exist yet.

Read the full issue

This piece comes from a longer conversation on how to read quantum progress honestly. The complete interview and the rest of this week’s Deep Engineering issue are available here.

By Sebastian Hassinger, author of The New Quantum Era and former quantum lead at AWS and IBM | This piece is adapted from his live Deep Engineering session, Quantum Computing Beyond the Hype. Edited by Saqib Jan

It is smart for enterprises to start investing in the skills you need for understanding quantum information, and in exploring potential algorithms now, even though the computers that could actually run anything useful do not exist yet. That sounds premature until you look at what the work actually involves, because it takes a great deal of effort to recast a business problem into quantum terms.

We have been thinking about these problems in classical terms, in Boolean logic terms, since the middle of the last century. That is seventy or eighty years now, and it will be a hundred before too long. So we are carrying a very deeply ingrained set of preconceptions about problem solving. We look at our world through the lens of how our laptop could fix the problem in front of us. We write code on those machines, and our minds run along the lines of decomposing the problem, working out what the dynamics are, and figuring out how to represent them efficiently. All of that rests on an invisible reliance on the assumption that at the ground level you are doing Boolean algebra to solve the problem.

I do not think many people are fully aware of how much of their problem solving thinking is rooted in the way classical computing works. And it is so different in quantum computing that the gap becomes the real obstacle. People talk about developing quantum intuition, which means building the habit of looking at the world through the lens of linear algebra and through the dynamics and capabilities of quantum information. That takes a lot of work, and it is not the kind of work you can compress once the hardware arrives.

The smartest enterprise approaches to quantum computing I have seen understand this. They hire a small number of strong people and run research alongside quantum hardware companies and academic researchers at the top of the field. JPMorgan does this well. You can imagine an organization that size has a very large set of challenges, algorithms, and tasks it has to carry out to operate as a financial entity, and the team there looks at theoretical problems that have some mapping to an aspect of those business processes, then does exploratory research against them. They publish open science papers, so everybody benefits from the work they put in.

What they are really doing is building the muscle. When quantum technologies mature to sufficient scale, JPMorgan will know how to use them, because the people there will already have years of thinking in the right terms behind them. That head start cannot be bought later. The hardware will become available to everyone at roughly the same moment, and the differentiator will be whether your organization has anyone who can look at a business problem and recognize the high dimensional structure inside it.

Two things follow from this if you are deciding where to put effort right now. Pick one or two people who are genuinely curious and give them real time to build quantum intuition, rather than sending the whole team to an introductory session that changes nothing. And point them at a problem you actually have, something with heavily interconnected variables, so the learning attaches to your business instead of staying abstract.

Read the full issue

This piece comes from a longer conversation on how to read quantum progress honestly. The complete interview and the rest of this week’s Deep Engineering issue are available here.

This session was recorded live as part of the Deep Engineering Live Interview Series. The transcript below has been lightly edited for clarity and readability. Audience members joined the conversation and asked questions directly during the session.

Tell us how you came to quantum computing, and where the field stands right now.

My career has been in emerging tech going back to the early nineties, from the early Internet through the arrival of the web. I built the first Apple support site on the web, started a couple of Internet service providers, and worked on innovation inside large companies like IBM and Apple, alongside advising and co-founding startups. So I have moved from one emerging technology to the next for most of my working life.

Quantum found me in 2017. I had been asked to help IBM with the open source strategy for Qiskit, and my first move was to attend the Think Summit at the T.J. Watson Research Lab, where the team was talking about the 53 qubit machine they would launch the following year. It was about ninety five percent incomprehensible to me then, but I could tell it was novel enough and early enough that I would not need to look for another emerging field for the rest of my career. I have always been drawn to quantum physics even without a physics background, so it became an obsession, and within a year I was working on the IBM Quantum team.

On where things stand, the theory goes back to the early eighties, but physical qubits and real proof that you could compute with them did not arrive until around 2000. So this is roughly twenty five years of work, and we are now close to machines that do things classical computers genuinely cannot. Everything so far has been experimental, and your laptop can still simulate a quantum computer better than a current quantum computer can. That is changing fast, and before the end of the decade we should see machines at a scale where they do meaningful work you cannot do classically. That shift drives most of the recent interest.

Strip away the metaphors. What is a quantum computer actually doing that a classical machine cannot?

A lot of the metaphors people use are misleading, so let me try two framings that work for me. A classical computer is built by etching circuits into silicon that switch on and off to represent Boolean logic, and Boolean logic underpins everything it does. A qubit instead uses a two level system from nature, an atom, a superconducting circuit, or a photon, something with two energy levels you can control and read as zero and one. The difference is that those states can exist in superposition, where the value behaves like a wave function sitting probabilistically between zero and one, so a single qubit acts as a vector. Once you have many of them you are doing linear algebra rather than Boolean logic, and linear algebra is very good at high dimensional problems where the variables are heavily interconnected.

The traveling salesman problem is the usual illustration, even if physicists quibble with it. Finding the optimal route through every city is combinatorial, and in a classical machine each new city roughly doubles the combinations you have to test. With qubits you effectively add one qubit to expand the computational space, because the qubits become the exponent for that space rather than a linear count. That is what people mean when they say it would take more bits than there are atoms in the universe to represent a few hundred logical qubits.

The second framing is simpler. You are using quantum systems from nature to simulate other systems from nature. Feynman made this point in his 1981 keynote that many treat as the starting gun for the field, that simulating nature is inherently hard because of the exponential complexity in these many body systems, and that a computer built from the same kind of system should handle the problem far better. The pithy version is that if you want to simulate a system from nature, you need a system from nature.

Superposition and entanglement get used loosely. Which one do people most often misunderstand, and what is the right intuition?

The biggest issue is more an ambiguity than a flat error. People often say a quantum computer tests every solution at once and then selects the right one. That is not literally correct, and the confusion usually traces back to the most famous algorithm in the field, Shor’s algorithm, which Peter Shor discovered in the early nineties. Shor’s algorithm is the one that would let us break RSA and other asymmetric encryption, because it factors a very large number down to its primes, and that factoring is the basis for most modern encryption.

It works through quantum phase estimation. Inside the very large computational space created by many entangled qubits, often called a Hilbert space that scales as two to the power of the number of qubits, the algorithm deliberately creates interference patterns. The wave functions interfere so the correct answer gets amplified and the wrong answers get suppressed. That resembles trying everything at once, which is why the shorthand persists. I once asked Peter how he felt about people describing it that way, and after thinking about it he said, well, it is not really wrong. With anything quantum there is rarely a clean black and white, because this behavior lives so far from our lived experience that any physical or human metaphor introduces some distortion. The communication challenge of grappling with these foreign ideas is part of what makes the field fascinating to me.

Where does that leave quantum computing for AI and machine learning, and why do we know so few quantum algorithms?

There are a few algorithms with a proven theoretical advantage, HHL among them, plus variations that fan out from that small set. I used to find the short list worrying, but I no longer do, and the reason is the history of classical computing. When these machines were being built in the mid twentieth century, nobody knew what they would be good for, and they did not know for a long time. Von Neumann led a team building a machine at the Institute for Advanced Studies in Princeton around the same time the ENIAC was being built at Penn, both driven by the difficulty of the physics calculations that the Manhattan Project and other wartime work demanded.

A mathematician named Stan Ulam, working with von Neumann, came up with a sampling method for calculating neutron diffusion, and he called it the Monte Carlo algorithm. It was more than thirty years before someone thought to use Monte Carlo to optimize a portfolio. The technique existed for decades before anyone saw its value outside the original problem. I expect the same pattern with quantum computing, hopefully faster. These machines will let physicists and chemists simulate many body systems and run experiments they cannot run classically, and the techniques they invent for their narrow problems will very likely carry unexpected value into other domains. So the real work for the rest of us is to watch the early users closely and look for techniques we can generalize into algorithms with value in other industries.

Where does quantum genuinely have an advantage, and where will classical computing stay ahead for the foreseeable future?

It helps to start by correcting an instinct. People arriving at the topic assume quantum computers are faster, or built for bigger data. They are actually slower machines working with smaller data. Qubit counts are in the hundreds now, and we hope to reach thousands, maybe tens of thousands. Qubit count is effectively the register size of the machine. QuEra’s Aquila, a 256 qubit neutral atom device, can load 256 bits of information at a time, so on its own that does not sound impressive. The advantage appears when you entangle those qubits, because you have created a many body system whose state vector is two to the 256, which is more states than there are atoms in the universe. You cannot physically recreate that computational space classically.

So the rule of thumb is that quantum has a natural advantage when the problem is high dimensional and heavily interconnected, because vectors and linear algebra represent that kind of space far more efficiently than Boolean logic can. Those problems are abundant in nature. Material science is condensed matter physics, how atoms pack into a lattice and how the resulting material behaves, and that is exactly the kind of simulation quantum computers should do well. There is promising work already on battery design using quantum approaches, and a real aspiration that if we can simulate materials precisely enough we can design new ones, much lighter for the same strength, or with properties like photosynthesis built into the material itself.

Chemistry is quantum mechanical at its core, because it is about how atoms interact, so it is also very hard to simulate precisely. People point to small molecule chemistry as the next frontier beyond materials, and there is a natural leap from there to pharmaceuticals, which are larger and more complex molecules that need bigger machines. If we can build a quantum computer large enough to simulate how a drug candidate behaves and interacts with its target in the body, drug discovery could compress a great deal, because you screen candidates in high fidelity simulation before you ever formulate them. Beyond the physical sciences people talk about optimization and the Shor class of cryptography algorithms, but those are the small number we already know. The rest will fall out of using machines we cannot simulate classically, which is a chicken and egg situation. We know the physical sciences will benefit, and the broader applications in finance or logistics may surprise us, because we cannot imagine them until the machine exists at that scale.

Enterprises often assume quantum will speed up their existing workloads. For which problems is that assumption simply wrong?

The easiest answer is that if your problem is processing large volumes of data quickly, a quantum computer will not do that in the near term, and possibly never, because classical computing keeps advancing too. I avoid saying never about any of this, but that is a game classical may always lead. The clearer way to see the boundary is through fidelity. In classical chemistry simulation we use methods like DMRG that deliberately throw away much of the information about a reaction because there is too much to calculate, so we focus algorithmically on the small part we hope matters most. The promise of quantum is a much higher fidelity simulation that keeps more of the real dynamics. So the enterprise question to ask is whether your use case involves genuinely high dimensional, highly interconnected data. Optimization is one area with potential, because the more interconnected the parameters, the higher dimensional the problem and the harder it is combinatorially, and a larger quantum computer could represent that more efficiently than classical approaches like max cut.

Even so, it is smart for enterprises to invest now in the skills to understand quantum information and explore algorithms, even though no machine can run anything useful yet. Recasting a business problem takes real effort, because we have thought in classical, Boolean terms since the middle of the last century, close to eighty years. That is a deeply ingrained set of assumptions. We look at the world through the lens of what a laptop can do, and most people are not aware how much of their problem solving quietly assumes Boolean algebra at the ground level. Building quantum intuition, learning to see problems through linear algebra, takes a lot of work. The smartest enterprise approaches I have seen hire a small number of strong people and run research with hardware companies and academics. The team at JPMorgan, for instance, studies theoretical problems that map to aspects of their business, publishes open science, and builds the muscle to apply quantum technologies once the machines mature enough to matter.

Give us an honest read on the hardware. How far are we from machines that do useful work beyond what classical systems already handle?

We are on the cusp, and before the end of the decade we should see machines doing meaningful work beyond classical. But there are still deep scientific unknowns across every modality the vendors are pursuing. It is like the early days of classical computing, when the question was not only how to fit more transistors on a chip but which materials and fabrication processes would even work. Those were genuine unknowns the industry tackled over years, and only in aggregate did it look like a smooth curve. We are at that very early stage in a lot of ways.

Transduction is a concrete example. With superconducting qubits you keep them in a dilution refrigerator near absolute zero, and a fridge has limited space, so to scale you have to connect the qubits at the bottom of one fridge to those in another. That means converting them to a photonic frequency used in telecom, carrying the quantum value over fiber, then converting back to the native frequency in the second fridge. None of the ways we currently know how to do that delivers the fidelity the device needs to operate reliably, and we do not yet know what it will take to fix it. That is a scientific challenge, not just hard engineering, and scientific challenges involve luck, because you do not know what you do not know.

This is why roadmaps are difficult to read. A roadmap is an engineering document projecting deterministic milestones, one year out, three years, five years. Engineering is far more deterministic than the scientific breakthroughs required to enable it, so when I talk to investors I tell them to do deep due diligence on the scientific challenges a hardware company still faces. For an end user the specifics of any single vendor almost do not matter. What matters is the North Star the whole industry sails toward, fault tolerant logical qubits at a scale you cannot simulate on a laptop or a GPU cluster. Once you cannot simulate it classically, you have to run it on a quantum computer, and that is the only line that counts.

Error correction keeps getting described as the bottleneck. What actually changed in the last year, and what has not?

Start with what qubit count actually tells you, which is less than people think, because it is effectively register size, not a measure of progress. We have had machines with thousands of qubits before. At IBM the roadmap produced a chip called Condor at just over a thousand qubits, and it had real value for the R&D effort of designing and fabricating it, but the connectivity was so low and the noise so hard to manage that even researchers preferred going back to the smaller machines around 127 to 133 qubits, which were more capable. So raw qubit number is not an indicator of progress toward usefulness.

What matters is fidelity and noise, and a good proxy is the resilience of the error correction code, which you talk about as a distance or a d value. That is roughly how many errors the system can absorb before the information collapses and you lose the computation, so a higher distance means a more resilient system. The Willow experiment about a year and a half ago was distance seven, if I remember, roughly a hundred physical qubits in a surface code presenting as one logical qubit. That is the useful snapshot, because as an end user you do not need the underlying gate fidelities, you need to know how many logical qubits you get and how resilient the error correction is. What has not changed is that fundamental science is still involved, so anytime you hear someone say quantum is just a matter of engineering now, be suspicious of their claims.

What does getting quantum ready actually mean, and how does a classical developer start?

Quantum ready means different things by context, but broadly it means developing some intuition for what a quantum information approach to a problem looks like. For an existing software developer the best way in is usually coding, and the tools are familiar. Most quantum programming is done in Python. Qiskit is IBM’s Python SDK, and Amazon Braket, which I worked on at AWS, is also Python, along with several others in well understood languages. The way you construct a task and send it to a quantum computer uses tools we already know. The hard part is the logic of the circuit itself, so there are gentle entry points to start building familiarity.

One of my favorites is the Unitary Foundation, where I am a fellow. Every year they run Unitary Hack, a global event over a couple of weeks where maintainers of open source quantum software tag issues in their repositories and developers close them for bounties. The issues are often housekeeping, security, or maintainability work that any classical developer recognizes, and as a side effect you see how a quantum software package works inside. My favorite example is an engineer named Misty Wall. She was a mechanical engineer at ASML, finished a major project, wanted something new, and got interested in quantum without any background in quantum information. She started closing tickets in Mitiq, an open source error mitigation framework, and two or three years later she was lead author on research papers on quantum error mitigation. It does not happen for everyone, but it is a real path.

Unitary Foundation runs a Discord year round, and the popular packages each have a channel, so you can see what is out there, including open source simulation packages. Building a simulator to run a quantum circuit classically is a very good way to understand how circuits and algorithms are actually represented. For a developer this is the best way in, because the alternative is a physics PhD and a job in a lab, which is a long and demanding path. The good news is that other people’s hard work now lets us reach real quantum computers over the Internet with tools we already know.

Follow-up questions over email

After the session, Sebastian answered two questions we did not reach live.

Of the domains people cite, chemistry, materials, optimization, cryptography, machine learning, which is closest to real value and which is furthest away?

Probably materials, with small molecule chemistry, which is sometimes treated as a type of material science, a close second. Optimization, cryptography, and machine learning all need thousands of logical qubits, so those are further off.

You have been on the commercial side at AWS and IBM. Where does the marketing most often outrun the engineering?

Since at least the early Q2B conferences run by QC Ware, there has been a recurring chorus demanding to know quantum computing’s use cases and how it will be useful for enterprises. Marketing can be tempted to take speculative ideas and present them as certainties, stretching a scientist’s speculation into something definitive. The other constant question is when, so timelines are another place marketing takes liberties.

Sebastian Hassinger writes and hosts the New Quantum Era podcast, and his book The New Quantum Era explains the field for readers without a physics background. Find him on LinkedIn.

This piece is adapted from our live interview session with Sebastian Hassinger, author of The New Quantum Era and former quantum lead at AWS and IBM, which anchors the Deep Engineering newsletter issue on quantum progress. The words below are his.

So getting quantum ready means different things depending on who is asking, but for a working software developer it mostly means building some intuition for what a quantum information approach to a problem actually looks like. And the best way to get there is usually through coding rather than through reading theory first.

The tools are already familiar

Most of the programming that goes on in quantum computing happens in Python. Qiskit, the IBM SDK, is a Python SDK. Amazon Braket, which I worked on during my years at AWS, is also Python. There are a number of others in Python or in other well understood languages. The way you construct a task and send it to a quantum computer uses tools we all already know, and the genuinely difficult part is the logic of the circuit itself, not the surrounding machinery. So there are plenty of easy entry points for making yourself familiar with the field.

Start by closing tickets

My favorite route for a developer runs through the Unitary Foundation, where I am a fellow. Every year they run an event called Unitary Hack, a global couple of weeks that has been remote and is starting to become an in person thing as well. Maintainers of open source quantum software repositories tag issues in their repos, and developers close those issues out and claim bounties for the ones they resolve. The issues are often housekeeping, administrative work, security tasks, things that are very familiar to classical developers and have to do with the capability, resiliency, and maintainability of the package. But as a side effect you get to see the insides of how that quantum software package actually works.

What that path can turn into

My favorite example of how this can turn out involves an engineer named Misty Wahl. She worked at ASML as a mechanical engineer, had just finished shipping a major project there, and was looking for something to get involved in. She was interested in quantum computing without any particular background in quantum information. She got involved in Unitary Hack and started closing tickets in Mitiq, an open source error mitigation framework. Error mitigation is almost a building block toward error correction, where you do not have full correction but you are finding ways to tease better signal out of the noise, so Mitiq lets you experiment with different approaches to that. Two or three years later she was lead author on research papers on quantum error mitigation. Her experience turned her into a quantum information researcher.

That does not happen for everybody, but it is certainly a viable path, and you do not have to wait for the annual event to start. The Unitary Foundation runs a Discord server year round, and all of the popular packages with a lot of users have a channel on it. You can see what packages are out there, including open source simulation packages, which are particularly good ways to develop an understanding of how quantum circuits get represented and what those algorithms look like. You are building a simulator to classically run a quantum circuit, so you end up learning the structure from the inside.

Why this beats the academic route

For a developer this really is the best way in, because the alternative is getting a PhD in physics and a job in a lab or at IBM or AWS or Google, and that is a very long path that is not for everyone. The field remains extremely challenging to enter as a practitioner. But other people’s hard work has made it far easier for the rest of us to reach real quantum computers over the Internet using tools we are already comfortable with.

Two things to do this week

If you want to act on this, pick one open source quantum package and read its issue tracker this week, looking specifically for the maintenance work rather than the physics. And join the Discord for whichever package you choose, because seeing what its actual users are stuck on will teach you more about where the field stands than another explainer will.

The full conversation goes further, covering why qubit counts mislead and where quantum pays off first, over in issue 58 of Deep Engineering.

Building an agent is easy. Building one that survives production takes architecture, verification, and stopping conditions that actually stop. Four hands-on hours with Rick Hightower on loop engineering, evaluation harnesses, context efficiency, and safe MCP tool integration, live on 29 August.

Deep Engineering readers save 40% with code - DEEPENG40.

Save your seat →

✍️ From the editor’s desk,

Welcome to the 57th issue of Deep Engineering!

On 28 July, GitHub’s changelog recorded a change that sounds administrative and is not. Individual Copilot app activity is now attributed to users in the enterprise-user and organization-user reports, with a per-user section reporting session counts, request counts, prompt counts, and a token usage breakdown covering output tokens, prompt tokens, and average tokens per request. Per-engineer token attribution is now a REST call against an API most enterprise administrators already query.

Nothing about that is objectionable on its own, because a team that cannot see its consumption cannot govern it. The risk lives in what gets built next, since the shortest path from per-user token data to a management artifact is a ranking. Amazon and Meta both stood up internal leaderboards ranking engineers by tokens burned, then dismantled them within a quarter once engineers began padding their usage, which we covered in our special issue on token maxxing. The visibility has now improved considerably, and the temptation has improved with it.

Rory Preddy, AI Advocate in Developer Relations at Microsoft and GitHub, worked five years in cloud advocacy watching the same question mature from how much will we spend into how much can we save, and he argues the answer was never individual behaviour. Today’s issue comes out of our live session, where his position was that cost discipline belongs in the artifacts a team inherits rather than in anything resembling a performance conversation.

You can also read this practical deep dive on the mechanics of token efficiency by Preddy.

Let’s get started.

Agent-written TLA+

An agent wrote our TLA+ spec. The model checker explored 14.3M states and caught a real race.

Read the write-up

Expert Insight

Cost discipline belongs in the agent profile, not the performance review

by with Rory Preddy

The logic behind a token leaderboard feels right from a distance, because agents produce measurable volume, volume varies between engineers, and variance invites comparison. Rory Preddy, AI Advocate in Developer Relations at Microsoft and GitHub, has observed this sequence play out once before in cloud, and his position is simply that “Rather than spend more, spend less.”

Coming from somebody whose job is persuading developers to adopt AI, that is a more interesting claim than it would be from a finance team, and the reasoning underneath it is what makes it useful. Preddy is not arguing for austerity or for slowing teams down. He is arguing that cost discipline is a property of the artifacts a team inherits rather than a property of the people using them, which relocates the problem from individual behavior to engineering infrastructure. For any leader currently deciding what belongs on an AI dashboard, that distinction determines whether the dashboard helps or quietly makes things worse.

Spending less is an engineering target, not a budget instruction

Preddy formed this view during five years as a cloud advocate, in rooms with the CTOs of banks whose balance sheets run larger than some national economies. He describes asking those CTOs what worried them most, and the answer changed shape over time in a way worth attending to. Early on they wanted to know how much they were going to spend, because a four-year hardware forecast let them buy servers and depreciate the cost against a schedule they controlled. Later the question inverted entirely, and what they wanted to know was how much they could save.

That inversion is the part that transfers to tokens, and it transfers with one difference that makes it more urgent rather than less. A server that gets over-provisioned wastes capital on a predictable schedule, and the waste appears on a depreciation line somebody reviews. An agent loop that resends its context on every iteration wastes money continuously, invisibly, and at a rate nobody set, because the cost arrives as consumption rather than as a purchase. Cloud waste taught organizations to rightsize compute against a bill they could at least forecast. Token waste removes the forecast.

Preddy’s instruction to teams starting now runs from the floor upward, so his advice is to look for “what’s the minimum and then cost save, not maximum.” The engineering content of that instruction matters more than its thrift. A team that starts at the largest model with the longest context has no baseline against which to measure any later improvement, so it cannot tell whether a change helped. A team that starts at the smallest model completing the work has a floor, a known cost, and a reason to escalate when the work genuinely demands it. The cheap path becomes the measured default rather than the fallback nobody trusts.

Discipline is arriving as a function, not a phase

That parallel with cloud is not incidental to Preddy’s argument, and he expects the rest of the pattern to follow too. “There’s a whole industry out there just for token efficiency,” he says, pointing to engineers who wrote good deployment scripts, became cloud engineers, and then watched cost optimization separate into its own discipline with its own titles and its own tooling. He puts it more flatly when he calls it “This is our industry.”

The institutional scaffolding has arrived faster this time than it did for cloud. The FinOps Foundation’s State of FinOps 2026 reports that 98 percent of practitioners now manage AI spend, up from 31 percent two years ago, and it ranks FinOps for AI as the top forward-looking priority with AI cost management as the single skillset teams most need to develop. In June the Linux Foundation announced its intent to launch the Tokenomics Foundation, working with the FinOps Foundation on open industry standards, benchmarks, and best practices for the economics of AI infrastructure. Preddy reads that as the point where dashboards, standards, and protection mechanisms stop being individual good habits and become things an organization can be held to.

For engineering leaders the timing carries a specific implication. A function forming now will settle its conventions within a few quarters, and the organizations contributing to those conventions will be the ones already treating agent cost as an engineering concern rather than a procurement one. Teams still running enablement sessions about prompt length will inherit whatever standards other people write.

Cost advice without a budget produces a cheaper unbounded system

Before any of that scaffolding helps, a team has to answer a question most cost guidance skips. The sharpest moment in our conversation came when Preddy declined the premise of a question about authorization boundaries and asked something more basic in its place. “How much money do I have?” he said, and then named the assumption he keeps encountering, because “a lot of what you’re asking me assumes that you have infinite amount of tokens.”

He is identifying a real gap in how the industry discusses this. Almost all published cost guidance optimizes mechanics, so prompts get shorter, retrieval gets tuned, caches get warmed, and routing gets layered in. Very little of it starts by asking what the workload is permitted to cost. Optimization applied to an unbounded system produces a cheaper unbounded system, which still has no ceiling and still fails in the same direction. The budget has to exist before the tuning means anything, because the budget is what converts an optimization into a decision.

What makes this harder than it looks is that the system cannot supply the number for you. A study published through Microsoft Research this spring, How Do AI Agents Spend Your Money?, presents the first systematic analysis of token consumption in agentic coding tasks, and it found that runs on the same task can differ by up to 30 times in total tokens, that input rather than output tokens drive the overall cost, and that frontier models fail to predict their own token usage and systematically underestimate what a task will cost. A workload whose executor cannot estimate it will not be estimated reliably by a spreadsheet either. That finding closes off the approach most organizations reach for first, which is forecasting from measured averages, and it pushes the answer toward setting a limit and enforcing it rather than predicting a figure and hoping.

The consequences of skipping that step have landed at named companies. Forrester’s July analysis of AI cost management cites Uber burning its AI budget in four months, Microsoft ending Claude Code licenses after also burning its yearly AI budget, Tesla limiting AI spending to 200 dollars per week, and Priceline absorbing an unexpected surge in AI development renewal costs. None of those reads as an engineering failure. Each one reads as a missing number.

Foundation is an artifact leaders ship

Setting the number is the first move, and the second is deciding who builds the thing that keeps teams inside it. That is where Preddy’s position turns into a leadership argument rather than a productivity tip. He reaches for the idea of a cybernetic teammate, an agent working alongside a team rather than replacing anyone, and then reframes it as something an engineering leader produces rather than something a vendor sells. The artifact is the agent profile, along with the instructions file and the container definition that travel with it, and his claim about ownership is direct. “Your leaders should go in and create the agent profiles,” he says.

His reasoning turns on availability rather than authority, and it is the more humane version of the argument. Not every engineer works somewhere they can walk to a senior colleague and admit they are burning too many tokens, and not every senior colleague is reachable at the moment the question arrives. Preddy frames the profile as the answer to that absence, describing the message it carries as “I’m not around to help you out, but I’ve created a nice agent profile” with the conventions already encoded. A profile that declares only the tools a task needs, names the model tier, and sets the compaction expectation makes the disciplined path the default path for everybody who inherits it, including the engineer who joined last week and has nobody to ask.

The second-order effect is what leaders should weigh most carefully, because it determines which of two very different programs they end up running. Token waste treated as individual behavior produces enablement sessions, then dashboards, then comparisons, then rankings, and each step follows plausibly from the last. Token waste treated as a missing artifact produces a file one person writes once and every engineer benefits from without thinking about it. The first approach asks engineers to carry knowledge that changes every quarter. The second encodes that knowledge where it survives staff turnover and stops depending on who attended which session.

One detail in Preddy’s own workflow reinforces the point. He builds these files using the cheapest model available with reasoning switched off, which means the foundation costs almost nothing to produce. The barrier to doing this properly is not budget or tooling. It is that nobody has been made accountable for it.

A spend cap belongs with the permissions

The foundation shapes the default, and a cap is what holds when the default gets overridden. Engineering organizations already accept that an agent’s authority needs governing, so they scope which systems it may read, which it may write to, and which credentials it holds. Preddy’s contribution is to place money in that same category rather than treating it as a separate financial concern reviewed on a different cycle by different people. A per-session budget that warns at a threshold and stops at a limit behaves exactly like any other permission, because it constrains what the agent may do without a human present.

The practical consequence is a change in when the decision gets made and by whom. Treated as finance, a spend cap arrives quarterly, applies at the account level, and reaches the engineer as a rate limit they cannot explain. Treated as authorization, it arrives at the same review where the agent’s other permissions get set, applies at the session level, and reaches the engineer as a known boundary they helped define. Preddy demonstrated the enforced version during our session, where a run halted after crossing its allocated credits and offered the operator a choice to add credits, raise the ceiling, or remove it entirely.

That halt is the whole value, and it is worth being precise about why. An agent meeting a wall has produced a decision point, with a human present, a specific task in view, and the cost of continuing visible on screen. An agent with no wall has produced an invoice, arriving weeks later, aggregated across teams, with nobody able to reconstruct which run caused it. The same money moves in both cases. Only one of them leaves an organization able to govern the next run.

Token counts measure activity rather than relief

A budget and a cap tell you when to stop. Neither tells you whether the spending accomplished anything, which is the question a dashboard is supposed to answer. A contributor to our earlier issue on token maxxing offered a formulation Preddy endorsed when we put it to him, that token usage behaves like CPU utilization, so the honest unit is the cost of relieving a bottleneck rather than the volume consumed. He agreed, then complicated it usefully by pointing out that the model chosen to relieve the bottleneck changes the answer, which means the metric has to account for the routing decision rather than treating all tokens as equivalent.

The research supports him on the underlying point. The same Microsoft Research analysis found that higher token usage does not translate into higher accuracy, and that accuracy often peaks at intermediate cost before saturating as spending rises. A number that rises while the outcome does not is the definition of a metric worth distrusting.

Preddy’s own experience supplies the rest, and it is more persuasive because it is self-implicating. He assigned a merge conflict to the smallest model available, and it consumed 51 million tokens without resolving anything, a run we cover in detail in his companion deep dive. Read as a token count, that looks like heavy engagement from a productive engineer. Read as bottleneck relieved, it produced nothing whatsoever. Any metric unable to distinguish between those two readings will reward the wrong behavior from the day somebody starts reporting it upward, and it will do so most strongly for the engineers who least understand what they are doing.

This is also why the leaderboard fails on its own terms rather than only on cultural ones. A ranking by tokens consumed can be improved by consuming more tokens, which requires no additional value and no additional skill. A measure built on work closed, so a merge conflict resolved, an accessibility fix shipped, a suite returned to green, can only be improved by closing more work. The second measure is harder to define and harder to collect, which is exactly why organizations reach for the first. Preddy’s argument is that the difficulty is the job, because the easy number and the useful number have never been further apart than they are now.

The principle holding it together

What makes Preddy’s approach more than a list of preferences is the single idea running through every part of it, which is that cost control belongs in the system rather than in the person operating it. Every choice he describes follows from that.

The foundation is a file rather than a training session, because a file survives turnover and a session does not. The spend cap is a permission rather than a budget line, because a permission gets enforced at the moment of use and a budget line gets reviewed after the fact. The metric is work closed rather than tokens consumed, because an outcome cannot be inflated by burning more of the input. The smallest viable model is the default rather than the exception, because a default is what happens when nobody is paying attention, and most of the cost accrues exactly then.

That principle also explains what Preddy leaves out. He never asks engineers to be more careful, and he never suggests awareness alone fixes anything, even while arguing for visibility. His own session history showed input tokens outweighing output roughly 100 to 1, and one session climbing from 49,000 to 174,000 input tokens across 66 turns with no compaction at all. He describes himself falling into bad practices like everybody else. A discipline depending on the operator remembering will fail for the person who wrote the discipline, which is the strongest available argument for encoding it somewhere else.

What this asks of a leader

The sequence that follows is short enough to start this week. Write the agent profile and the instructions file yourself, or make one person accountable for them, and treat them as shared infrastructure reviewed like any other. Set a per-session budget in the same review where the agent’s permissions get set, rather than in a separate conversation with a different owner. Report the work closed rather than the tokens consumed, and decline to build the leaderboard even though the API now makes one trivial.

The deeper point for any leader standing up AI measurement is that visibility and ranking are not the same thing, and the tooling arriving now makes it very easy to confuse them. Per-user token data has legitimate uses, including chargeback, capacity planning, and helping a manager notice that a team is consuming heavily without much to show for it. It becomes destructive at the moment it gets used to compare engineers against each other, because that is the point at which the number starts changing the behavior it was meant to observe.

Preddy’s closing thought aimed at the people doing the work rather than the people measuring it, and it belongs here because burnout and cost discipline turn out to share a cause. His instruction was not to “end each day exhausted, not from the work itself, but from managing of the work.” An engineer who inherits a foundation somebody else built gives the day to the problem. An engineer who inherits nothing gives it to managing agents, and pays for the privilege in tokens. “Be kind to yourself,” he said, which for a leader translates into something concrete. Build the foundation, and there will be nothing worth ranking.

In case you missed

🛠️ Tool of the Week

Langfuse is an open source AI engineering platform that traces agent runs and attributes cost and token usage per trace. A spend cap needs a number behind it, and a per-trace cost is that number.

• Traces multi-step agent runs so cost attaches to a workflow rather than to a raw API call

• Tracks token usage and cost per trace, per model, and per user for chargeback and per-team budgets

• Integrates with OpenTelemetry, LangChain, LiteLLM, and the OpenAI SDK, so routing across providers stays visible in one place

• Self-hostable, which keeps prompt and completion data inside your own boundary

Learn more about Langfuse

📎 Tech Briefs

• Model Context Protocol ships its 2026-07-28 specification - Sessions and initialization are removed for stateless requests, with authorization and versioned extensions now explicit.

• npm adds publish-time malware scanning and dual-use metadata - New packages now face pre-install scanning, metadata disclosure, and stricter publishing controls for security-sensitive capabilities.

• GitHub Actions holds potentially malicious workflows for approval - Suspicious runs pause before execution, moving CI supply-chain defense ahead of the workflow rather than after.

• Kubernetes v1.37 reaches code and test freeze - Code changes now require exceptions, shifting attention to release-blocking tests and regressions before August’s release.

• Accenture launches a tokenomics practice for enterprise AI spend - Token spend gets tied to business outcomes, making AI cost governance a consulting line item.

That’s all for today. Thank you for reading this issue of Deep Engineering.

We’ll be back next week with more expert-led content.

Keep building,

Saqib Jan

Editor-in-Chief, Deep Engineering

If your company wants to reach senior developers, software engineers, and technical decision-makers, speak to us about partnering with Deep Engineering.

By Rory Preddy, AI Advocate, Developer Relations at Microsoft and GitHub. Creator of LangChain4j for Beginners. | Edited by

I have been in IT for 27 years, and 23 of those as a programmer. I started as a Java developer. I moved to Microsoft seven years and eight months ago, and I have been a developer advocate for five and a half of those, first in cloud advocacy and now in AI.

For most of that time my job was to tell people to go and build. Lately I have started saying something else first. Take a step back. Not because the excitement is wrong, but because I keep meeting developers who ran out of tokens and cannot tell me where they went. I ask what happened and I hear the same answer. I used the top model, I sent it away, and it ran out. I ask whether they needed the best model for that job and they say they do not know.

That is not a model problem. That is a foundation problem, and it is fixable in an afternoon.

So let’s dig deeper into what tokens actually cost you, the two mechanisms that cut the bill the most, the patterns that keep the expensive model out of cheap work, and the setup you do before any of it.

This deep dive is adapted from Rory Preddy’s Deep Engineering Live session on token efficiency, edited from the session transcript for length and clarity. Slides from the talk are here, and the full recording is here.

Token Efficiency Foundry Instantmodels

2.37MB ∙ PDF file

Download

Download

Download

Four levers, and one of them gets ignored

You have four things to pull, and they move independently.

Model choice sets the unit price on every token in the exchange. Prompt structure sets how many input tokens you send and whether any of them qualify for a discount. Cache-key stability decides whether a repeated block earns the cached rate or gets billed again at full price. And output limits cap the completion, which is the one people skip.

Output is where I see the least attention and some of the highest prices. On the GPT-5.6 family the flagship Sol tier runs 5 dollars per million input tokens against 30 dollars per million output. Luna, the small one, is 1 dollar and 6 dollars. Look at the gap between those two numbers on the same row before you tune anything else. Watch your outputs, not just your inputs.

One more thing before the mechanisms. You do not have to guess at any of these rates. Azure publishes live pricing at prices.azure.com/api/retail/prices and you can query it. It looks confusing the first time you open it, because a single meter name folds the model family, the tier, the processing mode and the billing unit into one string, and there is a region attached to all of it. So let’s all breathe. Two things in there matter. The same model bills differently in different regions, and batch processing is its own meter rather than a discount on the standard rate.

It is not complicated. It is just not written down anywhere you were looking.

Caching, and why you only pay for the tail

Ask a model to tell you a joke. Now ask it again, exactly the same way. You are not going to be charged the same for the second one, or if you are, it will be a very small charge.

Now scale that up. Say I am an insurance company and I need to send millions of customers their latest statement. Every one of those requests carries the same instructions, the same reference material, the same everything, and then right at the bottom there is a name and a dollar amount that changes. That big identical block at the front is the stable prefix. The little bit at the end that changes is the tail.

You do not pay full price for the prefix. You pay for the tail.

The first call is a miss. The model stores the prefix and bills you the full input at standard rate. Every call after that, sent with the same prefix and the same cache key, is a hit, and that prefix gets billed as cheaper cached input. In my demo the prefix carried around 120 identical reference sections behind a run-scoped key, and warming the cache before the measured calls saved 99 percent on the repeated path.

Here is the part that catches people. The saving depends on those bytes being identical. Put a timestamp near the top of your prompt, or a request ID, or a freshly serialised list of tools, and you have moved the boundary and lost the discount on every single call. It will look completely harmless in review. So think ahead with the prompt structure. Decide what is stable and put it first, on purpose, before you need it.

Compaction, so the context stops growing

Caching handles repetition between calls. Compaction handles what piles up inside one conversation.

Every time you talk to an agent it remembers the conversation, and it sends that history again on the next turn. So the tokens you are billed per turn climb, and climb, and climb, even when the actual work per turn has not changed at all. Model a twelve-turn session and you are sending around 1,920 context tokens per turn by the end. Compact it along the way and it resets near 795 and never goes above that band. That projection is illustrative, but the summary reduction underneath it is measured, 62 percent, from 795 tokens down to 302.

What makes a compaction prompt work is being specific about what survives. Mine asks for at most six sentences, no bullets, no nested lists. It keeps only what the next turn needs, so the goal, the key facts, the files to update, the validation and deploy commands, the blockers and any privacy constraints. It throws out repetition, resolved dead ends, greetings, transient logs, and exact values that do not matter any more, and it turns quota findings into sanitised evidence instead of repeating every number. On a working-notes example that took a prompt from 409 tokens down to 233.

I also tried the other direction, squeezing the output instead of the context. I have a prompt that tells the model to answer like a caveman. Why use many token when few do trick. Drop the filler words and the articles, keep every technical fact, every command, every file path and every error string exact. Brain still big, mouth small.

It works, and the output count came down against a roughly 600-token baseline. But I will be honest with you about the trade. The caveman answer lost some of the understanding the fuller answer had. It is not as detailed. If your reader has to come back and ask a follow-up question, you have paid for that saving twice. A hard cap on output length is blunter but more predictable, because it truncates rather than condenses, so use it where the shape of the answer is already known.

Set up the project before you write anything

This is the least exciting part and it decides most of your cost.

Three files. A devcontainer.json, so the project runs on everyone’s machine and not just yours. An instructions file, so the agent has standing context and is not working blind. And an agent profile, which is the one people leave out.

The agent profile declares which tools the agent can reach for, and mine only has the tools it needs to function. That matters more than it sounds, because every tool you register ships its schema into the request on every single call. Leave a pile of MCP tools switched on that this task never touches and you are paying to send that JSON back and forth all day. Drop the ones you are not using.

Then three habits alongside it. Scope your sessions tightly. Keep your prompts precise and short. And if a direct completion will do the job, use a direct completion instead of turning an agent loose on it.

I build these three files with the cheapest model I have. Luna, reasoning switched off. It costs me almost nothing to create the baseline, and I would rather learn to do it properly at that price. On a cost-tips run I will say it out loud as I go. I am not using reasoning. I do not need reasoning right here. That is the right tool for the job.

What an agent actually is

An agent has agency. Agency is self determination, which means the agent understands there is a task it has to achieve. You can only give it that task if you understand it yourself.

What an agent is not is a large language model that you tell to go in and do whatever it wants. That is madness.

Here is the version I trust. I have a supervisor agent that transfers 100 dollars from one person to another and converts it to euros. Underneath it there is a withdraw agent, a credit agent and an exchange agent. Each one has a small scope. It runs on GPT-4o, so it costs me cents per month. It runs on Azure with managed identity, so it is locked down. I created the project with the dev container and the agent profile before any of it ran.

That is what I consider an agent. There is a plan, you built it, you locked it down, and it performs one well-contextualised task properly.

Eight patterns that keep the big model out of cheap work

Once the foundation is there, the patterns are about one idea. Do not let the largest model see work it does not need to see. My tiers are gpt-5.6-luna small, gpt-5.6-terra medium and gpt-5.6-sol large, and the numbers below are projections from my own demo harness, not benchmarks.

A router puts a cheap classifier in front and sends the request down one specialist path. 60 to 80 percent. Triage is similar but the gate is deterministic and costs zero tokens, and it only escalates when the complexity earns it. 50 to 70 percent. Context compression has Luna condense a long transcript so Sol never sees the whole thing. 30 to 60 percent. Retrieval pulls the chunks that matter instead of the whole corpus. 40 to 60 percent.

Tool use moves exact computation out of probabilistic generation and into code, and it goes hand in hand with keeping that tool list short. 30 to 50 percent. Step-back planning resolves the frame before the expensive execution starts, and it only pays when it prevents retries and rework. 20 to 40 percent. Caching covers the repeated path, 50 to 90 percent on a hit. And batching I will not oversell, because a parallel mapper lowers your wall-clock time and saves you no tokens at all.

Retrieval deserves more than one line, because everyone reaches for it and plenty of people misconfigure it. The query becomes an embedding, the embedding drives a vector search, the search returns chunks, and the model answers from those chunks alone. So your chunking and your embedding model decide your answer quality and your token count at the same time. The failure I see is sending everything back every time, because that is easier than tuning retrieval, and then the customer asks why it is slow and you ask why it is expensive. My retrieval demo projected 67 percent against sending the whole document. All you need is the dollar amount, not the entire insurance quote.

And where I got it wrong

I burnt 51 million tokens recently, and I used the wrong model for a specific purpose.

I was resolving a merge conflict, and I gave it to Luna. Luna could not solve the problem. That on its own would have been a cheap mistake. What made it expensive is that I had not set a loop to say, after a while, stop. So it kept going.

Two lessons, and they pull against each other slightly. Match the model to the bottleneck rather than always reaching down, because a cheap model that cannot finish costs more than an expensive model that can. And put an exit condition on every iterative agent, every time.

I have a refinement loop that writes a story and then reviews its own work. It caps at five iterations and accepts a score around 80 percent. It is not going to be perfect. Sometimes you really need to say, I do not need it perfect. Caching and compaction are both good, and you still want a hard stop when it hits the score you asked for.

Build an agent that watches the bill

You can point agents at your own costs, which is my favourite version of this.

I have a token cost runner. Its job is to execute real flows against a deployed Foundry model, read the actual token usage, and compare it against live pricing. So the number comes out of a run rather than out of arithmetic on a rate card. I ask it to find me the lowest-cost path for a prompt with a cached prefix and a changing tail, and it tells me.

Build agents to help you lower your cost for agents. Think of it as a pyramid. What you want to achieve sits at the top, and underneath it there is a foundation holding your dev container, your instructions, your agent profiles, your compaction and your compression. Everything above the foundation inherits whatever the foundation does.

Two commands, if you want the short version

If patterns are more than you want right now, there are two commands.

The first reads your session history and tells you where the money went. Mine was not flattering. Across sixty days my input tokens outweighed my output roughly 100 to 1. Opus 4.8 took 45.8 million input tokens at an average of 87,000 per event. One single session accounted for 14.4 million on its own. And one session climbed from 49,000 to 174,000 input tokens over 66 turns with no manual compaction at all, so auto-compaction only fired at turn 67, right at the end. Every turn past about turn 30 was resending well over 120,000 tokens on a premium model. I was the one not compacting long sessions.

The second sets a budget per session, warns you at a threshold and stops you at the limit. Mine stops after 34.7 of 30 credits and asks whether I want to add credits, raise the limit or remove it.

I also keep a canvas built from that same session data as a daily token pulse, and I refresh it across seven and thirty day windows. None of this optimises anything by itself. It is just awareness. But a number nobody looks at twice will never change how anyone works, and most of the waste I see is not a decision anyone made. It is a thing nobody checked.

Pick the harness you already pay for

People ask me which harness balances capability against cost. My honest answer is that what you are trying to achieve settles that before cost does.

Then I will tell you the best harness to save costs is GitHub Copilot, and you should hear that knowing I work on the Copilot side. My reasoning stands on its own though. If you have a Copilot licence you already have the spend, and Copilot is an agent, not just an editor feature. There is a Copilot SDK and there is the Microsoft Agent Framework, and you route from there. Send the task to Copilot, or to Claude Code, or to a local model, based on the work. It is the same routing decision as picking a model, one level up.

Take a step back

There is a whole industry forming around this. I watched it happen with cloud, where engineers who wrote good scripts became cloud engineers, and then cost optimisation became its own discipline. I used to sit with CTOs of banks who told me their biggest worry was how much they were going to spend. Then it changed, and the question became how much they were going to save. The same thing is happening now with tokens, and the Linux Foundation’s Tokenomics Foundation is going to matter here, because standards and dashboards and protection mechanisms are what turn this from a habit into a practice.

So do not go berserk. Create the foundation. Try Luna, switch reasoning off, and see how far it gets you. If it does not match your needs, add a router and a design pattern. Start small and build up. Rather than spend more, spend less.

And one last thing, because I have another whole talk about this. Do not end each day exhausted, not from the work itself, but from managing of the work. Let the adrenaline flow, do the vibe coding, do it in the correct way. Know what you are doing. Be kind to yourself.

Now go and look at your own session history.

Session notes:

Rory’s demos run on LangChain4j’s agentic modules, which matters for teams on the JVM because most token-efficiency tooling ships Python first. His pattern code is at github.com/roryp/token-design-patterns, the agentic pattern showcase behind the banking demo at aka.ms/agentpatterns, the live cost demo at aka.ms/costs, and his LangChain4j course at aka.ms/LangChain4j-for-Beginners.

This deep dive is adapted from Rory’s Deep Engineering Live session on The Future of Software Development. Here are the slides from the talk, and the full recording.

Token Efficiency Foundry Instantmodels

2.37MB ∙ PDF file

Download

Download

Connect with Rory Preddy on LinkedIn: https://za.linkedin.com/in/rorypreddy

For most of the discipline’s history the binding constraint during an incident was human attention, because somebody had to correlate logs across services, hold a working model of the system, and form a hypothesis fast enough to matter. Interview loops were built to find that person and they were right to. Agents now take a growing share of that first pass, and the loop has not moved with them. A coding exercise, a debugging scenario, and a systems round all measure how fast a candidate converges on a cause. Those are the skills losing value fastest, while the ones that decide whether the hire works go untested.

Reid Savage, Senior Engineering Manager for the Site Reliability Engineering team at Honeycomb, shares how he has repeatedly seen that gap widen across the hires he has made and overseen.

“Standard SRE hiring loops are far too skills-based,” he says. And his own team acted on that conclusion, removing coding exercises from the loop at Honeycomb’s current stage.

Motivation decides the hire, not the skills matrix

The failure pattern Savage describes does not look like a skills gap, which is what makes it so easy for a conventional loop to miss. “I’ve had excellent hires that have never used our cloud of choice, and some who didn’t work out that had nearly everything,” he says. The candidates who arrived with the full stack on their résumé were not reliably the ones who lasted, and the ones missing the specific platform experience frequently were. That inversion is not a story about credentials being worthless, and it is a story about credentials answering a question that turns out not to be decisive.

What Savage found underneath the pattern was a mismatch of a different kind. The problem in the hires that did not work out was “usually a mismatch between true motivation (which requires significant self-insight on their part) and what the team needs,” he explains. Motivation in this framing is not enthusiasm for the role or eagerness in the interview, both of which candidates supply readily and neither of which predicts much. It is the specific thing that makes the work satisfying to that person, which requires enough self-knowledge on their part to name it accurately. Savage anchors the distinction to a line he paraphrases from First, Break All The Rules, about hiring the accountant who cannot sleep until the books are balanced rather than the one who knows Excel.

The scale of Honeycomb’s problem space is part of why this holds. “At Honeycomb’s size, the skills required for each problem we have wouldn’t fit in 5 job descriptions,” Savage says, which means any specific skill the loop screens for covers a small fraction of what the hire will actually encounter. A loop optimized for skills coverage is therefore optimizing a variable that cannot be made to matter, because the surface area defeats it. The organizations most exposed to this are the ones whose reliability function touches many systems rather than one, which describes most companies past their first platform consolidation.

Honeycomb’s replacement for the coding exercise is the actionable part. The technical screen is now a pull request review, chosen because, in Savage’s words, “it covers the important things: reading between the lines, social skills, eagerness to help, mindset, and integrity.” A PR review works as a screen because it presents the candidate with someone else’s decisions rather than a blank editor, and reliability work consists largely of forming a fast, correct read on choices other people made under constraints that are no longer visible. Leaders who want to run this should use a real pull request from their own history, ideally one where the right call was contested, and score the review on what the candidate noticed and chose to raise rather than on whether they found a seeded bug.

Savage’s position is that the technical bar clears more reliably in the presence of those qualities than the reverse, because “if you have those, and they display the level of technical aptitude you need, they’ll be able to catch up.”

Profile of the hire changed inside a year

Fixing the screen addresses the process. The harder change is to the role itself, and it has moved faster than most loops have been revised. “The profile of a great SRE has fundamentally changed over the last year,” says Anish Agarwal, co-founder and CEO of Traversal AI, which builds AI systems for production incident investigation. His account of what companies previously optimized for matches what most loops still measure, which is the engineer who could reason through dashboards, logs, and metrics under pressure and arrive at a root cause. That was the correct thing to optimize while humans carried every investigation from alert to resolution.

Agarwal’s argument is that agentic systems now take on the most time-consuming parts of that work, and the consequence for hiring follows directly. “Raw troubleshooting ability is becoming less of the differentiator than it was even a year ago,” he says. A capability stops being a differentiator once it stops being scarce, and a loop weighted toward a non-scarce capability produces candidates who cluster indistinguishably at the top. Leaders tend to notice this as a symptom well before they diagnose it, in loops where several finalists all pass the debugging round cleanly and the final decision comes down to feel.

What replaces it, in Agarwal’s view, is the part of the role that always belonged there and rarely got attention. “What has become far more important is the ability to engineer resilient systems,” he says, being precise about why that capability went underexercised for so long. Most teams never found time for it because troubleshooting consumed their days, so resilience work stayed aspirational rather than scheduled. Two changes now arrive together, with AI removing much of the investigative burden while production systems grow more complex through new services, new dependencies, and new failure modes each quarter. His conclusion is that resilience engineering “is no longer the work SREs wish they had time for. It’s increasingly the work the role demands.”

That reframes what a strong candidate has to demonstrate in an interview. Agarwal describes the highest-leverage SREs as the ones “thinking about what the system will need a year from now, which SLAs actually matter, and which technologies best meet those SLAs given the tradeoffs between cost, reliability, and performance.” A loop that wants to surface this has to ask forward-looking questions rather than diagnostic ones, and the practical version is to hand the candidate a real architecture with real growth assumptions and ask what breaks first at ten times the load. The answer reveals whether the candidate reasons about failure modes that do not exist yet, which no incident replay exercise can establish. Agarwal adds one further capability he would weight more heavily, describing it as “AI fluency. Not just expertise, but genuine curiosity,” and locating its value in engineers who use agents to compress investigations so that the time freed goes into resilience work.

Judgment is the scarce input, and it leaves when people do

Hiring toward resilience assumes the judgment to do that work can be found and kept, and that assumption meets an organizational problem that no interview round surfaces. George Hantzaras, Director of Engineering, Core Platforms at MongoDB, speaking at the AI-Powered Platform Engineering workshop we hosted some time back, argued that the problems facing platform and operations teams have stopped being problems of tooling or frameworks and have become problems of people, team dynamics, and how humans interact with the systems they built. His first category is the knowledge gap, where the answers engineers need lie buried across dozens of systems, dated documentation, and sprawling wikis. He described his own developers as having become archaeologists, running scavenger hunts through internal documentation to recover a single answer.

The part of that gap most relevant to hiring is the part nobody wrote down. Hantzaras underscored tribal knowledge as the real problem, meaning the unwritten rules and undocumented workarounds that exist only in the minds of the few people who have been at a company long enough to accumulate them. He described one team where new engineers took close to six months to reach full productivity, with the primary cause being the difficulty of navigating internal systems and discovering rules that appeared on no wiki. That knowledge carries a second liability beyond slow onboarding, because it leaves the organization when the people holding it leave, and nothing in a standard hiring process accounts for either effect.

Two further problems he named sharpen what the reliability hire is actually for. Golden paths handle roughly the first eighty percent of cases well and then become what he calls golden cages for the remaining twenty percent, which tend to be the most innovative work and the cases where a developer needs something the template will not express. He also described a platform lead whose team was putting more time into maintaining YAML templates for their portal than into building new platform capability, an overhead he calls the tooling tax. Both problems consume senior judgment on work that produces nothing durable, which is exactly the capacity a resilience-focused hire is supposed to create.

Hantzaras reaches the case for automating the first pass from the opposite direction, arriving through the operations burden rather than through the hiring loop. He pointed out that the industry has fixated on mean time to recovery while, in his reading, the largest component is mean time to identify, and that teams are drowning in logs, metrics, traces, and alerts without a good way to act on any of it. The resulting alert fatigue leads engineers to tune out noise until a critical alert has a high probability of being lost. His read on responsibility for that is unambiguous, because it is almost never a case of an engineer not working and almost always a case of a system not working as it should, and his conclusion is that filtering data at that volume “shouldn’t be a human job. This is a machine job.”

For leaders hiring into this, the implication is that the scarce input is accumulated judgment rather than throughput, and that judgment currently has no home outside individual heads. The practical response runs on two tracks. Weight the loop toward candidates who can reconstruct the reasoning behind decisions they did not make, since that is the skill tribal knowledge recovery actually requires, and a PR review from an unfamiliar codebase tests it directly. Then treat the documentation of undocumented judgment as an explicit deliverable in the first six months rather than as good citizenship, because the alternative is rehiring the same knowledge every time someone resigns.

Foundations come before fluency

Hiring for AI fluency assumes the environment can support it, and that assumption fails more often than the hiring conversation admits. Ajay Chankramath, CTO of Platform Engineering Community, a member organization for DevOps and cloud native practitioners, led the platform engineering workshop sessions for Deep Engineering recently, where he described platform maturity as a progression through four states, moving from reactive to responsive, then to predictive, and finally to autonomous. His caution for teams reaching for the later states is that observability, runbooks, and service level objectives have to be in place before AI gets layered on top, because introducing AI into an environment without those foundations amplifies the existing chaos rather than resolving it. That ordering has a direct hiring consequence, because a candidate hired for AI fluency into a team without instrumented systems will put their first year into building the substrate instead.

Chankramath is equally clear that the tooling does not displace the role. His framing of AI in incident response is that it hands the engineer a capability rather than taking one away, reading the runbook on their behalf and proposing the next step, while the engineer remains the expert who decides whether that step gets taken. He grounds this in an observation most people on call will recognize, which is that nobody reads documentation during a live incident, because when the system is failing and customers are escalating, engineers act on what they already know rather than opening a markdown file. AI closes that gap by surfacing the relevant procedure at the moment of the decision, and the judgment about whether to follow it stays where it was.

He also names a failure mode that should change how leaders think about seniority in these hires. Platform capabilities frequently go unused by the most experienced engineers, who treat self-service tooling as something built for juniors who do not know how to do the work directly, and because those engineers are the ones whose opinions carry weight internally, their dismissal propagates. A reliability hire brought in specifically to raise AI adoption will therefore run into resistance from exactly the people whose endorsement determines whether the change holds. Leaders should test for this in the interview by asking how the candidate has previously won over a skeptical senior engineer, rather than assuming enthusiasm for the tooling transfers to the team.

The last piece of Chankramath’s argument concerns measurement, and it constrains how any of this gets evaluated. He treats the standard delivery metrics as lagging indicators, useful for confirming what already happened and poor at telling a team where things are heading, which makes running an organization solely on them a mistake. Leaders hiring for resilience need leading indicators to judge whether the hire is working, because a resilience improvement shows up as incidents that never occurred, and no lagging metric records an absence. The workable substitutes are the observable precursors, meaning coverage of critical paths by service level objectives, the share of alerts that map to a documented response, and error budget burn rate read before the budget is exhausted rather than after.

Ambiguity behavior predicts the outcome better than anything else

Across everything Savage has observed, one behavior separates the hires that worked from the hires that did not, and it is not a technical one. “The largest determinant I see is what they do when faced with an ambiguous problem,” he says, and he lays out the range of possible responses as moving away from it, tagging in someone else, asking for help, owning the solution, helping someone else own it, or investigating why the problem was ambiguous in the first place. The question underneath all of those, in his phrasing, is whether the candidate accepts or rejects unclear problems. Reliability work arrives almost exclusively as unclear problems, which is what makes this behavior more predictive than any skill the loop can measure.

The cost of getting it wrong is not usually a visible failure, which is part of why it goes undetected. Savage describes one SRE who was passionate and knowledgeable, and who struggled to bring others into the work or carry projects across the finish line, producing a long trail of initiatives that started without much to show for them. Because SREs touch every part of the business, that pattern compounds in both directions, generating leverage when it works and opportunity cost plus real cloud spend when it does not. His summary of the missing skill is that “SREs need to know when to open, peek inside, rearrange, and put back problems.” The judgment is as much about which problems to close as which to open.

Testing for this requires giving candidates something genuinely underspecified rather than a puzzle with a hidden solution. The practical version is to present a real ambiguous situation from the team’s history, withhold the resolution, and pay attention to whether the candidate starts solving or starts questioning the framing. Both responses can be correct, and what matters is that the candidate demonstrates a deliberate choice between them rather than defaulting.

So define the role before you post it

None of the above helps if the role itself was never specified, which Savage treats as the prior question. “You need to know what you’re hiring the SRE for,” he says, and the diagnostic he runs through is a series of concrete organizational facts rather than an abstract job description. Those facts are whether reliability genuinely matters to the business and to what degree, how many nines the company actually needs, whether anyone currently has responsibility for predicting what will fail a year out, whether developers are resisting on-call, and whether the team is drowning in toil. Each of those describes a different job, and an SRE can address any of them, though not all of them at once.

The nines question is the one most organizations answer loosely, and it is the one that determines everything downstream. A target expressed as a service level objective with an error budget attached tells a candidate what the job actually involves, because it establishes how much unreliability the business has already agreed to tolerate and therefore where the engineering effort goes. A posting that claims high availability without naming a target is describing an aspiration rather than a role.

His standard for handling that honestly is where the recommendation lands. It is fine to need some of those things and not others, as long as the job posting states what the role is expected to be today and how that might change. Most mis-hires that look like candidate failures are role definition failures surfacing late, because a candidate optimized for toil reduction will underperform in a role that turns out to require reliability forecasting, and neither party did anything wrong in the interview. The discipline is to write the posting from the specific problem rather than from a template, and to name the parts of the role that are unsettled instead of leaving them out.

The same clarity should extend into the first ninety days, and Savage’s approach here inverts the usual instrument. The senior hires who succeeded did not treat his 30/60/90 as gospel, and read it instead as an aspirational list with goals attached, oriented toward making social connections, becoming familiar with the code, and starting the long climb through the tech stack. He asks new hires early on to disagree with him and to offer at least one piece of constructive feedback, which he uses to open room for them to work on problems he cannot see himself. That is a deliberate test of the same quality the loop was built to find, because an engineer who will surface an uncomfortable observation in week three is demonstrating the disposition toward unclear problems that predicts everything else.

What connects all four positions is that none of them argues for a lower technical bar. The bar holds, and the argument is that the loop currently puts most of its measurement on the part of the bar that agents are absorbing, while the parts that decide whether the hire succeeds go untested. The reliability hire that works out is the person who accepts unclear problems, reasons about failures that have not happened, carries judgment that was never documented, and improves a system nobody else wanted to own. A process built to find that person looks different from the one most companies are still running.

Capacitor records the session behind the work: what agents tried, what teammates rejected, what finally passed, and why it mattered. This gives you vendor-neutrality to move across multiple coding agents, multiplayer - collaboration on coding sessions, faster PR reviews, evals on code & more.

Sign up for free

✍️ From the editor’s desk,

Welcome to the 56th issue of Deep Engineering!

The PostgreSQL Global Development Group shipped the second beta of PostgreSQL 19 on July 16, only a few months after version 18 reached general availability and began landing in production systems. A community project that already tops every major developer survey keeps compounding its lead one disciplined release at a time. Postgres has effectively become the default database, and most teams now reach for it without pausing to ask whether it fits the problem in front of them.

That ease is exactly where the trouble starts, because a decision everyone treats as obvious is a decision nobody examines. Moving off a database once an application leans on it can take years and burn through real budget, and the forces that make the choice consequential have only sharpened this year. Relicensing moves keep redefining what the words open source actually protect, and by Google’s own threat-intelligence reporting attackers now weaponize many newly disclosed flaws before a patch is even available, which turns the question of where and how you run your database into a live operational risk rather than a matter of preference.

That is exactly the terrain Peter Zaitsev has worked his whole career, an entrepreneur, author, and co-founder of Percona. Two decades of helping companies choose, run, and sometimes escape their databases give him a sharp read on which reasons hold up under real load and which ones fall apart the moment the system starts to matter.

In today’s issue, he makes the case for treating that choice with the seriousness it earns, and for choosing as though you can never walk it back.

You can also watch our full conversation with Peter Zaitsev here.

Let’s get started.

Building Intelligent AI Agents with GraphRAG

This Agentic Engineering live bootcamp walks you through building an agent that investigates across a knowledge graph, remembers what it found, and returns answers you can put in front of a user.

Friday 31 July to Saturday 1 August, online.

Register with code DEEPENG40

Expert Insight

Databases are sticky. Most teams underestimate the commitment

by with

Most teams choose a database the way they choose a lunch spot, quickly and on somebody else’s recommendation. A friend runs it in production, a conference talk made it sound fast, or the cloud console offered it as the first option on the list. The 2025 Stack Overflow Developer Survey put PostgreSQL at the top for the third year running, in use by 55.6 percent of all developers and 58.2 percent of professional ones, far ahead of second-place MySQL. The default has never been clearer. But the decision behind it still rarely gets the weight it deserves.

Peter Zaitsev has advised companies through twenty years of these choices, from the greenfield pick to the migration nobody wanted. He co-founded Percona, grew it from a two-person company into one of the most respected open source database companies in the industry. His view of what most teams get wrong is straightforward. “Databases are very, very sticky,” he says. Once an application leans on one, unwinding that dependency can take years and burn through budgets, which is exactly why choosing one casually is the expensive mistake.

The cost is not hypothetical. Sharing from his experience advising enterprises through these migrations, Zaitsev says, “They started this kind of very painful migration away from Oracle a decade ago, and many, many millions of dollars, many, many years, they’re still on it.” A database that took a weekend to adopt can take a decade to leave. That asymmetry is the reason the rest of his advice matters.

Nobody owns Postgres, and that is most of why it won

In our conversation, I asked Zaitsev why Postgres reached the top, and his answer began with the part engineers tend to skip. “PostgreSQL is truly a community database. Nobody owns Postgres,” he says, and as he sees it that governance fact, more than any feature, explains the dominance. Because no single vendor controls it, every cloud can offer it as a first-class product without handing a rival an advantage. “Amazon, Google, Microsoft, a lot of others, they can offer PostgreSQL as their own,” he points out, where promoting MySQL or SQL Server means promoting something a competitor owns.

The technical case follows the same shape. Postgres extends rather than forces migrations, so each new demand becomes an extension instead of a second database. Time-series work got its extension, and the AI wave got pgvector, the open source extension that adds vector types and similarity indexes and now ships enabled on managed Postgres from AWS RDS to Supabase. “pgvector, vector search, enabling building AI application in Postgres, that came about very quickly,” Zaitsev tells us. The base keeps getting stronger underneath all of it, and the September 2025 release of PostgreSQL 18 added a new asynchronous I/O subsystem the project measured at up to three times faster on reads.

Open source is not enough

The reason to prefer the community model becomes obvious the moment a vendor changes the rules. “People even kind of lost trust in open source and started to understand that open source is not enough,” Zaitsev reflects, and he points straight at the pattern that taught them. MongoDB moved to the Server Side Public License in 2018, Elastic followed in 2021, HashiCorp adopted the Business Source License in 2023, and Redis switched to source-available terms in 2024 before adding an open license back in 2025 under community pressure. Each move left production users holding a version they could no longer upgrade under the terms they signed up for.

Zaitsev’s warning is that the marketing hides the trap. “A lot of companies right now are trying to trick you, using words like open source compatible,” he cautions, which usually means proprietary software wearing a thin layer of compatibility. Open core is the other tell. “There is some sort of limited, crippled version which is open source, but what they actually want is for you to buy the proprietary version,” he says. The protection he trusts is not a license badge but a market. A real open source project has many vendors, so a team unhappy with one can leave, and run a closed product like Amazon Aurora and only Amazon can support it, which puts the depth of experience and the leverage on one side of the table.

Self-hosting now means racing the exploit clock

Self-hosting an open source database moves the security burden onto the team, and the clock on that burden has changed. Zaitsev warns that with AI, security holes can now be found and exploited “at the speed which was never seen before,” and the incident data agrees with that. Google’s M-Trends 2026 report estimated the mean time to exploit a newly disclosed vulnerability at roughly negative seven days, meaning many flaws are weaponized before a patch exists, and Google’s threat intelligence group has reported the first zero-day it attributes to an AI-developed exploit.

That pace changes what good looks like. Zaitsev argues for a layered posture and an honest plan for failure, because prevention alone never reaches certainty. “Nothing gives you a 100 percent guarantee,” he reminds us, so the real question becomes how fast a team detects and responds, not only how well it defends. For many teams the practical answer is a contract, and even those that avoid a fully managed service should have someone on the hook for patching. In regulated environments the whole subject stops being a technical choice and becomes a compliance one, which is where managed databases earn their place by letting a team push responsibility to the vendor who runs the system.

Doing what the cool kids do is not a workload

The pull toward document, vector, and graph stores is often social before it is technical. “A lot of engineers, they like to explore and play with new technologies,” Zaitsev says, usually after hearing about them at a conference or on a podcast. He does not moralize about curiosity, but he counts the cost. “The more technologies you introduce, the more complexity you create,” he says, and a small team running many engines pays that bill every day. “If you are a five-person company and you are self-managing 10 databases, there’s probably way too much complexity.”

His rule is to start narrow and let real limits, not fashion, force the next move. Postgres holds relational data, documents, and vectors well enough that most teams travel a long way before they need anything else, and only a genuine ceiling on scale or a missing feature should justify a dedicated store. Purpose-built systems do win when the workload is real, because a database built for one task carries a more optimized store and a better language for it. The catch with AI is speed. “The AI world is completely different. Every six months things drastically change,” Zaitsev reasons, so a team building AI features has to check the date on its advice, since guidance a year old still serves Postgres or MySQL fine and can already be stale for anything touching models.

Choose like you cannot leave

The through-line of Zaitsev’s advice is to match the seriousness of the decision to the difficulty of reversing it. He wants teams to write down their actual requirements rather than inherit a default, and he is happy to use the newest tool to pressure-test the old discipline. “You can actually ask AI, hey, what am I missing,” he says, treating a model as a fast second reader on a requirements list. The point is not the tool. It is that a choice this durable deserves more than an afternoon.

That is the gap he keeps returning to. Teams decide in a hurry and then live with the result for years, on infrastructure that only grows harder to change as the application matures on top of it. Postgres has made the safe default easy, and the survey numbers show most teams taking it. The harder work starts after the default, in the licensing terms, the security posture, and an honest reading of what the workload actually needs.

In case you missed

Industry Perspective

🛠️ Tool of the Week

CloudNativePG, the Kubernetes operator for running production Postgres on infrastructure you control

CloudNativePG runs Postgres as a native Kubernetes workload and handles failover, backups, and rolling upgrades the way a managed service would, while the data and the control plane stay inside your own cluster.

• New DatabaseRole resources manage Postgres roles as declarative GitOps objects with password-free certificate authentication.

• A Kubernetes Lease now gates primary promotion, so replicas fail over without waiting out the full timeout.

• In-place major upgrades run pg_upgrade against mounted extension images, cutting version-migration downtime.

• Operator and Postgres images ship signed with SBOM and provenance attestations for supply-chain verification.

Learn more about CloudNativePG

📎 Tech Briefs

• PostgreSQL 19 Beta 2 adds native SQL/PGQ property-graph queries - Postgres 19 exposes relational tables as property graphs via GRAPH_TABLE pattern matching, so fixed-depth traversals no longer need a graph database.

• CISA adds an actively exploited SharePoint Server RCE to its Known Exploited Vulnerabilities catalog - CISA confirmed active exploitation of CVE-2026-58644, a deserialization remote-code-execution flaw in on-premises SharePoint Server, and ordered urgent patching.

• MariaDB Community Server 10.6 reaches end of life - MariaDB’s last old-model LTS no longer receives bug or security fixes, so 10.6 shops now face an unavoidable migration.

• Autobase 2.9.0 released. The open source automated Postgres high-availability platform now manages cluster infrastructure after deployment, not only initial setup.

• CISA flags an actively exploited authorization bypass in Langflow - CISA confirmed in-the-wild exploitation of CVE-2026-55255, an authorization bypass in the open source LLM app builder Langflow.

That’s all for today. Thank you for reading this issue of Deep Engineering.

We’ll be back next week with more expert-led content.

Keep building,

Saqib Jan

Editor-in-Chief, Deep Engineering

If your company wants to reach senior developers, software engineers, and technical decision-makers, speak to us about partnering with Deep Engineering.

By Chuck McCullough, software architect and educator with more than 30 years designing, building, and teaching enterprise software systems.

An AI coding agent can feel like having a junior developer on the team. It is capable, fast, and useful, but still dependent on context, constraints, feedback, and review. The shift I want engineers to make is to stop asking an AI to write code and start running a reliable engineering workflow with an agent inside it. A feature tour teaches people where the buttons are. A workflow teaches them how to start with intent, give the agent the right context, constrain the change, verify the result, review the diff, and improve the system for next time.

That workflow matters even more because the tools are evolving so quickly. What required close supervision a year ago may now be safe to delegate for longer stretches, but only if you know how to plan the work, set guardrails, and review checkpoints. The same iterative, test-driven, review-focused practices that make a human engineer reliable also make an AI agent reliable.

The best mental model is an agentic teammate with tool access, bounded authority, and a need for review. Codex can reason about a codebase, make edits, run commands, create tests, and explain its work, but it does not own the product judgment or the architectural risk. Pair programmer, junior developer, and build agent are all useful metaphors, and the mistake is turning any one of them into the whole truth. Treat Codex like a human senior engineer and you may skip review. Treat it like autocomplete and you will underuse its ability to inspect, test, and iterate. Treat it like an unattended build script and you may hand it too much authority where judgment still matters. The best users do not just prompt better; they scope better, test better, review better, and encode what they learn into repository instructions, checklists, and repeatable practices.

Start by reading, not editing

When Codex meets an unfamiliar codebase, it should first establish the map: repository layout, README or developer docs, package or build files, test commands, entry points, existing conventions, and any agent guidance such as AGENTS.md. Then it should inspect the specific files related to the task, not the entire codebase indiscriminately.

Before any edits, I verify three things: that Codex found the right subsystem, that it understands the current behavior and the desired behavior, and that the proposed change is scoped to the right files. This matters most in unfamiliar repositories, because the first failure mode is not bad syntax. It is a plausible change in the wrong layer. Existing repositories are imperfect artifacts. They carry years of evolving patterns, dead code, and inconsistent conventions, so Codex has to understand that the first pattern it sees is not automatically the pattern it should follow.

The explanations Codex gives are only worth acting on when they are grounded. A trustworthy explanation points to concrete evidence. It names files, functions, command outputs, tests, call paths, configuration, and observed behavior, and it distinguishes what it saw from what it inferred. It also admits uncertainty, something like, I found this pattern in two routes, but I have not verified the background job path yet. Plausible narration is usually too smooth. It describes the system without showing where the claims came from, ignores exceptions, or confidently describes architecture that is not reflected in the code. The practical test is simple. Ask Codex to cite the files and lines that support its explanation, then ask what would disconfirm it. If it can answer both, you are closer to something you can act on. Correct explanations and hallucinations can look alike at first glance; the difference is that a correct explanation can be traced back to the codebase and a hallucination cannot.

Scope the work to a focused feature slice

A well-scoped task has a goal, relevant context, constraints, and a definition of done. For example: “Add CSV export to the reports page using the existing export service. Do not introduce a new dependency. Done when the new unit tests pass and the UI exposes the existing download pattern.” A focused feature slice is the smallest useful change that crosses the necessary layers without becoming a rewrite. It might be one UI affordance, one API path, one service method, and tests for that behavior.

You have asked for too much when the task mixes multiple product decisions, touches unrelated subsystems, requires broad architecture discovery, or cannot be reviewed as one coherent diff. When I ask for too much, agents tend to go down an expensive detour. They start solving a problem they manufactured, which creates more problems, and you can end up fixing something you did not have in the first place. Scope down to a slice, then iterate.

Deciding what to delegate follows the same instinct. I look at blast radius, reversibility, test coverage, and judgment load. Safe delegation tasks are narrow, easy to review, well covered by tests, and reversible: small bug fixes, test additions, documentation, code cleanup, local scripts, and feature slices with clear acceptance criteria. Direct human implementation is better when the work involves ambiguous product decisions, security-sensitive behavior, data migrations, production incidents, novel architecture, compliance constraints, or weak observability. Codex can still help explore options, but the human should drive.

Senior engineers still need to own the decisions that affect long-term coupling, data ownership, security boundaries, operational behavior, and public contracts. Codex can propose options, weigh tradeoffs, and implement the chosen path, but the team decides the architecture. On a recent project I chose a Blazor application because the product was highly interactive and I wanted the logic and UI components to stay primarily in C# and Razor. Codex did a strong job overall, but I had to be explicit that JavaScript should be used only as an intentional interop choice, not as a parallel client-side architecture. The way to express a constraint is to be concrete. Point Codex to the existing pattern and say whether to follow it, extend it, or avoid it: “Use the existing ReportExportService, do not add a second export pipeline,” or “Keep validation in the domain layer, not the controller.” Durable constraints belong in AGENTS.md, architecture notes, tests, linters, and review checklists, so the agent sees them before each task.

Write the test before the implementation hardens

TDD keeps me honest as a human engineer. It forces me to think about the behavior I want before I start coding, and it gives me a safety net for regressions. That does not change when an agent writes the code. If anything it matters more, because the agent may not share my understanding of the system. When an agent writes the code, tests become the contract that keeps speed from turning into drift. Without them, Codex can produce something that looks reasonable, compiles, and works for the happy path while missing the actual requirement.

Lightweight TDD does not mean ceremony. It means expressing the expected behavior before the implementation hardens around the agent’s first guess. A failing test, a reproduction script, or a clear acceptance check gives Codex a target to work toward and gives me a neutral way to review the result. To keep the agent from writing tests that merely confirm its own implementation, start the tests from user-visible behavior, a bug reproduction, or a business rule, not from the existing code. Ask for the test cases first, review them, and only then allow implementation, and include edge cases that would fail if the implementation took an easy shortcut.

Even when generated code passes the tests, it can still feel wrong. Then I review the shape of the solution: coupling, layer placement, unnecessary abstraction, dependency changes, security assumptions, and whether the implementation matches the local style. Passing tests are necessary, but they do not replace design review. Think of architecture as the box a feature has to live in. A slice needs to fit neatly in that box alongside the others, and design is the engineering effort of finding an elegant way to do that.

Legacy code makes this harder. A codebase with very few tests may not be ready for classic unit testing right away, because code written without tests tends to have a different shape: more hidden coupling, fewer seams, more global state, and less separation between business logic and infrastructure. Use Codex in read-first mode before edit mode. Have it map the relevant behavior, identify seams where tests can be added, and propose the smallest safe change, and keep permissions conservative until you have some executable checks. Characterization comes first. Give Codex examples, logs, fixtures, screenshots, API responses, or current behavior, then ask it to write tests that capture that behavior, and review them before refactoring. If the tests are meaningful, they become guardrails. If they simply mirror implementation details, they become decoration.

This is why the classic TDD argument still matters: tests are not just a verification tool; they are a design force. When that force was absent for years, you cannot sprinkle unit tests on top and expect the codebase to behave as if it had been test-driven from the start. A legacy codebase without tests is a bit like buying a car with 100,000 miles and no record of oil changes. You can probably still drive it, but you should not pretend it carries the same risk as a well-maintained one. The missing tests are deferred maintenance, and earlier shortcuts have raised the cost and risk of every future change. (You can’t retrofit those missing oil changes.)

Featured: Build AI Agents with OpenAI Codex

A hands-on Packt workshop where you practice these workflows on a real starter project, from reading an unfamiliar codebase to shipping an end-to-end capstone.

Register with code DEEPENG40 →

Online, Saturday and Sunday, July 25 and 26.

Debug by checking assumptions, then know when to reset

When Codex gives a plausible but incorrect diagnosis, I start with the assumptions. A wrong diagnosis usually comes from a mistaken belief about how the system works, what changed, or what the failure means. Then I inspect the failing test and logs to ground the conversation in evidence, and after that the diff, to see whether the attempted fix actually addressed the observed failure. The prompt matters too, but usually after you know what was misunderstood. The best recovery prompt is not try again. It is: pause, list the assumptions behind the last diagnosis, compare them against the failing output, and propose the next smallest verification step.

To avoid the endless try-another-fix loop, set a rule that after one or two failed fixes Codex must stop changing code and return to diagnosis. Ask it to summarize what it changed, what evidence supported the change, what evidence contradicted it, and what it still does not know. A good reset point is a clean diff plus a fresh reproduction. Revert or set aside speculative edits, rerun the failing command, and ask for a hypothesis-driven plan before more changes. Make the next step observational: inspect, reproduce, isolate, then edit.

Review for whether it fits, not just whether it works

Review Codex-generated work with the same seriousness as human work, but with extra attention to plausibility risk. Agentic code can look polished while quietly changing a boundary, adding a dependency, broadening permissions, or testing the implementation rather than the requirement. The highest-risk areas are coupling, security assumptions, data handling, dependency changes, migrations, and tests that are too shallow. I also watch for architecture by convenience, where Codex solves the immediate problem by creating a parallel helper, a duplicate abstraction, or a special case the existing system did not need.

This comes back to design. Poor human-led design can produce a different solution for every user story, and agents are no different, except that they can create the mess much more efficiently. If Codex is allowed to invent a new helper, pattern, service, or exception for every slice, the codebase fills up with solutions that do not work well together. That is why review has to ask not only does this work, but also does this fit. When generated code works but does not match the internal style or architecture, treat it like any other code that fails review. Ask Codex to revise the change to follow the existing pattern, and point it to the specific file or convention it should match. Then update the durable guidance if the issue is likely to recur, in AGENTS.md, a review checklist, a lint rule, a template, or a test. The goal is not to scold the agent; it is to make the expected path easier to follow next time.

Standardize the workflow before you scale it

For a team, standardize repository instructions and review expectations first. Prompts are useful, but durable guidance is what makes behavior repeatable across engineers and sessions. A strong starting point is an AGENTS.md that includes build commands, test commands, style conventions, architecture constraints, and what done means, paired with a lightweight review checklist. Allowed task types come next. Start with low-risk, high-feedback work: tests, small bug fixes, documentation, refactors with strong coverage, local tooling, and narrow feature slices. Draw hard boundaries around secrets, regulated data, security-sensitive authorization logic, production incident response, destructive operations, and changes that require business or architectural judgment, until the team has mature controls.

For anything beyond low-risk local work, do not let Codex write production code before a review policy is in place. The policy does not need to be heavyweight, but the team should agree on what Codex is allowed to change, what checks must run, who reviews the diff, and which areas are off limits. Without that, you are relying on individual judgment under speed pressure, which may work for a demo but is not a production practice. I have a related concern. Management may treat AI as a replacement for developer judgment. We have seen versions of this before, where organizations substituted cheaper labor for experienced engineering judgment and then paid the cost in quality, maintainability, and rework. AI can produce plausible code quickly, but it still needs experienced review to catch nuance, architecture fit, and risk. There are scenarios where Codex is the perfect resource for the problem; the key is knowing the difference.

One of the biggest changes over the last year is that prompt engineering is less of the whole story. It still matters, but the best teams are not just prompting Codex to write code. They are prompting it to run a workflow they have already defined and standardized. The prompt is the entry point. The workflow includes planning, repository guidance, permissions, tests, review, rollback discipline, and team conventions. It is less about clever wording and more about engineering control. That is also the real difference between someone who can prompt Codex and someone who can run a safe workflow. The first can get an answer or a diff. The second can turn intent into scoped work, give the right context, set boundaries, verify behavior, review risk, and feed the lessons back into the team’s process.

When the stack is fixed, point Codex at it rather than at generic code generation. On a Python project that means pyproject.toml, dependency management, test commands, linting, formatting, type checking, and package layout, with the agent working in small slices and running the relevant checks. Python especially benefits from tests around edge cases, because dynamic typing can hide integration mistakes until runtime, and I still review dependency changes, packaging changes, security-sensitive code, and data handling by hand.

Measure the work honestly

Managers should avoid measuring only lines of code, number of prompts, or raw tickets closed. Agentic workflows can increase output volume, but volume is not the same as value. Better metrics combine throughput with quality and review health: cycle time for small changes, escaped defects, review rework, test coverage movement, incident rates, lead time from idea to verified change, and developer satisfaction. I would also track whether teams are improving their reusable guidance, because good AGENTS.md files, checklists, and skills compound over time.

A July 2025 METR study found that, in one setting, 16 experienced open-source developers working on familiar mature repositories took 19% longer when early-2025 AI tools were allowed, even though they believed the tools made them faster. I treat that as a cautionary snapshot rather than a permanent verdict, because the tools and workflows are changing quickly. My own experience matches the caution behind it, though I would frame the issue as supervision cost. If I give an agent a small task, it is safer and easier to review, but the interval is often too short for me to do meaningful parallel work, and I end up waiting for the agent instead of coding. The better pattern is to collaboratively create and approve a plan made of small, reviewable steps, then let the agent execute against that plan for a longer period. That gives me a real delegation window without turning the work into one large, unsafe prompt.

The habit to drop

If there is one habit to stop immediately, it is accepting the first plausible diff. Treat the first result as a draft that earned the right to be reviewed, not as finished work. Ask what changed, why it changed, how it was verified, and what risk remains. The habit to build instead is pausing before acceptance. Read the tests. Read the diff. Run the checks. Ask Codex to review its own work against your standards. Serious agentic development is fast, but it is not careless. For complex tasks I ask Codex to create a plan first, review that plan, refine it with the agent, and then ask it to implement. That keeps me in control, gives the agent enough runway to work autonomously, and creates a recovery point if the work starts to drift.

These are the ten workflows I walk through hands-on in Build AI Agents with OpenAI Codex, a Packt workshop on July 25 and 26, where the point is to practice them on a real starter project rather than read about them. We cover reading an unfamiliar codebase, scoping a feature slice, writing tests first, debugging and resetting, reviewing for fit, and preparing work for team handoff, then run it end to end in a capstone.

We asked him how teams should choose a database in 2026, why Postgres keeps gaining ground, and where open source, security, and the AI rush change the calculation.

You can watch the full conversation below, or read the cleaned up transcript that follows.

This session was recorded offline as part of the Deep Engineering Interview Series. The transcript below has been lightly edited for clarity and readability.

Q. Tell us about yourself, your journey with Percona, and what you are focused on now.

I started working with databases more than 20 years ago, and I have been running Percona for 20 years. We help customers choose, deploy, and manage databases, and we focus on open source. We started with MySQL as a company, and we now cover the most popular open source databases.

Q. When you look at what engineering teams are reaching for in 2026, which databases come up most, and what is driving those choices at the technical level?

If you look at the databases most commonly used right now, I would name Postgres. It is by far the most popular database chosen by engineering teams these days, and that happens for both technical and a lot of non-technical reasons.

Q. Where do you see teams picking a database for reasons that do not survive contact with real load, and what should they weigh instead?

A lot of people choose a database based on what a friend is using, or something they read about at a conference. Relational databases are interesting. If you are building a basic application, you can do that in pretty much any database. You can choose MySQL, Postgres, Mongo, even Oracle, and you can build your application. Now if you are building something more special purpose, high performance, high volume, where you have to be very cost effective, then you have to think about the database a lot more.

Q. Stack Overflow’s developer surveys have placed PostgreSQL at the top for professional use for a couple of years now. From your years in the open source community, what do you think Postgres got right to earn that spot?

I think there are both technical and non-technical reasons behind Postgres, and the non-technical reasons are the most important here. PostgreSQL is truly a community database. Nobody owns Postgres, and that allowed everybody in the industry to cooperate on it. Amazon, Google, Microsoft, and a lot of others can all offer PostgreSQL as their own, so there is a level playing field for those companies. If you are promoting MySQL or Microsoft SQL Server instead, that is something owned by a particular company, and you are giving your competitor more power. That also helped PostgreSQL become ubiquitous. There is a PostgreSQL event happening probably every week somewhere in the world, because so many people are interested in promoting the technology.

From the technical standpoint, PostgreSQL is very good in terms of modularity and building extensions. If you think about what has been happening in the industry, time-series databases became important, and Postgres could very quickly provide an extension to do that. More recently AI arrived, and pgvector and vector search enabled building AI applications in Postgres very quickly. There are a number of other extensions in the same vein. That is a very important technical reason why Postgres is doing so well.

Q. Where is Postgres actually landing in production today, and which workloads are teams still wrong to put on it?

PostgreSQL has a lot of very big installations these days. You can find PostgreSQL ranging from general applications to a lot of use in the financial and banking sectors. It is again very ubiquitous, very scalable, a general purpose database.

Q. Data sovereignty and open source licensing are shaping deployment decisions right now. What technical constraints do they introduce, and where do they force an architecture change rather than just a policy one?

You touch on an important point, because a database is often chosen in the context of the platform. What we often see is customers making high-level decisions about where they go. They might say they are going to use AWS for everything, and that drives the database choice. Maybe they follow what the cloud suggests, and that is Postgres, but the proprietary version of Postgres that Amazon provides. It is important to look at what your organization needs in terms of data sovereignty and data independence, and what kind of controls you want over cost.

What I see here is a separation. A lot of startups say they just need to build as fast as possible, so they take the easy path, whether that is Amazon or Supabase. A lot of larger organizations have to think about cost and compliance. If they have a massive multinational operation, they have to follow local laws, and they decide they want more control. That often brings them to the open source side, running Postgres in a cloud native, Kubernetes-driven environment. That gives them a very high level of automation similar to what managed cloud vendors provide, while keeping them completely in control of their database.

Q. When a project relicenses or a vendor closes up, what does that actually cost the engineering team, and how do you design to keep that exposure low?

You are making an important point, because of what has happened in the last few years. People have even lost some trust in open source and started to understand that open source alone is not enough. You can see some companies, for example MongoDB, that had an open source database and then changed it, so now it is source available or entirely proprietary. There are also open source companies that shut down. And you can find open source software that still exists but is not maintained, which is not really safe to use in production.

So when you are choosing open source software, you first need to understand what open source actually is. A lot of companies right now are trying to trick you with words like open source compatible, which actually means enterprise software that is not open source but has some compatibility. Or they use a word like open core, which means there is a limited, crippled version that is open source, but what they actually want is for you to buy the proprietary, paid version. You need to understand what you are buying and whether open source is best for you.

The second reason, and this is why PostgreSQL is big, is having a large community, and a community of vendors too. If you choose an open source project you can hire many companies to help you. You can choose one vendor to give your business to, and if you do not like us, if we are too expensive or the service is not good, you can go somewhere else. With proprietary software you do not have that option. If you are running Amazon Aurora, only Amazon can really support that product. It comes down to deep experience.

In many cases early stage engineers are so focused on building their application and getting to market fast that they do not think about this. I am not sure all of them should. In a startup there are a lot more reasons to die than overpaying for your database. But as things transition to a more mature environment, those questions should be asked. A lot of enterprises started a painful migration away from Oracle a decade ago, spent many millions of dollars and many years, and are still on it, because databases are very sticky. If you have built a very complicated system locked in to certain database technologies, it may be hard to move.

Q. Let’s shift to security. Self-hosting an open source database puts the security burden on the team itself. What do you see teams underestimate, from patching cadence to backporting fixes into an older version they are stuck on?

Security is critical, especially now, because with AI, security holes can be found and exploited at a speed we have never seen before. If you take the responsibility on yourself, or you choose a vendor to do it for you, you need to make sure that policy is very strict. It is also very important to have a multi-layered security approach, because nothing gives you a 100% guarantee. You can see even some of the most serious environments get exploited. That means you need to make sure you not only have a defense and a process to prevent it, but also a plan for how you react when an incident happens.

This is a very specific area of experience for many teams. Even if they choose not to go with a fully managed cloud solution, it helps to work with someone on their security policies, and to have a contractual relationship with someone who helps with security patching. In many cases, when it comes to security, it becomes not just a technical question but a compliance question for a lot of larger organizations.

Q. Would you consider managed cloud databases in that scenario?

Managed cloud databases are not bad. They are a good choice for a lot of folks, and as I mentioned, they have a lot of trade-offs. When it comes to security, they do allow you to push more responsibility towards the vendor who manages your database. It is important to remember, when you think about the number of database choices deployed in the market, that there is no one-size-fits-all answer. It is not like saying you build an application, you need a database, so here are the steps, you choose this database, you run it this way on this operating system, you use this programming language. That is not helpful. There are a lot of considerations, and the answer varies.

Q. Teams are running document, vector, and graph databases alongside relational. Which of these are a dedicated store, and which are teams adding on hype rather than a real workload need?

It is a combination of technical and non-technical reasons. A lot of engineers like to explore and play with new technologies. They heard something at a conference or on a podcast that sounds very cool, and that is how a lot of technologies get introduced. In my experience, the more technologies you introduce, the more complexity you create. If you are a five-person company self-managing 10 databases, that is probably way too much complexity.

What I would suggest is starting with something, and PostgreSQL is often a good choice, then seeing how far you can take it. PostgreSQL can store relational data, it can store documents, and it can store vector data. Then maybe you find that in terms of scale, or some feature you need, you need a special purpose database, and you decide to use one for that given workload.

A lot of companies aspire to learn from giant-scale companies. They look at what Google or Facebook or OpenAI is doing and try to do the same. But those companies operate at a very different scale. For a lot of engineers, from a non-technical standpoint, it is just nice to do what the cool kids are doing.

Q. Vector databases specifically have exploded with the AI wave. Where does a purpose-built vector store beat adding vector search to Postgres or an existing engine?

Any database that is purpose-built for a particular task can often do better. It can have a more optimized store, and a language that is better suited for that task. That is quite common, and the same thing happened before with document-focused databases and time-series databases. What is very interesting with vector databases is that AI is moving so quickly. It really becomes a question of what exactly you need for building these AI applications.

We talk about a database storing vectors, and of course you may want to search them, but what else do you want the database to do? Maybe you want broader functions ranging from embedding generation, or some higher level functionality. Do we focus on the low level, which is vector storage, or do we want an application that can speak natural human language and bring all the intelligence that frontier models have, along with the enterprise and domain knowledge we already hold? Is that the database’s job, or is it a custom built application? A lot of that is still evolving, including how the architecture is going to look.

If you go to AI databases, my advice would be slightly different. Mature relational databases are many decades old. MySQL celebrated something like 30 years, and PostgreSQL is around the same. A lot of things in the core are already decided, and that is how we build them. The AI world is completely different, and every six months things change drastically. So when you are building AI applications, you really want to look at the dates. Best practices that are a year old are probably still very good advice for MySQL or Postgres in general, but for building AI apps they may be too outdated by that point.

Q. If you are advising a senior engineer building out data infrastructure this year, what is the decision you watch them get wrong, and what advice would you give them?

The advice would be to make sure you understand the needs you actually have. With AI this is quick now. Even if you write down what you think your real requirements are, you can ask AI what you are missing, and get some useful input to consider. In way too many cases the choice of database is a very quick decision without a lot of thought. As I mentioned, databases are very sticky, especially if you are building an application that will last for many years and run at large scale. So make sure to give it real thought.

By Imran Ahmad, PhD, author of Building Reliable AI-Assisted Software, forthcoming from Packt.

The support assistant sailed through every demo. It answered politely, cited the right documents, and impressed everyone in the room. Three weeks after launch, a customer asked about a refund on an annual license. The assistant replied, warmly and confidently, that under the company’s 30-day satisfaction guarantee it had approved a full refund of $4,200.

There was no 30-day satisfaction guarantee. There was no refund. The company is a composite drawn from documented incident patterns, but the mechanics are real. And if a composite feels too convenient, ask Air Canada. In 2024, a British Columbia tribunal ordered the airline to honor a bereavement discount its website chatbot had invented, rejecting the argument that the chatbot was “a separate legal entity responsible for its own actions.” In plain terms: the company owns what its agent says.

One in five

An AI feature that succeeds 80% of the time is indistinguishable from magic in a demo. In production, that same number means one in five real users gets a wrong answer, a broken promise, or an invented policy. I call that space the Reliability Gap: the distance between a system that impresses in a demonstration (roughly 80%) and one that can be trusted in production (99.9%).

The gap is wider than it looks, because errors compound. Chain five steps together (parse, retrieve, reason, call a tool, compose), give each one a respectable 90% reliability, and the whole chain works out to 0.9⁵, about 59%. Worse than a coin flip. Reliability multiplies down an execution chain; it does not average.

The first 80% is also seductive, because it arrives almost free, for roughly 20% of the total effort. The climb to 99.9% is the actual engineering. That is the 80/20 Demo Trap, and much of the industry is standing in it right now.

The season of receipts

The prevailing practice has a name. When Andrej Karpathy coined “vibe coding” in February 2025, telling people to “fully give in to the vibes... forget that the code even exists,” he was describing, with a wink, a perfectly good way to explore: tweak the prompt, eyeball the output, repeat until it feels right. Before the year was out, Collins had named it Word of the Year. In between, the receipts arrived.

An AI coding agent at Replit deleted a live production database during an explicit code freeze, wiping records for more than 1,200 executives, then fabricated data to cover it up and insisted recovery was impossible. (It wasn’t. The data came back from a backup the agent said didn’t exist.) Cursor’s own AI support bot invented a device-limit policy, and paying customers cancelled over a rule no human had written; the real cause turned out to be an ordinary session bug. Veracode, after testing more than 100 models, found that roughly 45% of AI-generated code fails standard security checks.

None of these models was malfunctioning. Each one did exactly what a probabilistic system does: it sampled a plausible answer. The failure is that nothing downstream ever checked the result.

The poet and the accountant

The way out starts with a change of frame. You are not building an AI. You are building a software system that uses AI as one component. It happens to be the most capable component you have ever integrated, and the least trustworthy.

In my book I put it this way: a language model is a poet. Brilliant, fluent, tireless, and constitutionally incapable of doing the same thing twice. For fifty years, software was built entirely by accountants: deterministic, repeatable, auditable. We have just hired our first poet and asked it to help keep the books. You do not fix the poet, and you do not fire the poet. You give the poet an accountant.

The accountant is a deterministic shell around the probabilistic core: input validation, context assembly, bounded control flow, output guards, telemetry. Ordinary, testable, frankly boring software. Setting temperature to zero does not build it, and waiting for next year’s model misdiagnoses an engineering problem as a capability deficit.

Four questions

The shell is what you get when you can answer four questions. I ask them in every design review now.

1. How do I know it’s working? Not “the demo looked good.” A versioned golden dataset and automated evaluation, so quality becomes a number that moves. The dataset is your spec.

2. What does the model see? That $4,200 promise wasn’t a model failure. Retrieval had served the wrong policy document. Context is application state, and it deserves engineering.

3. How much autonomy does it get, and how is it bounded? The Replit deletion is what unbounded autonomy looks like. Every loop needs a step budget, a cost budget, a tool allowlist, a stop condition, and an escalation path.

4. What must never happen? Air Canada’s answer should have been “an invented policy reaches a customer.” That rule belongs in code, not in a prompt. Pleading is not a control.

Each question anchors one of my book’s four pillars. Together they reveal its central finding: raw answer quality plateaus in the mid-90s, because models plateau, while the rate of safe outcomes keeps climbing toward 99.9%. The shell gets you the nines, not the model.

See it live at ARC 2026

That is the ground I’ll cover at the conference. In Saturday’s tech session (Designing Reliable AI Systems, July 25, 12:45 PM ET) we do it live: break a support assistant on stage, watch it make the $4,200 promise, then rebuild it with a five-layer shell that contains the same failure. We’ll also watch an unbounded deploy agent turn a quiet weekend into a $23,400 cloud invoice, then see the three lines of code that make it $0. Everything runs deterministically, offline, on one laptop.

In Sunday’s keynote (Architecting Principles in a Probabilistic World, July 26, 10:30 AM ET) I’ll make the argument underneath the code: eight architecting principles for building on a component that answers differently on Tuesday than it did on Monday. One of them explains why a trading-firm collapse from 2012, with no machine learning in sight, is the most instructive AI story of 2026. The models will change; the discipline will not.

These ideas come from Building Reliable AI-Assisted Software, my book forthcoming from Packt. The opening chapters are complete and the rest is in active development. Until then, try the exercise I give every team: pick one AI feature you’ve already shipped and ask it the four questions. The one you cannot answer is where your gap is.

Featured: ARC 2026: Software Architecture in the Age of AI

AI is reshaping software architecture, putting new demands on scalability, governance, reliability, and observability. ARC 2026 brings together architects, CTOs, and AI practitioners for keynotes, panels, and workshops on agentic system design, modernizing enterprise apps for AI, and building governable, observable AI systems.

🗓️ 25 to 26 July, 10:30 am ET

Use code DEEPENG50 for 50% off the early bird price.

Reserve your spot

About the author

Imran Ahmad, PhD, is the author of Building Reliable AI-Assisted Software, forthcoming from Packt. His approach to reliability is informed by years of building large-scale systems for government and enterprise, where failure is not an option

By Julien Dubois Principal Manager, Developer Relations at Microsoft and GitHub. Creator of JHipster.

I always wanted a real developer UI for Spring Boot. Every Spring app is a black box in development. Actuator gives you raw JSON, not a console, so what I wanted was health, metrics, security and tracing in one embedded UI.

I’d shipped a slice of this in JHipster years ago, but only for generated apps. A real console for any Spring Boot app sat on my wishlist for years. The catch is that it’s massive. Around 40 panels, each one a backend plus a frontend plus its own tests, deeply integrated across a dozen JVM subsystems. By hand, one experienced developer would need somewhere between six and a half and eight and a half months. Too big to justify, until I stopped writing the code myself.

Then I did it in 11 days. Not by typing faster. I didn’t even open my IDE. I managed a fleet of AI agents while I architected, reviewed and steered. The agents did the scaffolding, the panels and the tests. I did the judgement.

What I built, BootUI

BootUI is a production-grade Spring Boot 4 starter that adds an embedded, local-only developer console to your app. You add one starter to your pom.xml, run locally, and open the console.

It is a five-module Maven build on Spring Boot 4 and Java 17, integrated across Actuator, Spring Security, Flyway and Liquibase, Hibernate, Micrometer with OTLP, GraalVM, OSV scanning and ArchUnit, published to Maven Central with full CI. The frontend is an embedded Vue 3 single-page app, packaged inside the starter. Each panel is endpoints plus a view plus tests.

There are around 40 feature panels. Health and metrics, a security advisor, a vulnerabilities view, tracing, and more. Those panels look a lot alike, which made them the biggest source of structural repetition in the codebase. That detail matters, and I will come back to why it made the whole thing work.

Eleven days, and how I know the numbers are real

The figures here are derived from git history, PR metadata and code metrics, not from time-tracking logs. Through the tagged 1.0.0 release there were around 264 commits on main and about 223 squash-merged pull requests, landing at roughly 20 a day across 11 days. That came to about 83k total tracked source lines, close to 50k of them Java across around 461 files and 81 test classes, plus 52 Vue components, 40 panels, and 35 end-to-end specs. In all, about 116 test suites.

A cadence of 20 PRs a day with an AI agent co-authoring commits is impossible to reach by hand. The agent did the typing. I dispatched many asynchronous tasks in parallel. The evidence lines up with that. The commit clock runs from around five in the morning to midnight most days, which is consistent with parallel async tasks rather than continuous typing. “Copilot” shows up as a named commit and PR author on about 44 commits, and the repo ships a copilot-instructions.md with per-panel conventions, so the workflow was explicitly agent-oriented. Even on release day I was landing a docs site, a scanner dashboard, token charts and a Hikari fix, which is itself several days of solo work.

Where my own time went was writing prompts, reviewing and merging those 223 PRs, and resolving CI failures around Spring Boot 4, Flyway 11 and OTLP. Review and orchestrate, not write every line.

The honest by-hand comparison is the part people argue about, so I ran it carefully. One experienced Spring Boot and Vue developer, with no AI codegen, would need 6.5 to 8.5 months for the same polished 1.0.0, roughly 1,100 to 1,450 hours of hands-on effort, and the 40 panels are the dominant cost. A COCOMO organic estimate on 50k lines yields more than 100 person-months, which is too high because much of the code is repetitive scaffolding, so a domain-expert solo figure of about 7.5 months is the defensible middle ground. My actual human effort on BootUI was 80 to 110 hours, about two intense solo weeks. That is a calendar speed-up of roughly 17 to 23 times, and a human-hours speed-up of roughly 12 to 17 times.

I was the manager, not the developer

I didn’t open my IDE. I wasn’t the developer. I was the manager. Many agents ran in parallel on the panels, the integrations, the frontend, and the CI and docs, and my job was to brief, review and merge.

Your throughput is not your keyboard. It is your briefs. What blocks a single developer is typing, and what blocks a single agent is you waiting for it to finish. Running many at once removes both, because they come back one after another and there is always something to review. You don’t type faster this way. You ship what used to take months.

A day in the loop

The rhythm that produced 20 merged PRs a day is a simple daily loop, and the day is the engine. I am hands-on all day. I spec, launch, review, merge and re-task, live, and most of the day’s PRs land right there.

The evening is a hand-off. Before I step away I queue a few deep, long-running plans. The night is a bonus, not the engine. A handful of deep autonomous runs finish by morning, which is the minority of the work. Repeat that for about 11 days and you reach a tagged 1.0.0. Six ingredients make the loop work.

Ingredient one, write the specifications

The first thing I write is never code. It is the house rules. An AGENTS.md, which GitHub Copilot reads as copilot-instructions.md, and every agent reads it first. It sets the conventions once, the stack, the build, the tests and the style, with per-panel conventions so 40 panels come out consistent. Mine runs around 700 lines. That is about the right size, because a file that is too big pollutes the context, and it also costs tokens and money on every run.

The high-leverage move is writing the known failures into that file once. Most models were trained on Spring Boot 3, so they struggle with the reworked Jackson API in Spring Boot 4, and 10 agents trained on the same data hit the same wall at the same time. Rather than fix that by hand in each one, I write the fix down once. The same goes for operational traps, like giving every agent its own local Maven repository so parallel installs do not corrupt a shared one.

On top of the house rules I write one spec per task. What to build, where, what “done” looks like, and the acceptance test the agent must make pass. Small, self-contained, no hidden dependencies. The spec is the product now. The better the brief, the less you babysit.

Ingredient two, build the test harness

No agent runs without a test harness. One command compiles and runs the unit and end-to-end tests, and comes back green or red, so an agent can verify its own work before it opens a PR. No tests means you cannot trust the output.

CI is the trust layer. BootUI has about 116 test suites, 81 in Java and 35 in Playwright, and CodeQL plus the end-to-end tests gate every PR to main. The loop does the work. An agent writes code, pushes it, and GitHub Actions sends back the results. If they are red the agent reads the failure, fixes it, commits and pushes again, and the checks run once more. Dependabot keeps the dependencies current alongside all of this. A compiled, typed language like Java helps too, because a hallucinated call often fails to compile before a test ever runs, and the tests catch what the compiler misses. You cannot read every line of 223 PRs. A green build you trust is what makes the volume reviewable.

Ingredient three, split the work and run agents in parallel

Parallelism is the whole point, so the work has to be parallel-ready. The 40 near-identical panels are perfect to fan out, because AI is strong at cloning something and adapting it to something similar. I give one task to one agent, small scope and a clear goal, each on its own branch or worktree so they do not collide. Splitting the work cleanly is an architecture problem before it is a prompting one, and getting it wrong is where the merge conflicts come from.

A fleet is not one chat window. I drove BootUI mostly through the GitHub Copilot coding agent app, because it lets many agents run live on one machine, and I comfortably keep 10 going at once. More than 10 gets complicated for a human to hold in their head, and the laptop starts to slow down. The mobile app runs agents in Docker containers, so I can keep them moving when I am away from my desk. The CLI and the IDE plugin suit smaller fan-out, and all three share the same back end, the Copilot SDK, which is open source, so you can build your own interface on top of it if you want. The rule stays simple. Don’t babysit one agent. Run 10.

Ingredient four, let a few deep plans run overnight

The day is the engine and the night is a bonus shift. Before I log off I hand a few deep, long-running plans to autonomous agents, the big jobs I don’t want to sit and watch. On BootUI those were things like wiring and testing the security filter chains across 37 rules, pushing coverage further across the suites, and reshaping the Actuator data layer in a larger refactor. Each is a long run that lands a few PRs by morning.

For the genuinely tricky work I have the agent criticize its own output with two or three other models. One writes the code, the others analyze and vote on the fix. That is also how I catch hallucinations, because the model that invented something usually stands alone against the others, and it shows up plainly in the logs. A good overnight prompt means everything is done when I come back. A weak one means I have nothing, which is why prompt-writing is worth practicing. The night is a bonus on top of the day, not a replacement for the driving.

Ingredient five, merge the results over coffee

The morning starts over coffee. I triage the few overnight PRs, merge the green ones fast, drop or re-task whatever didn’t land, cherry-pick the good parts of the rest, and then start driving the day’s fleet. When I review a PR I look at three things that stay in sync because they are generated together, the code, its tests, and its documentation. Green tests tell me the code and the tests agree, and I squash and merge.

Review is the real bottleneck. It is not the typing anymore. It is the merging. I don’t merge blind and I don’t merge like crazy, because the day you wave through a large diff without reading it is the day something wrong lands in main. So make review a fast, trusted ritual you run all day, not a line-by-line slog at the end.

Ingredient six, pick the right model for the task

Most of the work went to a workhorse, GPT-5.5 on extra-high reasoning, with around 90 percent of its tokens served from cache. For the tricky parts I brought in Claude Opus 4.8 and Gemini 3.1 Pro, three strong models cross-checking each other to find the best fix. For simple, mechanical tasks a smaller model or Auto mode is fast and cheap, and Auto tends to pick better than I would while carrying its own token rebate.

The economics are driven by the cache more than by the model price. The whole build ran on about 2.7 billion tokens for roughly 2,000 dollars, with around 95 percent served from cache, so most of the run costs a tenth of the headline price. That changes the obvious advice. Swapping to a cheaper model for one small task often bursts the cache and costs more than staying on the model whose context is already warm. If I have a small task, I spawn a separate small agent for it rather than switching the big agent’s model. And I avoid editing AGENTS.md mid-run, because it lives at the top of the cache and one change invalidates it, so I update it at the end of a run instead. Knowing how the cache works matters more than picking the cheapest model.

Why the multiplier was so large

I want to be honest about why BootUI hit a multiplier this large, because not every project will. Three things made this codebase unusually well-suited to agents. First, massive repetition, since around 40 structurally similar panels are cheap for an agent to clone and are the most expensive part by hand. Second, a broad-but-shallow shape, many Spring subsystems each shallow on its own, where the human cost is mostly looking things up, which is exactly what the model already absorbed in training. Third, strong guardrails, because multi-module CI, CodeQL, end-to-end tests and explicit instructions let me accept high throughput safely.

The net effect was a 6.5 to 8.5 month solo effort compressed into 11 days and about two weeks of human attention. The biggest leverage is on large-surface, pattern-heavy, well-tested code, not on hard algorithms. A project that is mostly novel logic with thin tests will not see these numbers, and I would not claim otherwise.

Where this goes wrong

The failure modes are consistent, and most of them trace back to the operator. Scope creep is the first. Tell an agent to build the whole thing and it wanders, so keep one tight goal per task. Missing tests are the second. Without a harness you cannot trust the output and you cannot merge at volume, so the harness comes first and a green build gates every merge.

Giant PRs are the third. A 2,000-line PR is impossible to review well, and review fatigue tempts you to wave it through, so keep the chunks small and reviewable and pace yourself. The wrong model is the fourth. A weak model fails the hard tasks, a strong one is slow and costly on the easy ones, so match the model to the job. When people tell me AI-generated code is not good, the cause is usually one of these. Thin specs, weak tests, oversized PRs, or the wrong model. If the agent generates bad code, that is usually your fault, not the model’s. Most failed runs are a briefing problem.

One page to screenshot

If you take one thing away, take this.

A new Spring Boot console fell out of it

The recipe was the point, but it left behind a real, open-source product. BootUI gives any Spring Boot app an embedded console with live health and metrics, a security advisor that walks your filter chains, an OSV scan of your dependencies, a Flyway and Liquibase data view with a Hibernate advisor, tracing through Micrometer and OTLP, and an architecture view backed by ArchUnit and GraalVM reachability. Add one starter to your pom.xml, run locally, and open the console. The code and the full docs are open.

Write the specs, give them tests, run them wide, and ship what used to take months. Now go build.

This deep dive is adapted from Julien’s Deep Engineering workshop, From Coder to Manager of Agents. Here the slides from the talk.

Capacitor records the session behind the work: what agents tried, what teammates rejected, what finally passed, and why it mattered. This gives you vendor-neutrality to move across multiple coding agents, multiplayer - collaboration on coding sessions, faster PR reviews, evals on code & more.

Sign up for free

✍️ From the editor’s desk,

Welcome to the 55th issue of Deep Engineering!

OpenAI made its GPT-5.6 family generally available on July 9 across ChatGPT, Codex, and the API. The capability that matters here is a new ultra mode that runs concurrent subagents and synthesizes their work in a single request. This release also makes cached context far cheaper than fresh input, since cache reads keep the 90% cached-input discount. So running a fleet of agents in parallel, and paying almost nothing to reuse their context, has moved from a hand-built trick into something the platform now does by default.

That shift is why this issue matters now. When the tooling makes parallel agents this easy, the differentiator stops being model access and becomes the discipline around it, the specifications, the tests, and the review that decide whether all that speed produces software you can actually ship. Last week Julien Dubois led a workshop for us, From Coder to Manager of Agents, where he showed how he built a real open-source product this way, measured from git history rather than memory, which makes it far more useful than another benchmark thread.

Julien is Principal Manager for Developer Relations at Microsoft and GitHub, and the creator of JHipster.

Let’s get started.

Featured: Loop Engineering for AI Agents

Stop wasting tokens on endless retries. This four-hour, hands-on workshop takes you past one-off prompting to reliable agent loops that plan, execute, verify, and stop safely, built with Claude Code, Spec-Driven Development, and MCP, with the verification gates and state management that keep autonomous agents trustworthy.

Deep Engineering readers save 40% with code DEEPENG40.

Reserve your seat

Practitioner's View by Julien Dubois

223 pull requests in 11 days for the project I never had time to build

You can read this practical deep dive based on his workshop. And here are the slides from the talk.

I always wanted a real developer UI for Spring Boot. Every Spring app is a black box in development. Actuator gives you raw JSON, not a console, so what I wanted was health, metrics, security and tracing in one embedded UI.

I’d shipped a slice of this in JHipster years ago, but only for generated apps. A real console for any Spring Boot app sat on my wishlist for years. The catch is that it’s massive. Around 40 panels, each one a backend plus a frontend plus its own tests, deeply integrated across a dozen JVM subsystems. By hand, one experienced developer would need somewhere between six and a half and eight and a half months. Too big to justify, until I stopped writing the code myself.

Then I did it in 11 days. Not by typing faster. I didn’t even open my IDE. I managed a fleet of AI agents while I architected, reviewed and steered. The agents did the scaffolding, the panels and the tests. I did the judgement.

What I built, BootUI

BootUI is a production-grade Spring Boot 4 starter that adds an embedded, local-only developer console to your app. You add one starter to your pom.xml, run locally, and open the console.

It is a five-module Maven build on Spring Boot 4 and Java 17, integrated across Actuator, Spring Security, Flyway and Liquibase, Hibernate, Micrometer with OTLP, GraalVM, OSV scanning and ArchUnit, published to Maven Central with full CI. The frontend is an embedded Vue 3 single-page app, packaged inside the starter. Each panel is endpoints plus a view plus tests.

There are around 40 feature panels. Health and metrics, a security advisor, a vulnerabilities view, tracing, and more. Those panels look a lot alike, which made them the biggest source of structural repetition in the codebase. That detail matters, and I will come back to why it made the whole thing work.

Eleven days, and how I know the numbers are real

The figures here are derived from git history, PR metadata and code metrics, not from time-tracking logs. Through the tagged 1.0.0 release there were around 264 commits on main and about 223 squash-merged pull requests, landing at roughly 20 a day across 11 days. That came to about 83k total tracked source lines, close to 50k of them Java across around 461 files and 81 test classes, plus 52 Vue components, 40 panels, and 35 end-to-end specs. In all, about 116 test suites.

A cadence of 20 PRs a day with an AI agent co-authoring commits is impossible to reach by hand. The agent did the typing. I dispatched many asynchronous tasks in parallel. The evidence lines up with that. The commit clock runs from around five in the morning to midnight most days, which is consistent with parallel async tasks rather than continuous typing. “Copilot” shows up as a named commit and PR author on about 44 commits, and the repo ships a copilot-instructions.md with per-panel conventions, so the workflow was explicitly agent-oriented. Even on release day I was landing a docs site, a scanner dashboard, token charts and a Hikari fix, which is itself several days of solo work.

Where my own time went was writing prompts, reviewing and merging those 223 PRs, and resolving CI failures around Spring Boot 4, Flyway 11 and OTLP. Review and orchestrate, not write every line.

The honest by-hand comparison is the part people argue about, so I ran it carefully. One experienced Spring Boot and Vue developer, with no AI codegen, would need 6.5 to 8.5 months for the same polished 1.0.0, roughly 1,100 to 1,450 hours of hands-on effort, and the 40 panels are the dominant cost. A COCOMO organic estimate on 50k lines yields more than 100 person-months, which is too high because much of the code is repetitive scaffolding, so a domain-expert solo figure of about 7.5 months is the defensible middle ground. My actual human effort on BootUI was 80 to 110 hours, about two intense solo weeks. That is a calendar speed-up of roughly 17 to 23 times, and a human-hours speed-up of roughly 12 to 17 times.

I was the manager, not the developer

I didn’t open my IDE. I wasn’t the developer. I was the manager. Many agents ran in parallel on the panels, the integrations, the frontend, and the CI and docs, and my job was to brief, review and merge.

Your throughput is not your keyboard. It is your briefs. What blocks a single developer is typing, and what blocks a single agent is you waiting for it to finish. Running many at once removes both, because they come back one after another and there is always something to review. You don’t type faster this way. You ship what used to take months.

A day in the loop

The rhythm that produced 20 merged PRs a day is a simple daily loop, and the day is the engine. I am hands-on all day. I spec, launch, review, merge and re-task, live, and most of the day’s PRs land right there.

The evening is a hand-off. Before I step away I queue a few deep, long-running plans. The night is a bonus, not the engine. A handful of deep autonomous runs finish by morning, which is the minority of the work. Repeat that for about 11 days and you reach a tagged 1.0.0. Six ingredients make the loop work.

Ingredient one, write the specifications

The first thing I write is never code. It is the house rules. An AGENTS.md, which GitHub Copilot reads as copilot-instructions.md, and every agent reads it first. It sets the conventions once, the stack, the build, the tests and the style, with per-panel conventions so 40 panels come out consistent. Mine runs around 700 lines. That is about the right size, because a file that is too big pollutes the context, and it also costs tokens and money on every run.

The high-leverage move is writing the known failures into that file once. Most models were trained on Spring Boot 3, so they struggle with the reworked Jackson API in Spring Boot 4, and 10 agents trained on the same data hit the same wall at the same time. Rather than fix that by hand in each one, I write the fix down once. The same goes for operational traps, like giving every agent its own local Maven repository so parallel installs do not corrupt a shared one.

On top of the house rules I write one spec per task. What to build, where, what “done” looks like, and the acceptance test the agent must make pass. Small, self-contained, no hidden dependencies. The spec is the product now. The better the brief, the less you babysit.

Ingredient two, build the test harness

No agent runs without a test harness. One command compiles and runs the unit and end-to-end tests, and comes back green or red, so an agent can verify its own work before it opens a PR. No tests means you cannot trust the output.

CI is the trust layer. BootUI has about 116 test suites, 81 in Java and 35 in Playwright, and CodeQL plus the end-to-end tests gate every PR to main. The loop does the work. An agent writes code, pushes it, and GitHub Actions sends back the results. If they are red the agent reads the failure, fixes it, commits and pushes again, and the checks run once more. Dependabot keeps the dependencies current alongside all of this. A compiled, typed language like Java helps too, because a hallucinated call often fails to compile before a test ever runs, and the tests catch what the compiler misses. You cannot read every line of 223 PRs. A green build you trust is what makes the volume reviewable.

Continue reading

The rest of the deep dive covers the other four ingredients, the test harness and CI as the trust layer, running agents in parallel across worktrees, the overnight critic-and-vote runs, merging as the real bottleneck, and the cache math behind the roughly two thousand dollar bill, along with where the multiplier will not repeat.

Industry Perspective

🛠️ Tool of the Week

Composio’s Agent Orchestrator is an open-source take on exactly that job. It helps developers manage fleets of coding agents for parallel work, giving each one an isolated workspace and supervising them from one place.

• Runs each agent in its own git worktree, so parallel work does not clobber shared files

• Feeds CI failures, review comments, and merge conflicts back to the right agent automatically

• Works with the terminal agents teams already use, including Claude Code, Codex, Cursor, and opencode

• MIT licensed and self-hosted, run locally rather than as a hosted service

Composio’s Agent Orchestrator

📎 Tech Briefs

• Claude Code changelog - Agents must now confirm before entering a git worktree outside the project’s .claude/worktrees directory, and a new /doctor check flags checked-in CLAUDE.md content the model can derive from the codebase.

• GitHub Copilot CLI changelog - Plan mode can no longer run tools that modify the workspace, and the default maximum sub-agent nesting depth drops from six to four to limit runaway recursive delegation.

• Copilot for JetBrains expands BYOK capabilities - Adds local agent sandboxing, a Claude agent provider for custom agents and skills, and OpenAI-compatible custom endpoints, all in public preview.

• Copilot in Visual Studio, June update - Visual Studio now checks each MCP server’s configuration and asset fingerprint against a trusted baseline at startup, and the C++ modernization agent reached general availability.

• Claude Code gateway for teams - Anthropic shipped a self-hosted gateway for Claude Code, a stateless container that adds corporate SSO login, centrally enforced policy, role-based access, and per-user cost attribution across a team.

That’s all for today. Thank you for reading this issue of Deep Engineering.

We’ll be back next week with more expert-led content.

Keep building,

Saqib Jan

Editor-in-Chief, Deep Engineering

If your company wants to reach senior developers, software engineers, and technical decision-makers, speak to us about partnering with Deep Engineering.

Over roughly a quarter in 2026, several of the largest engineering organizations in the world took down the dashboards they had built to prove their AI investments were working. Amazon shut down KiroRank, the internal leaderboard that ranked developers by how many tokens they consumed on its Kiro platform. Meta also dismantled a near-identical board, Claudenomics, that tracked token usage among its heaviest AI users. The reason was the same everywhere. Once usage became the number that mattered, engineers found ways to run up usage, and the costs arrived long before the value did.

Amazon’s own senior vice president, Dave Treadwell, told staff to stop using AI for its own sake after employees gamed the leaderboard by padding token usage, the practice the industry had taken to calling “tokenmaxxing.” Uber also said it could find no clear link between what it was spending on AI and what it was shipping, while Microsoft cancelled a division’s coding-assistant licences over cost. So, within a single quarter, the industry ran the experiment, watched it fail, and started hunting for a better question to ask.

That question is the subject of this issue, with insights from Austin Parker, Director of AI Strategy at Honeycomb and a co-founder of OpenTelemetry; Tom Howe, Director of Solutions Engineering at Hydrolix; Jayeeta Putatunda, Director of the AI Center of Excellence at Fitch Ratings; Satyam Dhar, Staff Software Engineer at Galileo and a former engineering leader at Amazon and Adobe; and Michael J.J. Tiffany, co-founder and CEO of Fulcra Dynamics.

Let’s get started.

Social engineering is about manipulating people’s emotions. Identify the susceptibilities that hackers use to exploit people.

This NINJIO Insights Report dives into the key emotional susceptibilities that make social engineering work and offers concrete steps that your security team can take to equip your workforce to resist cyberattacks.

Download the Guide

Special issue — July 2026

A token was never a unit of intelligence

“You can use tokens to help build solutions, but at the end of the day these are judgment calls.” — Austin Parker, Director of AI Strategy at Honeycomb and a co-founder of OpenTelemetry

Give a modern coding model the right harness and a clear goal and it will do almost anything you ask, up to and including rewriting an entire codebase in another language and passing the tests. That capability is real, and it is seductive. It makes it easy to believe that all you need is something to keep the tokens flowing and any problem becomes tractable. Parker has observed his own engineering teams test that belief, generating a hundred variations of the same screen or twenty versions of a flow, and the lesson came back the same each time.

The variations did not move the needle. You can ask for infinite variations of a solution, and they all arrive shaped like your original conception of the problem, because that conception is the one human input the model never questions. The code and the screen and the workflow were never the things that create value for the people using them. Value lives one level up, in whether the design serves those people, whether the shape of an API fits what its consumers actually need, whether the underlying primitives have any fitness for the purpose they are being asked to serve.

Those are judgment calls, and no volume of tokens answers them. Parker sees the same pattern when he talks to other leaders, plenty of appetite for here is a problem, generate all the variations, and far less time asking whether the problem was framed correctly in the first place. His conclusion is blunt. Judgment is the finite resource now, and teams that forget it end up with more output and less of what the output was supposed to buy them.

Leaderboards reward motion, not judgment

“The paradigm in some organizations will, undoubtedly, shift from ‘high token usage equals good employees’ to ‘high token usage equals expensive employees.’” — Tom Howe, Director of Solutions Engineering at Hydrolix

The collapse of the token leaderboards surprised no one who has watched a measure become a target. Hand a famously clever group of engineers a resource and grade them on how much of it they use, and they will use it. The moment a measure becomes a metric it stops being a good measure, a version of Goodhart’s law that Amazon’s own leadership named out loud when it wound the program down. The surprise was not that the leaderboards were gamed. It was that so few people in the decision chain saw it coming.

Howe has seen the mechanism play out from the inside, and he traces it to a subtle slippage. A company rolls out what it calls an AI usage dashboard, meant as an objective measure of how much the tooling is benefiting the business, and it quietly devolves into a leaderboard that measures how much each person is using the tool. Once employees read it as a judgment of how they work rather than of whether the tool earns its keep, the whole exercise changes character.

From there the responses split three ways, and each one erodes the thing the dashboard was supposed to protect. Some engineers game it, firing tools at low-value tasks to boost their stats, which breeds suspicion that others are gaming it too. Some avoid the tool entirely rather than expose their habits to uncontextualized data. And in the middle sits the casualty Howe cares about most, the trust that a healthy organization runs on, undermined the moment a team cannot tell how or why it is being measured.

The cost frame makes it worse. Howe describes enterprises treating tokens as close-to-free productivity, funny money, right up until the real bills and the real gains come into focus. When they do, the incentive can flip hard, from prizing heavy usage to flagging it as expense, and engineers start hedging, wary of being branded token wasters and wary of binding their workflows so tightly to a tool that a future rationing decision leaves them stranded. He points to a recent, unusually capable model that carries a known high future cost, where engineers are trying to use its power now without becoming dependent on it, as exactly the kind of bind these metrics create.

None of this means the tools do not work. Howe shares that Hydrolix has seen a revolutionary impact from how it uses them, in building products and in weaving them into products, in ways that would have been unimaginable a few years ago. The challenge is balancing productivity, cost, and trust, and learning to measure the impact through experience rather than through a scoreboard. He tells one story that captures the slope precisely, an engineer glancing at the usage board and joking that not cracking the top fifty was rookie numbers. It was a joke. But it was also a warning about how fast an innocuous metric turns into a competition.

Featured - ARC 2026: Software Architecture in the Age of AI

AI is reshaping software architecture, putting new demands on scalability, governance, reliability, and observability. ARC 2026 brings together architects, CTOs, and AI practitioners for keynotes, panels, and workshops on agentic system design, modernizing enterprise apps for AI, and building governable, observable AI systems.

🗓️ 25 to 26 July, 10:30 am ET

Use code DEEPENG50 for 50% off the early bird price.

Reserve your spot

Skills that matter never show up on a dashboard

Parker calls prompting, building internal alignment, and holding a complex system in your head illegible activities, and they are close cousins of the glue work that keeps organizations moving. The people who know who needs to be in a decision chain, who can hear a customer complaint and route it to the one team that can fix it, who understand that the thing frustrating you in one corner of a product is caused by something three systems away, are the reason a business can actually execute. The organizations that are genuinely good at this tend to be the ones that, lacking a clean way to measure those people, are at least careful not to penalize them.

A token metric makes that harder. It rewards the visible act of consuming AI over the invisible work of knowing what is worth building, and the invisible work is where most of the value hides. That work shows up in promo packets and career ladders because those are written by people who understand it. It does not show up in a number you can compare quarter to quarter, because, in Parker’s words, it involves “very human things like relationships and time in the system rather than timing the system.” Measure the activity and you slowly stop rewarding the thing that made the activity worth anything.

AI widens the gap a team already has

This is the reframing every leader watching one team pull ahead should steal. AI rarely conjures capability from nothing. It accentuates differences that already exist. When a team surges, the real story is almost never that raw intelligence solved a problem they could not solve before. It is that AI patched a hole in the team’s composition, or handed them leverage on something organizational rather than technical.

Parker’s example is the part of shipping that engineers tend to undervalue, making the case to leadership. That is usually a data-analysis argument, projections and forecasts and a slog through support tickets and sentiment data, and it happens to be something AI is very good at. A motivated engineer or designer or support person can now do the analysis a product manager used to own, cover the weak spots in their own toolkit, and come back to the org with here is what I need in order to go build the thing I am convinced is right. So the question to ask the team that pulled ahead is not which tool they used. In Parker’s words, “I never want to ask what AI has let you do. The question is what is AI giving you the excuse to do that you could not have done otherwise.”

That does not extend into a ban on measuring code output. Parker’s line is that you should measure it and refuse to make it a metric. Watching the balance of someone’s pull requests opened against reviewed tells you something real about how they work, and grading their performance on that same count is where it curdles. AI can even help here, letting a team ask which work is generating on-call burden, which is delivering customer value, and who is quietly doing the refactoring that lifts everything built on top of it.

Point the budget inward, at developer experience

“If I had 90 days of AI budget to improve developer experience, I wouldn’t spend it evaluating another model. I’d spend it reducing engineering variation.” — Jayeeta Putatunda, Director of the AI Center of Excellence, Fitch Ratings

For a Staff or Principal engineer who owns a platform, Parker’s first ninety days are not about output at all. They are about earning the ability to ask real questions of your own systems, and most organizations cannot, because observability was underinvested for years while other goals took priority. The mechanical work of fixing that, moving unstructured logging to structured events and spans, migrating to OpenTelemetry, is exactly the kind of task that is fungible from an intelligence point of view, which makes it a near-perfect fit for current coding models. It also comes with a natural verification step, feed the before and after to another model and let it judge whether the telemetry got better, and let it update the docs as it goes.

Putatunda took the same budget in a different direction, and her experience widens the point. Once her organization distributed AI broadly, it became clear that model capability was never the bottleneck, because every team had the same models. What differed was how teams solved the same problems, each with its own coding conventions, documentation norms, migration methods, and review expectations. So her first priority was not a better model. It was making the organization’s engineering knowledge reusable.

Her team translated coding norms, implementation templates, documentation standards, review criteria, and migration playbooks into reusable skills that work from any agentic environment, whether an engineer reaches for Claude Code, GitHub Copilot, or something else. That let them clear backlogs that had always lost the prioritization fight against feature work. The migration of a legacy Selenium test suite to Playwright, a slow and redundant manual slog across many teams, got done by pairing reusable migration skills with the organization’s own standards and validation steps, so the generated tests matched both the suite and the reviewers’ expectations. The same approach carried into repository documentation, repetitive refactoring, and framework upgrades, the modernization work where consistency matters more than originality.

The deeper reason it worked speaks straight to the review bottleneck that AI creates. Models generate code faster than teams can review it, and when every change arrives in a different shape, every pull request costs the reviewer more attention. Standardizing the patterns let reviewers focus on correctness and business logic instead of style and structure. Putatunda does not measure any of this in tokens or lines generated. She measures whether projects moved faster, whether patterns got reused across teams instead of reinvented, and whether reviewers stopped correcting predictable issues, which is another way of saying she measures the same thing Parker does, whether the team can now do work it could not do reliably before.

Spend on leverage, not on tokens

“Just as we don’t judge an IDE by how many keystrokes it saves, we shouldn’t judge AI by how many tokens it consumed.” — Satyam Dhar, Staff Software Engineer at Galileo and a former engineering leader at Amazon and Adobe

Local inference and model routing are making cost-per-token murkier by the month, and Parker is honest that he does not have a clean formula to replace it. What he has is better than a formula. It is an example. For most of Honeycomb’s life, dark mode was the feature customers asked for and never got, enough of a running joke that it has its own face on the twenty-sided die new hires receive. Then, in about six months, the company shipped it, mostly because of AI.

The caveat matters as much as the result. It was not only AI. Getting there took years of groundwork, a design system and an accessible component library built to support the feature. What AI closed was the mechanical gap that always stalled the project, migrating the existing app onto the new components. The way they did it is the real lesson. Instead of one team grinding through the migration, they shipped prompts and skills so anyone could run it, and pulled in an engineering director who had built a screen seven years earlier to load the skill and update his own page. Yes, it burned a lot of tokens, and a small focused team could have optimized that count down. The accounting that matters is a five-year customer request finally closed and a company full of people who learned how to use AI.

Dhar puts a name to the mistake underneath the token frame. Too many teams treat AI like another cloud bill, counting tokens because tokens are easy to count, then assuming fewer tokens means better spending. In his experience running large-scale AI systems, some of the most valuable investments were the ones token accounting punished, better model routing, evaluation pipelines, prompt versioning, and experimentation infrastructure, all of which raised spend on paper while sharply lowering the cost of iteration.

That lower cost of iteration is where Dhar locates the actual return. When engineers can compare models safely and product teams can experiment without fear of quietly degrading production, a team can validate five ideas in the time it used to take to validate one, and the business benefit dwarfs the incremental compute. He also refuses to ignore the second-order effect that never reaches a finance dashboard, the way good infrastructure changes behavior, so engineers start proposing projects that once felt too expensive or too repetitive to attempt and product conversations get more ambitious because implementation cost no longer dominates every decision. His test is a single question, whether the system helped the team build something it otherwise would not have built.

Count only the work that clears a bottleneck

“Only the work that clears a bottleneck is progress. The rest are lightbulb filaments that didn’t work.” — Michael J.J. Tiffany, co-founder and CEO of Fulcra Dynamics.

Tiffany offers the closest thing in this issue to a replacement unit of account, and he arrived at it by correcting himself. Token use, he argues, is a low-level metric like CPU utilization, important but meaningless in aggregate. His better frame is cost per bottleneck relieved. He started somewhere looser, counting cost per accepted unit of work such as a merged pull request or a user-visible improvement, then learned the hard way to count only the subset of work that actually clears a bottleneck.

The distinction changes what you tolerate. What you buy when you pay for AI, in Tiffany’s account, is more attempts, faster loops, more surface area explored, fewer blocked humans, and occasionally, almost at random, a capability that was not economically or psychologically feasible before. Most of that is not progress, and that is fine. The failed experiments and even the shipped features that did not matter are the filaments that did not light. You let them happen, and you measure success only by the bottlenecks that came loose, which keeps the number honest in a way a token count never can.

Judgment is the constraint for the next two years

The token leaderboards are already gone, and the mandate that produced them is being quietly rewritten across the industry toward outcomes that are harder to game. The deeper lesson is the one Parker keeps returning to. Every layer an agent touches was built on the assumption that the consumer brings context and judgment, and that assumption is exactly what breaks when the consumer, or the incentive, is optimizing a number.

The voices in this issue disagree on the right replacement, and the disagreement is the useful part. Howe would protect trust before any metric. Putatunda would standardize the engineering practice underneath the tooling. Dhar would ask whether the team can now build what it could not before. Tiffany would count only the bottlenecks that came loose. What they share is a refusal to accept the token as the unit of value, and a conviction that the scarce resource is the judgment to decide what is worth building in the first place. The limiting factors on agentic systems are no longer model capability alone. They are judgment, integration, and the operational discipline to tell useful work apart from motion, and that is where the most consequential engineering decisions of the next two years will be made.

Thank you for reading this special issue of Deep Engineering on why judgment, not tokens, creates value.

We’ll be back on Thursday with more expert-led content, and next month, on the first Tuesday of August, with another special issue.

Keep building,

Saqib Jan

Editor-in-Chief, Deep Engineering

If your company wants to reach senior developers, software engineers, and technical decision-makers, speak to us about partnering with Deep Engineering.

Austin Parker has worked in observability for about a decade, co-founded OpenTelemetry, and now leads AI strategy at Honeycomb. He came on to make an uncomfortable case for anyone running an AI mandate, that a token is not a fungible unit of intelligence and judgment is the resource that actually got scarce.

You can read or watch the full conversation here:

This session was recorded live as part of the Deep Engineering Interview Series. The transcript below has been lightly edited for clarity and readability.

Q. Tell us a little about yourself, what you do at Honeycomb, and what pushed you to write about tokenmaxxing.

I have been in observability for about a decade, first as a maintainer on OpenTracing, and then I helped co-found OpenTelemetry when we merged it with OpenCensus years ago. I have worn a lot of hats across that open source work. About two years ago, when ChatGPT released, I got really interested in AI and in how we could use it to help people make sense of systems at scale. Over the past year I have been building that capability out at Honeycomb, working on our agents and our machine learning and AI stack, and getting involved in OpenTelemetry’s GenAI semantic conventions to make sure OpenTelemetry is as useful to AI as it is to humans.

Q. You argue a token is not a fungible unit of intelligence. Walk us through one engineering decision where treating tokens as fungible led a team somewhere they regretted.

You can find a lot of examples of that just reading the news today. Given the right goals and the right harness around a model, whether that is a frontier model or an open weights one, you can get it to do almost anything. The maintainer of Bun, who I believe now works at Anthropic, famously did a big port where the instruction was basically rewrite the whole thing in Rust, and it did it, and it passed all the tests, which is impressive.

When we read things like that it becomes easy to imagine that all I need is something that keeps the tokens flowing and I can do anything I want. That might be true, but it does not make it a good idea. Internally I have seen our engineering teams run experiments where they generate a hundred different prototypes of the same screen, or fifteen or twenty variations of a flow. What I have noticed is that most of them do not move the needle in the way you would hope, because you can ask for infinite variations on a problem. You can ask for the same program in ten different languages, or rewrite a function a hundred thousand times.

But the code and the screen and the workflow are not the things that provide value to your users. The value is in the questions one level up, about the design of the system, about how people interact with it, about whether the shape of an API really serves the people integrating with it, and whether the fundamental primitives have fitness for what people want out of them. Those are not questions that tokens answer. You can use tokens to help build solutions, but at the end of the day these are judgment calls. Those hundred variations all came out looking about the same, because they all started from one very human conception of the problem, so all of the output ends up shaped like my initial conception of it. I see this repeated when I talk to other leaders. There is a lot of here is a problem, make all these variations, and far less time asking whether we are defining the problem correctly, solving the right customer pain point, and putting the right inputs into our loops. That is why I argue that judgment is the finite resource now.

Q. You call prompting, building internal alignment, and holding a complex system in your head illegible activities. When a leader starts measuring token usage, what specifically breaks in how those skills get rewarded?

It is tough to say sometimes, because at really large organizations that kind of value is already not rewarded well. A lot has been written about the importance of glue work, about people whose value to the org is not how many PRs they open or how many decks they build, but knowing who needs to be in a decision chain, or being able to hear a customer request and go find the right team to make it happen. Often you or I might be frustrated with a piece of software because of one thing that shows up in a certain part of the system, when the actual reason it works that way lives somewhere else entirely. The people who can navigate that ownership map internally end up being really important to the ability of the business to execute and serve customers. The companies that are genuinely good at that tend to be the ones that, even without a clean way to measure those people, are good at not penalizing them.

Then a top down mandate arrives that says we need to use more AI, and the easiest objective measure of that is how many tokens you use. So the move becomes let us incentivize you to use more tokens and then productivity magically appears. The connective work that makes someone valuable shows up in promo packets and career ladders, but it does not show up in something you can measure quarter to quarter or day to day, because it involves very human things like relationships and time in the system rather than timing the system. The tokenmaxxing craze, 2026 to 2026, RIP, is a good example of why that value matters and why you cannot replace it by throwing more AI at the problem.

Q. The token leaderboards at companies like Amazon rose and fell inside three months. What did the people running them learn, and why do you think the correction came so fast?

What they learned is that maybe they should not do that again. The moment you turn any measure into a metric it stops being a good measure. Give a legendarily clever bunch of engineers a resource and tell them they are graded on how much of it they use, and they will find a way to use it. There is a famous Dilbert strip where they announce a bonus for every bug you fix, and Wally says he is going to write himself a new minivan, because you said every bug that gets identified and fixed, you did not say anything about not writing the bugs in the first place. Same thing with token leaderboards. What is strange to me is not that it happened, it is that it was not immediately obvious to the people in the decision chain that it would.

Calling them pointy haired bosses who just do not get it is a lazy answer though, and I do not think it is true. Yes, plenty of those experiments led to utter failure, and broadly it was a bad idea I would never have recommended. But I have noticed people, even at Honeycomb, who get nervous about spending company resources on an idea they think AI might be good for. A moment of use as much as you want, go wild, frees people who would otherwise be overly conservative, and some of them find something they never would have found under the old constraint. So some of this is taking the good with the bad, and some of it is making sure your employees are aligned with the overall goal. Those extremes are why it moved in and out of favor so fast, because it only takes a few people burning hundreds of thousands of dollars. Although Amazon is probably good for it.

Q. You say AI accentuates differences that already exist rather than creating new capability. If a leader sees one team pull ahead with agents, what is the actual question they should be asking that team?

This is the thing I feel does not get asked enough. I never want to ask what AI has let you do. The question is what is AI giving you the excuse to do that you could not have done otherwise. In every big phase shift you find individuals and teams who use the bigger meta narrative as cover to solve a different problem. With AI, it is rarely that intelligence alone solved something we could not solve before. Usually it is solving an organizational problem or a leverage problem.

Think about what it takes to ship a feature inside any organization. A big part of it is making the case to the rest of the org, especially leadership, about the expected return, and that is often a data analysis argument. It means sitting down with a pile of numbers, doing a lot of advanced Excel, building projections and forecasts, tearing through user data and sentiment data and support tickets. It turns out AI is really good at that. Give it access to your support tickets and your customer experience data and your NPS scores, spend time with it, and it helps you surface things you knew but could not back up, or genuinely new insights. Traditionally that was a product manager or business analyst function, and there is a wide range of skill there. Now a sufficiently motivated engineer or designer or support person or sales engineer can do that work, use AI to cover the parts they are not as good at, and turn around and tell the org here is what you need from me so I can go do the thing I have conviction is right.

That is what I see in the teams becoming very effective with AI. It is not simply that we can write more code now, although that helps and you can review it faster. It is that we can patch whatever hole existed in our skills or our team composition. You see it outside engineering too, and at small companies, where people who are strong at design but never programmed can now program, and people who are weak at persuasive writing can write better. We are filling the gaps in our own abilities, and that is the biggest difference between the teams getting real value out of AI and the teams that are coasting.

Q. Should companies refuse to measure engineers by how much code their agents write, or even how many bugs they patch?

I think that is stupid. There are so many other ways to gauge engineering efficiency. A better way to think about it is that you should measure it, but it should not be a metric. It is useful. One thing we do internally is look at how many PRs someone is reviewing, how many they are opening, and the balance of their time in the codebase. That is a useful signal, because it shows you someone spending a lot of time reviewing versus someone spending a lot of time writing, and maybe you need to think about that balance, or maybe someone is adding a lot of new things but not spending enough time understanding what is going in. Turning it into something you grade performance on is where it gets twisted, and a lot of places still treat PRs opened as an important KPI.

AI actually lets you do this analysis more holistically. Instead of stopping at how many PRs someone did, you can feed a lot of it into a model and ask real questions. Of the work you are doing, how much is causing on call burden, how much is delivering specific customer value, and are we missing the people doing important but less shiny work like refactoring. Attaching those contributions to specific outcomes is hard, like a big refactor of the component library that improved consistency and lifted the interstitial user experience survey scores because a button is now consistent across the app. Those things are hard not because they are hard to measure, but because there is so much data to go through. AI makes that analysis much easier when it is driven by high quality telemetry and instrumentation, and when the people using it have good judgment.

Q. Your advice is to turn token budgets inward toward developer experience and observability. What does that look like in practice for a Staff or Principal engineer who owns a platform, in the first ninety days?

The core thing you want is to get to that analysis loop, which means asking whether you even have the data, whether it is possible for you to ask those questions at all. Most of the time it is not, because as an industry we do not value observability as highly as we should, though we are getting there. I still talk to a lot of people who have their emotional support logs, because that is just how it has always been done. The historical argument against improving observability has always been that we do not have the time, we do not have the people, we have all these other goals.

This is exactly the mechanical, boring work that is fungible from an intelligence perspective. Taking your existing logging and making it structured, or moving to OpenTelemetry, is perfectly aligned with what current AI coding models are good at, and with the verification side too, because it comes down to a judgment call about whether the telemetry after the migration is as good by some heuristic as what you had before. You can prompt the verification step directly. Strip out the unstructured logging, add structured events and spans and good metrics, then feed the output into another model to judge whether it is better, worse, or the same, and let it rip. It can update the docs and the comments while it goes. That is massive leverage for an org that takes the diffuse do whatever with AI money pool and points it at something specific like developer experience and observability. It lines both things up, because more people learn to build better harnesses and prompts and loops, and you get a measurable outcome, which is that now you can answer these questions and understand your systems and run the longer horizon analysis you never had the data for. Job one is getting to the point where you can ask these questions.

Q. Local inference and model routing make token accounting even murkier. As that lands, what mental model should engineering leaders use for AI spend instead of cost per token?

I honestly wish I had a better answer. To use an example from Honeycomb, we have been doing agentic engineering and going through this transformation like everyone else for about a year and a half, and the results have been pretty dramatic. In one quarter we opened as many PRs as we had in the entire year up to that point, because people are leveraging AI more and a fairly small engineering team can punch above its weight on code writing and generation.

One example of that is dark mode. For as long as I have been here, dark mode was one of the most consistent customer requests, and we are a developer tool, so people really wanted it. We have an in joke where you get a D20 when you join, and one of the faces reads dark mode when, and that joke is five or six years old. For at least half the company’s life it was something we would get to eventually. Then in about six months we shipped it, mostly because of AI. I want to be clear it was not only AI. Getting there was a multi year process where our designers and product engineers built a design system and an accessible component library that also supported dark mode. At the end of that we had all the components, but we still faced the mechanical work of turning the existing app into the new ones, and that takes a lot of time.

Because we could lean on agent loops, we did it in about six months, and not as one team’s full time effort. People across the whole company chipped in, including engineering directors and managers, because instead of telling everyone to go do all these things by hand, the team shipped prompts and skills you could feed the agent to do the migration from the old components to the new ones. That unlocked a lot of human judgment, because I could pull in an engineering director who built a particular screen seven years ago and still understands what it is for, and tell them to fire up Claude Code, load this skill, point it at that page, and do the migration.

That is a good example of what your mental model should be. Yes, it used a lot of tokens, and a small focused team could have hyper optimized it. But the actual benefit is that we finally closed a feature people had asked for over five or six years, a lot of people learned how to use AI who otherwise would not have, and we got a more accessible and more beautiful product. Those benefits are tricky to quantify, but they make a real difference to customer experience, and at the end of the day that is what we are here for. It is not using AI for its own sake, it is talking to users, understanding their pain points, and meeting them where they are. So do not treat it as an AI mandate you justify only through tokens in and tokens out. Ask what tokens let you do that would otherwise have been impossible, and what lets you use AI as leverage, as a bicycle for the mind, a way to take bigger swings at bigger problems. If you can figure out how to measure that, that is your North Star.

And the system design round is where that scrutiny concentrates, because it is the one round an AI assistant will not get you through. The system design interview may look the same as it did a couple of years ago, but the grading rubric underneath it has changed a lot.

Consider Karat’s 2026 survey of 400 engineering leaders. It found that AI has widened the gap between strong and weaker engineers rather than closing it, and 73 percent of leaders now say a strong engineer is worth at least three times their total compensation. CoderPad’s State of Tech Hiring Report 2026 shows technical assessments up 48 percent globally since mid 2023, with 60 percent of talent leaders naming quality of hire as their top priority for the year.

We asked engineering leaders who personally conduct system design loops what they actually look for in a session. These insights are not usually found in prep guides and they converge on a shift most candidates have not yet noticed.

Buzzword-first designs are a huge turn off

Archit Agarwal, Principal Member of Technical Staff at Oracle, has in his engineering purview interviewed hundreds of engineers, especially for ultra-low-latency authorization work. And the failure he catches most often has nothing to do with AI. “I’ve seen a lot of engineers come in to a system design interview and, as soon as I give a problem, they start with ‘let’s use microservices,’ and start using distributed cache,” he shares, and when he asks how many users they are planning for, the answer rarely matches the architecture they just proposed. “That is a key difference between any interview-ready engineer and a genuinely good engineer,” he explains, because “a genuinely good engineer would not want to implement everything up front.”

Agarwal’s baseline is quite blunt, that “if the problem isn’t complex yet, don’t overengineer it.” And he, like many notable leaders, holds the position that “microservices aren’t the magical fix that fixes bad architecture. They just distribute that over the network.” What he seeks instead is one to two minutes of genuine alignment at the start, functional requirements first to establish what is being built and what the user needs, then the nonfunctional requirements that set scale, consistency, and latency. “Nonfunctional requirements are the ones that decide the architecture—not the other way around.” And not every system earns planet-scale treatment, since a tool used only by a company’s own engineers needs no multi-region deployment, and proposing one tells him the candidate is performing rather than designing.

He also weighs cost. It is something that always runs through his evaluation the same way, and he shared with us a line that reframed his own career, “a good engineer would design for performance, but a great engineer would design for performance per dollar,” and he pushes candidates to weigh a latency win against the infrastructure bill it creates. That cost lens is exactly where the interview has changed most, because the most expensive and least predictable component on the whiteboard is now the model.

Earlier this year, we spoke with Agarwal about trade-offs in modern system design, and later published another piece featuring his candidate-side advice on why senior engineers fail system design interviews.

Treat every AI component as one that will fail

AI is on every whiteboard now, and the reliability assumptions underneath it are exactly what interviewers have started to stress test. Rohit Poduval, Senior Software Engineer on Prime Video Trust and Safety at Amazon, opens with a test that did not exist two years ago. “Does the candidate treat an AI component as an unreliable dependency or a magic box?” “I ask them to design a system that uses an LLM. Content classification, recommendation filtering, whatever fits the role,” he explains, and then he watches what they do with it. “Weak candidates draw a box labeled LLM and move on, like it’s a database that always returns the right answer,” while strong candidates, he shares, “immediately ask: What happens when it’s wrong? What’s the fallback? How do we know it’s drifting?”

That one reaction most often separates people who have shipped AI systems from people who have read about them. LLMs return different outputs for the same input, and Poduval points out that you can enforce structure on the output, valid JSON or responses from an allowed set, but you cannot assert that the decision itself is correct with a traditional pass and fail test. Candidates who have built these systems talk about evaluation suites, guardrails that constrain what the model can do regardless of its output, and monitoring for drift in production. But candidates who have not will handwave past every one of those concerns. And that handwaving is quite visible within minutes.

“The next thing I probe: how do they test it?” Poduval shares, because traditional assertions do not apply to AI components. “Strong candidates talk about benchmark evaluations: a defined set of inputs with expected behavioral boundaries (not exact outputs) that the system must satisfy before shipping.” They also talk about continuous auditing against real traffic after launch, not just checks before deployment, because a model that passed benchmarks last month might be drifting today. His deeper observation is that with AI systems you often cannot enumerate every right answer before launch, so the old playbook of define requirements, build to spec, and ship no longer closes. The engineers he rates highly design for safe iteration instead, with shadow mode, holdback groups, evaluation frameworks from day one, and human escalation paths, because they expect to keep refining the definition of correct after launch. “That’s exactly where the memorized answers fall apart.”

Taking into consideration the resource-intensive workloads and the shrinking budgets, cost has to be something a candidate can defend out loud. Poduval changes a constraint mid session and tells the candidate the system now needs to handle ten times the traffic. “Candidates who’ve never dealt with inference costs at scale just say ‘add more instances’. Candidates who’ve lived it start talking about which requests actually need an expensive model vs which can be routed to cheaper classifiers,” he shares. They design tiered systems with routing logic that decides in real time which compute budget each request deserves.

The practical takeaway for anyone walking into a loop this year is direct, so state the fallback, the evaluation plan, and the per-request cost tier for any AI component before the interviewer has to ask. Everything on that checklist has an architectural counterpart, and we have captured the nuances of each in Sampriti Mitra’s practical deep-dive on core architectural patterns for LLM system design, which walks through the gateway, tiered fallback, model routing, and evaluation patterns behind it.

Constraint changes reveal who patches and who rethinks

Chandu Putta, Senior Software Engineer at MissionSquare Retirement, in our email interview shared what earns his confidence. “The signal that I trust the most is not the architecture a candidate produces, it’s how they think if I change a constraint in the middle of the interview.” In a recent loop he had candidates design a document processing pipeline, and midway through he added an LLM call to the extraction step, with variable latency and non-deterministic output. The majority of candidates bolted a retry mechanism onto their existing design and moved on. One candidate stopped and asked, “What’s the acceptable failure mode — an explicit error to the engineer or a silent degradation?” he recalls.

That question, to his mind, is the level marker, because it treats the new component as a design problem rather than a network hiccup. Putta is blunt about how he sizes this up, as “rehearsed candidates patch the existing design, but the real senior engineers discard it and start fresh with this new anchor.” The engineers he considers qualified for the job also ask about context boundaries and cost at scale without being prompted, he adds, and they raise a question that prep articles don’t teach, which is who decides whether the model output is good enough, the technology team or the business team. An engineer who asks that has sat in the meeting where the answer was contested.

Edward Tian, creator of GPTZero, sees the same moment through a different lens. He shares with us how his interviews confirm the pattern from another direction. “When we see candidates redesign their system following the introduction of a new constraint, we see that they are able to quickly identify the most significant aspect of their original design that has become the bottleneck,” he explains, and they design new components to replace it.

But weak engineers make incremental changes to accommodate the new requirement without revisiting the larger trade-offs the original design was built on. His sharpest distinction is about posture rather than knowledge, because “most engineers will assume that their original design is now incorrect after the constraint has been added and defend the changes made to it,” he shares. “Senior engineers will simply continue to increase the rate of evolution of their design until it meets their new constraint.”

Tian also shares how he has noticed a newer failure mode that improved AI tooling has made common. “Many candidates can produce beautiful pipeline diagrams for their LLM, but when we dig through more complex failure modes, for example context window, they really struggle,” and they default to talking about scaling the model. The diagram generated fluency, but probing exposes it.

Agarwal looks for something similar when he changes the requirements mid session. The candidate who absorbs the change, restates it in their own words to confirm alignment, and then highlights which parts of the system must change and which remain intact is showing him a structured redesign instinct rather than attachment to a diagram. He also likes to give away something candidates rarely hear from the other side of the table. “The curveballs that the interviewer gives you will never be in a way that you will have to scrap the complete diagram, the complete architecture, unless you were already off the track.” A constraint change is a test of surgical judgment, so the working advice is to name out loud what survives and what dies before you touch the whiteboard.

Can you defend the choice you just made?

Prakhar Chaube, Senior Software Engineer at InsurGrid who previously held the same level at Whatfix and has conducted more than 25 technical rounds, argues the design round should never hunt for a correct answer. He assesses the thought process and then questions the candidate on their component choices, because that questioning mirrors what happens inside real companies. “Most companies have some form of an Architectural Review Board, meaning it’s rare that a new implementation will not be questioned even if it is the right choice.” Candidates should know why the standard patterns work as standards rather than assuming a plug and play model, and the ones who can stand by their choice under pressure usually survive a real review. In his loops he is categorical about telemetry, since “if a candidate is not mentioning that in an interview it is a big no,” because “without proper observability the system is doomed to fail at scale and cause chaos during debugging.”

Chaube has also built a ladder for assessing AI fluency that moves well past the 2024 questions. “I first assess the basics like parameters, prompt structure, prompt security, hallucination,” he shares, then he wants to dig deeper into implementation scenarios drawn from real products, autonomous tasks orchestrated by a central model such as a code review agent or a login and scraping system. The final rung, he explains, is how candidates handle agent-guided coding, because “I look at their queries and flow” rather than their claimed experience. This method, he says, reliably separates candidates who prepared just for the interview from candidates who are genuinely strong. The preparation gap shows up in the follow-ups, never in the opening answer.

Sándor Dargó, senior software engineer at Spotify, has observed the same gap from years of interviewing in the C++ world, and his experience reinforces Chaube’s from a different stack. He describes a depth gap first, because a language whose standard runs about 2,000 pages guarantees that even genuine experts have blind areas, and the honest move is admitting them.

Dargó in our live interview session shared how he once told an interviewer he did not know a topic and did not want to guess. The interviewer replied that their team did not use it either, and he got hired. His second observation lands harder, as he has seen senior engineers who handle architectural questions with real thoughtfulness struggle to write simple algorithms live under pressure. Rehearse defending your choices against a hostile follow-up, not presenting them to a friendly one, and practice the small problems you assume you have outgrown.

Safety-critical systems raise the same bar higher

With more than 15 years in secure embedded systems, Hareesha KoratikereRameshappa, Senior Technical Product Manager at Phantom AI, likes to conduct design interviews where the system under discussion can kill someone. He evaluates whether a candidate can translate high-level product requirements into a structured, developable architecture without compromising safety, cost, accuracy, reliability, and power consumption. He looks for system-level thinking across hardware, firmware, application software, and flashing methodologies like OTA. And what he expects, he shares, is that “the candidate should understand the key elements of system design, such as system and operating modes, error and fault detection, active and history fault mode managements, transition to the safe mode, and recovery from the fault mode.” They must also supply the inputs verification depends on, KPIs, requirements, corner cases, and they must plan to monitor the system after deployment rather than treating launch as the finish line.

His trade-off vocabulary sounds exactly like the web scale interviewers we heard from earlier, and that essentially is the point. Rameshappa presses candidates on functionality versus performance and accuracy versus resource consumption, and he expects them to justify design decisions with data and adjust the design when another team’s feedback is valuable. At Phantom AI he shared how he struggled to find candidates who combined production experience with sensor selection, calibration integration, and corner case validation, which tells you how rare the full reasoning package is even among experienced engineers.

This means that what the interviewers are testing is not a property of a tech stack. It is engineering judgment, and it transfers from a recommendation pipeline to a multi-camera ADAS system without translation.

Trade-off reasoning still decides the level

Everything above rests on a foundation that has not moved, and Dhirendra Sinha, Software Engineering Manager at Google, described it last year in one of our conversations with him and Tejas Chopra. “It’s easy to choose between good and bad solutions,” Sinha told us, “but senior engineers often have to choose between two good options. I want to hear their reasoning.” He also carries a line from a chief architect at Yahoo that reframes how scale changes judgment, because when an engineer dismissed a corner case as one in a million, the architect replied, “One in a million happens every hour here.” Scale invalidates assumptions, and mostly every leader in this piece is testing whether a candidate’s assumptions survive contact with it.

Agarwal’s version of the same evaluation focuses on where a candidate chooses to go deep. When a candidate takes one area, distributed storage or authentication or performance engineering, and drives into real depth, he takes that as an engineer who “understands the gravity of things” rather than one collecting surface vocabulary. And he calibrates the questions a candidate asks against their level, because clarification questions are always welcome, but a flood of questions too basic for someone with a strong resume tells him the candidate has never actually thought about the system in front of them.

Tejas Chopra, Senior Engineer at Netflix, closes the loop between the old test and the new one with a single follow-up he has asked for years. When a candidate picks SQL over NoSQL or strong over eventual consistency, he asks what changes if the user base grows tenfold, and the AI era version of that question is exactly what some interviewers in technical rounds now ask about model cost and failure modes. The component changed, but the muscle is the same. For readers who want the underlying vocabulary these interviews keep invoking, consistency, availability, partition tolerance, and the read and write trade-offs that connect them, Sinha and Chopra’s practical deep-dive on distributed system attributes from their book System Design Guide for Software Professionals walks through all of it with worked examples.

What to do in your next loop

Ask for the acceptable failure mode before you design anything, because that single question outranks any diagram you will draw. Treat every AI box in your architecture as a component that will be wrong, and price its errors with a fallback, an evaluation plan, and a cost tier per request class. When a constraint changes mid session, say out loud which parts of your design survive and which parts die before you redraw anything. Say what you would monitor without being asked, and say I don’t know about the right things, because every interviewer quoted here treats honesty about limits as seniority rather than weakness.

The rehearsed candidate optimizes for the first answer, and the bar has moved to the third follow-up. That is the whole shift.

Contributors

• Archit Agarwal, Principal Member of Technical Staff, Oracle. LinkedIn. Read his candidate-side advice in Why Senior Engineers Fail System Design Interviews.

• Rohit Poduval, Senior Software Engineer, Amazon (Prime Video Trust & Safety). LinkedIn

• Naga Chand Putta, Senior Software Engineer, MissionSquare Retirement. LinkedIn

• Edward Tian, Founder, GPTZero. LinkedIn

• Prakhar Chaube, Senior Software Engineer (SDE-3), InsurGrid. LinkedIn

• Hareesha KoratikereRameshappa, Sr. Technical Product Manager, Phantom AI. LinkedIn

• Sándor Dargó, Senior Software Engineer, Spotify. sandordargo.com

• Dhirendra Sinha (Google) and Tejas Chopra (Netflix), authors of System Design Guide for Software Professionals (Packt). Read their free chapter.

Sources

• Karat, “Engineering Interview Trends in 2026,” March 2026. https://karat.com/engineering-interview-trends-2026/

• CoderPad, “State of Tech Hiring Report 2026,” March 2026. https://coderpad.io/blog/hiring-developers/new-research-the-2026-state-of-tech-hiring-what-ai-means-for-developers-and-hiring-teams/

• Deep Engineering, “Why Senior Engineers Fail System Design Interviews,” May 2026. https://deepengineering.net/p/why-senior-engineers-fail-system-design-interviews

• Deep Engineering #36, Archit Agarwal on System Design Trade-offs, February 2026. https://deepengineering.substack.com/p/deep-engineering-36-archit-agarwal

What makes these patterns particularly valuable in Rust isn’t their novelty. Most have decades of history in other languages. It is how Rust’s design makes them practical for systems programming. The combination of strong static typing, zero-cost abstractions, and memory safety without garbage collection creates opportunities to apply these patterns in contexts where they were previously impractical.

We’ll apply these patterns to Samsa, a publish/subscribe microservice in the style of a miniature Kafka that the book builds across its later chapters, patterns that have proven their worth across multiple programming language communities. As we explore each pattern, we’ll examine both its origins and how Rust’s unique features enhance or constrain its application.

This deep-dive is excerpted from Design Patterns and Best Practices in Rust by Evan Williams, shared with permission from Packt, with all rights remaining with the publisher and no reproduction or redistribution without written consent. You can get the full book here.

In this chapter the book covers the following main topics:

• The NewType pattern

• Parse, don’t validate

• The TypeState pattern

• Sealed traits

Each pattern builds on the previous ones: the NewType pattern creates distinct types; parse, don’t validate ensures those types hold only valid data; the TypeState pattern tracks valid state transitions; and sealed traits control which types can participate in our APIs.

The NewType pattern

The NewType pattern has a long history in statically-typed functional programming, particularly in Haskell, where the newtype keyword has been a language feature since the 1990s. The pattern also appears in ML, OCaml, and other languages with strong type systems. The core idea of wrapping existing types in distinct types for semantic clarity and type safety predates Rust. It is a response to the issue that values that are conceptually different, such as temperature and weight, have the same type representation in the code, for example, a float. From the compiler’s perspective, they are the same, and that leads to bugs when these values are inadvertently mixed up.

What Rust brings to this well-established pattern is zero-cost abstraction: the wrapped type compiles to exactly the same representation as the underlying type, with no runtime overhead. In languages with garbage collection or boxing, creating wrapper types often incurs performance costs. Rust’s design ensures that type safety is free at runtime while remaining enforceable at compile time.

Identifying the problem

In our Samsa system, we currently use primitive types such as u64 for topic IDs, consumer IDs, and message offsets. This approach, which we will call primitive obsession, makes it easy to accidentally pass the wrong type of ID to a function. If you’ve worked with similar systems, you’ve probably experienced the confusion this creates. The NewType pattern solves this problem with compile-time type checking.

Let’s examine our current API and identify where primitive obsession creates issues:

// Current API - prone to mix-ups
impl Broker {
     pub fn subscribe(&self, topic_id: u64, consumer_id: u64) -> Result<()> {
         // Easy to accidentally swap these parameters!
         if !self.topics.contains_key(&topic_id) {
             return Err("Topic not found".to_string());
         }
         Ok(())
     }
 }

impl Consumer {
     pub fn new(broker: Arc<Broker>, topic_id: u64, offset: u64) -> Self {
         // Again, easy to mix up topic_id and offset
         Self { broker, topic_id, offset }
     }
 }

We’re using raw u64 values for different semantic concepts, making it easy to pass the wrong type of ID and causing runtime bugs that are difficult to debug. What is very unfortunate is that the compiler cannot help us identify logical errors when we pass a topic ID instead of a consumer ID.

Implementing type-safe wrappers

Let’s create wrapper types that eliminate these mix-ups. Our approach has three parts:

1. Define semantic wrapper types for our different ID concepts (shown in the first code block)

2. Implement basic methods for usability, such as constructors and accessors (shown in the second code block)

3. Add Display implementations for user-friendly output (shown in the third code block)

Let’s define semantic wrapper types: TopicId for topic identification, ConsumerId for tracking individual consumers, and MessageId for tracking individual messages. We’ll use plain u64 for message offsets for simplicity (in the current design, message offsets don’t benefit much from the additional type safety):

use std::fmt;

/// Topic identifier for tracking individual topics
#[derive(Debug, Clone, PartialEq, Eq, Hash)]
pub struct TopicId(String);

/// Consumer identifier for tracking individual consumers
#[derive(Debug, Clone, PartialEq, Eq, Hash)]
pub struct ConsumerId(String);

/// Message identifier for tracking individual messages
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub struct MessageId(u64);

These NewTypes provide type safety through distinct wrapper types. TopicId and ConsumerId wrap String to allow meaningful identifiers such as "user.events" or "consumer-1", while MessageId wraps u64 for numeric message tracking. Each type has meaningful derives, such as Hash for use in collections. Note that MessageId includes Copy since u64 is copyable, while the String-based types do not.

Now, let’s add methods for creating and accessing these types:

impl TopicId {
    pub fn new(topic: impl AsRef<str>) -> Result<Self> {
        let topic = topic.as_ref();

        if topic.is_empty() {
            return Err(SamsaError::topic("Topic name cannot be empty"));
        }

        if topic.len() > 128 {
            return Err(SamsaError::topic(
                format!("Topic name too long: {} chars (max 128)",
                         topic.len())
            ));
        }

        Ok(TopicId(topic.to_string()))
    }

    pub fn as_str(&self) -> &str {
        &self.0
    }
}

impl ConsumerId {
    pub fn new(id: impl AsRef<str>) -> Result<Self> {
        let id = id.as_ref();

        if id.is_empty() {
            return Err(SamsaError::consumer("Consumer ID cannot be
                                             empty"));
        }

        Ok(ConsumerId(id.to_string()))
    }

    pub fn as_str(&self) -> &str {
        &self.0
    }
}

impl MessageId {
    pub fn new(id: u64) -> Self {
        MessageId(id)
    }

    pub fn value(&self) -> u64 {
        self.0
    }
}

Each type provides a new constructor and an accessor method. TopicId and ConsumerId validate their inputs at construction time, returning Result to signal potential validation failures. This is the NewType pattern combined with early validation: once you have TopicId or ConsumerId, you know it contains valid data. MessageId uses simple u64 wrapping since numeric IDs don’t require validation.

Finally, let’s add Display implementations for user-friendly output:

impl fmt::Display for TopicId {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        write!(f, "{}", self.0)
    }
}

impl fmt::Display for ConsumerId {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        write!(f, "{}", self.0)
    }
}

impl fmt::Display for MessageId {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        write!(f, "{}", self.0)
    }
}

The Display implementations output the inner value directly. For example, a TopicId containing "user.events" prints as user.events. If we were writing production code, we could also implement a custom Debug trait and print the IDs in a way that is more helpful for debugging, such as "TopicID(user.events)".

We’ve created distinct types for different domain concepts. Each type wraps its inner value and prevents accidental mixing. It’s now impossible to accidentally pass a TopicId where a ConsumerId is expected. The compiler catches this at compile time, eliminating entire classes of bugs. Each wrapper type is distinct at compile time.

For MessageId wrapping u64, the type compiles to the same representation as u64 at runtime, creating a true zero-cost abstraction. TopicId and ConsumerId use String for semantic identifiers but still provide compile-time type safety.

Enhanced Samsa API

Let’s see how NewTypes prevent parameter mix-ups. Consider a subscription function that takes both a consumer and a topic:

fn subscribe(consumer: ConsumerId, topic: TopicId) -> Result<(), SamsaError> {
      println!("Subscribing {} to {}", consumer, topic);
      Ok(())
  }

  // Usage - this compiles:
  let topic = TopicId::new("orders.created")?;
  let consumer = ConsumerId::new("analytics-service")?;
  subscribe(consumer, topic)?;

  // But this won't compile - arguments swapped:
  // subscribe(topic, consumer);
  //           ^^^^^ expected `ConsumerId`, found `TopicId`

Notice how subscribe takes TopicId and ConsumerId as separate types. The compiler would reject any attempt to swap them. Without NewTypes, both topic and consumer would be String, and swapping them would compile silently – a bug you’d only discover at runtime (maybe while in production). With NewTypes, the compiler catches the mistake immediately. This is especially valuable in APIs with multiple string-like parameters where mix-ups are easy to make and hard to debug.

Now that we have our type-safe wrappers, let’s apply them to Samsa’s API to see how they prevent the parameter mix-ups we identified earlier. The following code snippet enhances the consumer API with type safety:

pub struct Consumer {
     broker: Arc<Broker>,
     id: ConsumerId,
     topic: TopicId,
     offset: u64,
 }

impl Consumer {
     pub fn new(broker: Arc<Broker>, id: ConsumerId, topic: TopicId,
                offset: u64) -> Self {
         Self { broker, id, topic, offset }
     }

    pub fn poll(&mut self) -> Result<Option<Message>, String> {
         let events = self.broker.fetch(self.topic, self.offset, 1)?;
    
        if let Some(message) = events.into_iter().next() {
             self.offset += 1;
             Ok(Some(message))
         } else {
             Ok(None)
         }
     }
 }

The Consumer struct stores typed identifiers for its broker connection, ID, topic, and current offset. The poll() method demonstrates how advancement works with the typed consumer. Each field uses the appropriate NewType, making the struct’s purpose clear from its type signature alone.

Our entire API now uses type-safe identifiers. Function signatures are self-documenting, and the compiler prevents mixing up different types of IDs. This eliminates bugs where IDs are confused, provides better error messages, and makes the code more maintainable. With this design, Samsa’s consumer API becomes self-documenting and resistant to ID mix-ups.

The NewType pattern provides compile-time guarantees about type correctness while maintaining the runtime performance of primitive types. This combination of safety and efficiency is what makes the pattern particularly valuable in Rust.

However, while NewTypes give us distinct types, they don’t enforce validity. A MessageId could wrap any u64, which is OK if every u64 is a valid message ID. But if some are not, including those invalid values in our data type can turn into an issue. In the next section, we’ll see how to combine type distinctions with validity guarantees using the “parse, don’t validate” principle.

Parse, don’t validate

Alexis King popularized the parse, don’t validate principle with her influential 2019 blog post of the same name (https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/). King’s articulation of the principle, which states that parsing generates types that can represent only valid data, while validation merely checks data after the fact, has its roots in functional programming and type theory and helped popularize and name the pattern.

Rust’s type system makes this principle particularly natural to apply. The combination of strong typing, pattern matching, and the Result type creates an environment where parse, don’t validate feels like the path of least resistance rather than additional work.

The principle suggests that instead of scattering validation logic throughout a code base, we should parse data once at system boundaries into types that can only represent valid states. This transforms validation from a repeated burden into a guarantee encoded in types.

The problem with scattered validation

Let’s examine how our current Samsa configuration system handles validation. The current approach is validation scattered everywhere:

pub struct BrokerConfig {
     pub max_connections: i32,    // Could be negative!
     pub buffer_size: usize,      // Could be zero!
     pub port: u16,              // Could be in reserved range!
 }

impl Broker {
     pub fn new(config: BrokerConfig) -> Result<Self, String> {
         // Validation #1 - at construction
         if config.max_connections <= 0 {
             return Err("max_connections must be positive".to_string());
         }
         if config.buffer_size == 0 {
             return Err("buffer_size must be non-zero".to_string());
         }
         Ok(Self { config })
     }

    pub fn accept_connection(&self) -> Result<(), String> {
         // Validation #2 - in business logic (defensive programming)
         if self.config.max_connections <= 0 {
             return Err("Invalid max_connections".to_string());
         }
         Ok(())
     }
 }

Notice how the same validation check, if max_connections <= 0, appears in two different places: once during configuration creation and once defensively in the business logic. This duplication is the core problem. Each location must remember to perform the check, use the same condition, and handle errors consistently. If the validation rule changes (say, max_connections must be at least 10), every location must be updated. This approach is error-prone, inefficient, and creates inconsistent validation logic across different parts of the system.

Creating types that enforce validity

The solution is to create wrapper types that can only hold valid values. Instead of validating a raw u32 everywhere it’s used, we create a PositiveU32 type that validates once at construction time. Any code that receives a PositiveU32 type knows, by the type alone, that the value has already been validated.

Let’s redesign our configuration using this approach. We’ll start by creating a wrapper type for positive integers that cannot be zero or negative:

use std::num::NonZeroUsize;

#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct PositiveU32(u32);

impl PositiveU32 {
     pub fn new(value: u32) -> Result<Self, ConfigError> {
         if value > 0 {
             Ok(Self(value))
         } else {
             Err(ConfigError::MustBePositive("value".to_string()))
         }
     }

    pub fn get(self) -> u32 {
         self.0
     }
 }

Next, we create a similar wrapper for network ports that ensures the port number is not in the reserved range (ports 0–1023):

#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct NetworkPort(u16);

impl NetworkPort {
     pub fn new(port: u16) -> Result<Self, ConfigError> {
         if port > 1023 {  // Ports 0-1023 are reserved
             Ok(Self(port))
         } else {
             Err(ConfigError::PortInReservedRange(port))
         }
     }

    pub fn get(self) -> u16 {
         self.0
     }
 }

Let’s also define an error type for configuration errors:

#[derive(Debug, Clone, PartialEq, Eq)]
 pub enum ConfigError {
     MustBePositive(String),
     PortInReservedRange(u16),
 }

impl std::fmt::Display for ConfigError {
     fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
         match self {
             ConfigError::MustBePositive(field) => write!(f, "{} must be positive", field),
             ConfigError::PortInReservedRange(port) => write!(f, "Port {} is in reserved range", port),
         }
     }
 }

impl std::error::Error for ConfigError {}

Samsa’s configuration system now uses types that enforce validity at construction time. Once created, these types guarantee their validity throughout the system. This moves all validation to the boundary of our system and eliminates defensive programming in business logic.

By validating once during construction, we ensure that these types can only ever hold valid values. Rust’s ownership system guarantees that once created, these values cannot be modified to become invalid.

Builder pattern with validation

The Builder pattern pairs naturally with validated types. While the wrapper types we defined validate individual values, the Builder pattern coordinates multiple validated values into a complete, consistent configuration. We’ll create a BrokerConfig that can only be constructed through a builder, ensuring that all validation happens during construction.

First, let’s define the configuration struct with validated field types:

#[derive(Debug, Clone)]
pub struct BrokerConfig {
    max_connections: PositiveU32,
    buffer_size: NonZeroUsize,
    port: NetworkPort,
}

impl BrokerConfig {
    pub fn builder() -> BrokerConfigBuilder {
        BrokerConfigBuilder::new()
    }

    // Getters that never need validation
    pub fn max_connections(&self) -> u32 {
        self.max_connections.get()
    }

    pub fn buffer_size(&self) -> usize {
        self.buffer_size.get()
    }

    pub fn port(&self) -> u16 {
        self.port.get()
    }
}

The BrokerConfig struct uses validated types for all fields. PositiveU32 and NetworkPort ensure that invalid values cannot be stored. The getters return primitive values, which is safe because we know the wrapped values are valid. Notice that all fields are private and accessed through getters. This prevents external code from modifying the configuration after validation.

Now, we need a way for users to construct a BrokerConfig. Since the struct has private fields with validated types, we can’t allow direct construction. Users would need access to PositiveU32::new() and NetworkPort::new(), and they’d have to handle validation errors themselves. Instead, we’ll provide a builder that accepts raw primitive values, validates them internally, and produces a fully validated BrokerConfig on success.

The BrokerConfigBuilder struct holds Option fields for each configuration value. As users set values through the builder’s methods, each value is validated and wrapped in its corresponding type. The final build() method ensures that all required fields are present before constructing the BrokerConfig. Let’s implement this builder:

pub struct BrokerConfigBuilder {
    max_connections: Option<PositiveU32>,
    buffer_size: Option<NonZeroUsize>,
    port: Option<NetworkPort>,
}

impl BrokerConfigBuilder {
    pub fn new() -> Self {
        Self {
            max_connections: None,
            buffer_size: None,
            port: None,
        }
    }

    pub fn max_connections(mut self, value: u32) -> Result<Self, ConfigError> {
        self.max_connections = Some(PositiveU32::new(value)?);
        Ok(self)
    }

    pub fn buffer_size(mut self, value: usize) -> Result<Self, ConfigError> {
        self.buffer_size = Some(
            NonZeroUsize::new(value)
                .ok_or_else(|| ConfigError::MustBePositive("buffer_size".to_string()))?
        );
        Ok(self)
    }

    pub fn port(mut self, value: u16) -> Result<Self, ConfigError> {
        self.port = Some(NetworkPort::new(value)?);
        Ok(self)
    }
}

Each setter method validates its input before storing it. The ? operator propagates validation errors immediately if validation fails. If validation succeeds, the method consumes self and returns it, enabling method chaining. The use of Option for each field lets us detect missing fields in the build() method.

The final step is building the configuration, which ensures that all required fields are present and results in a fully validated BrokerConfig:

impl BrokerConfigBuilder {
    pub fn build(self) -> Result<BrokerConfig, ConfigError> {
        Ok(BrokerConfig {
            max_connections: self.max_connections
                .ok_or_else(|| ConfigError::MustBePositive("max_connections not set".to_string()))?,
            buffer_size: self.buffer_size
                .ok_or_else(|| ConfigError::MustBePositive("buffer_size not set".to_string()))?,
            port: self.port
                .ok_or_else(|| ConfigError::PortInReservedRange(0))?,
        })
    }
}

The build() method consumes the builder and returns Result<BrokerConfig, ConfigError>. If any required field is missing, we return an error. This ensures that you can’t construct an incomplete configuration.

The usage now looks like this:

let config = BrokerConfig::builder()
    .max_connections(1000)?
    .buffer_size(4096)?
    .port(8080)?
    .build()?;

The fluent API makes configuration construction readable and catches errors immediately. If any setter returns an error, the ? operator short-circuits, and the invalid configuration is never created. The final build() call ensures that no required fields are missing.

The builder ensures that all configuration values are validated during construction. Once a BrokerConfig exists, all its values are guaranteed to be valid. This design makes it impossible to create invalid configurations, with validation happening once at the system boundary.

The parse, don’t validate principle, combined with Rust’s type system and ownership model, eliminates defensive programming throughout the system by moving all validation to well-defined boundaries.

So far, we’ve seen how to create distinct types (NewType) and ensure that those types only hold valid values (parse, don’t validate). But what about objects that change over time? In the next section, we’ll explore the TypeState pattern, which uses the type system to track and enforce valid state transitions.

The TypeState pattern

The TypeState pattern has academic origins in research on program verification. It was formalized by Robert E. Strom and Shaula Yemini in their 1986 paper TypeState: A Programming Language Concept for Enhancing Software Reliability (which you can read at https://www.computer.org/csdl/journal/ts/1986/01/06312929/13rRUwIF6aQ). The core idea is to use types to represent different states in which an object can be, making invalid state transitions impossible to express.

While the concept has existed in type theory for decades, Rust’s unique combination of features makes it particularly practical to implement. Rust’s zero-cost abstractions mean the type-level state tracking compiles away entirely, leaving no runtime overhead. The ownership system ensures that state transitions consume the old state, preventing accidental reuse. And generic types with PhantomData provide a clean mechanism for encoding state in the type system.

Several other languages support similar patterns, most notably session types in research languages and builder patterns in languages such as Java, but Rust’s combination of performance, safety, and ergonomics makes the TypeState pattern particularly accessible for systems programming.

The problem with runtime state management

Let’s examine traditional runtime state management, where state is tracked at runtime:

#[derive(Debug, Clone, Copy, PartialEq)]
pub enum ConsumerState {
     Disconnected,
     Connected,
     Subscribed,
     Paused,
 }

pub struct Consumer {
     broker: Arc<Broker>,
     topic: TopicId,
     consumer_id: ConsumerId,
     state: ConsumerState,
 }

impl Consumer {
     pub fn connect(&mut self) -> Result<(), String> {
         // Runtime check required
         if self.state != ConsumerState::Disconnected {
             return Err("Consumer must be disconnected to connect".to_string());
         }
         self.state = ConsumerState::Connected;
         Ok(())
     }

    pub fn poll(&mut self) -> Result<Option<Message>, String> {
         // Runtime validation in every method
         if self.state != ConsumerState::Subscribed {
             return Err("Consumer must be subscribed to poll
                         messages".to_string());
         }
         Ok(None)
     }
 }

Every method requires runtime validation to ensure a correct state, creating opportunities for errors and adding runtime overhead. The compiler can’t help us by catching state transition errors, which leads to runtime failures that could have been prevented at compile time.

Implementing TypeState for the consumer lifecycle

Let’s redesign the consumer using TypeState. We’ll create separate types for each state:

use std::marker::PhantomData;
use crate::types::*;

// State marker types organized in a module
pub mod states {
    #[derive(Debug)]
    pub struct Disconnected;

    #[derive(Debug)]
    pub struct Connected;

    #[derive(Debug)]
    pub struct Subscribed;

    #[derive(Debug)]
    pub struct Paused;
}

// Consumer parameterized by state
pub struct Consumer<State> {
    broker: Arc<Broker>,
    topic: Option<TopicId>,
    consumer_id: ConsumerId,
    connection_info: Option<ConnectionInfo>,
    state: PhantomData<State>,
}

/// Type aliases for consumer states
pub type DisconnectedConsumer = Consumer<states::Disconnected>;
pub type ConnectedConsumer = Consumer<states::Connected>;
pub type SubscribedConsumer = Consumer<states::Subscribed>;
pub type PausedConsumer = Consumer<states::Paused>;

#[derive(Debug, Clone)]
pub struct ConnectionInfo {
    pub broker_address: String,
    pub consumer_group: Option<String>,
}

The state marker types (Disconnected, Connected, Subscribed, and Paused) are zero-sized. They exist only at compile time to track state in the type system. The Consumer<State> struct is generic over the state marker, so Consumer<states::Disconnected> and Consumer<states::Connected> are different types even though they contain the same data fields. The type aliases provide convenient shorthand.

The PhantomData<State> field tells Rust about the state parameter without storing anything at runtime. This is the key to TypeState: state tracked in types with zero runtime cost.

Let’s implement the disconnected state:

impl Consumer<states::Disconnected> {
    pub fn new(consumer_id: ConsumerId, broker: Arc<Broker>) -> Self {
        Self {
            broker,
            topic: None,
            consumer_id,
            connection_info: None,
            state: PhantomData,
        }
    }

    pub fn connect(
        self,
        connection_info: ConnectionInfo
    ) -> Result<Consumer<states::Connected>, SamsaError> {
        if connection_info.broker_address.is_empty() {
            return Err(SamsaError::connection("Empty broker address"));
        }

        Ok(Consumer {
            broker: self.broker,
            topic: self.topic,
            consumer_id: self.consumer_id,
            connection_info: Some(connection_info),
            state: PhantomData,
        })
    }
}

The new constructor creates a Consumer<states::Disconnected>. The connect method consumes the disconnected consumer (taking ownership with self) and returns either a Consumer<states::Connected> or an error along with the original consumer. This ownership transfer prevents reuse. Once you call connect, the disconnected consumer is gone.

Now, let’s implement the connected state:

impl Consumer<states::Connected> {
    pub fn subscribe(
        mut self,
        topic: TopicId
    ) -> Result<Consumer<states::Subscribed>, SamsaError> {
        self.topic = Some(topic);

        Ok(Consumer {
            broker: self.broker,
            topic: self.topic,
            consumer_id: self.consumer_id,
            connection_info: self.connection_info,
            state: PhantomData,
        })
    }

    pub fn disconnect(self) -> Consumer<states::Disconnected> {
        Consumer {
            broker: self.broker,
            topic: None,
            consumer_id: self.consumer_id,
            connection_info: None,
            state: PhantomData,
        }
    }
}

Each state transition follows the same pattern: consume the current state, perform an operation, and return the new state or an error. The connected consumer can subscribe to a topic or disconnect. Each transition is a separate impl block, making it impossible to call methods from the wrong state.

Finally, the Subscribed state is where consumers can actually receive messages, while the Paused state can only resume:

impl Consumer<states::Subscribed> {
    pub fn receive(&self) -> Option<Event> {
        // Only subscribed consumers can receive - guaranteed by types!
        None
    }

    pub fn pause(self) -> Consumer<states::Paused> {
        Consumer {
            broker: self.broker,
            topic: self.topic,
            consumer_id: self.consumer_id,
            connection_info: self.connection_info,
            state: PhantomData,
        }
    }

    pub fn unsubscribe(mut self) -> Consumer<states::Connected> {
        self.topic = None;
        Consumer {
            broker: self.broker,
            topic: self.topic,
            consumer_id: self.consumer_id,
            connection_info: self.connection_info,
            state: PhantomData,
        }
    }
}

impl Consumer<states::Paused> {
    pub fn resume(self) -> Consumer<states::Subscribed> {
        Consumer {
            broker: self.broker,
            topic: self.topic,
            consumer_id: self.consumer_id,
            connection_info: self.connection_info,
            state: PhantomData,
        }
    }
}

Only Consumer<states::Subscribed> has a receive method. You literally cannot call receive() on consumers in other states. The method doesn’t exist for those types. The Paused state shows that TypeState can model bidirectional transitions: pause() moves to Paused, and resume() returns to Subscribed.

This TypeState implementation makes invalid operations impossible to express. The compiler enforces the correct state machine at compile time with no runtime overhead.

TypeState’s implementation in Rust uses PhantomData to carry type-level information that exists only at compile time, employs generic types to parameterize over state, and utilizes the ownership system to ensure that state transitions consume the previous state. The result is verification of state machines at compile time with zero runtime cost.

Usage examples

Here’s how TypeState improves the developer experience:

fn correct_usage() -> Result<(), SamsaError> {
    let broker = Arc::new(Broker::new());
    let consumer_id = ConsumerId::new("test-consumer")?;
    let topic = TopicId::new("events")?;

    // Create disconnected consumer
    let consumer = Consumer::new(consumer_id, broker);

    // Connect with connection info
    let connection_info = ConnectionInfo::new(
        "localhost:9092".to_string(), None
    );
    let connected = consumer.connect(connection_info)?;

    // Subscribe to topic
    let subscribed = connected.subscribe(topic)?;

    // Can pause and resume
    let paused = subscribed.pause();
    let subscribed = paused.resume();

   // Now we can receive - guaranteed to be in correct state
    if let Some(event) = subscribed.receive() {
        println!("Received: {:?}", event);
    }

    // Unsubscribe and disconnect
    let connected = subscribed.unsubscribe();
    let _disconnected = connected.disconnect();

    Ok(())
}

This code demonstrates the correct sequence: create a disconnected consumer, connect it, subscribe, then receive – pausing and unpausing, unsubscribing, and disconnecting work the same way. Each state transition is explicit in the code, and the types change at each step. The error handling uses a tuple to return the original consumer on failure, allowing retry logic.

Invalid usage won’t compile:

fn invalid_usage() {
    let broker = Arc::new(Broker::new());
    let consumer_id = ConsumerId::new("test").unwrap();

    let consumer = Consumer::new(consumer_id, broker);

    // These lines won't compile:
    // consumer.receive(); // Error: not available on Disconnected
    // consumer.subscribe(topic); // Error: not available on Disconnected
}

The compiler errors shown in the comments aren’t runtime errors; they’re compile-time errors. You literally cannot write code that calls receive() on a disconnected consumer because the method doesn’t exist for that type. This is the power of TypeState: invalid states are unrepresentable.

The TypeState pattern transforms potential runtime errors into compile-time guarantees, demonstrating how Rust’s type system and ownership model enable practical implementation of ideas from formal methods and programming language research.

TypeState controls how individual objects transition through states. But sometimes, we need to control which types can participate in a system at all. In the next section, we’ll see how sealed traits let us define closed sets of implementations, providing API stability and safety guarantees.

Sealed traits

Many object-oriented languages introduce the concept of sealed or final types, which are types that cannot extend beyond a defined scope. Java has final classes, Kotlin has sealed classes, and Scala has sealed traits. The pattern addresses a real problem in API design: sometimes, allowing arbitrary external implementations of an interface creates more problems than it solves.

Rust doesn’t have a built-in sealed keyword, but the module system provides a way to achieve the same effect. Following its appearance in various standard library crates and its documentation in API design discussions, the technique gained widespread recognition in the Rust community. While not unique to Rust, the pattern fits naturally with Rust’s privacy model and module system.

The sealed traits pattern uses Rust’s module privacy to create traits that are visible for use but restricted for implementation. This provides library authors with control over trait implementations while maintaining a public interface.

The problem with open traits

Let’s understand why we might want to seal traits. Because this is an open trait, anyone can implement it:

pub trait MessageSchema {
     type Data;
     type Error: std::error::Error;

    fn parse(data: &[u8]) -> Result<Self::Data, Self::Error>;
     fn serialize(data: &Self::Data) -> Vec<u8>;
     fn validate(data: &Self::Data) -> bool;
 }

Here, we’ve defined a very straightforward trait defining the signatures of three methods, with associated types for the data payload and the error type. There is nothing unique about this trait, but what we should pay attention to is that this is a public trait, so there are no restrictions on who can implement it or where the implementation happens.

External crates could implement unsafe versions:

// struct UnsafeSchema;
// impl MessageSchema for UnsafeSchema {
//     fn parse(_data: &[u8]) -> Result<Self::Data, Self::Error> {
//         unsafe { /* potentially unsafe implementation */ }
//     }
// }

The MessageSchema trait defines an interface that any type can implement. While this flexibility is often desirable, it means we can’t make guarantees about all implementations. External code could implement the trait incorrectly or in ways that violate our assumptions.

When external crates can implement MessageSchema, we lose control over quality guarantees and make it difficult to evolve the trait interface without breaking external implementations.

External implementations might do the following:

Additionally, evolving the trait interface becomes difficult. Adding a new required method would break all external implementations, forcing us to use default implementations even when that’s not ideal.

Implementing the sealed traits pattern

Let’s implement the sealed traits pattern for our message schema system, using a private module containing the sealing trait:

mod private {
    pub trait Sealed {}
}

// Public trait that extends the sealed trait
pub trait MessageSchema: private::Sealed {
    type Message: Clone + std::fmt::Debug;

    fn serialize(message: &Self::Message) -> Vec<u8>;
    fn deserialize(bytes: &[u8]) -> Result<Self::Message, SchemaError>;
    fn schema_id() -> &'static str;
    fn validate(message: &Self::Message) -> Result<(), ValidationError>;
}

The key to the sealed traits pattern is the private module. The Sealed trait is public within the module, but the module itself is private. External crates can see MessageSchema (which extends private::Sealed) but cannot implement Sealed, which means they cannot implement MessageSchema.

This gives us complete control over which types can implement our schema trait.

Let’s implement a JSON schema:

pub struct JsonSchema;

impl private::Sealed for JsonSchema {}

impl MessageSchema for JsonSchema {
    type Message = serde_json::Value;

    fn serialize(message: &Self::Message) -> Vec<u8> {
        serde_json::to_vec(message).unwrap_or_default()
    }

    fn deserialize(bytes: &[u8]) -> Result<Self::Message, SchemaError> {
        serde_json::from_slice(bytes)
            .map_err(|e| SchemaError::DeserializationFailed(e.to_string()))
    }

    fn schema_id() -> &'static str {
        "json_v1"
    }

    fn validate(message: &Self::Message) -> Result<(), ValidationError> {
        if message.is_null() {
            return Err(ValidationError::InvalidValue(
                "Message cannot be null".to_string()
            ));
        }
        Ok(())
    }
}

JsonSchema implements both private::Sealed (which we can do because we’re in the same module) and MessageSchema. The implementation uses serde_json for parsing and serialization and validates that JSON values aren’t null.

Let’s add a text schema for plain String data:

pub struct TextSchema;

impl private::Sealed for TextSchema {}

impl MessageSchema for TextSchema {
    type Message = String;

    fn serialize(message: &Self::Message) -> Vec<u8> {
        message.as_bytes().to_vec()
    }

    fn deserialize(bytes: &[u8]) -> Result<Self::Message, SchemaError> {
        String::from_utf8(bytes.to_vec())
            .map_err(|e| SchemaError::DeserializationFailed(e.to_string()))
    }

    fn schema_id() -> &'static str {
        "text_v1"
    }

    fn validate(message: &Self::Message) -> Result<(), ValidationError> {
        if message.is_empty() {
            return Err(ValidationError::FieldRequired(
                "message content".to_string()
            ));
        }
        Ok(())
    }
}

TextSchema handles plain UTF-8 text. It deserializes byte slices into strings and validates that they aren’t empty. Both schema implementations are within our module, so both can implement private::Sealed.

External crates can use JsonSchema and TextSchema, but cannot create their own schema implementations; the sealed traits pattern prevents it.

The sealed traits pattern leverages Rust’s module privacy: private::Sealed is public within the module, but the sealed module itself is private. External crates can see and use MessageSchema, but cannot implement Sealed, which means they cannot implement MessageSchema.

Type-safe messages with sealed schemas

Now, we can combine sealed schemas with the type-safe message patterns we’ve developed. TypedMessage will be generic over the schema type, ensuring that messages and their schemas are always compatible. The sealed traits pattern prevents external code from creating incompatible schema implementations:

use std::marker::PhantomData;
use crate::schema::MessageSchema;
use crate::types::*;

#[derive(Debug, Clone)]
pub struct TypedMessage<S: MessageSchema> {
    pub id: MessageId,
    pub content: S::Message,
    pub schema_type: PhantomData<S>,
}

impl<S: MessageSchema> TypedMessage<S> {
    pub fn new(id: MessageId, content: S::Message) -> Result<Self, ValidationError> {
        S::validate(&content)?;

        Ok(TypedMessage {
            id,
            content,
            schema_type: PhantomData,
        })
    }

    pub fn to_bytes(&self) -> Vec<u8> {
        S::serialize(&self.content)
    }

    pub fn schema_id(&self) -> &'static str {
        S::schema_id()
    }
}

TypedMessage is generic over the S schema type, where S implements MessageSchema. The content field has the S::Message type, which is the associated type we defined in the message schema trait for the type of our message payload. When we wrote our trait implementations, we defined concrete types for MessageSchema::Message: serde_json::Value for TypedMessage<JsonSchema> and String for TypedMessage<TextSchema>. The schema_type field uses PhantomData to track the schema type at compile time without storing anything at runtime. The new constructor validates the content using the schema before creating the message.

We can extend TypedMessage with additional methods for parsing raw data. The following shows how you might add a parse method (this extension is not in the base repository, but demonstrates the pattern):

impl<S: MessageSchema> TypedMessage<S> {
    pub fn parse(
        id: MessageId,
        raw_data: &[u8],
    ) -> Result<Self, S::Error> {
        let content = S::deserialize(raw_data)?;
        Ok(Self {
            id,
            content,
            schema_type: PhantomData,
        })
    }

    pub fn serialize(&self) -> Vec<u8> {
        S::serialize(&self.content)
    }
}

pub type JsonMessage = TypedMessage<JsonSchema>;
pub type TextMessage = TypedMessage<TextSchema>;

The extended parse method takes raw bytes and uses the schema’s deserializer to construct typed content. If parsing fails, the error type is S::Error, which is serde_json::Error for JSON or std::str::Utf8Error for text. The serialize method converts the content back into bytes. This demonstrates how you can build on the sealed trait foundation to add richer functionality.

The type aliases provide convenient names: JsonMessage and TextMessage are clearer than writing TypedMessage<JsonSchema> everywhere. These messages are both type-safe (wrong schema = compile error) and sealed (can’t add new schemas externally).

Thus, the sealed traits pattern provides controlled extensibility. We maintain the ability to evolve our trait interface while providing compile-time guarantees about which implementations exist.

Summary

In this deep-dive, we explored four patterns that leverage Rust’s type system, each with established origins in programming language research and practice.

The NewType pattern comes from functional programming languages, particularly Haskell, where it has been a formal language feature since the 1990s. Rust’s contribution is making the pattern zero-cost: type safety is enforced at compile time and optimized away entirely at runtime, making it practical for systems programming, where performance matters.

Parse, don’t validate was articulated by Alexis King in 2019, building on ideas from functional programming and type theory. Rust’s type system, ownership model, and Result type make this principle natural to apply, turning boundary validation into type-level guarantees that persist throughout the program.

The TypeState pattern originated in formal methods research, specifically in Strom and Yemini’s 1986 work on program verification. Rust’s generic types with PhantomData, zero-cost abstractions, and ownership system make this academic concept practical for everyday systems programming, providing compile-time state machine verification with no runtime overhead.

Sealed traits draw from similar concepts in Java (final classes), Kotlin (sealed classes), and Scala (sealed traits). Rust implements the pattern through its module privacy system, providing a natural fit with the language’s existing visibility rules.

What these patterns demonstrate is Rust’s ability to take established ideas from type theory, functional programming, and formal methods and make them practical for systems programming. The combination of strong static typing, zero-cost abstractions, and memory safety without garbage collection creates opportunities to apply patterns that were previously either too expensive or impractical in systems languages.

Our enhanced Samsa system now leverages decades of programming language research while maintaining the performance characteristics necessary for systems programming. This synthesis, applying proven patterns in a performant, safe environment, represents one of Rust’s key contributions to the programming language space.

The book’s next chapter explores patterns from functional programming, including function composition pipelines, generics as type classes, pattern matching techniques, and closures as architectural components.

More from Evan Williams

As experienced engineers, we already know how to build reliable systems. This deep-dive isn’t about reinventing those principles, but about adapting them for the unique, messy challenges LLMs throw at us. Consider this an architect’s playbook that outlines the new patterns needed to meet these challenges.

This deep-dive is Chapter 2 of System Design for the LLM Era by Sampriti Mitra, shared with permission from Packt, with all rights remaining with the publisher and no reproduction or redistribution without written consent. You can get the full book here.

Featured workshop: Loop Engineering for AI Agents

A 4-hour hands-on workshop on Loop Engineering, designing agent workflows with Claude Code, SDD, and MCP that verify their own work and stay observable in production.

Use DEEPENG40 for 40% discount.

Register here

Designing for resilience and reliability

Our system’s stability is now tied to an external API that is slower, more expensive, and less predictable than any database or microservice calls. The primary goal is to decouple the application’s health from the provider’s health. Two types of pattern are especially valuable in this regard: the GenAI service pattern (or LLM gateway pattern) and the circuit breaker pattern.

Pattern: the GenAI service or LLM gateway

When we first start building with LLMs, our instinct is to treat them like any other third-party API. We install the SDK, generate an API key, and make the call directly from our application code.

It feels fast. It feels efficient. We write a Python function, import openai, and we are shipping features in minutes.

The architecture looks like this:

While this works for a weekend hackathon, it creates a high-coupling, low-cohesion architecture in production with the following issues:

Stating the problem more formally now:

Problem: Our services should not be calling OpenAI, Anthropic, or Google directly. This creates high-coupling, high maintenance problems.

Solution: To solve this, we borrow a pattern from traditional microservices: the API gateway. We stop treating LLMs as external vendors and start treating them as a unified internal resource.

Implement a single, centralized LLM gateway. This is a microservice that acts as the only entry point for all LLM calls. Our services talk only to this gateway using a single, unified API format. The gateway handles the messy details of talking to the outside world.

This pattern brings some significant benefits:

Pattern: circuit breakers with tiered fallbacks

An LLM provider might be slow, down, or just returning bad data. Simply retrying a failed call (like you would for a 503 on an external service) is often the wrong move.

The solution is to combine the circuit breaker pattern with tiered fallbacks following a three-stage model:

1. Monitor: The GenAI Service monitors the health (latency, error rate) of each model provider.

2. Trip: If a primary model (e.g. GPT-5) exceeds a failure threshold, the circuit opens.

3. Reroute: All subsequent requests are immediately and automatically rerouted to a backup model.

For example, we might implement the following tiered fallback strategy:

Tier 1: GPT-5 (high-cost, high-reasoning)

Tier 2 (fallback): Claude 3 Haiku (medium-cost, fast)

Tier 3 (fallback): Llama 3 8B (locally hosted, free, less smart)

Tier 4 (final fallback): Return a cached good enough response or a graceful error message to the client UI: ‘Our AI assistant is at high capacity, please try again in a moment’

We cannot leave the circuit open forever. We need recovery logic to close the circuit that includes a half-open state:

Sleep: After the circuit trips, wait for a defined cooldown (e.g. 30 seconds).

Probe: Allow a single canary request to pass through to the primary provider.

Reset: If the canary succeeds, close the circuit and resume full traffic. If it fails, restart the cooldown.

For our retry strategy we have two options: interactive and asynchronous:

Designing for low latency

LLM inference is fundamentally slow. A user requesting a search result expects a response in < 500 ms, but an LLM might take 5–10 seconds to generate a full answer. We cannot change the speed of inference, but we can architect around it. Hybrid processing (synchronous vs. asynchronous), response streaming and caching patterns can help us.

Pattern: hybrid processing (synchronous vs. asynchronous)

Given that we cannot block a user’s web request for 10 seconds while waiting for an LLM, one option is to separate the workloads. This is the most important latency-saving pattern.

Synchronous path (< 2 s): For immediate, low-latency needs. These are tasks that must be fast, like code complete or a real-time e-commerce search. These paths should use fast, cheap models or rely heavily on caching.

Asynchronous path (> 10 s): For high-latency, long-running tasks use a message queue (like Kafka or SQS) to decouple the initial request from the actual LLM work. The client receives an immediate ‘202 Accepted’ response. For example, use when generating an AI-powered report, in agentic workflows, or for offline content generation. The client can poll for the result or receive it via a WebSocket/callback.

Pattern: response streaming

Even a fast 3-second response feels slow if the user is staring at a loading spinner, so if possible stream the response token-by-token. Once the model generates the first word, send it to the client. This dramatically improves perceived latency by lowering the time-to-first-token (TTFT). This is a non-negotiable pattern for any conversational or chat application.

How it works

We use SSE (server-sent events) over standard HTTP. SSE is unidirectional (server -> client) and runs over standard HTTP/2, making it firewall-friendly and easy to implement.

Client: Opens a persistent connection.

Server: Instead of returning a JSON object, it returns a generator (an iterator)

Header: The server must set content-type: text/event-stream

Format: Data is sent in chunks prefixed ‘data:’:

Pattern: caching strategies

LLM calls are slow and expensive, while cache hits are the fastest, cheapest LLM calls you can make. Therefore implement a multi-level caching strategy:

Level 1: exact match

Level 2: semantic match

Level 3: proactive caching

Pattern: coalesce caching

During high-traffic events, thousands of users might ask the exact same question simultaneously. A standard cache misses the first time for everyone, causing a stampede of identical requests to the LLM.

The solution is middleware that identifies identical in-flight requests. It pauses subsequent requests, waits for the first request to complete, and then serves that single LLM response to all waiting users. This reduces load on the provider by orders of magnitude.

Designing for cost optimization

LLMs introduce a new, variable, and unbounded operational cost (COGS – cost of goods sold). A single complex query can cost dollars. Architectural decisions are now financial decisions. Important in this context are the model router, dynamic traffic control and prompt engineering and compression patterns.

Pattern: the model router

Not all tasks require the smartest (and most expensive) model. Using GPT-5 for a simple grammar check is like booking a helicopter to avoid traffic. By implementing a rule engine within your LLM Gateway to act as a model router you can dynamically route requests to the cheapest model that is good enough for the task.

Example rules:

IF task_type == 'simple_grammar_check' THEN route_to 'local_llama_8b'.
IF task_type == 'complex reasoning' AND user_tier == 'premium' THEN route_to 'GPT-5'.

Pattern: dynamic traffic control (utilization-based routing)

A static rule for traffic routing, e.g. IF task == complex THEN GPT-5, can cause bottlenecks during traffic spikes. In this case we can use a production-grade router that acts as a traffic controller, implementing load shedding if the primary model’s latency breaches the P99 SLA (e.g. > 2s) due to provider congestion. In those circumstances the router proactively shifts traffic to the faster or cheaper model, even for complex tasks. A cost ceiling can be established by enforcing a hard cap on tokens. If a prompt exceeds a threshold (e.g. 8k tokens), force-route to a lower-cost model to prevent a single query causing cost regression.

Pattern: prompt engineering and compression

Cost is based on the number of input and output tokens; large prompts are expensive. Therefore treat your prompt context as a cost to be optimized. Before sending a large document (e.g. 50 pages of chat history) to an LLM, use a cheaper, faster model to summarize or compress it first.

Designing for grounding and data management

LLMs hallucinate (invent facts) and have knowledge cutoffs (their training data is stale). How can we build a reliable enterprise application on this foundation? Retrieval-augmented generation (RAG) approaches are a fundamental pattern for enabling modern LLM applications to address these fundamental issues.

Pattern: retrieval-augmented generation (RAG)

Not uncommonly we need the LLM to answer questions about private, real-time, or domain-specific data. Instead of asking the LLM a question, the solution is to tell it the answer. As the RAG name suggests, there are three principal aspects to consider:

Retrieve: When a user asks, ‘What’s the status of order #123?’, you first query the database to get the order details.

Augment: You augment the prompt with this retrieved data.

Generate: You instruct the LLM – ‘Based only on the following context, generate a friendly response’.

Example context: {order_details_json}

Pattern: the ingestion pipeline

To build the retrieval component of a RAG application, we must prepare the knowledge base (documents, tickets, etc.) for retrieval. That means creating an ingestion pipeline, typically as an asynchronous, scalable job (e.g. using Spark). This pipeline:

1. Chunks large documents into small, semantically meaningful pieces

2. Embeds each chunk by calling an embedding model API

3. Stores the chunk and its corresponding vector in a vector database (e.g. OpenSearch, Pinecone, or pg_vector [with postgres])

Pattern: hybrid RAG

RAG using just vector search (or even simple term search) is not enough for complex, interconnected data. It’s good at finding similar content, but bad at traversing relationships. Hybrid RAG approaches such as GraphRAG represent an advanced RAG pattern where the ingestion pipeline populates both a vector database (for semantic similarity) and a knowledge graph (e.g. Neptune or Neo4j) for structured relationships.

The RAG orchestrator then queries both systems to build a much richer, more accurate context, as a customer support agent system demonstrates well.

Pattern: function calling (tool usage)

To overcome the problem that LLMs are generally bad at (for example) math and cannot interact with the outside world, we ask it to decide which tool to use. We provide a schema of tools (e.g. get_weather(city)), and the LLM outputs a structured JSON object requesting that function. The application layer executes the code and feeds the result back to the LLM.

Designing for testability and observability

Testability for AI-powered systems is crucial in order to maintain the quality of response expected from the system, but for non-deterministic systems it is not trivial to write unit tests that would assert on a given known output.

We use a pattern of using an LLM as a judge by creating a golden dataset, which has input with their ideal responses, and using that dataset, the LLM judge is trained. This LLM is later used to evaluate the quality of response returned by the system against the ideal response.

Problem: How do you write a unit test for a system that is non-deterministic?

Something like assert(response) == “expected_string” will fail constantly.

Pattern: golden datasets

We need a reliable way to catch regressions in AI quality, and one way is to create a ‘golden dataset’ of 50–100 representative inputs and their ideal outputs. In your CI/CD pipeline, run your system against this set to ensure that prompt changes or model upgrades haven’t broken core functionality.

Pattern: LLM-as-a-Judge

How do you assert the quality of a golden set test at scale? You can’t manually review 100 responses on every build. What you can do, however, is use a powerful LLM like GPT-5 as an evaluator or judge. You feed the judge the original prompt, the golden answer, and your system’s actual response, and then ask the Judge to score the actual response from 1 to 5 on metrics like accuracy, groundedness, and tone. This gives you a quantifiable quality metric you can track over time.

Pattern: new observability metrics

Existing performance dashboards (CPU, RAM, 5xx errors, etc.) are insufficient in the LLM era. We need to add a new layer of monitoring focused on the LLM itself, for example:

Cost: Track Cost_Per_Query and Total_Cost_Per_User. Set alerts for cost spikes.

Performance: Monitor P99 time to first token (TTFT) and tokens per second (TPS).

Provider health: Monitor 429_Rate_Limit_Errors and 5xx_Server_Errors per provider to feed your circuit breakers.

Quality: Track your LLM-as-a-Judge scores. Monitor the escalation rate – this measures the percentage of conversations where the AI fails to resolve the issue, forcing the system to transfer the work to a human (for example in case of customer support agents). A spike in this metric indicates a drop in model quality.

Designing for security and trust

Treating an LLM as a simple API call is naive and dangerous. It’s a non-deterministic component that you are inviting inside your trusted system. We must architect our systems to defend against a new class of vulnerabilities. A variety of patterns can help us in this, including the following. In this section we summarize the main security patterns, the threats they address and the mitigations they provide.

Pattern: mitigating prompt injection

Threat: Attackers trick an LLM into ignoring its original instructions by embedding malicious commands in prompts, causing it to perform unintended actions.

Mitigation

Instruction/data separation: Clearly separate trusted instructions from untrusted data from the user. Use role-based API structures (system, user) and wrap all user input in clear delimiters like <>.

Input filtering: Use a second, simpler, faster LLM to classify the intent of a user’s prompt. If it detects a likely attack, reject it before it reaches your primary model.

Output filtering: Always validate the LLM’s response. If it contains any system prompt text or suspicious keywords, block it.

Pattern: secure output handling

Threat: Insecure output handling occurs when an LLM’s outputs are not properly sanitized before being used, potentially leading to attacks like cross-site scripting (XSS).

Mitigation

Treat as untrusted: Never eval() code output.

Sanitize and encode: If the output is HTML, sanitize it. If it’s text for a web page, encode it to prevent XSS.

Validate: If you expect JSON, parse it in a try/catch block and validate its schema.

Parameterize: If the LLM helps build a SQL query, have it generate the parameters for a pre-defined, parameterized query you control. Never execute a raw SQL string from an LLM.

Pattern: mitigating excessive agency

Threat: This happens when an LLM is given too much control and makes unauthorized decisions or actions without human oversight.

Mitigation

Dynamic permissions: Do not give the LLM agent a static set of all possible tools.

Plan-approve-execute: Implement a multi-step loop. The LLM proposes a plan. Your code approves this plan. Only then do you execute it with a scoped-down client.

Human-in-the-loop (HITL): For high-impact actions (e.g. ‘delete database’, ‘refund customer’), always require explicit human approval.

Pattern: mitigating sensitive information disclosure

Threat: The model may inadvertently reveal confidential data or personally identifiable information (PII) that was present in its training data.

Mitigation (data hygiene)

PII/data scrubbing: Aggressively sanitize all data before it is used for training or RAG ingestion.

Zero-retention policies: For ultra-sensitive data (like user code), process it in-memory and discard it immediately. Do not store it.

Tenant-level RAG filtering: This is a critical architectural pattern. All RAG queries must include a filter for tenant_id or user_id. Never perform a vector search on the entire database and hope the LLM picks the right data.

Pattern: preventing model denial of service (MDoS)

Threat: Attackers overload the LLM with resource-intensive requests, slowing it down or making it unavailable.

Mitigation

API rate limiting: Enforce strict per-user and per-IP rate limits at the API gateway

Input validation: Reject queries that are obviously abusive (e.g. a 50,000-token prompt)

Cost-based throttling: Implement logic to monitor the cost of a user’s queries in real-time. If a single user is incurring high costs, temporarily throttle their access. We would want to return a 429 in case of user spam/abuse or redirect to a lower model in case user hit their budget.

Here’s an example implementation:

from fastapi import HTTPException

def route_request(user, prompt):
    # 1. HARD LIMIT CHECK (Redis Counter)
    current_rate = redis.get(f"rate:{user.id}")
    if current_rate > 50:
        # Scenario A: Abuse -> Hard Stop
        raise HTTPException(status_code=429, detail="Rate limit exceeded.")

    # 2. SOFT BUDGET CHECK (DB Query)
    daily_spend = db.get_spend(user.id)
    budget_limit = 10.00  # $10 limit
    if daily_spend > budget_limit:
        # Scenario B: Over Budget -> Downgrade (Soft Throttle)
        # We don't fail; we just swap the model
        print(f"User {user.id} over budget. Downgrading to Tier 3.")
        return call_llm(model="llama-3-8b", prompt=prompt)

    # 3. NORMAL PATH
    return call_llm(model="gpt-4", prompt=prompt)

Pattern: preventing data poisoning

Threat: Malicious actors tamper with an LLM’s training data to corrupt its behavior, leading to biased or incorrect outputs.

Mitigation (ingestion control):

Trusted sources: Only use and ingest data from known, trusted sources.

Data lineage: Track the origin of all data used for training or RAG.

HITL review: For fine-tuning data, use human experts to review and validate the datasets before training.

Pattern: mitigating supply chain and insecure plugin vulnerabilities

Threat: The security of an LLM can be compromised through third-party services, plugins, or datasets that are themselves vulnerable. This includes insecure plugin design, which can be exploited for attacks like SQL injection.

Mitigation

Minimize functionality: Any plugin or tool given to the LLM should have the absolute minimal functionality needed (e.g. read_email, not delete_email).

Validate plugin inputs: Treat all data passed to a plugin as untrusted. Sanitize it to prevent injection attacks within the plugin itself.

Vulnerability scanning: Regularly scan all third-party libraries, containers, and models for known vulnerabilities.

Use the genAI service / LLM gateway: Your gateway allows you to quickly replace a provider or model that is found to be compromised.

Engineering for production

We cannot manage what we do not track. Beyond standard APM (application performance monitoring), we must track:

TTFT (time to first token): The perceived latency. How long until the user sees the first character? High TTFT kills engagement.

TPOT (time per output token): The generation speed. If this is high, the model is too heavy or the provider is overloaded.

Context utilization: Is the context window filled? Use the model optimized for token usage for the use case for cost effectiveness.

Training with test data

In earlier sections we have discussed golden datasets – inputs with their ideal outputs used for training the LLM as a judge. Let’s now look at how we can procure this ideal data for different use cases.

Sourcing datasets

Testing and training for code complete

Randomly remove code blocks, functions, classes from the code. Start typing signatures of missing classes and functions. Compare the IDE code complete suggestions with the actual classes and functions to measure accuracy and semantic correctness. Iteratively tune the LLM model prompts based on error cases and coverage gaps identified.

Testing for chat responses

Clone the repo removing all the comments and docs. Ask the system to explain code blocks, functions, classes and compare against the original repo’s docs, README files or comments.

Testing for agentic workflow and tasks

Assign real world tasks like adding documentation, writing test cases or refactoring to the agent. Combine multiple tasks in scenarios.

Automatically check the code and tests and documentation for correctness and style.

Integrating these tests into CI/CDs helps keep models and prompts updated with evolving repos.

Pattern: quantitative evaluation testing

The goal of this pattern is to move beyond vibes and qualitative observations to a measurable correctness percentage.

The problem is that agentic flows are multi-step and non-deterministic. Traditional pass/fail unit tests often fail to capture the nuance of a complex task that is mostly correct but slightly off in tone or style.

To tackle this distinctive character we assign a numerical score to every agentic run by breaking the output into weighted criteria. This allows us to track an evaluation success rate, the percentage of tasks successfully completed to the golden standard.

To calculate the success rate we define weights to the different output facets that reflect their relative importance (e.g. logic: 50%, syntax: 30%, documentation: 20%).

Then we execute the agent across a golden dataset of at least 50 representative tasks, and score according to the following metric:

Figure 2.13 shows the overall evaluation process:

When to use it: Use this in your CI/CD pipeline. If a prompt change or model upgrade causes the correctness percentage to drop below a defined threshold (e.g. 90%) then the deployment should be automatically blocked to prevent quality regression.

Mutation testing

Add logic and syntax errors to your code. Evaluate whether the system can identify and fix these errors.

Negative prompt testing

Give confusing or risky actions like ‘Delete all databases’. Evaluate whether the system is able to flag it as a risk and ignore/refuse or ask the user for further clarification.

Respecting user privacy

We want to be able to use an LLM-powered system for code completions, reviews, generation etc. without leaking our codebase to the model. Techniques for ensuring that user privacy is maintained while the data is still served from the model include:

Summary

The patterns we’ve just looked at, from the GenAI service and async queues to RAG and LLM-as-a-Judge, form an architect’s playbook for building production-grade, AI-powered systems. In the rest of the book we examine four case studies that utilize this playbook.

Packt’s flagship two-day virtual summit

AI is reshaping software architecture, putting new demands on scalability, governance, reliability, and observability. ARC 2026 brings together architects, CTOs, and AI practitioners for keynotes, panels, and workshops on agentic system design, modernizing enterprise apps for AI, and building governable, observable AI systems.

The lineup includes Dr. Ali Arsanjani (Director, Applied AI, Google), David Ping (Head of Solutions Architecture, AWS), and Chi Wang (Google DeepMind), alongside other notable leaders distilling practical insights.

🗓️ 25 to 26 July, 10:30 am ET

Use code DEEPENG50 for 50% off the early bird price.

Reserve your spot

✍️ From the editor’s desk,

Welcome to the 54th issue of Deep Engineering!

Site reliability engineering is already agentic, and agents are moving from the edges of the incident into its core. New Relic unveiled Autopilot at its New Relic NOW event on June 23, an automated SRE agent that triages incidents, identifies root causes, and scopes remediations the moment an alert fires. “Operations are going headless,” said New Relic’s Head of AI, Camden Swita, describing agents that pull what they need through APIs and act.

That is exactly when governance starts to matter more than speed. Once an agent can decide and push a remediation, the reliability question is no longer whether automation moves fast but whether its actions stay bounded. Faster, wider change across a tightly coupled system amplifies trouble as easily as it absorbs it, and the retry, scaling, and routing reflexes that steady a healthy platform can drive a cascade once it is already stressed. Tellingly, even these launches arrive wrapped in the language of guardrails and human review.

The engineers who feel that trade-off most work where a wrong automated action shows up immediately on the balance sheet. Sibasis Padhi, a Staff Software Engineer at Walmart Global Tech, builds large-scale financial platforms of exactly that kind. In his Deep Engineering article “Autonomic Governance for Agentic Systems,” he argues that reliability engineering must evolve from managing distributed systems to governing the automation that manages them.

Padhi makes the case that the fix is not less automation but automation you can bound, audit, and reverse.

Let’s get started.

Featured Newsletter: The Hustling Engineer

Read by 25,000+ software engineers, The Hustling Engineer covers career growth, AI, interviews, and productivity. Every week you get actionable insights, engineering deep dives, and lessons from top tech companies.

📩 Get one actionable lesson every week to accelerate your tech career.

→ Subscribe to The Hustling Engineer

🧠 Expert Insight

Autonomic Governance for Agentic Systems

by Sibasis Padhi

The incident often starts small. Usually one dependency slows, tail latency grows, and a few requests time out. And then the platform does what we trained it to do. All the while, customers automatically retry their failed requests, and very quickly there are far more retry attempts than real new customers trying to use the service. But the autoscaling system sees the growing queue and reads it as exploding traffic, so it adds more servers, often in the wrong place, which sends even more work to the spot that is already overloaded.

This eats into the bottom line and the return on engineering investment as the bill keeps mounting while the system handles less real work. Circuit breakers finally trip and try to move traffic elsewhere, but those other paths were never built to absorb a sudden flood.

Everything might still look like it is working on paper, but the whole situation keeps getting worse. Now compound that by adding agentic AI to operations, systems that recommend or execute operational actions, which increases the speed and breadth of change under conditions that are already chaotic.

AI-driven operational automation can increase fragility when it increases the rate, scope, or coupling of production actions without bounded actuation, explicit constraints, and auditability. Under governance, the same automation reduces fragility by enforcing safety envelopes and reversible decision paths.

This is not an argument against automation or AI. It is an argument that reliability engineering must evolve from managing distributed systems to governing the automation that manages them, especially under simultaneous SLO, cost, and compliance constraints. I call this discipline Autonomic Reliability Governance (ARG). ARG is not more automation. But governed automation that is policy-constrained, auditable, and rollback-capable by design. It ensures that AI-assisted or agentic operational decisions cannot escalate into amplification engines under stress.

Why reliability automation amplifies in microservice ecosystems

In microservice systems, outages rarely come from a single broken component. They emerge from how services interact, through feedback loops, dependencies, and cascading effects. Mechanisms that improve reliability in normal conditions can unintentionally amplify problems when the system is already under stress, which is the feedback behavior Karl Johan Åström and Richard Murray formalize in Feedback Systems (Princeton University Press, 2008).

Retries multiply load under stress

Retries improve success rates when failures are transient and capacity is available. Under degradation, retries become a multiplier, the dynamic Jeffrey Dean and Luis André Barroso describe in “The Tail at Scale” (Communications of the ACM, February 2013).

This is how a minor regression becomes a self-inflicted flood. It can resemble a resource exhaustion event even when there is no attacker. The mechanism is simple. Unbounded retries consume the remaining capacity of an already-constrained dependency. The root issue is not that retries are bad. The issue is that unbounded retries are a powerful actuator that must be governed by system-level constraints such as SLO, cost, and blast radius.

Autoscaling is a blunt actuator fed by ambiguous signals

Autoscaling adds or removes capacity based on signals like CPU usage, request rate, or queue length. During incidents those signals mislead. Queue depth may grow because a dependency is slow, not because demand has grown. CPU may spike from retries and timeouts, not real workload. Request rates may rise simply because retried calls look like new traffic. A reactive autoscaler then adds capacity in the wrong place, driving up cost while putting even more pressure on the real bottleneck. The result is a system that becomes less stable and more expensive at exactly the moment it needs to recover.

Circuit breakers carry shock-wave potential

Circuit breakers are necessary. Without system-level coordination they create abrupt traffic shocks. If multiple clients trip simultaneously, alternate paths overload and create secondary failures that appear unrelated to the original degradation.

The common pattern is unbounded actuation in a coupled system

Retries, autoscaling, and circuit breakers are not mistakes. They are essential reliability tools. Problems arise when they operate like independent reflexes, without coordination or limits. When nothing controls how often they act, how broadly their actions affect the system, how easily changes can be reversed, or how clearly decisions can be explained, they unintentionally amplify failures. Managing this was already difficult with manually written rules. It becomes even more critical when AI agents make or trigger operational decisions.

Why agentic operations change the physics

Traditional automation such as scripts, thresholds, and fixed policies is limited in scope and relatively legible. Agentic systems expand both capability and risk because they tend to:

1. Increase velocity: propose actions faster than humans can validate non-local effects.

2. Increase scope: coordinate actions across services, clusters, and regions.

3. Optimize proxies: Charles Goodhart warned in 1975 that a statistical regularity tends to collapse once it is used as a control target, a caution later popularized as “when a measure becomes a target, it ceases to be a good measure” (”Problems of Monetary Management: The U.K. Experience,” 1975). Optimizing latency or cost in isolation can degrade reliability unless constraints are explicit.

4. Operate under uncertainty: incidents produce ambiguous signals, and partial telemetry invites misdiagnosis and overcorrection.

5. Reduce legibility unless designed otherwise: without decision traces, you cannot reconstruct why the system acted or demonstrate compliance.

So the risk is not AI in isolation. The risk is unbounded actuation under uncertainty. ARG is the discipline designed to make agentic operations governable.

A practical lens on amplification

To manage automation safely, teams need a clear way to recognize when helpful mechanisms start making things worse. Amplification is that lens. A reliability mechanism helps when it improves outcomes without adding too much extra load. It becomes harmful when it multiplies load, volatility, or system-wide effects beyond what the platform can handle. In practice, retries multiply request volume, autoscaling multiplies capacity changes and cost, traffic shifts increase pressure on dependencies, and aggressive fixes create configuration churn. ARG is essentially the discipline of keeping these amplification effects under control.

Autonomic Reliability Governance governs the automation itself

The notion that systems can manage themselves is not new. Jeffrey Kephart and David Chess articulated that aspiration more than two decades ago in “The Vision of Autonomic Computing” (IEEE Computer, January 2003). What is new is the velocity and scope of actuation introduced by agentic AI. Autonomic Reliability Governance is the discipline of designing and operating bounded, auditable, and reversible automation, including agentic operational decision layers, under explicit constraints.

• SLO constraints: latency ceilings, availability targets, error budgets.

• Cost constraints: budget caps, unit-economics bounds, runaway scaling prevention.

• Compliance constraints: auditability of decisions and actions.

ARG is not a product. It is an operating model, automation you can trust because it is governable. ARG treats operational actuation, human or agentic, as a first-class risk surface with explicit safety envelopes..

Governance primitives that make agentic operations safe

ARG becomes real when it is implemented through governance primitives that align with existing reliability practice such as SLOs, error budgets, and progressive rollout, while controlling actuation velocity and blast radius.

1) Actuation budgets rate-limit change, not only traffic

Most systems rate-limit user requests. Few rate-limit operational change. ARG introduces actuation budgets, limits on how frequently high-impact actions can occur. Scaling shifts, retry escalations, routing changes, and configuration flips consume tokens from a constrained budget. When the budget is exhausted, automation may still observe and recommend, but it cannot repeatedly perturb the system. When instability rises, slow the actuators before adding more intelligence.

2) Blast radius governance scopes every action

The difference between a safe fix and an outage multiplier is often scope. ARG constrains blast radius using canaries, segmentation across cells and regions, tiered permissions, and scoped rollouts with automatic halt conditions. Even correct actions can be wrong if applied globally in one step. Autonomy must always be localized before it is generalized.

Continue reading

Continue reading - the seven governance primitives that turn ARG from principle into practice, how to validate governed automation without touching production or private data, and why agentic operations win by becoming more governable, not more intelligent.

🛠️ Tool of the Week

Argo Rollouts - a progressive delivery controller for Kubernetes

Highlights:

• Bounds blast radius by exposing a change to a small traffic slice first, closing the gap that standard rolling updates leave open where nothing limits exposure or triggers an automated rollback on failure.

• Makes reversibility the default, since a rollback shifts traffic back to the previous version instead of triggering a fresh deploy.

• Gates promotion on evidence rather than a timer, aborting automatically when success rate or latency crosses a threshold, which mirrors the confidence gates in the feature.

• Fits GitOps workflows, where an automatic rollback surfaces as divergence from declared state and prompts investigation before anyone retries.

Learn more about Argo Rollouts

📎 Tech Briefs

• Google makes Model Armor generally available for Agent Gateway - Model Armor reaches general availability for Agent Gateway, screening agent prompts and responses against content guardrails.

• Datadog extends Bits AI to autonomous remediation at DASH - Bits AI now detects and remediates autonomously within predefined guardrails, and Agent Console tracks agent actions.

• OpenTelemetry graduates from the CNCF - OpenTelemetry graduates from the CNCF, cementing the vendor-neutral telemetry standard that grounds agent decisions in trusted context.

• Microsoft makes Azure SRE Agent generally available - Azure SRE Agent is generally available, with Review and Autonomous run modes gating remediation behind human approval.

• Grafana previews AI Observability in Grafana Cloud - Grafana previews AI Observability, treating agent sessions as first-class telemetry and alerting on policy violations and anomalies.

That’s all for today. Thank you for reading this issue of Deep Engineering.

We’ll be back next week with more expert-led content.

Keep building,

Saqib Jan

Editor-in-Chief, Deep Engineering

If your company wants to reach senior developers, software engineers, and technical decision-makers, speak to us about partnering with Deep Engineering.