This hands-on masterclass takes you from LangGraph fundamentals to supervisor and hierarchical multi-agent systems, with live debugging in LangSmith throughout.

Deep Engineering readers save 50% with code - DEEPENG50.

Register here →

✍️ From the editor’s desk,

Welcome to the 58th issue of Deep Engineering!

On August 3, NTT announced that it has signed a capital and business alliance with OptQC, the University of Tokyo spinout building optical quantum processors, with both companies aiming at a fault-tolerant machine of one million qubits. The announcement from NTT lays out a phased roadmap. The companies aim to complete the system architecture and key component technologies by fiscal 2027 alongside a practical 10,000 qubit system, then begin verification work in fiscal 2028 and deliver a platform for running optical and classical machines together the year after.

One million is the largest number the field has yet attached to a headline, and it arrives in a year when the reported metric already shifted once. Through the first half of 2026 vendors moved from physical qubit counts to logical qubit counts as the figure worth announcing. Both are real results, and both leave out the properties that decide what a machine can actually compute, which is where this issue picks up.

Sebastian Hassinger has read claims like these from inside the companies making them, first on the IBM Quantum team and later leading go to market for AWS Quantum Technologies. He wrote The New Quantum Era for readers without a physics background, and today he walks us through which numbers carry the information and which ones do not.

You can watch the full session or read the transcript here.

Let’s get started.

Agent-written TLA+

An agent wrote our TLA+ spec. The model checker explored 14.3M states and caught a real race.

Read the write-up

Expert Insight

Qubit Count Measures Register Size, Not Capability

by with

Three logical qubit results landed in the first half of 2026, and in each one the number worth reading is not the number in the headline. QuEra published 96 logical qubits encoded across 448 neutral atoms, a ratio near five to one. Quantinuum reported 48 logical qubits drawn from 98 trapped ions, closer to two to one. IBM’s published fault tolerance roadmap targets 200 logical qubits from roughly 10,000 physical ones by 2029, a ratio near fifty to one.

Those ratios differ by an order of magnitude because the underlying codes differ, and the choice of code decides whether a logical qubit corrects errors or only detects them. A count reported on its own collapses all of that into a single integer, which is why the integer tells you very little about what the machine computes.

Sebastian Hassinger worked on the IBM Quantum team and later led go to market for AWS Quantum Technologies, and he wrote The New Quantum Era to give engineers without a physics background enough grounding to read results like these directly. During our interview when I asked him what actually carries information in a milestone result, he began by taking apart the metric the field has reported for a decade. “Qubit count is effectively the register size of that computer,” he said. Then came the harder line, aimed at a claim the field now repeats freely. “Anytime you hear somebody saying quantum computing is just a matter of engineering now, be suspicious of that person’s claims.”

Register size bounds information, not computation

The clearest demonstration that a raw count says little about capability comes from IBM’s own hardware history. And Hassinger was there for it. The technical roadmap produced a chip called Condor at just over a thousand qubits, which he describes as genuinely valuable for the research and fabrication effort it took to build. But it saw little use. Connectivity between qubits on the chip was low and the noise proved very difficult to manage, so researchers went back to the smaller machines in the 127 to 133 qubit range, which were more capable in practice.

The reason is architectural rather than numerical. Register size sets an upper bound on how much information you can load, and nothing beyond that. Hassinger points out that QuEra’s 256 qubit Aquila holds 256 bits at a time, which sounds unremarkable until you entangle those qubits and produce a state vector of two to the 256, a computational space you cannot physically recreate on classical hardware. The capability lives in the entanglement structure and in how well the problem maps onto it, so a machine with more qubits and worse connectivity computes less than a smaller machine with better ones.

That same logic now applies one level up. A logical qubit is an encoding, not a unit, and its value depends on the code family, the physical to logical ratio, and the error model the code assumes. Codes at distance two detect errors without correcting them, which is a different guarantee from correction, and the difference produced considerable argument when Microsoft and Quantinuum reported reliable logical qubits in 2024 using error detection with post-selection. Two systems reporting the same logical qubit count can therefore be doing categorically different things.

Code distance carries the information a count discards

Hassinger’s proposed substitute is quite specific, and it happens to be exactly the property that separates those cases. Fidelity and noise are what matter, he reasons, particularly the fidelity of one and two qubit gates, where two qubit operations mean entanglement. Those figures are hard to extract from a published result, so he offers a proxy that survives summarization.

Read the resilience of the error correction code, expressed as a distance or a d value. That is roughly how many errors the system absorbs before the encoded information collapses and the computation is lost, so a higher distance means a more resilient machine. He points to the Willow experiment as the useful reference, roughly a hundred physical qubits arranged in a surface code presenting as one logical qubit at distance seven. That snapshot carries what you need without the underlying gate fidelities, because the only two questions that determine what you can run are “how many logical qubits do I get, and how resilient is that error correction.”

Pair a logical qubit count with its code, its distance, and its encoding ratio and you have something you can reason about. Take the count alone and you have an integer that happens to increase.

Speculation hardens into certainty before it reaches you

There is a structural reason the public record runs ahead of the results, and it operates on the way from the lab to the summary rather than inside the science. Hassinger named the pressure that drives it, and he was unusually direct about where the gap opens.

“Since at least the beginning of the Q2B conferences put on by QCWare, there has been a recurring chorus demanding to know what are quantum computing’s use cases, how will it be useful for enterprises,” he told us. “Marketing can be tempted to take speculative ideas and present them as certainties, stretching the truth about a scientist’s speculation to reframe it as definitive. The other question is always when, so timelines are also something that marketing can take liberties with.”

Both distortions are directional, which makes them correctable. A researcher’s conditional loses its condition, and a scientific dependency acquires a date. Reading a result back through those two transformations usually recovers something close to the original claim.

Roadmaps model engineering determinism onto unsolved physics

The deeper issue Hassinger identifies is a category problem. A roadmap projects milestones one year out, three years, five years, and he is blunt about what kind of document that is. “That’s an engineering document,” he says, “and engineering is much more deterministic than the underlying scientific breakthroughs that are required to enable the engineering to deliver those milestones.”

Transduction is the concrete case. Superconducting qubits operate inside a dilution refrigerator near absolute zero, and a refrigerator has finite volume, so scaling past one fridge means entangling qubits across separate cryostats. That requires converting the quantum state to a photonic frequency used in telecom, carrying it over fiber as what the field calls flying qubits, then converting back at the far end. None of the known conversion methods delivers the fidelity a reliable device needs, and nobody yet knows what closing that gap requires. “It’s not just hard work,” he says of that class of problem. “It’s a lot of hard work, but it’s also luck, because we don’t know what we don’t know.”

This is the reason he treats specifications and milestone dates as the least informative part of any hardware program, and the unsolved science underneath as the part that determines whether the dates mean anything. It is also why his sharpest formulation of the field’s position lands where it does. “A qubit is a very interesting device with no intrinsic commercial value,” he says, and converting it into something useful still depends on physics nobody has finished.

Classical simulability is the only threshold that changes anything

Hassinger’s position does not end in skepticism, because the field is converging on one measurable target regardless of which architecture reaches it. “The consensus is we need to deliver fault tolerant logical qubits at a scale that is not simulatable by a classical computer,” he says. “That’s the North Star we’re all sailing towards.” Once a system passes the point where your laptop or your GPU cluster can reproduce its output, running it on quantum hardware becomes necessary rather than interesting, and nothing before that crossing changes what you can compute.

That threshold also tells you where the physics pays off first, and his answer is narrower than the general coverage implies. Materials science arrives first because condensed matter behaviour maps naturally onto these systems, with small molecule chemistry close behind, while optimization, cryptography, and machine learning all wait on thousands of logical qubits.

So the technical reading is straightforward. When a new result publishes, work out its encoding ratio and its code distance before you compare it to anything, since those two numbers determine what the machine tolerates and the logical qubit count does not. And when a roadmap updates, separate the engineering milestones from the scientific dependencies underneath them and check which unsolved physics the far dates rest on. Then put the effort into building quantum intuition inside your own team, because recognizing the high dimensional structure in your own problems transfers whichever architecture crosses the threshold first.

In case you missed

🛠️ Tool of the Week

Stim is an open source stabilizer circuit simulator maintained under Google’s quantumlib organisation, built for analysing quantum error correction circuits at speed.

Highlights

• Derives a circuit’s actual code distance instead of relying on the number a vendor publishes.

• Turns a noisy circuit into a detector error model ready to configure matching-based decoders.

• Samples circuits with thousands of qubits and millions of operations at kilohertz rates.

• Installs as a Python package and also runs as a C++ library or a command line tool.

Learn more about Stim

📎 Tech Briefs

• Anthropic ships inference hooks for Claude Enterprise - Governed prompts now route to customer security servers before inference, centralizing DLP across Claude Enterprise surfaces.

• OpenAI cuts GPT-5.6 prices and adds Fast mode - Luna and Terra get lower API pricing, while Sol Fast mode offers 2.5× speed at 2× cost.

• D-Wave and Nasdaq Verafin agree a quantum proof of concept for financial crime detection - Nasdaq Verafin will test D-Wave quantum-hybrid workflows on hundreds of financial-crime signals and network relationships.

• Cloud SQL makes PSC reconciliation default - New or newly enabled PSC instances now close existing connections after project removal from allowed lists.

• Palo Alto Networks integrates Prisma AIRS with Claude Enterprise - Prisma AIRS can inspect Claude prompts before inference, applying existing DLP policies across Claude Enterprise surfaces.

That’s all for today. Thank you for reading this issue of Deep Engineering.

We’ll be back next week with more expert-led content.

Keep building,

Saqib Jan

Editor-in-Chief, Deep Engineering

If your company wants to reach senior developers, software engineers, and technical decision-makers, speak to us about partnering with Deep Engineering.

Building an agent is easy. Building one that survives production takes architecture, verification, and stopping conditions that actually stop. Four hands-on hours with Rick Hightower on loop engineering, evaluation harnesses, context efficiency, and safe MCP tool integration, live on 29 August.

Deep Engineering readers save 40% with code - DEEPENG40.

Save your seat →

✍️ From the editor’s desk,

Welcome to the 57th issue of Deep Engineering!

On 28 July, GitHub’s changelog recorded a change that sounds administrative and is not. Individual Copilot app activity is now attributed to users in the enterprise-user and organization-user reports, with a per-user section reporting session counts, request counts, prompt counts, and a token usage breakdown covering output tokens, prompt tokens, and average tokens per request. Per-engineer token attribution is now a REST call against an API most enterprise administrators already query.

Nothing about that is objectionable on its own, because a team that cannot see its consumption cannot govern it. The risk lives in what gets built next, since the shortest path from per-user token data to a management artifact is a ranking. Amazon and Meta both stood up internal leaderboards ranking engineers by tokens burned, then dismantled them within a quarter once engineers began padding their usage, which we covered in our special issue on token maxxing. The visibility has now improved considerably, and the temptation has improved with it.

Rory Preddy, AI Advocate in Developer Relations at Microsoft and GitHub, worked five years in cloud advocacy watching the same question mature from how much will we spend into how much can we save, and he argues the answer was never individual behaviour. Today’s issue comes out of our live session, where his position was that cost discipline belongs in the artifacts a team inherits rather than in anything resembling a performance conversation.

You can also read this practical deep dive on the mechanics of token efficiency by Preddy.

Let’s get started.

Agent-written TLA+

An agent wrote our TLA+ spec. The model checker explored 14.3M states and caught a real race.

Read the write-up

Expert Insight

Cost discipline belongs in the agent profile, not the performance review

by with Rory Preddy

The logic behind a token leaderboard feels right from a distance, because agents produce measurable volume, volume varies between engineers, and variance invites comparison. Rory Preddy, AI Advocate in Developer Relations at Microsoft and GitHub, has observed this sequence play out once before in cloud, and his position is simply that “Rather than spend more, spend less.”

Coming from somebody whose job is persuading developers to adopt AI, that is a more interesting claim than it would be from a finance team, and the reasoning underneath it is what makes it useful. Preddy is not arguing for austerity or for slowing teams down. He is arguing that cost discipline is a property of the artifacts a team inherits rather than a property of the people using them, which relocates the problem from individual behavior to engineering infrastructure. For any leader currently deciding what belongs on an AI dashboard, that distinction determines whether the dashboard helps or quietly makes things worse.

Spending less is an engineering target, not a budget instruction

Preddy formed this view during five years as a cloud advocate, in rooms with the CTOs of banks whose balance sheets run larger than some national economies. He describes asking those CTOs what worried them most, and the answer changed shape over time in a way worth attending to. Early on they wanted to know how much they were going to spend, because a four-year hardware forecast let them buy servers and depreciate the cost against a schedule they controlled. Later the question inverted entirely, and what they wanted to know was how much they could save.

That inversion is the part that transfers to tokens, and it transfers with one difference that makes it more urgent rather than less. A server that gets over-provisioned wastes capital on a predictable schedule, and the waste appears on a depreciation line somebody reviews. An agent loop that resends its context on every iteration wastes money continuously, invisibly, and at a rate nobody set, because the cost arrives as consumption rather than as a purchase. Cloud waste taught organizations to rightsize compute against a bill they could at least forecast. Token waste removes the forecast.

Preddy’s instruction to teams starting now runs from the floor upward, so his advice is to look for “what’s the minimum and then cost save, not maximum.” The engineering content of that instruction matters more than its thrift. A team that starts at the largest model with the longest context has no baseline against which to measure any later improvement, so it cannot tell whether a change helped. A team that starts at the smallest model completing the work has a floor, a known cost, and a reason to escalate when the work genuinely demands it. The cheap path becomes the measured default rather than the fallback nobody trusts.

Discipline is arriving as a function, not a phase

That parallel with cloud is not incidental to Preddy’s argument, and he expects the rest of the pattern to follow too. “There’s a whole industry out there just for token efficiency,” he says, pointing to engineers who wrote good deployment scripts, became cloud engineers, and then watched cost optimization separate into its own discipline with its own titles and its own tooling. He puts it more flatly when he calls it “This is our industry.”

The institutional scaffolding has arrived faster this time than it did for cloud. The FinOps Foundation’s State of FinOps 2026 reports that 98 percent of practitioners now manage AI spend, up from 31 percent two years ago, and it ranks FinOps for AI as the top forward-looking priority with AI cost management as the single skillset teams most need to develop. In June the Linux Foundation announced its intent to launch the Tokenomics Foundation, working with the FinOps Foundation on open industry standards, benchmarks, and best practices for the economics of AI infrastructure. Preddy reads that as the point where dashboards, standards, and protection mechanisms stop being individual good habits and become things an organization can be held to.

For engineering leaders the timing carries a specific implication. A function forming now will settle its conventions within a few quarters, and the organizations contributing to those conventions will be the ones already treating agent cost as an engineering concern rather than a procurement one. Teams still running enablement sessions about prompt length will inherit whatever standards other people write.

Cost advice without a budget produces a cheaper unbounded system

Before any of that scaffolding helps, a team has to answer a question most cost guidance skips. The sharpest moment in our conversation came when Preddy declined the premise of a question about authorization boundaries and asked something more basic in its place. “How much money do I have?” he said, and then named the assumption he keeps encountering, because “a lot of what you’re asking me assumes that you have infinite amount of tokens.”

He is identifying a real gap in how the industry discusses this. Almost all published cost guidance optimizes mechanics, so prompts get shorter, retrieval gets tuned, caches get warmed, and routing gets layered in. Very little of it starts by asking what the workload is permitted to cost. Optimization applied to an unbounded system produces a cheaper unbounded system, which still has no ceiling and still fails in the same direction. The budget has to exist before the tuning means anything, because the budget is what converts an optimization into a decision.

What makes this harder than it looks is that the system cannot supply the number for you. A study published through Microsoft Research this spring, How Do AI Agents Spend Your Money?, presents the first systematic analysis of token consumption in agentic coding tasks, and it found that runs on the same task can differ by up to 30 times in total tokens, that input rather than output tokens drive the overall cost, and that frontier models fail to predict their own token usage and systematically underestimate what a task will cost. A workload whose executor cannot estimate it will not be estimated reliably by a spreadsheet either. That finding closes off the approach most organizations reach for first, which is forecasting from measured averages, and it pushes the answer toward setting a limit and enforcing it rather than predicting a figure and hoping.

The consequences of skipping that step have landed at named companies. Forrester’s July analysis of AI cost management cites Uber burning its AI budget in four months, Microsoft ending Claude Code licenses after also burning its yearly AI budget, Tesla limiting AI spending to 200 dollars per week, and Priceline absorbing an unexpected surge in AI development renewal costs. None of those reads as an engineering failure. Each one reads as a missing number.

Foundation is an artifact leaders ship

Setting the number is the first move, and the second is deciding who builds the thing that keeps teams inside it. That is where Preddy’s position turns into a leadership argument rather than a productivity tip. He reaches for the idea of a cybernetic teammate, an agent working alongside a team rather than replacing anyone, and then reframes it as something an engineering leader produces rather than something a vendor sells. The artifact is the agent profile, along with the instructions file and the container definition that travel with it, and his claim about ownership is direct. “Your leaders should go in and create the agent profiles,” he says.

His reasoning turns on availability rather than authority, and it is the more humane version of the argument. Not every engineer works somewhere they can walk to a senior colleague and admit they are burning too many tokens, and not every senior colleague is reachable at the moment the question arrives. Preddy frames the profile as the answer to that absence, describing the message it carries as “I’m not around to help you out, but I’ve created a nice agent profile” with the conventions already encoded. A profile that declares only the tools a task needs, names the model tier, and sets the compaction expectation makes the disciplined path the default path for everybody who inherits it, including the engineer who joined last week and has nobody to ask.

The second-order effect is what leaders should weigh most carefully, because it determines which of two very different programs they end up running. Token waste treated as individual behavior produces enablement sessions, then dashboards, then comparisons, then rankings, and each step follows plausibly from the last. Token waste treated as a missing artifact produces a file one person writes once and every engineer benefits from without thinking about it. The first approach asks engineers to carry knowledge that changes every quarter. The second encodes that knowledge where it survives staff turnover and stops depending on who attended which session.

One detail in Preddy’s own workflow reinforces the point. He builds these files using the cheapest model available with reasoning switched off, which means the foundation costs almost nothing to produce. The barrier to doing this properly is not budget or tooling. It is that nobody has been made accountable for it.

A spend cap belongs with the permissions

The foundation shapes the default, and a cap is what holds when the default gets overridden. Engineering organizations already accept that an agent’s authority needs governing, so they scope which systems it may read, which it may write to, and which credentials it holds. Preddy’s contribution is to place money in that same category rather than treating it as a separate financial concern reviewed on a different cycle by different people. A per-session budget that warns at a threshold and stops at a limit behaves exactly like any other permission, because it constrains what the agent may do without a human present.

The practical consequence is a change in when the decision gets made and by whom. Treated as finance, a spend cap arrives quarterly, applies at the account level, and reaches the engineer as a rate limit they cannot explain. Treated as authorization, it arrives at the same review where the agent’s other permissions get set, applies at the session level, and reaches the engineer as a known boundary they helped define. Preddy demonstrated the enforced version during our session, where a run halted after crossing its allocated credits and offered the operator a choice to add credits, raise the ceiling, or remove it entirely.

That halt is the whole value, and it is worth being precise about why. An agent meeting a wall has produced a decision point, with a human present, a specific task in view, and the cost of continuing visible on screen. An agent with no wall has produced an invoice, arriving weeks later, aggregated across teams, with nobody able to reconstruct which run caused it. The same money moves in both cases. Only one of them leaves an organization able to govern the next run.

Token counts measure activity rather than relief

A budget and a cap tell you when to stop. Neither tells you whether the spending accomplished anything, which is the question a dashboard is supposed to answer. A contributor to our earlier issue on token maxxing offered a formulation Preddy endorsed when we put it to him, that token usage behaves like CPU utilization, so the honest unit is the cost of relieving a bottleneck rather than the volume consumed. He agreed, then complicated it usefully by pointing out that the model chosen to relieve the bottleneck changes the answer, which means the metric has to account for the routing decision rather than treating all tokens as equivalent.

The research supports him on the underlying point. The same Microsoft Research analysis found that higher token usage does not translate into higher accuracy, and that accuracy often peaks at intermediate cost before saturating as spending rises. A number that rises while the outcome does not is the definition of a metric worth distrusting.

Preddy’s own experience supplies the rest, and it is more persuasive because it is self-implicating. He assigned a merge conflict to the smallest model available, and it consumed 51 million tokens without resolving anything, a run we cover in detail in his companion deep dive. Read as a token count, that looks like heavy engagement from a productive engineer. Read as bottleneck relieved, it produced nothing whatsoever. Any metric unable to distinguish between those two readings will reward the wrong behavior from the day somebody starts reporting it upward, and it will do so most strongly for the engineers who least understand what they are doing.

This is also why the leaderboard fails on its own terms rather than only on cultural ones. A ranking by tokens consumed can be improved by consuming more tokens, which requires no additional value and no additional skill. A measure built on work closed, so a merge conflict resolved, an accessibility fix shipped, a suite returned to green, can only be improved by closing more work. The second measure is harder to define and harder to collect, which is exactly why organizations reach for the first. Preddy’s argument is that the difficulty is the job, because the easy number and the useful number have never been further apart than they are now.

The principle holding it together

What makes Preddy’s approach more than a list of preferences is the single idea running through every part of it, which is that cost control belongs in the system rather than in the person operating it. Every choice he describes follows from that.

The foundation is a file rather than a training session, because a file survives turnover and a session does not. The spend cap is a permission rather than a budget line, because a permission gets enforced at the moment of use and a budget line gets reviewed after the fact. The metric is work closed rather than tokens consumed, because an outcome cannot be inflated by burning more of the input. The smallest viable model is the default rather than the exception, because a default is what happens when nobody is paying attention, and most of the cost accrues exactly then.

That principle also explains what Preddy leaves out. He never asks engineers to be more careful, and he never suggests awareness alone fixes anything, even while arguing for visibility. His own session history showed input tokens outweighing output roughly 100 to 1, and one session climbing from 49,000 to 174,000 input tokens across 66 turns with no compaction at all. He describes himself falling into bad practices like everybody else. A discipline depending on the operator remembering will fail for the person who wrote the discipline, which is the strongest available argument for encoding it somewhere else.

What this asks of a leader

The sequence that follows is short enough to start this week. Write the agent profile and the instructions file yourself, or make one person accountable for them, and treat them as shared infrastructure reviewed like any other. Set a per-session budget in the same review where the agent’s permissions get set, rather than in a separate conversation with a different owner. Report the work closed rather than the tokens consumed, and decline to build the leaderboard even though the API now makes one trivial.

The deeper point for any leader standing up AI measurement is that visibility and ranking are not the same thing, and the tooling arriving now makes it very easy to confuse them. Per-user token data has legitimate uses, including chargeback, capacity planning, and helping a manager notice that a team is consuming heavily without much to show for it. It becomes destructive at the moment it gets used to compare engineers against each other, because that is the point at which the number starts changing the behavior it was meant to observe.

Preddy’s closing thought aimed at the people doing the work rather than the people measuring it, and it belongs here because burnout and cost discipline turn out to share a cause. His instruction was not to “end each day exhausted, not from the work itself, but from managing of the work.” An engineer who inherits a foundation somebody else built gives the day to the problem. An engineer who inherits nothing gives it to managing agents, and pays for the privilege in tokens. “Be kind to yourself,” he said, which for a leader translates into something concrete. Build the foundation, and there will be nothing worth ranking.

In case you missed

🛠️ Tool of the Week

Langfuse is an open source AI engineering platform that traces agent runs and attributes cost and token usage per trace. A spend cap needs a number behind it, and a per-trace cost is that number.

• Traces multi-step agent runs so cost attaches to a workflow rather than to a raw API call

• Tracks token usage and cost per trace, per model, and per user for chargeback and per-team budgets

• Integrates with OpenTelemetry, LangChain, LiteLLM, and the OpenAI SDK, so routing across providers stays visible in one place

• Self-hostable, which keeps prompt and completion data inside your own boundary

Learn more about Langfuse

📎 Tech Briefs

• Model Context Protocol ships its 2026-07-28 specification - Sessions and initialization are removed for stateless requests, with authorization and versioned extensions now explicit.

• npm adds publish-time malware scanning and dual-use metadata - New packages now face pre-install scanning, metadata disclosure, and stricter publishing controls for security-sensitive capabilities.

• GitHub Actions holds potentially malicious workflows for approval - Suspicious runs pause before execution, moving CI supply-chain defense ahead of the workflow rather than after.

• Kubernetes v1.37 reaches code and test freeze - Code changes now require exceptions, shifting attention to release-blocking tests and regressions before August’s release.

• Accenture launches a tokenomics practice for enterprise AI spend - Token spend gets tied to business outcomes, making AI cost governance a consulting line item.

That’s all for today. Thank you for reading this issue of Deep Engineering.

We’ll be back next week with more expert-led content.

Keep building,

Saqib Jan

Editor-in-Chief, Deep Engineering

If your company wants to reach senior developers, software engineers, and technical decision-makers, speak to us about partnering with Deep Engineering.

Capacitor records the session behind the work: what agents tried, what teammates rejected, what finally passed, and why it mattered. This gives you vendor-neutrality to move across multiple coding agents, multiplayer - collaboration on coding sessions, faster PR reviews, evals on code & more.

Sign up for free

✍️ From the editor’s desk,

Welcome to the 56th issue of Deep Engineering!

The PostgreSQL Global Development Group shipped the second beta of PostgreSQL 19 on July 16, only a few months after version 18 reached general availability and began landing in production systems. A community project that already tops every major developer survey keeps compounding its lead one disciplined release at a time. Postgres has effectively become the default database, and most teams now reach for it without pausing to ask whether it fits the problem in front of them.

That ease is exactly where the trouble starts, because a decision everyone treats as obvious is a decision nobody examines. Moving off a database once an application leans on it can take years and burn through real budget, and the forces that make the choice consequential have only sharpened this year. Relicensing moves keep redefining what the words open source actually protect, and by Google’s own threat-intelligence reporting attackers now weaponize many newly disclosed flaws before a patch is even available, which turns the question of where and how you run your database into a live operational risk rather than a matter of preference.

That is exactly the terrain Peter Zaitsev has worked his whole career, an entrepreneur, author, and co-founder of Percona. Two decades of helping companies choose, run, and sometimes escape their databases give him a sharp read on which reasons hold up under real load and which ones fall apart the moment the system starts to matter.

In today’s issue, he makes the case for treating that choice with the seriousness it earns, and for choosing as though you can never walk it back.

You can also watch our full conversation with Peter Zaitsev here.

Let’s get started.

Building Intelligent AI Agents with GraphRAG

This Agentic Engineering live bootcamp walks you through building an agent that investigates across a knowledge graph, remembers what it found, and returns answers you can put in front of a user.

Friday 31 July to Saturday 1 August, online.

Register with code DEEPENG40

Expert Insight

Databases are sticky. Most teams underestimate the commitment

by with

Most teams choose a database the way they choose a lunch spot, quickly and on somebody else’s recommendation. A friend runs it in production, a conference talk made it sound fast, or the cloud console offered it as the first option on the list. The 2025 Stack Overflow Developer Survey put PostgreSQL at the top for the third year running, in use by 55.6 percent of all developers and 58.2 percent of professional ones, far ahead of second-place MySQL. The default has never been clearer. But the decision behind it still rarely gets the weight it deserves.

Peter Zaitsev has advised companies through twenty years of these choices, from the greenfield pick to the migration nobody wanted. He co-founded Percona, grew it from a two-person company into one of the most respected open source database companies in the industry. His view of what most teams get wrong is straightforward. “Databases are very, very sticky,” he says. Once an application leans on one, unwinding that dependency can take years and burn through budgets, which is exactly why choosing one casually is the expensive mistake.

The cost is not hypothetical. Sharing from his experience advising enterprises through these migrations, Zaitsev says, “They started this kind of very painful migration away from Oracle a decade ago, and many, many millions of dollars, many, many years, they’re still on it.” A database that took a weekend to adopt can take a decade to leave. That asymmetry is the reason the rest of his advice matters.

Nobody owns Postgres, and that is most of why it won

In our conversation, I asked Zaitsev why Postgres reached the top, and his answer began with the part engineers tend to skip. “PostgreSQL is truly a community database. Nobody owns Postgres,” he says, and as he sees it that governance fact, more than any feature, explains the dominance. Because no single vendor controls it, every cloud can offer it as a first-class product without handing a rival an advantage. “Amazon, Google, Microsoft, a lot of others, they can offer PostgreSQL as their own,” he points out, where promoting MySQL or SQL Server means promoting something a competitor owns.

The technical case follows the same shape. Postgres extends rather than forces migrations, so each new demand becomes an extension instead of a second database. Time-series work got its extension, and the AI wave got pgvector, the open source extension that adds vector types and similarity indexes and now ships enabled on managed Postgres from AWS RDS to Supabase. “pgvector, vector search, enabling building AI application in Postgres, that came about very quickly,” Zaitsev tells us. The base keeps getting stronger underneath all of it, and the September 2025 release of PostgreSQL 18 added a new asynchronous I/O subsystem the project measured at up to three times faster on reads.

Open source is not enough

The reason to prefer the community model becomes obvious the moment a vendor changes the rules. “People even kind of lost trust in open source and started to understand that open source is not enough,” Zaitsev reflects, and he points straight at the pattern that taught them. MongoDB moved to the Server Side Public License in 2018, Elastic followed in 2021, HashiCorp adopted the Business Source License in 2023, and Redis switched to source-available terms in 2024 before adding an open license back in 2025 under community pressure. Each move left production users holding a version they could no longer upgrade under the terms they signed up for.

Zaitsev’s warning is that the marketing hides the trap. “A lot of companies right now are trying to trick you, using words like open source compatible,” he cautions, which usually means proprietary software wearing a thin layer of compatibility. Open core is the other tell. “There is some sort of limited, crippled version which is open source, but what they actually want is for you to buy the proprietary version,” he says. The protection he trusts is not a license badge but a market. A real open source project has many vendors, so a team unhappy with one can leave, and run a closed product like Amazon Aurora and only Amazon can support it, which puts the depth of experience and the leverage on one side of the table.

Self-hosting now means racing the exploit clock

Self-hosting an open source database moves the security burden onto the team, and the clock on that burden has changed. Zaitsev warns that with AI, security holes can now be found and exploited “at the speed which was never seen before,” and the incident data agrees with that. Google’s M-Trends 2026 report estimated the mean time to exploit a newly disclosed vulnerability at roughly negative seven days, meaning many flaws are weaponized before a patch exists, and Google’s threat intelligence group has reported the first zero-day it attributes to an AI-developed exploit.

That pace changes what good looks like. Zaitsev argues for a layered posture and an honest plan for failure, because prevention alone never reaches certainty. “Nothing gives you a 100 percent guarantee,” he reminds us, so the real question becomes how fast a team detects and responds, not only how well it defends. For many teams the practical answer is a contract, and even those that avoid a fully managed service should have someone on the hook for patching. In regulated environments the whole subject stops being a technical choice and becomes a compliance one, which is where managed databases earn their place by letting a team push responsibility to the vendor who runs the system.

Doing what the cool kids do is not a workload

The pull toward document, vector, and graph stores is often social before it is technical. “A lot of engineers, they like to explore and play with new technologies,” Zaitsev says, usually after hearing about them at a conference or on a podcast. He does not moralize about curiosity, but he counts the cost. “The more technologies you introduce, the more complexity you create,” he says, and a small team running many engines pays that bill every day. “If you are a five-person company and you are self-managing 10 databases, there’s probably way too much complexity.”

His rule is to start narrow and let real limits, not fashion, force the next move. Postgres holds relational data, documents, and vectors well enough that most teams travel a long way before they need anything else, and only a genuine ceiling on scale or a missing feature should justify a dedicated store. Purpose-built systems do win when the workload is real, because a database built for one task carries a more optimized store and a better language for it. The catch with AI is speed. “The AI world is completely different. Every six months things drastically change,” Zaitsev reasons, so a team building AI features has to check the date on its advice, since guidance a year old still serves Postgres or MySQL fine and can already be stale for anything touching models.

Choose like you cannot leave

The through-line of Zaitsev’s advice is to match the seriousness of the decision to the difficulty of reversing it. He wants teams to write down their actual requirements rather than inherit a default, and he is happy to use the newest tool to pressure-test the old discipline. “You can actually ask AI, hey, what am I missing,” he says, treating a model as a fast second reader on a requirements list. The point is not the tool. It is that a choice this durable deserves more than an afternoon.

That is the gap he keeps returning to. Teams decide in a hurry and then live with the result for years, on infrastructure that only grows harder to change as the application matures on top of it. Postgres has made the safe default easy, and the survey numbers show most teams taking it. The harder work starts after the default, in the licensing terms, the security posture, and an honest reading of what the workload actually needs.

In case you missed

Industry Perspective

🛠️ Tool of the Week

CloudNativePG, the Kubernetes operator for running production Postgres on infrastructure you control

CloudNativePG runs Postgres as a native Kubernetes workload and handles failover, backups, and rolling upgrades the way a managed service would, while the data and the control plane stay inside your own cluster.

• New DatabaseRole resources manage Postgres roles as declarative GitOps objects with password-free certificate authentication.

• A Kubernetes Lease now gates primary promotion, so replicas fail over without waiting out the full timeout.

• In-place major upgrades run pg_upgrade against mounted extension images, cutting version-migration downtime.

• Operator and Postgres images ship signed with SBOM and provenance attestations for supply-chain verification.

Learn more about CloudNativePG

📎 Tech Briefs

• PostgreSQL 19 Beta 2 adds native SQL/PGQ property-graph queries - Postgres 19 exposes relational tables as property graphs via GRAPH_TABLE pattern matching, so fixed-depth traversals no longer need a graph database.

• CISA adds an actively exploited SharePoint Server RCE to its Known Exploited Vulnerabilities catalog - CISA confirmed active exploitation of CVE-2026-58644, a deserialization remote-code-execution flaw in on-premises SharePoint Server, and ordered urgent patching.

• MariaDB Community Server 10.6 reaches end of life - MariaDB’s last old-model LTS no longer receives bug or security fixes, so 10.6 shops now face an unavoidable migration.

• Autobase 2.9.0 released. The open source automated Postgres high-availability platform now manages cluster infrastructure after deployment, not only initial setup.

• CISA flags an actively exploited authorization bypass in Langflow - CISA confirmed in-the-wild exploitation of CVE-2026-55255, an authorization bypass in the open source LLM app builder Langflow.

That’s all for today. Thank you for reading this issue of Deep Engineering.

We’ll be back next week with more expert-led content.

Keep building,

Saqib Jan

Editor-in-Chief, Deep Engineering

If your company wants to reach senior developers, software engineers, and technical decision-makers, speak to us about partnering with Deep Engineering.

Capacitor records the session behind the work: what agents tried, what teammates rejected, what finally passed, and why it mattered. This gives you vendor-neutrality to move across multiple coding agents, multiplayer - collaboration on coding sessions, faster PR reviews, evals on code & more.

Sign up for free

✍️ From the editor’s desk,

Welcome to the 55th issue of Deep Engineering!

OpenAI made its GPT-5.6 family generally available on July 9 across ChatGPT, Codex, and the API. The capability that matters here is a new ultra mode that runs concurrent subagents and synthesizes their work in a single request. This release also makes cached context far cheaper than fresh input, since cache reads keep the 90% cached-input discount. So running a fleet of agents in parallel, and paying almost nothing to reuse their context, has moved from a hand-built trick into something the platform now does by default.

That shift is why this issue matters now. When the tooling makes parallel agents this easy, the differentiator stops being model access and becomes the discipline around it, the specifications, the tests, and the review that decide whether all that speed produces software you can actually ship. Last week Julien Dubois led a workshop for us, From Coder to Manager of Agents, where he showed how he built a real open-source product this way, measured from git history rather than memory, which makes it far more useful than another benchmark thread.

Julien is Principal Manager for Developer Relations at Microsoft and GitHub, and the creator of JHipster.

Let’s get started.

Featured: Loop Engineering for AI Agents

Stop wasting tokens on endless retries. This four-hour, hands-on workshop takes you past one-off prompting to reliable agent loops that plan, execute, verify, and stop safely, built with Claude Code, Spec-Driven Development, and MCP, with the verification gates and state management that keep autonomous agents trustworthy.

Deep Engineering readers save 40% with code DEEPENG40.

Reserve your seat

Practitioner's View by Julien Dubois

223 pull requests in 11 days for the project I never had time to build

You can read this practical deep dive based on his workshop. And here are the slides from the talk.

I always wanted a real developer UI for Spring Boot. Every Spring app is a black box in development. Actuator gives you raw JSON, not a console, so what I wanted was health, metrics, security and tracing in one embedded UI.

I’d shipped a slice of this in JHipster years ago, but only for generated apps. A real console for any Spring Boot app sat on my wishlist for years. The catch is that it’s massive. Around 40 panels, each one a backend plus a frontend plus its own tests, deeply integrated across a dozen JVM subsystems. By hand, one experienced developer would need somewhere between six and a half and eight and a half months. Too big to justify, until I stopped writing the code myself.

Then I did it in 11 days. Not by typing faster. I didn’t even open my IDE. I managed a fleet of AI agents while I architected, reviewed and steered. The agents did the scaffolding, the panels and the tests. I did the judgement.

What I built, BootUI

BootUI is a production-grade Spring Boot 4 starter that adds an embedded, local-only developer console to your app. You add one starter to your pom.xml, run locally, and open the console.

It is a five-module Maven build on Spring Boot 4 and Java 17, integrated across Actuator, Spring Security, Flyway and Liquibase, Hibernate, Micrometer with OTLP, GraalVM, OSV scanning and ArchUnit, published to Maven Central with full CI. The frontend is an embedded Vue 3 single-page app, packaged inside the starter. Each panel is endpoints plus a view plus tests.

There are around 40 feature panels. Health and metrics, a security advisor, a vulnerabilities view, tracing, and more. Those panels look a lot alike, which made them the biggest source of structural repetition in the codebase. That detail matters, and I will come back to why it made the whole thing work.

Eleven days, and how I know the numbers are real

The figures here are derived from git history, PR metadata and code metrics, not from time-tracking logs. Through the tagged 1.0.0 release there were around 264 commits on main and about 223 squash-merged pull requests, landing at roughly 20 a day across 11 days. That came to about 83k total tracked source lines, close to 50k of them Java across around 461 files and 81 test classes, plus 52 Vue components, 40 panels, and 35 end-to-end specs. In all, about 116 test suites.

A cadence of 20 PRs a day with an AI agent co-authoring commits is impossible to reach by hand. The agent did the typing. I dispatched many asynchronous tasks in parallel. The evidence lines up with that. The commit clock runs from around five in the morning to midnight most days, which is consistent with parallel async tasks rather than continuous typing. “Copilot” shows up as a named commit and PR author on about 44 commits, and the repo ships a copilot-instructions.md with per-panel conventions, so the workflow was explicitly agent-oriented. Even on release day I was landing a docs site, a scanner dashboard, token charts and a Hikari fix, which is itself several days of solo work.

Where my own time went was writing prompts, reviewing and merging those 223 PRs, and resolving CI failures around Spring Boot 4, Flyway 11 and OTLP. Review and orchestrate, not write every line.

The honest by-hand comparison is the part people argue about, so I ran it carefully. One experienced Spring Boot and Vue developer, with no AI codegen, would need 6.5 to 8.5 months for the same polished 1.0.0, roughly 1,100 to 1,450 hours of hands-on effort, and the 40 panels are the dominant cost. A COCOMO organic estimate on 50k lines yields more than 100 person-months, which is too high because much of the code is repetitive scaffolding, so a domain-expert solo figure of about 7.5 months is the defensible middle ground. My actual human effort on BootUI was 80 to 110 hours, about two intense solo weeks. That is a calendar speed-up of roughly 17 to 23 times, and a human-hours speed-up of roughly 12 to 17 times.

I was the manager, not the developer

I didn’t open my IDE. I wasn’t the developer. I was the manager. Many agents ran in parallel on the panels, the integrations, the frontend, and the CI and docs, and my job was to brief, review and merge.

Your throughput is not your keyboard. It is your briefs. What blocks a single developer is typing, and what blocks a single agent is you waiting for it to finish. Running many at once removes both, because they come back one after another and there is always something to review. You don’t type faster this way. You ship what used to take months.

A day in the loop

The rhythm that produced 20 merged PRs a day is a simple daily loop, and the day is the engine. I am hands-on all day. I spec, launch, review, merge and re-task, live, and most of the day’s PRs land right there.

The evening is a hand-off. Before I step away I queue a few deep, long-running plans. The night is a bonus, not the engine. A handful of deep autonomous runs finish by morning, which is the minority of the work. Repeat that for about 11 days and you reach a tagged 1.0.0. Six ingredients make the loop work.

Ingredient one, write the specifications

The first thing I write is never code. It is the house rules. An AGENTS.md, which GitHub Copilot reads as copilot-instructions.md, and every agent reads it first. It sets the conventions once, the stack, the build, the tests and the style, with per-panel conventions so 40 panels come out consistent. Mine runs around 700 lines. That is about the right size, because a file that is too big pollutes the context, and it also costs tokens and money on every run.

The high-leverage move is writing the known failures into that file once. Most models were trained on Spring Boot 3, so they struggle with the reworked Jackson API in Spring Boot 4, and 10 agents trained on the same data hit the same wall at the same time. Rather than fix that by hand in each one, I write the fix down once. The same goes for operational traps, like giving every agent its own local Maven repository so parallel installs do not corrupt a shared one.

On top of the house rules I write one spec per task. What to build, where, what “done” looks like, and the acceptance test the agent must make pass. Small, self-contained, no hidden dependencies. The spec is the product now. The better the brief, the less you babysit.

Ingredient two, build the test harness

No agent runs without a test harness. One command compiles and runs the unit and end-to-end tests, and comes back green or red, so an agent can verify its own work before it opens a PR. No tests means you cannot trust the output.

CI is the trust layer. BootUI has about 116 test suites, 81 in Java and 35 in Playwright, and CodeQL plus the end-to-end tests gate every PR to main. The loop does the work. An agent writes code, pushes it, and GitHub Actions sends back the results. If they are red the agent reads the failure, fixes it, commits and pushes again, and the checks run once more. Dependabot keeps the dependencies current alongside all of this. A compiled, typed language like Java helps too, because a hallucinated call often fails to compile before a test ever runs, and the tests catch what the compiler misses. You cannot read every line of 223 PRs. A green build you trust is what makes the volume reviewable.

Continue reading

The rest of the deep dive covers the other four ingredients, the test harness and CI as the trust layer, running agents in parallel across worktrees, the overnight critic-and-vote runs, merging as the real bottleneck, and the cache math behind the roughly two thousand dollar bill, along with where the multiplier will not repeat.

Industry Perspective

🛠️ Tool of the Week

Composio’s Agent Orchestrator is an open-source take on exactly that job. It helps developers manage fleets of coding agents for parallel work, giving each one an isolated workspace and supervising them from one place.

• Runs each agent in its own git worktree, so parallel work does not clobber shared files

• Feeds CI failures, review comments, and merge conflicts back to the right agent automatically

• Works with the terminal agents teams already use, including Claude Code, Codex, Cursor, and opencode

• MIT licensed and self-hosted, run locally rather than as a hosted service

Composio’s Agent Orchestrator

📎 Tech Briefs

• Claude Code changelog - Agents must now confirm before entering a git worktree outside the project’s .claude/worktrees directory, and a new /doctor check flags checked-in CLAUDE.md content the model can derive from the codebase.

• GitHub Copilot CLI changelog - Plan mode can no longer run tools that modify the workspace, and the default maximum sub-agent nesting depth drops from six to four to limit runaway recursive delegation.

• Copilot for JetBrains expands BYOK capabilities - Adds local agent sandboxing, a Claude agent provider for custom agents and skills, and OpenAI-compatible custom endpoints, all in public preview.

• Copilot in Visual Studio, June update - Visual Studio now checks each MCP server’s configuration and asset fingerprint against a trusted baseline at startup, and the C++ modernization agent reached general availability.

• Claude Code gateway for teams - Anthropic shipped a self-hosted gateway for Claude Code, a stateless container that adds corporate SSO login, centrally enforced policy, role-based access, and per-user cost attribution across a team.

That’s all for today. Thank you for reading this issue of Deep Engineering.

We’ll be back next week with more expert-led content.

Keep building,

Saqib Jan

Editor-in-Chief, Deep Engineering

If your company wants to reach senior developers, software engineers, and technical decision-makers, speak to us about partnering with Deep Engineering.

Over roughly a quarter in 2026, several of the largest engineering organizations in the world took down the dashboards they had built to prove their AI investments were working. Amazon shut down KiroRank, the internal leaderboard that ranked developers by how many tokens they consumed on its Kiro platform. Meta also dismantled a near-identical board, Claudenomics, that tracked token usage among its heaviest AI users. The reason was the same everywhere. Once usage became the number that mattered, engineers found ways to run up usage, and the costs arrived long before the value did.

Amazon’s own senior vice president, Dave Treadwell, told staff to stop using AI for its own sake after employees gamed the leaderboard by padding token usage, the practice the industry had taken to calling “tokenmaxxing.” Uber also said it could find no clear link between what it was spending on AI and what it was shipping, while Microsoft cancelled a division’s coding-assistant licences over cost. So, within a single quarter, the industry ran the experiment, watched it fail, and started hunting for a better question to ask.

That question is the subject of this issue, with insights from Austin Parker, Director of AI Strategy at Honeycomb and a co-founder of OpenTelemetry; Tom Howe, Director of Solutions Engineering at Hydrolix; Jayeeta Putatunda, Director of the AI Center of Excellence at Fitch Ratings; Satyam Dhar, Staff Software Engineer at Galileo and a former engineering leader at Amazon and Adobe; and Michael J.J. Tiffany, co-founder and CEO of Fulcra Dynamics.

Let’s get started.

Social engineering is about manipulating people’s emotions. Identify the susceptibilities that hackers use to exploit people.

This NINJIO Insights Report dives into the key emotional susceptibilities that make social engineering work and offers concrete steps that your security team can take to equip your workforce to resist cyberattacks.

Download the Guide

Special issue — July 2026

A token was never a unit of intelligence

“You can use tokens to help build solutions, but at the end of the day these are judgment calls.” — Austin Parker, Director of AI Strategy at Honeycomb and a co-founder of OpenTelemetry

Give a modern coding model the right harness and a clear goal and it will do almost anything you ask, up to and including rewriting an entire codebase in another language and passing the tests. That capability is real, and it is seductive. It makes it easy to believe that all you need is something to keep the tokens flowing and any problem becomes tractable. Parker has observed his own engineering teams test that belief, generating a hundred variations of the same screen or twenty versions of a flow, and the lesson came back the same each time.

The variations did not move the needle. You can ask for infinite variations of a solution, and they all arrive shaped like your original conception of the problem, because that conception is the one human input the model never questions. The code and the screen and the workflow were never the things that create value for the people using them. Value lives one level up, in whether the design serves those people, whether the shape of an API fits what its consumers actually need, whether the underlying primitives have any fitness for the purpose they are being asked to serve.

Those are judgment calls, and no volume of tokens answers them. Parker sees the same pattern when he talks to other leaders, plenty of appetite for here is a problem, generate all the variations, and far less time asking whether the problem was framed correctly in the first place. His conclusion is blunt. Judgment is the finite resource now, and teams that forget it end up with more output and less of what the output was supposed to buy them.

Leaderboards reward motion, not judgment

“The paradigm in some organizations will, undoubtedly, shift from ‘high token usage equals good employees’ to ‘high token usage equals expensive employees.’” — Tom Howe, Director of Solutions Engineering at Hydrolix

The collapse of the token leaderboards surprised no one who has watched a measure become a target. Hand a famously clever group of engineers a resource and grade them on how much of it they use, and they will use it. The moment a measure becomes a metric it stops being a good measure, a version of Goodhart’s law that Amazon’s own leadership named out loud when it wound the program down. The surprise was not that the leaderboards were gamed. It was that so few people in the decision chain saw it coming.

Howe has seen the mechanism play out from the inside, and he traces it to a subtle slippage. A company rolls out what it calls an AI usage dashboard, meant as an objective measure of how much the tooling is benefiting the business, and it quietly devolves into a leaderboard that measures how much each person is using the tool. Once employees read it as a judgment of how they work rather than of whether the tool earns its keep, the whole exercise changes character.

From there the responses split three ways, and each one erodes the thing the dashboard was supposed to protect. Some engineers game it, firing tools at low-value tasks to boost their stats, which breeds suspicion that others are gaming it too. Some avoid the tool entirely rather than expose their habits to uncontextualized data. And in the middle sits the casualty Howe cares about most, the trust that a healthy organization runs on, undermined the moment a team cannot tell how or why it is being measured.

The cost frame makes it worse. Howe describes enterprises treating tokens as close-to-free productivity, funny money, right up until the real bills and the real gains come into focus. When they do, the incentive can flip hard, from prizing heavy usage to flagging it as expense, and engineers start hedging, wary of being branded token wasters and wary of binding their workflows so tightly to a tool that a future rationing decision leaves them stranded. He points to a recent, unusually capable model that carries a known high future cost, where engineers are trying to use its power now without becoming dependent on it, as exactly the kind of bind these metrics create.

None of this means the tools do not work. Howe shares that Hydrolix has seen a revolutionary impact from how it uses them, in building products and in weaving them into products, in ways that would have been unimaginable a few years ago. The challenge is balancing productivity, cost, and trust, and learning to measure the impact through experience rather than through a scoreboard. He tells one story that captures the slope precisely, an engineer glancing at the usage board and joking that not cracking the top fifty was rookie numbers. It was a joke. But it was also a warning about how fast an innocuous metric turns into a competition.

Featured - ARC 2026: Software Architecture in the Age of AI

AI is reshaping software architecture, putting new demands on scalability, governance, reliability, and observability. ARC 2026 brings together architects, CTOs, and AI practitioners for keynotes, panels, and workshops on agentic system design, modernizing enterprise apps for AI, and building governable, observable AI systems.

🗓️ 25 to 26 July, 10:30 am ET

Use code DEEPENG50 for 50% off the early bird price.

Reserve your spot

Skills that matter never show up on a dashboard

Parker calls prompting, building internal alignment, and holding a complex system in your head illegible activities, and they are close cousins of the glue work that keeps organizations moving. The people who know who needs to be in a decision chain, who can hear a customer complaint and route it to the one team that can fix it, who understand that the thing frustrating you in one corner of a product is caused by something three systems away, are the reason a business can actually execute. The organizations that are genuinely good at this tend to be the ones that, lacking a clean way to measure those people, are at least careful not to penalize them.

A token metric makes that harder. It rewards the visible act of consuming AI over the invisible work of knowing what is worth building, and the invisible work is where most of the value hides. That work shows up in promo packets and career ladders because those are written by people who understand it. It does not show up in a number you can compare quarter to quarter, because, in Parker’s words, it involves “very human things like relationships and time in the system rather than timing the system.” Measure the activity and you slowly stop rewarding the thing that made the activity worth anything.

AI widens the gap a team already has

This is the reframing every leader watching one team pull ahead should steal. AI rarely conjures capability from nothing. It accentuates differences that already exist. When a team surges, the real story is almost never that raw intelligence solved a problem they could not solve before. It is that AI patched a hole in the team’s composition, or handed them leverage on something organizational rather than technical.

Parker’s example is the part of shipping that engineers tend to undervalue, making the case to leadership. That is usually a data-analysis argument, projections and forecasts and a slog through support tickets and sentiment data, and it happens to be something AI is very good at. A motivated engineer or designer or support person can now do the analysis a product manager used to own, cover the weak spots in their own toolkit, and come back to the org with here is what I need in order to go build the thing I am convinced is right. So the question to ask the team that pulled ahead is not which tool they used. In Parker’s words, “I never want to ask what AI has let you do. The question is what is AI giving you the excuse to do that you could not have done otherwise.”

That does not extend into a ban on measuring code output. Parker’s line is that you should measure it and refuse to make it a metric. Watching the balance of someone’s pull requests opened against reviewed tells you something real about how they work, and grading their performance on that same count is where it curdles. AI can even help here, letting a team ask which work is generating on-call burden, which is delivering customer value, and who is quietly doing the refactoring that lifts everything built on top of it.

Point the budget inward, at developer experience

“If I had 90 days of AI budget to improve developer experience, I wouldn’t spend it evaluating another model. I’d spend it reducing engineering variation.” — Jayeeta Putatunda, Director of the AI Center of Excellence, Fitch Ratings

For a Staff or Principal engineer who owns a platform, Parker’s first ninety days are not about output at all. They are about earning the ability to ask real questions of your own systems, and most organizations cannot, because observability was underinvested for years while other goals took priority. The mechanical work of fixing that, moving unstructured logging to structured events and spans, migrating to OpenTelemetry, is exactly the kind of task that is fungible from an intelligence point of view, which makes it a near-perfect fit for current coding models. It also comes with a natural verification step, feed the before and after to another model and let it judge whether the telemetry got better, and let it update the docs as it goes.

Putatunda took the same budget in a different direction, and her experience widens the point. Once her organization distributed AI broadly, it became clear that model capability was never the bottleneck, because every team had the same models. What differed was how teams solved the same problems, each with its own coding conventions, documentation norms, migration methods, and review expectations. So her first priority was not a better model. It was making the organization’s engineering knowledge reusable.

Her team translated coding norms, implementation templates, documentation standards, review criteria, and migration playbooks into reusable skills that work from any agentic environment, whether an engineer reaches for Claude Code, GitHub Copilot, or something else. That let them clear backlogs that had always lost the prioritization fight against feature work. The migration of a legacy Selenium test suite to Playwright, a slow and redundant manual slog across many teams, got done by pairing reusable migration skills with the organization’s own standards and validation steps, so the generated tests matched both the suite and the reviewers’ expectations. The same approach carried into repository documentation, repetitive refactoring, and framework upgrades, the modernization work where consistency matters more than originality.

The deeper reason it worked speaks straight to the review bottleneck that AI creates. Models generate code faster than teams can review it, and when every change arrives in a different shape, every pull request costs the reviewer more attention. Standardizing the patterns let reviewers focus on correctness and business logic instead of style and structure. Putatunda does not measure any of this in tokens or lines generated. She measures whether projects moved faster, whether patterns got reused across teams instead of reinvented, and whether reviewers stopped correcting predictable issues, which is another way of saying she measures the same thing Parker does, whether the team can now do work it could not do reliably before.

Spend on leverage, not on tokens

“Just as we don’t judge an IDE by how many keystrokes it saves, we shouldn’t judge AI by how many tokens it consumed.” — Satyam Dhar, Staff Software Engineer at Galileo and a former engineering leader at Amazon and Adobe

Local inference and model routing are making cost-per-token murkier by the month, and Parker is honest that he does not have a clean formula to replace it. What he has is better than a formula. It is an example. For most of Honeycomb’s life, dark mode was the feature customers asked for and never got, enough of a running joke that it has its own face on the twenty-sided die new hires receive. Then, in about six months, the company shipped it, mostly because of AI.

The caveat matters as much as the result. It was not only AI. Getting there took years of groundwork, a design system and an accessible component library built to support the feature. What AI closed was the mechanical gap that always stalled the project, migrating the existing app onto the new components. The way they did it is the real lesson. Instead of one team grinding through the migration, they shipped prompts and skills so anyone could run it, and pulled in an engineering director who had built a screen seven years earlier to load the skill and update his own page. Yes, it burned a lot of tokens, and a small focused team could have optimized that count down. The accounting that matters is a five-year customer request finally closed and a company full of people who learned how to use AI.

Dhar puts a name to the mistake underneath the token frame. Too many teams treat AI like another cloud bill, counting tokens because tokens are easy to count, then assuming fewer tokens means better spending. In his experience running large-scale AI systems, some of the most valuable investments were the ones token accounting punished, better model routing, evaluation pipelines, prompt versioning, and experimentation infrastructure, all of which raised spend on paper while sharply lowering the cost of iteration.

That lower cost of iteration is where Dhar locates the actual return. When engineers can compare models safely and product teams can experiment without fear of quietly degrading production, a team can validate five ideas in the time it used to take to validate one, and the business benefit dwarfs the incremental compute. He also refuses to ignore the second-order effect that never reaches a finance dashboard, the way good infrastructure changes behavior, so engineers start proposing projects that once felt too expensive or too repetitive to attempt and product conversations get more ambitious because implementation cost no longer dominates every decision. His test is a single question, whether the system helped the team build something it otherwise would not have built.

Count only the work that clears a bottleneck

“Only the work that clears a bottleneck is progress. The rest are lightbulb filaments that didn’t work.” — Michael J.J. Tiffany, co-founder and CEO of Fulcra Dynamics.

Tiffany offers the closest thing in this issue to a replacement unit of account, and he arrived at it by correcting himself. Token use, he argues, is a low-level metric like CPU utilization, important but meaningless in aggregate. His better frame is cost per bottleneck relieved. He started somewhere looser, counting cost per accepted unit of work such as a merged pull request or a user-visible improvement, then learned the hard way to count only the subset of work that actually clears a bottleneck.

The distinction changes what you tolerate. What you buy when you pay for AI, in Tiffany’s account, is more attempts, faster loops, more surface area explored, fewer blocked humans, and occasionally, almost at random, a capability that was not economically or psychologically feasible before. Most of that is not progress, and that is fine. The failed experiments and even the shipped features that did not matter are the filaments that did not light. You let them happen, and you measure success only by the bottlenecks that came loose, which keeps the number honest in a way a token count never can.

Judgment is the constraint for the next two years

The token leaderboards are already gone, and the mandate that produced them is being quietly rewritten across the industry toward outcomes that are harder to game. The deeper lesson is the one Parker keeps returning to. Every layer an agent touches was built on the assumption that the consumer brings context and judgment, and that assumption is exactly what breaks when the consumer, or the incentive, is optimizing a number.

The voices in this issue disagree on the right replacement, and the disagreement is the useful part. Howe would protect trust before any metric. Putatunda would standardize the engineering practice underneath the tooling. Dhar would ask whether the team can now build what it could not before. Tiffany would count only the bottlenecks that came loose. What they share is a refusal to accept the token as the unit of value, and a conviction that the scarce resource is the judgment to decide what is worth building in the first place. The limiting factors on agentic systems are no longer model capability alone. They are judgment, integration, and the operational discipline to tell useful work apart from motion, and that is where the most consequential engineering decisions of the next two years will be made.

Thank you for reading this special issue of Deep Engineering on why judgment, not tokens, creates value.

We’ll be back on Thursday with more expert-led content, and next month, on the first Tuesday of August, with another special issue.

Keep building,

Saqib Jan

Editor-in-Chief, Deep Engineering

If your company wants to reach senior developers, software engineers, and technical decision-makers, speak to us about partnering with Deep Engineering.

Packt’s flagship two-day virtual summit

AI is reshaping software architecture, putting new demands on scalability, governance, reliability, and observability. ARC 2026 brings together architects, CTOs, and AI practitioners for keynotes, panels, and workshops on agentic system design, modernizing enterprise apps for AI, and building governable, observable AI systems.

The lineup includes Dr. Ali Arsanjani (Director, Applied AI, Google), David Ping (Head of Solutions Architecture, AWS), and Chi Wang (Google DeepMind), alongside other notable leaders distilling practical insights.

🗓️ 25 to 26 July, 10:30 am ET

Use code DEEPENG50 for 50% off the early bird price.

Reserve your spot

✍️ From the editor’s desk,

Welcome to the 54th issue of Deep Engineering!

Site reliability engineering is already agentic, and agents are moving from the edges of the incident into its core. New Relic unveiled Autopilot at its New Relic NOW event on June 23, an automated SRE agent that triages incidents, identifies root causes, and scopes remediations the moment an alert fires. “Operations are going headless,” said New Relic’s Head of AI, Camden Swita, describing agents that pull what they need through APIs and act.

That is exactly when governance starts to matter more than speed. Once an agent can decide and push a remediation, the reliability question is no longer whether automation moves fast but whether its actions stay bounded. Faster, wider change across a tightly coupled system amplifies trouble as easily as it absorbs it, and the retry, scaling, and routing reflexes that steady a healthy platform can drive a cascade once it is already stressed. Tellingly, even these launches arrive wrapped in the language of guardrails and human review.

The engineers who feel that trade-off most work where a wrong automated action shows up immediately on the balance sheet. Sibasis Padhi, a Staff Software Engineer at Walmart Global Tech, builds large-scale financial platforms of exactly that kind. In his Deep Engineering article “Autonomic Governance for Agentic Systems,” he argues that reliability engineering must evolve from managing distributed systems to governing the automation that manages them.

Padhi makes the case that the fix is not less automation but automation you can bound, audit, and reverse.

Let’s get started.

Featured Newsletter: The Hustling Engineer

Read by 25,000+ software engineers, The Hustling Engineer covers career growth, AI, interviews, and productivity. Every week you get actionable insights, engineering deep dives, and lessons from top tech companies.

📩 Get one actionable lesson every week to accelerate your tech career.

→ Subscribe to The Hustling Engineer

🧠 Expert Insight

Autonomic Governance for Agentic Systems

by Sibasis Padhi

The incident often starts small. Usually one dependency slows, tail latency grows, and a few requests time out. And then the platform does what we trained it to do. All the while, customers automatically retry their failed requests, and very quickly there are far more retry attempts than real new customers trying to use the service. But the autoscaling system sees the growing queue and reads it as exploding traffic, so it adds more servers, often in the wrong place, which sends even more work to the spot that is already overloaded.

This eats into the bottom line and the return on engineering investment as the bill keeps mounting while the system handles less real work. Circuit breakers finally trip and try to move traffic elsewhere, but those other paths were never built to absorb a sudden flood.

Everything might still look like it is working on paper, but the whole situation keeps getting worse. Now compound that by adding agentic AI to operations, systems that recommend or execute operational actions, which increases the speed and breadth of change under conditions that are already chaotic.

AI-driven operational automation can increase fragility when it increases the rate, scope, or coupling of production actions without bounded actuation, explicit constraints, and auditability. Under governance, the same automation reduces fragility by enforcing safety envelopes and reversible decision paths.

This is not an argument against automation or AI. It is an argument that reliability engineering must evolve from managing distributed systems to governing the automation that manages them, especially under simultaneous SLO, cost, and compliance constraints. I call this discipline Autonomic Reliability Governance (ARG). ARG is not more automation. But governed automation that is policy-constrained, auditable, and rollback-capable by design. It ensures that AI-assisted or agentic operational decisions cannot escalate into amplification engines under stress.

Why reliability automation amplifies in microservice ecosystems

In microservice systems, outages rarely come from a single broken component. They emerge from how services interact, through feedback loops, dependencies, and cascading effects. Mechanisms that improve reliability in normal conditions can unintentionally amplify problems when the system is already under stress, which is the feedback behavior Karl Johan Åström and Richard Murray formalize in Feedback Systems (Princeton University Press, 2008).

Retries multiply load under stress

Retries improve success rates when failures are transient and capacity is available. Under degradation, retries become a multiplier, the dynamic Jeffrey Dean and Luis André Barroso describe in “The Tail at Scale” (Communications of the ACM, February 2013).

This is how a minor regression becomes a self-inflicted flood. It can resemble a resource exhaustion event even when there is no attacker. The mechanism is simple. Unbounded retries consume the remaining capacity of an already-constrained dependency. The root issue is not that retries are bad. The issue is that unbounded retries are a powerful actuator that must be governed by system-level constraints such as SLO, cost, and blast radius.

Autoscaling is a blunt actuator fed by ambiguous signals

Autoscaling adds or removes capacity based on signals like CPU usage, request rate, or queue length. During incidents those signals mislead. Queue depth may grow because a dependency is slow, not because demand has grown. CPU may spike from retries and timeouts, not real workload. Request rates may rise simply because retried calls look like new traffic. A reactive autoscaler then adds capacity in the wrong place, driving up cost while putting even more pressure on the real bottleneck. The result is a system that becomes less stable and more expensive at exactly the moment it needs to recover.

Circuit breakers carry shock-wave potential

Circuit breakers are necessary. Without system-level coordination they create abrupt traffic shocks. If multiple clients trip simultaneously, alternate paths overload and create secondary failures that appear unrelated to the original degradation.

The common pattern is unbounded actuation in a coupled system

Retries, autoscaling, and circuit breakers are not mistakes. They are essential reliability tools. Problems arise when they operate like independent reflexes, without coordination or limits. When nothing controls how often they act, how broadly their actions affect the system, how easily changes can be reversed, or how clearly decisions can be explained, they unintentionally amplify failures. Managing this was already difficult with manually written rules. It becomes even more critical when AI agents make or trigger operational decisions.

Why agentic operations change the physics

Traditional automation such as scripts, thresholds, and fixed policies is limited in scope and relatively legible. Agentic systems expand both capability and risk because they tend to:

1. Increase velocity: propose actions faster than humans can validate non-local effects.

2. Increase scope: coordinate actions across services, clusters, and regions.

3. Optimize proxies: Charles Goodhart warned in 1975 that a statistical regularity tends to collapse once it is used as a control target, a caution later popularized as “when a measure becomes a target, it ceases to be a good measure” (”Problems of Monetary Management: The U.K. Experience,” 1975). Optimizing latency or cost in isolation can degrade reliability unless constraints are explicit.

4. Operate under uncertainty: incidents produce ambiguous signals, and partial telemetry invites misdiagnosis and overcorrection.

5. Reduce legibility unless designed otherwise: without decision traces, you cannot reconstruct why the system acted or demonstrate compliance.

So the risk is not AI in isolation. The risk is unbounded actuation under uncertainty. ARG is the discipline designed to make agentic operations governable.

A practical lens on amplification

To manage automation safely, teams need a clear way to recognize when helpful mechanisms start making things worse. Amplification is that lens. A reliability mechanism helps when it improves outcomes without adding too much extra load. It becomes harmful when it multiplies load, volatility, or system-wide effects beyond what the platform can handle. In practice, retries multiply request volume, autoscaling multiplies capacity changes and cost, traffic shifts increase pressure on dependencies, and aggressive fixes create configuration churn. ARG is essentially the discipline of keeping these amplification effects under control.

Autonomic Reliability Governance governs the automation itself

The notion that systems can manage themselves is not new. Jeffrey Kephart and David Chess articulated that aspiration more than two decades ago in “The Vision of Autonomic Computing” (IEEE Computer, January 2003). What is new is the velocity and scope of actuation introduced by agentic AI. Autonomic Reliability Governance is the discipline of designing and operating bounded, auditable, and reversible automation, including agentic operational decision layers, under explicit constraints.

• SLO constraints: latency ceilings, availability targets, error budgets.

• Cost constraints: budget caps, unit-economics bounds, runaway scaling prevention.

• Compliance constraints: auditability of decisions and actions.

ARG is not a product. It is an operating model, automation you can trust because it is governable. ARG treats operational actuation, human or agentic, as a first-class risk surface with explicit safety envelopes..

Governance primitives that make agentic operations safe

ARG becomes real when it is implemented through governance primitives that align with existing reliability practice such as SLOs, error budgets, and progressive rollout, while controlling actuation velocity and blast radius.

1) Actuation budgets rate-limit change, not only traffic

Most systems rate-limit user requests. Few rate-limit operational change. ARG introduces actuation budgets, limits on how frequently high-impact actions can occur. Scaling shifts, retry escalations, routing changes, and configuration flips consume tokens from a constrained budget. When the budget is exhausted, automation may still observe and recommend, but it cannot repeatedly perturb the system. When instability rises, slow the actuators before adding more intelligence.

2) Blast radius governance scopes every action

The difference between a safe fix and an outage multiplier is often scope. ARG constrains blast radius using canaries, segmentation across cells and regions, tiered permissions, and scoped rollouts with automatic halt conditions. Even correct actions can be wrong if applied globally in one step. Autonomy must always be localized before it is generalized.

Continue reading

Continue reading - the seven governance primitives that turn ARG from principle into practice, how to validate governed automation without touching production or private data, and why agentic operations win by becoming more governable, not more intelligent.

🛠️ Tool of the Week

Argo Rollouts - a progressive delivery controller for Kubernetes

Highlights:

• Bounds blast radius by exposing a change to a small traffic slice first, closing the gap that standard rolling updates leave open where nothing limits exposure or triggers an automated rollback on failure.

• Makes reversibility the default, since a rollback shifts traffic back to the previous version instead of triggering a fresh deploy.

• Gates promotion on evidence rather than a timer, aborting automatically when success rate or latency crosses a threshold, which mirrors the confidence gates in the feature.

• Fits GitOps workflows, where an automatic rollback surfaces as divergence from declared state and prompts investigation before anyone retries.

Learn more about Argo Rollouts

📎 Tech Briefs

• Google makes Model Armor generally available for Agent Gateway - Model Armor reaches general availability for Agent Gateway, screening agent prompts and responses against content guardrails.

• Datadog extends Bits AI to autonomous remediation at DASH - Bits AI now detects and remediates autonomously within predefined guardrails, and Agent Console tracks agent actions.

• OpenTelemetry graduates from the CNCF - OpenTelemetry graduates from the CNCF, cementing the vendor-neutral telemetry standard that grounds agent decisions in trusted context.

• Microsoft makes Azure SRE Agent generally available - Azure SRE Agent is generally available, with Review and Autonomous run modes gating remediation behind human approval.

• Grafana previews AI Observability in Grafana Cloud - Grafana previews AI Observability, treating agent sessions as first-class telemetry and alerting on policy violations and anomalies.

That’s all for today. Thank you for reading this issue of Deep Engineering.

We’ll be back next week with more expert-led content.

Keep building,

Saqib Jan

Editor-in-Chief, Deep Engineering

If your company wants to reach senior developers, software engineers, and technical decision-makers, speak to us about partnering with Deep Engineering.

A hands-on, build-along workshop where you build and run 10 production-inspired AI agents across finance, healthcare, and education. Walk away with a complete GitHub repo and implementations for OpenAI, Claude, Gemini, and local models.

🗓️ September 12th · 11:00 AM EDT onwards

Use code DEEPENG50 for 50% off the early bird price. First 10 sign-ups only.

Register here

✍️ From the editor’s desk,

Welcome to the 53rd issue of Deep Engineering!

At London Tech Week this month, NVIDIA made it exceedingly clear, showcasing a wave of UK companies building AI for environments where sovereignty is not optional. Building capable AI models is no longer the hardest problem. The real challenge is running them in environments where security, compliance, and cost aren’t negotiable. Organizations want AI they can trust, control, and operate on their own terms.

That’s exactly the challenge Rick Spencer has been solving at SUSE. As General Manager for Technology and Product at SUSE, he works with the teams behind its enterprise Linux and cloud native platforms for regulated organizations. His perspective is simple: engineers will use AI wherever it helps them move faster. But the real question isn’t whether teams should adopt AI; in fact, it is how to match the right model to the right kind of work.

In today’s issue, Spencer shares the framework his teams use to make those decisions. We discuss where agentic AI is delivering real value, why frontier models should be reserved for high-impact problems, and how SUSE keeps AI adoption practical without letting costs spiral.

You can read or watch the full Q&A interview here.

Let’s get started.

Featured Newsletter: AI Agents Simplified

AI Agents Simplified cuts through the noise with clear, actionable breakdowns of agents, automation, and what's actually worth your attention. Trusted by 58,000+ subscribers, with new issues every week and no hype.

→ Subscribe to AI Agents Simplified

Expert Insights

Not whether engineers use AI, but which AI for which work

by with Rick Spencer

Most engineering organizations start their AI adoption conversation with limits. Should engineers use it, and how much should they be allowed to? Rick Spencer sees that as the wrong question. As General Manager for Technology and Product at SUSE, he works with the teams building enterprise Linux and cloud native infrastructure for companies operating under strict compliance requirements. His starting point is practical: engineers will use AI wherever it helps them move faster.

“It’s not like, oh, don’t use AI,” he says. “That would just not be workable.”

For Spencer, the real question is not whether engineers should use AI. It is which AI belongs to which kind of work. Getting that decision right is what separates useful AI adoption from adoption that quietly burns money, trust, and control.

Three kinds of AI work, and why the distinction matters

SUSE breaks engineering use of AI into three categories, each calling for different tools, cost profiles, and levels of oversight. Spencer describes the first as daily work: statement completion and debugging that engineers rely on throughout the day. The second is agentics, where agents take care of repetitive work and interruptions that would otherwise consume engineering time. The third is what he calls curve jumping, and it is the most consequential of the three.

“That’s like when you’re just going from zero to infinity,” he explains. “You can do things with AI that you wouldn’t have tried before, like solve really deep problems in one big step.”

The value of this distinction is that it helps teams make more deliberate tooling and cost decisions instead of relying on guesswork. As Spencer puts it, the framework helps engineering managers pattern-match the right AI to the right kind of work. If a task only needs statement completion and debugging, that points to one set of tools. If it involves data sovereignty requirements, that points to another. Frontier models, the most capable and most expensive, are reserved for curve jumping, where their cost is justified by the scale and complexity of the problem.

“It sounds very organized now, but there was a lot of experimentation, a lot of really rapid innovation from the engineers,” Spencer says of how the framework came together.

The framework emerged from engineers first. Management then built structure around what those early adopters had learned so the approach could be shared across the organization. It’s a practical reminder that successful AI frameworks often evolve from experimentation before they become formal processes.

Frontier models are for curve jumping, not code completion

One of Spencer’s most practical observations is about matching model capability to the task at hand. Routing every request through the most capable model is both expensive and unnecessary, and he is clear about where the line sits.

“You do not need a frontier model to understand your Python module and give you code completion,” he says. “You just don’t need it for that.”

Frontier models earn their cost in the curve-jumping category, where the problems are strategic and complex.

“We have projects where we spend tens of thousands of dollars on frontier models, but they generated, who knows, a million or two million dollars in value,” he notes.

When the value is that high, the investment makes sense. The discipline is in not using frontier models for work that a much cheaper model can handle just as well. Spencer’s teams apply the same thinking in another way. They use frontier models to build agents that later run on much cheaper models.

“We use the frontier model to create an agent that can then be run on a much lower-cost model,” he explains.

The frontier model writes the Python scripts and prepares the context the agent needs. After that, a lower-cost model handles the repeated execution. The expensive model does the design work once, while the cheaper model runs the workflow repeatedly. For teams looking to bring frontier-level capabilities into production without paying frontier-level costs every time, it’s a practical approach that balances capability with efficiency.

Agentics is where the toil goes to die

The agentics category is where Spencer’s examples become most concrete. More importantly, they show what relieving engineering toil looks like in a production environment rather than a demo. His favorite example isn’t about writing code at all. After a series of software supply chain attacks, SUSE’s security team built an agent that scans for newly reported compromised packages every hour.

“It finds those, and then it scans all of our open source code to see if we’re using it anywhere,” he says. “If we are, it writes a report and notifies us on Slack.”

The benefit is immediate.

“If you see a report of a tool chain attack, our agent was on it before we even knew about it,” Spencer says.

Work that once required engineers to manually search through repositories now happens automatically, often before anyone has even seen the news.

His second example focuses on CVE triage, another task that has become increasingly difficult to manage manually at enterprise scale. CVEs arrive faster than teams can assess them, and many turn out not to be relevant.

“A lot of times the CVE comes in and the package is in the repo, but it is not being exposed in any way that it would matter,” Spencer says.

An agent reviews each CVE for applicability and helps generate the VEX file that documents whether the vulnerability actually affects the product. The result is that engineers spend less time sorting through reports and more time addressing the vulnerabilities that matter.

“We’re focusing our attention not on keeping up with the crush of CVE reports, but on the actual vulnerabilities,” he explains. “Our attention is reserved for actually keeping our customers safe.”

That’s the hallmark of a good agentic use case. It removes repetitive work without taking engineers away from the decisions that require human judgment.

The context that does not survive the session

As engineers move from AI-assisted coding to autonomous agents, Spencer points to a challenge his senior engineers have had to learn to manage. Context that isn’t made explicit often ends up buried in conversation history. When that history disappears, the agent’s behavior changes.

“Those history sessions can embed context which you have not made explicit in your markdown,” he says. “The next time you go and you don’t have all that history, you get different behavior than you were expecting.”

The lesson, Spencer says, is to make recurring context explicit instead of letting it live inside a session that will eventually disappear.

The challenge becomes even more significant once agents begin operating autonomously. Unlike traditional infrastructure tools, agents are not deterministic.

“This is a big change in infrastructure management,” he notes. “If I give this input, I know exactly what’s going to happen.”

With conventional tools, unexpected inputs usually produce predictable failures. Agents behave differently. When they encounter something unexpected, they try to solve the problem, and that often requires additional context, including organizational policies or guidance on how a situation should be handled.

This is where MCP servers become important. They give agents a way to retrieve the right context when they need it. At that point, context management is no longer just about writing better prompts; it becomes part of the infrastructure itself.

Cost control is a design decision, not an afterthought

Running AI at scale makes cost something teams have to design for, not react to later. Spencer sees this as part of the architecture.

At SUSE, part of the answer is structural. Self-hosted AI gives the organization a clearer cost ceiling. The question becomes how well the team can observe and use that fixed capacity, rather than whether a usage-based bill might suddenly run away.

Spencer connects this directly to sovereignty.

“Sometimes they call it cost sovereignty,” he says, “because no one can come back later and say, oh, by the way, we’re changing our model.”

He has seen suppliers move from seat-based pricing to usage-based pricing, leaving engineering teams with a cost model they did not control. Hosting your own AI infrastructure changes that equation. It gives teams more control over where AI runs, how it is governed, and what it can cost.

The governance side of that argument, including how SUSE keeps agents and their costs inside a boundary it can stand behind, is covered in the companion piece, How SUSE Runs AI Without Losing Control.

The more tactical layer is circuit breakers, which cap runaway agent spend in real time.

“We just noticed our Claude usage in the last minute was way too high,” he says, describing the trigger.

Spencer acknowledges the trade-off. Aggressive rate limiting can frustrate engineers who are trying to get work done, but it is a necessary safeguard when autonomous agents can generate costs without a human in the loop.

Like the three-tier framework, the goal is simple: match the cost of the tool to the value of the work and put clear limits around the situations where agents can spend without delivering value.

Ultimately, Spencer’s goal isn’t to slow engineers down. It’s to give them the confidence to use AI effectively without sacrificing governance or cost control.

“The penny drops for them that they’re in a new paradigm,” he says, describing the moment developers realize they can be ten or even a hundred times more productive.

The framework, the tooling, and the guardrails exist to support that shift, not to get in its way.

“You don’t want to stop them from getting that 100X improvement,” he says. “You need to give them the right tools for the job.”

For Spencer, that’s what successful AI adoption looks like: giving engineers the freedom to move faster while making sure the right tool is used for the right work.

In case you missed

🛠️ Tool of the Week

ToolHive is an open source platform from Stacklok for running and governing Model Context Protocol (MCP) servers in production.

Highlights

• Secure isolation: Runs each MCP server in its own isolated container with minimal permissions.

• Enterprise access control: Enforces per-request identity and access policies with OIDC integration.

• Self-hosted deployment: Keeps the MCP registry, gateway, and servers on your own infrastructure.

• Lower token usage: Uses semantic tool discovery to reduce token usage by up to 85%.

ToolHive

📎 Tech Briefs

• Grok Build opens a plugin marketplace - New plugin marketplace featuring tools from MongoDB, Vercel, Sentry, and Cloudflare.

• Gartner Forecasts Worldwide AI Spending - Enterprise spending is expected to grow 47% year-over-year to reach $2.59 trillion in 2026.

• Anthropic study links AI coding success to domain understanding - Domain expertise proved a stronger predictor of success than coding ability.

• Mastra npm supply chain attack disclosed - Over 144 packages were compromised through the easy-day-js typosquat dependency.

• OpenAI Codex adds granular internet access controls - Users can now restrict internet access by domain and HTTP method.

That’s all for today. Thank you for reading this issue of Deep Engineering.

We’ll be back next week with more expert-led content.

Keep building,

Saqib Jan

Editor-in-Chief, Deep Engineering

If your company wants to reach senior developers, software engineers, and technical decision-makers, speak to us about partnering with Deep Engineering.

Join this interactive workshop to learn how to turn Claude Code from a session-by-session assistant into a repeatable engineering system, using structured context, reusable skills, scoped rules, hooks, and guardrails that work across real codebases and team workflows.

🗓️ Friday, June 20 · 10:30 AM EDT onwards

Use code DEEPENG50 for 50% off.

Register here

✍️ From the editor’s desk,

Welcome to the 52nd issue of Deep Engineering!

A recent study pulled apart the architecture of Claude Code and found that only 1.6 percent of the codebase is actual AI decision logic, while the remaining 98.4 percent is the deterministic infrastructure that surrounds the model, including the permission gates, the context management, the tool routing, and the recovery logic that keep the whole thing usable. The agent loop at the center of it turns out to be a simple while loop, which means the genuine engineering effort sits in the systems built around the model rather than in the model itself.

This analysis lines up almost exactly with what most engineers working with AI tools are discovering through daily practice, which is that the thing slowing them down is rarely the intelligence of the model and is far more often the fact that none of the context they carefully supply in one session survives into the next. Supplying that missing context has quietly become the engineer’s job, and it gets paid at the start of every single session without anyone ever counting it.

This week Sam Keen, an agentic engineering researcher, and former engineer at AWS and Nike, and the author of Clean Architecture with Python, shares a practical way to stop paying that cost for good. His piece walks through how to convert the context you re-explain on repeat into a system that compounds across sessions, using the mechanisms Claude Code already gives you, and how to recognize the moment that system starts quietly working against you rather than for you.

Let’s get started.

Featured Newsletter: Engineering At Scale

A weekly column that makes databases, system design, and architecture easy to follow, with clear explanations, practical insights, and career advice for engineers building at scale.

→ Subscribe to Engineering At Scale

The Hidden Cost of Starting From Scratch

Submitted by

When you open a fresh Claude Code session, the assistant knows nothing about your project. It does not know where your tests live, it does not know the patterns this codebase uses, and it does not remember the conventions you walked it through the day before. So you explain all of it again, and then you do the same thing again tomorrow.

That re-teaching is a tax, and it is the easiest one to overlook because it never shows up on any gauge, which means you pay it at the start of every session without ever once counting what it costs you.

The bottleneck isn’t the model

The real limiting factor in AI-assisted development right now is not how clever the model is, because the models are already more than capable enough for the work most teams are asking of them. The limiting factor is that none of that intelligence carries from one session into the next, and the job of supplying the missing context every single time has quietly been handed to you without anyone naming it as work.

Picture a senior engineer who forgets everything about your codebase overnight, every single night, and shows up the next morning brilliant and fast and genuinely helpful but starting again from absolute zero. You would not describe that person as a force multiplier, you would describe the arrangement as exhausting, and yet that is the default relationship most people have with their coding assistant. They end up blaming the model for the drag when the actual problem is that nothing they explained yesterday is still present today.

Write the context down once

The fix is to stop starting from scratch, which means writing the recurring context down once in a place the harness reads automatically so that it is present in every session without you having to lift a finger. Claude Code gives you several mechanisms for doing exactly this, and three of them are foundational. The most useful way to think about the three is by the specific kind of cost each one removes from your day.

Agent files, written as CLAUDE.md, are your project’s standing memory, the place where the conventions and the layout and the way things get done here all live. You write them once and they load into every session, so you stop re-explaining the project from the beginning each time. The part most people underuse is that these files load hierarchically, which means a personal file can ride along on every session on your machine, a broader file can cover all of your coding work, and a project-specific file can sit on top of both, each one layering onto the last. The thing to remember is to keep them lean, somewhere around a couple hundred lines each rather than letting them sprawl.

Skills capture the procedures you would otherwise walk through by hand every time, the multi-step moves where you first do one thing and then check another before continuing. A procedure you would normally re-explain becomes a procedure you simply invoke, and because a skill is not limited to instructions alone, it can bundle the scripts the agent runs, which means the repeatable move can carry real executable code rather than prose describing what the code should do.

Hooks handle the corrections you would otherwise find yourself repeating, the lint nit and the formatting rule and the check you keep having to ask for. They fire at fixed points in the harness lifecycle, before or after a tool runs for instance, and because they sit outside the model’s control they run every single time regardless of whether the model would have remembered to do them. You give the note once and then you never have to give it again.

The pattern underneath all three mechanisms is the same one, because each of them converts a recurring cost into a single one-time write. That is what compounding actually means in this context, and it is the direct opposite of starting from scratch, since the investment you make is small and the payback lands in every session that follows it.

Your setup can rot, here is how to catch it

There is an honest catch worth naming here, and it is the place where the /context command earns its keep, because a compounding system can quietly rot over time. You install a skill pack and then forget it is even there. A CLAUDE.md that started out tight slowly fills up with things that mattered once and no longer do. The investment turns into freight without you noticing, and freight is really just the from-scratch tax wearing a slightly nicer costume.

The /context command is how you tell the difference, and it does one genuinely useful thing, which is that it makes the invisible visible. One command gives you a colored grid and a per-category breakdown of exactly what you are carrying before the conversation has even started. You do not need to master it, you only need to glance at it often enough to notice when something is wrong.

The last time I ran it, the single biggest chunk of my standing context was not the project memory I had carefully written, it was 14.4k tokens of skill packs I had installed on a whim and then never used even once. I had assumed my context was quietly working in my favor, and a five-second look told me otherwise. Culling them took about a minute, using /skills to deactivate the ones I never reach for and /plugin to drop a whole pack that had arrived bundled with something else.

That same readout flagged a second problem with a different fix, because the CLAUDE.md in my working directory had grown to 3,500 tokens without my noticing. I sat down with Claude and compacted it, cutting the irrelevant and the quietly duplicated, and it came back at 1,200 tokens, which is the same project memory carried at roughly a third of the weight.

Clarity helps the agent for exactly the same reason it helps a human reader, because a bloated and half-contradictory CLAUDE.md does not only cost you tokens, it actively muddies the very instructions you are leaning on to get good work out of the model. A lean file is easier for the model to follow in the same way that a tight brief is easier for a colleague to follow.

The specific number you land on does not really matter, but the habit does, so skim /context the way you would skim a credit-card statement, not obsessively but often enough to catch the recurring charge you forgot you ever signed up for.

Do the upfront work once

Starting from scratch feels free because the cost is smeared so thin across every session you will ever run, but it is not actually free, and it is in fact one of the largest and quietest line items in the way you work.

So put in the upfront work of writing a lean memory file, building a few skills you genuinely use, and setting a couple of hooks that hold the line for you. Then keep curating it, because models change and your projects change, and a glance at /context now and then is what tells you which parts of your setup are still earning their place. The goal was never a clever prompt. The goal is a setup that already knows your project before you say a single word to it.

🛠️ Tool of the Week

Repomix — an open source tool that packs an entire repository into a single, AI-friendly file ready to feed to a coding agent, with token counting built in.

Highlights

• Packs a whole repository into a single structured file optimized for LLM consumption, removing the manual copy-paste that eats the start of every session.

• Counts tokens per file and for the whole pack, so you see what context costs before you spend it rather than after.

• Ships an official skill for Claude Code, Cursor, Codex, and Copilot, letting agents run Repomix directly inside the workflow.

Learn more about Repomix

📎 Tech Briefs

• Anthropic suspends Fable 5 and Mythos 5 worldwide - A US export control directive citing national security forced Anthropic to disable both frontier models for all customers, days after launch, with all other models unaffected.

• OpenAI Codex 0.140.0 ships - Codex adds Claude Code imports, unified mentions, and encrypted Amazon Bedrock API-key authentication.

• GitHub Copilot CLI 1.0.63 released - A new deferTools option reduces MCP context bloat when tool search is enabled.

• Xiaomi open-sources MiMo Code - Xiaomi’s terminal coding agent targets long tasks and offers free limited-time MiMo Auto access.

• ChatGPT adds memory summary controls - Users can delete memories, turn memory off, and directly correct their memory summary.

That’s all for today. Thank you for reading this issue of Deep Engineering.

We’ll be back next week with more expert-led content.

Keep building,

Saqib Jan

Editor-in-Chief, Deep Engineering

If your company wants to reach senior developers, software engineers, and technical decision-makers, speak to us about partnering with Deep Engineering.

Learn how to use Claude and Codex as development partners for building reliable Rust applications faster. This hands-on workshop with Francesco Ciulla shows how to scaffold, refactor, debug, test, and productionize AI-assisted Rust code with confidence.

Use code DEEPENG50 for 50% off.

Register here

✍️ From the editor’s desk,

Welcome to the 51st issue of Deep Engineering!

The Rust team released version 1.96.0 on May 28, shipping new Copy-compatible range types, stabilized assert macros, and Cargo security fixes. It is the kind of release that tells you something important about where the language is. Now on a steady six-week release cadence, Rust’s toolchain keeps getting more integrated, and each release moves the language further from its reputation as a difficult, specialist tool and closer to something engineering teams can simply rely on.

That maturity is part of what is changing the production calculus for engineering teams this year. Francesco Ciulla, author of The Rust Programming Handbook and head of developer relations at Zerops, has used both Rust and Go in production. His view on the Rust versus Go debate is neither dismissive of Go nor evangelistic about Rust.

Ciulla discussed how Rust and Go solve different backend problems, where Go still wins, where Rust’s flat latency and binary size arguments become genuinely decisive, and why his thinking on committing to Rust for a production backend has changed.

Today’s expert insights are based on the broader conversation about Rust adoption we had with Ciulla. You can read our previous issue or watch the full Q&A here.

Let’s get started.

Featured Newsletter: Java Tips and Tricks

Subscribe to Java Tips and Tricks on Substack for practical Java insights, modern Java features, software design discussions, and updates from the Java ecosystem.

→ Subscribe to Java Tips and Tricks

Expert Insights

Rust and Go Are Better Compared Service by Service

by Saqib Jan with Francesco Ciulla

Most engineering teams debating Rust versus Go for their next backend service are usually trying to answer the question too early, assuming the decision starts with the language rather than the service.

Francesco Ciulla, author of The Rust Programming Handbook and head of developer relations at Zerops, thinks that framing misses the point. The useful question is not whether Rust is better than Go, or whether Go is more practical than Rust. It is whether the specific service being built has constraints that make one language’s trade-offs more valuable than the other’s.

The distinction Ciulla draws is practical. Go still earns its place in cloud infrastructure, CLI tooling, hiring, and teams that need a service working quickly without introducing a new adoption burden. Rust, in his view, becomes much harder to ignore when a service is constrained by memory use, latency predictability, high-concurrency performance, or the need to remove whole classes of runtime failure from the system. That is why the Rust versus Go debate becomes less useful the longer it stays at the language level. The decision only starts to make sense when it moves down to the service level.

Most teams are asking the wrong question

Most engineers who have followed the Rust versus Go conversation online will have encountered it as a tribal argument, with advocates on both sides treating the choice as a matter of identity rather than engineering judgment. Ciulla rejects that framing, and part of what makes his view useful is that he is not arguing from a Rust-only position. He currently works with Go at Zerops, where the company’s CLI is written in Go, and he has run Rust services in production on his own projects.

“I would never say that Go is a bad programming language,” Ciulla says. “You can feel how powerful Rust is because we are talking about completely different scenarios and still Rust has something to say. But that does not make Go bad.”

The more useful framing, he argues, is to stop treating Rust versus Go as a general-purpose language comparison and start treating it as a service-level engineering decision. The question is not whether an organization should adopt Rust instead of Go. The question is whether a specific service in the system has properties where Rust’s trade-offs are worth paying for, or whether Go’s simplicity, ecosystem, and hiring advantages matter more.

That reframe changes the conversation because it moves the decision away from preference and toward evidence. Once the team is looking at the service rather than the language, the relevant questions become much more concrete. Is memory use a real constraint? Is tail latency a business problem? Does the service sit on a critical path? Does the team have anyone who can review Rust code well enough to put it into production safely? Without those questions, the debate quickly becomes ideology dressed up as architecture.

Go wins on hiring, tooling, and the Docker ecosystem

Ciulla is careful not to turn the comparison into an anti-Go argument. Go is a natural fit for CLI tools, cloud infrastructure tooling, and the Docker and Kubernetes ecosystem. Docker is written in Go and Kubernetes is built on Go. For teams building tools that live inside that ecosystem, Go is the default for reasons that have little to do with language preference and everything to do with integration, community, and the availability of patterns the team can learn from.

The hiring argument also runs in Go’s favor, and Ciulla thinks engineering leaders should be honest about it. “In terms of finding Go engineers, probably at the moment it is easier,” he notes, “because there are probably more of them.” For a company that needs to move quickly and cannot afford a long search for specialized Rust talent, that matters. Staffing is not separate from engineering judgment. It is one of the constraints that determines whether a technical choice can survive contact with production.

Go also has lower adoption friction in general-purpose backend contexts where the performance ceiling is not the binding constraint. If a team is building a service that needs to be working by the end of the week, deployed reliably, and understood by the next engineer who touches it, Go’s simplicity and the breadth of its ecosystem are real advantages. Ciulla makes the same point more generally when talking about technology choices under deadline pressure. “When you need something simple, and you’re familiar already with Java or JavaScript, why don’t you use it?” The same principle applies to Go for teams already operating in that ecosystem.

That matters because Rust adoption is not free. It requires a different mental model, a compiler that forces decisions earlier, and at least one person on the team who knows the language well enough to validate what is being shipped. For a service that does not need Rust’s performance profile, those costs may not be worth paying.

Rust wins on performance and it is not a close contest

Where Ciulla becomes more direct is performance. On raw performance, he does not see much ambiguity. “Check the benchmarks yourself and send me the link where Go beats Rust,” he says. “Sometimes they are at the same level. In terms of pure performance, there is no story.”

That bluntness is useful, but the more important argument in his interview is not about benchmark wins. It is about latency predictability. Languages that rely on garbage collection, including Go, Java, and Node.js, can introduce pauses when the collector runs. An HTTP request that arrives during one of those pauses may experience higher latency than one that does not, even though the service logic is identical.

“By not having a garbage collector on the backend side, you basically have flat latency,” Ciulla explains. “You don’t rely on luck, or on the user not being the unlucky one. It’s a problem that is removed.”

For most web applications running at moderate scale, that distinction may not matter enough to justify a language change. Ciulla’s point is not that every API should be rewritten in Rust. His argument is that services with strict latency requirements, high concurrency, or service-level objectives tied to consistent tail latency should be evaluated differently from ordinary backend services. In those cases, the lack of a garbage collector is not an aesthetic language feature. It changes the runtime behavior the team has to reason about.

The resource efficiency argument is equally concrete. Ciulla describes running a Rust web server in production and observing roughly four megabytes of RAM in development and five megabytes in production. On a one-gigabyte droplet, that means many small Rust services can sit idle at once without creating the same memory pressure a heavier runtime might introduce. That is not a benchmark designed to win an online argument. It is an operational fact that changes what an infrastructure team has to provision, monitor, and pay for.

That is where Rust’s case becomes strongest. Not when the team wants a language that is fashionable, and not when someone wants to win the Rust versus Go debate, but when a specific service is expensive, latency-sensitive, memory-constrained, or sitting on a path where runtime predictability matters.

Deployment changes the operational argument

One of the more practical points Ciulla makes is that Rust changes the shape of the deployment artifact. When you run cargo build on a Rust project, you get an architecture-specific executable binary. A Windows machine produces a Windows executable. A Mac produces a Mac executable. A Linux machine produces a Linux executable. You can also cross-compile by changing the target architecture through the compiler, which lets a team produce a Linux binary from another environment when the workflow requires it.

Ciulla’s preferred production workflow is to build the Rust binary directly inside the Docker image build process. “My flow is that I prefer to build the Rust binary directly when I build the Docker image,” he explains, “so I have something which is just deployable everywhere, a Linux executable running in a Docker container.”

The appeal is straightforward. The team gets a lightweight binary compiled for the right architecture, packaged inside a portable container. “The dream for operations is having an executable inside a Docker container,” he says. “You get something which is lightweight and can run everywhere Docker is installed.”

That matters for teams already containerizing services, which is most teams building at meaningful production scale. A Rust binary inside a container keeps the deployment artifact small while preserving the operational consistency teams expect from Docker. The container still matters because real systems rarely deploy one service by hand. They orchestrate many services, restart them, replace them, and scale them across environments. As Ciulla puts it, “We are not in the 90s anymore.”

The argument is not that Rust removes the need for containers. It is that Rust and containers fit together cleanly. Rust gives the team a compact executable artifact. Docker gives the team a repeatable deployment boundary. Together, they reduce some of the runtime and packaging overhead that teams accept as normal in other ecosystems.

Rust is ready for more production backends

The most interesting part of Ciulla’s position is that it has changed. Two years ago, he would not have committed to building a paid SaaS product with a Rust backend. A year ago, he was still hedging. In 2026, he removes the qualification entirely. “If I had a paid product, I would use Rust,” he says. “Let’s remove the probably.”

That shift is not based on general enthusiasm for the language. Ciulla describes himself as skeptical by default, and his advice throughout the interview is to try technologies directly rather than rely on what advocates or critics say online. His confidence comes from the maturity he now sees in the Rust backend ecosystem, especially around Axum, which he says he would now use in production if he were building a SaaS or paid product.

The surrounding toolchain also strengthens the case. Cargo handles dependency management, building, testing, and documentation in a single integrated workflow. There is no equivalent of the npm versus yarn versus pnpm decision that JavaScript teams often navigate before they even get to the application itself. Running tests is cargo test. The integration is not a small ergonomic convenience. For teams that have spent years carrying build-system complexity, dependency churn, and fragmented tooling across projects, a coherent toolchain reduces the amount of process overhead attached to the language.

Ciulla sees that as part of why Rust’s reputation can lag behind its current reality. The language still has a learning curve, especially around ownership, lifetimes, and the borrow checker, but the ecosystem around the language has become more integrated rather than more fragmented. For experienced developers willing to learn Rust on its own terms, that changes the adoption equation.

Start the decision with the service

Ciulla’s adoption advice was consistent across our interview. Do not rewrite everything in Rust. Do not introduce it because the internet says it is the future. Do not make the language the strategy. Start with one service.

The right starting point, in his view, is a critical service with a real performance, latency, or memory problem. It might be a login service. It might be an API under heavy load. It might be a component that slows the rest of the system down. The point is to choose the service where Rust’s advantages are tied to an actual constraint rather than a general preference.

That is also where organizational readiness becomes unavoidable. If a team has no one who understands Rust well enough to review AI-generated code or validate a production deployment, the language can become a liability. “Who decides if the AI-generated Rust service is okay to put in production?” Ciulla asks. “You need the validation of an expert.”

That expert does not make Rust special. It makes Rust normal. Every new technology needs someone inside the organization who can tell the difference between code that compiles and code that should be shipped. Without that person, the team is not adopting a better tool. It is creating a new category of production risk.

For teams that do have the right service and the right internal expertise, Ciulla thinks the timing has changed. Rust is no longer only a systems language that backend teams admire from a distance. It is becoming a practical choice for the services where its properties matter most. The mistake is treating that as a universal recommendation. The opportunity is knowing exactly where it applies.

The Rust versus Go decision, then, is not really a Rust versus Go decision at all. It is a question about constraints. If the service needs simplicity, staffing depth, cloud tooling familiarity, and quick delivery, Go may be the better engineering choice. If the service needs predictable latency, low memory overhead, strong runtime control, and a deployment artifact that maps cleanly to containers, Rust deserves serious consideration. The senior engineering move is not to pick a side. It is to know which job each language is being asked to do.

Go deeper with Francesco Ciulla’s book

The Rust Programming Handbook: An End-to-End Guide to Mastering Rust Fundamentals, a practical guide to Rust’s ownership model, memory safety guarantees, concurrency patterns, trait system, and real-world use in systems and web programming.

Explore the book here

In case you missed

🛠️ Tool of the Week

cargo-nextest — a next-generation test runner for Rust that replaces cargo test with faster, more reliable test execution for production Rust projects.

Highlights

• Runs each test in its own process, surfacing hidden state sharing and enabling better parallelism than cargo test’s single-process model.

• Retries flaky tests automatically, reducing noise in CI pipelines without manual intervention.

• Outputs machine-readable JUnit XML, compatible with most CI systems out of the box.

• Version 0.9.137 added a JSON schema for user configuration, standardizing nextest settings across a workspace.

Learn more about cargo-nextest

📎 Tech Briefs

• Docker Desktop 4.77.0 released - Marketplace extensions now install by pinned manifest digest, reducing tag-mutation risk after publication.

• Rust 1.96.0 released - New Copy-compatible range types, assert_matches! macro stabilization, and two Cargo security fixes for third-party registry users.

• Go 1.26.4 released - Security fixes ship for crypto/x509, mime, and net/textproto alongside compiler and runtime bug fixes.

• Docker Desktop security update - Two container-to-host code execution CVEs in the Model Runner inference backend patched, with a Linux kernel backport fixing a container privilege escalation.

• rust-analyzer update - New diagnostics, predicate evaluation, and completion fixes improve daily Rust IDE feedback loops.

That’s all for today. Thank you for reading this issue of Deep Engineering.

We’ll be back next week with more expert-led content.

Keep building,

Saqib Jan

Editor-in-Chief, Deep Engineering

If your company wants to reach senior developers, software engineers, and technical decision-makers, speak to us about partnering with Deep Engineering.

Join this interactive workshop to learn how to turn Claude Code from a session-by-session assistant into a repeatable engineering system, using structured context, reusable skills, scoped rules, hooks, and guardrails that work across real codebases and team workflows.

🗓️ Friday, June 20 · 10:30 AM EDT onwards

Use code DEEPENG50 for 50% off.

Register here

✍️ From the editor’s desk,

Welcome to the 50th issue of Deep Engineering!

Anthropic expanded Project Glasswing on June 2, extending Claude Mythos Preview to approximately 150 new organizations for codebase vulnerability scanning, after initial partners found more than 10,000 high-severity security flaws in production code.

AI can now find vulnerabilities in production systems that were presumably tested, reviewed, and shipped by engineering teams. The gap between code that compiles and code that holds up under real-world pressure is no longer theoretical.

That gap also has a cause. Brian Allbee, Staff Software Engineer at Cleerly and author of Hands-On Software Engineering with Python (Packt), argues that programming focuses on the correctness of the code itself, while software engineering expands that focus to sustainability as change occurs. Too many developers optimize for code that works today, without enough attention to whether that code can be changed, tested, maintained, and handed off tomorrow.

Allbee joined Deep Engineering Live to discuss what closing that gap looks like in practice. Today’s expert insights are based on that conversation, and you can read or watch the full Q&A here.

Let’s get started.

Featured: All the dev content that matters, in one personalized feed

daily.dev is a professional network for developers, built around a personalized feed of the best content from across the dev ecosystem. Millions of developers use it to stay current with their stack, discover new tools and frameworks, and connect with a global community that shares what they’re learning.

Join for free at daily.dev

Expert Insights

Building Better Python Software Is Not About Writing Better Code

by Saqib Jan with Brian Allbee

Most Python developers measure their work by whether the code runs, assuming the job is done once the function returns the right value, the tests pass, and the build is green. Brian Allbee, Staff Software Engineer at Cleerly and author of Hands-On Software Engineering with Python (Packt), thinks that measure is correct but incomplete, noting that the gap between correct code and sustainable software is where most Python developers stop growing without realizing it.

The distinction Allbee draws is precise. Programming, he explained in our live interview, is focused on “the correctness of the code itself,” whereas software engineering “starts expanding out into more of a focus on sustainability as change occurs.” That shift in focus sounds subtle, but it changes almost every decision an engineer makes, from how they structure a module and handle a growing codebase to how they talk about technical debt with the people who control the roadmap.

The discipline that architecture cannot replace

The instinct when a Python codebase starts to grow is to reach for architecture by breaking things into services, introducing abstractions, and redesigning the data model. Allbee’s experience points in a different direction. “I think most of the paths to success in that context, at least the ones that I can think of that I’ve seen, don’t really start with the architecture, but with discipline behind the process,” he shares.

The discipline he describes is specific and unglamorous, emphasizing the need to keep things as simple as possible while wrapping repeated processes into functions or methods, stressing “that teams should agree on documentation standards and stick to them until something unexpected comes up, and that developers must write code with testability in mind from the beginning, even when there is no immediate requirement for tests.” These practices do not require a new framework or a redesign because they rely on consistency, which is often much harder to achieve than architecture.

The reason discipline comes before architecture is that architecture without discipline produces complexity without clarity. Allbee, in our interview, shared a vivid example with the audience from his own experience regarding a system he encountered that had been written in Python by an engineer who came from a C# background, resulting in an architecture where every function and every class had its own isolated module. The functional layers of the system were seven or eight deep depending on the context, creating a project that was “ridiculously huge,” he recalls, and “way more complicated than it needed to be, and it was hard to manage... hard to maintain.”

The problem was not the language or incompetence, but rather a mental model built for a different environment being applied wholesale to Python. Allbee points to a concept from the book Code That Fits In Your Head to explain why this matters, because humans can only keep “five to seven bits of information in the front of their memory at a given point in time.” A system with seven layers of depth saturates that capacity before a developer has even started reasoning about what any individual layer does.

Allbee argues that developers must “keep it simple” and “collapse things down to the point where you don’t have to have 19 different classes and 15 different instances of, you know, all these other classes to deal with something that really should be capable of being managed as a single function.”

Technical debt is a product decision, not a technical one

One of the more practically useful things Allbee explains about managing an evolving Python system is that technical debt is not primarily a technical problem. “Technical debt is one of those product-level priorities,” he reasons, adding that “whoever’s making the prioritization decisions is going to be in control of when those get tackled, if they get tackled.”

That framing shifts where an engineer should focus their energy when technical debt is accumulating, meaning the work is not just to identify the debt but to communicate its consequences clearly enough that the people controlling the roadmap can make an informed decision. “Making sure that you can communicate effectively, here’s what the impact of this technical debt is to your product-level people or whoever’s making those decisions, is gonna be a key thing,” he adds. That requires being able to sit down and say clearly that if the team does not deal with a bug, it is going to lead to cascading issues, and the longer they put it off, the more likely it is to lead to a really significant problem that will take even longer to get past.

The teams that handle technical debt well, in Allbee’s experience, are the ones that treat it as a first-class concern rather than an emergency. The difference between those two approaches is almost entirely about communication, where debt that gets communicated early and framed in terms of product risk gets prioritized, while debt that surfaces as a crisis gets managed badly.

Testing is a design decision

The most common framing of testing in Python projects is that it is something you add to code that already exists, but Allbee’s position is that testability is a property of the code itself, meaning that designing for it from the beginning changes the shape of the code in ways that make it easier to understand, change, and hand off to other engineers.

His testing approach for his own projects is a method that “exercises valid and invalid inputs for all of the parameters of every callable in the project.” He shares, “You combine that with judicious monitoring of missing lines and a code coverage report, that has served really well for me in making sure that the targets of those tests are being both thoroughly and realistically exercised.” The more important principle underneath the practice is that tests are most valuable when they reflect how the system is actually used, not just how the code is structured.

In team contexts, Allbee advocates for explicit agreement about how tests are organized and what tools are involved. “I’ve seen what happens when different engineers who aren’t communicating with each other each go their own way,” he points out, noting that “the tests that result, even if they’re rigorous and well thought out, are oftentimes difficult to follow across different test modules.” The investment in agreement upfront produces a test suite that the whole team can confidently read, maintain, and extend.

On AI-generated code in testing contexts, Allbee recommends defining a test suite that only humans are permitted to modify, making it as rigorous and complete as possible, and then allowing AI to generate implementation code that must pass that suite. He explains the boundary by stating that you can tell the AI to “write all the code you want,” but it “must pass this test suite” and it does “not get to modify that test suite.” That boundary, he reasons, provides about as much coverage as can realistically be achieved when AI is involved in production code.

Concurrency is a design problem first

Python’s performance limitations and its Global Interpreter Lock have been a recurring concern for engineers building high-throughput systems, and CPython’s free-threaded build has stirred interest in what Python might make possible beyond the GIL. Allbee is measured about expectations, highlighting that most Python code is IO-bound rather than CPU-bound, which is where the GIL has its most significant impact, though he is hopeful that the free-threaded model will open doors for more CPU-bound work to be written in Python.

The framing that matters most to many developers is not about the runtime at all. “Concurrency is a design problem before it’s a runtime problem,” he underscores, adding that having better concurrency support in the language really does not eliminate the need to understand how your processes are going to contend against each other, how to deal with data ownership at the scope of the code, or how failures can happen. His practical advice on concurrency reflects this directly by recommending that developers add it sparingly and only when there is an actual benefit that outweighs the overhead of handling errors, data contention, and coordination costs. “Optimize your clarity and correctness first,” he recommends, and “really only reach for concurrency when you understand where the time is actually being spent.”

Cloud readiness is designing for volatility

The question of what makes a Python application cloud-ready is one Allbee addresses in terms of design principles rather than tools or platforms. The containerized application is cloud-ready, he acknowledges, but so are function-as-a-service constructs like AWS Lambda functions, proving that the specific mechanism matters less than the underlying design orientation.

“The key concept that ties almost every cloud-resident system together, containerization, stateless design, any of those, is that they are inherently disposable,” he explains. Because a container can be killed at any time, a Lambda invocation could be terminated before it reaches a successful completion, and Kubernetes pods restarting are probably routine events, designing for that reality means building processes around the expectation that the hardware can disappear at any point in time.

Statelessness in that context is about making failure cheap. There is no state to manage and no need to write code to reacquire that state, meaning a process simply ends and is restarted, making recovery from a failure as simple as starting a new instance. “Statelessness and containerization matter more because they make failure cheap and recovery routine than for any other purpose or reason,” he says, arguing that this principle should sit near the top of the list of factors shaping design decisions for any system built to run in a cloud environment.

What senior engineers actually do

The question of what separates an engineer ready for senior work from one who is not comes back to the same systems-oriented thinking that distinguishes engineering from programming, where the indicator is not technical mastery but curiosity about the system rather than just the isolated function.

“If they started demonstrating that they’re concerned with more than just is the code doing what it’s supposed to do,” he explains, “if there’s a certain amount of curiosity, why are we doing it this way, do they recognize the trade-offs, those are the things that I think start really indicating somebody is actually ready to go beyond just I’ve written this function, and it’s done, and it’s tested, and it works. Done. I’m finished.”

The senior engineers Allbee has tried to emulate and seen do their best work are not defined by the code they write but by the systems they shape and the teams they are enabling. That involves asking questions that guide less senior engineers to ask those same questions on their own, such as why the team is going down a certain road, what the benefits are, and what trade-offs exist. “There are always trade-offs,” he notes, emphasizing, “Always, always, always trade-offs.”

The advice he offers to Python developers trying to grow in an AI-accelerated world collapses to three core principles, which are to “think in systems,” to “design for change,” and to “optimize for your team.” If you come away thinking differently about why you write the code that you are writing and not just how, then that is the shift that matters. Since the language, tools, and expectations placed on Python engineers will inevitably keep growing, the engineers who hold up under those pressures are the ones who stopped measuring their work by whether the code runs and started asking what it will take for the system to survive.

Go deeper with Brian Allbee’s book

Brian Allbee explores these ideas in more depth in Hands-On Software Engineering with Python, a practical guide to building Python systems that are easier to test, maintain, evolve, and hand off.

Explore the book here

In case you missed

🛠️ Tool of the Week

Ruff - fast Python linting and formatting for teams trying to keep quality gates enforceable without slowing every commit.

Highlights

• Consolidates linting, import sorting, upgrade checks, and formatting behind one configuration surface.

• Runs fast enough for pre-commit and CI workflows, which makes quality checks more likely to stay enabled.

• Supports monorepos and hierarchical configuration, helping larger teams avoid one-off project rules.

• Already used across major Python projects, making it a practical default rather than a niche experiment.

Learn more about Ruff

📎 Tech Briefs

• Copilot SDK is now generally available - GitHub made Copilot SDK stable across six languages, letting teams embed agent workflows into internal tools.

• Anthropic expands Project Glasswing - Project Glasswing now extends to 150 organizations, shifting AI vulnerability discovery toward coordinated patching capacity.

• Python pip 26.1.2 - Pip 26.1.2 shipped with Trusted Publishing attestations, tightening provenance for standard Python installation workflows across teams.

• Using uv in GitLab CI/CD - Astral added GitLab CI guidance for uv images and cache pruning, simplifying reproducible Python pipelines outside GitHub.

• Pyright 1.1.410 - Pyright 1.1.410 refreshed the Python wrapper package, keeping CLI and editor type checks aligned automatically.

That’s all for today. Thank you for reading this issue of Deep Engineering.

We’ll be back next week with more expert-led content.

Keep building,

Saqib Jan

Editor-in-Chief, Deep Engineering

If your company is interested in reaching an audience of senior developers, software engineers, and technical decision-makers, you may want to advertise with us.

With agents discovering and calling APIs at runtime rather than developers hardcoding them at build time, that design problem has become one of the most urgent infrastructure questions in enterprise engineering.

This month’s special issue digs into how APIs built for developers need to become discoverable, understandable, governed, and safe runtime capabilities for agents, with commentary from Erik Wilde, Head of Enterprise Strategy at Jentic; Nandita Giri, Senior Software Engineer at Microsoft; Rohan Gupta, Principal Product Manager at Harness; and Mayank Bhola, Co-Founder and Head of Products at TestMu AI.

Let’s get started.

Special issue — June 2026

Your APIs were built for developers, not agents

“You don’t just look for APIs when you’re writing an app. You kind of look for APIs every time you solve a problem.” — Erik Wilde, Head of Enterprise Strategy at Jentic and OpenAPI Ambassador

Most large enterprises have no idea how many APIs they have. Ask them and the honest answer is usually somewhere between a guess and a shrug. What they do know is that the number is large, the endpoints are larger, and the documentation is somewhere between incomplete and missing. For years that was manageable because the people consuming those APIs could compensate. They had context, experience, and enough judgment to work around the gaps in a poorly written spec or an ambiguous parameter name.

For the better part of two decades, engineering teams designed APIs for a specific kind of consumer, a developer sitting at a keyboard, reading documentation, and making deliberate decisions about which endpoints to call and in what order. That consumer had context, experience, and the judgment to fill in the gaps that a poorly written spec inevitably left open. The API did not need to be perfect because the developer compensated for its imperfections at design time, before a single line of integration code was written.

That assumption breaks down when the consumer is an agent, says Erik Wilde, Head of Enterprise Strategy at Jentic and OpenAPI Ambassador. Agents do not read between the lines, compensate for ambiguous parameter names, or infer the intent behind a generic error response. They act on what the contract says, at runtime, every time they need to solve a problem, and the gap between what most enterprise APIs offer and what agents actually need is where many AI projects begin to fail before they deliver measurable value.

Anthropic’s 2026 State of AI Agents Report found that 46% of engineering teams cite integration with existing systems as their primary challenge when deploying agents, placing it above model capability, prompt quality, and every other factor on the list. The bottleneck is infrastructure, and that infrastructure depends on APIs that were never designed for this kind of consumer.

Masterclass: Building AI-Ready APIs with Agent Skills

Join OpenAPI Ambassadors Erik Wilde and Frank Kilcommins for a hands-on masterclass on building AI-ready APIs with agent skills, covering OpenAPI, Overlay, Arazzo, semantic discovery, deterministic workflows, and governance guardrails for agent-driven integrations.

🗓️ July 1, 2026 · 10:30 AM – 1:30 PM ET · Online

Use code DEEPENG50 for 50% off.

Register here

Agents discover APIs at runtime, developers do not

In our live interview, Wilde gave one of the clearest framings for how agent consumption differs from developer consumption. Developers search for APIs when building an application, make a decision, and hardcode the integration so it stays consistent for the lifetime of the application. Agents search for APIs at runtime, every time they encounter a problem they need to solve, against a catalog that may contain hundreds of thousands of options across a large enterprise.

That changes the API problem from documentation quality to runtime selection. The consumer is no longer a skilled person who can fill in the gaps of a poorly written spec. It is a machine that acts on exactly what the contract says, nothing more and nothing less, and it does so without the accumulated context that a developer brings to the integration process.

Wilde illustrated the scale of this problem by sharing a recent experience with a car manufacturer operating roughly 50,000 APIs and 500,000 endpoints across the organization. The point is not that this number is exceptional. For a large enterprise with decades of accumulated systems and services, it is closer to the normal condition than most teams would like to admit. What changes with agents is the cost of that normal condition. When the consumer needs to find the right capability at runtime, the selection problem alone can make the API landscape effectively unusable without a serious restructuring of how capabilities are described, organized, and exposed.

Agents cannot compensate for spec drift

“Think of the API as a contract with a very literal, very curious machine.” — Nandita Giri, Senior Software Engineer at Microsoft

Nandita Giri, Senior Software Engineer at Microsoft with prior engineering experience at Meta and Amazon, works across agentic AI and automation, and the pattern she observes across organizations working to become AI-ready is consistent and predictable. Teams invest in producing a good OpenAPI specification at launch, treat it as a first-class deliverable at the time of release, and then watch the specification and the actual API behavior silently diverge over the following months as the code evolves faster than the documentation does.

For developer-facing APIs, this drift is a manageable nuisance because developers notice the discrepancy, ask questions in Teams, Slack or a GitHub issue, and someone eventually updates the documentation before the next consumer runs into the same problem. But for agent-facing APIs, spec drift is not a nuisance. It is a silent failure mode that is exceptionally difficult to trace because agents have no mechanism for noticing the discrepancy between what the spec says and how the API actually behaves. They act on what the spec says, encounter failures they cannot interpret without the surrounding context that a developer would have, and either produce incorrect results or abandon the task entirely without surfacing a meaningful error to the system that called them.

The only way to stop that drift from compounding, Giri argues, is to treat the specification as a first-class part of the release process on every change, with CI pipelines that validate spec fidelity against actual runtime behavior before deployment proceeds, not as a quarterly audit task but as a gate that blocks release when the spec and the actual behavior have diverged.

Giri is equally specific about what good specifications actually require for agent consumption, and her examples are concrete enough to apply immediately. A field called status that returns values 1, 2, and 3 is useless to an agent unless the spec also documents that 1 means New, 2 means In Progress, and 3 means Completed, because the agent has no way to infer that mapping from the field name or the values themselves. An endpoint that documents only that it returns a 400 error for bad input, without specifying which input combinations trigger that response, leaves an agent unable to prevalidate its requests or recover gracefully when the error occurs. A rate limit that appears only in external documentation and not in the spec itself is invisible to any agent that has not been specifically trained on that external documentation. These are not edge cases that organizations can deprioritize. They are the normal state of most enterprise API specifications, and they are a primary reason why agents fail in ways that produce poor results without surfacing a clear explanation of what went wrong.

The same distinction applies on the API producer side. Standard linting tools check structure, including whether a description field exists, whether it meets a minimum length, and whether required parameters are present. That structural check is genuinely useful as a first line of defense, but it cannot evaluate whether a description is written in a way that helps an agent understand what the operation is actually for.

A field that passes every linting rule can still be useless to an agent if it describes what the endpoint does technically without explaining the intent a consumer would bring to it. Descriptions need to represent intent, including what somebody would use the operation for, what constraints apply, and how the agent should reason about the result. The gap between a description that passes a linting check and a description that an agent can act on reliably is the gap that most teams are not yet closing, and closing it requires evaluation mechanisms that go beyond pattern matching on the specification itself.

Cross-service inconsistency breaks agent workflows

“It’s not just about connecting the dots for AI agents. It’s about making sure they understand what those dots mean.” — Rohan Gupta, Principal Product Manager at Harness

Rohan Gupta, Principal Product Manager at Harness, approaches the same problem from the perspective of an organization managing APIs across many teams and many services. His concern extends beyond the quality of any individual specification to the consistency of API design across the entire landscape. When agents operate in enterprise environments, they rarely interact with a single service in isolation. They move through workflows that cross multiple services, passing data and decisions from one system to another, and every inconsistency between how different teams have designed their APIs adds friction at the exact points where agents need to reason about how to connect things together.

Gupta’s view is that API specifications must be well-annotated and thoroughly documented so that agents can understand and execute the tasks they are given with accuracy and clarity, and that the design sloppiness which developers could historically compensate for becomes a structural blocker when the consumer is a machine reading a schema as its only source of truth. Missing descriptions, vague parameter names, inconsistent error handling patterns, and exposed implementation quirks that make no sense outside the context of the original development team all force agents into guesswork, and agents that guess tend to fail in ways that are difficult to reproduce and harder to debug than the original error would have been.

The governance problem becomes harder at the cross-service level. If one service in an agent’s workflow provides ambiguous or outdated information, the agent can be misled into triggering actions on a completely separate system in ways that no individual team would have anticipated or authorized. Lifecycle management for APIs that agents consume cannot focus only on backward compatibility within a single service anymore. It has to account for the cross-platform consistency and auditability of changes across every service in every workflow that agents are permitted to traverse, which is a meaningful expansion of what API governance has historically required.

APIs that work for developers fail agents in production

“If you can’t explain why your agent made a decision, you’re not ready to go live.” — Mayank Bhola, Co-Founder and Head of Products at TestMu AI

Mayank Bhola, Co-Founder and Head of Products at TestMu AI, has a practitioner’s view of where the failure patterns actually surface when organizations move from building agentic systems in development to running them in production. The pattern he observes is consistent across teams and organizations. APIs that worked reliably for developer consumption fail at meaningful rates when agents start calling them, and the root cause is almost always constraints and rules that were documented in external guides or tribal knowledge rather than encoded explicitly in the specification itself, leaving agents with no mechanism for knowing those rules exist until they violate them and encounter a failure they cannot interpret.

The fix Bhola advocates for is not simply better documentation, because better documentation that lives outside the machine-readable contract is still invisible to agents. It requires rethinking how APIs surface information about their own behavior, making all constraints explicit within the spec itself and building API surfaces that are structured to reduce the cognitive overhead agents face when trying to understand what an endpoint does, when to call it, and what the consequences of calling it incorrectly might be. For organizations with established API landscapes, he recommends maintaining two parallel layers, with a legacy developer API preserving backward compatibility for existing integrations and an AI-optimized layer built on top of it that flattens nested data structures, makes all constraints and relationships explicit, and exposes capabilities at a level of abstraction that agents can act on without needing to combine multiple lower-level calls to accomplish a single business task.

Bhola believes the industry’s biggest blind spot is assuming that successful API consumption automatically leads to reliable agent behavior. In practice, many failures emerge after the API call succeeds. The agent selects the wrong tool, misinterprets context, follows an invalid reasoning path, or takes an action that technically satisfies the request but violates business intent. This is why validation infrastructure must be designed before deployment rather than after incidents occur.

Testing agentic systems requires teams to evaluate decision quality, tool selection accuracy, reasoning traceability, and behavioral consistency under changing conditions. The goal Bhola highlights is not just to verify outputs, but to understand whether the agent arrived at those outputs for the right reasons.

Too many endpoints, not enough intent

The structural problem underneath all of this is that most enterprise APIs are too fine-grained for agents to use reliably, even when every individual specification is perfectly written and maintained. As Wilde frames it, accomplishing anything meaningful often requires combining many different endpoints in a specific order that encodes implicit business logic which is obvious to a developer who understands the domain but entirely opaque to an agent that has only the API contracts to work from.

When doing something meaningful requires chaining thirty endpoints in the right sequence, agents become confused about how to combine them, inventive in ways that produce incorrect results, or they make errors partway through the sequence that cascade into larger failures that are difficult to unwind. Wilde’s position is that AI readiness requires reducing the number of endpoints agents are exposed to and improving the business alignment and intent-based nature of the APIs that remain, so that a workflow that wants to accomplish a task ideally needs only a single tool call rather than having to orchestrate many lower-level calls in the correct order. The solution he and his colleagues at Jentic are working toward is a workflow layer that sits above the existing fine-grained API landscape, exposing business-level capabilities that are designed for runtime discovery and agent consumption rather than for developer integration at build time.

This pattern already shows up in enterprise partner integrations. Organizations with complex APIs that they expose to partners face a specific version of the fine-grained problem, where a partner integrating with a large API surface has to understand the full landscape even when they only need a small part of it, and the engineering effort of that integration is significant enough to slow or block adoption entirely.

The solution Wilde describes is building purpose-built workflows for specific partners, so that a partner only needs to understand the workflows that were designed for their particular use cases rather than navigating the full API surface independently. The underlying APIs do not change. What changes is the layer of business-level capabilities that sits above them, designed for a specific consumer’s needs rather than for maximum flexibility across all possible consumers. The benefit for agents is the same as the benefit for partners, with fewer options to navigate, clearer intent at each step, and a much lower chance of combining things incorrectly.

The insight that makes this approach worth pursuing beyond its value for agents alone is one that Wilde makes explicit. This improvement is not only valuable for agents. Any developer who currently has to call fifteen underlying APIs to accomplish a task that should conceptually be a single operation would also benefit from a better-designed capability API on top of those underlying services. The investment in agent-readiness is an investment in the overall quality and usability of the API landscape, and the returns compound across every consumer of those APIs whether that consumer is a human developer or an autonomous agent running at runtime.

The API layer is where the next two years are decided

Wilde’s view of API lifecycle management is the right closing frame for this issue. Agents do not consume APIs the way developers do. They discover capabilities at runtime, decide whether a tool looks useful in the moment, and need machine-readable signals about what the API does, what constraints apply, what side effects it may trigger, and whether it is safe to keep using.

That changes how organizations need to think about versioning, deprecation, and governance. The old model assumes that a developer reads the documentation, notices a migration notice, and updates an integration on a schedule the team can manage. Agent-facing APIs need more of that information to be visible at runtime. If an API is being deprecated, if a capability is nearing sunset, or if a safer replacement exists, the consuming system needs a way to discover that signal before it makes a decision.

This is where API lifecycle management needs to move, and organizations that invest in the governance structures to support it now will be better positioned than those that wait for the pressure to become unavoidable. The agents are already in production, and the limiting factors are no longer model capability alone but integration, security, and operational scalability, which means the API layer is where the most consequential infrastructure work of the next two years will happen for most engineering organizations.

The same design assumption that broke enterprise APIs, that the consumer has context, judgment, and the ability to fill in gaps, is present in every other infrastructure layer that agents call at runtime. Wilde’s framing brings the issue back to a practical rule. Agents should not be used to compensate for infrastructure that fails to express intent, constraints, lifecycle state, or safe operating boundaries. The teams that build on infrastructure designed to make those signals explicit will ship more reliable agentic systems than those still working around infrastructure that was never designed for this kind of consumer.

Thank you for reading this special issue of Deep Engineering on why the API layer has become the most consequential infrastructure problem in enterprise AI.

We’ll be back on Thursday with more expert-led content, and next month, on the first Tuesday of July, with another special issue.

Keep building,
Saqib Jan
Editor-in-Chief, Deep Engineering

daily.dev is a professional network for developers, built around a personalized feed of the best content from across the dev ecosystem. Millions of developers use it to stay current with their stack, discover new tools and frameworks, and connect with a global community that shares what they're learning.

Whether you're an early-career engineer levelling up or a senior dev tracking what's next, daily.dev makes sure the signal reaches you - without the noise.

Join for free at daily.dev

✍️ From the editor’s desk,

Welcome to the 49th issue of Deep Engineering!

Earlier this month, CISA and its international cybersecurity partners released Careful Adoption of Agentic AI Services, a guide for organisations adopting AI systems that can plan, use tools, access data, and act across digital environments. That changes the risk model because AI systems operating inside real workflows inherit risk from the surrounding data, permissions, tools, and context.

That risk is becoming easier to understand in practice. On 23 May 2026, Rohan Pandey of DigitalOcean and Archit Bhujang of Arizona State University published Poisoning the Watchtower, which shows how logs, alerts, URLs, payloads, DNS queries, and usernames can carry attacker written instructions into LLM assisted security workflows.

AI systems do not only consume clean prompts from users. They consume context from operational systems, open web sources, documents, logs, tools, and knowledge bases that the model does not control. Once that context includes contradiction, deception, malicious text, or attacker controlled content, relevance alone becomes an unsafe target for retrieval and summarisation.

David Knickerbocker, founder of Verdant Intelligence and author of Network Science with Python (Packt), builds systems for Open Source Intelligence (OSINT) environments where messy and adversarial data is normal. His perspective matters in this issue because the systems he builds separate observing from judging, treat claims as claims rather than facts, and preserve minority signals that simpler retrieval pipelines often discard.

In issue 43, we looked at Knickerbocker’s work on real-time knowledge graphs and AI systems that treat knowledge as a live stream of claims. Today’s issue continues that conversation with what OSINT teaches engineers about messy, adversarial data. You can also watch our interview or read the full Q&A here.

Let’s get started.

Claude Code for Software Engineering

Learn how to structure Claude Code with context, reusable skills, scoped instructions, and guardrails so it works reliably across real codebases and team workflows.

🗓️ Friday, June 20 · 10:30 AM EDT onwards

Use code DEEPENG50 for 50% off.

Register here

Expert Insights

Building AI Systems That Handle Contradiction at Scale

by Saqib Jan with David Knickerbocker

Most engineers building AI systems have never had to question whether their data source is working against them. The data comes, is processed, retrieved, and the system responds. The assumption underneath all of that is that the source is cooperative, that it was created to convey information accurately, stored in a format designed for retrieval, and that what comes back when you query it is at least an honest attempt at an answer. The problem is that assumption is so embedded in how most AI systems are designed that it never gets examined.

David Knickerbocker, founder of Verdant Intelligence and author of Network Science with Python (Packt), builds systems for environments where data is not clean, settled, or cooperative. His AI systems ingest from the open web, across sources that contradict each other, where some information may be misleading, incomplete, or adversarial. The engineering challenge is not only making retrieval accurate. It is making the system useful when the real world refuses to behave like a clean dataset.

The assumption that data is helpful

Engineers who have worked primarily with internal databases, structured APIs, or carefully assembled training sets carry a baseline assumption that data is cooperative. It was created to convey information accurately, stored in a format designed for retrieval, and accessed through interfaces that return what was asked for. The job of the retrieval system is to find the right thing efficiently.

Open-source intelligence does not work this way. When ingesting from the open web at scale, some fraction of what arrives is wrong, some is deliberately misleading, and some represents one side of a contested claim. For Knickerbocker, the ingestion layer is not the right place to decide what is true. “You can have two different groups that are in opposition from each other,” he says. “One group will say this is the truth, and another group will say this is the truth, and they will be in direct conflict with each other.” The system’s job, in that moment, is to capture what is being claimed and preserve enough context for judgment to happen later.

The practical design response is to treat the system as an observer rather than an adjudicator. Knickerbocker draws that line clearly. “My systems do not care who is right or wrong,” he explains. “They just do not. My systems are observers.” The point is not neutrality as a value statement. It is an architectural boundary. The system captures what is being said, keeps competing claims visible, and avoids collapsing observation into judgment too early.

This distinction matters far beyond open-source intelligence. Any AI system that draws on user-generated content, social media, news, or unstructured enterprise data is working with material that was not created to be machine-readable and was not vetted before ingestion. The assumption that the data is trying to help is not just wrong in those environments. It is a liability.

Bigger clusters are not more important than smaller ones

One of the quieter failures in production NLP systems is the treatment of minority signals as noise. A similarity-based retrieval system returns the most representative results, which in practice means the most common results. A clustering pipeline that surfaces the largest groups first will consistently deprioritize small but significant signals. In a world where the interesting thing is often the outlier, that is a serious problem.

In open-source intelligence specifically, this failure mode has consequences. A small cluster of claims pointing toward something dangerous is not less important because it is small. A single source saying something that contradicts the majority view is not less worth capturing because it is in the minority. “Bigger clusters are not more important than smaller clusters,” Knickerbocker observes. “In open source intelligence, everything matters, top to bottom.”

Drawing from his engineering experience building these systems, Knickerbocker ensures his APIs return full context rather than a ranked shortlist. “If you use a tool to do a search to find out something, you are getting a snapshot of time,” he says. His systems are designed to capture what he calls the heartbeat of the internet. “If I use my API... it is going to come back with 10,000 things. My APIs do not return 10. They return full context.” That creates a harder downstream problem because the question is no longer how to retrieve the best few results. It is how to make a large, shifting body of claims usable without discarding the signals that do not look dominant at first.

The parallel for general AI systems is specific and direct. Any retrieval or summarisation pipeline that privileges majority signal is making a judgment call that the most common view is the most relevant one. That judgment call is often wrong, and it is invisible because the discarded minority signal never surfaces.

The difference between a claim and a fact

Engineers trained on factual datasets tend to build systems that treat retrieved content as facts to be combined and presented. The underlying assumption is that if the source is credible and the retrieval is accurate, what comes back is true. In a contested information environment that assumption collapses immediately, and the design has to change with it.

Knickerbocker’s approach separates the task of capturing claims from the task of evaluating them. What a source says is observable. Whether what it says is correct requires judgment that depends on context, corroboration, and often human expertise that the system does not have. Turning that claim into an evaluated fact requires a different layer of judgment, and Knickerbocker is careful not to build that decision into the first act of ingestion. “I do not make that decision, and I do not allow my AI to make the decision what is true or what is false either,” he says. “I am more interested in what people are claiming is what is going on in the world.”

This design choice has a significant downstream consequence. It means the system can handle contradiction without breaking. Two sources saying opposite things about the same event are not a problem to resolve at the retrieval layer. They are two data points, both of which belong in the response. Knickerbocker simply logs these varied claims as parallel ribbons of information. The human or the downstream system that receives them can then apply judgment about which to act on, in what context, and with what confidence.

The verification boundary

One of the hardest design decisions in any AI system that works with real-world data is where to draw the line between surfacing an insight and making an actionable claim. The two feel similar at the output layer but require very different things from the system that produces them.

In our live interview, Knickerbocker was specific about where that line sits. “Everything that I do is intentional,” he shares. His real-time intelligence layer is built for awareness. It captures what is happening and surfaces it without making the final judgment on what should be done next. If a piece of intelligence looks actionable, the system does not automatically act on it. It surfaces the signal so a human or downstream process can decide whether it matters, who should see it, and what level of confidence is appropriate.

In practice, this means that even when a piece of intelligence looks clearly actionable, the system does not act on it. It surfaces it. The routing of that intelligence to the right person or the right downstream process is a separate engineering and organisational problem, and conflating it with the retrieval problem produces systems that are either too conservative to be useful or too confident to be trusted.

This is a principle with broad application. AI systems that are asked to be both the observer and the actor tend to perform neither role well. Keeping the observation layer and the action layer separate, with a clear boundary between them, is one of the most reliable ways to build something that stays trustworthy as it scales.

Entity extraction gets easier but never clean

Entity extraction from clean text is comparatively well understood. The models are good, the cleanup is manageable, and the output is reliable enough for most downstream uses. Entity extraction from the open web at scale is a different challenge, not because the models are worse but because the data has properties that laboratory text does not.

Knickerbocker began this work in 2018, starting with part-of-speech tagging before NER models were mature, moving to spaCy as those models improved, and more recently using LLMs for extraction. The trajectory is one of improving reliability rather than changing fundamentals. “Entity extraction has improved a lot since 2015,” he notes. “I mostly have to just throw away less. I have less cleaning to do, and it gets things right a lot easier.”

What has not changed is the messiness. Natural language processing at scale on real-world text always produces noise. The question is how much noise is acceptable for the downstream use case and how to handle the cleaning efficiently. At the scale he describes from previous work, including entity extraction across internet-scale datasets, the cleaning cannot be purely manual. It has to be part of the pipeline rather than an editorial step applied after the fact.

He also flags a risk in the current extraction approach that is worth understanding. Older NLP models produced visible noise that engineers learned to catch and correct. LLM-based extraction produces outputs that look clean even when they are wrong, because the model is good at generating confident-looking text regardless of underlying accuracy.

The implication for engineers is that moving to LLMs for extraction does not reduce the need for validation. It makes validation harder to remember because the outputs no longer look like they need it.

Building for the world that actually exists

The thread running through Knickerbocker’s work is a commitment to grounding. He builds systems for the world as it is, not a cleaned version of it. That leads to a specific set of design choices: treat data as claims rather than facts, preserve minority signals, separate awareness from judgment, and let the system observe before any person or downstream workflow decides what to do next.

Those principles come from the kinds of environments Knickerbocker has worked in: data operations, cybersecurity, open-source intelligence, and production systems where the cost of getting something wrong is real. “The real world is a messy space,” he says. “Natural language processing is just messy. I have not seen it get really cleaned up yet.”

For engineers who have worked mostly with clean internal systems, that might sound like a warning about a narrow class of hard problems. It is broader than that. Any AI system that deals with content created by people, pulled from the web, generated by users, or routed through operational systems eventually has to confront the same condition. Real-world data is messy by default. The systems that handle it well are intentionally designed for that mess before it becomes a production failure.

In case you missed

🛠️ Tool of the Week

GraphRAG — A graph-based retrieval pipeline for unstructured text

GraphRAG helps teams preserve relationships across messy documents, conflicting claims, and large text collections before asking an LLM to answer.

Highlights:

• Builds knowledge graphs from unstructured text instead of relying only on isolated chunks.

• Links entities, claims, and topics so retrieval can use structure, not just similarity.

• Supports local and global search for both narrow evidence lookup and corpus-level synthesis.

• Gives engineers a practical starting point for testing graph-based RAG patterns.

Learn more about GraphRag

📎 Tech Briefs

• Claude Compliance API Integrations - Compliance API integrations help IT and security teams govern Claude across connected enterprise workflows.

• MCP Events Working Groups - Gateway, transport, registry, and agents groups advanced protocol work around tool-connected AI systems.

• RAGFlow v0.25.6 - Browser agents and RAPTOR AHC mode expand RAGFlow from document retrieval into web-aware ingestion workflows.

• Qdrant v1.18.1 — Vector dimension validation before WAL writes reduces ingestion failure risk during async upserts.

• Weaviate v1.38.0-rc.0 - Nested object filtering and namespace support improve retrieval precision for structured, multi-tenant corpora.

That’s all for today. Thank you for reading this issue of Deep Engineering.

We’ll be back next week with more expert-led content.

Keep building,

Saqib Jan

Editor-in-Chief, Deep Engineering

If your company is interested in reaching an audience of senior developers, software engineers, and technical decision-makers, you may want to advertise with us.

A hands-on workshop covering how to build production-grade AI agents using Java and LangChain4J.

🗓️ Friday, June 13 · 10:00 AM – 1:30 PM ET · Online

2 for 1 deal is live. Use code DEEPENG50 for 50% off.

Register here

✍️ From the editor’s desk,

Welcome to the 48th issue of Deep Engineering!

Google announced Managed Agents in the Gemini API two days ago at Google I/O, making it possible to spin up an agent that can reason, use tools, execute code, and browse the web with a single API call. The infrastructure work that previously required teams to build and manage sandboxes, scaffolding, and execution environments is being abstracted away. The capability is in public preview and Google is clear that outputs should be reviewed before use in sensitive workflows, but the direction is quite clear. Deploying agents is getting significantly easier.

What is not getting easier at the same pace is making the APIs those agents will call worth calling. APIs designed for actual developers, who can tolerate ambiguous descriptions, infer intent from sparse documentation, and navigate hundreds of operations to find the right one, do not work the same way for agents. Agents are less reliable at resolving ambiguous API semantics, choosing among many overlapping operations, and safely composing actions without machine-readable contracts and guardrails.

Erik Wilde, Head of Enterprise Strategy at Jentic and OpenAPI Ambassador at the OpenAPI Initiative, has spent considerable time on solving for that gap. We spoke with Wilde about what agent-ready actually means in practice, and he explained from his engineering purview why MCP will not fix a poorly designed API foundation, and what platform engineers should start planning for today.

The expert insights in today's issue are based on our recent live interview with Wilde and you can read or watch the full Q&A here.

Let’s get started.

Featured Newsletter: Machine Learning at Scale.

Are you a SWE looking to upskill into ML systems? Get high quality ML system design content delivered to your inbox. Learn how to design and scale Machine Learning Systems.

Subscribe to Machine Learning at Scale

🧠 Expert Insights

Your APIs Are Not Ready for Agents, and MCP Will Not Fix That

by with

The conversation about AI agents in enterprise software dominates engineering mindshare as to how agents will consume APIs and what it actually takes to make that consumption work reliably. Most organisations have taken the shortcut. They have built an MCP server, pointed it at their existing API landscape, and told themselves the agent problem is solved. Erik Wilde, Head of Enterprise Strategy at Jentic and OpenAPI Ambassador at the OpenAPI Initiative, thinks that is the wrong bet, and his reasoning is specific enough to be useful.

“Whatever you invest in better APIs becomes useful for everybody,” Wilde affirms. “If you invest specifically in MCP, that investment is effectively scoped to LLM consumers.” The point is not that MCP is useless. It is that MCP is a delivery mechanism, and delivery mechanisms change. The API foundation underneath it does not change nearly as quickly, and if that foundation is poorly designed for the agents that will eventually consume it, no amount of tooling stacked on top of it will compensate. The organisations that will be in the strongest position in two years are the ones investing in the foundation now, not the ones chasing the current delivery protocol.

The abstraction level problem

The clearest way to understand what makes an API agent-ready is to look at a concrete example, and Wilde in our interview offered one that makes the problem immediately legible. The GitHub REST API currently has around 1,100 operations. That is not unreasonable for a product as complex as GitHub. A developer can navigate 1,100 operations because they bring context, experience, and the ability to read documentation and infer intent. They know roughly what they are looking for and they can work toward it even when the path is not obvious.

An agent does not work that way. “For an agent to work directly with that GitHub API is pretty complex,” Wilde points out, “because a lot of those operations need to be combined in a certain way to result in the workflows that you really want to accomplish on GitHub.” The agent has to figure out not just what each individual operation does but how they compose, in what order, under what conditions, and with what dependencies. With 1,100 operations, the combinatorial space of possible workflows is enormous, and agents navigating it without guidance will produce unreliable results.

Now look at the GitHub MCP server, which has around 70 tools. Each of those tools represents a higher-level workflow, something a developer might actually want to accomplish on GitHub rather than a low-level operation that contributes to that accomplishment. The reduction from 1,100 to 70 is not a loss of capability. It is a gain in usability for the specific class of consumer that is trying to get things done rather than explore a surface. “What I would say,” Wilde argues on this point, “is that if you had a genuinely agent-friendly GitHub API, it might also just have around 70 operations.” The MCP server is not adding something new. It is providing the abstraction level that the underlying API should have provided in the first place.

This is the abstraction level problem, and it is the most important design question for engineering teams building API infrastructure that agents will consume. The APIs that were designed for developer flexibility, with many fine-grained operations that compose in powerful ways, are exactly the wrong shape for agents that need to accomplish specific goals reliably. The discipline of designing for agents is the discipline of asking what a consumer actually wants to accomplish and surfacing that at the API level, rather than exposing every atomic capability and leaving the composition to the consumer.

What agent-ready actually means

The properties that follow from the abstraction level insight are consistent and actionable. An API designed for agent consumption should not be too fine-grained, and its descriptions should be intent-based and written at a level that is meaningful for a language model rather than just technically accurate for a developer who already knows the domain. It should have examples, ideally multiple examples per operation rather than one, because examples are one of the most reliable ways for a model to understand what an operation actually does in practice. Its error messages should be meaningful enough that an agent encountering a failure has enough information to understand what happened and what it might do next.

“If an AI agent looks at a poorly described API and cannot figure out how it works, it will just move on to the next one,” Wilde notes. “It has less context. It has less experience. It does not really know as well as an actual developer what to do.” This is the practical consequence of the abstraction level problem at the description level. A developer reading a sparse API description can fill in the gaps from domain knowledge and engineering experience. An agent cannot do that reliably, and the result is not a helpful error or a clarifying question. It is a silent failure or a wrong action.

Wilde and his team have built a scoring mechanism for API readiness that makes these dimensions concrete. The scoring uses a combination of standard linting, running tools like Spectral and Redocly to check structural conditions, and LLM-based checks that evaluate whether descriptions are written in a way that is genuinely useful for an agent rather than just present. The distinction matters because a description that exists and passes a structural check may still be useless for an agent if it describes what an operation does technically without explaining what a consumer would use it to accomplish. “These descriptions need to represent intent,” Wilde highlights. “What is the intent of somebody who would use this operation?”

Linting though necessary is not sufficient

Linting has become standard practice in well-run API programs, and Wilde endorses it as a first line of defense. The popular tools are capable and in some cases open source, and the practice of defining shared rule sets that teams can discuss, extend, and maintain in version control is genuinely useful. But in our conversation he was clear that linting alone does not get you to agent-ready, and teams that treat it as the complete solution are leaving the most important problems unaddressed.

The structural checks that linting tools perform are exactly that. They can tell you whether a description field exists and whether it meets a minimum length requirement, but they cannot tell you whether the description is written in a way that helps an agent understand what the operation is for. They can flag a missing example but cannot evaluate whether the examples present give a model enough signal to use the operation correctly in a novel context. The gap between what linting checks and what agent readiness requires is the gap between structure and meaning, and closing it requires evaluation mechanisms that go beyond pattern matching on OpenAPI descriptions.

Wilde also makes an important point about rule set governance that is worth taking seriously. “I am not a big fan of just reusing existing rule sets,” he contends. “I would always say start owning this, build up your own in a collaborative fashion.” The Zalando and Adidas rule sets that circulate in the API community are useful references, but they were built for specific contexts and specific quality standards. Adopting them wholesale means inheriting decisions that were made for a different organisation’s constraints. The value of a rule set comes not just from the rules it contains but from the process by which those rules were agreed upon, which is a process that builds shared understanding of what good API design actually means in a particular context.

MCP is a delivery mechanism, not a foundation

MCP has been growing fast. It is now under the Linux Foundation, major model providers support it, and a growing number of enterprise vendors are shipping MCP servers as a standard part of their product offering. For engineers deciding where to invest, it looks like an obvious answer to the question of how to make APIs accessible to agents.

Wilde’s skepticism is not about MCP’s current momentum. It is about what MCP is and what it is not. “MCP is the current delivery mechanism,” he says. “You need a delivery mechanism, but I would not build too many things that are MCP-specific.” At Jentic, the team supports MCP because it is what the market expects right now, but they have deliberately avoided deep investment in MCP-specific infrastructure. If MCP were replaced by something else, the transition would be straightforward because the underlying work, making APIs well-described, well-structured, and semantically rich, would carry over entirely. That work is not MCP-dependent. It is foundational.

The risk for teams that invert this priority is real. Building an MCP server on top of a poorly designed API landscape means the MCP server inherits all of those same problems. Operations that are too fine-grained stay too fine-grained, descriptions that lack intent stay unreadable to a model, and error messages that tell a human nothing tell an agent even less. The wrapper changes the protocol by which those problems reach the agent, not the problems themselves.

Open standards outlast any delivery protocol

One of the clearest threads in Wilde’s thinking is the value of building on open standards rather than specific tools or protocols. This is not an abstract preference for openness. It is a practical argument about optionality. Teams that build their API practices on OpenAPI, Arazzo, and Overlays are building on specifications that are independent of any vendor, any model provider, and any current delivery protocol including MCP. When the next delivery mechanism arrives, or when the current tooling landscape shifts, the foundation remains.

Arazzo is worth understanding in this context. It is a workflow language published by the OpenAPI Initiative that allows you to describe sequences of API interactions in a standardised format. If accomplishing a particular goal requires calling five endpoints in a specific order with specific dependencies, Arazzo is the language for expressing that. For agents, which struggle with exactly this kind of multi-step composition, a well-constructed Arazzo workflow is one of the most useful things an API producer can provide. “Figuring out multi-step workflows is one of the hardest things for agents to do right now,” Wilde says, “and Arazzo is genuinely good at describing those. We just need to make it discoverable.”

Overlays, the third specification from the OpenAPI Initiative, provides a way to express changes to an OpenAPI description in a standardised diff format. “We use overlays,” Wilde shares, “to deliver improvement suggestions alongside API scores. When the scoring mechanism identifies that an API is not well-designed for AI consumption, it also produces an Overlay that shows exactly what would need to change to improve it.” That makes the gap between current state and agent-ready state concrete and actionable rather than a list of abstract recommendations.

The APIs you design today will still be running in two years

The practical implication of everything Wilde argues is a specific recommendation about timing. API landscapes change slowly. Whatever is designed or changed today will likely remain largely unchanged for one to three years. Agents are arriving in enterprise contexts incrementally but consistently. The customer support and HR agents that are already deployed broadly are the early wave, and the business agents with genuine decision-making authority are behind them.

“API landscapes evolve slowly,” Wilde says. “Whatever you design or change today, you will probably have around for a year or two or three before you touch it again.” The teams that start building API readiness for agents now are the teams whose infrastructure will be in the right shape when agents with more capability and more authority arrive. The teams that wait for agents to become mainstream before improving their APIs will find themselves doing expensive remediation work on a landscape that is already in production and already depended upon.

The recommendation is not to stop shipping features or to redesign everything at once. It is to make agent readiness a standard consideration in the decisions that are already being made. When writing a new operation, write the description for an agent as well as for a developer. When adding examples, add enough that a model can generalise. When defining error responses, add enough context that a consumer without domain knowledge can understand what happened. These are not large investments per decision and they compound over time into an API landscape that agents can actually use.

To this end, “All the platform people out there who are building API platforms or doing platform engineering,” Wilde says, “think about how all of this will change if you have more and more agentic actors and consumers in your organisation, and start planning for that today, even if you can say that right now you do not have it this much and it is going to be another year or two. It is going to arrive.”

In case you missed

Here’s the full Q&A with the interview video featuring Erik Wilde.

🛠️ Tool of the Week

Spectral — open-source JSON and YAML linter with built-in support for OpenAPI, Arazzo, and AsyncAPI

• Validates OpenAPI v3.1, v3.0, v2.0, Arazzo v1.0, and AsyncAPI v2.x out of the box.

• Supports fully custom rule sets, letting teams build and own their own governance standards.

• Integrates with VS Code, JetBrains, GitHub Actions, and Azure API Center for shift-left linting.

Learn more about Spectral

📎 Tech Briefs

• Google Managed Agents now in Gemini API - A single API call now provisions an ephemeral Linux sandbox with code execution, web browsing, and tool use built in.

• Kyverno 1.18 released post-CNCF graduation - First post-graduation release patches two SSRF CVEs and adds cleanup policy support to the Kubernetes policy engine.

• OpenAI and Dell bring Codex to on-premises enterprise - The partnership makes the Codex coding agent available in hybrid and air-gapped enterprise environments for the first time.

• A2A protocol underpins Google’s full agent stack - Agents built at any abstraction level can be called as sub-agents across the entire Google Cloud agent platform.

• 43% of AI-generated code fails in production - Survey of 200 SRE leaders finds teams average three production redeploy cycles to verify a single AI-suggested fix.

That’s all for today. Thank you so very much for reading this issue of Deep Engineering.

We’ll be back next week with more expert-led content.

Stay awesome,

Saqib Jan

Editor-in-Chief, Deep Engineering

If your company is interested in reaching an audience of senior developers, software engineers, and technical decision-makers, you may want to advertise with us.

This hands-on workshop teaches you to build reliable, production-ready AI systems using eval-driven development. Taught by Imran Ahmad, Data Scientist and author of 50 Algorithms Every Programmer Should Know.

🗓️ May 30 · 11:00 AM – 3:30 PM ET

Use code EDD50 for 50% off.

Register here

✍️ From the editor’s desk,

Welcome to the 47th issue of Deep Engineering!

Debian’s APT package manager is moving toward the Rust threshold its maintainers set more than six months ago. APT maintainer Julian Andres Klode announced on the Debian developer mailing list that hard Rust dependencies and Rust code would be introduced no earlier than May 2026, citing memory safety and stronger unit testing as reasons to move core parsing and signature-verification paths toward Rust and the Sequoia ecosystem. For a tool that underpins Debian, Ubuntu, and their many derivatives, this is not a marginal adoption story. It is Rust moving into infrastructure that enormous numbers of systems rely on every day.

That kind of decision does not get made because Rust is fashionable. It gets made because the language can shift whole classes of errors from production failures into compile-time constraints. That is precisely the argument Evan Williams, senior software engineer and author of Design Patterns and Best Practices in Rust, makes in this week’s issue. We spoke with Williams about what it actually takes to think in Rust, why the borrow checker is a design tool rather than a compiler obstacle, and why he found it harder to write bad Rust than good Rust when working on the book. You can watch our interview or read the full Q&A here.

Let’s get started.

Featured Newsletter: DevOps Bulletin

If you work across DevOps, Cloud Native, AI and security and want a weekly read that surfaces the most relevant open-source tooling, stories, and insights in the space, DevOps Bulletin is worth adding to your reading list.

Subscribe to DevOps Bulletin

Expert Insights

Rust Makes It Harder to Write Bad Code Than Good Code

by Saqib Jan with Evan Williams

Most engineers who struggle with Rust describe the same experience. The compiler rejects code that would compile without complaint in C++ or Java, and the borrow checker surfaces errors that feel arbitrary until, gradually, they start to feel like something else entirely. Evan Williams, author of Design Patterns and Best Practices in Rust (Packt), has a precise name for what they are. They are design feedback, not feedback about syntax or style, but about the structure of the program itself, the shape of data flow, the discipline of ownership, and the decisions about who controls what and for how long. “Rust is your partner in doing that,” Williams says. “You can still write code with bugs in it, but Rust makes it harder to do that and easier to write code that’s going to be solid.”

That framing changes how to think about the borrow checker, and it changes how to think about what Rust actually is. Most languages make it easy to write code that works in isolation. Rust makes it hard to write code that fails in combination, and the difference matters more as systems grow.

The borrow checker is enforcing design, not syntax

Most engineers who pick up Rust treat borrow checker errors as obstacles to route around. The instinct is understandable because in Java or Python, the path from a failing compiler error to working code runs through adjustment: find what the compiler dislikes, change it, move on. Rust works differently, and engineers who apply the same strategy find that routing around the borrow checker is possible in the short term and damaging in the long term.

“The borrow checker is your friend because it prevents you from making a messy design. It prevents you from making a broken design. It prevents you from writing whole classes of bugs that you will then spend many hours trying to find,” Williams says. “I have found it to be an incredible partner in writing code that allows me to sleep at night.”

The reason the borrow checker behaves this way is structural. Java and Python allow data to be accessed from many places at once, which gives engineers flexibility but leaves the responsibility of managing that access entirely with the programmer. Rust removes that flexibility. A value has one owner. References are either shared and immutable or exclusive and mutable, never both at the same time. This constraint forces the programmer to be explicit about who owns what and when, because the compiler will not let the program proceed otherwise. The practical consequence is that programs which compile in Rust tend to have a quality that programs in other languages achieve only through discipline: their data flows are explicit. You can read a Rust program and understand, without running it, who controls which piece of state, when that control transfers, and what happens at the boundary.

“The principles that the borrow checker forces you to adhere to in Rust are the exact principles that you should be using in every programming language,” Williams reasons. “But you don’t have to. So it’s very easy to not think about those things.”

The object-oriented trap

The single most common mistake engineers make when getting started with Rust is treating it as an object-oriented language. It resembles one superficially, with structs, methods, and something that looks like encapsulation, but it has no inheritance, no abstract base classes, and no shared mutable state by default. An engineer who brings a Java or C++ mental model will find that the things they reach for instinctively are either unavailable or actively counterproductive.

“If you carry with you an object-oriented language mindset, then you’re going to have nothing but trouble,” Williams says. “The more experienced you are, the more years you have doing something in some other language, the more trouble you’re likely to have, because you have patterns of thought that come from those languages that you don’t even realize are there.”

This is a precise observation about how expertise transfers, or fails to. An engineer with ten years in Java has a large inventory of solutions to common problems, and most of those solutions depend on inheritance, shared mutable references, or runtime polymorphism through interfaces. In Rust, none of those approaches work as expected. The patterns are not wrong in their original context. They are wrong for this one, and the difficulty is that the engineer applying them does not recognize the mismatch until the borrow checker makes it unavoidable.

The design patterns that experienced engineers carry into Rust need to be examined before use, not applied by default. Some evolve into new forms because Rust’s enums and advanced generics make several classical patterns either less necessary or unnecessary entirely. Others require fundamental rethinking. The Singleton pattern, useful enough in Java and Python that engineers reach for it without deliberation, tends to become either redundant or actively problematic in Rust. “In Rust, it tends to be either completely unnecessary because other features of the language make it unneeded, or it tends to encourage designs that are really not necessary and where a much better approach could be used,” Williams says.

The replacement for inheritance in most cases is traits, which provide polymorphism without the coupling that comes from sharing a class hierarchy. The discipline required to work with traits well is the same discipline the borrow checker enforces on data: think about the boundaries, be explicit about what crosses them, and design the structure before writing the code.

Ownership as architecture

The ownership model does more than prevent bugs at the function level. It shapes the architecture of the system, because the rules that apply to individual values apply at every level of scale. A program that handles data ownership correctly in a single function has to handle it correctly across modules, across threads, and across the boundaries between components. The borrow checker enforces this at compile time, which means architectural decisions that in other languages can be deferred until the system grows large enough to break start being made from the beginning.

“You need to think about who controls what, how it is controlled, and you need to start from the very beginning thinking about the boundaries of your program and the system architecture, dividing things up into areas of responsibility,” Williams says. “Because unlike Python or Java, you can’t have links going all over the place. The borrow checker is never going to accept that.”

This constraint produces a specific architectural tendency in well-written Rust systems: data flows in one direction. Rather than components that hold references to each other in a web of mutual dependency, Rust systems tend toward chains of ownership that move in one direction and do not loop back. “By saying, I have a chain of ownership that moves down but never moves back up, you are now much more likely to have a system that is going to work,” Williams says. “Data flowing down is something that feels natural and smooth and just works. Data trying to fight the stream back up is going to end up giving you problems because the borrow checker is not going to like you.”

The architectural benefit of this tendency is legibility as much as correctness. A system where data flows in one direction is a system where behavior is predictable from the structure, and debugging does not require reconstructing who might have modified a value and when, because the ownership model makes that history explicit.

Williams illustrates this with an example from his book, a miniature publish-and-subscribe system built to resemble Kafka at a much smaller scale. “Because Rust has move semantics, you know that if something leaves here and goes here, it’s now not here anymore. It’s there. There’s no question about things like having references dangling or anything like that. The clarity of things moving through the system, the clarity of being able to have immutable data in a lot of places and knowing who can and can’t modify any piece of data, it just makes the design of the system so clear and it makes it so much harder to make a system that doesn’t work,” he says.

The typestate pattern

The most underutilized expression of this architectural discipline in Rust is one that Williams returns to with visible enthusiasm. The typestate pattern uses the type system to encode the state of a value at compile time in a way that makes invalid state transitions not just errors but programs that will not compile.

“It’s a way of developing state machines and systems that have state that evolves where invalid state transitions aren’t just errors, they’re impossible to write. The compiler won’t compile them,” Williams says. “It represents a huge advance in the way that such systems are written because now instead of runtime errors, you have a state machine that is guaranteed to work because every transition either is a valid transition or it won’t even compile. That’s an amazing thing.”

The typestate pattern was not invented for Rust, but the language’s ownership system and its handling of types make it practical in a way that other languages do not. The result is that a class of bugs that normally surfaces at runtime, invalid transitions through a state machine, surfaces instead at compile time, before the program runs. For systems where correctness is not optional, this is a material improvement. “Not invented for Rust, but it fits Rust so perfectly, it’s hard to believe it,” Williams says.

What this requires in practice

None of this comes without a cost. The discipline that Rust enforces at compile time is discipline that engineers have to supply at design time, and for teams moving from other languages the transition is genuinely difficult. Williams is specific about where the difficulty concentrates. Velocity drops during the learning period, often enough that teams take it as a signal that the decision was wrong, and it usually is not. “Once the team becomes very well acquainted with Rust, velocity can increase dramatically, but there is a period of time where it seems like things have gotten worse,” he says.

The answer for most teams is to start with a small, non-critical piece of work rather than a rewrite of an existing system, with the goal of building familiarity in a context where the cost of roadblocks is low and then expanding from there. “What you don’t want to do is jump into saying, we’re just going to rewrite our project in Rust now. Pick a small piece, focus on that, gain confidence and mastery of the language, and then use that to build upon it and start bringing in more things,” Williams says.

There are also cases where Rust is the wrong tool. Prototyping benefits from the flexibility that Python provides and that Rust does not. Environments where the tooling is incomplete are not the right place to fight the language and the Rust ecosystem, while growing rapidly, still has gaps where Java or C libraries are well established. User interfaces are the clearest current example. But in systems where failure is expensive, where correctness cannot be approximated, and where the code has to remain understandable as the team around it changes, Rust’s constraints are not a cost. They are the point.

The harder thing to write

The most revealing observation Williams made came not from a question about patterns or architecture but from the experience of writing the early chapters of his book, the ones about what not to do. He went back and tried to write bad Rust deliberately, the kind of code that would illustrate the mistakes he was cautioning against, and it was harder than he expected.

“When I went back and tried to write bad code in Rust, it was much harder than writing the good code,” Williams says. “That’s an interesting perspective that just didn’t even occur to me.”

That observation captures something important about what the language is doing. Rust is not just a language with a strict compiler. Its constraints push code toward a specific shape, one that is explicit about data flow, deliberate about ownership, and structured around clear boundaries of responsibility. The engineers who find Rust most difficult are often the engineers with the most experience, because they have the most deeply held instincts to unlearn. And the engineers who find it most rewarding tend to be the ones who stop treating the borrow checker as an obstacle and start reading it as design feedback. The language is not rejecting their code. It is asking them to think more clearly about what the code is actually doing.

In case you missed

Here’s the full Q&A with the interview video featuring Evan Williams.

🛠️ Tool of the Week

rust-analyzer — Rust language server that provides IDE functionality for writing Rust programs.

Highlights:

• Surfaces Rust diagnostics, including ownership and borrow-checker errors from compiler checks, inline during editing.

• Supports major LSP-compatible editors such as VS Code, Vim, Emacs, and Zed, with regular stable releases.

• Widely used across the Rust ecosystem as the standard Rust IDE backend, including in workflows at organizations that build with Rust.

Learn more about rust-analyzer

📎 Tech Briefs

• Dirty Frag vulnerabilities disclosed in the Linux kernel - Two CVEs in Linux ESP/IPsec and RxRPC components allow unprivileged local users to gain root on affected systems.

• Linux 7.0.5 stable released with partial Dirty Frag fix - Linux 7.0.5 ships a partial XFRM/ESP patch for Dirty Frag, with a second required fix still in development at release time.

• GitHub secret scanning via MCP Server now generally available - Credential scanning is now available in MCP-compatible coding agents before commits or pull requests, requiring GitHub Secret Protection to be enabled on the repository.

• Linux 7.1-rc2 published — KVM selftest renaming drove the unusual patch volume in rc2, with functional work covering driver and networking fixes throughout the tree.

• MySQL 9.7.0 LTS generally available - New MySQL LTS line ships the Hypergraph Optimizer in Community Edition, with Dynamic Data Masking remaining Enterprise-only in this release.

That’s all for today. Thank you so very much for reading this issue of Deep Engineering.

We’ll be back next week with more expert-led content.

Stay awesome,

Saqib Jan

Editor-in-Chief, Deep Engineering

If your company is interested in reaching an audience of senior developers, software engineers, and technical decision-makers, you may want to advertise with us.

Thanks for reading Packt Deep Engineering! Subscribe for free to receive new posts and help grow our work.

See what’s new for the HubSpot Developer Platform!

Ship faster with AI coding tools like Cursor, Claude Code, and Codex. Build MCP-powered AI connectors, run serverless functions with support for UI extensions, and use date-based versioning to streamline roadmap planning.

Explore Updates

✍️ From the editor’s desk,

Welcome to the 46th issue of Deep Engineering!

This week, InfoQ analyzed what it actually took for Cloudflare to run large language models efficiently on their global network. The team built a custom inference engine called Infire from scratch in Rust, split model processing into two separate hardware stages because a single machine could not handle both efficiently, and compressed model weights by 15 to 22 percent to reduce what GPUs need to load and move during inference. The reason they had to do all of this is the same one that matters to every engineering team building AI systems: the hardware layer is not an abstraction you can ignore. It is the constraint that every other architectural decision is made around.

This pattern, where standard approaches to running AI workloads break down under real production constraints and the fix requires going back to the hardware layer, is one most engineering teams will eventually encounter. The engineers who avoid it are the ones who understood the hardware constraints before they started building, not after they hit them. Cloudflare’s engineering blog post goes into the technical detail for teams who want to dig further.

This week we are featuring Jim Ledin, CEO of Ledin Engineering and author of Modern Computer Architecture and Organization, now in its third edition published by Packt. Ledin has over thirty years of experience working on embedded systems, safety-critical hardware, and cybersecurity. In this issue he breaks down what engineers building AI systems get wrong about the hardware layer and why it costs them.

Let’s get started.

Architecting Production-Ready APIs for Agents

Most API ecosystems were not built for autonomous agent usage. This hands-on masterclass covers governed API design, OpenAPI specifications, and multi-API workflow modelling with Arazzo so your platform stays predictable and safe under automated usage.

2 FOR 1 deal is also live. Bring a colleague free and learn how to design AI-ready, governed APIs

Use code DEEPENG50 for 50% off.

Register here

Expert Insights

Hardware Is Not Someone Else’s Problem

by with

For most software engineers working in the cloud, hardware is an abstraction managed by someone else. You provision compute, write code, deploy, and pay the bill. What happens between the instruction and the silicon is not your concern. That assumption has always had a cost. In an AI-accelerated world, that cost is becoming visible in ways that are harder to ignore.

Jim Ledin, CEO of Ledin Engineering, has been working at the boundary where software meets silicon for over thirty years. His entry point into computer architecture was not a formal computer science curriculum. It was a Commodore 64, a joystick, and a drawing program so slow you could watch it move one pixel at a time.

“That episode really cemented for me how important it is to understand what is going on in the hardware of a system, and not just write what you want to do in your favourite language,” Ledin reflects.

He rewrote the inner loops of that drawing program in 6502 assembly, poking opcodes directly into memory, and the line shot across the screen faster than he could see. The lesson stayed with him across thirty years of embedded systems work, electric vehicle software, and cybersecurity testing on safety-critical hardware. Understanding what the hardware is actually doing is not an optimization exercise. It is the difference between software that works and software that works reliably under real constraints.

That distinction matters more now than it ever has, because the hardware layer is where most AI system performance problems actually live, and most of the engineers building those systems have never had to care about it before.

Where the GPU consensus breaks down

The idea that GPUs are the right architecture for AI workloads has become so widely accepted that most teams treat it as settled. Ledin’s view is more specific, and the specificity matters. For local and personal use, running models on a consumer GPU like an Nvidia RTX 4090, GPUs are the right choice. For large-scale deployments running the largest models, the picture is different.

The distinction comes down to what GPUs were actually designed to do. The “G” in GPU stands for graphics, and consumer GPUs still carry silicon dedicated to real-time video generation and gaming workloads. TPUs, by contrast, are built entirely around the tensor operations that dominate AI model processing. At least 80% of the execution time in a transformer-based model is matrix multiplications, and TPUs concentrate every transistor on exactly that work.

The more pressing constraint, though, is memory bandwidth. “AI workloads are becoming increasingly memory bandwidth limited. That means it is taking more time to bring data into the GPU or TPU memory than it is taking for the computation itself to complete,” Ledin explains.

This is the reason high-end AI systems use high bandwidth memory, or HBM, stacked RAM modules with far higher data rates than anything available on a consumer GPU. “It is also,” Ledin notes, “part of why DDR5 is becoming harder to find. Production capacity for memory is increasingly going into HBM modules for AI infrastructure rather than into consumer components.”

And so, for engineering teams choosing hardware for AI deployments, the implication is concrete: the GPU consensus is correct for a specific part of the problem space, and incomplete for the rest of it.

Data movement is the real cost

The performance conversation in AI engineering tends to focus on compute: cores, clock speed, parallelization. Ledin redirects it toward something that gets less attention and causes more problems.

“Data movement can often be more expensive than the actual computation steps. The latency of moving large data structures across different levels of the memory hierarchy can dominate and leave a lot of compute bandwidth idle,” he emphasizes.

This is not a new insight in systems engineering, but it is one that most application developers have never had to internalize because the abstractions they work with hide it. In a modern PC, reading a single byte from DRAM causes 64 bytes to be transferred into the CPU cache. If the code then bounces to other memory locations, causes those to be loaded into cache, and pushes that first block out, the next access to that original data requires fetching it again from DRAM. The latency compounds across every cache miss, and in AI workloads operating on large data structures, those misses accumulate fast.

The practical recommendation follows directly. Iterating across large data structures multiple times in an algorithm should be avoided wherever possible. Working through memory linearly, in a way that keeps recently accessed data in cache rather than evicting it, is the single most impactful optimization available to most AI system code. It does not require a new framework or a different hardware platform. It requires understanding what the hardware is doing with the data you give it.

In cloud environments, this understanding has a direct financial translation. “You are paying for the usage of the system whether the CPU is actually crunching instructions or sitting idle waiting for a data item to come in from memory,” Ledin warns. This is because inefficient memory access patterns do not just slow down a system. They inflate the bill for it.

When abstraction becomes the problem

Abstractions are one of the most effective tools available to software teams. They accelerate development, limit mistakes, and allow large teams to work on complex systems without every engineer needing to understand every layer. Ledin does not dispute any of this. His concern is more specific: abstractions that obscure hardware costs, in performance-critical applications, are not just unhelpful. They actively create risk.

“Where it becomes dangerous is when abstraction obscures what is happening with the data layout in memory and the execution patterns, basically how the processor is interacting with data as the algorithm proceeds,” he cautions.

The failure mode is not that abstractions break. It is that they make costs invisible until those costs produce an incident. An engineer works within an abstraction layer, the code looks correct at that level, and the performance problem lives underneath it in a layer the abstraction was designed to hide. By the time the problem surfaces in production, the context needed to diagnose it is buried.

Ledin’s recommendation is a two-layer design. Use the most expressive code at the edges of the system, where the abstractions are doing the most valuable work. Use performance-aware code in the core, where the hardware interaction is most consequential. The boundary between those layers is not fixed, and finding it requires benchmarking rather than intuition. But knowing the boundary needs to exist is the starting point. Teams that treat the expressive outer layer as the whole system tend to discover, under load, that the core was never designed for the hardware it runs on.

The CPU versus GPU distinction, for engineers who have never had to care

Most senior software engineers working today have built careers without ever needing to think about the difference between a CPU and a GPU. That is changing, and Ledin’s framing of the distinction is the most useful one available for engineers coming to it for the first time.

A CPU is optimized for low-latency execution of complex branching code. It is built to handle conditional logic, to predict branches and recover when predictions are wrong, and to minimize the latency cost of that work. A GPU is optimized for high-throughput execution of linear code across massively parallel workloads, and it works best when it is running the same instruction across thousands of data streams simultaneously with as little branching as possible.

The implication, therefore, for algorithm design is practical. “The GPU only really becomes attractive when you have enough work for it to do that it can be parallelised, and enough that it will amortise the costs associated with moving data onto the GPU, launching the kernels to execute the code, and doing the management work to transfer data to and from the GPU,” he points out.

That last point is the one most teams miss. A GPU is not a general purpose computer. It cannot run a program on its own. It needs to be started and managed from a CPU, and the overhead of moving data onto the GPU, scheduling the kernels, and moving results back is real. If the workload is not large enough and parallel enough to amortize that overhead, the CPU implementation wins, not because GPUs are slow, but because the cost of using them correctly exceeds the benefit for that specific workload.

Knowing where that line sits, for a specific algorithm running on specific hardware, is the kind of judgment that requires understanding what the hardware is actually doing. It cannot be read off from a benchmark or inferred from a framework’s documentation. It comes from the same place Ledin’s understanding came from: going one level deeper than the abstraction, and learning what happens when the instruction meets the silicon.

In case you missed

Here’s the full Q&A with the interview video featuring Jim Ledin.

If the hardware layer argument resonates, the article below by Lee Peterson, VP of Secure WAN Product Management at Cisco, covers the same constraint from the networking and distributed compute angle.

🛠️ Tool of the Week

vLLM - high-throughput, memory-efficient inference and serving engine for large language models

Cloudflare referenced it as the baseline they benchmarked their custom Infire engine against when building hardware-optimized inference at scale.

Highlights:

• PagedAttention eliminates the memory waste that causes most GPU out-of-memory failures in production inference.

• Continuous batching processes requests in a dynamic stream rather than static batches, keeping GPUs saturated under real load.

• Disaggregated prefill/decode runs compute-bound and memory-bound stages on separate hardware for better throughput.

• Supports tensor parallelism, FP8 and NVFP4 quantization across multi-GPU deployments.

Learn more about vLLM

📎 Tech Briefs

• Inference gives AI chip startups a second chance - Disaggregated inference, splitting prefill and decode across purpose-built silicon, is making GPU-only inference architectures look like the wrong default for large-scale production deployments.

• OpenAI releases MRC for AI training networks - OpenAI’s MRC shows frontier training now depends on failure-tolerant network design, making the interconnect layer a first-class engineering constraint rather than an infrastructure afterthought.

• Anthropic opens Claude Security public beta - Claude Security moves vulnerability scanning closer to code review, triage, and patch creation, shifting security work earlier into the engineering workflow rather than treating it as a downstream audit step.

• Google opens Workspace MCP server preview - Google is turning enterprise agents into a governed API and access-control problem, with MCP making the boundary between agent capability and enterprise data policy the next infrastructure challenge for platform teams.

• vLLM v0.20.1 ships with DeepSeek V4 stabilization and FP4 improvements - The patch release stabilizes DeepSeek V4 serving and improves FP32-to-FP4 conversion speed.

That’s all for today. Thank you for reading this issue of Deep Engineering.

We’ll be back next week with more expert-led content.

Stay awesome,

Saqib Jan

Editor-in-Chief, Deep Engineering

If your company is interested in reaching an audience of senior developers, software engineers, and technical decision-makers, you may want to advertise with us.

Thanks for reading Packt Deep Engineering! Subscribe for free to receive new posts and help grow our work.

See what’s new for the HubSpot Developer Platform!

Ship faster with AI coding tools like Cursor, Claude Code, and Codex. Build MCP-powered AI connectors, run serverless functions with support for UI extensions, and use date-based versioning to streamline roadmap planning.

Explore Updates

✍️ From the editor’s desk,

Welcome to the 45th issue of Deep Engineering!

The TIOBE Index for April 2026 puts Rust at number 16, up from 18 last year. While the community widely expected it to break into the top 10, that momentum has slowed. TIOBE attributes this to adoption friction, noting that broader mainstream uptake has proven harder to achieve than the language’s early trajectory suggested.

The institutional picture, however, tells a different story. The NSA and CISA issued joint guidance in June 2025 urging organisations to adopt memory-safe languages for national security systems and critical infrastructure. Google’s Android security team reported that memory safety vulnerabilities, which accounted for 76% of Android bugs in 2019, fell below 20% for the first time in 2025 after the team prioritised writing new code in memory-safe languages.

The Linux kernel maintainers also made Rust permanent, making it a core part of the kernel. The argument about whether memory-safe systems programming languages belong in production is settled at the institutional level. The harder argument, therefore, is how engineering teams adopt it without the expensive, disruptive projects that give the language an undeserved reputation for being hard to introduce.

Francesco Ciulla, author of The Rust Programming Handbook and head of developer relations at Zerops, directly challenges that perception. His framework starts not with the language’s strengths but with the one service in your system where Rust would make an undeniable, measurable difference. That is what this issue covers.

Let’s get started.

Architecting Production-Ready APIs for Agents

Most API ecosystems were not built for autonomous agent usage. This hands-on masterclass covers governed API design, OpenAPI specifications, and multi-API workflow modelling with Arazzo so your platform stays predictable and safe under automated usage.

2 FOR 1 deal is also live. Bring a colleague free and learn how to design AI-ready, governed APIs

Use code DEEPENG50 for 50% off.

Register here

Expert Insights

Rust Does Not Need to Replace Your Stack to Make It Better

by with

Every engineering team that considers Rust tends to circle the same concerns. The language is difficult to learn, expensive to adopt, and more practically useful to systems programmers than to the teams building and running services at scale.

Francesco Ciulla, author of The Rust Programming Handbook, Docker Captain and head of developer relations at Zerops, says, "I've heard that conversation many times, and my response has always been that the framing is wrong from the start."

Ciulla has been building with Rust since 2022, has spoken about it internationally at conferences, and his perspective on Rust adoption is shaped less by enthusiasm for the language and more by a practitioner’s view of where it actually earns its place in a production system. That starting point matters, he says, because the teams that struggle with Rust adoption tend to start from the wrong question.

A joke that contains a kernel of truth

People in the Rust community have long joked about rewriting everything in Rust, and the memes around it have become something of a cultural shorthand for over-enthusiastic adoption. Like most good jokes, Ciulla acknowledges, it contains a kernel of truth. But the practical lesson is the opposite of what it implies. “The best way to introduce Rust in a big project is to find that hard part that is slowing things down, the bottleneck of all your services, and try to write one single service in Rust.” The rewrite everything instinct is how adoption projects become expensive and difficult to justify. The bottleneck-first instinct is how teams get a proof of concept that demonstrates real value before committing to anything broader.

The practical implication is that the adoption decision is not a language decision at the organizational level. It is an engineering decision at the service level. The question is not whether the organization should adopt Rust. The question is whether there is one service in the system that is slow, resource-intensive, or difficult to keep stable, where the properties Rust offers would make a measurable difference. If that service exists, it is the right place to start. If it does not, the case for introducing Rust at all is weaker than it might appear.

Ciulla’s production experience makes this concrete. Running a Rust web server on his own machine, he shared during our conversation that it used four megabytes of RAM in development and five in production. On a one-gigabyte droplet, that means more than 200 services running simultaneously in idle. That number is the kind of resource profile that changes what is economically viable to deploy, and it is the kind of argument that lands differently with an ops team than a language comparison ever could.

Flat latency is a real engineering argument

One of the most underappreciated technical arguments for Rust in production systems is not about speed in the raw throughput sense. It is about predictability. Languages that rely on garbage collection, including Go, Java, and Node.js, introduce periodic pauses when the collector runs. Those pauses can last hundreds of milliseconds. An HTTP request that arrives during a GC cycle experiences higher latency than one that does not. The user on the receiving end did not do anything differently. They were just unlucky.

Ciulla is candid about what this means in practice. “By not having a garbage collector on the back end side, you basically have flat latency. You don’t rely on luck, or on the user not being the unlucky one. It’s a problem that is removed.” For most web applications running at moderate scale, this distinction is invisible. For services with strict latency requirements, high concurrency, or SLAs that depend on consistent tail latency rather than average response time, it is one of the more significant architectural arguments available.

This connects to a broader point about where Rust earns its place and where it does not. The resource efficiency and latency predictability are not arguments for using Rust everywhere. They are arguments for using Rust in the specific services where those properties matter. A service that scrapes a website once a month does not need flat latency. A service handling a million concurrent users does. Knowing the difference is what separates a good adoption strategy from an expensive experiment.

Rust, when it is the wrong choice

Ciulla is honest about the cases where Rust is not the right tool, which is part of what makes his advocacy for it credible. If a team needs something simple and the deadline is tomorrow, Rust is probably the wrong choice. If a developer needs a working API by the end of the day and has no Rust experience, this is not the moment to start learning the language under delivery pressure. “When you need something simple, and you’re familiar already with Java or JavaScript, why don’t you use it?” The question is not rhetorical. It is the right question to ask before any technology adoption decision.

The ecosystem argument is also honest. Python has better libraries for data science. JavaScript has a larger package ecosystem for certain kinds of web work. Rust integrates well with other languages, but if what a team needs is native to another ecosystem, that is a real constraint rather than a preference. Good engineers use the right tool for the problem. The case for Rust is strongest when the problem involves performance, memory efficiency, or concurrency at a level where other languages start showing their limits.

The shepherd principle

One of the more practical observations Ciulla makes about organizational adoption is about knowledge rather than tooling. The bottleneck to Rust adoption at scale is rarely the language itself. It is whether the organization has someone who knows it deeply enough to validate the work being done in it. He draws the parallel to Docker adoption at the European Space Agency, where he worked and observed the tool move slowly not because of anything wrong with Docker, but because it was not well understood internally. The technology is never the problem, he points out. The knowledge is.

“You need the validation of an expert,” Ciulla says. In the era of AI-accelerated development, this point is sharper than it has ever been. Teams can now generate Rust code with AI assistance far faster than their ability to validate it has grown. That gap between generation speed and validation depth is where production incidents come from. Having at least one engineer on the team who understands the ownership model, the borrow checker, and the concurrency primitives well enough to review what the AI produces is not a nice-to-have. It is the thing that determines whether the Rust service is a genuine improvement or a liability waiting to surface.

Concurrency without the trauma

Most engineers who have worked with concurrency in Java or C++ carry a specific kind of wariness about it. The mental model for concurrency in older languages is that it is an advanced topic requiring extra care, specialized libraries, and a heightened awareness of race conditions and deadlocks. Ciulla describes learning concurrency in Java at university as the final, difficult session of the course, something treated as inherently dangerous and saved for the end of the degree.

His first attempt at a concurrency example in Rust produced the opposite experience. “When I had to teach concurrency in Rust in a YouTube video, I made an example in three minutes. I was done. I say okay, the basic example really lasted like two or three minutes because you just declare a couple of threads and literally done.” That experience reflects something structural about how Rust was designed. The language was created after multi-core processors were already standard. Concurrency was not retrofitted onto a model designed for single-threaded execution. It was built in from the start, and the ownership system that prevents data races at compile time is the same ownership system that governs memory safety everywhere else in the language. There is no separate concurrency model to learn. The properties that make Rust memory safe are the same properties that make concurrent code safe.

For teams building services that need to use available CPU resources efficiently, this is not a minor ergonomic improvement. It means that the gap between writing concurrent code and writing correct concurrent code is substantially smaller in Rust than in the languages most engineers have used before. The cost of concurrency, measured in debugging time and production incidents rather than lines of code, is genuinely lower.

The compiler is the most patient teacher on your team

The reputation Rust has for being difficult to learn is real, and Ciulla does not dismiss it. But his explanation for where the difficulty actually comes from is different from the common framing. The problem is not that the concepts are inherently harder than those in other languages. It is that Rust forces you to unlearn patterns that other languages allowed. Engineers who have spent years in C++ or JavaScript carry assumptions about how memory works, how mutability is managed, and what the runtime will silently fix for them. Rust does not fix those things silently. It surfaces them at compile time and requires you to address them explicitly before the code runs.

That shift in where the pain lands is the key insight. “Rust is not hard to learn,” Ciulla says. “It’s different. And this is how we should advocate for it.” The difficulty is front-loaded by design, because the language makes a deliberate trade of more friction during development in exchange for fewer failures in production. Teams that have spent significant time debugging null pointer exceptions, race conditions, or memory leaks in production understand this trade intuitively. The hours lost to a null pointer exception in production dwarf the hours spent fighting the borrow checker upfront.

The Rust compiler, which is the primary source of that upfront friction, is also the primary teaching tool the language provides. Error messages in Rust are unusually detailed and specific. They do not just tell you that something is wrong. They explain what rule was violated, show the relevant code, and often suggest a fix. Ciulla describes the compiler as a teacher rather than a gatekeeper, one that overcommunicates in the same way a good mentor does. “The errors in Rust are basically tutorials. They are helping you to write better code.” For teams introducing Rust to engineers who have not used it before, this property matters practically. The compiler is doing a significant part of the knowledge transfer work that would otherwise fall on the senior Rust engineer on the team.

Rust in the Linux kernel is paying off

The decision by the Linux kernel maintainers to not only allow Rust in the kernel but to plan for components that require it going forward is the kind of institutional endorsement the language community has been waiting for. Ciulla frames it clearly. Even if the experiment had failed, just the fact that Rust was considered a viable option for kernel-level work would have been a meaningful milestone. The language was competing in a domain that had been exclusively C and C++ territory for decades, and it earned a permanent place there.

For engineering leaders tracking where the industry is moving, this matters beyond the kernel itself. Government systems, military applications, and other security-critical domains are beginning to treat Rust as a default rather than an experiment. “Rust is slowly getting adopted at bigger and bigger levels,” Ciulla says. The adoption curve is not linear, but the direction is consistent. Teams that build internal expertise now are not chasing a trend. They are positioning ahead of a transition that is already underway in the most demanding environments in the industry.

The ecosystem argument is changing

One of the historically valid objections to Rust for web development was tooling maturity. Two years ago, Ciulla would not have committed to shipping a production SaaS product in Rust. Today he would, and the reason is specific rather than general. The Axum framework has matured to the point where it is a production-grade choice for web APIs, and the broader ecosystem around async Rust has improved substantially. “In 2026, I will use it,” he says of building a paid product with Rust as the backend, dropping the qualification he would have applied even a year earlier.

The toolchain story is also one of Rust’s genuine advantages for teams evaluating the full cost of adoption. Cargo handles dependency management, building, testing, and documentation in a single integrated tool. There is no equivalent of the npm versus yarn versus pnpm decision that teams arriving to JavaScript have to navigate before writing a single line of code. Running tests is cargo test. The integration is native to the language, not an ecosystem of competing choices layered on top of it. For teams that have spent time debugging JavaScript build configurations, this is not a small thing.

Rust in the AI accelerated development era

Ciulla makes an argument about Rust and AI that is worth sitting with. The claim is not that Rust is better for writing AI applications, though he has views on that too. The claim is that Rust may be one of the best languages to work in during the current period of AI-assisted development, specifically because of what it requires from the engineer reviewing AI-generated code.

When AI writes Rust code, the engineer validating it still has to understand ownership, borrowing, and the type system well enough to know whether the generated code is correct. The compiler will catch a large category of errors, but the human reviewer still needs to understand why the compiler is happy with a piece of code before shipping it. “If you have no control, either you are useless or you cause a problem. So in both cases, it’s not a good time for you.” That discipline, the requirement to understand what the code actually does rather than just accepting output that compiles, is not a burden unique to Rust. But it is more explicitly enforced by the language than in most alternatives, and that enforcement is valuable at a moment when the volume of AI-generated code is increasing faster than the average team’s ability to review it carefully.

🛠️ Tool of the Week

wrkflw — open-source tool for validating and running GitHub Actions locally

wrkflw lets you validate and execute GitHub Actions workflows on your local machine before pushing, catching configuration errors and pipeline failures before they reach CI. Version 0.8.0 shipped this week. Built in Rust.

• Validates GitHub Actions workflow syntax locally before pushing to CI

• Runs multi-step jobs and matrix builds without cloud dependency

• Fast startup and low resource overhead from Rust’s binary compilation model

• Catches pipeline failures early, reducing the feedback loop between code and CI

Learn more about wrkflw

📎 Tech Briefs

• Linux 7.0 ships with Rust as an official core kernel language - Rust loses its experimental tag in Linux 7.0, reaching full parity with C for kernel development after the Linux Kernel Maintainers Summit decision in December 2025.

• Rust 1.95.0 released - The latest stable release introduces cfg_select!, a compile-time macro for conditional configuration, and removes unstable support for custom target specifications on stable toolchains.

• crates.io opens new Svelte-based frontend for public testing - The Rust team invites the community to test the rebuilt crates.io frontend ahead of the planned production migration.

• Cargo stabilises build.warnings configuration - The build.warnings field is now stable, giving teams a standardised way to configure compiler warning behaviour across workspace builds.

• Zed 1.0 ships - The Rust-built code editor reaches stable release with GPU-accelerated rendering, real-time collaborative editing, Git integration, and native AI assistant support across macOS, Windows, and Linux.

That’s all for today. Thank you for reading this issue of Deep Engineering.

We’ll be back next week with more expert-led content.

Stay awesome,

Saqib Jan

Editor-in-Chief, Deep Engineering

If your company is interested in reaching an audience of senior developers, software engineers, and technical decision-makers, you may want to advertise with us.

Thanks for reading Packt Deep Engineering! Subscribe for free to receive new posts and help grow our work.

Hone your skills to design, implement, and ship an AI-powered internal developer platform from scratch with Ajay Chankramath.

📅 25th and 26th April 2026 | 11:00 AM to 3:00 PM ET

Use code DEEPENG50 for 50% off, or grab the 2-for-1 deal

Register now

Includes access to The Platform Engineer’s Handbook by Ajay Chankramath upon release (eBook worth $35.99)

✍️ From the editor’s desk,

Welcome to the 44th issue of Deep Engineering!

C++ just had its most consequential standard in years finalized. Contracts and static reflection are expected after a decade of committee work, and the async execution model is in. This has the conference circuit energised, blog posts are multiplying, and most production teams are continuing to ship C++17 or C++20 while they wait for compiler support to catch up and patterns to settle.

The ISO C++ Foundation also opened its 2026 annual developer survey this week, the single biggest opportunity the global C++ community gets each year to tell the standards committee and tooling vendors what actually matters in practice. If you write C++ at any scale it is worth ten minutes of your time before it closes next week.

This week we are featuring Sándor Dargó, senior software engineer at Spotify, on what responsible C++26 adoption decisions should look like in production, what most teams get wrong, why fallback plans need to be on every adoption checklist, and what the gap between a finalized standard and a production system actually costs. The feature is based on our live interview with him and you can watch or read the full Q&A here.

Dargó also spoke at length in that session about clean code, cognitive load, and what it actually costs teams to optimize for the wrong things. We have collated those insights alongside our previous conversation with Sam Morley, mathematician and C++ researcher at the University of Oxford, into this separate piece “Clean Code Is a Trap, Decompose Instead for Physics and Performance” published this week.

Let’s get started.

View the latest HubSpot Developer Platform updates in Spring Spotlight

See what’s new for the HubSpot Developer Platform!

Ship faster with AI coding tools like Cursor, Claude Code, and Codex. Build MCP-powered AI connectors, run serverless functions with support for UI extensions, and use date-based versioning to streamline roadmap planning.

Check → HubSpot Developer Platform

Most C++ Teams Get Feature Adoption Wrong, and What the Pragmatic Ones Do Differently

by with Sándor Dargó

C++26 is heading to its final approval ballot. Contracts and static reflection are expected. The feature set is the largest the language has seen in years, and the C++ community is already deep into discussing what it means. And yet, while they wait for compiler support to catch up and patterns to settle, engineering teams are starting to ask how to prepare for when they can actually ship it.

That gap between what the standard says and what teams can actually ship is not a failure of ambition, but the normal condition of C++ in production. It has looked roughly the same across every major standard transition for the past fifteen years. “The engineering teams that navigate it well have learned to keep understanding and adoption on separate timelines,” says Sándor Dargó, senior software engineer at Spotify. “It serves well to invest deeply in knowing a feature years before you need to ship it, because conflating the two is where most adoption mistakes gain foothold.”

Dargó has a different set of questions he likes to ask first. His framework is intentional and straightforward. The feature set is not the starting point. What matters is whether the team can live with the code a year from now, when the engineer who introduced it has moved on and compiler support is still uneven across platforms. Proven and maintainable is not playing it safe. It is what keeps production systems running while the language moves underneath them.

At Spotify, where he works on large-scale C++ systems with a hard requirement that the code stays clean, maintainable, and operable over time. He has observed teams navigate standard transitions from C++11 through C++23, and the pattern he sees repeat is always the same one. “Teams focus on what the standard says, and they underestimate the distance between the standard and the compiler, between the compiler and the toolchain, and between the toolchain and the production system that has to run reliably when all of this settles,” he said. “Everyone is talking about contracts and reflection. That’s going to change everything. I’m not sure about the time scale though. If you look at C++23 support right now, even that is not complete yet, especially if you look at the differences across compilers. You go on cppreference, check what’s implemented on which compiler, and we are simply not there yet.”

That observation shapes everything that follows in Dargó’s approach to feature adoption, and it is worth understanding precisely what he means by it.

The gap between the standard and the production system

The C++ standard finalizes on a schedule. Compiler vendors implement on a different one, and they do not always agree on approach, especially for genuinely new features that require significant infrastructure rather than incremental additions. Small library extensions arrive with almost no implementation lag because compilers already have the machinery to support them. But contracts and reflection are not small library extensions. Both require compilers to build something they have never built before, and different vendors will make different choices along the way, as they did with modules in C++20.

Dargó cited modules in our live interview as the clearest recent example of how this plays out in practice. Vendors diverged, build system integration took years to stabilize, and teams that moved early discovered that the standard’s guarantees did not insulate them from the fragmentation that came from uneven implementation across compilers. “There might be a bigger gap, just like we saw with C++20 modules. I hope it won’t be that problematic. But I don’t think I’ll be able to use those in a production environment in the next one or two years.” Dargó, drawing on his years of engineering experience leading high-velocity teams at Spotify, a company with engineering resources that most teams do not have. If Dargó is not planning to ship contracts or reflection to production in the next two years, the average multi-platform C++ team operating with a mixed compiler pipeline and a long-lived codebase should be thinking in similar terms.

This does not mean the features are not worth understanding. It means the time horizon for understanding and the time horizon for shipping are different, and most adoption conversations conflate the two.

Cautious is not the same as resistant

Dargó’s posture toward new features is not skepticism. It is a more specific kind of discipline that he applies before any adoption decision, and it comes down to two distinct questions that most teams skip. The first is whether the usage pattern is proven in a production environment with real operational constraints, not in a blog post or a conference demo. The second is whether the feature actually solves a problem in the specific codebase, rather than just offering a more modern way to write something that was already working. “If they only bring a different syntax, that’s not really good. If they actually solve a problem in your codebase, in your use case, then it’s good. And let’s discuss why and how that’s the right solution.”

The distinction between a genuinely useful adoption and a syntactic upgrade matters more than most teams want to admit, because every feature adoption comes with a cost that does not appear in the initial pull request. The team has to understand the feature well enough to review code that uses it, to debug it when it breaks, and to maintain it when the engineer who introduced it has moved to a different team. “You cannot just push the new way of coding. You should share the knowledge and discuss what these new features actually bring. Deep down, everyone wants to learn about these things, and we all want to start using them as soon as possible.” Dargó acknowledges that impulse is not wrong. But the enthusiasm of the early adopter on the team and the readiness of the rest of the team to live with the decision are two different things, and adoption decisions that ignore the second tend to create the kind of complexity that shows up as maintenance debt two years later.

No one puts the fallback plan on the checklist

The most operationally concrete principle Dargó applies to feature adoption is also the one that almost no team’s adoption checklist includes. Before shipping code that uses a new compiler feature, make sure you can roll back the compiler version without having to change the code. This constraint is not about being conservative for the sake of it. It is about what happens when a compiler update introduces a regression or a new feature behaves differently than expected on a specific platform, and the fastest path to stability is to revert the compiler version rather than to fix the code.

Dargó walked through the alternative. “Move to a new version, start using concepts from C++20, and then in two weeks they say there’s a problem, we must go back. And then you realize it’s not just updating the compiler version, you actually have to change the code.” At that point the team is no longer choosing between a rollback and a fix. They are choosing between a broken build and a risky code change under pressure, and neither is a good position to be making decisions from. The fallback plan requirement changes how the team thinks about adoption from the start because it sets a concrete constraint on which features can be used and how deeply they can be integrated before the team has sufficient confidence in the compiler support. For single-platform, single-compiler environments, that constraint is loose. For multi-platform pipelines with mixed compiler versions, it is the gate that determines what gets shipped and what gets saved for a later standard cycle.

Making trade-offs explicit before someone else makes them implicit

The broader philosophy Dargó brings to C++ at scale is that complexity is always the enemy, and the job of a senior engineer is to reduce it through the daily decisions that accumulate into the character of a codebase over time. Clean code reduces cognitive load. Smaller binaries reduce operational cost. Eliminating undefined behavior reduces hidden risk. New language standards reduce boilerplate and enable safer abstractions. And all of these things connect, in Dargó’s framing, to the same underlying goal: making large C++ systems more maintainable and more evolvable over the years that teams and requirements will inevitably change.

But maintainability does not happen by default. It requires making trade-offs explicit at the moment they are made, in the code itself, not just in the code review thread that will be forgotten. Dargó has been on the receiving end of what happens when this does not happen. He came into a codebase, saw code that looked wrong, began cleaning it up, and realized too late that the seemingly redundant choice was affecting binary size in a way that mattered operationally. Some pull requests had already merged before the context became clear. “Trade-offs there will be. But make them conscious and share the knowledge.” That principle applies to binary size decisions and it applies equally to feature adoption decisions. The team that adopts contracts without documenting why, which problem it was solving, what compiler version it was verified on, and what the fallback plan is, has introduced complexity that is invisible until the moment it becomes a production incident.

What C++26 actually changes and when it matters

Contracts and reflection are genuinely significant features. Contracts bring the language its first formal precondition and postcondition system, giving teams a way to enforce interface discipline through the language rather than through documentation and convention. Static reflection enables code generation, serialization frameworks, and tooling that the language simply could not support before in any ergonomic way. Both of these things will change how C++ gets written over the next decade, and engineers who understand them deeply before the production window opens will be in a stronger position than those who try to learn them under delivery pressure.

But a decade is closer to the right time scale for thinking about the full impact than a release cycle is. The standard has finalized, but the compilers have not caught up. And the community has not yet established the patterns worth following or the misuse patterns worth avoiding. For engineering leaders building roadmaps, Dargó’s framework points toward a clear separation between understanding and shipping. Invest in understanding contracts and reflection now, run experiments on non-production code, track compiler support as it lands, and build team knowledge before the toolchain is ready. Then, when the support is solid, the patterns are clear, and the fallback plan can be written, adoption becomes a technical decision rather than a leap of faith.

Dargó likes to put it in simple words. “Being among the first adopters is sometimes good. Sometimes it’s better to be in the second line.” For most production C++ systems with multi-platform requirements, long-lived codebases, and teams where knowledge has to be shared rather than siloed, the second line is where proven and maintainable gets built. The first line is where the lessons come from that make the second line possible.

🔍 In case you missed it…

🛠️ Tool of the Week

vcpkg — open-source C and C++ package manager

vcpkg removes the dependency management friction that makes compiler and toolchain transitions harder than they need to be, letting teams verify library support across targets without manually managing platform-specific build configurations.

• Parallel file installation for faster builds

• 2,773 ports in the curated registry, each validated across all major triplets

• Dependabot support for automated dependency vulnerability tracking

• OpenSSL packaging security fix on Windows

Learn more about vcpkg

📎 Tech Briefs

• Spotify publishes how it used Honk, Backstage, and Fleet Management to migrate thousands of datasets - Details how background coding agents reduced the manual overhead of large cross-repository migrations at scale.

• GitHub ships C++ Language Server for Copilot CLI in public preview - Extends semantic C++ code intelligence to the command line, giving Copilot symbol definitions, call hierarchies, and type data beyond what grep returns.

• Clang/LLVM 22.1.4 released - Patch release of Clang 22, the compiler with the most active C++26 feature tracking. Teams on Clang should update.

• MSVC Build Tools 14.51 release candidate ships in Visual Studio 2026 Insiders - Adds expanded C++23 conformance, CWG and LWG issue resolutions, and runtime performance improvements. Stable release expected in May.

• cppreference.com returning to read-write after ISO C++ Foundation takes over infrastructure - Herb Sutter announced the Foundation is taking over hosting and maintenance for cppreference, the primary compiler support reference for the C++ community.

That’s all for today. Thank you for reading this issue of Deep Engineering.

We’ll be back next week with more expert-led content.

Stay awesome,

Saqib Jan

Editor-in-Chief, Deep Engineering

If your company is interested in reaching an audience of senior developers, software engineers, and technical decision-makers, you may want to advertise with us.

Thanks for reading Packt Deep Engineering! Subscribe for free to receive new posts and help grow our work.

Learn ownership, RAII, smart pointers & allocators to eliminate leaks/UB—live hands-on with Patrice Roy—#1 bestselling author of C++ Memory Management and ISO C++ Standards Committee (WG21) member.

📅 Online: Apr 18 & Apr 19. Limited seats.

Register now

✍️ From the editor’s desk,

Welcome to the 43rd issue of Deep Engineering!

On April 15, Neo4j ran NODES AI 2026, a full-day virtual conference dedicated to AI with knowledge graphs, with tracks covering GraphRAG, graph-based agent memory, and context engineering. Interestingly, the opening keynote framed the central challenge directly around how most AI systems behave as if they have never seen the data that the organisations deploying them have spent years accumulating. The gap between what a system has access to and what it can actually use at query time is, in the view of the practitioners at NODES, the defining engineering problem of this moment.

And so, in today’s issue we want to address exactly that gap. David Knickerbocker, founder of Verdant Intelligence and author of Network Science with Python (Packt), has spent years building knowledge graph systems that stay current with a different design constraint than most teams start from: not how to make retrieval faster, but how to make the system aware of what changed a minute ago. His system (Verdant Eye) treats knowledge as a continuous stream of claims rather than a static store of facts, and that distinction has significant consequences for how you handle freshness, hallucination, temporal drift, and testing.

This is the first of two features from our conversation with David, the second will cover what building AI on top of messy, adversarial, real-world data teaches you that clean-dataset engineers never have to confront. You can read the full interview and watch the conversation here.

Let’s get started.

30k+ DevOps Engineers Read EverythingDevOps

The cloud-native ecosystem moves on its own schedule. New tooling, shifting practices, team structures that look nothing like they did two years ago. Therefore, keeping up is not optional when your job is to build and run reliable systems.

EverythingDevOps‘ weekly digest is built specifically for engineers who want the signal without the scroll.

• What’s changing in the ecosystem — and what it actually means for your work

• Career thinking for engineers moving from senior to staff to leadership

• Events, opportunities, and community conversations worth your Frida

Join 30,000+ engineers

AI That Sees the World as It Is, Not as It Was

by with David Knickerbocker

Most AI systems are answering questions about a world that no longer quite exists. They draw from training data frozen at a point in time, from retrieval systems that return whatever was most recently indexed, from pipelines that treat knowledge as a stable object rather than a moving one. For David Knickerbocker, founder of Verdant Intelligence, this is not a limitation to work around, but a design error to avoid from the start.

The problem starts at problem definition

Building a knowledge graph that stays fresh does not require a different database. It requires a different question. For Knickerbocker the starting point is always the same, “what is the system actually trying to know, and at what resolution of time?”

He explains, “If you want to build a world AI and be able to answer questions about things that happened a minute ago, then that is your problem statement. And then you think about how to get that data into the database so that it is there and it is fresh. But then you also have to get AI to be able to use that data. There are two sides to this coin.”

The engineering philosophy he applies is KISS (Keep it Super Simple), and YAGNI (You Aren't Gonna Need It"). Both push in the same direction, he reasons “build the minimum thing that works, validate that it works, and expand from there.” His system (Verdant Eye) maintains data fresh up to a minute old, not by doing something architecturally exotic, but by treating freshness as a first-class constraint from day one rather than an optimization to add later.

“The AI industry feels very shiny and very new, but there is a lot of old school discipline that is still extremely useful. You start with the idea, you go through the ideation, from ideation you create your spec, from the spec you do your project management, you assign tasks and do the work. It feels like vanilla old school engineering to me.”

Awareness before truth

The instinct when building a real-time knowledge system is to treat freshness and accuracy as a tension to manage: newer data is less verified, older data is more reliable, and the system needs some formula for weighing one against the other. But Knickerbocker rejects this frame entirely.

He explains from his engineering experience that why his systems are not trying to determine what is true. But instead trying to capture what is being claimed, and those are fundamentally different engineering problems.

“In the world of open source intelligence, it has less to do with right and wrong. It has less to do with facts. What I am looking for is really claims of what is going on in the world. I do not make that decision, and I do not allow my AI to make the decision about what is true or false either.”

In adversarial open source intelligence environments, two sources in direct conflict with each other are not a problem to resolve. But they are both signal. And what matters is that both claims are captured, clustered, and surfaced. Knickerbocker describes this structure using a frame he developed from years of NLP work: not clusters in the geometric sense, but ribbons, layered bands of related information that emerge from the data.

“You have a whole bunch of information and this top ribbon might be this bad thing happened. The next ribbon might be this event is happening at the library. The next ribbon might be a punk rock show is happening at this nightclub.”

The consequence of this design is that awareness comes before truth, and that is by intention. For the applications the Verdant Eye is built for, knowing what is being said and where is actionable before you know whether any of it is correct. Adjudicating truth is a downstream problem, one that can be handled by human judgment or purpose-built downstream systems, not by the ingestion layer.

Snapshot versus movie

What separates a real-time knowledge graph from an agent that runs searches is not a matter of degree. It is a structural difference in what kind of object the system is.

An agent with search tools takes a snapshot. It queries for something, gets back the most relevant indexed results, and presents them. The system only knows what it was asked to look for, and it only looks when prompted. A knowledge graph that continuously ingests from the open web is more like a movie: it is always recording, and when you query it, you are not taking a new photograph but pulling a frame from a film that has been running the whole time.

“If you use a tool to do a search to find out something, you are getting a snapshot of time. My systems capture the heartbeat of the Internet themselves, and they are always listening. It would be closer to a movie compared to a photograph.”

The practical difference shows up clearly in demos with clients who need urgent situational information. Running a calibration query against the Verdant Eye, asking what is the latest information, returns results anywhere from a few seconds old to a minute and a half old. That behaviour is not possible with a search-tool approach at any level of optimization.

An empty result matters too. If nothing has been reported in the last minute on a given topic, the system returns nothing. That discipline is load-bearing.

“An empty dataset,” Knickerbocker underscores, “is better than a hallucination.”

The rush to use a tool before understanding it

Knickerbocker spent years watching teams fail with graph databases in 2020 and 2021, and he sees the same pattern repeating now with agents and AI. Teams adopt the tool before building the understanding, and then cannot tell whether it is working.

“There is a rush to use agents before even understanding AI. And if the understanding is not there, then it is just wishful. If you do not know how it works, you cannot tell the difference between it ran and it ran correctly. Those are very different things.”

His own relationship with graphs predates the graph database trend by years. At Intel, from 2015 to 2019, he was using graph theory for data flow mapping, tracing inputs to outputs across thousands of scripts and hundreds of servers, using centralities to find important nodes and shortest paths to understand flow. He was never using graph databases. He was using the underlying science.

“I was never invited to the cool kid graph database parties. I was always just doing stuff with graphs and using it to map out data flows and fix production outages. Dead serious stuff.”

That gap between tool and understanding is what drove him to write Network Science with Python. Teams were populating graph databases and then stopping, treating the populated database as the deliverable. The value, in his view, starts the moment the graph is populated and you begin running network science on top of it.

“If you do not know about the basics of network science, then what are you gonna do with the graph database? You have put your data in the graph database. Whereas if you come to graph databases and you have learned about centralities, community detection, shortest paths, simulations, then it can really have an impact on your network thinking.”

Similarity is not the same as identical

The central limitation of vector-based retrieval is one that Knickerbocker has framed the same way since at least 2017. Semantic similarity is a probabilistic measure, not a logical one. Two pieces of text can be highly similar in embedding space while referring to entirely different things. A graph replaces probabilistic similarity with structural traversal.

“Similarity in language is not equal to same. I will say that one more time. Similarity is not equal to same. Similar sounding things can be very, very different from each other. A graph anchors things into a piece of context.”

In a GraphRAG system, the query does not ask what is most like this input. It traverses the graph: if you are asking about jazz events in Portland, Oregon, you are connected to the Oregon node, to the Portland node, to the jazz node. The hallucination space collapses because the answer space is bounded by structure rather than probability.

“In a GraphRAG system, if there is no match then the output is that there is no match. There is no hallucination opportunity. With a similarity-based system, there could be similarity even if it is only a single word in a paragraph.”

This intuition was clear to him during his data operations years at Intel. Log files across hundreds of servers written by dozens of engineers contain enormous amounts of natural language. Working with that language to understand what a production system was doing made it obvious that the structure, the graph, and the meaning, the language, were inseparable.

“Graphs show you where things go,” Knickerbocker adds. “But all of the context about what that node even is is often carried by language itself.”

Deliberate forgetting

Temporal drift is usually framed as a correctness problem: facts become outdated, relationships change, the graph silently goes wrong. The standard response is validation pipelines, contradiction detection, freshness scores. Knickerbocker builds around a different principle.

Sharing from his experience building the Verdant Eye, Knickerbocker explains that the Verdant Eye system is not called the Verdant Brain because the metaphor is not storage. It is perception.

“The Verdant Eye sees, and it does not contain eternal memory, because that is not what an eye does. An eye sees. When the scene changes, the scene changes. Your eyes do not need to be recalibrated. The thing has just changed.”

The design principle is biological: living systems do not maintain eternal memory, and a knowledge system that tracks a living world should not either. The practical implementation mirrors transactional database patterns he worked with throughout his career in data operations. Data that is no longer operationally relevant gets archived. The live layer runs on what is current, and the cost of maintaining it stays proportional to what the system actually needs to know.

“In a transactional database, you operate off of what you need, and data that is not needed eventually gets archived. I want to build AI that does not boil the ocean, that can be bootstrapped by individuals. Infinite memory is not just philosophically wrong for this kind of system. It is economically unworkable.”

The pass butter philosophy

Knickerbocker shares how he currently runs three production GraphRAG systems, each built for a distinct purpose with its own testing criteria. Verdant Intelligence operates at high altitude, tracking events across states and regions. Grooveseeker (which he created to put his AI system to a different and specific kind of use) operates at street level, finding events in specific cities on specific nights. A third system contains thirty years of AI research, useful for tracing the intellectual lineage of ideas when building new things.

The principle connecting all three comes from a scene in Rick and Morty. Rick builds a robot. The robot asks what its purpose is. Rick says: you pass butter.

“There is no testing framework anybody else can give me that is going to be fit for purpose for what I am trying to build, because I am not trying to build general intelligence. I am trying to build intelligence that serves a specific purpose.”

Each system is therefore tested against the specific failures that would make it useless for its purpose. For Grooveseeker, that means verifying the right city, not a city of the same name in a different state; the right date, not a historical recurrence of the same event; and a URL that actually leads somewhere you can buy a ticket.

The final test of the Grooveseeker system was to stop writing articles proving it worked and simply use it. The system returned events in Portland between March 10 and March 13. Knickerbocker even attended one of them, bought a ticket at the door, saw the band, and spoke to one of the musicians.

His AI did not send him to a venue that did not exist. “That is how I know it works,” Knickerbocker affirms.

🔍 In case you missed it…

🛠️ Tool of the Week

Graphiti — open-source temporal knowledge graph engine for AI agents

Highlights:

• Temporal fact management: Facts carry explicit validity windows — invalidated when superseded, never deleted, queryable at any point in time.

• Incremental ingestion: New data integrates immediately without batch recomputation, keeping the graph current as the world changes.

• Hybrid retrieval: Combines semantic search, BM25, and graph traversal in a single query, typically under 100ms.

• MCP-native: Ships an MCP server for direct integration with Claude, Cursor, and other MCP clients.

Learn more about Graphiti

📎 Tech Briefs

• Neo4j NODES AI 2026 closing keynote: From Data to Knowledge to Action - Sudhir Hasbe makes the case that most enterprise AI behaves as if it has never seen the organisation's own data, and how graphs help close that gap.

• Graph memory for LLM agents: the relational blind spots that flat vectors miss - Zep's LongMemEval evaluation shows 18.5% accuracy gains over vector baselines, context tokens dropping from 115,000 to 1,600, and latency falling from 29 seconds to under 3 seconds.

• FalkorDB Browser ships April 2026 update for GraphRAG developers - Adds favourite query saving, one-click connection string copying, enhanced graph statistics, and configurable table views..

• Neo4j Aura Agent Hackathon opens - Developers building knowledge-graph-grounded AI agents can register for cloud credits and managed GraphRAG platform access.

• Four GraphRAG papers accepted at ACL 2026 - PolyU’s Awesome-GraphRAG repo confirmed acceptances including ProbeRAG for retrieval faithfulness, LegalGraphRAG for legal reasoning, and LinearRAG for efficient graph construction.

That’s all for today. Thank you for reading this issue of Deep Engineering.

We’ll be back next week with more expert-led content.

Stay awesome,

Saqib Jan

Editor-in-Chief, Deep Engineering

If your company is interested in reaching an audience of senior developers, software engineers, and technical decision-makers, you may want to advertise with us.

Thanks for reading Packt Deep Engineering! Subscribe for free to receive new posts and support my work.

Hone your skills to design, implement, and ship an AI-powered internal developer platform from scratch with Ajay Chankramath.

📅 25th and 26th April 2026 | 11:00 AM to 3:00 PM ET

Use code DEEPENG50 for 50% off, or grab the 2-for-1 deal

Register now

Includes access to The Platform Engineer’s Handbook by Ajay Chankramath upon release (eBook worth $35.99)

✍️ From the editor’s desk,

Welcome to the 42nd issue of Deep Engineering!

Multi-agent AI is widely being adopted in production, but the engineering reality is messier than the demos suggest. GitHub Engineering published a post by Gwen Davis laying out the most common ways multi-agent workflows fail in practice. Most failures come down to missing structure, not model capability.

Agents make implicit assumptions about state, ordering, and validation at the points where they hand off work to one another, and without explicit data formats, typed interfaces, and defined action schemas at those boundaries, the system breaks in ways that are hard to reproduce and harder to debug.

The architectural decisions you make around the model matter more than the model itself. That is precisely what today’s issue covers. Fitz Nowlan, VP of AI and Architecture at SmartBear, has spent years building agentic systems in production.

Let’s get started.

Featured Newsletter: The Cloud Playbook

If you own reliability, cost, and compliance in production on AWS, The Cloud Playbook newsletter is worth your attention.

It is a weekly read for engineering leaders building platforms where incidents, audits, and surprise AWS bills are real problems. It covers AWS architecture, platform engineering, FinOps, security, observability, and compliance frameworks like FedRAMP, HIPAA, and ISO 27001.

Subscribe to The Cloud Playbook

Multi-Agent Systems Need Rules to Stay Reliable in Production

by with Fitz Nowlan

As agents pass tasks to one another, reason autonomously, and compose solutions on the fly, it is easy to construct a version of multi-agent AI that looks technically sound on a whiteboard. In practice, however, that version tends to fall apart when context gets lost between handoffs, when loops run without terminating, and when open-ended systems become progressively harder to test, debug, or trust.

Fitz Nowlan, VP of AI and Architecture at SmartBear, has spent the past two years building agentic systems in production, and his view, drawn from that engineering experience, is that the reliability of a multi-agent system is determined less by the model you choose and more by the architectural decisions you make around it.

Preserving facts across the handoff boundary

The first place multi-agent systems break down is at the handoff. When agent A passes context to agent B, the question is not just what to pass but how to pass it. Raw string prompts are the path of least resistance, but Nowlan argues they are also the path most likely to introduce errors.

“We are almost always using some form of structured data,” he points out. “It’s generally going to be JSON. What we’ll often try to do on top of that JSON is define, effectively, a domain-specific language, or DSL, or our own schema, to cache, kind of, to preserve truth that we’ve already determined.”

The distinction matters because of what happens when you do not preserve that truth. Suppose an agent is working within the context of a browser and identifies that two elements appear inside the same container on a page. That relationship has been syntactically proven through a screenshot, the HTML, or the DOM. If you pass that context as plain text to the next agent, that agent has to decompose the text and then probabilistically reconstruct whether the relationship exists at all.

“That other agent has to effectively decompose and then probabilistically, i.e., potentially hallucinate, that relationship is there,” Nowlan notes. “So what we try to do is communicate over JSON and lock in to the JSON in our custom schema the facts that we’ve identified or extracted from one context or world to another.”

The practical principle is simple enough. Anything that has been established as a fact should be encoded as a fact, not as prose. Structured contracts at the handoff boundary are not just a formatting preference. They are a mechanism for preventing the downstream agent from having to guess at things the upstream agent already knew.

When MCP outperforms a static API wrapper

Model Context Protocol (MCP) has become a standard topic in agentic architecture discussions, and anyone planning to use it should understand where it actually adds value and where it does not.

“If you knew all five workflows that your customers ever did in your application, well, then you should just statically code them up,” Nowlan reasons. “Make it a single API endpoint, take the inputs at the outset, chain them all together, and spit the outputs back out to your user. That would be highly efficient. You wouldn’t then need MCP or agents at all.”

The benefit of MCP is the emergent use case. That is the workflow your users need that you did not anticipate, in the sequence they need it, composed on the fly from the tools you have exposed. By giving the model a set of tools rather than a fixed pipeline, you allow the system to compose solutions for problems you never explicitly programmed for.

“If you have an application that’s very diverse, with a diverse set of data, with a wide-ranging type or class of data, and maybe even different user roles that are coming in and using your product, that’s where MCP can really be a massive unlock, because the composition artifacts are effectively infinite,” Nowlan observes.

The corollary is equally important. If your application is relatively rigid and your users reliably need the same five things, MCP introduces complexity without adding value. The decision to use it should follow directly from the diversity of your use cases, not from the fact that it is available.

The 80-20 rule for DAG versus autonomous orchestration

One of the more practically useful frameworks Nowlan describes is how his team decides when to use a fixed Directed Acyclic Graph (DAG)-based workflow versus an open-ended MCP-style loop, and how that decision evolves over time.

The starting point is openness. When a new feature or product area is introduced, the team tends to begin with the full MCP loop. The premise is straightforward. Here is the set of tools. Let the system compose whatever the user needs. This preserves maximum flexibility while the team learns how customers actually use the system.

Over time, patterns emerge. Certain workflows appear repeatedly. Those are the candidates for promotion into a DAG.

“There’s kind of an 80-20 rule here, where 80% of the time your customers are looking to solve their problems with 20% of these key workflows that you’ve identified,” Nowlan notes. “Those should then be translated into DAGs, into workflows, into a little bit stricter information flow architecture.”

The payoff is significant, Nowlan affirms. Once a workflow is on a DAG, you can optimise aggressively. You can use a cheaper model for nodes that do not require frontier-level reasoning. You can shed context that is irrelevant to a particular step rather than packing everything into the window. You can bound latency because the execution path is known in advance.

“When you get on that track, that predefined workflow, that’s where you can save cost, you can potentially use a cheaper model for a particular node that you know doesn’t need the most expensive model. And you can also shed information from context in that workflow when you know it’s not necessary for the outcome.”

The open-ended MCP loop remains available for everything else, for the 20% of use cases that do not fit a known pattern. The architecture supports both, and the team actively monitors usage to identify when a new workflow has crossed the threshold and is ready to be promoted. There is no firm cutoff, just the observation that a workflow is appearing often enough to justify the investment in making it faster and cheaper.

The infinite loop problem

Bounding execution is one of the more fundamental challenges in agentic systems, and it is one that the industry is still working through. The early solutions were basic by necessity. At the most fundamental level, Nowlan explains, it comes down to two questions: has the system hit a timeout, and has it exceeded a set number of attempts.

The deeper issue is that the problem did not originate with modern agents. In the early days of GPT-3-era pipelines, a common approach was to ask the model for one action at a time, execute it, and return for the next. The loop would never terminate because the model had no reliable sense of when it was done.

“We quickly realized they would literally go on forever. The loop would just never terminate, would never know that it was done, or it would know that it was done way too early.”

The response was to shift from reactive step-by-step prompting to upfront scoping. Rather than asking the model what to do next after each action, you ask it at the outset to decompose the overall task into a bounded set of subtasks, and then you execute those subtasks in sequence and stop.

“We pivoted from the early days toward putting more bounds on the problem space. What are the things you think I should do? Break this down into a set of tasks, a set of subtasks, and then I’ll take each of those subtasks in turn, but when I get to that last subtask, I’m done. I’m not going back and asking you for more.”

As models have improved, this has relaxed somewhat. The team now allows a final evaluation pass, a double or triple check at the end, before closing the loop. But the underlying principle has not changed. Get the model to scope the work before execution begins, and treat that scope as a constraint rather than a suggestion.

“You want to get the AI to put bounds and scope around the overall work that you’ll be doing to solve or complete some task, and try to stick to that as a guide so that you don’t run off into the infinite space of querying forever.”

Testing non-deterministic flows

Testing a system where the execution path changes based on the model’s output requires a different approach from conventional integration testing. Nowlan and his team lean toward trace-based evaluation, grounded in a close understanding of realistic user inputs.

“All of our evaluations are, we think, we hope, reasonably close to the reality of those inputs that we’re going to get from our end users.”

The logic is that if you understand the domain well enough, you can anticipate the shape of the tasks users will bring to the system even if you cannot predict the exact inputs. In a web testing context, for example, that means understanding that users are going to log in, fill out forms, scroll through lists, and navigate between pages. Those logical actions become the basis for evaluations, and the evaluations can then check whether the agent followed a reasonable path to completion rather than whether it followed an exact predetermined path.

At the same time, Nowlan acknowledges that no evaluation suite fully covers the space of real-world inputs. The complement to pre-built evaluations is comprehensive logging and tracing of every prompt and response the system exchanges with the model.

“We log and trace all of our inputs and outputs that we exchange to the LLMs, and then we can go back and debug those, and examine those, and we can obviously use AI to probabilistically parse and understand those inputs and outputs.”

This creates a feedback loop that pre-built evaluations cannot replicate. When a user reports a bad experience or churns, the team can go back to the traces for that user, examine the prompt and response sequences, and use a model to evaluate whether the quality of the AI outputs degraded at any point. The evaluation happens after the fact, using the actual production inputs rather than synthetic ones.

Nowlan’s broader point is that evaluation in non-deterministic systems is not a gate you run before deployment. It is an ongoing process that runs in parallel with production, using real data to surface quality issues that no pre-deployment test suite would have caught.

🛠️ Tool of the Week

smolagents — Hugging Face’s open-source library for building agents that think in code

Highlights:

• Code-first agents: Generates and executes Python code instead of structured tool calls, which can reduce back-and-forth with the model in some workflows.

• Model-agnostic: Works with local Transformers models, Hugging Face Inference API, and providers via LiteLLM.

• Supports sandboxing: Can run code in environments like Docker, E2B, or Pyodide, but requires proper setup for safety.

• Hub integration: Tools and agents can be shared and reused via the Hugging Face Hub.

Learn more about smolagents

📎 Tech Briefs

• Anthropic moves Claude Mythos into controlled early access under Project Glasswing - Limited rollout to partners including Amazon, Apple, and Microsoft for defensive security work, with early reports pointing to strong capability in identifying software vulnerabilities

• Microsoft Copilot Studio ships multi-agent orchestration to general availability - Now supports A2A protocol for agent-to-agent delegation and cross-app agent reuse via the Microsoft 365 Agents SDK.

• Google releases Gemma 4 under Apache 2.0 - Ranges from lightweight edge models to a 31B-parameter variant with strong performance on reasoning, agentic workflows, and multilingual tasks.

• LangGraph ships async subagents in latest Deep Agents update - Adds non-blocking background subagents alongside type-safe streaming and Pydantic coercion in the new v2 API.

• Microsoft releases open-source Agent Governance Toolkit - A seven-package system that intercepts every agent action before execution at sub-millisecond latency, with native integrations for LangChain, CrewAI, Google ADK, and Microsoft Agent Framework.

That’s all for today. Thank you for reading this issue of Deep Engineering.

We’ll be back next week with more expert-led content.

Stay awesome,

Saqib Jan

Editor-in-Chief, Deep Engineering

If your company is interested in reaching an audience of senior developers, software engineers, and technical decision-makers, you may want to advertise with us.

Thanks for reading Packt Deep Engineering! Subscribe for free to receive new posts and support my work.

Learn ownership, RAII, smart pointers & allocators to eliminate leaks/UB—live hands-on with Patrice Roy—#1 bestselling author of C++ Memory Management and ISO C++ Standards Committee (WG21) member. Online: Sat Apr 11, 10:30 AM–Sun Apr 12, 4:00 PM ET. Limited seats.

Register now

✍️ From the editor’s desk,

Welcome to the 41^st issue of Deep Engineering!

As AI scaling hits the memory wall, system performance is increasingly limited by data movement rather than raw compute power. This shifts the optimization focus toward data locality because adding cores provides no benefit when bandwidth bottlenecks the system. Writing fast code now requires precise control over memory layout and concurrency, yet these techniques often introduce complexity that becomes unmanageable as codebases grow.

That specific tension defines this week’s feature with Sam Morley, Research Software Engineer and Mathematician at the University of Oxford and author of The C++ Programmer’s Mindset.

Building on our earlier discussion in Part 1 around decomposition and abstraction costs, today's issue digs deeper into the guiding principles for scaling C++ systems. It covers how teams can keep complexity under control as codebases grow, why template metaprogramming deserves extreme caution, and why thinking with the machine is non-negotiable when optimising cache layout.

Let’s get started.

Stop Building Vault

Secrets, PKI, & PAM in one platform. Postgres-backed. No custom orchestration. Flexible deployment.

Start for free today →

Adopting the C++ Programmer’s Mindset (Part 2) with Sam Morley

Before listing any specific practices, Morley introduces a concept he returns to repeatedly. He calls it the future you.

“Future you is your future self and for all intents and purposes this is a different person,” Morley says. “When you’re writing some code, you understand things the way they are in the context of what you’re doing at the moment. Future you will have lost this context. So, when you come back to your code in a month, six months, a year’s time and you look at it and you think what was I thinking — almost surely the answer to that is: I don’t know.”

The practical implication is easy to skip under deadline pressure, but it could be costly to ignore. Comments are not just for teammates. They are messages to yourself, written at the moment of maximum understanding, for a future reader who no longer has that understanding. Morley is explicit that this does not mean narrating the obvious. What he does instead, particularly on mathematically intricate work, is write large block comments that describe where the process is, how the next section works, and what the algorithm is intended to achieve. “These comments save me so much pain when I jump off the project for a week and then go back and have to remember exactly what I was trying to do.”

Beyond comments, Morley points to three C++ specific practices that matter most for scalability. The first is a strict separation of concerns. Numerical computation should not live in the same place as user-facing code. Components should be modular enough to be tested in isolation. This also pays dividends when distributing computation across clusters, because reusable tight-loop routines can be dropped into different distribution mechanisms without rework.

The second practice is thinking about thread safety earlier than you think you need to. Designing class members with that future in mind costs almost nothing early and can be enormously expensive to retrofit later. The third is keeping your build system clean. Morley uses CMake and is emphatic about this. “Having a broken build system is far worse than having broken code. It’s much harder to figure out what exactly has gone wrong if your build system is broken.” Build systems accumulate debt quietly and the consequences arrive at the worst possible moment, usually when you need to extract a component into its own library under time pressure.

When metaprogramming becomes the problem

“I’ve seen some horrendous template metaprogramming in my life,” Morley reflects. “I’ve written some horrendous template metaprogramming in my life. I’m going to be the first one to admit that it’s never worth it.”

The target here is the elaborate kind: deeply nested type machinery and SFINAE-heavy enable_if chains. Instantiating a complex template metaprogram can easily double compile time for a single translation unit. At scale, across thousands of files, the cost becomes structural. Morley notes this is exactly why Google kept metaprogramming to an absolute minimum when writing Abseil. Modern C++ has reduced the genuine need for TMP significantly. Concepts and constexpr functions cover much of what engineers used to reach for TMP to solve, with readable syntax and without the compile time penalty.

Lambdas are a different conversation. Used correctly, Morley thinks they are one of the most readability-enhancing tools in the language. Used carelessly, they introduce exactly the kind of invisible coupling that makes the future you miserable. His specific warning is about lambdas that capture and modify values defined far away from where the lambda is used. “Every time you think what is this lambda doing, it’s modifying something that you’ve not looked at for a long time because your screen has been further down the page.” In cases where a lambda is the only place that reads or writes a particular value, Morley argues the value almost certainly belongs inside a class, where the ownership and mutation path are explicit.

Thinking with the machine

Morley uses a road analogy to introduce hardware-aware programming. If you are driving down an unfamiliar road in the dark, you slow down because you cannot see what is ahead. Knowing the road means you can go faster without risk. The system you are running the code on is the road, the code you are writing is the car, and understanding the road conditions means you can drive faster with confidence.

The most practically important piece of hardware to understand is the cache hierarchy. Cache behavior is invisible in the code but determines a large part of actual runtime performance. The canonical example comes from the games industry: the debate between Array of Structs (AoS) and Struct of Arrays (SoA).

If you represent each game entity as a large struct and store those structs in a vector, then iterating over position data means striding across the full width of each struct to reach the next position value. “Big strides are bad for the cache. What you really want is all of the position data to be close together.”

The alternative is to separate data by type, storing all position values together and all velocity values together, so that each field is laid out contiguously in memory rather than interleaved across structs. As Morley puts it: “This transformation basically doubles or quadruples your throughput because now you don’t have to step over all of the useless data in order to update your position.”

Matrix multiplication illustrates the same principle at a more concrete level. In one direction, data access is sequential and cache-friendly. In the other direction, moving between rows in a column means jumping across memory in steps as wide as the entire matrix, which is exactly what the cache is designed to avoid. The standard mitigation is tiling: load a small tile of the matrix, do as much computation as possible on that tile, then move to the next one. Morley walks through a tiled implementation in the book and notes roughly a four-times improvement over the naive approach from tiling alone, before any SIMD or pipelining optimisations are applied.

Branch prediction and SIMD matter too, but Morley is careful about context. For general-purpose application code, the compiler will usually make reasonable decisions, and the latency bottleneck is more often a network call or a disk write than an instruction pipeline stall. These details start to matter seriously when throughput is the primary constraint: training large models, running physics simulations, anything where a microsecond saved per operation compounds into meaningful wall-clock time across billions of invocations. “Taking an extra microsecond to do a calculation is devastating when you have to do that a billion times.”

Concurrency: two models, two risk profiles

Morley distinguishes two fundamentally different multi-threaded scenarios that require different mental models.

The first is data parallelism. A large dataset split across threads, each thread operating on its own independent range with no shared state. “There’s never any overlap. Each thread goes away, does its work, and the results are put in the buffer.” This is relatively safe territory. Parallel algorithms and OpenMP handle much of the machinery, and as long as the data partition is clean and there is no self-referential access, data races are structurally prevented.

The second scenario is harder. Multiple worker threads operating on shared state within a larger system. A dispatch queue is the typical pattern, where a main thread stacks work items and worker threads pull from the queue. Here, ownership discipline becomes non-negotiable.

Morley’s design goal is simple to state and difficult to achieve: only one thread, one function, one whatever should be able to modify a value at any given time. There are two ways to achieve this. The first is architectural: design the program so that each thread exclusively owns its data and never touches another thread’s values. The second is synchronisation: use atomics, mutex-locked values, or other thread-safe mechanisms to control access.

Morley urges engineers to read the synchronisation documentation carefully before reaching for any of these tools. Deadlocks happen when engineers use them without fully understanding their semantics. The architectural solution is always preferable when achievable: if you can design the program so that each thread never touches another thread’s data, you eliminate the entire class of problem. When you cannot, synchronisation primitives are available, but the burden of correctness falls entirely on the engineer.

Memory safety and the Rust argument

The issue closes on a topic that provokes strong reactions in C++ communities. With around 70% of serious security vulnerabilities attributed to memory safety failures in C and C++ code, the question of whether the language itself is the problem is not academic.

Morley’s answer is direct. “Go and learn some Rust.”

He anticipates the objection. “A lot of C++ programmers turn their nose up when Rust is mentioned. Generally, the feeling is: we don’t need Rust, we can do all of this in C++. But that’s not the point.” The point is that the Rust compiler forces a specific kind of thinking about ownership and lifetime that C++ leaves entirely to the developer. Rust’s sync and send traits enforce thread safety at the type system level. Unsafe code is marked explicitly, which means the developer makes a deliberate choice to step outside safe boundaries rather than doing so accidentally. “Learning a bit of Rust will make you better at writing safe C++. The reverse is not true.”

For engineers who are not yet ready to invest in Rust, Morley's immediate C++ recommendations are practical and specific. Prefer std::array over C-style arrays. Use smart pointers instead of manual memory management; writing operator new directly in application code is an antipattern at this point. Use std::span rather than raw pointers when passing data around. Avoid the C standard library IO functions entirely: gets, puts, sprintf, and their relatives have documented vulnerabilities and safer equivalents exist.

“The best-case scenario for a bad memory access is a crash. If it goes unnoticed, it could happen for months before you notice that this has been producing garbage the entire time. By which point you’ve wasted months.”

The throughline from Part 1 holds here. Whether the subject is decomposition, abstraction, maintainability, or memory safety, Morley keeps returning to the same discipline: be conscious of what you are doing, understand the costs of your choices, write for the person who will read the code next, and treat the machine as a collaborator rather than a black box.

🔍 In case you missed it…

The complete Chapter 1 from Sam Morley’s book, The C++ Programmer’s Mindset 👇

🛠️ Tool of the Week

Google Benchmark — A microbenchmark support library for C++

Google Benchmark is an open-source framework for measuring the performance of isolated C++ code so you can test cache-aware designs and tight-loop optimizations with high precision.

Learn more about Google Benchmark

📎 Tech Briefs

• C++26 Finalizes Technical Work — The ISO committee completed the C++26 draft in London, locking in features like compile-time reflection, memory safety improvements, and contracts ahead of final approval.

• Rust 1.94.1 Shipped — A point release that fixes compiler regressions, resolves clippy false positives on match arms, and updates Cargo dependencies to address critical CVEs.

• RustRover 2026.1 Released — JetBrains introduced cargo-nextest integration, a call hierarchy view, LLDB 21 upgrades, and improved DWARF indexing for procedural macros.

• CppCon 2026 Opens Registration — Registration is open for the Colorado event, with sessions covering AI infrastructure, low-latency design, and hardware-aware performance.

• Linux Kernel 7.0 Approaches — After Linux 6.19, the merge window for 7.0 is now open, pointing to a major version release expected later this April.

That’s all for today. Thank you for reading this issue of Deep Engineering.

We’ll be back next week with more expert-led content.

Stay awesome,

Saqib Jan

Editor-in-Chief, Deep Engineering

If your company is interested in reaching an audience of senior developers, software engineers, and technical decision-makers, you may want to advertise with us.