Featured - LangGraph Masterclass: From Beginner to Professional
This hands-on masterclass takes you from LangGraph fundamentals to supervisor and hierarchical multi-agent systems, with live debugging in LangSmith throughout.
Deep Engineering readers save 50% with code - DEEPENG50.
✍️ From the editor’s desk,
Welcome to the 60th issue of Deep Engineering!
CNCF published the KubeCon North America schedule on 10 August and added a new AI Inference and Agentic track, with sessions on GPU scheduling, model serving and production observability across vLLM, KServe, Ray and OpenTelemetry. A conference track is a lagging indicator, which is what makes it useful. It means enough organizations are running agents in production that the operational problems have settled into a shared vocabulary.
Engineering teams still lack a reliable method for deciding how many agents a problem needs and which coordination pattern should connect them. They often choose a topology that resembles the problem, whether supervisor-and-worker, debate, or parallel attempts with convergence, then use production behavior to determine whether the structure fits. The patterns are well documented, but the evidence that should decide among them is not.
We put that question to Chi Wang and Xiao Ma during a roundtable at ARC 2026, Packt’s virtual summit on software architecture in the age of AI. Wang developed AutoGen during a decade at Microsoft Research, later joined Google DeepMind and now works on AG2, MassGen and Sutando. Ma leads the teams behind Splunk Observability Cloud at Splunk, a Cisco company, after seven years as Medium’s director of engineering and chief architect.
Neither believes teams can choose the right topology from a diagram alone. They start with the simplest credible system, study where it fails and add structure only when the evidence justifies it.
Today’s issue lays their framework out as a sequence, starting from a single-agent baseline, splitting only when a specific failure justifies it, and using traces to check whether the split helped.
Let’s get started.
Thor.ai — One Source of Truth for Your Coding Agents
Thor is a context brain for teams running coding agents like Claude Code, Cursor and Codex. It keeps one current, sourced record of what is true across your tools, so agents stop guessing and specs stop going stale the moment a decision changes.
🧠 Expert Insight
How to Choose Between Single-Agent and Multi-Agent Architecture
by Saqib Jan Jan with Chi Wang and Xiao Ma
Multi-agent architecture often begins with a diagram that assigns a supervisor, several workers and a path toward consensus, yet the diagram says little about whether the task needs those roles or whether one capable agent could complete the work with fewer coordination costs. Debate, hierarchical delegation, role-based teams and parallel attempts all solve different problems, so choosing among them requires more than matching a familiar pattern to a new use case. A team needs a baseline that exposes the limitation its next architectural choice must address.
Chi Wang’s early work on AutoGen at Microsoft Research shows why the baseline matters. His first design placed one coding agent in a loop that wrote code, executed it, revised the result and continued, but the agent could not handle the growing instructions and exceptions reliably. The failure justified decomposition at that point, while later improvements in model capability suggest that the same design could travel much further today, which means the original boundary belonged to a particular combination of model, tools and task rather than to the problem for all time.
Multi-agent topology therefore works best as a design hypothesis that a team tests against observed behavior. The problem establishes the first credible architecture, evaluation defines whether it succeeds, and execution traces explain why it falls short. That sequence preserves architectural judgment while giving evidence the authority to change the design.
One Agent Should Establish the Baseline
Wang favors the simplest credible starting point because every additional agent creates another context boundary, another exchange to inspect and another opportunity for coordination to fail. One capable agent with appropriate tools provides the cleanest baseline when it can receive the required context, use that context effectively and complete the task within the required time. The team can then add structure in response to a demonstrated limitation instead of paying for complexity in advance.
Context can rule out the single-agent option before execution begins. Two people who want their personal agents to coordinate a meeting may trust each agent with a private calendar while refusing to share both calendars with a common service, so the information boundary requires multiple agents even though the scheduling operation remains simple. In that case, decomposition protects context rather than compensating for weak reasoning, and forcing one agent into the center would weaken the design.
Efficiency creates a different boundary because an agent may receive all the relevant material and still fail to use it consistently. Instructions can accumulate until the agent stops following them, while long-running work can compete with urgent one-off tasks and produce unstable priorities. A context window that can contain everything does not guarantee an agent that can reason across everything, so the architecture must account for how the agent uses context rather than measuring capacity alone.
Ma, who leads teams building Splunk Observability Cloud at Splunk, a Cisco company, adds parallelism as a separate reason to introduce more agents. A task may require several paths of investigation within the same time window, either because latency matters or because independent attempts provide useful diversity. One agent could complete those paths sequentially, but the execution would no longer meet the operational requirement, which makes parallelism an architectural need rather than an embellishment.
Start with the simplest credible system, then add structure only when context, reliability or parallelism creates a reason for the split.
Paraphrased from ARC 2026
The single-agent baseline should remain a default rather than a doctrine. Privacy constraints, incompatible contexts and time-sensitive parallel work can justify a multi-agent design from the beginning, while a task that lacks those constraints should first prove that one agent cannot handle it. This distinction keeps simplicity useful without turning it into another pattern that teams apply without evidence.
Failure Modes Should Choose the Next Topology
A failed baseline does not automatically justify decomposition because different failures require different coordination patterns. An agent that nearly completes the task but makes inconsistent mistakes presents a reliability problem, while an agent that rarely completes the task presents a capability or decomposition problem. Treating both cases alike adds agents without identifying the work those agents need to perform.
Wang, who created MassGen, uses parallel agents when each agent can attempt the same problem successfully but may fail in a different way. The agents inspect and refine one another’s answers, so the architecture seeks reliability through diverse attempts without dividing the task into separate specialties. This pattern works when the baseline already demonstrates substantial capability, while it adds cost without repairing a task that no agent can solve.
A low baseline success rate calls for smaller units of work with narrower contexts, distinct tools or specialized responsibilities. The team first raises the reliability of each unit, then adds verification or coordination to close the remaining gap, which shifts the design from repeated attempts toward role-based decomposition. More agents alone do not produce this improvement because the gain comes from changing the problem each agent receives.
Parallel execution introduces another distinction because some tasks need multiple workers even when a single worker remains capable. Independent investigations can proceed together and feed a synthesis step, while dependent work may require a coordinator that controls sequence and handoffs. Both topologies use several agents, but one optimizes exploration and latency while the other manages dependency, so the same agent count can conceal a different architecture.
Coordination cost complicates every choice because each split creates more messages, context transfer and failure paths. A team should therefore classify the limitation before changing the topology, then test whether the new structure improves the system-level result enough to justify its additional work. Success rate, latency and trajectory quality give that decision a firmer basis than the number of agents in the diagram.
Context and Expertise Define the Decomposition
AutoGen’s early decomposition followed the limits that appeared inside one agent rather than a predetermined organizational chart. Wang added instructions as the coding loop encountered errors and exceptions until the accumulated guidance became difficult to follow, then separated responsibilities so that each agent could work with a smaller set of concerns. He later separated continuous work, including self-improvement, from temporary work such as producing one piece of content because the two time horizons disrupted one another’s priorities.
Ma uses an incident war room to show how context and expertise can define the equivalent split in production systems. An incident commander coordinates communication while service, infrastructure, database and Kubernetes specialists investigate with different tools and knowledge, then each finding changes the work that follows. The roles matter because they encode distinct context and capability, not because a multi-agent framework needs several named participants.
The analogy also has limits because software agents do not inherit every organizational constraint that shaped a human team. An existing human role can still provide a useful starting hypothesis when it owns a clear context, toolset or decision, but copying an org chart without those boundaries reproduces titles rather than engineering logic. Every proposed agent should therefore own a distinct reason for existing, while responsibilities that cannot meet that test should remain together.
This approach turns decomposition into a response to context pressure, expertise and dependency rather than a search for the ideal number of agents. It also makes later changes easier to justify because a team can identify which boundary moved when a model improved, a tool changed or two responsibilities began to interfere. Under-Decomposition Is the More Common Mistake develops this test through the war-room example and the three conditions that support a split.
Observability Has Become Architecture Work
Traditional systems often treat observability as an operational layer that helps developers and SREs diagnose latency, errors and outages after the software exists. Ma argues that agent systems require evaluation and observability earlier because prompts, context assignment and topology now form part of the build process, while source code alone cannot establish how those choices will behave during execution. A team needs evidence from the agent trajectory before it can decide whether to change the topology, the model, the tools or the context.
System-level evaluation establishes whether the agent team completed the task to the required standard, while traces explain how the team reached that result. A useful trajectory records which agent acted, which context it received, which tool it selected, what it returned and how the next agent used that result. Those details expose missing context, repeated work, poor tool selection, prolonged deliberation and sequential work that could proceed in parallel.
Evaluation establishes whether the agent team solved the task, while traces show how its coordination helped or failed.
Paraphrased from ARC 2026
Trace data does not choose an architecture by itself because a team still needs a task-level definition of success and enough judgment to distinguish a topology problem from a model or prompt problem. More telemetry can create more detail without creating a decision when the evaluation criteria remain vague. Observability becomes architecture work only when the team connects a trajectory to an outcome and uses that relationship to test a specific change.
This requirement expands the audience for observability beyond developers and SREs. Product managers and designers may understand expected customer behavior better than the engineers who implemented the agent, which gives them a direct role in evaluating production trajectories and refining the product. Teams that restrict trace access and interfaces to operations staff exclude people who now contribute to the build loop, so the tooling must support technical diagnosis and product judgment together.
Every Topology Remains Provisional
A topology that succeeds today can regress because agent systems combine models, tools, prompts, context and external conditions that change at different rates. Wang treats regression as a central production problem because delegation reduces control over the route an agent takes, while each new task can expose a case that the earlier evaluation set never covered. One successful execution therefore proves that one configuration handled one situation, not that the architecture has established a permanent baseline.
Continued evaluation protects the design from that false confidence by showing when an agent begins taking longer paths, when coordination adds cost without improving quality and when a task that once fit inside one agent now requires a split. Traces make those changes visible, but the team still needs versioned evaluations and a stable definition of acceptable behavior to compare one configuration with another.
A durable process begins with a task-level measure of success and the simplest architecture that respects known context, expertise and latency constraints. The team captures the execution trajectory, classifies each failure before adding structure and repeats the evaluation after every material change to the topology, model, tools or context. This process does not eliminate design judgment, but it prevents an early diagram from becoming permanent without evidence.
Multi-agent architecture therefore develops through a continuing exchange between design and observation. The initial topology expresses the team’s best hypothesis about the problem, while evaluation and traces test whether that hypothesis holds under execution and remains valid as the system changes. A team still designs the architecture, but evidence decides how long that design deserves to survive.
In case you missed
Multi-Agent Architectures in Production with Chi Wang and Xiao Ma
Multi-agent gets attached to almost anything now, which makes it hard to design against.
Forward Deployed Engineer Jobs Are Out There. Hiring Is Harder Than The Postings Suggest
As companies integrate advanced AI systems into core workflows, they need engineers who can turn persuasive pilots into reliable production systems.
🛠️ Tool of the Week
Arize Phoenix — open source tracing and evaluation for agent systems
Captures full multi-agent traces so you can see which agent called which, in what order, and with what context
Runs evaluations over traces rather than single responses, including LLM-as-judge and custom evaluators
Built on OpenTelemetry and the GenAI semantic conventions, so it fits existing pipelines rather than replacing them
Self-hostable, which keeps prompt and completion data inside your own boundary
📎 Tech Briefs
Microsoft ships its August security updates - Microsoft fixed 421 CVEs, including one exploited zero-day and two publicly disclosed vulnerabilities.
CNCF graduates Cloud Native Buildpacks - Graduation follows a Quarkslab and OSTIF security review and an OpenSSF Best Practices badge.
Kubernetes explains native KYAML output - KYAML makes structure and value types explicit while remaining compatible with existing YAML tooling.
CNCF adds an AI Inference and Agentic track to KubeCon - The track covers agentic workflows, GPU scheduling, model serving and production observability.
GCC 16.2 fixes regressions in GCC 16.1 - The release corrects more than 102 regressions and serious bugs in GCC 16.1.
That’s all for today. Thank you for reading this issue of Deep Engineering.
Keep building,
Saqib Jan - Editor-in-Chief, Deep Engineering