Multi-agent gets attached to almost anything now, which makes it hard to design against. The question underneath is narrower. When does a problem stop fitting inside one agent, and what do you build once it does.
We hosted a fireside chat on exactly that at ARC 2026 with Chi Wang and Xiao Ma, who have built these systems from opposite ends.
Chi created AutoGen at Microsoft Research, then AG2, MassGen and Sutando. He previously led agentic AI work as a Senior Staff Research Scientist at Google DeepMind and teaches at Stanford, Berkeley and DeepLearning.AI.
Xiao was chief architect at Pattern Insight and Medium, and now leads the teams building Splunk Observability Cloud at Splunk, a Cisco company, including its enterprise multi-agent systems.
Their book Multi-Agent AI Engineering is available on Amazon. We talked about when a problem needs more than one agent, how to choose a coordination pattern, and what production does to systems that looked fine in a demo.
Multi-agent is now attached to almost everything. What does multi-agent architecture actually mean to you this year, and is the term being used too loosely?
Chi Wang: The meaning of it is quite broad, and the most important paradigm can also change over time. When we talk about multi-agent, the meaning we refer to can evolve, and the reason for that change is basically that the meaning of agent, and what it encompasses, has changed.
Back in 2023 the model was GPT-4, and an agent essentially was a model plus a few tool calls. We kept making model inference using the tool, sometimes with tool calls, sometimes without. At that time, multi-agent was about how you address the weakness of these initial powerful language models. They were already powerful, but there were also a lot of limitations, so we needed to build harnesses on top of the model. Making the model reliably use tools was one of the hard problems then. We often needed multiple agents to check the results of the tool call, and to perform different tasks, because a single model is hard-pressed to follow complex instructions. We needed to decompose a big task into smaller ones and have each agent focus on a specific problem.
In 2025, most of the model providers offered their agents with built-in tool calls, and in many cases multiple different ways to configure the models to access different kinds of tool calls. Search and coding are the two common ones across different model providers, but each of them has slightly different dialects, and different strengths and weaknesses. When we solved very hard problems like the Humanity’s Last Exam benchmark, each single model plus tool at that time made a different kind of mistake. So multi-agent could mean we ask each agent to solve the same problem, then ask them to check each other’s answers and iterate on top of that. That is what MassGen tries to do, mimicking a study group kind of architecture.
This year there is even more progress in personal AI, so we can design one personal AI agent for everyone. You have your agent, I have mine. When we use them they interact with us, and they develop unique strengths. My agent may be very good at research, your agent may be very good at communication. When we put them together and let them interact, we have an even higher level of multi-agent system, because each personal AI may already contain multiple agents inside. Those powerful agents can still talk to each other, and we can compose even stronger systems. So there are different levels of architecture we can combine, and we can do this recursively. Over time we may see this abstraction go higher.
Shed some light on the landscape too, and tell us what multi-agent is not, in your view.
Xiao Ma: The term may become confusing because there is a huge debate about how many agents you really need for your task. At some point this becomes a semantic discussion rather than the real discussion to solve problems. Multi-agent does not mean you literally have multiple repositories, multiple processes. Even with one single agent, if you give that agent different context, different memory, different roles, it can do multiple things.
There is a version where it does mean many systems, which is the concept of an internet of agents. You have very diverse agents built by different companies, built by different frameworks, and they talk to each other. That by definition is multi-agent. But even within your own system you could still have the same agent play different roles, or have parallelism of multiple agents doing the same thing at the same time, and you could still call it multi-agent.
So I would say do not fixate on multi versus single too much. Think more about the problem you want to solve. If you need parallelism, you could have the same agent doing multiple things at the same time. If you need a debate, if you need agents to check each other’s results, there is an architecture for that as well. And if you need different stages, where you have planning, you have doing the work, you have synthesizing, that is another multi-agent architecture pattern you can follow.
The standard advice is to start with one capable agent and some good tools. When does a problem genuinely require multiple agents rather than one agent with more tools?
Chi Wang: The simplest answer is that when the problem is too complex for a single agent to handle, you need multiple agents. But if we break that down and get more specific, and if we refer to a single agent as, for example, a personal AI agent that I can delegate my daily tasks to, like writing, developing code, communication, then there are two questions you need to check.
The first question is, can you easily provide all the context and information needed for the agent to finish the task. The second is, can the agent effectively and efficiently use all the information you provide. If the answer to both is yes, then a single agent is enough. If either answer is no, then probably you need multiple agents.
An example for the first question is scheduling a meeting. I have my schedule, you have your schedule. Can we easily use a single agent to collect both? It is kind of hard, because we probably want to retain the information boundary. I am comfortable sharing my information with my agent, you are comfortable sharing with yours, but we probably would not be comfortable sharing with a common agent. And during the coordination, agents sometimes need to communicate back with us, and we probably want a private conversation with each of our own agents. So even when the problem sounds very simple, just scheduling a meeting, the answer to the first question is already hard.
For the second question, I often need to provide context in different modalities. Sometimes I want to just speak to my agent, or share screen, or share my camera, share vision in real time. Other times I share information by chatting. Although an agent can potentially combine different sources of information, a single agent may not necessarily do all the things I want efficiently at one time. Sometimes I want a real-time result back without long waiting, and there are certain agents that are very good at that, but a single model plus tools cannot do both kinds of requirement with today’s technology. So I often need to use different agents designed for slightly different purposes, and also have a way to connect them together to solve the problem.
You see thousands of teams building with AG2. What is the most common mistake in how people decompose a problem into agents?
Chi Wang: In general there are two kinds of mistake, over-decomposing and under-decomposing. But I think under-decomposing is not necessarily a bad choice in the beginning, because if the problem indeed can be solved by a single agent, why should you create more? You should always prefer to start from the simplest setting.
That is what I did, and you could say that was a mistake, but I would not know it was a mistake if I had not tried it. When I started building AutoGen, what I wanted was just to build a single simple agent that could write code and iterate on the code. Write code, revise code, keep iterating in a loop. That was the initial design of AutoGen, and it apparently failed. But it was still a good thing to try, because it might have worked. Today, with a more powerful model, I think that pattern can get very far already. So you should always keep the simple design as the initial starting point, and only decompose when you need to.
That is what happened with AutoGen. Initially we tried to use a single agent and kept adding more instructions to handle all the errors and exceptions encountered in different situations, until the instructions became too long and the agent could not follow them. Then I started to slowly decompose, so each agent does a smaller set of things and can focus.
This year, that loop of using one agent to keep trying things and keep refining is quite good at solving much more complex tasks than in 2023. But if you use that single pattern for everything, the problem is that you probably have tasks of high diversity. Some tasks take very long. If I want to design an agent to rewrite itself and improve itself, that is a continuous process that always needs to be on. Other tasks are temporary, like writing a particular piece of content one time. If you mix these types without good decomposition they will hurt each other, and they will be confused about when to prioritize what. So then you gradually add more decomposition to handle it. But always start from the simplest.
Chi has covered under-decomposition. What is the signal that a team has over-decomposed, that they have split work into agents that now only add coordination overhead?
Xiao Ma: I would build on what Chi just mentioned. I probably see more under-decomposition than over-decomposition. In my opinion there are three major questions we should ask ourselves when we decide how many agents we need.
The first is what Chi mentioned, can you really fit the context into a single agent. I think about this with humans as well. Our brain is not really good at multitasking and context switching. Can we really do a lot of things in one setting effectively? I do not think so. A similar thing applies to a large language model.
The second is whether you need parallelism. If you really need to explore different paths, to try different things at the same time for the sake of latency or efficiency, that is another reason to have multiple agents.
Let me give one particular example. In Splunk Observability Cloud we tackle how we observe multi-agent systems, but we also tackle the problem of how we use multi-agent systems to help people triage their system problems. All of the audience are engineers, and we all know that when the system is down we have a so-called war room, a group of people coming together to triage an outage. Think about what different roles we have in that setting. Usually we have an incident commander, and that person is really good at customer communication, really good at organizing information to post updates in different channels. And we have maybe multiple subject experts from different teams. Someone knows the service really well. Someone may know the infrastructure database really well. Someone may know Kubernetes really well. They all come together, they bring different expertise, they use different tools, they do different exploration in parallel. And then they also converge. If I find something interesting I will post it in a certain format, other people can read it, and they may continue their triage based on what I found.
Picture that kind of collaboration in a real production outage war room, and apply it to a multi-agent system. You have different agents doing different things, their context is more focused on what they are really good at, so they can do a really good job very fast. Then we gather the information back, and there may be another agent to synthesize and do the next stage of triage. If your problem fits into that model, if you would have multiple people doing the work, and you need them to collaborate and coordinate while focusing on their own expertise, that is a very strong signal you need a multi-agent system. The way you would decompose a task to human teams, you can decompose in a similar way to machine teams.
AG2 grew out of conversational multi-agent patterns and MassGen pushes in a different direction with parallel agents converging on an answer. How should an architect choose among conversational, hierarchical manager and worker, and parallel topologies? What actually decides the shape?
Chi Wang: Most of the time it is just about the nature of the problem you are trying to solve, and it is often easy to tell just by thinking about how humans would approach it. Suppose you have unlimited resources and you can let agents do anything you want. What is the most effective way, from your experience, that the problem has been solved? It is quite often just common sense.
The non-obvious part is often about knowing what a single agent can do in your existing system. By single agent there are also different granularities, different levels. If you have already built agents at different levels, that is good knowledge you can use, because if you start with the most powerful agent you have built, the type of multi-agent system will be very different from when you start with low-level agents that do simpler things.
If you already have a very strong single agent that can almost solve the task but makes some errors sometimes, then you could use a parallel pattern and design several diverse agents. Each of them may make different kinds of mistakes, but they can all fundamentally solve the same problem, and just sometimes succeed and sometimes fail. That is the scenario where you apply parallel agents and converge to refine each other’s answers.
If the problem looks very much unsolvable by a single agent, not just making mistakes sometimes but fundamentally having a very low success rate, like single-digit success rate, that means you probably need more complex patterns that do more decomposition. You need to make each single agent reach at least more than ninety percent, and then use some verification pattern, or an independent checker agent, to fill the gap of the last mile. During that conversation each agent might be doing very heterogeneous things, and you probably need a more complex conversation pattern to piece them together.
How much coordination logic should live in explicit orchestration code, and how much should be left to the agents to negotiate themselves? Where is that line today, and is it moving?
Chi Wang: It is always moving. The most tricky scenario is when you do not own all the agents in the system. If you own all of them, then you can basically profile each of them, because you know what level of complexity each one is, and choose a pattern in the way I just described.
But what if you cannot see their internals? What if you only know that I have an agent, you have an agent, and each of our teams has their own agents, but we do not know how they are implemented? Or if you are dealing with some external party’s agents, and we have totally no clue how complex they are and what they can do. What assumptions should we make about dealing with these types of agents?
When you do not have a lot of information about them, you have to make it flexible. You probably need to design your agent to be able to deal with different types of other agents, and often you cannot make too many assumptions about how capable they are. Only if you verify what they can do do you decide the most efficient way. If you find that they are less capable, then when you communicate with them you must first decompose on your side and then give them simpler tasks. Otherwise you could just provide a very high-level goal and assume they can stick to it. So figuring out the profiling of the agents is quite important.
Xiao Ma: I do not want to sell too hard, but that is a big topic in our book. In reality there is a lot of theoretical discussion we could have before we build. I am not trying to trivialize the architecture design side of things, but based on Chi’s experience and my own, a lot of this also comes from just trying it out.
This is where evaluation and observability really come in. You could try one architecture, one topology, one pattern, and just see how these agents work together. Do they really solve your problem? The only way to know that is to have already thought about evaluation and observability beforehand. You need to understand how these agents are talking to each other. These are not just static code where you can take a call stack dump and know all the details, or where you can read the source code. When agents communicate, a lot of things are non-deterministic during production time.
Once you have a really good evaluation, not just for individual LLM call and response but for the whole system, you know the trajectory, how agents talk to each other, what kind of information they exchange. And then you have the observability to have all the traces you can analyze. That gives you the information to really decide whether you need to pivot on the topology, whether you need a different model, or whether you need a different context setting for the agents. By trying different things, that is probably the best answer to really carve that line between more decomposition and more consolidation.
From the enterprise position, we have MCP for tools and the emerging agent-to-agent standards. Are we heading toward real interoperability, or just another round of framework lock-in with better branding?
Xiao Ma: Things are evolving so fast that my answer may be different in just a week or so. But on the practical view, MCP and A2A are the two things you should always start with. There are many other protocols you could consider, but most production systems are not that complex. MCP and A2A are, in my opinion, the bare minimum. One is to really abstract the tool use, the other is to facilitate the communication between agents.
If you expand to what I mentioned at the beginning, a so-called internet of agents where you have many agents from different companies and different teams, then there are a lot of other questions you should answer. One of the framework infrastructures developed by Cisco’s Outshift team, AGNTCY, tries to solve some of those problems. There is identity, authorization, even how you run different agents and how you measure their performance. Once your problem scope becomes that large you need to worry about other protocols. But to start with, have MCP, have A2A, and you can do a lot with just those two.
Xiao Ma leads Splunk Observability Cloud at Splunk, a Cisco company, and AGNTCY originated at Cisco’s Outshift incubator.
Chi Wang: If we look back at the framework environment, every time the model is upgraded, or the harness on top of the model is upgraded, that brings a whole new set of challenges. It changes the focal point about where interoperability is required.
MassGen is one example. When we started building MassGen we did not have real interoperability across the model providers’ most powerful agent offerings at that time. Initially it was more about interoperability across models, because we wanted a way to delegate the same task to different models. That layer was relatively easy before last year. But when each model provider started to offer their own agent offering, it was not just making model inference standard, it was about their internal reasoning, their tool use, and what gets exposed to developers. There were lots of issues in that regard, so we had to build our own interoperable layer to make these agents able not only to solve the problem by themselves but also to share and check each other’s answers. That already required more interoperability challenges to be solved.
This year the single agent’s power has moved higher. That does not mean the previous protocols are not useful. At the relatively lower level of abstraction, how you abstract models and tools, you should use MCP. And for combining relatively simple agents talking to each other, you probably use A2A. But as the single harness becomes more and more complex, like Codex and Claude Code, these coding agents already have MCP built in, they already have other protocols built in, but they present themselves as even more complex agents than before. How do we make them interoperate with each other? How do we combine multiple such coding agents, or customize them to do things beyond coding, and make them a personal AI that does everything, or add non-coding capability like processing real-time requests, processing audio and visual? Right now they are scattered across different types of models and different types of systems.
What we really want is a single standard that can process all of these, and use some unique protocols to piece them together. And when you make a personal AI usable by other people as well, and even let different personal AI agents talk to each other, then we need higher-level protocol standards. So this is a constantly evolving landscape. The previous protocols will still be useful at a certain layer, and we will then need new protocols for the higher layer.
You build AI observability as a product. What does observing a multi-agent system require that metrics, logs and traces do not already give us?
Xiao Ma: The biggest shift from observing multi-agent systems compared to traditional systems is that for traditional systems, observability is more of a production concern and less of a building concern. It is more on the operations side, because you observe because you want to know latency, you want to triage an outage. You use MELT, metrics, events, logs and traces, mostly for triaging systems, and you occasionally use the data for building. If I know there is a performance bottleneck, if I know there are some bugs, I can use the data for my building. But I would say ninety-five percent of the case is build first and observe second.
The multi-agent system is the other way around, because the building part essentially relies on the observability data and the evaluation data. One example we mentioned earlier is that you do not even know the best topology, the best way to decompose your system. You need observability to teach you how to do this. And there are many cases where your building is essentially iterating on the topology and the prompt, or skills, whatever term we use today, rather than writing a bunch of code. The way to iterate on the topology and the architecture of your agents is using the evaluation and observability data.
It has evolved even to a point where observability has different personas now. Previously the personas were just SRE, just operations folks, and sometimes the developer. But today the agent observability persona is operations folks, developers, and even PMs and designers. There are many other functions that need to use observability data to really build the system.
For example, if you are building a customer support agent system, and I am not an engineer, I am a PM on that system, I want to know if my system is responding to customer support questions properly, because I know the product requirement and I know how the system should behave. Previously I would not write code as a PM, I do not know how to write code. But today my job is literally that I go to the observability product, I look through some of the dataset collected from the production system, and I know if the system is answering questions in a proper way. As a PM I literally have a role to use evaluation and observability products to improve my system, to build my system. So that is a reverse of build first, observe second. Everyone starts building with observability and evaluation data. That is the biggest shift I have seen.
What do the characteristic production failure modes look like? Things like cascading errors, agent looping, silent drift. How do you catch them before your customers do?
Xiao Ma: Everything you mentioned could happen, and this is where I would add another one, which is cost. If you read industry news you probably already see a lot of news about cost. We did not really think about cost when people first built LLM-based systems. They felt, let us solve the customer pain first, and if cost becomes a problem that would be a nice problem to have. But today it is a problem, and it is a very big problem. It is not a nice problem to have, it is actually a very painful problem, both for internal usage of AI tooling and when we build systems based on LLMs, and the cost is going through the roof.
In order to identify these problems, again this goes back to evaluation and observability, because these are not problems you can just design out of. You have to observe and iterate. A lot of things you can observe from traces. If you have proper instrumentation and an observability product, you can literally see that maybe some agents are just debating for too long, way too long, before they converge, or that there is not enough parallelism to be more effective.
We are also applying AI to identify these things. We call it observability for AI, and AI for observability, and these two sides of the coin are converging very fast. Once you have all these agent traces you can apply AI to understand what the potential problems are. Sometimes they are not converging fast enough. Sometimes they do not have the right context. Sometimes they make wrong tool calls. Once you have enough observing data you can easily identify these anomalies with the power of AI.
So all the things you mentioned are real production failure modes, and the only way to identify them is to have a solid evaluation and observability system up front. That is not an afterthought anymore. That is encoded into your building, your entire development cycle.
Chi Wang: One common type of failure I observe is regression. Often, initially, we do not know whether an agent can do a certain type of task very well, so we try it and we find it works, then we assume it always works. But at some time it suddenly breaks down and you have a hard time finding out why.
I think the reasons are twofold. One is that agents by definition are something we can delegate many different types of things to without worrying about the details. The less you worry about details, the more diverse the type of task they solve, and they need to figure it out by themselves. You cannot predict all the types of problems or difficulties they encounter. So even if they work today, it just means that for the scenarios they ran today it works, but there can be many other situations they could not handle, and you cannot know all of them in front.
The second reason is that an agent system is often a big system that contains many moving parts. The model can change, the tools underneath the model can change, the context can change. So a few working points just means that for that particular combination at around that time, things work. But when things change, and even when all the things do not change, the external world might change. Because of this there are so many moving pieces, so regression is also harder to manage than in traditional software.
Audience questions
What is the biggest engineering challenge you face in production that most people building multi-agent systems do not anticipate?
Chi Wang: I have encountered many such examples myself, and the regression problem I just mentioned is probably one big one. As I mentioned, the challenge is how you make the system not just work once, but always work, moving on with all the dynamic changes.
It is not very hard to give it one particular task and say this is my task I want to solve, and if it cannot, just keep revising until it can. Today’s agents are quite good at that. If you give them a particular task, even if it fails in the beginning, it often can figure out ways to address it. But whether the way they address a problem is general enough to handle all future types of task is not necessarily easy. When we talk about multi-agent systems, it often means that the particular decomposition and the particular way of coordinating them need to account for all the different future situations. I think that is still a very open problem.
Xiao Ma: I would only add to Chi’s point. Chi mentioned regression. I would say even one step forward, the biggest question is, do you even know the first time whether your system works or not in real production? It probably works for demos, but does it actually work in production systems, solving real problems?
For traditional systems, if you pass your test, pass your QA, whatsoever, you roughly know it solves the problem, because it is well defined and it is deterministic. But for multi-agent systems, knowing whether it works in the first place is one thing which most people did not anticipate. And that is where, again, evaluation and observability come in to help.
What would you recommend for observability tooling, and will the GenAI semantic conventions become the standard?
Xiao Ma: Obviously I have a strong bias, and I would recommend Splunk Observability Cloud, which is the flagship Cisco product for observability.
On the GenAI semantic conventions, the answer is certainly yes. We are a big advocate for OpenTelemetry. For people who do not know what OTel means, it is basically an open standard to instrument and collect data for observability, for both traditional systems and agentic systems. As a matter of fact, Splunk is one of the inventors of OTel, six or seven years ago, and we are still the biggest contributor in the open source community. And yes, the GenAI semantic conventions will be the standard.
Most of what these models know comes from English. How do you correct for that bias?
Chi Wang: In general, one good way to correct bias is to use the MassGen type of approach, to not have only one agent solving a problem but to have multiple of them, each with a different kind of bias. The more diverse you make it, the chance for all of them to be wrong is smaller. So in this particular example, if you have models that have a really different kind of bias than the common ones, add them into the system.
Another way is to not only rely on language models but also add tools. How you configure the tools and how you configure the language models are also ways to increase the diversity. When you configure them differently, they can behave very differently.
And then you use a MassGen type of approach, to not trust any single one of them but have the different agents join together to check each other’s answers. It is still non-trivial to find out the correct one out of many different answers, because sometimes the truth is in the minority. If you use traditional voting you may not necessarily get things right. So it is not simple voting. It needs to be some careful reasoning, and not simple majority voting, to reach consensus. Often you do not just simply vote, you think about whether there is a better answer, whether none of the existing answers is good enough, whether you need to iterate based on the existing answers.
When you keep asking an agent to do that, sometimes they will identify their own bias and rethink whether they should do it in a different way. So even when all the answers are wrong, if they are wrong in different ways they realize they need to rethink the problem. As long as you can trigger that, there is hope that you can iterate and eventually get a better answer, even if you could not solve the problem completely.
Chi Wang is the creator of AutoGen, AG2, MassGen and Sutando, and teaches at Stanford, Berkeley, Coursera and DeepLearning.AI.
Xiao Ma leads the teams building Splunk Observability Cloud at Splunk, a Cisco company. Their book Multi-Agent AI Engineering publishes on 24 September 2026.