Retries and autoscaling keep most microservice platforms upright on an ordinary day, but under stress the same mechanisms can feed each other until a small slowdown turns into an outage. Sibasis Padhi, a Staff Software Engineer at Walmart Global Tech who builds agentic microservices for finance applications, is concerned with what happens when an AI agent starts making those calls without any limit on how often it acts or how far its actions spread.
His answer is a discipline he calls Autonomic Reliability Governance, which he first set out in an article for Deep Engineering in July. He is candid that it is still research and not yet part of his current projects, and in this conversation he walks through the local Kubernetes experiments behind it, where agents made better retry decisions once they could see the retry settings of every downstream service instead of only the closest one.
He also explains why an agent has to be told its API cost budget explicitly and handed SLO agreements as structured data, and why audit trails matter most to the engineer who arrives the next morning to work out what the agent did overnight. For financial systems his advice is to start each agent as a recommender and score its suggestions against what analysts actually decide, handing over authority only once that record has earned the confidence of the people who use and operate the system.
Below is the cleaned up transcript of our conversation.
Before we get started, tell us about your background and what you’re working on right now at Walmart.
My name is Sibasis, and I have around 19 years of industry experience. I’ve been building microservices from day one, which I believe was around 2016, and we have been building distributed software ever since. Things have changed since AI arrived in the last couple of years, and now we build agents everywhere. The industry is looking for agents, and for how agents can be built and supported inside microservices to deliver high-volume applications.
I build financial software, finance applications for Walmart, and I work on how we can help finance executives make better decisions when they need immediate action on every decision. That’s where my background comes from. I lead a high-impact engineering team here that is focused on building agentic AI microservices.
Your recent ACM TechTalk focused on preventing retry storms in AI-assisted microservices. Why are retry storms still a serious risk in microservices?
Retries have always been a problem, whether a system is AI-assisted or not, and industry experts like Google and the Kubernetes community have already come out with various options for handling retry storms. The challenge now is that we are bringing AI into every solution. We are asking AI to solve those retry problems for us, but we are leaving it ungoverned, so the retry problem has been amplified since AI came into the picture. That’s why we have to be very careful about how we design our agents and how we govern them. It has become a much more important consideration in our designs.
As I said, every solution today involves AI. Otherwise we are not getting projects and we are not earning client trust. Clients want AI to solve their problems, and they need their applications to stay stable without breaking. AI can take those decisions very fast, so when we leave it to AI to help us solve retry storms, we need to be very careful.
In cloud-native systems, retries and failover work well on a normal day, but under heavy traffic and stress the same tools can work against you. How does an AI-assisted operations tool end up turning a small slowdown into a bigger incident?
Take an example with three microservices talking to each other, where every microservice has its own retries. Microservice A knows about microservice B, and it decides to retry B three times. But A has no idea what B does next. B might be talking to C, D and E, another three microservices that A isn’t aware of. So the calls cascade and everything is linked to everything else, but each of us worries only about our own application and the next one in line, not about what happens outside of that. That’s the major problem today.
It was a problem before as well, but once AI is brought in to solve these retries, the complete picture has to be given to the LLM or the AI agent so it can work out how to act. If A points to B, and B points to another three microservices, the agent for A needs to know what all their retries are, including B’s. Without that information it is going to mess things up completely. I did that experiment, and the models performed better with the complete picture than when they only knew about their closest microservice. That’s one of the ways we can solve this problem.
Many teams add automation expecting it to make their systems more resilient. How can a team tell whether a new layer of automation is actually adding resilience or exposing the system to more risk?
Resilience has always been a problem, and people have been solving it for a long time. The question is how to make it smooth once AI comes into the picture. Say I’m building microservice A, and the first thing I do is give my agent the job of solving my retry problem. We can’t leave that open-ended and just tell it to go and solve the retry problem. It has to be governed.
The agent needs to know the retry settings of all the cascading microservices, B, C, D and so on, so that it knows how many retries each one makes. It also needs the timing, meaning when each retry happens and how long each one takes, because timing is a very important factor in whether your microservices stay resilient. Then every step has to be traceable, which is a kind of auditing. Auditing was always part of the picture, but it has become more important now, because every step my agent takes should be audited, and the agent should have a guardrail.
The last and very important piece is rollback. If the AI takes an action and there is a problem, there has to be a way to roll it back. This is what teams miss, how quickly they can roll back their steps. And you can only roll back easily if your audit steps are clearly measured and kept somewhere.
That’s why I defined the ARG concept in the article I wrote for Deep Engineering. I call it ARG, Autonomic Reliability Governance. These are the steps that have to be followed for AI agents to perform better. The agent has a guardrail and, most importantly, it has a rollback. When you combine all of these, resilience comes into the picture, your microservices perform well, and your clients are happy using the AI.
In your article for us, you describe a shift from self-healing systems to self-governing systems, and you call that discipline Autonomic Reliability Governance, or ARG. What does that shift look like in practical terms for a site reliability engineering team today?
What I have seen in the industry, whether it’s software development or site reliability engineering, is that there is no solution without AI. Even your CTO or your SVPs put you on a path where your project solution has to go through AI. That’s the first step. So site reliability engineering is also moving toward using AI agents to solve its problems.
ARG is a concept I brought in. To be very frank, I have not used it in my current projects yet. It comes from my research, which I have been publishing in my articles, and industry leaders have commented that it is one of the solutions that can be put into site reliability engineering.
I did the experiment on my local laptop with a Kubernetes configuration and multiple microservices talking to each other. Take the same retry storm as an example. I created a burst in microservice one and watched how that burst cascaded to all the other microservices. My focus was on microservice A calling B, but I had no idea what all the other microservices were doing. So how can I trace each microservice’s audit log? I should know whether microservices B and C are using AI agents. If they are, how can I trace them? If they aren’t, they will have their own logging system where they keep their audit records. Then the question is how all that information can be provided to my microservice’s agent so that it knows what is going on.
What I found is that it works if you give the agents complete audit logs and you have a solid rollback mechanism. Here is what I mean by rollback. Say a burst of input requests comes in and your systems are slowing down, and I give my AI agent the job of correcting all these cascading failures. If it goes and tries to correct everybody else’s microservices at once, it is going to cause a bigger failure. So it should take small steps to correct each issue rather than correcting all the microservices in one go.
Coming to financial microservices, we should not give our AI agents complete authority to take action. The agent has to be tested first as a recommendation engine. It’s the same in site reliability. As a hypothetical example, if my CPU is at 70%, the agent restarts the service. If I give that rule to my agent, it will do it. But is that the right way to start? In the first phase of a release, I should not give my agent the authority to restart a service when CPU hits 70%. It should give a recommendation instead, saying CPU is at 70%, and send it to the site reliability engineer, who checks it and makes the decision. That should be the first phase of an AI implementation. Once we have that confidence, I open it up for the agent to take the action itself.
So from a site reliability point of view, and from an ARG point of view, take small steps. Start with recommendations and let a human take the action. Once you have all the audit features in place and you have the confidence, let the AI agent take automatic action.
Two of the controls you describe are actuation budgets, which cap how often an agent can take high-impact actions, and blast radius limits, which cap how far any single action reaches. Realistically, as you manage your systems and your engineering teams, how would a team work out where to set those limits?
One important point I missed earlier is budgets. We often forget about them. In a retry storm, for example, if there is a burst of API calls, each call has a budget. When an engineer writes an API or a retry mechanism, we never think about a budget. It becomes an important feature once AI is in the picture, because AI is non-deterministic. We just ask the agent to solve a problem and leave it to the agent. It might try ten API calls to solve one problem, and it never knows that each API call has a budget. There is a network cost, an API cost, and the cost of parsing whatever JSON you send over the network. All of it is cloud cost, and that is cost to the company.
When we give AI a completely free hand to go and solve a problem, and it calls your APIs ten times, that’s where your costs go up. So how do we control that? AI doesn’t know your budget by itself. We have to feed it in and tell the agent what the budget is and that it should not exceed it. Half of our engineering teams and developers never think about what is happening underneath an API call, what the network costs are, how much traffic we are using, how much data we are sending over the network. It is all cost to the cloud, so it has to be considered in the design. And it has become more important now that AI is in the picture, because we are paying for AI as well as for the cloud, and both costs are going to double if we don’t keep this in mind.
That ties closely to FinOps. Another control you mention is SLO-aware decision boundaries. For our audience, an SLO is the reliability target a service commits to, and an error budget is how much failure a team can absorb before it has to slow down. How should an agent use observability data to decide when to stop acting?
As you rightly said, when to stop is one of the most important features of an AI agent, along with when to roll back. Those are the two things we should always consider whenever we design an application that uses an AI agent. When to stop depends on what the application is trying to do. When we design a system today, we think only about our own application. I call an API, I get my response, my work is done, and we never look beyond that at which system we are calling and what its SLO agreement is.
We already have SLO agreements everywhere. Service one calls two, three and four, and everywhere there is an agreement and an SLO. We still have the problem, and that problem gets amplified with AI. We think everything is deterministic and our SLOs are deterministic, but when AI comes into the picture, how do we design it so that we still stay within the SLO?
That’s where ARG, what I proposed, is one of the important pieces. There is a tabular format, or you can simply think of it as an agreement defined in a JSON structure, where you define all your steps. What is my SLO, what is my target between microservices A and B, and how much time should each step take. If you put that complete structure in JSON and give your AI agents access to it, they go through that path every time they route a request to another microservice. That makes it deterministic for the AI. The more deterministic input the AI uses, the better the solution and the output from the model, and we remain within that guardrail.
Let’s talk about adopting this in FinTech. Financial workflows raise the stakes, since sending a payment request twice can mean charging a customer twice unless the operation is idempotent, meaning running it a second time has no extra effect. What does safe adoption of agentic operations look like for FinTech teams specifically?
That’s a very interesting and important question. I come from the financial domain and build microservices for finance operations and applications, and the most important thing in finance is that we are dealing with numbers. Even a dollar here and there at the lower level, where we build the application, turns into millions or billions once the problem reaches the finance calculations at the upper level. And there is pressure to use AI to solve all our problems underneath. Our applications should use AI, and forecasting and planning applications should use AI to produce better forecasts. On top of that, we are building agents.
So here are a couple of inputs for anyone building financial applications with AI. The first is not to allow your AI agents to do everything in the first place. If you have a planning and forecasting application and you are forecasting a number, let your AI agents recommend first rather than take an action. Use AI for all your analysis and calculations, then have it make a recommendation. Your financial analyst comes in, reviews it and acts on it, and then you record what the analyst did. Over time you build a list of recommendations alongside what your financial manager or analyst did with each one, whether they approved it or requested a change. You are auditing and capturing all of that data in your systems.
After some time, once your confidence level increases, allow your agent to take that action on behalf of the analyst. My point is never to give your agent full authority to take action, especially in financial services. Always have a human in the loop, but slowly reduce that intervention as you move along and your confidence in your agentic solution grows.
Last but not least is rollback capability. Whether your AI agent is approving or rejecting a transaction, always have rollback in place. You can ask your agent to roll back, so you don’t need a person to go and do it. If your agent can approve or reject a financial transaction, it should also be able to roll it back. All of these things have to be carefully considered when you design your applications, and that’s how your AI agents will learn, improve and take appropriate action in the future.
In financial systems, audit trails and rollback plans often exist because compliance requires them. How do they help the engineer handling an incident?
That’s a very important point. We are giving our AI agents complete authority to solve problems so that they don’t disturb us while we are asleep or outside working hours. Say there is an incident. In one case, my AI agent takes an action and solves the problem. In the second case, it solves the problem, but not correctly, or in a different way. When my engineer comes in the next day and checks what happened, how do they know? Without an audit trail, that is not going to help anyone. The future is AI doing all our incident management and problem solving, but every step it takes needs to be checked, and you can’t do that without a complete audit trail.
We have various tools for that, observability tools like OpenObserve or OpenSearch, and they have to be integrated into our systems. Otherwise it is going to be very difficult for an engineer to figure out what the AI did yesterday or the day before during an incident. And I can’t go back and roll back a problem that has happened, even if we have an AI agent with a rollback feature. A person still needs those audit trails to go and check what just happened. Those logs are also very useful for training agents in the future so they can make better decisions. That’s why I would say rollback and audit trails are very important in Autonomic Reliability Governance.
Most teams start with advisory agents, which suggest a fix and wait for an engineer to approve it. You describe progressive autonomy, where agents earn more authority in steps. What evidence should move an agent from advising to acting, and who should make that call?
That’s a very good question. The move from advising to acting has to come from confidence, the confidence of your engineers, your leadership and your users or clients. All three have to have that confidence before an AI agent moves from advisory to acting. So how does that confidence come? When my agent takes an action, I write it to my audit trail. When users take recommendations, I have all those details too. My agent’s actions are documented, my users’ actions are documented, and all of it is available to the engineer.
To track that confidence from beginning to end, we need to give it a score, a bit like reinforcement learning. I have to give a score to my audit logs, to my user recommendations, and to my engineers’ and clients’ decisions. Let me take a simple example. My agent’s actions are all in the audit log, and it gives a recommendation. My client comes and accepts that recommendation, and I give it a score of 90. Tomorrow something else happens, and my client doesn’t give it a good rating, so I give it 50. I keep tracking all those actions over a quarter, for example, or whatever period the client and the engineering team define.
Once I reach that date, after monitoring all these recommendations with their numbers for the quarter, I come up with a score. Say my confidence level is 95 now, and it was 80 last quarter. That’s how we can track our progress. Once all three parties, your users, your engineers and your leadership, are confident, that’s when you decide to move from advisory to acting. But even when you are 100% confident, I always suggest keeping a human in the loop. That’s how, someday, perhaps a year from now, we can give agents full authority to take action for us.
Last question. You’ve said agentic operations will only become useful in enterprise systems when they are bounded, auditable and reversible. You meet a lot of engineering leaders at conferences. What do most of them still struggle with?
When I talk to leaders at conferences, or wherever I meet them in my company, the first ask is speed. How fast are we delivering our applications to the client? The second is to get close to the customer, develop your app and deliver it as soon as possible. Those are the two asks that keep coming from leaders, because AI is helping us build applications fast. What they are saying is use AI, go to your customers, sit beside them, develop the solutions and deploy them to production. Everyone has that ask, but hardly anyone talks about governance.
I believe governance is going to be the problem in the future, because now anyone can build an application within minutes using coding agents or AI models. I have sometimes seen, even with financial applications, people directly download data and upload it to a model to solve a problem. There is no security or governance in between, because the clients and users who are now building applications don’t have the knowledge that this is insecure, that they shouldn’t upload financial data to a model, and that there should be guardrails and security in between.
That’s the missing piece, governance, with guardrails put in place by senior leadership. It has to come from the top level of a company, whether it’s a startup or any other company. Otherwise we are going to expose more and more sensitive data to AI models, and we might face problems in the future. So governance is something leaders, engineers and even clients need to start thinking about now.
