by Rohit Bhardwaj, Director of AI and Data Architecture at Salesforce and founder of ProductiveCloudInnovation.com. This piece is adapted from his ARC 2026 workshop, slides and lab materials, and was written up by the Deep Engineering editors in his own words.
If your team says any of these things, there is a gap in how you run inference. “We don’t know why P99 spikes.” “We don’t know what the agents actually did.” “We don’t know what caused the GPU bill.” “We don’t know our carbon footprint per inference.”
A lot of people think creating an agent is just putting it in production. It’s really not. Once an agentic system meets real traffic, we have to take care of four things, which are reliability, observability, cost and sustainability. I call it the ROCS-Loop, and it comes from what I’ve learned putting more than 100 agents into production. It has four loops and 19 patterns, with six for reliability, five for observability, four for cost and four for sustainability.
All the workshop materials, including the labs, checklists and scorecard, are at tinyurl.com/AIInferenceScale. You can run the same simulations yourself with my public ROCS-Loop Reasoning Engine agent at tinyurl.com/AIInferenceAgent.
As a special provision for Deep Engineering readers, we’re privately sharing the full recording of Rohit’s ARC 2026 workshop below.
Green dashboards, then Monday morning
This is the most common modern AI production incident I see. Everything looks fine over the weekend. Then Monday morning traffic hits, and the system melts. P99 latency doubles, a $60,000 bill arrives and a carbon spike is detected. It hits RAG pipelines, multi-agent systems, GPU-backed inference, vector databases and caching layers in every industry.
The weekend hides the problem because caches, vector partitions and GPU pools are all warm, and traffic is light. In my simulation, a weekend request looks like this.
{
"gpu": {"queue_depth": 0.05, "batch_window_ms": 18},
"vector_db": {"partition":"hot","latency_ms":5},
"cost":{"total_usd":0.00006},
"sustainability":{"sci":0.0013,"region":"us-west1"}
}On Monday morning, the caches have decayed, the GPU pools are cold, the vector partitions are cold, retries cascade and the router picks the wrong model tier.
{
"gpu": {"queue_depth": 1.6, "batch_window_ms": 0},
"vector_db": {"partition":"cold","latency_ms":120},
"cost":{"total_usd":0.00097},
"sustainability":{"sci":0.0081,"region":"us-east1"},
"tokens_out": 1100,
"retries": 2
}GPU queueing and cold vector partitions trigger first. Latency multiplies because cold model loads, retries and the lack of batching compound each other. Cost surges with token drift and large-model misrouting, and the carbon score spikes because the work shifts to peak-hour, dirtier electricity. After the ROCS patterns are in place, the same Monday request looks like this.
{
"gpu": {"queue_depth": 0.2, "batch_window_ms": 24},
"vector_db": {"partition":"hot","latency_ms":8},
"cost":{"total_usd":0.00014},
"sustainability":{"sci":0.0020,"region":"us-west1"},
"tokens_out": 350,
"model_tier":"3B_INT4"
}Calm periods hide structural weaknesses, because warm caches, warm GPUs and low load keep tail behavior from showing up. Without ROCS, teams rely on intuition. With ROCS, they rely on signals.
Inference is a DAG of hidden loops
There is no such thing as “call the LLM.” Take a simple request. I want to buy shoes endorsed by Messi that can reach my home in two days for less than $100. The user thinks they made one call. Behind the scenes, a planner agent interprets the intent and works out that the ZIP code is missing. A retriever rewrites the query and hits the vector database, a re-ranker orders the results, and a small model drafts an answer and escalates to a larger one. Tools look up price and inventory, an evaluator revises the answer, a synthesizer merges everything and a logger stores it. Some of those steps loop. The planner retries, the re-ranker loops twice, and a tool call triggers another vector query.
The user perceives one request. The system executes a DAG of 12 to 15 hops, and each one affects reliability, observability, cost and sustainability. That’s why I say your monitoring shows only about 20% of the system. Planner spans, retry storms, vector partitions, embedding generation, batch jitter, GPU queueing, token drift, agent loops and tool calls all live in the other 80%.
Tail latency does not add, it multiplies
Here’s the physics nobody explains. If the retriever is at its P99, the vector store is at its P99, the LLM is at its P99 and the tool API hits its own P99, your pipeline doesn’t degrade gracefully. It collapses into several seconds of latency. This is why your P50 is perfectly fine but your P99 is catastrophic.
In my tail latency lab, each hop looks reasonable on its own.
Hop P50 P95 P99
----------- --------------------- ------- -------
Planner 12ms 22ms 80ms
Retriever 8ms 18ms 70ms
Vector DB 5ms hot / 120ms cold - 150ms
Reranker 10ms 40ms 120ms
LLM 90ms 300ms 900ms
Tool 12ms 40ms 150ms
Evaluator 20ms 90ms 200msEnd to end, P50 lands around 150 to 200 milliseconds, which seems fine. When the planner’s P99 collides with the vector store’s P99 and the LLM’s P99, end-to-end P99 lands somewhere between two and five seconds.
Most architects suddenly realize they don’t measure P99 per hop, don’t trace critical paths, and don’t know their GPU queueing or vector shard latency. So start there. Measure P95 and P99 per hop, and measure hops per second, because requests per second won’t show you where the tail is exploding.
Two hops deserve special attention. The first is GPU queueing. Most teams monitor GPU utilization, but users experience GPU queueing, and those are completely different realities. A GPU can be at 60% utilization while the queue in front of it is 100% full, and that queue depth is what drives tail latency explosions at P95 and P99. The second is the vector store, which behaves like a cache. A warm partition answers in 4 milliseconds, a cold partition spikes to 80 milliseconds, poor ANN tuning can double or triple latency, and a re-indexing event can stall the whole system for 300 milliseconds. Many pipelines break here silently, because teams don’t monitor partition-level latency.
For capacity planning, I use total allocated GPU-seconds divided by successfully completed business tasks. Then, for each hop, look at the arrival rate, input token rate and output token rate.
Reliability means absorbing the load
The reliability loop exists so that unpredictable input no longer produces unpredictable output. It has six patterns.
R1, async queueing. Put every request in a queue and process the queue. It smooths bursts, enables batching, decouples the user from GPU latency and avoids ad hoc retry storms. If you don’t buffer the traffic, your GPU will pay for it.
R2, backpressure. Use gateway rejections, token caps, agent loop limits and vector query rate limits. We have API gateways for rate limiting, and VIP customers can be tiered so they still get through. Most teams don’t have an LLM gateway yet, but they should, so they can cap total tokens and catch agentic loops where one agent calls billing, billing calls invoicing and the loop keeps going.
R3, GPU pooling. Autoscaling GPUs sounds magical until you need them instantly during a burst. GPUs take minutes to warm up, load weights, allocate memory and connect to the inference tier. During those minutes, retries multiply, GPU queues explode and P99 collapses. Warm GPU pools shared across models give you predictable latency and predictable cost. You don’t want to go into the shower with cold water.
R4, dynamic batching. Batching is reliability’s cheat code. GPUs don’t want tiny, random requests. They want consistent micro-batches every 5 to 20 milliseconds. That fixes P99 without changing your model or your hardware.
R5, model tiering. Not every request needs the most powerful model. Search needs fast recall, product Q&A needs a mid-size model, and complex comparisons need expert-level reasoning. Tiering ensures you don’t burn a 70B model on a 7-token search query. Many tasks need no model at all, since numerical calculations, permission checks and schema validation can be deterministic at close to zero cost. I also route on confidence. Above 85% intent confidence, accept the request. Below that, retry once with a constrained prompt, and if confidence stays low, escalate to a human or another agent. Financial transactions, and anything where lives are involved, always keep a human in the loop.
R6, multi-level caching. Caching isn’t just cost optimization, it’s latency stabilization. Cache at the prompt and intent level, since “shoes endorsed by Messi” and “Messi shoes from Adidas” have the same intent, and cache embeddings, vector searches and final answers too. Without caching, your system fights the load. With caching, your system absorbs the load.
Observability means tracing the prompt
You are not tracing text. You are tracing compute events, including prompt expansion, token generation, retries and template versions. Five patterns cover it.
O1, prompt-level tracing. Ask for “breathable running shoes under $100, compared to my last purchase,” and in the background the planner adds metadata, tools add price, stock and personalization, the retriever fetches more context than needed, and sub-agents revise and regenerate. The prompt grows by 2× or 3× without anyone noticing. Version every prompt template, every model change and every routing decision. Trace the prompt, or you trace nothing.
O2, vector query telemetry. Log latency per partition, so cold partitions show up before Monday does.
O3, GPU telemetry. Utilization is the most misleading GPU metric. Your dashboard says the GPU is at 70% and looks fine. Inside it, the queue is full, batch sizes are all over the place, kernels are slowing down and memory is heavily fragmented. Watch queue depth, batch size variance, kernel execution drift and memory fragmentation.
O4, multi-agent span correlation. Multi-agent systems aren’t linear, they’re DAGs, and your biggest outages happen in the spaces between the components. Give every request a trace ID, a conversation ID, a chain ID for each agent reasoning cycle, a span ID for each hop and a parent span ID for whoever made the call. Then build a span correlation table from them, using tools like OpenTelemetry, Dynatrace or New Relic.
O5, structured prompt logging. Unstructured prompts are unreadable, but structured prompts are analyzable and traceable. When token counts grow, retrieval context drifts or the model escalates from a 7B to a 70B, you see it. That’s how you catch runaway prompts before they bankrupt you.
Cost means knowing why the bill moved
Inference cost is not linear per request. During a Black Friday surge, traffic increased 10% and cost increased between 3× and 9×. It compounds through token drift, retry cascades, cold vector partitions, GPU queue depth, autoscaling lag and retrievers returning far more documents than needed.
Retries are the silent killer, especially in financial systems. One timeout in a risk engine triggers three agent retries, and each retry performs another vector query, another embedding, another LLM call and another risk scoring pass. That’s 3× the tokens, GPU time, cost and carbon, and without observability into retries you will never know. These cascades cause the most expensive AI outages in banking, and they can also distort risk scores.
C1, cost attribution. Without it, the CFO sees only “GPU costs increased 4× this weekend,” but not why. Attribute cost per model, prompt, vector query, agent hop, retry and tier escalation. Track unit economics too, meaning cost per request and cost per successful outcome, because the industry is moving toward outcome-based pricing.
C2, model right-sizing. Distillation shrinks a giant model into fast, specialized students that learn its patterns. Around 80% of e-commerce workloads don’t need deep reasoning, distilled models can cut cost 5× to 10×, and latency can drop from 400 milliseconds to 20 to 80. Quantization moves a model from FP32 to INT8 or INT4, reducing precision while keeping almost all of its intelligence. In my experience it cuts cost 3× to 8× and latency 2× to 5×. INT8 is the safe middle ground, and INT4 unlocks maximum efficiency.
C3, token budgets. Set maximum input tokens, maximum output tokens, retry caps and context window limits. Do only one retry.
C4, spot and preemptible GPUs. Embeddings, batch jobs, index refreshes and simulation batches don’t need on-demand A100s or H100s. Spot GPUs cost 70% to 90% less, and workloads that autosave and resume on interruption can take a $20,000-a-month pipeline down to $3,000 or $4,000. If a job can wait for nighttime, run it at night.
In my workshop simulation, using a large model for everything costs about 6 cents per request, or $6,000 per 100,000 requests. With intent extraction, routing and tiering, the same workload comes out at $237.
Sustainability you can measure
You can’t optimize what you can’t measure. One small retrieval failure in a manufacturing digital twin causes more documents to be fetched, bigger prompts, more sub-agent hops, more GPU time and longer queue occupancy, until jobs spill into a high-carbon region and burn 4× to 10× more carbon.
S1, Software Carbon Intensity. The Green Software Foundation’s SCI score combines the energy a system uses, the carbon intensity of that energy and the hardware’s embodied emissions, per unit of output. Energy times carbon intensity is the part routing and scheduling can change, so that’s what I watch most closely.
S2, carbon-aware scheduling. Schedule non-urgent work at night, route it to low-carbon regions and keep GPU pools warm so you don’t waste energy on cold starts.
S3, hardware efficiency. Large FP32 or FP16 models on older GPUs make energy consumption and SCI skyrocket. Combine INT4 quantization, FP8 precision, newer GPU generations and distillation, and models get smaller, faster, cheaper and far more carbon-efficient.
S4, do less work. Fetch only the three documents you need, limit vector recall to what matters, stop agents from rewriting themselves to death and cache everything possible. Agent-to-agent calls should form a directed acyclic graph, and a cycle in that graph is wasted carbon.
One retry, an idempotency key and clear ownership
People do five retries, and that’s a bad design to begin with. Do only one. If agent one is calling agent two and agent two goes down, back off at one second, two, four and eight, with jitter. Make writes idempotent too. An order purchase should return an idempotency key on the first call, so a repeated call with the same key returns the saved result instead of creating a second order.
Then decide which layer owns retry authority. The agent owns workflow retries, including the maximum attempts and the maximum loop depth, where one loop is fine and a second is not. The gateway owns admission control, meaning rate limits, concurrency, per-tenant queue depth and quotas, deadlines, load shedding and bulkheads. The inference tier owns transport retries. To reduce the blast radius from a noisy tenant, or one attempting prompt injection, use traffic shaping with shuffle sharding.
When the spike outruns autoscaling
New capacity doesn’t arrive right away when queue depth climbs, so don’t rely on autoscaling at first. I use what I call the hurricane pattern, which means supporting only essential services. Checkout and add to cart are essential, and a complex search is not. Cap the pool at around 80% for essential traffic, increase queue depth and worker threads, and set guardrails on P95 queue wait time and the age of the oldest request. Bring nonessential services back once things settle. This is a physics problem, and you can’t solve physics right away.
Degrade without changing the decision
When GPUs run short, preserve correctness and approved behavior first. Sacrifice speed and convenience before you change the decision model. A smaller model is not automatically a safe fallback, because it may be cheaper and faster while changing the accuracy and explainability of the answer. Keep the normal model for normal operation, remove optional work, use bounded queues, define approved and prohibited tasks, and refuse or defer anything you don’t understand. Graceful degradation means reducing functionality while preserving accuracy. Correctness comes first.
Start with the readiness checklist
Repetition is the mother of skill, so everything is ready to go at tinyurl.com/AIInferenceScale. Start with the readiness checklist. Check whether P95 latency is measured per hop, whether queue depth and queue wait time are visible to your GPU scheduler, and whether your retry budgets use exponential backoff and jitter. Then score your system with the ROCS scorecard, assign an owner to every gap, and set service level objectives for each hop.
Then run the labs with the ROCS-Loop Reasoning Engine. Each one simulates a broken pipeline, applies the patterns and shows you the before and after. Here’s the first prompt from the tail latency lab.
Simulate a broken AI inference pipeline to demonstrate Tail Latency
Physics, why P99 multiplies across hops. Use a pipeline with these hops:
Planner, Retriever, Vector DB (hot/cold partitions), Reranker, LLM model,
Tool call, Evaluator. Give each hop different P50, P95, P99 values.
Run this pipeline in broken mode (no R1–R6 patterns). Show a hop-by-hop
latency table, combined end-to-end P50, P95, P99, telemetry logs, a ROCS
reliability failure analysis and a DAG diagram of all hops.If you want a plan, take 30 days. Spend week one on quick reliability and observability wins, weeks two and three on FinOps controls, and week four on your first sustainability improvements. ROCS belongs in every inference architecture review.
Editorial note: This article is adapted from Rohit Bhardwaj’s ARC 2026 workshop, AI Inference at Scale, along with his slides and lab materials. Simulation figures and telemetry samples come from his ROCS-Loop labs.