Prepare for Anthropic’s CCAR-F certification in two days
Join Puria Izady on November 21 and 22 for 10 hours of live preparation for Anthropic’s Claude Certified Architect Foundations exam. Work through all five domains with six runnable labs, a mock exam and a 60-question practice test. Recordings are included
🗓️ Sat 21 – Sun 22 Nov · 40% off Standard passes
✍️ From the editor’s desk
Welcome to the 71st issue of Deep Engineering!
On November 3, the Cloud Native Computing Foundation published Understanding AI Workloads for Kubernetes Scheduling, the first paper in a five-part series. It explains why inference needs careful scaling decisions, since new replicas can spend seconds or minutes loading model weights before they serve traffic. Keeping substantial state in GPU memory adds another constraint when those replicas scale up or down.
Agentic systems make this harder to manage because one user request can trigger several model calls, retrieval steps and tool calls. The paper describes that demand as difficult to predict and says effective scaling needs coordination between autoscalers, cluster schedulers and request routers. Next week’s KubeCon + CloudNativeCon North America also includes a dedicated AI Inference + Agentic track, with production inference firmly on the agenda.
Rohit Bhardwaj, Director of AI and Data Architecture at Salesforce, approaches these problems through lessons from putting more than 100 agents into production. In his Deep Engineering workshop, he brings those lessons together in the ROCS-Loop, with 19 patterns across reliability, observability, cost and sustainability.
We’re sharing an excerpt from the practical deep dive, where you’ll also find the full workshop recording and links to his labs.
Let’s get started.
📚 Secure your multi-agent systems
Secure long-running agents with Ken Huang’s Harness Engineering. This Packt guide covers agent permissions, trusted context, governed memory and recovery, with runnable labs for every chapter.
🧠 Expert Insight
AI Inference at Scale With 19 Patterns for Agentic Systems
If your team says any of these things, there is a gap in how you run inference. “We don’t know why P99 spikes.” “We don’t know what the agents actually did.” “We don’t know what caused the GPU bill.” “We don’t know our carbon footprint per inference.”
A lot of people think creating an agent is just putting it in production. It’s really not. Once an agentic system meets real traffic, we have to take care of four things, which are reliability, observability, cost and sustainability. I call it the ROCS-Loop, and it comes from what I’ve learned putting more than 100 agents into production. It has four loops and 19 patterns, with six for reliability, five for observability, four for cost and four for sustainability.
I’ve put all the workshop materials online, including the labs, checklists and scorecard. You can also run the same simulations yourself with my public ROCS-Loop Reasoning Engine agent.
Green dashboards, then Monday morning
This is the most common modern AI production incident I see. Everything looks fine over the weekend. Then Monday morning traffic hits, and the system melts. P99 latency doubles, a $60,000 bill arrives and a carbon spike is detected. It hits RAG pipelines, multi-agent systems, GPU-backed inference, vector databases and caching layers in every industry.
The weekend hides the problem because caches, vector partitions and GPU pools are all warm, and traffic is light. In my simulation, a weekend request looks like this.
{
“gpu”: {”queue_depth”: 0.05, “batch_window_ms”: 18},
“vector_db”: {”partition”:”hot”,”latency_ms”:5},
“cost”:{”total_usd”:0.00006},
“sustainability”:{”sci”:0.0013,”region”:”us-west1”}
}On Monday morning, the caches have decayed, the GPU pools are cold, the vector partitions are cold, retries cascade and the router picks the wrong model tier.
{
“gpu”: {”queue_depth”: 1.6, “batch_window_ms”: 0},
“vector_db”: {”partition”:”cold”,”latency_ms”:120},
“cost”:{”total_usd”:0.00097},
“sustainability”:{”sci”:0.0081,”region”:”us-east1”},
“tokens_out”: 1100,
“retries”: 2
}GPU queueing and cold vector partitions trigger first. Latency multiplies because cold model loads, retries and the lack of batching compound each other. Cost surges with token drift and large-model misrouting, and the carbon score spikes because the work shifts to peak-hour, dirtier electricity. After the ROCS patterns are in place, the same Monday request looks like this.
{
“gpu”: {”queue_depth”: 0.2, “batch_window_ms”: 24},
“vector_db”: {”partition”:”hot”,”latency_ms”:8},
“cost”:{”total_usd”:0.00014},
“sustainability”:{”sci”:0.0020,”region”:”us-west1”},
“tokens_out”: 350,
“model_tier”:”3B_INT4”
}Calm periods hide structural weaknesses, because warm caches, warm GPUs and low load keep tail behavior from showing up. Without ROCS, teams rely on intuition. With ROCS, they rely on signals.
Inference is a DAG of hidden loops
There is no such thing as “call the LLM.” Take a simple request. I want to buy shoes endorsed by Messi that can reach my home in two days for less than $100. The user thinks they made one call. Behind the scenes, a planner agent interprets the intent and works out that the ZIP code is missing. A retriever rewrites the query and hits the vector database, a re-ranker orders the results, and a small model drafts an answer and escalates to a larger one. Tools look up price and inventory, an evaluator revises the answer, a synthesizer merges everything and a logger stores it. Some of those steps loop. The planner retries, the re-ranker loops twice, and a tool call triggers another vector query.
The user perceives one request. The system executes a DAG of 12 to 15 hops, and each one affects reliability, observability, cost and sustainability. That’s why I say your monitoring shows only about 20% of the system. Planner spans, retry storms, vector partitions, embedding generation, batch jitter, GPU queueing, token drift, agent loops and tool calls all live in the other 80%.
Tail latency does not add, it multiplies
Here’s the physics nobody explains. If the retriever is at its P99, the vector store is at its P99, the LLM is at its P99 and the tool API hits its own P99, your pipeline doesn’t degrade gracefully. It collapses into several seconds of latency. This is why your P50 is perfectly fine but your P99 is catastrophic.
In my tail latency lab, each hop looks reasonable on its own.
Hop P50 P95 P99
----------- --------------------- ------- -------
Planner 12ms 22ms 80ms
Retriever 8ms 18ms 70ms
Vector DB 5ms hot / 120ms cold - 150ms
Reranker 10ms 40ms 120ms
LLM 90ms 300ms 900ms
Tool 12ms 40ms 150ms
Evaluator 20ms 90ms 200msEnd to end, P50 lands around 150 to 200 milliseconds, which seems fine. When the planner’s P99 collides with the vector store’s P99 and the LLM’s P99, end-to-end P99 lands somewhere between two and five seconds.
Most architects suddenly realize they don’t measure P99 per hop, don’t trace critical paths, and don’t know their GPU queueing or vector shard latency. So start there. Measure P95 and P99 per hop, and measure hops per second, because requests per second won’t show you where the tail is exploding.
Two hops deserve special attention. The first is GPU queueing. Most teams monitor GPU utilization, but users experience GPU queueing, and those are completely different realities. A GPU can be at 60% utilization while the queue in front of it is 100% full, and that queue depth is what drives tail latency explosions at P95 and P99. The second is the vector store, which behaves like a cache. A warm partition answers in 4 milliseconds, a cold partition spikes to 80 milliseconds, poor ANN tuning can double or triple latency, and a re-indexing event can stall the whole system for 300 milliseconds. Many pipelines break here silently, because teams don’t monitor partition-level latency.
For capacity planning, I use total allocated GPU-seconds divided by successfully completed business tasks. Then, for each hop, look at the arrival rate, input token rate and output token rate.
Reliability means absorbing the load
The reliability loop exists so that unpredictable input no longer produces unpredictable output. It has six patterns.
R1, async queueing. Put every request in a queue and process the queue. It smooths bursts, enables batching, decouples the user from GPU latency and avoids ad hoc retry storms. If you don’t buffer the traffic, your GPU will pay for it.
R2, backpressure. Use gateway rejections, token caps, agent loop limits and vector query rate limits. We have API gateways for rate limiting, and VIP customers can be tiered so they still get through. Most teams don’t have an LLM gateway yet, but they should, so they can cap total tokens and catch agentic loops where one agent calls billing, billing calls invoicing and the loop keeps going.
Continue reading → all 19 ROCS-Loop patterns, with the workshop recording, lab materials and readiness checklist.
🛠️ Tool of the Week
Agent Router (formerly Envoy AI Gateway), an open source gateway for routing model requests and MCP traffic through Envoy Gateway, with controls for shared inference usage.
Set token limits and backend quotas to control usage across shared models and tenants.
Configure provider fallback when a streaming response times out before its first token.
Route self-hosted requests through InferencePools, with an endpoint picker selecting the model server.
Track token usage and response latency per backend with Prometheus metrics and optional OpenTelemetry tracing.
📎 Tech Briefs
CNCF publishes guidance on scheduling AI jobs - The new report covers gang scheduling, fairness, preemption and reservation for GPU workloads on Kubernetes.
Buoyant open-sources Kinference for vLLM serving - The new control plane provides token rate limiting, per-tenant cost tracking and KV-cache-aware routing.
Supermicro, IBM and NVIDIA launch a shared KV cache tier - Reusing cached context instead of recomputing it cut time-to-first-token up to 56x on 130K-token prompts in reference tests.
OpenKruise Agents 0.6.0 adds sandbox autoscaling - Warm pools now scale on schedules, and idle sandboxes pause automatically.
NVIDIA Triton 2.74.0 improves cancellation handling - Cancelled requests are now caught before they reach the backend.
That’s all for this week. If you take one thing from Rohit, start measuring P99 per hop before your next traffic spike, and work through his readiness checklist with your team.
Stay awesome, Keep Building
Saqib Jan, Editor-in-Chief, Deep Engineering
Partner with Deep Engineering
If your company wants to reach senior developers, software engineers, and technical decision-makers, speak to us about partnering with Deep Engineering