I burned 51 million tokens on one merge conflict, and the model was not the problem
Cache the prefix, compact the context, route to the smallest model that fits, and put a ceiling on every loop
By Rory Preddy, AI Advocate, Developer Relations at Microsoft and GitHub. Creator of LangChain4j for Beginners. | Edited by Saqib Jan
I have been in IT for 27 years, and 23 of those as a programmer. I started as a Java developer. I moved to Microsoft seven years and eight months ago, and I have been a developer advocate for five and a half of those, first in cloud advocacy and now in AI.
For most of that time my job was to tell people to go and build. Lately I have started saying something else first. Take a step back. Not because the excitement is wrong, but because I keep meeting developers who ran out of tokens and cannot tell me where they went. I ask what happened and I hear the same answer. I used the top model, I sent it away, and it ran out. I ask whether they needed the best model for that job and they say they do not know.
That is not a model problem. That is a foundation problem, and it is fixable in an afternoon.
So let’s dig deeper into what tokens actually cost you, the two mechanisms that cut the bill the most, the patterns that keep the expensive model out of cheap work, and the setup you do before any of it.
This deep dive is adapted from Rory Preddy’s Deep Engineering Live session on token efficiency, edited from the session transcript for length and clarity. Slides from the talk are here, and the full recording is here.
Four levers, and one of them gets ignored
You have four things to pull, and they move independently.
Model choice sets the unit price on every token in the exchange. Prompt structure sets how many input tokens you send and whether any of them qualify for a discount. Cache-key stability decides whether a repeated block earns the cached rate or gets billed again at full price. And output limits cap the completion, which is the one people skip.
Output is where I see the least attention and some of the highest prices. On the GPT-5.6 family the flagship Sol tier runs 5 dollars per million input tokens against 30 dollars per million output. Luna, the small one, is 1 dollar and 6 dollars. Look at the gap between those two numbers on the same row before you tune anything else. Watch your outputs, not just your inputs.
One more thing before the mechanisms. You do not have to guess at any of these rates. Azure publishes live pricing at prices.azure.com/api/retail/prices and you can query it. It looks confusing the first time you open it, because a single meter name folds the model family, the tier, the processing mode and the billing unit into one string, and there is a region attached to all of it. So let’s all breathe. Two things in there matter. The same model bills differently in different regions, and batch processing is its own meter rather than a discount on the standard rate.
It is not complicated. It is just not written down anywhere you were looking.
Caching, and why you only pay for the tail
Ask a model to tell you a joke. Now ask it again, exactly the same way. You are not going to be charged the same for the second one, or if you are, it will be a very small charge.
Now scale that up. Say I am an insurance company and I need to send millions of customers their latest statement. Every one of those requests carries the same instructions, the same reference material, the same everything, and then right at the bottom there is a name and a dollar amount that changes. That big identical block at the front is the stable prefix. The little bit at the end that changes is the tail.
You do not pay full price for the prefix. You pay for the tail.
The first call is a miss. The model stores the prefix and bills you the full input at standard rate. Every call after that, sent with the same prefix and the same cache key, is a hit, and that prefix gets billed as cheaper cached input. In my demo the prefix carried around 120 identical reference sections behind a run-scoped key, and warming the cache before the measured calls saved 99 percent on the repeated path.
Here is the part that catches people. The saving depends on those bytes being identical. Put a timestamp near the top of your prompt, or a request ID, or a freshly serialised list of tools, and you have moved the boundary and lost the discount on every single call. It will look completely harmless in review. So think ahead with the prompt structure. Decide what is stable and put it first, on purpose, before you need it.
Compaction, so the context stops growing
Caching handles repetition between calls. Compaction handles what piles up inside one conversation.
Every time you talk to an agent it remembers the conversation, and it sends that history again on the next turn. So the tokens you are billed per turn climb, and climb, and climb, even when the actual work per turn has not changed at all. Model a twelve-turn session and you are sending around 1,920 context tokens per turn by the end. Compact it along the way and it resets near 795 and never goes above that band. That projection is illustrative, but the summary reduction underneath it is measured, 62 percent, from 795 tokens down to 302.
What makes a compaction prompt work is being specific about what survives. Mine asks for at most six sentences, no bullets, no nested lists. It keeps only what the next turn needs, so the goal, the key facts, the files to update, the validation and deploy commands, the blockers and any privacy constraints. It throws out repetition, resolved dead ends, greetings, transient logs, and exact values that do not matter any more, and it turns quota findings into sanitised evidence instead of repeating every number. On a working-notes example that took a prompt from 409 tokens down to 233.
I also tried the other direction, squeezing the output instead of the context. I have a prompt that tells the model to answer like a caveman. Why use many token when few do trick. Drop the filler words and the articles, keep every technical fact, every command, every file path and every error string exact. Brain still big, mouth small.
It works, and the output count came down against a roughly 600-token baseline. But I will be honest with you about the trade. The caveman answer lost some of the understanding the fuller answer had. It is not as detailed. If your reader has to come back and ask a follow-up question, you have paid for that saving twice. A hard cap on output length is blunter but more predictable, because it truncates rather than condenses, so use it where the shape of the answer is already known.
Set up the project before you write anything
This is the least exciting part and it decides most of your cost.
Three files. A devcontainer.json, so the project runs on everyone’s machine and not just yours. An instructions file, so the agent has standing context and is not working blind. And an agent profile, which is the one people leave out.
The agent profile declares which tools the agent can reach for, and mine only has the tools it needs to function. That matters more than it sounds, because every tool you register ships its schema into the request on every single call. Leave a pile of MCP tools switched on that this task never touches and you are paying to send that JSON back and forth all day. Drop the ones you are not using.
Then three habits alongside it. Scope your sessions tightly. Keep your prompts precise and short. And if a direct completion will do the job, use a direct completion instead of turning an agent loose on it.
I build these three files with the cheapest model I have. Luna, reasoning switched off. It costs me almost nothing to create the baseline, and I would rather learn to do it properly at that price. On a cost-tips run I will say it out loud as I go. I am not using reasoning. I do not need reasoning right here. That is the right tool for the job.
What an agent actually is
An agent has agency. Agency is self determination, which means the agent understands there is a task it has to achieve. You can only give it that task if you understand it yourself.
What an agent is not is a large language model that you tell to go in and do whatever it wants. That is madness.
Here is the version I trust. I have a supervisor agent that transfers 100 dollars from one person to another and converts it to euros. Underneath it there is a withdraw agent, a credit agent and an exchange agent. Each one has a small scope. It runs on GPT-4o, so it costs me cents per month. It runs on Azure with managed identity, so it is locked down. I created the project with the dev container and the agent profile before any of it ran.
That is what I consider an agent. There is a plan, you built it, you locked it down, and it performs one well-contextualised task properly.
Eight patterns that keep the big model out of cheap work
Once the foundation is there, the patterns are about one idea. Do not let the largest model see work it does not need to see. My tiers are gpt-5.6-luna small, gpt-5.6-terra medium and gpt-5.6-sol large, and the numbers below are projections from my own demo harness, not benchmarks.
A router puts a cheap classifier in front and sends the request down one specialist path. 60 to 80 percent. Triage is similar but the gate is deterministic and costs zero tokens, and it only escalates when the complexity earns it. 50 to 70 percent. Context compression has Luna condense a long transcript so Sol never sees the whole thing. 30 to 60 percent. Retrieval pulls the chunks that matter instead of the whole corpus. 40 to 60 percent.
Tool use moves exact computation out of probabilistic generation and into code, and it goes hand in hand with keeping that tool list short. 30 to 50 percent. Step-back planning resolves the frame before the expensive execution starts, and it only pays when it prevents retries and rework. 20 to 40 percent. Caching covers the repeated path, 50 to 90 percent on a hit. And batching I will not oversell, because a parallel mapper lowers your wall-clock time and saves you no tokens at all.
Retrieval deserves more than one line, because everyone reaches for it and plenty of people misconfigure it. The query becomes an embedding, the embedding drives a vector search, the search returns chunks, and the model answers from those chunks alone. So your chunking and your embedding model decide your answer quality and your token count at the same time. The failure I see is sending everything back every time, because that is easier than tuning retrieval, and then the customer asks why it is slow and you ask why it is expensive. My retrieval demo projected 67 percent against sending the whole document. All you need is the dollar amount, not the entire insurance quote.
And where I got it wrong
I burnt 51 million tokens recently, and I used the wrong model for a specific purpose.
I was resolving a merge conflict, and I gave it to Luna. Luna could not solve the problem. That on its own would have been a cheap mistake. What made it expensive is that I had not set a loop to say, after a while, stop. So it kept going.
Two lessons, and they pull against each other slightly. Match the model to the bottleneck rather than always reaching down, because a cheap model that cannot finish costs more than an expensive model that can. And put an exit condition on every iterative agent, every time.
I have a refinement loop that writes a story and then reviews its own work. It caps at five iterations and accepts a score around 80 percent. It is not going to be perfect. Sometimes you really need to say, I do not need it perfect. Caching and compaction are both good, and you still want a hard stop when it hits the score you asked for.
Build an agent that watches the bill
You can point agents at your own costs, which is my favourite version of this.
I have a token cost runner. Its job is to execute real flows against a deployed Foundry model, read the actual token usage, and compare it against live pricing. So the number comes out of a run rather than out of arithmetic on a rate card. I ask it to find me the lowest-cost path for a prompt with a cached prefix and a changing tail, and it tells me.
Build agents to help you lower your cost for agents. Think of it as a pyramid. What you want to achieve sits at the top, and underneath it there is a foundation holding your dev container, your instructions, your agent profiles, your compaction and your compression. Everything above the foundation inherits whatever the foundation does.
Two commands, if you want the short version
If patterns are more than you want right now, there are two commands.
The first reads your session history and tells you where the money went. Mine was not flattering. Across sixty days my input tokens outweighed my output roughly 100 to 1. Opus 4.8 took 45.8 million input tokens at an average of 87,000 per event. One single session accounted for 14.4 million on its own. And one session climbed from 49,000 to 174,000 input tokens over 66 turns with no manual compaction at all, so auto-compaction only fired at turn 67, right at the end. Every turn past about turn 30 was resending well over 120,000 tokens on a premium model. I was the one not compacting long sessions.
The second sets a budget per session, warns you at a threshold and stops you at the limit. Mine stops after 34.7 of 30 credits and asks whether I want to add credits, raise the limit or remove it.
I also keep a canvas built from that same session data as a daily token pulse, and I refresh it across seven and thirty day windows. None of this optimises anything by itself. It is just awareness. But a number nobody looks at twice will never change how anyone works, and most of the waste I see is not a decision anyone made. It is a thing nobody checked.
Pick the harness you already pay for
People ask me which harness balances capability against cost. My honest answer is that what you are trying to achieve settles that before cost does.
Then I will tell you the best harness to save costs is GitHub Copilot, and you should hear that knowing I work on the Copilot side. My reasoning stands on its own though. If you have a Copilot licence you already have the spend, and Copilot is an agent, not just an editor feature. There is a Copilot SDK and there is the Microsoft Agent Framework, and you route from there. Send the task to Copilot, or to Claude Code, or to a local model, based on the work. It is the same routing decision as picking a model, one level up.
Take a step back
There is a whole industry forming around this. I watched it happen with cloud, where engineers who wrote good scripts became cloud engineers, and then cost optimisation became its own discipline. I used to sit with CTOs of banks who told me their biggest worry was how much they were going to spend. Then it changed, and the question became how much they were going to save. The same thing is happening now with tokens, and the Linux Foundation’s Tokenomics Foundation is going to matter here, because standards and dashboards and protection mechanisms are what turn this from a habit into a practice.
So do not go berserk. Create the foundation. Try Luna, switch reasoning off, and see how far it gets you. If it does not match your needs, add a router and a design pattern. Start small and build up. Rather than spend more, spend less.
And one last thing, because I have another whole talk about this. Do not end each day exhausted, not from the work itself, but from managing of the work. Let the adrenaline flow, do the vibe coding, do it in the correct way. Know what you are doing. Be kind to yourself.
Now go and look at your own session history.
Session notes:
Rory’s demos run on LangChain4j’s agentic modules, which matters for teams on the JVM because most token-efficiency tooling ships Python first. His pattern code is at github.com/roryp/token-design-patterns, the agentic pattern showcase behind the banking demo at aka.ms/agentpatterns, the live cost demo at aka.ms/costs, and his LangChain4j course at aka.ms/LangChain4j-for-Beginners.
This deep dive is adapted from Rory’s Deep Engineering Live session on The Future of Software Development. Here are the slides from the talk, and the full recording.
Connect with Rory Preddy on LinkedIn: https://za.linkedin.com/in/rorypreddy