Deep Engineering Specials: Judgment, Not Tokenmaxxing, Creates the Value
The next limit on engineering output is not how many tokens your teams burn, but whether anyone can still tell motion apart from judgment.
The next enterprise AI bottleneck is not model capability. It is whether leaders can tell the difference between a team using AI well and a team running up a number.
Over roughly a quarter in 2026, several of the largest engineering organizations in the world took down the dashboards they had built to prove their AI investments were working. Amazon shut down KiroRank, the internal leaderboard that ranked developers by how many tokens they consumed on its Kiro platform. Meta also dismantled a near-identical board, Claudenomics, that tracked token usage among its heaviest AI users. The reason was the same everywhere. Once usage became the number that mattered, engineers found ways to run up usage, and the costs arrived long before the value did.
Amazon’s own senior vice president, Dave Treadwell, told staff to stop using AI for its own sake after employees gamed the leaderboard by padding token usage, the practice the industry had taken to calling “tokenmaxxing.” Uber also said it could find no clear link between what it was spending on AI and what it was shipping, while Microsoft cancelled a division’s coding-assistant licences over cost. So, within a single quarter, the industry ran the experiment, watched it fail, and started hunting for a better question to ask.
That question is the subject of this issue, with insights from Austin Parker, Director of AI Strategy at Honeycomb and a co-founder of OpenTelemetry; Tom Howe, Director of Solutions Engineering at Hydrolix; Jayeeta Putatunda, Director of the AI Center of Excellence at Fitch Ratings; Satyam Dhar, Staff Software Engineer at Galileo and a former engineering leader at Amazon and Adobe; and Michael J.J. Tiffany, co-founder and CEO of Fulcra Dynamics.
Let’s get started.
This NINJIO Insights Report dives into the key emotional susceptibilities that make social engineering work and offers concrete steps that your security team can take to equip your workforce to resist cyberattacks.
Special issue — July 2026
A token was never a unit of intelligence
“You can use tokens to help build solutions, but at the end of the day these are judgment calls.” — Austin Parker, Director of AI Strategy at Honeycomb and a co-founder of OpenTelemetry
Give a modern coding model the right harness and a clear goal and it will do almost anything you ask, up to and including rewriting an entire codebase in another language and passing the tests. That capability is real, and it is seductive. It makes it easy to believe that all you need is something to keep the tokens flowing and any problem becomes tractable. Parker has observed his own engineering teams test that belief, generating a hundred variations of the same screen or twenty versions of a flow, and the lesson came back the same each time.
The variations did not move the needle. You can ask for infinite variations of a solution, and they all arrive shaped like your original conception of the problem, because that conception is the one human input the model never questions. The code and the screen and the workflow were never the things that create value for the people using them. Value lives one level up, in whether the design serves those people, whether the shape of an API fits what its consumers actually need, whether the underlying primitives have any fitness for the purpose they are being asked to serve.
Those are judgment calls, and no volume of tokens answers them. Parker sees the same pattern when he talks to other leaders, plenty of appetite for here is a problem, generate all the variations, and far less time asking whether the problem was framed correctly in the first place. His conclusion is blunt. Judgment is the finite resource now, and teams that forget it end up with more output and less of what the output was supposed to buy them.
Leaderboards reward motion, not judgment
“The paradigm in some organizations will, undoubtedly, shift from ‘high token usage equals good employees’ to ‘high token usage equals expensive employees.’” — Tom Howe, Director of Solutions Engineering at Hydrolix
The collapse of the token leaderboards surprised no one who has watched a measure become a target. Hand a famously clever group of engineers a resource and grade them on how much of it they use, and they will use it. The moment a measure becomes a metric it stops being a good measure, a version of Goodhart’s law that Amazon’s own leadership named out loud when it wound the program down. The surprise was not that the leaderboards were gamed. It was that so few people in the decision chain saw it coming.
Howe has seen the mechanism play out from the inside, and he traces it to a subtle slippage. A company rolls out what it calls an AI usage dashboard, meant as an objective measure of how much the tooling is benefiting the business, and it quietly devolves into a leaderboard that measures how much each person is using the tool. Once employees read it as a judgment of how they work rather than of whether the tool earns its keep, the whole exercise changes character.
From there the responses split three ways, and each one erodes the thing the dashboard was supposed to protect. Some engineers game it, firing tools at low-value tasks to boost their stats, which breeds suspicion that others are gaming it too. Some avoid the tool entirely rather than expose their habits to uncontextualized data. And in the middle sits the casualty Howe cares about most, the trust that a healthy organization runs on, undermined the moment a team cannot tell how or why it is being measured.
The cost frame makes it worse. Howe describes enterprises treating tokens as close-to-free productivity, funny money, right up until the real bills and the real gains come into focus. When they do, the incentive can flip hard, from prizing heavy usage to flagging it as expense, and engineers start hedging, wary of being branded token wasters and wary of binding their workflows so tightly to a tool that a future rationing decision leaves them stranded. He points to a recent, unusually capable model that carries a known high future cost, where engineers are trying to use its power now without becoming dependent on it, as exactly the kind of bind these metrics create.
None of this means the tools do not work. Howe shares that Hydrolix has seen a revolutionary impact from how it uses them, in building products and in weaving them into products, in ways that would have been unimaginable a few years ago. The challenge is balancing productivity, cost, and trust, and learning to measure the impact through experience rather than through a scoreboard. He tells one story that captures the slope precisely, an engineer glancing at the usage board and joking that not cracking the top fifty was rookie numbers. It was a joke. But it was also a warning about how fast an innocuous metric turns into a competition.
Featured - ARC 2026: Software Architecture in the Age of AI
AI is reshaping software architecture, putting new demands on scalability, governance, reliability, and observability. ARC 2026 brings together architects, CTOs, and AI practitioners for keynotes, panels, and workshops on agentic system design, modernizing enterprise apps for AI, and building governable, observable AI systems.
🗓️ 25 to 26 July, 10:30 am ET
Use code DEEPENG50 for 50% off the early bird price.
Skills that matter never show up on a dashboard
Parker calls prompting, building internal alignment, and holding a complex system in your head illegible activities, and they are close cousins of the glue work that keeps organizations moving. The people who know who needs to be in a decision chain, who can hear a customer complaint and route it to the one team that can fix it, who understand that the thing frustrating you in one corner of a product is caused by something three systems away, are the reason a business can actually execute. The organizations that are genuinely good at this tend to be the ones that, lacking a clean way to measure those people, are at least careful not to penalize them.
A token metric makes that harder. It rewards the visible act of consuming AI over the invisible work of knowing what is worth building, and the invisible work is where most of the value hides. That work shows up in promo packets and career ladders because those are written by people who understand it. It does not show up in a number you can compare quarter to quarter, because, in Parker’s words, it involves “very human things like relationships and time in the system rather than timing the system.” Measure the activity and you slowly stop rewarding the thing that made the activity worth anything.
AI widens the gap a team already has
This is the reframing every leader watching one team pull ahead should steal. AI rarely conjures capability from nothing. It accentuates differences that already exist. When a team surges, the real story is almost never that raw intelligence solved a problem they could not solve before. It is that AI patched a hole in the team’s composition, or handed them leverage on something organizational rather than technical.
Parker’s example is the part of shipping that engineers tend to undervalue, making the case to leadership. That is usually a data-analysis argument, projections and forecasts and a slog through support tickets and sentiment data, and it happens to be something AI is very good at. A motivated engineer or designer or support person can now do the analysis a product manager used to own, cover the weak spots in their own toolkit, and come back to the org with here is what I need in order to go build the thing I am convinced is right. So the question to ask the team that pulled ahead is not which tool they used. In Parker’s words, “I never want to ask what AI has let you do. The question is what is AI giving you the excuse to do that you could not have done otherwise.”
That does not extend into a ban on measuring code output. Parker’s line is that you should measure it and refuse to make it a metric. Watching the balance of someone’s pull requests opened against reviewed tells you something real about how they work, and grading their performance on that same count is where it curdles. AI can even help here, letting a team ask which work is generating on-call burden, which is delivering customer value, and who is quietly doing the refactoring that lifts everything built on top of it.
Point the budget inward, at developer experience
“If I had 90 days of AI budget to improve developer experience, I wouldn’t spend it evaluating another model. I’d spend it reducing engineering variation.” — Jayeeta Putatunda, Director of the AI Center of Excellence, Fitch Ratings
For a Staff or Principal engineer who owns a platform, Parker’s first ninety days are not about output at all. They are about earning the ability to ask real questions of your own systems, and most organizations cannot, because observability was underinvested for years while other goals took priority. The mechanical work of fixing that, moving unstructured logging to structured events and spans, migrating to OpenTelemetry, is exactly the kind of task that is fungible from an intelligence point of view, which makes it a near-perfect fit for current coding models. It also comes with a natural verification step, feed the before and after to another model and let it judge whether the telemetry got better, and let it update the docs as it goes.
Putatunda took the same budget in a different direction, and her experience widens the point. Once her organization distributed AI broadly, it became clear that model capability was never the bottleneck, because every team had the same models. What differed was how teams solved the same problems, each with its own coding conventions, documentation norms, migration methods, and review expectations. So her first priority was not a better model. It was making the organization’s engineering knowledge reusable.
Her team translated coding norms, implementation templates, documentation standards, review criteria, and migration playbooks into reusable skills that work from any agentic environment, whether an engineer reaches for Claude Code, GitHub Copilot, or something else. That let them clear backlogs that had always lost the prioritization fight against feature work. The migration of a legacy Selenium test suite to Playwright, a slow and redundant manual slog across many teams, got done by pairing reusable migration skills with the organization’s own standards and validation steps, so the generated tests matched both the suite and the reviewers’ expectations. The same approach carried into repository documentation, repetitive refactoring, and framework upgrades, the modernization work where consistency matters more than originality.
The deeper reason it worked speaks straight to the review bottleneck that AI creates. Models generate code faster than teams can review it, and when every change arrives in a different shape, every pull request costs the reviewer more attention. Standardizing the patterns let reviewers focus on correctness and business logic instead of style and structure. Putatunda does not measure any of this in tokens or lines generated. She measures whether projects moved faster, whether patterns got reused across teams instead of reinvented, and whether reviewers stopped correcting predictable issues, which is another way of saying she measures the same thing Parker does, whether the team can now do work it could not do reliably before.
Spend on leverage, not on tokens
“Just as we don’t judge an IDE by how many keystrokes it saves, we shouldn’t judge AI by how many tokens it consumed.” — Satyam Dhar, Staff Software Engineer at Galileo and a former engineering leader at Amazon and Adobe
Local inference and model routing are making cost-per-token murkier by the month, and Parker is honest that he does not have a clean formula to replace it. What he has is better than a formula. It is an example. For most of Honeycomb’s life, dark mode was the feature customers asked for and never got, enough of a running joke that it has its own face on the twenty-sided die new hires receive. Then, in about six months, the company shipped it, mostly because of AI.
The caveat matters as much as the result. It was not only AI. Getting there took years of groundwork, a design system and an accessible component library built to support the feature. What AI closed was the mechanical gap that always stalled the project, migrating the existing app onto the new components. The way they did it is the real lesson. Instead of one team grinding through the migration, they shipped prompts and skills so anyone could run it, and pulled in an engineering director who had built a screen seven years earlier to load the skill and update his own page. Yes, it burned a lot of tokens, and a small focused team could have optimized that count down. The accounting that matters is a five-year customer request finally closed and a company full of people who learned how to use AI.
Dhar puts a name to the mistake underneath the token frame. Too many teams treat AI like another cloud bill, counting tokens because tokens are easy to count, then assuming fewer tokens means better spending. In his experience running large-scale AI systems, some of the most valuable investments were the ones token accounting punished, better model routing, evaluation pipelines, prompt versioning, and experimentation infrastructure, all of which raised spend on paper while sharply lowering the cost of iteration.
That lower cost of iteration is where Dhar locates the actual return. When engineers can compare models safely and product teams can experiment without fear of quietly degrading production, a team can validate five ideas in the time it used to take to validate one, and the business benefit dwarfs the incremental compute. He also refuses to ignore the second-order effect that never reaches a finance dashboard, the way good infrastructure changes behavior, so engineers start proposing projects that once felt too expensive or too repetitive to attempt and product conversations get more ambitious because implementation cost no longer dominates every decision. His test is a single question, whether the system helped the team build something it otherwise would not have built.
Count only the work that clears a bottleneck
“Only the work that clears a bottleneck is progress. The rest are lightbulb filaments that didn’t work.” — Michael J.J. Tiffany, co-founder and CEO of Fulcra Dynamics.
Tiffany offers the closest thing in this issue to a replacement unit of account, and he arrived at it by correcting himself. Token use, he argues, is a low-level metric like CPU utilization, important but meaningless in aggregate. His better frame is cost per bottleneck relieved. He started somewhere looser, counting cost per accepted unit of work such as a merged pull request or a user-visible improvement, then learned the hard way to count only the subset of work that actually clears a bottleneck.
The distinction changes what you tolerate. What you buy when you pay for AI, in Tiffany’s account, is more attempts, faster loops, more surface area explored, fewer blocked humans, and occasionally, almost at random, a capability that was not economically or psychologically feasible before. Most of that is not progress, and that is fine. The failed experiments and even the shipped features that did not matter are the filaments that did not light. You let them happen, and you measure success only by the bottlenecks that came loose, which keeps the number honest in a way a token count never can.
Judgment is the constraint for the next two years
The token leaderboards are already gone, and the mandate that produced them is being quietly rewritten across the industry toward outcomes that are harder to game. The deeper lesson is the one Parker keeps returning to. Every layer an agent touches was built on the assumption that the consumer brings context and judgment, and that assumption is exactly what breaks when the consumer, or the incentive, is optimizing a number.
The voices in this issue disagree on the right replacement, and the disagreement is the useful part. Howe would protect trust before any metric. Putatunda would standardize the engineering practice underneath the tooling. Dhar would ask whether the team can now build what it could not before. Tiffany would count only the bottlenecks that came loose. What they share is a refusal to accept the token as the unit of value, and a conviction that the scarce resource is the judgment to decide what is worth building in the first place. The limiting factors on agentic systems are no longer model capability alone. They are judgment, integration, and the operational discipline to tell useful work apart from motion, and that is where the most consequential engineering decisions of the next two years will be made.
Thank you for reading this special issue of Deep Engineering on why judgment, not tokens, creates value.
We’ll be back on Thursday with more expert-led content, and next month, on the first Tuesday of August, with another special issue.
Keep building,
Saqib Jan
Editor-in-Chief, Deep Engineering
If your company wants to reach senior developers, software engineers, and technical decision-makers, speak to us about partnering with Deep Engineering.