Tokens, Judgment, and AI Productivity with Austin Parker
On why a token is not a unit of intelligence, the illegible work token counts miss, pointing AI budgets at developer experience and observability, and judgment as the last scarce resource
A token is not a unit of intelligence, and judgment is the resource that actually got scarce.
Austin Parker has worked in observability for about a decade, co-founded OpenTelemetry, and now leads AI strategy at Honeycomb. He came on to make an uncomfortable case for anyone running an AI mandate, that a token is not a fungible unit of intelligence and judgment is the resource that actually got scarce.
You can read or watch the full conversation here:
This session was recorded live as part of the Deep Engineering Interview Series. The transcript below has been lightly edited for clarity and readability.
Q. Tell us a little about yourself, what you do at Honeycomb, and what pushed you to write about tokenmaxxing.
I have been in observability for about a decade, first as a maintainer on OpenTracing, and then I helped co-found OpenTelemetry when we merged it with OpenCensus years ago. I have worn a lot of hats across that open source work. About two years ago, when ChatGPT released, I got really interested in AI and in how we could use it to help people make sense of systems at scale. Over the past year I have been building that capability out at Honeycomb, working on our agents and our machine learning and AI stack, and getting involved in OpenTelemetry’s GenAI semantic conventions to make sure OpenTelemetry is as useful to AI as it is to humans.
Q. You argue a token is not a fungible unit of intelligence. Walk us through one engineering decision where treating tokens as fungible led a team somewhere they regretted.
You can find a lot of examples of that just reading the news today. Given the right goals and the right harness around a model, whether that is a frontier model or an open weights one, you can get it to do almost anything. The maintainer of Bun, who I believe now works at Anthropic, famously did a big port where the instruction was basically rewrite the whole thing in Rust, and it did it, and it passed all the tests, which is impressive.
When we read things like that it becomes easy to imagine that all I need is something that keeps the tokens flowing and I can do anything I want. That might be true, but it does not make it a good idea. Internally I have seen our engineering teams run experiments where they generate a hundred different prototypes of the same screen, or fifteen or twenty variations of a flow. What I have noticed is that most of them do not move the needle in the way you would hope, because you can ask for infinite variations on a problem. You can ask for the same program in ten different languages, or rewrite a function a hundred thousand times.
But the code and the screen and the workflow are not the things that provide value to your users. The value is in the questions one level up, about the design of the system, about how people interact with it, about whether the shape of an API really serves the people integrating with it, and whether the fundamental primitives have fitness for what people want out of them. Those are not questions that tokens answer. You can use tokens to help build solutions, but at the end of the day these are judgment calls. Those hundred variations all came out looking about the same, because they all started from one very human conception of the problem, so all of the output ends up shaped like my initial conception of it. I see this repeated when I talk to other leaders. There is a lot of here is a problem, make all these variations, and far less time asking whether we are defining the problem correctly, solving the right customer pain point, and putting the right inputs into our loops. That is why I argue that judgment is the finite resource now.
Q. You call prompting, building internal alignment, and holding a complex system in your head illegible activities. When a leader starts measuring token usage, what specifically breaks in how those skills get rewarded?
It is tough to say sometimes, because at really large organizations that kind of value is already not rewarded well. A lot has been written about the importance of glue work, about people whose value to the org is not how many PRs they open or how many decks they build, but knowing who needs to be in a decision chain, or being able to hear a customer request and go find the right team to make it happen. Often you or I might be frustrated with a piece of software because of one thing that shows up in a certain part of the system, when the actual reason it works that way lives somewhere else entirely. The people who can navigate that ownership map internally end up being really important to the ability of the business to execute and serve customers. The companies that are genuinely good at that tend to be the ones that, even without a clean way to measure those people, are good at not penalizing them.
Then a top down mandate arrives that says we need to use more AI, and the easiest objective measure of that is how many tokens you use. So the move becomes let us incentivize you to use more tokens and then productivity magically appears. The connective work that makes someone valuable shows up in promo packets and career ladders, but it does not show up in something you can measure quarter to quarter or day to day, because it involves very human things like relationships and time in the system rather than timing the system. The tokenmaxxing craze, 2026 to 2026, RIP, is a good example of why that value matters and why you cannot replace it by throwing more AI at the problem.
Q. The token leaderboards at companies like Amazon rose and fell inside three months. What did the people running them learn, and why do you think the correction came so fast?
What they learned is that maybe they should not do that again. The moment you turn any measure into a metric it stops being a good measure. Give a legendarily clever bunch of engineers a resource and tell them they are graded on how much of it they use, and they will find a way to use it. There is a famous Dilbert strip where they announce a bonus for every bug you fix, and Wally says he is going to write himself a new minivan, because you said every bug that gets identified and fixed, you did not say anything about not writing the bugs in the first place. Same thing with token leaderboards. What is strange to me is not that it happened, it is that it was not immediately obvious to the people in the decision chain that it would.
Calling them pointy haired bosses who just do not get it is a lazy answer though, and I do not think it is true. Yes, plenty of those experiments led to utter failure, and broadly it was a bad idea I would never have recommended. But I have noticed people, even at Honeycomb, who get nervous about spending company resources on an idea they think AI might be good for. A moment of use as much as you want, go wild, frees people who would otherwise be overly conservative, and some of them find something they never would have found under the old constraint. So some of this is taking the good with the bad, and some of it is making sure your employees are aligned with the overall goal. Those extremes are why it moved in and out of favor so fast, because it only takes a few people burning hundreds of thousands of dollars. Although Amazon is probably good for it.
Q. You say AI accentuates differences that already exist rather than creating new capability. If a leader sees one team pull ahead with agents, what is the actual question they should be asking that team?
This is the thing I feel does not get asked enough. I never want to ask what AI has let you do. The question is what is AI giving you the excuse to do that you could not have done otherwise. In every big phase shift you find individuals and teams who use the bigger meta narrative as cover to solve a different problem. With AI, it is rarely that intelligence alone solved something we could not solve before. Usually it is solving an organizational problem or a leverage problem.
Think about what it takes to ship a feature inside any organization. A big part of it is making the case to the rest of the org, especially leadership, about the expected return, and that is often a data analysis argument. It means sitting down with a pile of numbers, doing a lot of advanced Excel, building projections and forecasts, tearing through user data and sentiment data and support tickets. It turns out AI is really good at that. Give it access to your support tickets and your customer experience data and your NPS scores, spend time with it, and it helps you surface things you knew but could not back up, or genuinely new insights. Traditionally that was a product manager or business analyst function, and there is a wide range of skill there. Now a sufficiently motivated engineer or designer or support person or sales engineer can do that work, use AI to cover the parts they are not as good at, and turn around and tell the org here is what you need from me so I can go do the thing I have conviction is right.
That is what I see in the teams becoming very effective with AI. It is not simply that we can write more code now, although that helps and you can review it faster. It is that we can patch whatever hole existed in our skills or our team composition. You see it outside engineering too, and at small companies, where people who are strong at design but never programmed can now program, and people who are weak at persuasive writing can write better. We are filling the gaps in our own abilities, and that is the biggest difference between the teams getting real value out of AI and the teams that are coasting.
Q. Should companies refuse to measure engineers by how much code their agents write, or even how many bugs they patch?
I think that is stupid. There are so many other ways to gauge engineering efficiency. A better way to think about it is that you should measure it, but it should not be a metric. It is useful. One thing we do internally is look at how many PRs someone is reviewing, how many they are opening, and the balance of their time in the codebase. That is a useful signal, because it shows you someone spending a lot of time reviewing versus someone spending a lot of time writing, and maybe you need to think about that balance, or maybe someone is adding a lot of new things but not spending enough time understanding what is going in. Turning it into something you grade performance on is where it gets twisted, and a lot of places still treat PRs opened as an important KPI.
AI actually lets you do this analysis more holistically. Instead of stopping at how many PRs someone did, you can feed a lot of it into a model and ask real questions. Of the work you are doing, how much is causing on call burden, how much is delivering specific customer value, and are we missing the people doing important but less shiny work like refactoring. Attaching those contributions to specific outcomes is hard, like a big refactor of the component library that improved consistency and lifted the interstitial user experience survey scores because a button is now consistent across the app. Those things are hard not because they are hard to measure, but because there is so much data to go through. AI makes that analysis much easier when it is driven by high quality telemetry and instrumentation, and when the people using it have good judgment.
Q. Your advice is to turn token budgets inward toward developer experience and observability. What does that look like in practice for a Staff or Principal engineer who owns a platform, in the first ninety days?
The core thing you want is to get to that analysis loop, which means asking whether you even have the data, whether it is possible for you to ask those questions at all. Most of the time it is not, because as an industry we do not value observability as highly as we should, though we are getting there. I still talk to a lot of people who have their emotional support logs, because that is just how it has always been done. The historical argument against improving observability has always been that we do not have the time, we do not have the people, we have all these other goals.
This is exactly the mechanical, boring work that is fungible from an intelligence perspective. Taking your existing logging and making it structured, or moving to OpenTelemetry, is perfectly aligned with what current AI coding models are good at, and with the verification side too, because it comes down to a judgment call about whether the telemetry after the migration is as good by some heuristic as what you had before. You can prompt the verification step directly. Strip out the unstructured logging, add structured events and spans and good metrics, then feed the output into another model to judge whether it is better, worse, or the same, and let it rip. It can update the docs and the comments while it goes. That is massive leverage for an org that takes the diffuse do whatever with AI money pool and points it at something specific like developer experience and observability. It lines both things up, because more people learn to build better harnesses and prompts and loops, and you get a measurable outcome, which is that now you can answer these questions and understand your systems and run the longer horizon analysis you never had the data for. Job one is getting to the point where you can ask these questions.
Q. Local inference and model routing make token accounting even murkier. As that lands, what mental model should engineering leaders use for AI spend instead of cost per token?
I honestly wish I had a better answer. To use an example from Honeycomb, we have been doing agentic engineering and going through this transformation like everyone else for about a year and a half, and the results have been pretty dramatic. In one quarter we opened as many PRs as we had in the entire year up to that point, because people are leveraging AI more and a fairly small engineering team can punch above its weight on code writing and generation.
One example of that is dark mode. For as long as I have been here, dark mode was one of the most consistent customer requests, and we are a developer tool, so people really wanted it. We have an in joke where you get a D20 when you join, and one of the faces reads dark mode when, and that joke is five or six years old. For at least half the company’s life it was something we would get to eventually. Then in about six months we shipped it, mostly because of AI. I want to be clear it was not only AI. Getting there was a multi year process where our designers and product engineers built a design system and an accessible component library that also supported dark mode. At the end of that we had all the components, but we still faced the mechanical work of turning the existing app into the new ones, and that takes a lot of time.
Because we could lean on agent loops, we did it in about six months, and not as one team’s full time effort. People across the whole company chipped in, including engineering directors and managers, because instead of telling everyone to go do all these things by hand, the team shipped prompts and skills you could feed the agent to do the migration from the old components to the new ones. That unlocked a lot of human judgment, because I could pull in an engineering director who built a particular screen seven years ago and still understands what it is for, and tell them to fire up Claude Code, load this skill, point it at that page, and do the migration.
That is a good example of what your mental model should be. Yes, it used a lot of tokens, and a small focused team could have hyper optimized it. But the actual benefit is that we finally closed a feature people had asked for over five or six years, a lot of people learned how to use AI who otherwise would not have, and we got a more accessible and more beautiful product. Those benefits are tricky to quantify, but they make a real difference to customer experience, and at the end of the day that is what we are here for. It is not using AI for its own sake, it is talking to users, understanding their pain points, and meeting them where they are. So do not treat it as an AI mandate you justify only through tokens in and tokens out. Ask what tokens let you do that would otherwise have been impossible, and what lets you use AI as leverage, as a bicycle for the mind, a way to take bigger swings at bigger problems. If you can figure out how to measure that, that is your North Star.