Site Reliability Engineer (SRE) Interviews Measure a Depreciating Skill
Notable engineering leaders on motivation, ambiguity, tribal knowledge and the measurement problem behind every reliability hire that did not work out
Reliability teams are hiring against a job description that describes work their agents increasingly do. That mismatch surfaces as a strong candidate who cleared every round and then underperformed, as a loop where three finalists looked interchangeable and the decision came down to feel, or as a new hire who put their first year into building instrumentation nobody had told them was missing.
For most of the discipline’s history the binding constraint during an incident was human attention, because somebody had to correlate logs across services, hold a working model of the system, and form a hypothesis fast enough to matter. Interview loops were built to find that person and they were right to. Agents now take a growing share of that first pass, and the loop has not moved with them. A coding exercise, a debugging scenario, and a systems round all measure how fast a candidate converges on a cause. Those are the skills losing value fastest, while the ones that decide whether the hire works go untested.
Reid Savage, Senior Engineering Manager for the Site Reliability Engineering team at Honeycomb, shares how he has repeatedly seen that gap widen across the hires he has made and overseen.
“Standard SRE hiring loops are far too skills-based,” he says. And his own team acted on that conclusion, removing coding exercises from the loop at Honeycomb’s current stage.
Motivation decides the hire, not the skills matrix
The failure pattern Savage describes does not look like a skills gap, which is what makes it so easy for a conventional loop to miss. “I’ve had excellent hires that have never used our cloud of choice, and some who didn’t work out that had nearly everything,” he says. The candidates who arrived with the full stack on their résumé were not reliably the ones who lasted, and the ones missing the specific platform experience frequently were. That inversion is not a story about credentials being worthless, and it is a story about credentials answering a question that turns out not to be decisive.
What Savage found underneath the pattern was a mismatch of a different kind. The problem in the hires that did not work out was “usually a mismatch between true motivation (which requires significant self-insight on their part) and what the team needs,” he explains. Motivation in this framing is not enthusiasm for the role or eagerness in the interview, both of which candidates supply readily and neither of which predicts much. It is the specific thing that makes the work satisfying to that person, which requires enough self-knowledge on their part to name it accurately. Savage anchors the distinction to a line he paraphrases from First, Break All The Rules, about hiring the accountant who cannot sleep until the books are balanced rather than the one who knows Excel.
The scale of Honeycomb’s problem space is part of why this holds. “At Honeycomb’s size, the skills required for each problem we have wouldn’t fit in 5 job descriptions,” Savage says, which means any specific skill the loop screens for covers a small fraction of what the hire will actually encounter. A loop optimized for skills coverage is therefore optimizing a variable that cannot be made to matter, because the surface area defeats it. The organizations most exposed to this are the ones whose reliability function touches many systems rather than one, which describes most companies past their first platform consolidation.
Honeycomb’s replacement for the coding exercise is the actionable part. The technical screen is now a pull request review, chosen because, in Savage’s words, “it covers the important things: reading between the lines, social skills, eagerness to help, mindset, and integrity.” A PR review works as a screen because it presents the candidate with someone else’s decisions rather than a blank editor, and reliability work consists largely of forming a fast, correct read on choices other people made under constraints that are no longer visible. Leaders who want to run this should use a real pull request from their own history, ideally one where the right call was contested, and score the review on what the candidate noticed and chose to raise rather than on whether they found a seeded bug.
Savage’s position is that the technical bar clears more reliably in the presence of those qualities than the reverse, because “if you have those, and they display the level of technical aptitude you need, they’ll be able to catch up.”
Profile of the hire changed inside a year
Fixing the screen addresses the process. The harder change is to the role itself, and it has moved faster than most loops have been revised. “The profile of a great SRE has fundamentally changed over the last year,” says Anish Agarwal, co-founder and CEO of Traversal AI, which builds AI systems for production incident investigation. His account of what companies previously optimized for matches what most loops still measure, which is the engineer who could reason through dashboards, logs, and metrics under pressure and arrive at a root cause. That was the correct thing to optimize while humans carried every investigation from alert to resolution.
Agarwal’s argument is that agentic systems now take on the most time-consuming parts of that work, and the consequence for hiring follows directly. “Raw troubleshooting ability is becoming less of the differentiator than it was even a year ago,” he says. A capability stops being a differentiator once it stops being scarce, and a loop weighted toward a non-scarce capability produces candidates who cluster indistinguishably at the top. Leaders tend to notice this as a symptom well before they diagnose it, in loops where several finalists all pass the debugging round cleanly and the final decision comes down to feel.
What replaces it, in Agarwal’s view, is the part of the role that always belonged there and rarely got attention. “What has become far more important is the ability to engineer resilient systems,” he says, being precise about why that capability went underexercised for so long. Most teams never found time for it because troubleshooting consumed their days, so resilience work stayed aspirational rather than scheduled. Two changes now arrive together, with AI removing much of the investigative burden while production systems grow more complex through new services, new dependencies, and new failure modes each quarter. His conclusion is that resilience engineering “is no longer the work SREs wish they had time for. It’s increasingly the work the role demands.”
That reframes what a strong candidate has to demonstrate in an interview. Agarwal describes the highest-leverage SREs as the ones “thinking about what the system will need a year from now, which SLAs actually matter, and which technologies best meet those SLAs given the tradeoffs between cost, reliability, and performance.” A loop that wants to surface this has to ask forward-looking questions rather than diagnostic ones, and the practical version is to hand the candidate a real architecture with real growth assumptions and ask what breaks first at ten times the load. The answer reveals whether the candidate reasons about failure modes that do not exist yet, which no incident replay exercise can establish. Agarwal adds one further capability he would weight more heavily, describing it as “AI fluency. Not just expertise, but genuine curiosity,” and locating its value in engineers who use agents to compress investigations so that the time freed goes into resilience work.
Judgment is the scarce input, and it leaves when people do
Hiring toward resilience assumes the judgment to do that work can be found and kept, and that assumption meets an organizational problem that no interview round surfaces. George Hantzaras, Director of Engineering, Core Platforms at MongoDB, speaking at the AI-Powered Platform Engineering workshop we hosted some time back, argued that the problems facing platform and operations teams have stopped being problems of tooling or frameworks and have become problems of people, team dynamics, and how humans interact with the systems they built. His first category is the knowledge gap, where the answers engineers need lie buried across dozens of systems, dated documentation, and sprawling wikis. He described his own developers as having become archaeologists, running scavenger hunts through internal documentation to recover a single answer.
The part of that gap most relevant to hiring is the part nobody wrote down. Hantzaras underscored tribal knowledge as the real problem, meaning the unwritten rules and undocumented workarounds that exist only in the minds of the few people who have been at a company long enough to accumulate them. He described one team where new engineers took close to six months to reach full productivity, with the primary cause being the difficulty of navigating internal systems and discovering rules that appeared on no wiki. That knowledge carries a second liability beyond slow onboarding, because it leaves the organization when the people holding it leave, and nothing in a standard hiring process accounts for either effect.
Two further problems he named sharpen what the reliability hire is actually for. Golden paths handle roughly the first eighty percent of cases well and then become what he calls golden cages for the remaining twenty percent, which tend to be the most innovative work and the cases where a developer needs something the template will not express. He also described a platform lead whose team was putting more time into maintaining YAML templates for their portal than into building new platform capability, an overhead he calls the tooling tax. Both problems consume senior judgment on work that produces nothing durable, which is exactly the capacity a resilience-focused hire is supposed to create.
Hantzaras reaches the case for automating the first pass from the opposite direction, arriving through the operations burden rather than through the hiring loop. He pointed out that the industry has fixated on mean time to recovery while, in his reading, the largest component is mean time to identify, and that teams are drowning in logs, metrics, traces, and alerts without a good way to act on any of it. The resulting alert fatigue leads engineers to tune out noise until a critical alert has a high probability of being lost. His read on responsibility for that is unambiguous, because it is almost never a case of an engineer not working and almost always a case of a system not working as it should, and his conclusion is that filtering data at that volume “shouldn’t be a human job. This is a machine job.”
For leaders hiring into this, the implication is that the scarce input is accumulated judgment rather than throughput, and that judgment currently has no home outside individual heads. The practical response runs on two tracks. Weight the loop toward candidates who can reconstruct the reasoning behind decisions they did not make, since that is the skill tribal knowledge recovery actually requires, and a PR review from an unfamiliar codebase tests it directly. Then treat the documentation of undocumented judgment as an explicit deliverable in the first six months rather than as good citizenship, because the alternative is rehiring the same knowledge every time someone resigns.
Foundations come before fluency
Hiring for AI fluency assumes the environment can support it, and that assumption fails more often than the hiring conversation admits. Ajay Chankramath, CTO of Platform Engineering Community, a member organization for DevOps and cloud native practitioners, led the platform engineering workshop sessions for Deep Engineering recently, where he described platform maturity as a progression through four states, moving from reactive to responsive, then to predictive, and finally to autonomous. His caution for teams reaching for the later states is that observability, runbooks, and service level objectives have to be in place before AI gets layered on top, because introducing AI into an environment without those foundations amplifies the existing chaos rather than resolving it. That ordering has a direct hiring consequence, because a candidate hired for AI fluency into a team without instrumented systems will put their first year into building the substrate instead.
Chankramath is equally clear that the tooling does not displace the role. His framing of AI in incident response is that it hands the engineer a capability rather than taking one away, reading the runbook on their behalf and proposing the next step, while the engineer remains the expert who decides whether that step gets taken. He grounds this in an observation most people on call will recognize, which is that nobody reads documentation during a live incident, because when the system is failing and customers are escalating, engineers act on what they already know rather than opening a markdown file. AI closes that gap by surfacing the relevant procedure at the moment of the decision, and the judgment about whether to follow it stays where it was.
He also names a failure mode that should change how leaders think about seniority in these hires. Platform capabilities frequently go unused by the most experienced engineers, who treat self-service tooling as something built for juniors who do not know how to do the work directly, and because those engineers are the ones whose opinions carry weight internally, their dismissal propagates. A reliability hire brought in specifically to raise AI adoption will therefore run into resistance from exactly the people whose endorsement determines whether the change holds. Leaders should test for this in the interview by asking how the candidate has previously won over a skeptical senior engineer, rather than assuming enthusiasm for the tooling transfers to the team.
The last piece of Chankramath’s argument concerns measurement, and it constrains how any of this gets evaluated. He treats the standard delivery metrics as lagging indicators, useful for confirming what already happened and poor at telling a team where things are heading, which makes running an organization solely on them a mistake. Leaders hiring for resilience need leading indicators to judge whether the hire is working, because a resilience improvement shows up as incidents that never occurred, and no lagging metric records an absence. The workable substitutes are the observable precursors, meaning coverage of critical paths by service level objectives, the share of alerts that map to a documented response, and error budget burn rate read before the budget is exhausted rather than after.
Ambiguity behavior predicts the outcome better than anything else
Across everything Savage has observed, one behavior separates the hires that worked from the hires that did not, and it is not a technical one. “The largest determinant I see is what they do when faced with an ambiguous problem,” he says, and he lays out the range of possible responses as moving away from it, tagging in someone else, asking for help, owning the solution, helping someone else own it, or investigating why the problem was ambiguous in the first place. The question underneath all of those, in his phrasing, is whether the candidate accepts or rejects unclear problems. Reliability work arrives almost exclusively as unclear problems, which is what makes this behavior more predictive than any skill the loop can measure.
The cost of getting it wrong is not usually a visible failure, which is part of why it goes undetected. Savage describes one SRE who was passionate and knowledgeable, and who struggled to bring others into the work or carry projects across the finish line, producing a long trail of initiatives that started without much to show for them. Because SREs touch every part of the business, that pattern compounds in both directions, generating leverage when it works and opportunity cost plus real cloud spend when it does not. His summary of the missing skill is that “SREs need to know when to open, peek inside, rearrange, and put back problems.” The judgment is as much about which problems to close as which to open.
Testing for this requires giving candidates something genuinely underspecified rather than a puzzle with a hidden solution. The practical version is to present a real ambiguous situation from the team’s history, withhold the resolution, and pay attention to whether the candidate starts solving or starts questioning the framing. Both responses can be correct, and what matters is that the candidate demonstrates a deliberate choice between them rather than defaulting.
So define the role before you post it
None of the above helps if the role itself was never specified, which Savage treats as the prior question. “You need to know what you’re hiring the SRE for,” he says, and the diagnostic he runs through is a series of concrete organizational facts rather than an abstract job description. Those facts are whether reliability genuinely matters to the business and to what degree, how many nines the company actually needs, whether anyone currently has responsibility for predicting what will fail a year out, whether developers are resisting on-call, and whether the team is drowning in toil. Each of those describes a different job, and an SRE can address any of them, though not all of them at once.
The nines question is the one most organizations answer loosely, and it is the one that determines everything downstream. A target expressed as a service level objective with an error budget attached tells a candidate what the job actually involves, because it establishes how much unreliability the business has already agreed to tolerate and therefore where the engineering effort goes. A posting that claims high availability without naming a target is describing an aspiration rather than a role.
His standard for handling that honestly is where the recommendation lands. It is fine to need some of those things and not others, as long as the job posting states what the role is expected to be today and how that might change. Most mis-hires that look like candidate failures are role definition failures surfacing late, because a candidate optimized for toil reduction will underperform in a role that turns out to require reliability forecasting, and neither party did anything wrong in the interview. The discipline is to write the posting from the specific problem rather than from a template, and to name the parts of the role that are unsettled instead of leaving them out.
The same clarity should extend into the first ninety days, and Savage’s approach here inverts the usual instrument. The senior hires who succeeded did not treat his 30/60/90 as gospel, and read it instead as an aspirational list with goals attached, oriented toward making social connections, becoming familiar with the code, and starting the long climb through the tech stack. He asks new hires early on to disagree with him and to offer at least one piece of constructive feedback, which he uses to open room for them to work on problems he cannot see himself. That is a deliberate test of the same quality the loop was built to find, because an engineer who will surface an uncomfortable observation in week three is demonstrating the disposition toward unclear problems that predicts everything else.
What connects all four positions is that none of them argues for a lower technical bar. The bar holds, and the argument is that the loop currently puts most of its measurement on the part of the bar that agents are absorbing, while the parts that decide whether the hire succeeds go untested. The reliability hire that works out is the person who accepts unclear problems, reasons about failures that have not happened, carries judgment that was never documented, and improves a system nobody else wanted to own. A process built to find that person looks different from the one most companies are still running.