Rust Is Hard for the Engineers with the Most Experience

On the unlearning problem that trips up experienced engineers and the production payoff that makes it worth it

Rust, who would have thought, has ranked as the most loved programming language in the Stack Overflow developer survey for nine consecutive years. Honestly, I must admit this is an unusual kind of statistic because it measures not just adoption but retention. The engineers who use Rust want to keep using it, and that pattern has only deepened even as the language moved from systems programming curiosity to production infrastructure at companies including Amazon, Google, Meta, and Microsoft. Interestingly, the Linux kernel now carries Rust code without the experimental label it held for years. Debian’s APT package manager is also introducing hard Rust dependencies this year.

The performance benchmarks and the memory safety arguments have been made, tested in production, and largely validated. But what the benchmarks do not explain is why so many experienced engineers find Rust genuinely difficult to work with, why the teams that adopt it often go through a period where velocity drops before it recovers, and what it actually takes to get good at it rather than just competent. These are the questions that sit behind the adoption numbers and they matter more than the numbers do for any engineer thinking seriously about where Rust fits in their work.

Evan Williams, author of Design Patterns and Best Practices in Rust, has been writing software for more than 40 years and came to Rust while building a hardware system that needed to be rock solid and run without access in a remote location. Francesco Ciulla, author of The Rust Programming Handbook and a Docker Captain who previously worked at the European Space Agency on the Copernicus project, started publishing Rust content in 2022 and 2023, earlier than most, and has since watched the language’s adoption from the inside. Their perspectives on Rust come from different parts of the stack and different kinds of work, but on the questions that actually trip engineers up, they are in close agreement.

We interviewed Evan Williams and Francesco Ciulla separately for Deep Engineering Newsletter issues.

The engineer who struggles most is usually the most experienced one

The reasonable assumption when a team introduces Rust is that the senior engineers will pick it up fastest. They have the most context, the most pattern recognition, and the most experience navigating unfamiliar codebases. In practice, the opposite tends to happen, and both Williams and Ciulla have seen it play out firsthand.

“The more experienced you are, the more years you have doing something in some other language, the more trouble you’re likely to have,” Williams says, “because you have patterns of thought that come from those languages that you don’t even realize are there.” The problem is not that experienced engineers consciously try to apply Java or C++ patterns to Rust. The problem is that those patterns are invisible to them, baked in over years of use until they no longer register as choices at all. The engineer is not making a decision when they reach for inheritance or shared mutable state. They are doing what has always worked, and Rust will not let them.

Ciulla put it more directly. “Even if you are a senior developer, even if you have twenty years of experience, if you want to try to learn Rust comparing it to other programming languages, you will fail, because it’s like learning something which is completely new.” He makes the case that this is not a reason to avoid Rust but a reason to go in with a specific kind of openness, one that experienced engineers often find harder to maintain than junior ones do precisely because they have more to unlearn. A developer learning Rust as their second or third language has no competing mental model to discard. A senior engineer with a decade long experience in Java has to dismantle instincts that have been reliable for years before they can build new ones, and that dismantling is the work that most people underestimate going in.

Trusting the compiler is not a beginner tip

The first thing most engineers do when the borrow checker rejects their code is look for the minimum change that will make it compile. That is the right approach in almost every other language and the wrong one in Rust, and it is where a significant amount of early frustration comes from.

“The golden rule is to trust the compiler, especially at the beginning,” Ciulla says. What he means is not passive acceptance but active reading. The borrow checker is not producing noise. It is producing information about what the program’s structure requires, and engineers who learn to read it that way move through the learning curve faster than engineers who treat every error as an obstacle to clear. The difference is subtle at first and significant over time.

Williams argues that the borrow checker is doing something more useful than preventing bugs. “The borrow checker is your friend because it prevents you from making a messy design. It prevents you from making a broken design. It prevents you from writing whole classes of bugs that you will then spend many hours trying to find,” he explains. “I have found it to be an incredible partner in writing code that allows me to sleep at night.” The reason it works this way is that Rust’s ownership rules enforce a discipline that experienced engineers in other languages apply selectively and inconsistently because those languages do not require it. A value has one owner. References are either shared and immutable or exclusive and mutable, never both. The compiler will not proceed until the code is explicit about who owns what and when.

“The principles that the borrow checker forces you to adhere to in Rust are the exact principles that you should be using in every programming language,” Williams reasons. “But you don’t have to. So it’s very easy to not think about those things.” That observation reframes what the borrow checker is. It is not an imposed restriction. It is a discipline that good engineers apply in other languages by habit and judgment, made non-negotiable and automatic in Rust.

The discipline extends from individual functions to the shape of the whole system. A program that handles ownership correctly at the function level has to handle it correctly across modules, across threads, and across component boundaries, because the same rules apply everywhere. “You need to think about who controls what, how it is controlled, and you need to start from the very beginning thinking about the boundaries of your program and the system architecture, dividing things up into areas of responsibility,” Williams underscores. “Because unlike Python or Java, you can’t have links going all over the place. The borrow checker is never going to accept that.” The result is that well-written Rust systems tend toward a specific architectural shape: data flows in one direction, ownership chains move forward and do not loop back, and the behavior of the system is legible from its structure in a way that systems with shared mutable state often are not.

The most underutilized expression of what this makes possible is the typestate pattern. It uses the type system to encode the state of a value at compile time in a way that makes invalid state transitions not just errors but programs that cannot be compiled at all. Williams reflects on it with visible enthusiasm. “It’s a way of developing state machines and systems that have state that evolves where invalid state transitions aren’t just errors, they’re impossible to write. The compiler won’t compile them,” he says. “It represents a huge advance in the way that such systems are written because now instead of runtime errors, you have a state machine that is guaranteed to work because every transition either is a valid transition or it won’t even compile. That’s an amazing thing.” The pattern was not invented for Rust, but the language’s ownership system and type handling make it practical in a way that other languages do not, and for systems where invalid state transitions are genuinely dangerous rather than merely inconvenient, it is one of the most concrete expressions of what Rust makes possible.

What Rust actually gives you in production

Ciulla’s case for Rust is grounded in things he measured when running a Rust web server on his own machine. It was consuming four megabytes at rest and five in production. “If you have a droplet with one gigabyte of RAM, you can have 200 plus services,” he notes, “of course in idle, but this proves that if you have a service that consumes a lot of RAM, it is worth thinking about.” For teams running infrastructure where memory costs money and density matters, the difference between a Rust service and an equivalent service in a garbage-collected language is not marginal.

The latency story is also specific. “By not having a garbage collector on the back end side, you basically have a flat latency,” Ciulla observes. “If a user makes an HTTP request when the garbage collector starts, it will experience a higher latency. Rust removes that problem entirely.” Go and Node.js both have garbage collectors that pause for collection cycles, and even short pauses measured in hundreds of milliseconds are enough to introduce latency spikes that affect users who are unlucky enough to hit the request at the wrong moment. Rust’s absence of a garbage collector means the latency profile is predictable rather than probabilistic, which matters significantly for services where consistency is as important as average throughput.

The deployment model is simpler than most engineers expect going in. A Rust project built with cargo produces a standalone binary for the target architecture, which packages cleanly into a container image. “If you build the executable when you build the Docker image, you have something which is just deployable everywhere,” Ciulla says. “A Linux executable running in a Docker container. That’s the dream.” The operational benefit is smaller images, faster startup, and a runtime with almost no overhead beyond the binary itself.

Williams approaches the production question from the correctness angle rather than the performance angle. The systems where Rust earns its place most clearly are the ones where failure has a real cost. “Systems that are mission critical in some way or other are really key Rust use cases,” he says. “All of these features combine into a whole that make Rust a really powerful language for doing things that have to work. Things where failure is monetarily or in human cost even a terrible problem.” The memory safety and the ownership model and the compile-time guarantees are not separate features. They are different expressions of the same underlying commitment: the program either demonstrates its correctness to the compiler or it does not compile.

Williams also reflects on something unexpected he discovered while writing the early chapters of his book, the ones covering what not to do in Rust. He went back and deliberately tried to write bad code, the kind of code that would illustrate the mistakes he was cautioning against, and found it harder than he expected. “When I went back and tried to write bad code in Rust, it was much harder than writing the good code,” he recalls. “That’s an interesting perspective that just didn’t even occur to me.” The language’s constraints push code toward a particular shape so consistently that departing from it requires actively working against the grain of the language rather than simply making a poor choice.

“The biggest benefit in Rust is about the lack of the debugging depth. You spend more time thinking up front, but you spend almost zero time chasing segfaults or memory leaks in production,” Ciulla remarks. “And we always underestimate this part. We always talk about the efficiency of the code, but if you need less time to debug your code, you’re basically writing more logic at the end of the day.” The upfront investment in getting the types and the ownership right is real, but the downstream debugging cost it removes is larger and does not diminish as the team becomes more experienced. It is simply gone.

Where to start and where Rust is the wrong tool

On the practical question of how to bring Rust into an existing codebase, both Williams and Ciulla give advice that converges almost exactly despite coming from different engineering contexts. Neither recommends starting with a rewrite.

“The best way to introduce Rust in a big project is to find that hard part that’s the bottleneck and try to write one single service in Rust,” Ciulla says. “And then you will see, probably slowly, Rust might take over your code base, but I mean this in a good sense.” Williams makes the same point with a specific warning about the temptation to go faster. “What you don’t want to do is jump into saying, we’re just going to rewrite our project in Rust now. Pick a small piece, focus on that, gain confidence and mastery of the language, and then use that to build upon it and start bringing in more things,” he says. Starting with a bounded, non-critical component gives the team room to move through the learning curve without the pressure of a production incident concentrating everyone’s attention on the wrong things.

Ciulla adds something worth noting about the AI-assisted workflow that is becoming standard for many engineers. “In this AI era, everyone is rushing stuff with AI, but you still need the validation,” he says. “Okay, AI wrote this Rust service, but now who decides if this is okay to put in production? Of course, you need the validation of an expert.” The Rust compiler catches a large class of errors automatically, but the errors that survive it, logic errors rather than memory errors, still require someone who understands the language well enough to see what the code is actually doing. Having at least one engineer on the team who knows Rust well enough to review AI-generated code is not optional.

Both are also direct about when Rust is not the right choice. Ciulla points to tight deadlines and fast prototyping as the clearest case against it. “If you need fast prototyping, you are familiar already with Java, JavaScript, why don’t you use it?” he says. “When the deadline is so close, probably it’s not the best way to try something new because something would go wrong, especially if you’re not an expert.”

Williams points to user interfaces as an area where the ecosystem is still catching up and the tooling gaps are large enough to make other languages more practical. “Doing a website in Rust is still kind of a feat,” he notes, “and it’s an awful lot easier to use the tools that everybody else is using to accomplish that goal.” The Python data science ecosystem is another area Ciulla names directly: the libraries are simply better established there, and using Rust for data science work means building against a thinner set of available tools than Python provides.

Where Is the Ecosystem Headed

Ciulla expects Rust to grow most significantly in the near term, and his prediction lands in a direction that surprises most of the Rust community. “I think the next big wave might be in web development,” he says, adding that he is aware this is an unpopular position in a community that still thinks of Rust primarily as a systems language. His reasoning is grounded in what he has been seeing directly: companies with hundreds of developers reaching out to tell him they are moving services to Rust for their web backends. “I get this news because I’m well known for talking about Rust and being quite vocal about it,” he observes. “I’m not talking about a person just doing this on a random Saturday night. I’m talking about companies that have hundreds of developers.” He points to Axum as the framework that has matured to the point where he would now use it in a production SaaS product, which he says was not true two years ago. Embedded systems, in his view, have already crossed the threshold where Rust’s place is settled.

Williams takes a longer view on how the ecosystem will evolve. “The ecosystem is going to get richer and people are going to be branching out in the set of use cases, hitting areas that right now Rust has relatively weak support for,” he says. “As larger and larger projects are built, there is going to be more refinement of the language itself, but more importantly, more refinement of the use of the language.” The patterns that make Rust work well at scale are still being discovered and codified. The language is stable, but the understanding of how to use it well is still developing, and that development is happening inside the teams building the largest Rust codebases.

What both conversations point toward is a language whose difficulty and whose value come from the same source. Rust is hard to learn for experienced engineers because it refuses to accommodate the habits that made them experienced. It is valuable in production because that same refusal, enforced by the compiler on every build, produces code whose behavior is predictable, whose data flows are legible, and whose failure modes are constrained to things the language cannot check rather than things the engineer forgot to check. The engineers who get the most out of it are the ones who stop trying to carry their existing instincts across and start letting the compiler teach them what the program actually needs.

---

In case you missed

Here’s the full interview video featuring Evan Williams.