Use your agents. Keep your IDE. Stay the architect.
Air in JetBrains IDEs lets you run several AI coding agents in one place: you pick the agents, and you review every change before you accept it. Because agents can use IDE tools such as inspections, tests, and the debugger, they make fewer wrong turns.
For a limited time, try Air free with Junie Lite.
✍️ From the editor’s desk
Welcome to the 72nd issue of Deep Engineering!
On November 9, Cloudflare published an experimental Codex harness that preserves conversations, files, and completed model and tool results across restarts. Completed steps reuse their recorded results after an interruption, while unfinished steps may run again. The recovery behavior depends on a durable record of the work.
That history also matters when an agent hands off its work or compresses an old conversation. In a November 10 preprint on evidence-linked agent memory, Yukyeong Lee and colleagues describe WikiTrace, which connects each saved lesson to the original run behind it. The agent can inspect that record when a lesson leaves out details or conflicts with another. For teams building agent memory, this gives a practical reason to keep the underlying history accessible after summarization.
Tun Shwe, Staff AI Engineer at Lenses.io and co-organizer of PyData London, works with teams building agent harnesses on Kafka. In today’s issue, we feature an excerpt from his practical deep dive, where he shows how an append-only event log gives a harness the history to rebuild any context window, debug a failed session, recover what a handoff dropped and replay old workflows against new models.
Let’s get started.
Close the loop on agent-written code
Your coding agent writes the mobile feature, then waits for someone to test it on a phone. Expo Simulators removes that wait. Each agent gets its own cloud iOS simulator, where it installs the build and drives the app. You review a PR with a session recording attached.
🧠 Expert Insight
Build AI Agent Memory as an Append-Only Event Log
From a Deep Engineering live session with Tun Shwe.
I lead AI at a company called Lenses, and I love working on open source projects. That’s probably why you see me at a lot of conferences giving talks and workshops. I’m based on the coast in the southwest of England, about five hours from London, where I can go surfing, and I’ve created a PyData community down here called PyData Cornwall. I’m also a co-organizer of the annual PyData London conference.
Before we get to the event log, we need to talk about what a harness is, so we can see how an event log fits into the harness and into agent memory.
An agent is a model plus a harness
Vivek Trivedy, who leads product and research at LangChain, puts it like this. An agent is a model plus a harness, and if you’re not the model, you’re the harness. The harness is everything that sits around the model, which is the code, the config, the whole infrastructure. You and I probably rent the model from a frontier lab to do most of our work, and that doesn’t leave us with much control when we’re deploying agents. So we need to build ourselves a good harness that gives us that control and that reliability.
I talk about a harness in terms of five components. A context manager, a tool registry, guardrails, a loop and a verifier. The context manager is the system that builds the context window and manages everything around it, and its most important job is to create the context window for the agent at each turn. You can think of harness engineering itself as prompt engineering plus context engineering. The tool registry is where you register local tools, scripts, MCP servers and MCP tools. Guardrails block certain actions and make sure the system doesn’t behave other than how it should. The loop is how an agent keeps progressing through its work and knows when to stop once its goal is achieved. And the verifier is the system that checks that everything the agent says it has done is true. You’ve probably heard of hallucinations, and you’ve probably seen long-running agents claim to have done things that turned out not to be true when you checked manually. That’s where a verifier comes in.
The model only sees the window you assemble
The case for building agent memory in an event log starts with the fact that a model is stateless. Every single call starts empty, and the model knows only what the harness puts in the window. If you use coding harnesses like Claude Code or Codex, you’re leasing the model from Anthropic or OpenAI, and they’re providing you with a harness that lets you use that model correctly. But every single call you make still starts empty.
So the window is working memory for one single call. Long-running agents outlive any single window. They crash, they get redeployed, and sometimes they wait days for a human to review something and hand work off to other agents. Stuffing more memory and more history into that window makes the model measurably worse and makes it more expensive. So memory has to live outside the model, and that’s why I came up with the idea of an event log as the long-term memory part of the equation.
An event log is an append-only log of facts
The most useful way to think of an event log is as an append-only log of facts. The harness writes every single message, every tool call, every tool result, every guardrail decision and every human approval into the log, and nobody ever edits it. It’s immutable. You can’t change it or update it, so it remains a fact. If you’re thinking in terms of observability, you can register every conversation that’s happening, and you can log every tool call along with the inputs, the payloads, the outputs and the tokens consumed. You can log a stop token too, which is how a lot of the harnesses you use know when the job is done and when to hand back to you.
Above the log sits the window. You assemble it, it’s the context window for the agent, and it’s what you can consider short-term or working memory. You project it from the log itself. The log has the whole series of facts, everything that’s ever happened, and from that you project working memory. That includes the tool calls and the current conversation, but you also merge in things like your AGENTS.md and the skills you’re loading. As working memory grows, you’ll see the percentage of your context window growing in some of these tools, and it has to go through a compaction event to get back to something workable.
Having a log behind the window means you can rebuild the window from scratch. It answers questions like how we got here and where we are now. The beauty of a log is that it has timestamps, so you can go back in time and reconstruct anything from memory.
Why Kafka fits
A log is anything that’s an immutable, growing list. If you read back to when Anthropic talked about long-running agents and how its harness design evolved, you’ll see it started off with a log, using git log and local files that were being appended to. There are many systems you can use for this.
Kafka fits the design really well because Kafka is a distributed, replicated commit log. It maps directly onto an agent’s event log. You can think of a session topic keyed by session ID, which gives you strict ordering within a session, so events are read out by the consumer in the same order they happened. You can retain it for as long as you configure. Kafka’s default retention is seven days, but you can make it 365 days or as long as you want. And now that the Kafka architecture is moving to diskless, where retention is offloaded into object storage, you’re trading a couple of hundred milliseconds of latency for really cheap storage and long-term memory.
The decoupling between a producer and a consumer means any number of independent consumers, such as the context manager, an auditing agent or an archival agent, can all read the same history at their own pace. Kafka also plays a second role. A lot of large enterprises already have Apache Kafka somewhere in their infrastructure, and the job it’s already doing is carrying all the live business events. Being able to use existing infrastructure matters if you want to start building an event log and memory for your agents.
Realistically, if you don’t already use Kafka in your company or your team, you probably don’t need it on day one. It starts making sense when you need to add a second consumer. Teams building harnesses usually start by solving their own problems, because a harness solves the shortcomings of the model. It starts making sense when you build for a team, when this business team needs to tap into the same knowledge bases and that other team needs to tap into the same event log to understand what happened.
This is key for handoffs. It’s normal in an organization to have agents running in different teams, and you’re going to have handoffs. With an event log, the next agent in the chain doesn’t just get the context handed to it by the previous system. It can rebuild and seek the whole context, and see everything that led up to the point of handoff.
The lakehouse variant
The next view is something I came up with a few weeks ago while working with teams that use lakehouses, with Apache Iceberg over an object store. In some cases you project directly from the log itself. In other cases you sync the contents of the log, so all those messages go to Iceberg on S3 through an Iceberg Kafka Connect sink, and you project them into the window using a query engine and some code in between that also pulls in files like your AGENTS.md and your system prompt.
Continue reading→ the full practical deep dive on DeepSeek’s session log, the four types of agent memory, event schema design, debugging with offsets, snapshots at handoffs and replaying old logs against new models.
🛠️ Tool of the Week
Debezium publishes committed database changes and outbox events to Kafka, giving other agents and context managers a reliable record of what an agent changed.
Capture committed database changes from PostgreSQL, MySQL and other supported databases.
Publish explicit application events from an outbox table without a separate application-to-Kafka write.
Carry event IDs for deduplication and use application identifiers as Kafka message keys.
Route events into separate topics, with support for JSON and Avro payloads.
📎 Tech Briefs
Anthropic breaks down the components of agent memory - Raw transcripts form the base layer, and an offline pass corrects memory notes against them.
Codex CLI 0.167.0 keeps sessions readable across provider switches - Chats that can’t resume on a new provider still open as readable transcripts.
OpenShell and Agent Substrate join the CNCF Sandbox - Both are vendor-neutral runtimes that isolate agent code and gate its access to credentials.
Google adds a Memory Bank console to Agent Platform - Teams can inspect memory revisions and set retention for deployed agents without API calls.
Nanocode’s agent harness now runs in Cloudflare Durable Objects - Each turn’s progress and tool results persist to SQLite, so interrupted sessions resume.
That’s all for today. Thank you for reading this issue of Deep Engineering.
We’ll be back next week with more expert-led content.
Keep building,
Saqib Jan
Editor-in-Chief, Deep Engineering
Sponsor Deep Engineering
If your company wants to reach senior developers, software engineers, and technical decision-makers, speak to us about partnering with Deep Engineering