by Tun Shwe, Staff AI Engineer at Lenses.io and co-organizer of PyData London.
I lead AI at a company called Lenses, and I love working on open source projects. That’s probably why you see me at a lot of conferences giving talks and workshops. I’m based on the coast in the southwest of England, about five hours from London, where I can go surfing, and I’ve created a PyData community down here called PyData Cornwall. I’m also a co-organizer of the annual PyData London conference.
Before we get to the event log, we need to talk about what a harness is, so we can see how an event log fits into the harness and into agent memory.
An agent is a model plus a harness
Vivek Trivedy, who leads product and research at LangChain, puts it like this. An agent is a model plus a harness, and if you’re not the model, you’re the harness. The harness is everything that sits around the model, which is the code, the config, the whole infrastructure. You and I probably rent the model from a frontier lab to do most of our work, and that doesn’t leave us with much control when we’re deploying agents. So we need to build ourselves a good harness that gives us that control and that reliability.
I talk about a harness in terms of five components. A context manager, a tool registry, guardrails, a loop and a verifier. The context manager is the system that builds the context window and manages everything around it, and its most important job is to create the context window for the agent at each turn. You can think of harness engineering itself as prompt engineering plus context engineering. The tool registry is where you register local tools, scripts, MCP servers and MCP tools. Guardrails block certain actions and make sure the system doesn’t behave other than how it should. The loop is how an agent keeps progressing through its work and knows when to stop once its goal is achieved. And the verifier is the system that checks that everything the agent says it has done is true. You’ve probably heard of hallucinations, and you’ve probably seen long-running agents claim to have done things that turned out not to be true when you checked manually. That’s where a verifier comes in.
The model only sees the window you assemble
The case for building agent memory in an event log starts with the fact that a model is stateless. Every single call starts empty, and the model knows only what the harness puts in the window. If you use coding harnesses like Claude Code or Codex, you’re leasing the model from Anthropic or OpenAI, and they’re providing you with a harness that lets you use that model correctly. But every single call you make still starts empty.
So the window is working memory for one single call. Long-running agents outlive any single window. They crash, they get redeployed, and sometimes they wait days for a human to review something and hand work off to other agents. Stuffing more memory and more history into that window makes the model measurably worse and makes it more expensive. So memory has to live outside the model, and that’s why I came up with the idea of an event log as the long-term memory part of the equation.
An event log is an append-only log of facts
The most useful way to think of an event log is as an append-only log of facts. The harness writes every single message, every tool call, every tool result, every guardrail decision and every human approval into the log, and nobody ever edits it. It’s immutable. You can’t change it or update it, so it remains a fact. If you’re thinking in terms of observability, you can register every conversation that’s happening, and you can log every tool call along with the inputs, the payloads, the outputs and the tokens consumed. You can log a stop token too, which is how a lot of the harnesses you use know when the job is done and when to hand back to you.
Above the log sits the window. You assemble it, it’s the context window for the agent, and it’s what you can consider short-term or working memory. You project it from the log itself. The log has the whole series of facts, everything that’s ever happened, and from that you project working memory. That includes the tool calls and the current conversation, but you also merge in things like your AGENTS.md and the skills you’re loading. As working memory grows, you’ll see the percentage of your context window growing in some of these tools, and it has to go through a compaction event to get back to something workable.
Having a log behind the window means you can rebuild the window from scratch. It answers questions like how we got here and where we are now. The beauty of a log is that it has timestamps, so you can go back in time and reconstruct anything from memory.
Why Kafka fits
A log is anything that’s an immutable, growing list. If you read back to when Anthropic talked about long-running agents and how its harness design evolved, you’ll see it started off with a log, using git log and local files that were being appended to. There are many systems you can use for this.
Kafka fits the design really well because Kafka is a distributed, replicated commit log. It maps directly onto an agent’s event log. You can think of a session topic keyed by session ID, which gives you strict ordering within a session, so events are read out by the consumer in the same order they happened. You can retain it for as long as you configure. Kafka’s default retention is seven days, but you can make it 365 days or as long as you want. And now that the Kafka architecture is moving to diskless, where retention is offloaded into object storage, you’re trading a couple of hundred milliseconds of latency for really cheap storage and long-term memory.
The decoupling between a producer and a consumer means any number of independent consumers, such as the context manager, an auditing agent or an archival agent, can all read the same history at their own pace. Kafka also plays a second role. A lot of large enterprises already have Apache Kafka somewhere in their infrastructure, and the job it’s already doing is carrying all the live business events. Being able to use existing infrastructure matters if you want to start building an event log and memory for your agents.
Realistically, if you don’t already use Kafka in your company or your team, you probably don’t need it on day one. It starts making sense when you need to add a second consumer. Teams building harnesses usually start by solving their own problems, because a harness solves the shortcomings of the model. It starts making sense when you build for a team, when this business team needs to tap into the same knowledge bases and that other team needs to tap into the same event log to understand what happened.
This is key for handoffs. It’s normal in an organization to have agents running in different teams, and you’re going to have handoffs. With an event log, the next agent in the chain doesn’t just get the context handed to it by the previous system. It can rebuild and seek the whole context, and see everything that led up to the point of handoff.
The lakehouse variant
The next view is something I came up with a few weeks ago while working with teams that use lakehouses, with Apache Iceberg over an object store. In some cases you project directly from the log itself. In other cases you sync the contents of the log, so all those messages go to Iceberg on S3 through an Iceberg Kafka Connect sink, and you project them into the window using a query engine and some code in between that also pulls in files like your AGENTS.md and your system prompt.
A question that comes up is whether you can still replay a session once it ages out of the topic, or whether replay is limited to Kafka’s retention window. If you’re retaining lots of events in Kafka, that log is going to grow and get expensive, so a lot of teams have moved to diskless Kafka, which offloads the events that aren’t hot into object storage, and I’d recommend that. Some of those solutions can write it as Iceberg, so if you already have a lakehouse you can tap into it immediately. Others use a sink connector. Either way, you project the working session from that log, so the log is always your source of truth. You can always replay from a specific point in time using Kafka, and because Iceberg has time travel, you can replay in Iceberg as well.
A harness that validates the design
DeepSeek open sourced its harness in August, and it’s a really good one to study. It’s composed of plugins, which is a good way to think about a harness, because when you want to add other bits of context, you do it by adding plugins. Its session package is good validation for having an event log. You have a session, which is the event log I showed in the diagram, you append everything to it with session.append, and it has a session.deriveMessages function that projects the messages the model sees. Other harnesses follow the same pattern, and LangChain’s Deep Agents harness also uses a session log under the hood.
The model never reads the event log directly, because the log is part of the harness. Your tooling reads from the event log and provides it to the agent by building the context window. The model only sees the context window you built for it.
Memory is a whole system
For memory, you have to think of the whole system. A harness manages short-term memory, which is the working memory and is usually in process. It manages long-term memory, which lives in things like vector databases and MEMORY.md files. It manages episodic memory, which is what the event store holds, the events that form the episodes of what’s been happening. And it manages semantic memory, which is derived from the other three, mainly from the episodic memory. A good example is user preferences. A user takes an action that goes into the event log, say changing their preferred currency to GBP, and that gets pulled out into semantic memory so you can answer in the right format with the right preferences.
In a multi-agent handoff, what you generally do is compact or summarize the context, save it somewhere decoupled, such as a file or long-term storage, and hand off to the next agent, which reads the summary and continues. When that agent hits an error or a scenario it doesn’t expect, it has the option to go back in time. It goes directly to the event log, finds the offset from the start of the day or from when the first agent in the chain started its task, rewinds to that period, reads in all the messages, and figures out for itself what happened.
We know key information gets missed in handoffs. Chroma’s context rot report from July 2025 showed that models get less reliable as you stuff more tokens into the window, even on simple tasks, and the signal gets harder to find in all that noise. So you keep the most important pieces at the edges, with the system prompt at the start and the user request at the end. I’ve seen teams where fulfilling an order required three criteria, and the next agent received only two, because the summarization step dropped the third. You can rerun the whole chain from the beginning, burn more tokens and maybe get a different result. But when you’re building resilient and efficient agentic systems, you want some degree of replayability, and for that you sometimes have to go back to the log.
Designing the events
How much you log is up to you, because you build the context manager yourself. I always recommend logging the type of interaction, whether it’s a user request, a call to the LLM or a tool call, along with the decision the agent made and the context it had available to make it. I store tool results inline, directly in the log, even when they’re big JSON files, because it makes exploring and auditing easier. Binaries definitely go in the object store. If an agent produces an image, the image goes straight to S3 and the log holds a reference to it. In Kafka that’s called the claim check pattern, like checking in your coat at a conference and getting a ticket from the counter. There’s a limit on the size of a Kafka message, so for things like PDFs you write the file to object storage, get a URI, and put that into the message so you always have the pairing.
For ordering, I save the session ID and another identifier, such as the agent or job ID. When you have parallel agents writing to the log at the same time within the same session, you need two layers. When I’m debugging, the end-to-end sequence is what matters, so you need to know the order these things happened in. How many agents you run in parallel determines the design of your schema.
If agents produce a lot of events, you scale Kafka the way you always have, with partitioning. Banks run their transactions through it and manufacturers ingest telemetry from all their robots, so volume hasn’t been a problem. At IBM’s Think conference in May, I heard the ratio of engineers to agents put at one to twenty within the next twelve months, and that feels realistic when a simple research task in Claude Code can spawn four research sub-agents.
Debugging means rebuilding what the model saw
In a demo, when something goes wrong, you run it again until it succeeds. In production you can’t do that. The same inputs won’t necessarily give the same output, the model behind the agent might have changed since last week through a gateway you don’t control, and sometimes the failure happens one session in a thousand. So the event log becomes the only source of truth, the only thing that tells you this happened.
Debugging means rebuilding exactly what the model saw at the step where it went wrong. You resume a backlog of sessions after a model gets swapped or a provider has an outage, you find the offsets for the bad session’s events, you quarantine that one session, and through that isolation you rebuild the memory of what happened. Sometimes it was corruption, sometimes a bad payload, sometimes poisoned context, where something irrelevant got inserted into the window and led the agent down the wrong path. Those are the sessions where you see really long loops, like a rogue sub-agent in your coding harness that gets into a funk and can’t complete its task. Over time you get to stop trusting the agent’s own account of what happened, because you have the log and you’ve built a verifier. Reading the log is reading the truth.
Snapshots at handoffs
You tend to snapshot at handoff points, because that’s where you want to make sure the next agent has the right context. The event log stays the source of truth, but the semantic memory, the user preferences and the notable things that should be retained, needs a log of its own too, so it’s present when you snapshot and hand off. In the very early days, just write a handoff.md file. That’s the best thing you can do, and then build some evals from there, so at least you have artifacts you can go back to and read. When you get more confident, write it into another topic, something like a session summary topic.
Replay old logs against new models
Someone asked whether sub-agents could extract lessons from these logs instead of waiting for an agent to review them. That’s exactly how teams are building self-improving agents. The event log tells you everything that’s happened, and it’s also how you start building evals. The wrong thing to do is to start writing evals. What you need first is exploratory data analysis to figure out your ground truth, and the log is a good source for seeing previous failures and what worked.
When a new model comes out, run your old workflows through it, with exactly the same payloads, against your existing harness, and compare the outcome with your previous event logs. You might find it figures things out in fewer steps, with fewer tokens or with fewer agents. You might find a skill you built is now baked into the model, so the agent never calls it, which means you can remove that skill definition from the context window. Have a really good set of evals, because that’s the thing you fall back to every time a new model comes out.
Everything from the session is in the open source agentic-engineering-for-apache-kafka repo, which includes skills for Kafka you can use through any MCP server that taps into it. And if you want a local stack to play with, Lenses Community Edition is a Docker Compose that brings up a Kafka cluster with demo data, so you can start understanding what it means to have an event log and build long-term agent memory from there.
Editorial note: This article is adapted from Tun Shwe’s Deep Engineering live session, Event-Driven Agent Memory: Kafka as the Event Log for AI Agents, hosted by Saqib Jan. The session recording and slides were condensed and reordered for print.
One tension with "nobody ever edits it": a log full of user messages and tool payloads will eventually get a GDPR erasure request, and a delete-policy topic can't remove one session's records. Encrypting each session's events with its own key and deleting that key on request (crypto-shredding) keeps the log immutable and still erasable.
Watch the cleanup policy on that session topic. Someone flips it to compact to save storage, and since every event is keyed by session ID, Kafka keeps only the latest record per key, so a session's whole history shrinks to its last event. Pin cleanup.policy=delete with a long retention.ms on memory topics and alert if either changes.