By James Ward, Agent Experience at AWS. Creator of WebJars, co-author of Effect-Oriented Programming. | This deep dive is based on his Deep Engineering live session, edited by Saqib Jan.
I work on agent experience at AWS, which means making it easy for you to build on AWS from inside your AI agents. The other part of my job is the new Agentic AI Foundation, where MCP, AGENTS.md, Goose and Agent Gateway are being standardized under the Linux Foundation. I represent Amazon on that technical committee.
Let me start with the thing that makes all of this necessary.
Go to your agent and ask it what the current weather is. It will tell you it does not have access to real-time weather data or your location, and suggest you look out of the window. Ask it what time it is and you get the same shape of answer, because it does not have a clock. That is not a failure. By default, LLMs have no access to external things. They are just a model.
The way I think about what an LLM actually is: a knowledgeable translator. It translates natural language to natural language, natural language to an image, an image to natural language, natural language to structured data, structured data to structured data, natural language to a programming language. That is all it does. Everything else you want from an agent has to be built around it.
An agent, then, is not complicated. You take an environment, usually a history of messages plus whatever else you want to carry. You take tools, the things you want to give the model access to. You add a system prompt to give it a goal or a personality. And then the agent runs in a loop until it decides it has achieved what the user asked, or that it cannot.
Everything below is how you fill in those tools. All the code is at github.com/jamesward/agent-integration-demo, and I am using Spring AI and Java throughout.
Inference is the easy part, and it is worth seeing what is on the wire
Spring AI hit 1.0 a little over a year ago, and it gives you one abstraction across model providers. I am using AWS Bedrock with the Converse API and the Nova Pro model, but there are around 250 models on Bedrock and you could just as easily point this at Ollama running locally. The provider is a config change.
In the application you inject a ChatClient.Builder rather than a concrete client, and the reason matters. In a real system you will want several chat clients, one on a large model and one on something faster and cheaper, and your agentic architecture will route between them.
The basic call is a user prompt, .call(), and .content(). Ask it to say hello and it says hello.
More useful is structured output. Define a Java record, annotate a field with @JsonPropertyDescription("most popular food"), and ask for a List<City> back using .entity() with a ParameterizedTypeReference because of how Java generics are reified. What happens underneath is that the record’s metadata gets sent to the model so it knows how to shape a response that will deserialize cleanly. When you are building agentic applications, it very often makes more sense to interact with the model through structured data than through free-form text.
I want to demystify what is actually happening on those calls, because the API is concise and what goes over the wire is not. Run it in debug mode and you see the request carries the message, the media slots for images, the max tokens from your settings, the model name, a pile of defaults and the system message. The response carries metadata about rate limits, prompt tokens in, completion tokens out, total tracked, and one field worth knowing by name.
finish_reason. On a simple call it comes back as end_turn, which is the model saying it has done what you asked. Hold on to that, because the entire agentic loop turns on it becoming something else.
Spring AI wires token metadata into Actuator and Micrometer, so you can push those metrics wherever you already send metrics.
Streaming is a one-word change. Swap .call() for .stream() and you get a Flux of chunks instead of waiting for the whole response to assemble. And a system prompt gives the whole interaction a personality or some grounding. Mine was “you are a Wookiee from Star Wars,” and it growled at me. One caution: you cannot rely on system prompts to protect a system from being used for things you did not intend. More on that later.
Tool calling is a four-step conversation, and the model never touches your tool
Now ask what time it is, with no tools defined. The model tells you it has no access to a clock. This is the wall.
Here is what actually happens when you get past it.
You send the user message to the model, and alongside it you send metadata describing the tools you have. Just the name, the parameters, the description. The tool itself is not on the model’s side. It is on your application’s side. You are saying, here is what the user wants, and here are some things I can do if you decide you need them.
The model looks at the request and responds. And the finish_reason this time is not end_turn, it is tool_use. The model is telling you which tool it needs and what parameters to pass. For a weather question that is get_weather with a city name.
Your application invokes the tool. No model involvement at all in this step.
Then you call the model again with the whole history: the original question, the fact that it asked for a tool, and the result you got back. Now it assembles a real answer and comes back with end_turn.
The key thing to hold on to is that the LLM never invokes anything. You define and call the tools. The model only tells you which ones it wants.
In Spring AI this is short. Annotate a method with @Tool and a description, then pass the containing object to .defaultTools() on your chat client. The description is doing real work, because that is what the model reads when deciding whether the tool is relevant.
@Tool(description = "Get the current date and time in the user's time zone")
String getCurrentDateTime() {
return LocalDateTime.now()
.atZone(LocaleContextHolder.getTimeZone().toZoneId())
.toString();
}The API for calling this is identical to the basic inference call. Same .call(), same .content(). Spring AI uses the same API for a single inference as for a full agentic loop, and the loop just keeps going until it reaches end_turn, making however many tool calls it needs on the way.
MCP is the microservices version of the same thing
MCP has become the standard way to do tool calling, and you have probably used it in a code assistant with a local server. Plenty of businesses now publish their services as MCP too.
It works over HTTP, so servers can be remote. Underneath it is JSON-RPC carrying a method, a tool name and arguments. That is genuinely all it is. MCP is remoting for the tool calling I just described.
In Spring AI, configure the client with the streamable HTTP protocol and a URL. I pointed mine at a JavaDocs MCP server I built, which gives access to all the Javadocs on Maven Central. Then inject a ToolCallbackProvider, hand it to .defaultTools() alongside your local tools, and the wiring is done. Ask for the latest version of a library and it calls the tool rather than guessing from training data, which is the difference between a correct version number and a plausible one.
You can build MCP servers with Spring AI as easily as you consume them. But I would push back on wrapping everything. Most real architectures are a mix of MCP tools and local ones, and there are good reasons to keep tools local. Something like getting the current date, or doing arithmetic, belongs in the same process as your agent. My general approach to architecture is to start with a monolith, build it so it can become microservices, and only break things out when you need to. The same reasoning applies here exactly.
The portability is straightforward when you do need it. Take a tool written in Spring, pull it into a separate project, expose it as an MCP server, and the code barely changes. Swap @Tool for @McpTool if you want the MCP-specific features, though @Tool will work as it is. And the security model stays the same, so user identity flows to MCP tools the same way it flows to local ones, which is usually the painful part of that kind of migration and here is not.
Too many tools is a token problem and a reasoning problem
Here is what breaks at scale. Every request to the model carries the metadata for every tool you have. With a hundred tools that is a lot of tokens spent describing capabilities before the model has done anything. And it gets harder for the model to pick the right one as the list grows.
The technique that addresses both is tool search, and most AI coding agents now do this implicitly.
You take each tool description and generate a vector representation of it. That is what an embedding model does: you give it a string and it gives you back an array of numbers. I am using Titan Text Embeddings V1 on Bedrock for this, storing the results in Spring AI’s SimpleVectorStore, which is in-memory and fine for a demo but should be something persistent in production.
Then you use an advisor. An advisor in Spring AI is middleware for your model calls, letting you intercept the request on the way out and the response on the way back. The tool search advisor intercepts the outbound call and replaces your full tool list with exactly one tool: the tool search tool.
So the model sees one tool, decides it needs to find something that can generate a random string, calls the search tool, gets back a vector match, and then on the next round the random string tool is available and it calls that. More round trips to the model, considerably fewer tokens, because you were never shipping twenty-one tool descriptions on every request.
That is one strategy. There are others. You can group tools and give different parts of your agentic flow access to different groups, which is where multiple chat clients come in. Or look at Embabel, an agent framework built on top of Spring AI by Rod Johnson, who created the Spring Framework. It has a concept called unfolding tools that progressively loads domains of tools, similar in spirit to semantic search but organized around domains rather than similarity.
One note on where this lives. Tool search started in the Spring AI community repository, which is where more experimental work goes, and as of Spring AI 2.0 it is part of core. Christian Tzolov, who leads Spring AI, wrote up the migration notes with good detail on how it works.
Skills are progressive disclosure for instructions
Agent skills are a standard for describing additional knowledge or process you want to give an agent. You have probably added one to your code assistant. They work in a business domain too, as a way to encode process and guide an agent in a specific direction.
The format is markdown with front matter. The front matter is a small amount of metadata about what the skill is, and by default that metadata is all that gets sent to the model. The full body only loads when the model asks for it, through a companion tool.
My friend Josh Long and I built a dog adoption service demo called Pooch Palace, and it has a dog breed skill. Josh and I happen to know a lot about Chihuahuas, including what they say, which is “Chihuahua.” That is encoded in the skill, and no model is going to produce it from training data.
You can preload the whole skill file into the system message, and it works, and it costs tokens on every request whether or not the conversation has anything to do with dogs. The better approach is the skills tool, where you add a classpath resource directory and register the tool. The model gets the front matter, decides whether it needs the body, and calls the tool to fetch it.
There is a real caveat here, and I hit it live. Sometimes the model decides it does not need the skill. I asked whether Chihuahuas have demonic tendencies, and the model figured it knew enough about that already and never loaded my skill. If I had asked about an expense reporting policy and had a skill containing that policy, it would almost certainly have loaded it. But this is nondeterministic and you should expect it to be.
I also built something for reusing and versioning skills, which packages them into JAR files you can manage as normal dependencies. I published the Pooch Palace skills to Maven Central. Add the dependency, point a classpath resource at the META-INF/skills directory, and everything pulled in as a dependency becomes available through the same skills tool.
Memory is where this gets genuinely hard
Ask the model your name, then in a second call ask what your name is. It has no idea. There is no memory between requests.
There are two broad strategies. Short term memory assumes the last n messages matter and sends them every time. Long term memory takes messages as they pass through and does compaction, extraction or categorization to pull out what seems worth keeping, then sends that alongside the recent window.
Memory is one of the more challenging parts of building these systems, because deciding how many messages belong in the window, and what deserves to be preserved long term, is genuinely difficult. There are services like AgentCore Memory that handle a lot of it.
The simple version in Spring AI is MessageWindowChatMemory with a window of ten, wired in through an advisor, which is the natural place for it given advisors already intercept everything going both directions. You need a conversation ID as the key into the memory store. I hard-coded mine, but in a real system that is a user ID or whatever principal you have after authentication, and the only requirement is that it stays the same across the calls that should share memory.
Storage is pluggable. In-memory for a demo, JDBC against a database for anything real.
RAG decides for the model, tool calling lets the model decide
RAG has been around a while and it is still the standard way to inject data into a prompt.
The distinction that matters is this. With tool calling, the model asks you for data. With RAG, you decide before the model sees anything. You run a vector search against the user’s prompt, find data that looks relevant, and append it whether the model turns out to need it or not.
The mechanics: generate embeddings for your data, usually on create or update of the record. Store those vectors somewhere built for searching them. When a query arrives, embed the query, run a cosine similarity search, take the top few results, and append them to the prompt.
In Spring AI you put your data into a vector store as Document objects and add a QuestionAnswerAdvisor. I had three bank accounts, generated embeddings for each at startup, and asked for my checking account number. The advisor found the relevant accounts and injected them into the prompt with no tool calls at all.
I could have exposed accounts as a tool instead. The RAG style makes sense when you can reasonably assume the data will often be relevant. In a banking chat application, people ask about their accounts, so include them when the prompt looks like a match.
There is a further pattern in the repo I did not demo, where instead of treating embeddings as a copy of your data you correlate the RAG results back to actual rows in a database. Worth a look if your data changes underneath you.
Human in the loop, when the agent needs to ask
Sometimes you need more information from the user mid-flow.
MCP supports elicitation for this. A user asks to search flights from Denver to San Francisco tomorrow, the flight search tool gets invoked, and the tool itself says it needs a preferred airline it was not given. The MCP server elicits that from the user, gets a response, and continues the tool call.
The important qualifier: this is only for things you sometimes need. If you always want the preferred airline, make it a tool parameter and collect it every time. Elicitation is for the case where you might already know it and might not.
Spring AI has its own version through AskUserQuestionTool, which takes a question handler. Mine used standard in and standard out for the demo, but you would wrap this in WebSocket messages or whatever your interface actually is. With a system prompt listing the user’s accounts and that tool registered, asking for “my account balance” makes the model realize it does not know which account, ask, and continue with the answer.
Questions from the session
On tool search without embeddings. You can use a small model instead of an embedding model to do the same selection. It costs tokens every time, where an embedding only has to be generated once per tool description. Spring AI enables this through recursive advisers, which let you make another model call from inside the advisor chain, potentially to a different model. Part of the saving is that on that secondary call you do not have to send the whole message history.
On guardrails and the boundary between system and user prompts. There is some boundary, and how it is weighted depends on the model. You should not rely on it. System prompts give you a little protection and not much more. If you want real protection, use actual guardrails. Spring AI has a client-side guardrail system, and model providers have their own. Bedrock’s has several kinds depending on whether you want semantic guardrails or something more provable. And it is worth remembering tokens are a resource to protect. Put a chatbot on the internet and people will use it to do their homework.
On analyzing large log volumes. Expose search as a tool and let the agent probe. Watch what a code assistant does on a filesystem and you will see a lot of grepping and finding before it commits to anything. It will check how many results a search returns and refine until the set is small enough to work with. Build probing tools that let the agent narrow down. If you are on CloudWatch or Datadog, there are already MCP servers doing this.
On temperature. Temperature is one knob among several, and the way you find out where it should be set is evals. An eval sets a task, an environment of available tools, and a criteria for success. You run it, take the full transcript of what happened, and give that to a different model with the question of whether the user’s goal was accomplished. That is LLM as judge. You run it many times because of the nondeterminism. Evals are how you know anything about whether your agent is working, including whether a tool description needs rewriting. Spring AI has an eval system to build on.
On Spring AI’s maturity. The JVM enterprise community was late to this, and Python is where a lot of it started. But most enterprise business logic is already in Java and Spring, and those organizations do not want to move it. Spring AI is the congruent choice, and it is genuinely good technology rather than a compromise. If you want higher-level abstractions, Embabel adds unfolding tools and DICE, domain integrated context engineering. I also built ai4jvm.com, which catalogs the JVM AI ecosystem, and it is more extensive than people expect. We are not behind anymore.
Session notes
All the code is at github.com/jamesward/agent-integration-demo. The skills format is documented at agentskills.io. The JVM AI catalog is at ai4jvm.com.
I did not cover guardrails in depth, evals in depth, or the database-correlated RAG example, all of which deserve their own session.
Find me at jamesward.com or @JamesWard.