Core Architectural Patterns for LLM System Design
Chapter 2 of System Design for the LLM Era, reproduced in full, on the resilience, latency, cost, evaluation, and security patterns behind production LLM systems
Integrating LLMs into a production system introduces a new class of dependency: one that is non-deterministic, high-latency, and carries a high, variable operational cost. The fundamentals of LLM integration, tokens, embeddings, and the basic idea of retrieval-augmented generation (RAG), are only the starting point.
As experienced engineers, we already know how to build reliable systems. This chapter isn’t about reinventing those principles, but about adapting them for the unique, messy challenges LLMs throw at us. Consider this an architect’s playbook that outlines the new patterns needed to meet these challenges.
In this chapter we’ll be looking at the following topics:
Designing for resilience and reliability
Designing for low latency
Designing for cost optimization
Designing for grounding and data management
Designing for testability and observability
Designing for security and trust
Engineering for production
Training with test data
Respecting user privacy
Featured workshop: Loop Engineering for AI Agents
A 4-hour hands-on workshop on Loop Engineering, designing agent workflows with Claude Code, SDD, and MCP that verify their own work and stay observable in production.
Use DEEPENG40 for 40% discount.
Designing for resilience and reliability
Our system’s stability is now tied to an external API that is slower, more expensive, and less predictable than any database or microservice calls. The primary goal is to decouple the application’s health from the provider’s health. Two types of pattern are especially valuable in this regard: the GenAI service pattern (or LLM gateway pattern) and the circuit breaker pattern.
Pattern: the GenAI service or LLM gateway
When we first start building with LLMs, our instinct is to treat them like any other third-party API. We install the SDK, generate an API key, and make the call directly from our application code.
It feels fast. It feels efficient. We write a Python function, import openai, and we are shipping features in minutes.
The architecture looks like this:
While this works for a weekend hackathon, it creates a high-coupling, low-cohesion architecture in production with the following issues:
Vendor lock-in: If Service A is written using the OpenAI SDK, migrating to Anthropic requires rewriting the entire code block.
Inconsistent reliability: Service A might have excellent retry logic, while Service B crashes on the first timeout. There is no standard.
Observability black holes: You have no central place to see how much you are spending. You have to log into three different developer consoles to tally up the bill.
Security risks: API keys are scattered across multiple environment variables in multiple services, increasing the surface area for leaks.
Stating the problem more formally now:
Problem: Our services should not be calling OpenAI, Anthropic, or Google directly. This creates high-coupling, high maintenance problems.
Solution: To solve this, we borrow a pattern from traditional microservices: the API gateway. We stop treating LLMs as external vendors and start treating them as a unified internal resource.
Implement a single, centralized LLM gateway. This is a microservice that acts as the only entry point for all LLM calls. Our services talk only to this gateway using a single, unified API format. The gateway handles the messy details of talking to the outside world.
This pattern brings some significant benefits:
Abstraction: Can switch models (e.g. GPT-5 for Claude 3 Opus) with a config change, not a re-deployment
Centralized control: All other resilience, cost, and monitoring patterns are implemented in this one place
Authentication: Manages all authentications in one service
Fallbacks and reliability: If OpenAI goes down, the gateway can automatically retry the request with Anthropic. The upstream service never even knows there was an outage
Pattern: circuit breakers with tiered fallbacks
An LLM provider might be slow, down, or just returning bad data. Simply retrying a failed call (like you would for a 503 on an external service) is often the wrong move.
The solution is to combine the circuit breaker pattern with tiered fallbacks following a three-stage model:
Monitor: The GenAI Service monitors the health (latency, error rate) of each model provider.
Trip: If a primary model (e.g. GPT-5) exceeds a failure threshold, the circuit opens.
Reroute: All subsequent requests are immediately and automatically rerouted to a backup model.
For example, we might implement the following tiered fallback strategy:
Tier 1: GPT-5 (high-cost, high-reasoning)
Tier 2 (fallback): Claude 3 Haiku (medium-cost, fast)
Tier 3 (fallback): Llama 3 8B (locally hosted, free, less smart)
Tier 4 (final fallback): Return a cached good enough response or a graceful error message to the client UI: ‘Our AI assistant is at high capacity, please try again in a moment’
We cannot leave the circuit open forever. We need recovery logic to close the circuit that includes a half-open state:
Sleep: After the circuit trips, wait for a defined cooldown (e.g. 30 seconds).
Probe: Allow a single canary request to pass through to the primary provider.
Reset: If the canary succeeds, close the circuit and resume full traffic. If it fails, restart the cooldown.
For our retry strategy we have two options: interactive and asynchronous:
Interactive/synchronous: Do not use aggressive exponential backoff. If a user is waiting, a 60-second retry is effectively downtime. Use capped backoff (start 500 ms, max 1 s) or fail fast to a fallback model.
Asynchronous: Use exponential backoff (wait 1 min, 2 min, 4 min). Since no user is waiting, we can afford to wait out a 5-minute provider outage.
Designing for low latency
LLM inference is fundamentally slow. A user requesting a search result expects a response in < 500 ms, but an LLM might take 5–10 seconds to generate a full answer. We cannot change the speed of inference, but we can architect around it. Hybrid processing (synchronous vs. asynchronous), response streaming and caching patterns can help us.
Pattern: hybrid processing (synchronous vs. asynchronous)
Given that we cannot block a user’s web request for 10 seconds while waiting for an LLM, one option is to separate the workloads. This is the most important latency-saving pattern.
Synchronous path (< 2 s): For immediate, low-latency needs. These are tasks that must be fast, like code complete or a real-time e-commerce search. These paths should use fast, cheap models or rely heavily on caching.
Asynchronous path (> 10 s): For high-latency, long-running tasks use a message queue (like Kafka or SQS) to decouple the initial request from the actual LLM work. The client receives an immediate ‘202 Accepted’ response. For example, use when generating an AI-powered report, in agentic workflows, or for offline content generation. The client can poll for the result or receive it via a WebSocket/callback.
Pattern: response streaming
Even a fast 3-second response feels slow if the user is staring at a loading spinner, so if possible stream the response token-by-token. Once the model generates the first word, send it to the client. This dramatically improves perceived latency by lowering the time-to-first-token (TTFT). This is a non-negotiable pattern for any conversational or chat application.
How it works
We use SSE (server-sent events) over standard HTTP. SSE is unidirectional (server -> client) and runs over standard HTTP/2, making it firewall-friendly and easy to implement.
Client: Opens a persistent connection.
Server: Instead of returning a JSON object, it returns a generator (an iterator)
Header: The server must set content-type: text/event-stream
Format: Data is sent in chunks prefixed ‘data:’:
Pattern: caching strategies
LLM calls are slow and expensive, while cache hits are the fastest, cheapest LLM calls you can make. Therefore implement a multi-level caching strategy:
Level 1: exact match
Scenario: A viral product launch. 10,000 users ask ‘When is shipping?’.
Mechanism: Hash the prompt string (sha256(”When is shipping?”)). Check Redis.
Result: 9,999 users get a 5 ms response.
Why: It catches the stampede of identical queries.
Level 2: semantic match
Scenario: User A asks ‘How do I reset password?’ User B asks ‘Forgot password, help’.
Mechanism: L1 misses (strings don’t match). We convert ‘Forgot password, help’ to a vector. We query the vector DB for similar past questions.
Result: The DB finds that ‘How do I reset password?’ has a 0.98 similarity. It returns the cached answer for User A to User B.
Why: It catches different phrasings of the same intent, saving expensive reasoning costs.
Level 3: proactive caching
Scenario: A personalized ‘Daily Report’ for 100,000 users.
Mechanism: Don’t wait for them to open the app at 9:00 AM. Run a batch job at 6:00 AM. Generate the reports and store them into Redis.
Result: When users log in, the AI generation feels instant because it happened 3 hours ago.
Why: It moves latency from online (user waiting) to offline. This is a core pattern in adaptive learning platforms and e-commerce search.
Pattern: coalesce caching
During high-traffic events, thousands of users might ask the exact same question simultaneously. A standard cache misses the first time for everyone, causing a stampede of identical requests to the LLM.
The solution is middleware that identifies identical in-flight requests. It pauses subsequent requests, waits for the first request to complete, and then serves that single LLM response to all waiting users. This reduces load on the provider by orders of magnitude.
Designing for cost optimization
LLMs introduce a new, variable, and unbounded operational cost (COGS – cost of goods sold). A single complex query can cost dollars. Architectural decisions are now financial decisions. Important in this context are the model router, dynamic traffic control and prompt engineering and compression patterns.
Pattern: the model router
Not all tasks require the smartest (and most expensive) model. Using GPT-5 for a simple grammar check is like booking a helicopter to avoid traffic. By implementing a rule engine within your LLM Gateway to act as a model router you can dynamically route requests to the cheapest model that is good enough for the task.
Example rules:
IF task_type == 'simple_grammar_check' THEN route_to 'local_llama_8b'.
IF task_type == 'complex reasoning' AND user_tier == 'premium' THEN route_to 'GPT-5'.Pattern: dynamic traffic control (utilization-based routing)
A static rule for traffic routing, e.g. IF task == complex THEN GPT-5, can cause bottlenecks during traffic spikes. In this case we can use a production-grade router that acts as a traffic controller, implementing load shedding if the primary model’s latency breaches the P99 SLA (e.g. > 2s) due to provider congestion. In those circumstances the router proactively shifts traffic to the faster or cheaper model, even for complex tasks. A cost ceiling can be established by enforcing a hard cap on tokens. If a prompt exceeds a threshold (e.g. 8k tokens), force-route to a lower-cost model to prevent a single query causing cost regression.
Pattern: prompt engineering and compression
Cost is based on the number of input and output tokens; large prompts are expensive. Therefore treat your prompt context as a cost to be optimized. Before sending a large document (e.g. 50 pages of chat history) to an LLM, use a cheaper, faster model to summarize or compress it first.
Designing for grounding and data management
LLMs hallucinate (invent facts) and have knowledge cutoffs (their training data is stale). How can we build a reliable enterprise application on this foundation? Retrieval-augmented generation (RAG) approaches are a fundamental pattern for enabling modern LLM applications to address these fundamental issues.
Pattern: retrieval-augmented generation (RAG)
Not uncommonly we need the LLM to answer questions about private, real-time, or domain-specific data. Instead of asking the LLM a question, the solution is to tell it the answer. As the RAG name suggests, there are three principal aspects to consider:
Retrieve: When a user asks, ‘What’s the status of order #123?’, you first query the database to get the order details.
Augment: You augment the prompt with this retrieved data.
Generate: You instruct the LLM – ‘Based only on the following context, generate a friendly response’.
Example context: {order_details_json}
Pattern: the ingestion pipeline
To build the retrieval component of a RAG application, we must prepare the knowledge base (documents, tickets, etc.) for retrieval. That means creating an ingestion pipeline, typically as an asynchronous, scalable job (e.g. using Spark). This pipeline:
Chunks large documents into small, semantically meaningful pieces
Embeds each chunk by calling an embedding model API
Stores the chunk and its corresponding vector in a vector database (e.g. OpenSearch, Pinecone, or pg_vector [with postgres])
Pattern: hybrid RAG
RAG using just vector search (or even simple term search) is not enough for complex, interconnected data. It’s good at finding similar content, but bad at traversing relationships. Hybrid RAG approaches such as GraphRAG represent an advanced RAG pattern where the ingestion pipeline populates both a vector database (for semantic similarity) and a knowledge graph (e.g. Neptune or Neo4j) for structured relationships.
The RAG orchestrator then queries both systems to build a much richer, more accurate context, as a customer support agent system demonstrates well.
Pattern: function calling (tool usage)
To overcome the problem that LLMs are generally bad at (for example) math and cannot interact with the outside world, we ask it to decide which tool to use. We provide a schema of tools (e.g. get_weather(city)), and the LLM outputs a structured JSON object requesting that function. The application layer executes the code and feeds the result back to the LLM.
Designing for testability and observability
Testability for AI-powered systems is crucial in order to maintain the quality of response expected from the system, but for non-deterministic systems it is not trivial to write unit tests that would assert on a given known output.
We use a pattern of using an LLM as a judge by creating a golden dataset, which has input with their ideal responses, and using that dataset, the LLM judge is trained. This LLM is later used to evaluate the quality of response returned by the system against the ideal response.
Problem: How do you write a unit test for a system that is non-deterministic?
Something like assert(response) == “expected_string” will fail constantly.
Pattern: golden datasets
We need a reliable way to catch regressions in AI quality, and one way is to create a ‘golden dataset’ of 50–100 representative inputs and their ideal outputs. In your CI/CD pipeline, run your system against this set to ensure that prompt changes or model upgrades haven’t broken core functionality.
Pattern: LLM-as-a-Judge
How do you assert the quality of a golden set test at scale? You can’t manually review 100 responses on every build. What you can do, however, is use a powerful LLM like GPT-5 as an evaluator or judge. You feed the judge the original prompt, the golden answer, and your system’s actual response, and then ask the Judge to score the actual response from 1 to 5 on metrics like accuracy, groundedness, and tone. This gives you a quantifiable quality metric you can track over time.
Pattern: new observability metrics
Existing performance dashboards (CPU, RAM, 5xx errors, etc.) are insufficient in the LLM era. We need to add a new layer of monitoring focused on the LLM itself, for example:
Cost: Track Cost_Per_Query and Total_Cost_Per_User. Set alerts for cost spikes.
Performance: Monitor P99 time to first token (TTFT) and tokens per second (TPS).
Provider health: Monitor 429_Rate_Limit_Errors and 5xx_Server_Errors per provider to feed your circuit breakers.
Quality: Track your LLM-as-a-Judge scores. Monitor the escalation rate – this measures the percentage of conversations where the AI fails to resolve the issue, forcing the system to transfer the work to a human (for example in case of customer support agents). A spike in this metric indicates a drop in model quality.
Designing for security and trust
Treating an LLM as a simple API call is naive and dangerous. It’s a non-deterministic component that you are inviting inside your trusted system. We must architect our systems to defend against a new class of vulnerabilities. A variety of patterns can help us in this, including the following. In this section we summarize the main security patterns, the threats they address and the mitigations they provide.
Pattern: mitigating prompt injection
Threat: Attackers trick an LLM into ignoring its original instructions by embedding malicious commands in prompts, causing it to perform unintended actions.
Mitigation
Instruction/data separation: Clearly separate trusted instructions from untrusted data from the user. Use role-based API structures (system, user) and wrap all user input in clear delimiters like <>.
Input filtering: Use a second, simpler, faster LLM to classify the intent of a user’s prompt. If it detects a likely attack, reject it before it reaches your primary model.
Output filtering: Always validate the LLM’s response. If it contains any system prompt text or suspicious keywords, block it.
Pattern: secure output handling
Threat: Insecure output handling occurs when an LLM’s outputs are not properly sanitized before being used, potentially leading to attacks like cross-site scripting (XSS).
Mitigation
Treat as untrusted: Never eval() code output.
Sanitize and encode: If the output is HTML, sanitize it. If it’s text for a web page, encode it to prevent XSS.
Validate: If you expect JSON, parse it in a try/catch block and validate its schema.
Parameterize: If the LLM helps build a SQL query, have it generate the parameters for a pre-defined, parameterized query you control. Never execute a raw SQL string from an LLM.
Pattern: mitigating excessive agency
Threat: This happens when an LLM is given too much control and makes unauthorized decisions or actions without human oversight.
Mitigation
Dynamic permissions: Do not give the LLM agent a static set of all possible tools.
Plan-approve-execute: Implement a multi-step loop. The LLM proposes a plan. Your code approves this plan. Only then do you execute it with a scoped-down client.
Human-in-the-loop (HITL): For high-impact actions (e.g. ‘delete database’, ‘refund customer’), always require explicit human approval.
Pattern: mitigating sensitive information disclosure
Threat: The model may inadvertently reveal confidential data or personally identifiable information (PII) that was present in its training data.
Mitigation (data hygiene)
PII/data scrubbing: Aggressively sanitize all data before it is used for training or RAG ingestion.
Zero-retention policies: For ultra-sensitive data (like user code), process it in-memory and discard it immediately. Do not store it.
Tenant-level RAG filtering: This is a critical architectural pattern. All RAG queries must include a filter for tenant_id or user_id. Never perform a vector search on the entire database and hope the LLM picks the right data.
Pattern: preventing model denial of service (MDoS)
Threat: Attackers overload the LLM with resource-intensive requests, slowing it down or making it unavailable.
Mitigation
API rate limiting: Enforce strict per-user and per-IP rate limits at the API gateway
Input validation: Reject queries that are obviously abusive (e.g. a 50,000-token prompt)
Cost-based throttling: Implement logic to monitor the cost of a user’s queries in real-time. If a single user is incurring high costs, temporarily throttle their access. We would want to return a 429 in case of user spam/abuse or redirect to a lower model in case user hit their budget.
Here’s an example implementation:
from fastapi import HTTPException
def route_request(user, prompt):
# 1. HARD LIMIT CHECK (Redis Counter)
current_rate = redis.get(f"rate:{user.id}")
if current_rate > 50:
# Scenario A: Abuse -> Hard Stop
raise HTTPException(status_code=429, detail="Rate limit exceeded.")
# 2. SOFT BUDGET CHECK (DB Query)
daily_spend = db.get_spend(user.id)
budget_limit = 10.00 # $10 limit
if daily_spend > budget_limit:
# Scenario B: Over Budget -> Downgrade (Soft Throttle)
# We don't fail; we just swap the model
print(f"User {user.id} over budget. Downgrading to Tier 3.")
return call_llm(model="llama-3-8b", prompt=prompt)
# 3. NORMAL PATH
return call_llm(model="gpt-4", prompt=prompt)Pattern: preventing data poisoning
Threat: Malicious actors tamper with an LLM’s training data to corrupt its behavior, leading to biased or incorrect outputs.
Mitigation (ingestion control):
Trusted sources: Only use and ingest data from known, trusted sources.
Data lineage: Track the origin of all data used for training or RAG.
HITL review: For fine-tuning data, use human experts to review and validate the datasets before training.
Pattern: mitigating supply chain and insecure plugin vulnerabilities
Threat: The security of an LLM can be compromised through third-party services, plugins, or datasets that are themselves vulnerable. This includes insecure plugin design, which can be exploited for attacks like SQL injection.
Mitigation
Minimize functionality: Any plugin or tool given to the LLM should have the absolute minimal functionality needed (e.g. read_email, not delete_email).
Validate plugin inputs: Treat all data passed to a plugin as untrusted. Sanitize it to prevent injection attacks within the plugin itself.
Vulnerability scanning: Regularly scan all third-party libraries, containers, and models for known vulnerabilities.
Use the genAI service / LLM gateway: Your gateway allows you to quickly replace a provider or model that is found to be compromised.
Engineering for production
We cannot manage what we do not track. Beyond standard APM (application performance monitoring), we must track:
TTFT (time to first token): The perceived latency. How long until the user sees the first character? High TTFT kills engagement.
TPOT (time per output token): The generation speed. If this is high, the model is too heavy or the provider is overloaded.
Context utilization: Is the context window filled? Use the model optimized for token usage for the use case for cost effectiveness.
Training with test data
In earlier sections we have discussed golden datasets – inputs with their ideal outputs used for training the LLM as a judge. Let’s now look at how we can procure this ideal data for different use cases.
Sourcing datasets
Curate diverse set of open source projects with varied languages, sizes and domains
Refresh the dataset regularly to incorporate up to date coding style and practices
Create custom open-source repos with intentional gaps and edge cases to stress test behaviors
Testing and training for code complete
Randomly remove code blocks, functions, classes from the code. Start typing signatures of missing classes and functions. Compare the IDE code complete suggestions with the actual classes and functions to measure accuracy and semantic correctness. Iteratively tune the LLM model prompts based on error cases and coverage gaps identified.
Testing for chat responses
Clone the repo removing all the comments and docs. Ask the system to explain code blocks, functions, classes and compare against the original repo’s docs, README files or comments.
Testing for agentic workflow and tasks
Assign real world tasks like adding documentation, writing test cases or refactoring to the agent. Combine multiple tasks in scenarios.
Automatically check the code and tests and documentation for correctness and style.
Integrating these tests into CI/CDs helps keep models and prompts updated with evolving repos.
Pattern: quantitative evaluation testing
The goal of this pattern is to move beyond vibes and qualitative observations to a measurable correctness percentage.
The problem is that agentic flows are multi-step and non-deterministic. Traditional pass/fail unit tests often fail to capture the nuance of a complex task that is mostly correct but slightly off in tone or style.
To tackle this distinctive character we assign a numerical score to every agentic run by breaking the output into weighted criteria. This allows us to track an evaluation success rate, the percentage of tasks successfully completed to the golden standard.
To calculate the success rate we define weights to the different output facets that reflect their relative importance (e.g. logic: 50%, syntax: 30%, documentation: 20%).
Then we execute the agent across a golden dataset of at least 50 representative tasks, and score according to the following metric:
Figure 2.13 shows the overall evaluation process:
When to use it: Use this in your CI/CD pipeline. If a prompt change or model upgrade causes the correctness percentage to drop below a defined threshold (e.g. 90%) then the deployment should be automatically blocked to prevent quality regression.
Mutation testing
Add logic and syntax errors to your code. Evaluate whether the system can identify and fix these errors.
Negative prompt testing
Give confusing or risky actions like ‘Delete all databases’. Evaluate whether the system is able to flag it as a risk and ignore/refuse or ask the user for further clarification.
Respecting user privacy
We want to be able to use an LLM-powered system for code completions, reviews, generation etc. without leaking our codebase to the model. Techniques for ensuring that user privacy is maintained while the data is still served from the model include:
Ephemeral data handling: Code snippets and related info, like filenames, should never be saved to disk or databases. Encrypted code should only be decrypted in the computer’s live memory for processing and deleted the moment it’s no longer needed.
Embedding-only search: We don’t store the actual code. Instead, code snippets are converted into a mathematical format (vectors) for searching. This process is irreversible, so the original code cannot be reconstructed from the database. (We also scramble all metadata. Real file and function names are replaced with anonymous, hashed IDs.)
Strict access control
Minimized code transfer: Only code context required for a query or code complete is sent to the server, not the entire codebase.
Encryption policies: All requests sent to/from the client are encrypted, and all data stored on server side is encrypted.
Summary
The patterns we’ve just looked at, from the GenAI service and async queues to RAG and LLM-as-a-Judge, form an architect’s playbook for building production-grade, AI-powered systems. In the rest of the book we examine four case studies that utilize this playbook.
This article is an excerpt, Chapter 2, from System Design for the LLM Era: Patterns and principles for production-grade AI architecture by Sampriti Mitra, published by Packt. It is shared here with the publisher’s permission for knowledge sharing with the Deep Engineering community. All rights remain with Packt Publishing. This content may not be reproduced, redistributed, or remixed in any form without the publisher’s written consent. You can get the full book here.