It’s common to think of an agent as an LLM with a few tools attached. In practice, the architecture looks quite different.
The model provides the reasoning core, but most of the system’s capability lives outside it and the system composes that at runtime.
That last part is the bit people skip. The model isn’t holding the whole agent inside its weights. It’s a reasoning engine that gets handed the right context, the right procedures and the right interfaces at the moment it needs them. Change what you compose around it and you change what the agent can do, without touching the model at all. So when an agent feels smart, most of what you’re seeing is the system around the model doing its job, not the model alone.
The three layers that matter
At the core, three layers matter:
- Memory manages state the model shouldn’t carry (working context, semantic knowledge, past interactions, personalisation)
- Skills define how tasks get done (procedures, decision heuristics, constraints)
- Protocols structure interactions (agent-to-user, agent-to-agent and tool interfaces)
Memory is about not making the model hold what it has no business holding. A context window is not a database. If you stuff every past interaction, every fact and every user preference into the prompt, you pay for it on every call and the model still loses the thread. Memory pulls that state out. Working context for the task at hand, semantic knowledge it can look up, a record of past interactions, personalisation for this specific user. The model reads from it rather than carrying it.
Skills are how work actually gets done. Not “the model figures it out each time,” but defined procedures, decision heuristics and constraints. If there’s a right way to handle a refund, or a sequence of steps for onboarding, that shouldn’t be re-derived from scratch on every run. It’s a skill. Written down, reused, improved.
Protocols are the rules of interaction. How the agent talks to the user. How it talks to other agents. How it calls tools. These are interfaces, and interfaces need structure or everything downstream breaks in ways that are hard to trace.
The mediators in between
Between the model and these layers sits a set of mediators: sandboxing, observability, evaluation, compression, approval loops and orchestration. These control how information flows and how decisions are executed.
This is the layer that gets ignored until something goes wrong. Sandboxing keeps the agent from doing damage it shouldn’t be able to do. Observability lets you see what it did and why. Evaluation tells you whether it’s actually working or just looking like it. Compression keeps context manageable so you’re not drowning the model. Approval loops put a human in the path before consequential actions. Orchestration decides what runs, when, and in what order.
None of that lives in the model. All of it decides whether the model’s output turns into something safe and useful, or something you have to apologise for later. The mediators are where control lives.
Where things belong
For any new capability, decide where it belongs. Stable knowledge goes into memory. Repeatable workflows go into skills. Interaction rules go into protocols. Control and safety live in the mediators.
That’s the actual design discipline. When you want the agent to do something new, the first question isn’t “how do I prompt the model for this.” It’s “which layer does this belong in.” A fact that rarely changes is memory. A workflow you’ll run again and again is a skill. A rule about how the agent should behave in a conversation is a protocol. A guardrail or a checkpoint is a mediator. Put things in the wrong layer and you end up with brittle prompts trying to do a job the architecture should be doing.
Designing agents becomes less about the model itself and more about what you externalise and how you manage it.