Most AI agents don’t fail because the model cannot reason. They fail because the world they are supposed to reason about is invisible, fragmented, or constantly shifting beneath them. In other words, the data layer—not the model layer—is where production agents usually break.
Over the past few years, teams have poured energy into picking the “right” model and wiring it into demos that look impressive on stage. But as soon as those agents are asked to operate inside real organisations—with sprawling workspaces, partial integrations, and brittle permissions—the gap shows. The agent can talk, but it cannot really see.
What’s missing is not another prompt template or a clever tool‑calling trick. What’s missing is a persistent context layer: a structured, continuously updated representation of organisational activity that agents can query and act on in real time. Until you treat that context layer as a first‑class part of your data architecture, model upgrades will only get you marginal gains.
In my experience building and deploying agents in production, this pattern shows up in most of the failures I’ve seen: the model is fine, but the context is missing. One recent MIT study of enterprise generative AI projects puts a number on it, suggesting that the overwhelming majority of pilots stall before scaling because they lack the memory, connectivity, and integration needed to support real work. This isn’t a model failure, it’s a data problem.
More specifically, it’s a missing data layer: what I call an agent context layer.
Where Context Fails in Practice
A context layer is a persistent, structured representation of organisational activity that agents can query and act on in real time. At a basic level, the context layer is the full set of information an agent can view and interact with that describes your workspace. In a coding environment, that might be your codebase, comments, and logs. In a work platform, it translates to tasks, documents, chats, spreadsheets, and the relationships between them. Just as a human moves from a conversation to a document to a task while doing knowledge work, an agent needs to be able to navigate that same graph of information.
Imagine a critical customer escalation. A human engineer might scan the latest Slack messages, open the linked Jira ticket, check recent deploy logs, and skim a runbook before deciding what to do. An effective agent should be able to follow that same path: understand that the Slack thread, the ticket, the logs, and the runbook are all facets of the same incident, and move across them seamlessly. Without a coherent context graph, the agent is stuck looking at each artefact in isolation.
But most organisations don’t have a unified context layer. Instead, their information is fragmented. Communication data lives in one system, project data in another, and even when access technically exists, it often breaks down under real‑world constraints. APIs may allow long historical queries but impose strict rate limits or other limitations that make them impractical for agents operating at scale. In many cases, you don’t have true first‑party access to your own data, which makes assembling a complete picture of your organisation difficult in practice. The result is the same: the agent cannot reliably retrieve the context it needs.
ALSO READ: Intent: The Missing Data Layer in Generative AI
The typical workaround is to assemble context at runtime, pulling together whatever information can be retrieved in the moment. That often means refetching raw data, recomputing summaries, and passing incomplete context into the model. It works well enough for demos, but at scale it becomes expensive, slow, and inconsistent. Latency increases, cost per query rises, and agent behaviour becomes unpredictable because the underlying context is incomplete or stale.
Instead of reconstructing context at read time, a complete context layer continuously maintains a structured representation of what’s happening across your organisation. Every action in the system contributes to that representation. When a new message is added to a thread, for example, a compressed summary can be updated incrementally so the agent doesn’t need to reread the full history or reconstruct knowledge from scratch. This kind of compression—turning raw activity into compact, continuously updated representations—isn’t just an optimisation, it’s the core problem to solve when building effective agents.
Designing an Agent‑ready Data Layer
Building this is genuinely hard. Some of that difficulty is technical, but the more fundamental challenge is that most organisations don’t control their data in a unified way. Knowledge is fragmented across tools, and those tools often impose real constraints on access. Even when APIs exist, they may limit historical depth or restrict the kinds of queries that can be performed. That leads to an incomplete context graph, which in turn leads to unreliable agents.
From Fragmented Tools to a Context Graph
This is why simply integrating tools isn’t enough. You can’t take separate systems, declare them connected, and expect them to behave like a unified source of truth. The connections have to be native to how data is stored and structured. The context layer has to be built into the system itself. Systems that control their own primitives and data model have a fundamental advantage here, because they can provide agents with complete, high‑fidelity access to the underlying context.
If you’re moving agents from pilots into production, the context layer question will find you eventually. Start by auditing where your organisational knowledge actually lives and whether it’s truly accessible to agents—not just whether an API exists, but whether the data has the historical depth, structure, and permissions model required to support real queries.
ALSO READ: Agentic AI in Production: Why Better Prompts Won’t Bridge the Gap
Shifting Compute from Read Time to Write Time
Think carefully about what you’re precomputing versus what you’re reconstructing at runtime. Anything that agents need frequently and consistently—summaries of ongoing work, relationships between decisions and execution, the state of active projects, normalised representations of customers or services—is a candidate for precomputation. Shifting that work to write time allows you to pay the cost once, instead of repeatedly at query time.
That shift has second‑order effects. It improves latency and cost, but it also stabilises behaviour. When agents are querying against a well‑maintained, precomputed context graph, their decisions become more predictable because the inputs they see are more consistent. You’re no longer gambling on whether a just‑in‑time retrieval pipeline will assemble the right slice of reality under load.
Designing Primitives Agents can Actually Use
It’s also worth resisting the instinct to over‑engineer heuristics into the system. There’s a long‑standing lesson in AI that systems built around handcrafted rules tend to be overtaken by those that scale learning and computation. Instead of trying to encode what matters in advance, focus on building a complete and expressive set of primitives and let the model determine how to use them. Your job is to make the world legible to the model; the model’s job is to decide what to do with it.
Modern agents are increasingly good at deciding how to query, navigate, and act on top of these primitives if the underlying system is designed clearly. If you give them a coherent context graph, well‑designed tools, and reliable feedback signals, you can offload far more of the decision‑making to the model without falling back into brittle rule‑based systems.
The model debates will continue, but durable value from AI agents won’t come from marginal improvements in model capability. It will come from systems that give those models the right context, in the right structure, at the right time. If your systems aren’t structured for agents to reason over, the model won’t save you. Start with the context layer first.
