For two years, a recurring pattern has plagued AI development teams: building an agent, encountering reliability issues, upgrading the underlying model, achieving only marginal improvements, and then facing the same reliability problems in a slightly different guise. The diagnosis, consistently, has been a lack of model intelligence, leading to a predictable fix: deploying a newer, supposedly smarter model. The outcome, however, has remained stubbornly consistent: the system remains broken. This persistent cycle points to a fundamental misunderstanding of the problem. The issue is not, and never has been, the model itself.
Andrej Karpathy, a prominent figure in the AI research community, articulated this shift in perspective months ago. In a post on X, he detailed how his allocation of AI compute resources had changed, noting that "a large fraction of my recent token throughput is going less into manipulating code, and more into manipulating knowledge." This wasn’t about running a more sophisticated model. Instead, Karpathy focused on building a robust infrastructure around a constant model. He described an approach where raw data sources are indexed into a structured directory. A large language model (LLM) then incrementally compiles this data into a comprehensive wiki, complete with summaries, backlinks, and concept articles. Agents are equipped with tools, accessed via command-line interfaces (CLIs), and their outputs are fed back into the base system to enhance future queries. The model remained static; the surrounding infrastructure was the variable that drove progress.
The core of Karpathy’s insight, and the central argument for overcoming agent reliability issues, lies in the framing: "The model runs on context." The quality of an agent’s execution is directly dependent on the quality of the context it receives, the precision of the actions it is permitted to take, and the feedback loops that enable the system to learn from its mistakes. Crucially, none of these critical components reside within the model itself. They are all functions of the underlying infrastructure, an area that most development teams have historically neglected.
The Missing Compile-Step: Transforming Raw Data into Actionable Knowledge
Karpathy’s approach highlights an essential, often overlooked, preliminary step: a "compilation" phase. Raw data is ingested, and an LLM then transforms it into a structured, queryable format. This compiled version becomes the foundation upon which agents operate, utilizing their assigned tools. This compilation process is where the real work of making data usable for agents takes place. It’s the mechanism that converts a chaotic collection of internal data into something an agent can reason about, rather than merely guess at.
A significant number of production agent systems today bypass this crucial step. They connect models directly to raw data sources – databases, APIs, document stores – expecting the LLM to perform the compilation within its context window, under the intense pressure of query latency. The result is typically pattern-matched guesswork, drowned in a sea of noisy, unstructured information.
Teams that have achieved reliable agent performance have explicitly built this compilation step. This isn’t about creating a generic knowledge base; it’s about constructing a structured representation of their specific organizational reality. This includes internal naming conventions, actual decision-making pathways, and the outcomes of previous, similar operations. It is the organizational equivalent of Karpathy’s wiki, derived from operational data rather than static documentation.
While building and maintaining this structured context is more challenging than simply switching to a new model, it represents the critical difference between an agent that operates within an organization’s true operational landscape and one that confidently navigates a hallucinated version of it.
The Tool Retrieval Conundrum: Bridging Intent and Action
As Karpathy’s wiki grew, reaching approximately 100 articles and 400,000 words, its usability was maintained through LLM-generated indexes and summaries. The addition of tools, including a small search engine exposed to the LLM via a CLI, further enhanced its functionality. This demonstrates that context doesn’t disappear; rather, the engineering of retrieval mechanisms, indexing, and tool interfaces becomes paramount to system success.
Production agent systems frequently encounter significant challenges at this juncture. Consider a mid-sized engineering organization, which likely utilizes a complex ecosystem of tools including GitHub, Jira, Confluence, cloud providers, monitoring and alerting systems, CI/CD platforms, and internal deployment tooling. Each integration represents a family of tools with its own distinct schema, naming conventions, and expected invocation patterns. Attempting to fit all of this information into a single context window is not only slow and expensive but also leads to poor tool selection. The model, overwhelmed by noise, resorts to pattern matching.
Standard vector retrieval methods often exacerbate this problem. These systems match the semantic similarity between a user query and stored tool descriptions. While effective when vocabulary aligns, they falter when it doesn’t. For instance, a developer might ask, "Why did the deploy fail?" The correct tool might be get_pipeline_run_logs. However, the vector match between these two phrases is often poor. Consequently, the agent selects a plausible tool rather than the correct one.
A more effective solution involves "guessing the answer first." Given a query, the system attempts to predict what a working tool call would look like. It then matches against this hypothetical call, rather than the raw query. While "Why did the deploy fail?" and "fetch pipeline logs" may not appear textually similar, matching the shape of the intended action, rather than the literal words, creates a robust alignment.
This represents a critical translation layer, converting user intent into specific actions. Failures in agent systems frequently originate in this translation process, underscoring that it is an engineering challenge, not a model deficiency. Teams that have implemented hypothetical-invocation matching instead of relying solely on semantic similarity report significantly more reliable tool selection than those who have prioritized model upgrades.
The Guardrails Gap: Preventing Unintended Actions
A common failure mode in agent systems is capability without constraint. This manifests when an agent, possessing broad tool access and executing correctly from its own internal logic, takes an action that was never intended by its human operators. This is not a result of the model hallucinating, but rather a consequence of the absence of enforced boundaries between what the agent can do and what it should do.
Three distinct scenarios illustrate the pervasive nature of this problem:
-
State-Sponsored Cyberattacks: The GTG-1002 incident, disclosed by Anthropic in November 2025, serves as a stark example. A Chinese state-sponsored group successfully manipulated Claude Code in a cyber-espionage campaign targeting approximately 30 organizations. AI performed an estimated 80-90% of the tactical work, with human intervention limited to strategic decision points. At its peak, the AI was initiating thousands of requests per second. This was not a failure of model quality but a catastrophic breakdown in execution boundaries. Once attackers fragmented their operations and bypassed safeguards, the system was capable of executing high-risk actions at a speed that far outpaced human supervision.
-
Prompt Injection Vulnerabilities: Prompt injection represents a different vector but stems from the same structural failure. Researchers have repeatedly demonstrated agents executing injected instructions embedded within retrieved content. A malicious payload within a web page or document can redirect an agent’s subsequent tool calls. The model executes these injected instructions because the execution layer fails to distinguish between an "instruction from the user" and an "instruction found in retrieved content." The agent is functioning as designed based on the input it receives; the architecture, however, is fundamentally flawed.
-
Over-Permissioned Workflows: A quieter but more prevalent issue involves agents with excessive permissions operating across multiple write-capable systems. An agent with access to a CRM, an email client, and a calendar might perform an unexpected action – updating records, sending a draft email, or booking a meeting – because a workflow encountered an unanticipated branch, and no scoped permission model was in place to prevent it. These actions were not intended, and no explicit rules prohibited them. The agent acted simply because it had the access.
In each of these cases, the model is not the point of failure. The breakdown occurs in the absence of an execution layer that defines, for each specific action, what is permitted and enforces those boundaries, irrespective of the model’s internal decision-making.
Effective architectures intercept every tool call before execution. They mask sensitive data before it is processed by the LLM, block specific tool combinations at the execution layer rather than solely at the prompt level, enforce per-agent rate limits and role-based access, and generate comprehensive audit trails with explicit reasoning for every invocation. Furthermore, human checkpoints must be architecturally integrated into high-stakes workflows, not as mere fallbacks, but as fundamental components of the system’s design.
Companies like Mate have developed execution isolation layers that embody this principle. Every tool call undergoes validation, scoping, and logging before reaching the integration. The model never possesses direct write access to downstream systems; instead, it proposes actions to a layer that determines whether and at what scope those actions should be executed.
Karpathy’s wiki project, in miniature, illustrates a similar discipline. Health checks identify inconsistent data, impute missing fields, and suggest new connections. In an enterprise context, this data "linting" layer must incorporate permission boundaries. Some updates can be automated, while others should be routed through proposal, approval, or review queues. The distinction between what an agent can modify autonomously and what requires human judgment must be explicit and rigorously enforced.
The Real Engineering Effort Behind Reliable Agents
Teams consistently producing reliable production agents dedicate the majority of their engineering effort to infrastructure. Four key areas repeatedly emerge as critical:
The Context Graph: A Living Representation of Organizational Knowledge
Building and maintaining a compiled representation of organizational knowledge is not a one-time task. Schemas evolve, systems are renamed, and personnel and processes undergo continuous change. Successful teams treat the context graph as a product with dedicated ownership, a defined update cadence, and robust health checks, rather than a static setup step performed once at deployment.
Observability: Beyond Basic Tracing
Model-agnostic proxy layers are becoming standard. These layers capture full traces for every LLM call, regardless of the provider, enforce per-tenant cost and rate limits, and enable model swaps without requiring extensive re-architecting. For agents, comprehensive tracing means capturing not just requests but also the underlying reasoning: what the agent considered, which tool it selected and why, what data was returned, and how that data was utilized. Without this level of detail, debugging becomes an archaeological endeavor.
Continuous Evaluation: Real-World Performance Metrics
Effective evaluation relies on per-agent, per-workflow datasets derived from production traces, not synthetic benchmarks. This approach typically involves two tracks: deterministic checks for quantifiable aspects like tool call correctness, rate limit compliance, and scope violations; and a "model-as-judge" approach for qualitative aspects such as reasoning coherence and response quality. When promoting a new prompt version or upgrading a model, rigorous testing against actual production data is essential. The critical question is not "does it benchmark higher?" but rather "does it continue to function correctly within this organization’s specific operational context?"
Configuration Management: Independent Releases and Rollbacks
Prompt versions, model selections, and tool configurations must be independently releasable and independently rollback-able. Changing an agent’s underlying model should not necessitate a full code deployment. Rolling back a prompt regression should not trigger a system incident. Teams that establish this capability early are able to ship changes more rapidly and with significantly fewer disruptions. This practice mirrors the mature ML engineering discipline that emerged in recommendation systems years ago, now being applied to agent behavior, a discipline many organizations are building from the ground up.
The Differentiator Isn’t Reasoning, It’s Infrastructure
Karpathy’s observation about shifting from manipulating code to manipulating knowledge accurately describes where the substantial engineering effort in agent systems truly lies. The model operates based on context. The structure, retrieval, and scoping of that context are the primary determinants of the system’s outcome. The constraints placed on the model’s actions dictate its safety, and the mechanisms for measuring and improving the system’s performance ensure its reliability.
All of these critical functions are rooted in infrastructure. None of them can be solved by simply deploying a more capable model.
The commoditization of AI models is progressing far more rapidly than many teams realize. The reasoning gap between major LLM providers is narrow and continues to shrink. Conversely, the infrastructure gap between teams that have invested in robust context plumbing and guardrails and those that have not is wide and is steadily widening.
A more intelligent model will not imbue an agent with an understanding of your organization’s specific nuances. It will not prevent a prompt injection within a retrieved document. It will not enforce scope limitations on an over-permissioned workflow. And it will not alert you to a decline in tool retrieval accuracy caused by a recent integration name change.
These are the responsibilities of infrastructure. And it is infrastructure that must be built first.
