The rapid advancement and adoption of Artificial Intelligence (AI) within enterprises are encountering a significant obstacle, not in the sophistication of AI models themselves, but in the foundational data infrastructure that supports them. The core issue lies in the pervasive fragmentation of data across a multitude of disparate systems. Businesses today typically house their critical information in a patchwork of databases, Software-as-a-Service (SaaS) tools, data warehouses, and proprietary internal platforms. Each of these data silos is often protected by its own set of security controls, creating a complex and often inefficient web of data governance.
While this fragmented approach may have sufficed for traditional batch processing environments, it presents a substantial impediment for modern AI applications and agents that require real-time reasoning across live business data. Sean Falconer, Head of AI at Confluent, a leading data streaming platform, articulated this challenge in a recent discussion with The New Stack. "Most AI projects fail before they even reach a single customer because the data layer breaks down," Falconer stated. "Teams have the models and mandate, but security risks and fragmented data stop them from shipping."
This disconnect between the promise of AI and the reality of data accessibility means that many promising AI initiatives remain confined to proof-of-concept (POC) stages, unable to transition into production-ready solutions that deliver tangible business value. A report by McKinsey & Company underscores this sentiment, revealing that a staggering eight out of ten companies identify data limitations as a primary roadblock to scaling agentic AI—AI systems designed to act autonomously on behalf of users. It is precisely this critical gap that Confluent aims to bridge with its latest suite of capabilities, Confluent Intelligence and enhanced Confluent Cloud services, unveiled on May 19th in London. The company’s strategic objective is to establish real-time data streaming as the secure, robust foundation for AI applications and agents, equipping high-stakes industries with the necessary controls and developer tooling.
Historically, businesses have invested heavily in building robust security protocols and segmenting their data into distinct, isolated pools, a strategy that made logical sense in the pre-AI era. However, this established architecture now acts as a bottleneck for AI integration. Enterprise leaders are left with a difficult choice: either grant AI agents overly broad access, risking security breaches and data integrity, or attempt to centralize vast amounts of data, potentially compromising performance and control. Neither of these options presents an ideal long-term solution.
The Imperative of Live Context for AI
Falconer contends that the primary constraint for enterprise AI is not the inherent quality of the AI models, but rather the difficulty in accessing and utilizing up-to-the-minute data. He vividly illustrates this point: "I see a demo like a movie set, with a fake cityscape, where I’ve paid the extras to be in the background. Real cities are going to be a lot more complicated and unpredictable." The true test of an AI application lies not in its polished demonstration, but in its ability to perform reliably when interacting with actual users and dynamic, real-world data.
Consider the example of an airline aiming to deploy an AI-powered customer support agent to assist passengers with flight rebookings. If this agent relies on outdated data, it could inadvertently offer a passenger a seat on a flight that no longer exists or assign a seat that has already been occupied. Such errors, Falconer notes, directly undermine the goal of improving user experience and streamlining operations. For AI systems, real-time accuracy has evolved from a desirable feature to an absolute necessity, differentiating between a convincing illusion and a functional, trustworthy application.
Falconer draws an analogy to the act of driving to emphasize the critical need for real-time data in AI. "I’ve got 10,000 hours under my belt, so I have the historical reference on how to drive," he explains. "But if my camera feed of the road ahead only updates every two hours, how long is it before I crash?" This illustrates that while historical data provides the necessary experience and pattern recognition, it is insufficient without an immediate, up-to-date understanding of the surrounding environment. The same principle applies to enterprise AI. Companies have amassed extensive historical data and built sophisticated batch processing systems over decades, but have often neglected the crucial capability to process and act upon present-moment information. The pivotal leap for AI agents involves integrating both long-term memory and live contextual awareness to enable safe and effective operational decision-making.
Streamlining Data Streaming for Enhanced Security and Accessibility
Confluent’s mission is to guide organizations away from the complexities of manual data pipeline construction towards a more streamlined and intuitive approach to working with streaming systems. The company’s recently introduced features, including its managed MCP server, agent skills, and dbt adapter, are specifically designed to reduce the friction associated with configuration and setup. These tools aim to democratize access to streaming data for a broader range of developers, including those who may not be deeply immersed in streaming technologies on a daily basis. In practical terms, this means users can articulate their data requirements in natural language and then leverage familiar workflows to build, manage, and troubleshoot streaming operations.
The emphasis on ease of use is not merely about convenience; it is intrinsically linked to security. Falconer posits that when systems are cumbersome and opaque, developers are often incentivized to find workarounds, which can inadvertently compromise security protocols. Conversely, by providing opinionated and user-friendly tooling, Confluent encourages adherence to approved pathways. The "skills" feature, for instance, acts as pre-defined "recipes" that encapsulate Confluent’s best practices, saving developers the considerable effort of recreating this domain expertise from scratch.
However, Confluent acknowledges the inherent complexities of AI, particularly the non-deterministic nature of machine learning models. Therefore, maintaining control over AI operations remains paramount. The managed MCP server operates in a read-only mode and integrates seamlessly with existing Role-Based Access Control (RBAC), roles, and credentials. Simultaneously, the more flexible "skills" layer is recommended for staging or development environments rather than for direct production modifications.
A crucial distinction for enterprise AI development is the need for determinism in production environments, even while acknowledging the value of experimentation. "With production environments, use tools within a staging or deployment environment, and once you know it’s working correctly, use a system like Terraform to push them into production in a repeatable way," Falconer advises. This methodical approach ensures that AI deployments are both agile and reliable.
The same principles apply to data movement. Confluent’s stream processing layer is capable of connecting to external databases, ingesting contextual information for AI, enriching data streams, and interacting with large language models within a customer’s cloud environment. Critically, these interactions are designed to occur securely, avoiding the need to traverse the public internet. When Confluent Cloud operates in conjunction with a customer’s Azure-hosted systems, the communication between these environments remains within Microsoft’s private backbone, ensuring enhanced data protection and privacy.
This secure data transit is fundamental to the enterprise AI proposition, which increasingly centers on the secure orchestration of context rather than merely high-speed data transfer. Confluent is also reinforcing privacy controls deeper within its stack. Features such as built-in Personally Identifiable Information (PII) detection and redaction in Flink SQL allow sensitive data to be managed directly within the stream, eliminating the need to route it to external warehouses for processing and subsequent redaction.
Real-World Validation and the Path Forward
The financial services sector, which inherently operates on continuous streams of events—spanning card transactions, login attempts, wire transfers, and ATM activities—offers a compelling use case for Confluent’s advancements. "But historically, it’s been difficult to run advanced AI models directly within livestreams in a secure, governed way," Falconer observes. The traditional recourse has been to batch process this data into a warehouse for later analysis, a method that often results in detection delays of hours or even days, by which time fraudulent activities may have already incurred significant costs.
The recent launch, which notably includes support for the TimesFM time-series forecasting model, enables advanced AI capabilities to be applied closer to the point of data origination. This proximity facilitates more adaptive anomaly detection and reduces the need for extensive model tuning. Consequently, financial institutions can identify patterns indicative of account takeovers or detect unusual transaction spikes in near real-time, rather than retrospectively after financial losses have occurred.
Falconer’s overarching philosophy is rooted in his experience as Head of Developer Experience at Google: cumbersome tooling invariably hinders adoption. He recalls a powerful, yet difficult-to-implement API-first business messaging product. The introduction of a simplified click-through interface dramatically reduced onboarding times, leading to a threefold increase in the usage of advanced features and a doubling of product launches. As he eloquently puts it, "make it easy and fun to use, and developers will be safer and more successful." This guiding principle underscores Confluent’s commitment to creating solutions that not only enhance AI capabilities but also empower developers to implement them securely and effectively. The company’s sustained investment in data infrastructure modernization signals a critical step toward unlocking the full potential of AI across the enterprise landscape.
