Confluent Unifies Batch and Stream to Power Agentic AI at Scale

Shutterstock

Data streaming platform Confluent has introduced new improvements to Confluent Cloud, its fully managed service built on Apache Kafka. A key upgrade is the introduction of Snapshot queries that allow users to query both batch and streaming data using the same SQL interface. The company also introduced new private networking and security features that make stream processing more secure and enterprise-ready. 

Agentic AI, which refers to autonomous systems capable of reasoning, planning, and taking action on behalf of users, is swiftly transitioning from an emerging concept to a foundational element in enterprise technology. However, for AI agents to make the right decisions and deliver maximum value, they need streaming and historical datasets. This can be a problem if data is housed in disparate systems, such as legacy databases and separate cloud storage platforms. Complex ETL jobs may be needed to merge data, but that would add complexity, costs, and inefficiency to the overall workflow.  

Confluent aims to solve this issue through Snapshot queries by providing AI agents with unified access. Instead of relying on brittle pipelines that shuffle data between batch and stream systems, Snapshot queries are designed to bring everything together in one place. This means teams can work with a single query language and interface to analyze past trends and react to live events, without spinning up separate workloads or syncing across tools.

“Agentic AI is moving from hype to enterprise adoption as organizations look to gain a competitive edge and win in today’s market,” said Shaun Clowes, Chief Product Officer at Confluent. “But without high-quality data, even the most advanced systems can’t deliver real value. The new Confluent Cloud for Apache Flink features make it possible to blend real-time and batch data so that enterprises can trust their agentic AI to drive real change.”

According to Confluent, Snapshot queries could be particularly useful for generating reports that reflect data’s state at a specific time, analyzing historical data for auditing for compliance purposes, and debugging issues by examining past data states. Snapshot queries would also be useful for developers building agentic AI systems and event processing workflows that require historical data enrichment. 

Available in early access through Confluent Cloud for Apache Flink, Snapshot queries rely on the platform’s advanced SQL query optimizer that determines whether data should be fetched from Kafka topics or from open table formats like Apache Iceberg or Delta Lake. The feature uses Tableflow to materialize Kafka event streams into these tables, enabling efficient historical access alongside real-time processing. 

This translates to less complexity for users. For example, a developer building a fraud detection system no longer has to manually orchestrate pipelines to pull historical transaction patterns from one system and live activity from another. Instead, they can write a single SQL query, and the new snapshot query engine automatically determines where the relevant data lives and retrieves it efficiently.

Agentic AI needs more than speed, it needs context. As IDC’s Stewart Bond puts it, the goal is to “unify disparate data types, including structured, unstructured, real-time, and historical information, in a single environment.” That’s exactly what Confluent is aiming to deliver with its latest Flink-powered snapshot queries.

Many organizations hesitate to deploy real-time systems at scale due to security and compliance risks, especially in hybrid cloud environments. Confluent’s latest updates bring a new level of security and efficiency to Flink workloads, particularly through CCN (Confluent Cloud Network) routing and IP filtering.

(SkillUp/Shutterstock)

The CCN routing works by allowing teams to reuse their existing private networking configurations already in place for Kafka. This means they don’t have to create new network setups from scratch to run Flink workloads, saving time and reducing complexity. By extending the same secure connections to Flink, teams can maintain consistent security policies across both data systems. CCN routing is now generally available on Amazon Web Services (AWS) in all regions where Flink is supported.

Many organizations running in hybrid environments need tighter control over which data can be accessed publicly. IP filtering for Flink helps by limiting access to approved IP addresses and making it easier to track any unauthorized attempts. When paired with CCN routing, it gives teams more control over their Flink workloads and helps meet security and compliance needs in real-time settings. IP Filtering is now generally available for all Confluent Cloud users.

The updates to the platform show that Confluent is keen to move beyond Kafka and become a full streaming data platform built for modern AI needs. It is not just adding new capabilities, but also signaling a broader shift in strategy. 

While Snowflake built its foundation on batch processing and Databricks promotes the lakehouse model, Confluent is focused more on delivering real-time intelligence. Apache Flink is at the center of that push. For agentic AI and other fast-moving workloads, Flink gives systems the fresh context they need to make smart decisions on the fly.

Related Items

Five Drivers Behind the Rapid Rise of Apache Flink

Confluent Goes On Prem with Apache Flink Stream Processing

Yes, Real-Time Streaming Data Is Still Growing