RAG on Kafka: a technical reference architecture for streaming datalakes, with fresh embeddings and vector stores that are always up to date

· By José Manuel Román Fernández Checa
Contents

TL;DR

The piece that blocks enterprise GenAI projects most in 2026 is not the model, nor even the guardrails: it is data ingestion for RAG. Companies hold valuable information in OLTP databases, in operational logs, in SaaS systems, and all of it is quietly changing every second. Batch RAG systems that reindex every night arrive late (the model’s answer is backed by a snapshot from 18 hours ago) and open the door to operational hallucinations even when the retriever is perfect. The dominant answer in production in 2026 is to build the RAG piece on top of Kafka as the source of truth: an immutable log, massive throughput, managed schema evolution, and a mature stream processing ecosystem (Flink, Kafka Streams, RisingWave) that makes it possible to transform and embed events as they happen, delivering them to vector stores (Milvus, Qdrant, Weaviate, pgvector) in milliseconds. The canonical pattern: source → CDC with Debezium → Kafka topics → Flink SQL with an embedding UDF → sink connector to a vector store → serving with vLLM or equivalent. The 2026 developments that change the game: Confluent Tableflow turns Kafka topics into Iceberg/Delta tables automatically (read from Snowflake/Databricks/Trino with no ETL, 30-50 % lower TCO); native Flink SQL brings openai_embedding() and vector search integrated with Cosmos DB and Amazon S3 Vectors; Confluent’s official MCP server lets AI agents query Kafka/Flink/Tableflow in natural language. This post develops the end-to-end architecture with manifests, Flink SQL code and concrete numbers.

This is the second post in the MLOps series specific to LLMs. The first (2026 landscape) set out the framework. Here we go down to the most operational piece of the stack: how a real enterprise system connects to an LLM agent while keeping RAG fresh without falling into explosive complexity.

The analogy: Kafka as the “single source of truth”

Anyone who has spent time in distributed systems has seen the pattern again and again: an immutable, append-only, replicated, time-ordered log has become the canonical primitive for rebuilding complex systems. DBAs know it as the write-ahead log (PostgreSQL WAL, MySQL binlog). Event system developers know it as event sourcing. Data architects know it as the Kappa architecture. Kafka is the massive, distributed, mature implementation of that primitive: a log that lives on disk, partitioned to scale, replicated for durability, retained by time or size, readable from any point in history.

When you think about RAG, this is exactly what you need. A well-designed RAG system has two critical questions: how do you keep the index fresh? And how do you rebuild the index when something breaks? Kafka answers both naturally: fresh because every change at the source is published as an event to the log and the pipeline processes it in milliseconds; rebuildable because the whole log is there: you drop the vector store, you have the Kafka topic from offset 0, and you build the index back exactly as it was.

There is a second layer to the analogy as well. For a modern GenAI architecture, Kafka plays the role of the WAL of the entire system. Just as the Postgres WAL is the gospel of the database’s state (if you lose the DB but keep the WAL, you can rebuild it), the Kafka log is the gospel of the state of the business as a whole: orders, users, transactions, documents. Connecting your AI agent to Kafka means connecting it to the real pulse of the system, not to stale snapshots.

The static RAG problem

Before presenting the architecture, it is worth pinning down what concrete problem we are solving. The antipattern that trips up most GenAI projects:

  1. A team builds RAG on a static dataset: it dumps Confluence documents, product PDFs, database snapshots.
  2. It embeds the lot with a nightly cron that regenerates the index every 24 hours.
  3. It ships the product.
  4. Day 2: a user asks about a change that happened two hours ago. RAG does not have it; the model answers about the old version.
  5. The team adds fragile logic: “if the query mentions a recent date, escalate to a human agent”.
  6. Day 30: the dataset has moved so much that half the RAG is out of date. The team decides to refactor and migrate to streaming.

It is the repeated story of so many projects that the ecosystem has learned the lesson: streaming from day 1, even when the volume is low. The operational complexity of a well-designed streaming pipeline is constant; the complexity of migrating from batch to streaming on a live project is enormous.

From Lambda to Kappa to Streaming RAG

Three architectures in historical order:

Lambda (the classic big-data design from 2014): two parallel pipelines, one batch for accuracy and one streaming for freshness. The query combines both. It works, but it means maintaining two pipelines.

Kappa (Jay Kreps, 2014, mainstream since 2020): a single streaming pipeline. Batch is a special case of streaming (reprocessing from the beginning). It simplifies a great deal.

Streaming RAG (emerging in 2025-2026): a specific variant of Kappa where the pipeline’s output is embeddings indexed in a vector store that the LLM queries at runtime. The Kafka log is the source of truth, the vector store is a queryable projection.

The mental conversion: think of the vector store as the materialised view of the Kafka log. If the view is corrupted, you rebuild it from the log. If you want a new view (a different embedding model, a different chunking strategy), you create another consumer of the log and build a second view in parallel.

The reference architecture

On to the diagram. I am going to present the canonical architecture that stabilised in 2026, showing where each component fits:

[OLTP DB (Postgres)]                           [Other sources]
       │                                              │
       │  WAL via logical decoding                    │
       ▼                                              ▼
   ┌──────────────────────────────────────────────────────────┐
   │              Debezium / Kafka Connect (Sources)          │
   └──────────────────────────────────────────────────────────┘
                              │
                              ▼ produces events
   ┌──────────────────────────────────────────────────────────┐
   │                    Kafka cluster                          │
   │   ┌───────────────────────────────────────────────────┐   │
   │   │ topic: orders.raw          (3 partitions, RF=3)  │   │
   │   │ topic: users.raw           (3 partitions, RF=3)  │   │
   │   │ topic: documents.raw       (6 partitions, RF=3)  │   │
   │   └───────────────────────────────────────────────────┘   │
   │      + Schema Registry (Avro/Protobuf)                    │
   └──────────────────────────────────────────────────────────┘
                              │
                              ▼ consumes and transforms
   ┌──────────────────────────────────────────────────────────┐
   │     Flink SQL streaming jobs                              │
   │   - chunking text                                         │
   │   - calls to the embedding model (UDF)                    │
   │   - enrichment with metadata                              │
   │   - sink to a curated topic: documents.embedded           │
   └──────────────────────────────────────────────────────────┘
                              │
                  ┌───────────┼────────────────────┐
                  ▼           ▼                    ▼
         [Vector store]   [Tableflow]      [Iceberg/Delta]
         Milvus/Qdrant     auto-convert    for analytics
         /pgvector/        topics →
         Weaviate          tables
                  │
                  ▼ queried at runtime
   ┌──────────────────────────────────────────────────────────┐
   │   LLM serving (vLLM / SGLang)  +  Retriever              │
   │   - receives the agent's query                           │
   │   - searches top-K in the vector store                   │
   │   - builds prompt + context                              │
   │   - generates an answer with citations                   │
   └──────────────────────────────────────────────────────────┘

The five layers you can see (source, ingestion (CDC), transport (Kafka), processing (Flink), storage (vectors + tables)) are the ones that structure any serious RAG on a datalake in 2026. Let us go through each one.

Layer 1 — Sources: your OLTP as the starting point

The typical source is an OLTP database (Postgres, MySQL, SQL Server). It is where the live state of the business lives. The technique for extracting changes in real time is Change Data Capture (CDC): reading the database’s transaction log (PostgreSQL WAL, MySQL binlog) and turning each commit into a Kafka event.

The OSS standard is Debezium. It supports Postgres, MySQL, SQL Server, MongoDB, Oracle, Cassandra and others. The typical deployment is a Kafka Connect cluster with Debezium connectors.

An example Debezium configuration for PostgreSQL:

{
  "name": "postgres-orders-connector",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "tasks.max": "1",
    "database.hostname": "postgres.prod.internal",
    "database.port": "5432",
    "database.user": "debezium",
    "database.password": "${secret:postgres-creds}",
    "database.dbname": "ecommerce",
    "database.server.name": "ecommerce-prod",
    "table.include.list": "public.orders,public.users,public.products",
    "publication.autocreate.mode": "filtered",
    "slot.name": "debezium_slot",
    "plugin.name": "pgoutput",
    "topic.prefix": "ecommerce",
    "key.converter": "io.confluent.connect.avro.AvroConverter",
    "value.converter": "io.confluent.connect.avro.AvroConverter",
    "key.converter.schema.registry.url": "http://schema-registry:8081",
    "value.converter.schema.registry.url": "http://schema-registry:8081"
  }
}

For each commit in the database, this produces an Avro event on the corresponding topic (ecommerce.public.orders, ecommerce.public.users, and so on) carrying the change: type (INSERT/UPDATE/DELETE), values before and after, commit timestamp, position in the WAL.

A simpler alternative for 2026: RisingWave can read the Postgres WAL directly, with no Debezium or Kafka Connect in between. When the case is CDC only, with no other sources, it is operationally simpler. For architectures with multiple sources (CDC + APIs + scrapers + logs), Debezium remains the standard piece.

Layer 2 — Kafka as transport and persistence

The Kafka cluster is where all events land. Key operational decisions:

Topics: raw vs curated

The convention that settled in 2026:

  • *.raw: the raw event exactly as it arrived. Untransformed CDC, unparsed application logs.
  • *.cleaned: after dedup, schema validation, type normalisation.
  • *.enriched: after adding metadata (geolocation, cross identifiers, and so on).
  • *.embedded: the event with its vector embedding already computed.

Multi-stage topics allow debugging per layer and partial reprocessing: if you change the embedding model, discarding *.embedded and rebuilding from *.enriched costs hours; rebuilding from *.raw costs days.

Schema Registry

Without a schema registry, topics break silently when someone changes the schema at the source. Confluent Schema Registry or the OSS Apicurio are the dominant options.

Common formats:

  • Avro: versioned schema, strict evolution rules. The historical default.
  • Protobuf: compatible with gRPC stacks, good performance.
  • JSON Schema: textual, debuggable by eye, less efficient.

For RAG on Kafka we recommend Avro by default. Schema evolution matters because source tables change over time, and an unversioned schema breaks downstream consumers.

Partitions, replication and retention

Operational decisions for RAG topics:

  • Partitions: typically 3-12. More partitions means more parallelism in the Flink consumer, but more overhead. The rule of thumb: partitions = expected peak events/s ÷ 1000.
  • Replication factor: 3 as a minimum in production. Replication protects against broker failure; with RAG, the cost of losing a topic can be weeks of re-embedding.
  • Retention: for topics that feed RAG, use long retention or compaction by key. If document doc-42 changes 100 times, compaction keeps only the last state per key, leaving a smaller, rebuildable log. For data that is never updated (historical logs), retention by time (90 days, 1 year).

Cross-cluster replication

For multi-region or multi-cloud deployments, MirrorMaker 2 or Cluster Linking (Confluent) replicate topics between Kafka clusters. RAG can then query the local cluster without crossing regions.

Layer 3 — Flink as the streaming processor

Apache Flink is the dominant stream processing piece in 2026. Apache 2.0, a mature distribution, a broad ecosystem. The main alternative is Kafka Streams (simpler, Java-only); RisingWave is the emerging option for pure SQL cases.

What Flink adds to Kafka:

  • Stateful streaming: time-based aggregations, joins between streams, sessions.
  • Exactly-once semantics: with checkpoint coordination.
  • Watermarks: correct handling of out-of-order events.
  • UDFs in Python/Java: including calls to LLM models.

Flink SQL is the most usable part of Flink for data engineers who are not streaming experts. Here is a realistic RAG pipeline example:

-- 1. Define the source: Kafka topic with CDC events from documents
CREATE TABLE documents_raw (
  doc_id        STRING,
  title         STRING,
  body          STRING,
  category      STRING,
  updated_at    TIMESTAMP_LTZ(3),
  PRIMARY KEY (doc_id) NOT ENFORCED
) WITH (
  'connector' = 'upsert-kafka',
  'topic' = 'ecommerce.public.documents',
  'properties.bootstrap.servers' = 'kafka:9092',
  'key.format' = 'avro-confluent',
  'value.format' = 'avro-confluent',
  'value.fields-include' = 'EXCEPT_KEY'
);

-- 2. Define the sink: vector store via an intermediate Kafka topic
CREATE TABLE documents_embedded (
  doc_id         STRING,
  chunk_id       INT,
  title          STRING,
  chunk_text     STRING,
  category       STRING,
  embedding      ARRAY<FLOAT>,
  embedded_at    TIMESTAMP_LTZ(3),
  PRIMARY KEY (doc_id, chunk_id) NOT ENFORCED
) WITH (
  'connector' = 'upsert-kafka',
  'topic' = 'rag.documents.embedded',
  'properties.bootstrap.servers' = 'kafka:9092',
  'key.format' = 'json',
  'value.format' = 'json'
);

-- 3. UDF for chunking (defined in Python or Java)
-- CREATE TEMPORARY FUNCTION chunk_text AS 'com.example.ChunkingUDF';

-- 4. Pipeline: chunk, embed, write to the sink
INSERT INTO documents_embedded
SELECT
  doc_id,
  chunk_idx                            AS chunk_id,
  title,
  chunk                                AS chunk_text,
  category,
  OPENAI_EMBEDDING(chunk,
    'text-embedding-3-small')          AS embedding,
  CURRENT_TIMESTAMP                    AS embedded_at
FROM documents_raw
CROSS JOIN UNNEST(chunk_text(body, 512, 64))
  WITH ORDINALITY AS t(chunk, chunk_idx);

What happens here, line by line:

  • The documents_raw table reads the CDC topic in upsert-kafka mode (each new event on the same key replaces the previous one). This correctly reflects the semantics of “this is the latest version of doc 42”.
  • The documents_embedded table will be the intermediate topic where Flink writes the embedded chunks.
  • The chunk_text UDF (defined in Python or Java) splits each doc into 512-token chunks with an overlap of 64.
  • The INSERT INTO query runs continuously: every new event in documents_raw is chunked, each chunk is embedded with OPENAI_EMBEDDING (a built-in Flink SQL function in Confluent Cloud 2026), and written to the embedded topic.

OPENAI_EMBEDDING can be swapped for a custom function that calls a self-hosted model (vLLM with an encoder), SentenceTransformers, or a managed service. The syntax is the same; you change the provider.

Watermarks and late events

For cases where an event can arrive late (for example, the Postgres WAL falls behind because of a network blip), Flink lets you define watermarks:

CREATE TABLE documents_raw (
  doc_id        STRING,
  title         STRING,
  body          STRING,
  updated_at    TIMESTAMP_LTZ(3),
  WATERMARK FOR updated_at AS updated_at - INTERVAL '5' MINUTE
) WITH (...)

This tells Flink “assume no event arrives more than 5 minutes late relative to the event timestamp”. For temporal joins and aggregations, Flink uses the watermark to decide when to “close” a window.

Layer 4 — Sinks to vector stores

The last step is indexing the embeddings in a vector store. Three patterns in 2026:

Pattern A — direct Kafka Connect sink

Each vector store has its official connector:

  • Milvus: official Zilliz sink connector. Supports named/unnamed dense/sparse vectors.
  • Qdrant: official sink connector. Supports dense, sparse, multi-vector.
  • pgvector: has no dedicated connector, but the JDBC Sink Connector is used with custom SQL.
  • Weaviate: community connector.
  • LanceDB: community connector.

An example Milvus sink configuration:

{
  "name": "milvus-rag-embeddings-sink",
  "config": {
    "connector.class": "com.milvus.io.kafka.MilvusSinkConnector",
    "tasks.max": "3",
    "topics": "rag.documents.embedded",
    "milvus.host": "milvus.prod.internal",
    "milvus.port": "19530",
    "milvus.collection.name": "documents",
    "milvus.collection.dim": "1536",
    "milvus.collection.partition": "default",
    "key.converter": "org.apache.kafka.connect.storage.StringConverter",
    "value.converter": "org.apache.kafka.connect.json.JsonConverter",
    "value.converter.schemas.enable": false
  }
}

Three parallel tasks (tasks.max: 3) consume the embedded topic and write to the Milvus collection. Latency from “event in Kafka” to “indexable vector in Milvus” is typically <5 seconds.

Pattern B — pgvector with a direct CDC pipe

For teams that already live in PostgreSQL, pgvector is the lowest-friction option. The pattern: the same source Postgres cluster holds a second DB for embeddings with the pgvector extension enabled; the Flink pipeline writes to it directly over JDBC.

-- On the Postgres cluster with pgvector enabled
CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE document_embeddings (
  doc_id TEXT,
  chunk_id INT,
  chunk_text TEXT,
  category TEXT,
  embedding vector(1536),
  embedded_at TIMESTAMP,
  PRIMARY KEY (doc_id, chunk_id)
);

CREATE INDEX ON document_embeddings 
  USING hnsw (embedding vector_cosine_ops)
  WITH (m = 16, ef_construction = 64);

Advantages: the same DBA operates everything, transactionality across tables, trivial joins with relational metadata. Limitation: beyond 10M vectors, pgvector performance starts to give way compared with dedicated systems.

Pattern C — Confluent Tableflow → Iceberg + Flink SQL vector search

This is the 2026 development that changes the mechanics. Confluent Tableflow materialises Kafka topics automatically as Apache Iceberg or Delta Lake tables. Features:

  • No ETL pipeline: you do not write Flink/Spark jobs to move Kafka into a table. Tableflow does it.
  • Automatic schema evolution: changes to the topic schema are reflected in the table.
  • Unified catalogue: the table appears in Glue, Unity Catalog, Snowflake, Databricks. Any analytical engine queries it without copying data.
  • Native CDC: handles inserts, updates and deletes correctly.
  • 30-50 % lower TCO according to the figures Confluent publishes against traditional pipelines.

And since 2026, Tableflow + Flink SQL offer native vector search integrated with Cosmos DB and Amazon S3 Vectors. The RAG query can be made directly in Flink SQL:

SELECT doc_id, chunk_text, category
FROM documents_embedded
WHERE VECTOR_SEARCH(embedding, 
                    OPENAI_EMBEDDING('user query', 'text-embedding-3-small'),
                    top_k => 10) > 0.7
ORDER BY VECTOR_SEARCH_SCORE DESC;

This unifies layers that used to be separate (vector store + analytics). For many cases, it removes the need to maintain a dedicated vector store.

Confluent’s official MCP server

One piece added in 2026 deserves a mention: Confluent has published an official MCP server that exposes Kafka, Flink and Tableflow as tools accessible to AI agents over MCP. Any MCP client (Claude Desktop, Cursor, your own agents) can:

  • List topics, read recent messages, publish to topics.
  • Run Flink SQL queries in natural language (“give me the orders from the last 24 hours worth more than 1000 €”).
  • Query Tableflow Iceberg tables.
  • Manage Kafka Connect connectors.

This closes the circle: as well as reading data from the datalake through RAG (with vector search), your AI agent can write data to the log (through MCP) and trigger transformations (through Flink SQL in natural language). It is the deepest point of fusion between LLM ops and data ops this year.

A connection with the earlier series: this MCP server emits traces with the OpenTelemetry GenAI MCP semantic conventions we covered in the MCP observability post. The spans show up in Langfuse, Phoenix or your OTel backend with the right cardinality. Zero instrumentation code.

Vector stores: a 2026 comparison

The five dominant options:

Vector storeLicenceOperationWhere it fits
pgvectorPostgres ext, OSSYour DBA<10M vectors, Postgres-heavy team
QdrantApache 2.0Self-host or managedMid-scale, performance focus
MilvusApache 2.0Self-host or Zilliz CloudLarge-scale, scalability focus
WeaviateBSD-3Self-host or managedNative hybrid search, semantically rich
LanceDBApache 2.0Embedded or serverlessSmall-medium, simplicity

The choice depends on:

  • Scale: pgvector falls short beyond 10M vectors. Milvus and Qdrant scale to billions.
  • Hybrid search: Weaviate brings lexical + vector natively. Others support it but less integrated.
  • Operation: pgvector if you already run Postgres. Qdrant if you want simplicity. Milvus if you need maximum scale.
  • Cloud managed: Zilliz Cloud for Milvus, Qdrant Cloud for Qdrant, Pinecone if you want pure SaaS (with no OSS behind it).

Freshness vs accuracy: the operational trade-off

A critical decision any RAG system on Kafka must answer: when is a new document considered “live” in the index?

Three options:

Synchronous streaming: the event reaches Kafka, Flink embeds it, the sink writes it to the vector store, and only then is it considered live. Typical latency: 1-5 seconds. The best freshness. But if the embedding model fails or the vector store is slow, events pile up in the topic.

Asynchronous streaming with a baseline: the event is considered live immediately; a background process embeds it when it can. In the meantime, queries asking for that document do not find it. Typical latency: 5-60 seconds. Acceptable for most applications.

Micro-batch: processing happens in mini-batches every 1-5 minutes. Less efficient than continuous streaming but more stable under variable load. Latency: 1-5 minutes.

The decision depends on the product’s SLA. For customer support chatbots, 5-60 seconds is acceptable. For systems that react to critical events (financial prices, alarms), synchronous streaming is necessary.

Schema evolution and re-embedding

When the embedding model changes (you move from text-embedding-3-small to text-embedding-3-large, or from OpenAI to Cohere), the vectors already in the index are incompatible: different dimensions, different semantic spaces. The distance between an old vector and a new one means nothing.

The standard pattern for handling this: dual-index during the migration.

  1. T0: the active index is V1 (embedding model A).
  2. T1: a parallel pipeline starts writing to a V2 index (embedding model B), consuming the topic from offset 0 (reprocessing the whole log).
  3. T2: V2 has caught up with the present.
  4. T3: you switch the retriever over to V2.
  5. T4: a week later, you discard V1.

The Kafka log makes this pattern feasible because it is immutable and reproducible. Without the log, this pattern becomes a weeks-long data migration project.

Operational pitfalls

Topics without adequate retention

Configuring topics with 7-day retention on the grounds that “I already have the vector store” means losing the ability to rebuild if the vector store fails. Long retention (90+ days) or compaction by key for topics that feed RAG.

Heavy CDC during peak load

Debezium reading the WAL at peak hours can hit the source database’s performance. A dedicated read replica for Debezium, not the production primary. Or use logical replication restricted to just the tables you need.

Embedding cost run-away

OPENAI_EMBEDDING on every event of a topic with millions of messages a day comes to thousands of USD per month. Strategies: filter before embedding (only embed what adds value); deduplicate by content hash; use self-hosted open-source embedding models (BGE, E5, GTE) when the cloud cost is prohibitive.

Slow re-embedding due to limited throughput

Recomputing 10M embeddings with the OpenAI API at 3000 req/min takes 55 hours. If you wait for an incident to re-embed, that is two days without service. Embedding throughput is an explicit capacity planning item; reserve capacity or keep an offline job ready to start.

Downstream schema breaks

A change to the raw topic’s schema breaks Flink jobs downstream. Schema Registry with BACKWARD compatibility made mandatory; never ALLOW_ALL. And test schema evolution in CI.

A vector store with no backup

Your vector store holds 50M vectors. It is the only copy (the topics expired). A failure wipes it. Vector stores must be backed up like any primary persistence layer. For Milvus/Qdrant: periodic snapshots. For pgvector: plain pg_dump.

Multi-region without cross-cluster replication

Your RAG serves users in the US and the EU. The vector store is in US-east. Latency from the EU = 100 ms+ per query. MirrorMaker or Cluster Linking to replicate topics and vector stores in both regions.

What we have not covered

  • Hybrid search in production: combining BM25/lexical + vector + reranker. A topic for its own post.
  • Multimodal RAG: indexing images, audio and video alongside text. Multimodal embeddings (CLIP, Imagebind), a specific architecture.
  • GraphRAG: using structured knowledge (knowledge graphs) as well as vector retrieval. Microsoft GraphRAG, LlamaIndex KnowledgeGraphQueryEngine.
  • RAG with multi-tenant ACLs: filtering by permissions at runtime. The pattern uses metadata in the vector store + server-side filters.
  • Query rewriting with an LLM: using a first LLM to expand the query before retrieval (HyDE, multi-query, step-back prompting).

References

Kafka and stream processing:

Vector store connectors:

Tableflow and 2026 architecture:

Streaming RAG:

Cross-references: