Postgresql on lo0 — Tech Blog

LiteLLM on day 2: one worker per pod, six thousand seconds of timeout and the other five things to change before opening the traffic

Tue, 08 Sep 2026 07:00:00 +0200

Second article in the operational track of the control layer. The operational pair with Langfuse covered the seam between the gateway and observability; here everything stays inside the gateway. The prior decisions, which gateway and why, are in choosing the OSS gateway and in the L7 inference router.

TL;DR

LiteLLM Proxy starts from a twenty-line config.yaml and serves traffic that same afternoon. The operational work consists of correcting seven defaults designed for a scenario other than an in-house inference factory.

One uvicorn worker per pod. The project’s production guide asks for --num_workers 1 on Kubernetes and horizontal scaling, with 1 vCPU and 4 GiB per worker set at the same time as requests and as limits. The 4 GiB are a floor, not a target: the Prisma query engine marks a resident memory high-water mark that grows up to the largest statement it has executed and that glibc does not return to the system.

Autoscale on CPU, never on memory. A direct consequence of the above. Memory goes up and never down, so an HPA on memory scales and never scales back. The recommended target is targetCPUUtilizationPercentage: 60.

The Postgres connection arithmetic. database_connection_pool_limit is 10, and the number of connections is instances times workers times the pool. The Helm charts ship maxReplicas: 100, on the order of a thousand connections, far above what a stock Postgres accepts.

Redis stops being optional from around a thousand requests per second or ten instances. Without it, each instance applies its limits on its own, caches are not shared, and spend updates against the same rows produce locks and exhaust the connections with FATAL: sorry, too many clients already.

Three health endpoints and only two are usable as probes. /health/liveliness and /health/readiness require no authentication and answer fast. Bare /health demands a key and fires a real request against every model in the catalogue, with its cost in tokens.

The resilience numbers are 2, 3 and 5: two retries, three allowed failures and five seconds of cooldown, the last one despite the function’s own documentation saying the default is 1.

request_timeout is 6,000 seconds. A hundred minutes holding a connection against an engine that does not answer.

And a warning about the performance figures: the project publishes 8 ms of p95 overhead on one benchmark and 257.7 ms of p99 on another. Both are theirs, and the difference is in the shape of the load.

You are here: the gateway layer, the day after

In the seven-layer stack, this article lives entirely in the gateway layer. The difference with the selection article is the one that separates buying from maintaining: there the question was which piece to put in front of the fleet, and here it is what happens when that piece has been serving for six months, the team has tripled the traffic and someone has added an agent that fires twenty calls per interaction.

The gateway has an uncomfortable property worth keeping in mind: it sits on the critical path of every request and contributes not a single token. All it does is coordination, and any coordination that fails turns into an outage of the whole inference service, even with the GPUs perfectly healthy.

The analogy: the switchboard and its notebook

A hospital switchboard does three jobs at once. It puts calls through, which is the urgent part. It writes down in a notebook who called, to which extension and for how long, which is what sustains the billing. And it consults a list of extensions to know where to put each call through.

The three jobs compete. If the operator stops to write each line in the notebook before putting the next call through, the waiting queue grows. If there are three operators and each keeps her own notebook, the totals do not add up. And if the list of extensions is consulted in a filing cabinet on another floor, every call costs the trip.

The three tensions have an exact equivalent in LiteLLM. The notebook is the SpendLogs in Postgres, which is why the transaction buffer in Redis exists. The three operators with three notebooks are the pod replicas applying limits independently. And the filing cabinet on another floor is usage-based routing, which adds a Redis lookup inside the request path.

The process model: one worker per pod

The production documentation is explicit: on Kubernetes, --num_workers 1 and scale with replicas. On a virtual machine with no orchestrator, NUM_WORKERS equal to the number of vCPUs.

The reason is not ideological. An additional worker inside the same pod shares the container’s memory limit with the others, multiplies the database connections by the same factor, and registers a copy of every periodic job. With separate replicas, each one has its own resource budget, its own connection quota and a place where the scheduler can put it.

The recommended sizing is 1 vCPU and 4 GiB per worker, and the part that gets overlooked is that these go as requests and as limits at the same time. Setting less than 4 GiB produces an OOM restart loop that shows up late, once some large request has grown the query engine’s footprint.

resources:
 requests: { cpu: "1", memory: "4Gi" }
 limits: { cpu: "1", memory: "4Gi" }

That footprint needs to be described precisely, because it determines the autoscaling policy. The Prisma query engine keeps a resident memory that behaves as a high-water mark: it grows up to the largest statement the process has executed and does not come back down, because glibc does not return that freed memory to the system. A pod that served a spike yesterday still shows the footprint of that spike today.

Two rules come out of this. The first: autoscale on CPU and never on memory. An HPA on memory sees a metric that only goes up, scales at the peak and never scales back. The recommended target is targetCPUUtilizationPercentage: 60, lower than the 80 the charts ship by default, because the startup probe allows up to 300 seconds before passing the first readiness check and there needs to be headroom for the new pod to be ready before the saturated one goes down.

The second: bound the process lifetime. --max_requests_before_restart 10000 recycles the worker before the high-water mark matters.

Postgres: the arithmetic that blows up when you scale

database_connection_pool_limit is 10 by default. The total number of connections the deployment opens is replicas times workers times pool, and it has to be compared against the Postgres max_connections.

The formula the documentation gives is the inverse, and it is the one to apply when sizing:

database_connection_pool_limit = MAX_DB_CONNECTIONS / (instances × workers)

The detail that hurts in production is in the Helm charts, which ship autoscaling.maxReplicas and keda.maxReplicas at 100. With the default pool, that is on the order of a thousand connections at full scale-out, far above what an untouched Postgres accepts. And the moment the HPA reaches those replicas is exactly the moment of highest load, so connection exhaustion arrives when it can least be afforded.

There are two more valves on the same path. Provider errors are written to the database by default, and under a sustained engine failure that inflates the spend table without contributing anything that is not already in the metrics. They are switched off like this:

general_settings:
 disable_error_logs: true # stops writing provider errors
 proxy_batch_write_at: 60 # batches the spend writes

disable_spend_logs: true is the radical version, which removes the per-request detail from the interface and leaves the cost in Prometheus and in the tracing backend. Before going that far you have to decide whether that table is the system’s audit record, which is the discussion in the previous article: if it is, it cannot be switched off, and the way out is the Redis buffer.

Redis: when it stops being optional

Without Redis, LiteLLM works. Each instance keeps its own in-memory cache and applies its own counters. The consequences are three, and all of them show up as you grow.

Requests-per-minute limits are applied per instance, so a limit of 100 with five replicas is in practice a limit of 500. Cache hits are local, so the same repeated question hits once in five. And there is no leader election, so periodic jobs run in every process at once.

The threshold the documentation gives for enabling the transaction buffer is a thousand requests per second or ten instances. Below that, each instance updates the key, user and team rows directly; above it, they all write to the same rows, locks appear and Postgres starts rejecting with FATAL: sorry, too many clients already.

general_settings:
 use_redis_transaction_buffer: true

The metrics to watch when that is enabled are litellm_in_memory_spend_update_queue_size, litellm_redis_spend_update_queue_size and litellm_pod_lock_manager_size. If the first grows without the second coming down, the flush to Postgres is not keeping up.

A note about the routing strategy, which interacts with this. simple-shuffle consults nothing external. Usage-based routing does, and it adds a round trip to Redis inside the request path. On a homogeneous vLLM fleet behind the same model, the improvement it brings rarely pays for that latency; on a heterogeneous fleet, the conversation is different and the router article covered it.

Background jobs register per worker

This detail appears in no getting-started guide and produces considerable bewilderment when it is discovered. The proxy’s periodic jobs register per uvicorn worker, not per pod. With --num_workers 4 and ten replicas that is forty copies of every job, all of them running with no coordination if there is no Redis for leader election.

The variable that separates them is LITELLM_JOB_ROLE: value serving on the pods that serve traffic and a separate replica with value worker for the jobs. With the one-worker-per-pod recommendation the problem shrinks, but it does not go away while there are several replicas.

Probes: three endpoints and only two are usable

The three endpoints exist and do different things:

Endpoint	Authentication	What it does	What it is for
`/health/liveliness`	No	Returns `I'm alive!`, or 503 during shutdown	`livenessProbe`
`/health/readiness`	No	State of the process and the database. 503 if the configured Postgres does not answer	`readinessProbe`
`/health`	Yes	Fires a real request against every model in the catalogue	Manual diagnosis, never a probe

The last row deserves insisting on. /health spends tokens on every call, one per configured model. Set as a probe, with the Kubernetes default interval and a fleet of eight models, that is thousands of inference requests a day that serve nobody.

There are more endpoints useful for diagnosis: /health/readiness/details, /health/services?service=langfuse to check one specific callback, /health/history and /health/latest for the history, and /health/backlog, which returns the in-flight requests and shows up in the latency section. /health/drain exists but returns 404 unless it is enabled with enable_drain_endpoint: true, and it is protected with X-Drain-Token.

The periodic model checks are configured separately and not in the probe:

general_settings:
 background_health_checks: true
 health_check_interval: 300 # seconds
 background_health_check_model_groups: ["llama-70b", "qwen-30b"]

And one setting with consequences in deployments with no Internet egress: allow_requests_on_db_unavailable: true allows serving traffic with the database down, but leaves /health/readiness always returning 200. With that set, the readiness probe stops detecting the very failure it was designed to detect, and budget and model errors keep blocking just the same.

Retries, cooldowns and fallbacks: the real numbers

The defaults, read from the project’s code and not from the documentation:

Constant	Value	What it governs
`DEFAULT_MAX_RETRIES`	2	Retries before moving on to fallbacks
`DEFAULT_ALLOWED_FAILS`	3	Failures before cooling down a deployment
`DEFAULT_COOLDOWN_TIME_SECONDS`	5	Seconds a deployment stays out
`SINGLE_DEPLOYMENT_TRAFFIC_FAILURE_THRESHOLD`	1000	Minimum requests before applying the cooldown logic with a single deployment
`DEFAULT_REQUEST_TIMEOUT_SECONDS`	6000.0	Request timeout

Besides the failure counter, there is a proportional criterion: a deployment enters cooldown if 50% of its requests fail in any given minute.

On the cooldown there is a discrepancy worth knowing about before debugging blind. The docstring of the router function says cooldown_time defaults to 1, and the code uses the constant of 5. The code wins.

The fallbacks are three separate lists and are applied after the retries are exhausted:

router_settings:
 num_retries: 2
 timeout: 120
 fallbacks: [{"llama-70b": ["qwen-30b"]}]
 context_window_fallbacks: [{"llama-70b": ["llama-70b-128k"]}]
 content_policy_fallbacks: [{"llama-70b": ["llama-70b-sin-guardrail"]}]

A note on the behaviour when everything is in cooldown: if no deployment is left available in the group, LiteLLM falls back onto a specific model_info.id, skipping the cooldown check. It is a deliberate degradation, serve something rather than return 503, and it is worth knowing it exists because it masks the real state of the fleet.

And a capability lost along the way: since version 1.85.0, the mock_testing_fallbacks parameter and its two siblings are stripped from requests arriving through the proxy and have no effect. Fallbacks can no longer be tested against the proxy with a doctored request; it has to be done against litellm.Router directly, in a test. Anyone who had a smoke test built on that has it broken without warning.

The six thousand second timeout

It deserves its own section because it is the setting with the worst damage-to-effort ratio in the whole system. request_timeout is 6,000 seconds. A hundred minutes. An inference engine that stops answering without closing the connection holds a worker for that long, and with one worker per pod that is an entire pod out of service per trapped request.

general_settings:
 request_timeout: 600

Ten minutes is still generous for a long generation, and it bounds the damage.

Overhead: the project’s two figures

LiteLLM publishes two measurements of its own overhead that differ by a factor of thirty, and both are legitimate because they measure different loads.

The benchmarks page, with Locust, a thousand users with think time and a fake endpoint, gives for four instances an x-litellm-overhead-duration-ms of median 2 ms, p95 8 ms and p99 13 ms at 1,170 requests per second. With two instances, median 12 ms, p95 29 ms and p99 43 ms. The page itself warns that those figures correspond to around 130 in-flight requests, and that a closed-loop client with no think time keeps a thousand in flight and sees on the order of eight times that latency at the same rate, by Little’s law.

The AIGatewayBench benchmark, from July 2026, measures the Python proxy at 257.7 ms of added p99 and 329.5 MB of peak memory, with no callbacks, no spend tracking and no persistence. The Rust variant, in beta, gives 0.7 ms.

The reading for an in-house platform: the first figure is the best case with human clients, and the second is the order of magnitude to plan for agentic load, which arrives in a closed loop against fast responses. There is one reported case that resembles this scenario closely: an in-house OpenAI-compatible engine served around 16 requests per second directly and around 9 through LiteLLM, with the degradation growing with concurrency, on a 4 vCPU, 8 GB pod.

The latency gap the metrics cannot see

LiteLLM’s timers start when its handler starts. The time the request spends queued in the uvicorn event loop, before getting there, appears in none of its metrics. The example in its own troubleshooting guide is a case in which LiteLLM records 10 seconds and the user experiences 20.

It is detected by comparing two sources: GET /health/backlog or the litellm_in_flight_requests gauge against the response time measured by the load balancer in front. If they diverge, the gap is in the input queue and the answer is more replicas, not more tuning.

Two headers are always on and serve to watch it without instrumenting anything: x-litellm-overhead-duration-ms and x-litellm-callback-duration-ms. For the second, the documentation sets the suspicion threshold at 100 ms, above which the diagnosis is that the payloads are too large.

And the first cause of latency that guide mentions is none of the above: LITELLM_LOG=DEBUG serialises the payload with json.dumps(indent=4) synchronously, and with payloads over 2 MB that alone can cost between 2 and 5 seconds per request. A log level set to debug a problem and forgotten there produces exactly the problem it was meant to debug.

The failure mode to put in the runbook

There is an open issue describing a cascade with a characteristic shape. Under sustained 429s from the engine upstream, the pods stopped answering readiness probes during startup, Kubernetes killed them on failed probes, and the restart put more pressure on an upstream that was already saturated. Peaks of 500 requests per second, average of 60, between two and five replicas with 1.3 CPU and 4 GB.

The shape of the failure is what matters more than the specific case: the engine’s saturation turns into a restart loop of the gateway, and from the outside it looks like the problem is the gateway. The defences are the ones already in this article, applied together: startupProbe with a wide margin so readiness does not compete with startup, request_timeout bounded so trapped requests get released, cooldowns that take the deployment returning 429 out of rotation, and CPU autoscaling at 60% so the capacity exists before it is needed. It fits with what was covered in incident response runbooks.

Upgrading without cutting traffic

The deployment is a normal Deployment with RollingUpdate, maxUnavailable: 0 and maxSurge: 1, and there is no further mystery to it except on one point: schema migrations. LiteLLM uses Prisma and applies migrations at startup. With several replicas coming up at once after a version change, the migration must be run by a single process, which is another reason for the dedicated replica with LITELLM_JOB_ROLE: worker.

Before a version bump, two checks that pay for themselves: read the release notes looking for changes in the metadata emitted to the callbacks, because a change like that breaks saved dashboards without touching the service, and verify that the new configuration starts with --detailed_debug on a separate pod before applying it to the fleet.

Go-live checklist

--num_workers 1 and scaling by replicas.
requests equal to limits, 1 vCPU and 4 GiB per worker.
HPA on CPU at 60%. Never on memory.
database_connection_pool_limit computed from max_connections, and the chart’s maxReplicas lowered to something the database can take.
Redis as soon as there is more than one replica, and use_redis_transaction_buffer from ten instances or a thousand requests per second.
livenessProbe on /health/liveliness, readinessProbe on /health/readiness, and /health out of the probes.
startupProbe with a margin of up to 300 seconds.
request_timeout: 600.
LITELLM_LOG out of DEBUG, and disable_error_logs: true.
A dashboard with x-litellm-overhead-duration-ms, x-litellm-callback-duration-ms, litellm_in_flight_requests and the response time of the load balancer in front.

Traps and things that are not what they look like

The HPA scales up and never back down. It is measuring memory, which in this process only goes up.

The pods die in a loop right after deploying. Less than 4 GiB of limit. The footprint appears with the first large request, not at startup.

Postgres starts rejecting connections at the peak. The chart has maxReplicas: 100 and the pool defaults to 10.

The per-minute limits are enforced badly. There is no Redis, and each replica keeps its own count.

The health check consumes tokens. Someone set /health as the readiness probe.

The readiness probe returns 200 with the database down. allow_requests_on_db_unavailable is set.

A deployment keeps receiving traffic after failing. The cooldown is 5 seconds, not minutes, and with a single deployment in the group you have to pass a thousand requests for the logic to apply.

The fallback test no longer tests anything. The simulation parameters are stripped from requests arriving through the proxy since 1.85.0.

The p99 is ten times worse than the published benchmark. The benchmark has think time and the real load is agentic.

Closing

None of these settings is complicated and all of them are one line. What unites them is that their defaults describe a small deployment, with one instance, a human client on the other side and a commercial provider taking care of capacity. An in-house inference factory is the opposite on all four axes: several replicas, automated clients, and an engine whose capacity is finite and known.

Of the ten points in the checklist, three decide almost everything in an on-premise deployment: the process model with its sizing, the database connection arithmetic, and the request timeout. The other seven are the ones that avoid the Saturday phone call.

Sources

LiteLLM, Production best practices (topología, workers, memoria, pool de conexiones, LITELLM_JOB_ROLE, buffer de transacciones en Redis, request_timeout): https://docs.litellm.ai/docs/proxy/prod.
LiteLLM, Health checks (los tres endpoints, comprobaciones de fondo, enable_drain_endpoint): https://docs.litellm.ai/docs/proxy/health.
LiteLLM, Reliability (fallbacks, reintentos, deprecación de los parámetros de simulación en la 1.85.0): https://docs.litellm.ai/docs/proxy/reliability.
LiteLLM, Benchmarks y Load test advanced (las cifras de sobrecarga por número de instancias y la advertencia de la ley de Little): https://docs.litellm.ai/docs/benchmarks · https://docs.litellm.ai/docs/load_test_advanced.
LiteLLM, Rust AI Gateway benchmarks (los 257,7 ms de p99 del proxy en Python): https://docs.litellm.ai/blog/rust-ai-gateway-benchmarks.
LiteLLM, Latency overhead (el hueco de la cola de uvicorn, las cabeceras de diagnóstico, el coste de LITELLM_LOG=DEBUG): https://docs.litellm.ai/docs/troubleshoot/latency_overhead.
LiteLLM, DB info (disable_spend_logs, disable_error_logs y qué se pierde con cada uno): https://docs.litellm.ai/docs/proxy/db_info.
LiteLLM, litellm/constants.py (los valores por defecto de reintentos, fallos permitidos, enfriamiento y timeout): https://github.com/BerriAI/litellm/blob/main/litellm/constants.py.
LiteLLM, litellm/router.py (la discrepancia entre el docstring de cooldown_time y la constante que usa el código): https://github.com/BerriAI/litellm/blob/main/litellm/router.py.
LiteLLM, issue #15526 — cascada de 429 sostenidos, sondas de readiness fallidas y bucle de reinicios: https://github.com/BerriAI/litellm/issues/15526.
LiteLLM, issue #21046 — degradación de throughput contra un motor propio compatible con OpenAI: https://github.com/BerriAI/litellm/issues/21046.

Debezium and CDC: the notary who hears changes before anyone asks

Thu, 04 Jun 2026 11:00:00 +0200

TL;DR

Change Data Capture (CDC) with Debezium listens to the PostgreSQL Write-Ahead Log and turns every INSERT, UPDATE and DELETE into a structured Kafka event. Unlike traditional polling (SELECT ... WHERE updated_at > ?), it detects deletes, has latency in the tens of milliseconds and adds no extra load to the database. In RAG pipelines, this means that when a document is deleted from Postgres, the Qdrant chunks disappear too, automatically and in real time. The supporting infrastructure is modest: the connector uses 2-4 cores and 4-8 GB RAM to process thousands of events per second.

The master analogy: the land registry notary

Picture the land registry. Every time a flat is sold, mortgaged, or a mortgage is cancelled, the registrar records the operation in the registry ledger, a chronological and immutable journal. If you want to know what has changed in the registry, you have two options.

Option A (polling): you send someone every 5 minutes with a list of properties to ask “has anything changed?”. Problems: if a title was cancelled (DELETE), the property no longer exists by the time your envoy arrives, and there is no trace of it. If 20 different departments are doing the same thing, that is 20 people pestering the registrar every 5 minutes. And the minimum latency is the interval: 5 minutes.

Option B (Debezium): you hire a notary who sits right at the registrar’s desk. Every time the registrar signs an operation into the ledger, the notary notes it down immediately and notifies whoever needs to know. Title cancellations included, because the notary sees them as clearly as any other operation: he was there when it was signed.

In this analogy:

The registry ledger is the PostgreSQL WAL (Write-Ahead Log).
The notary is the Debezium connector.
The notary’s bookmark, which guarantees he loses no page even if he steps out for a moment, is the logical replication slot.
The messenger who carries the notifications to interested parties is Kafka (or Redpanda, or NATS JetStream).

We will pick this thread up in every section. When a technical detail is unclear, go back to the image of the notary.

1. The problem CDC solves

The most common synchronisation pattern between services sharing PostgreSQL is periodic polling:

SELECT id, content, updated_at
FROM documents
WHERE updated_at > $1
ORDER BY updated_at
LIMIT 1000;

This pattern has three structural problems.

DELETEs are invisible. When you delete a row, updated_at is not updated: the row is gone. The next time the poller queries, the row does not exist and there is no way to know it ever did. In a RAG pipeline, this means orphan chunks in Qdrant: the document no longer exists in Postgres, but its vectors keep polluting search results.

The minimum latency is the interval. If the poller runs every 5 seconds, average latency is 2.5 seconds. For near-real-time synchronisation (dashboards, alerts, RAG with frequently changing documents), that is too much.

Load scales with the number of consumers. If 10 services poll the same table every 5 seconds, that is 10 × 12 = 120 queries/minute producing no useful work; they only check whether there is anything new. On large tables with complex indexes, that is real load on the database.

CDC inverts the model: the database notifies, the consumers listen. Zero polling, zero extra load, DELETEs included, latency in the tens of milliseconds.

2. What the PostgreSQL WAL is

The operations journal

The Write-Ahead Log (WAL) is the chronological, immutable record of every operation Postgres performs. Before modifying any data page on disk, Postgres writes the operation to the WAL. That sequence, log first then data, is what guarantees durability (the D in ACID) and enables crash recovery: if Postgres dies mid-transaction, on restart it replays the WAL to return the database to a consistent state.

The WAL is the registry ledger of our analogy: chronological, immutable, complete.

Physical vs logical replication

PostgreSQL supports two WAL-based replication modes:

Physical replication: it replicates disk blocks verbatim. The standby receives the same bytes as the primary. It serves high availability and failover, but the target must be an exact copy of Postgres; you cannot send the changes to an external application.
Logical replication: instead of disk blocks, it replicates semantic operations: “the row with id=42 was inserted into table documents with these values”. The target can be anything that understands the protocol: another Postgres, Debezium, or any custom consumer.

CDC uses logical replication. It is what lets Debezium understand “what changed and in which table” rather than “which disk block changed at which offset”.

The replication slot: the notary’s bookmark

A logical replication slot is a persistent cursor in the WAL. Postgres keeps a record of how far into the WAL each slot has consumed. While a slot exists, Postgres guarantees it will not discard the WAL segments the slot has not yet read.

This is exactly the notary’s bookmark: even if the notary goes out for lunch, the ledger stays open at the last page he read. When he comes back, he carries on from exactly where he left off, having lost nothing.

The risk is the inverse: if the notary never comes back, the bookmark stops the registrar from archiving the old pages. If the Debezium connector goes down and does not recover for hours, the WAL grows indefinitely on disk until the slot is deleted manually or the consumer starts consuming again. This is called WAL disk blowup and is the most important operational risk of Debezium.

Mandatory monitoring:

SELECT slot_name,
 confirmed_flush_lsn,
 pg_current_wal_lsn(),
 pg_wal_lsn_diff(pg_current_wal_lsn(), confirmed_flush_lsn) AS lag_bytes
FROM pg_replication_slots
WHERE slot_type = 'logical';

The pgoutput plugin

The WAL stores operations in an internal binary format. For Debezium to understand them, Postgres needs to decode them into a readable format. The pgoutput decoding plugin, included in the Postgres core since version 10, does exactly that: it translates the binary WAL events into messages with the before/after structure of each row.

Debezium uses pgoutput by default. It requires no external extensions (unlike the wal2json plugin that was popular before Postgres 10).

3. Debezium architecture

The connector as a Kafka Connect plugin

Debezium is not a standalone service; it is a plugin for the Kafka Connect framework. Kafka Connect manages the connector’s lifecycle (start, stop, reconnection, offset tracking) and provides the parallelism and fault tolerance infrastructure.

The connector talks to Postgres through the logical replication protocol (not over JDBC), using the credentials of a user with the REPLICATION role.

PostgreSQL (WAL + pgoutput)
│
│ logical replication protocol
▼
Debezium Connector (Kafka Connect worker)
│
│ Kafka Producer API
▼
Kafka topic: rag.public.documents
│
▼
Consumer (Qdrant sync, audit log, fine-tuning pipeline...)

Structure of a Debezium event

Each change in the table becomes a JSON message with this structure:

INSERT ("op": "c" — create):

{
 "before": null,
 "after": {
 "id": 42,
 "content": "Contrato de arrendamiento...",
 "tenant_id": "acme",
 "updated_at": 1748934000000
 },
 "op": "c",
 "source": {
 "version": "2.7.0.Final",
 "connector": "postgresql",
 "db": "rag_db",
 "schema": "public",
 "table": "documents",
 "lsn": 29823948,
 "txId": 1047,
 "ts_ms": 1748934000123
 }
}

DELETE ("op": "d"):

{
 "before": {
 "id": 42,
 "content": "Contrato de arrendamiento...",
 "tenant_id": "acme",
 "updated_at": 1748934000000
 },
 "after": null,
 "op": "d",
 "source": { "lsn": 29824102, "txId": 1051, "ts_ms": 1748934060200 }
}

The before field holds the row’s previous state, available because Postgres can be configured with REPLICA IDENTITY FULL to include the complete row in the WAL on delete or update. Without that setting, before contains only the primary key.

This is the key for the RAG pipeline: the DELETE event carries the document id. The consumer uses it to delete every associated chunk in Qdrant with a doc_id = 42 filter. Without CDC, those chunks would never have been deleted.

Initial snapshot

When the connector starts for the first time (or after a reset), it cannot begin consuming the WAL from “the beginning of time”, only from the moment the slot was created. How does it guarantee the consistency of the initial state?

Through a transactional snapshot: the connector opens a transaction in REPEATABLE READ mode, exports the snapshot ID (pg_export_snapshot()), and does a full SELECT of the configured tables inside that transaction. It then starts consuming the WAL from the snapshot’s LSN. That way there is no gap: the snapshot covers the state up to an instant, and the WAL covers from that instant onwards.

SMT transformations (Single Message Transforms)

Before emitting the event to the Kafka topic, the connector can apply inline transformations called SMT. Common use cases:

Filtering sensitive columns (ReplaceField with blacklist): strip password_hash, phone_number before they reach the topic.
Adding metadata (InsertField): enrich the event with a tenant_id extracted from the original HTTP header (if it is in the row).
Conditional routing (Filter): drop events from rows with status = 'draft' before emitting them.

SMTs are pure configuration, requiring no code, and they are applied inside the connector process with no perceptible extra latency.

4. Debezium vs the Outbox pattern

The Outbox pattern is the most common alternative to pure CDC. Instead of emitting events directly to Kafka, the application writes to an outbox table in Postgres within the same transaction that modifies the data. A separate worker reads that table and publishes the events.

Criterion	Debezium (pure CDC)	Outbox pattern
Event latency	~50-200 ms from commit	Depends on the worker interval (typically 1-5 s)
Consistency	At-least-once	At-least-once
DELETE detection	Native (the DELETE event includes `before`)	Only if the app writes to the outbox on delete
Setup complexity	High (Kafka Connect, replication slot, permissions)	Low (an extra table plus a simple worker)
Infrastructure dependency	Requires Kafka/Redpanda/NATS JetStream	Postgres plus a worker; Kafka optional
WAL disk blowup risk	Yes, if the slot stops consuming	No
Schema visibility	Reads the table’s real schema	The event schema is defined by the app
Schema migration	Needs care (events reflect DDL changes)	More flexible (the event is whatever the app puts there)
When to use it	When you need DELETEs, low latency, or you cannot modify the app	When the app controls the event domain and infrastructure is limited

Rule of thumb: if you control the application code and do not need native DELETEs, the Outbox is simpler. If you do not control the code (legacy database, third-party application) or DELETEs are critical (a RAG pipeline with document deletion), Debezium is the right choice.

5. The maths

Throughput

Debezium on a connector with 4 workers can process between 10,000 and 50,000 events/second on modest hardware (4 cores, 8 GB RAM). The real bottleneck is not the connector but the Kafka broker: with 3 brokers and suitable partitions, Kafka can comfortably sustain 500,000 messages/second with 1 KB messages (source: public Confluent benchmarks, 2023).

For a typical RAG pipeline with 100 documents modified per minute:

$$\text{events/s} = \frac{100}{60} \approx 1{.}7 \text{ events/s}$$

That is 0.0034% of the connector’s capacity. Debezium will not be the bottleneck in any realistic RAG scenario.

End-to-end latency

The path from a commit in Postgres to an upsert in Qdrant has these stages:

Stage	Typical latency
Commit in Postgres → WAL written	< 1 ms (synchronous with the commit)
WAL written → Debezium reads it (WAL lag)	10-50 ms
Debezium → Kafka produce (ack)	5-20 ms
Kafka → Consumer (poll interval)	0-100 ms (configurable)
Consumer → Qdrant upsert/delete	5-15 ms
Typical total	30-200 ms

With fetch.min.bytes=1 and fetch.max.wait.ms=10 on the consumer, the Kafka poll latency drops to ~10 ms. The realistic range for an optimised pipeline is 30-100 ms.

WAL disk blowup risk

If the connector stops consuming, Postgres retains the WAL from the slot’s confirmed_flush_lsn onwards. The retained volume grows linearly with time and the write rate:

$$\text{retained WAL} = \text{write rate} \times \text{average WAL event size} \times \text{time without consuming}$$

Example with a moderate load (50,000 writes/hour, 500 bytes average per WAL event):

$$50{,}000 \times 500 \text{ B} \times 1 \text{ h} = 25 \text{ MB/h}$$

With a high load (1,000,000 writes/hour):

$$1{,}000{,}000 \times 500 \text{ B} \times 1 \text{ h} = 500 \text{ MB/h}$$

If the connector is down for 48 hours under high load: 24 GB of retained WAL. That can fill the disk and block Postgres completely.

Recommended alert: set an alert when lag_bytes > 1 GB or when confirmed_flush_lsn has not advanced for more than 15 minutes. See the monitoring query in section 2.

6. Use cases in LLMOps / RAG

RAG synchronisation with real deletion

This is the use case that most clearly justifies Debezium over polling. The flow:

A user deletes document id=42 from the document management interface.
Postgres runs DELETE FROM documents WHERE id = 42.
Debezium detects the DELETE in the WAL and emits the event with "op": "d" and "before": {"id": 42, ...}.

The consumer receives the event and runs:

qdrant_client.delete(
 collection_name="documents",
 points_selector=Filter(must=[FieldCondition(key="doc_id", match=MatchValue(value=42))])
)

Every chunk with doc_id=42 disappears from Qdrant in ~100 ms.

Without Debezium, those chunks would stay indefinitely, polluting retrieval results with fragments of documents that no longer exist in the source of truth.

Event sourcing for fine-tuning datasets

Every time a human annotator updates a row in the annotations table (correcting an LLM output), Debezium emits the UPDATE with before and after. The consumer writes the pair (original_output, correction) into the dataset curation pipeline, with no need for the annotator to do anything beyond saving in the interface. The fine-tuning pipeline knows exactly what changed and when, with no polling and no risk of duplicates from overlapping time windows.

Immutable audit log

WAL events are, by definition, the most faithful record of what happened in the database: they are the same data Postgres uses for crash recovery. Kafka with long retention (90 days, or size-based retention) serves as an immutable audit log without modifying the application schema or adding triggers. This is particularly useful in regulated environments where traceability of data modifications is required.

7. Architecture diagram

8. Minimum configuration

PostgreSQL: enable logical replication

-- Requires restarting Postgres
ALTER SYSTEM SET wal_level = logical;
ALTER SYSTEM SET max_replication_slots = 10;
ALTER SYSTEM SET max_wal_senders = 10;

-- Reload configuration (wal_level requires a full restart)
SELECT pg_reload_conf();

-- Dedicated user for Debezium
CREATE USER debezium WITH REPLICATION LOGIN PASSWORD 'cambiar_esto';
GRANT SELECT ON TABLE public.documents TO debezium;

-- REPLICA IDENTITY FULL to get a complete 'before' on DELETEs and UPDATEs
ALTER TABLE public.documents REPLICA IDENTITY FULL;

Debezium connector (Kafka Connect REST API)

{
 "name": "postgres-debezium",
 "config": {
 "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
 "database.hostname": "postgres",
 "database.port": "5432",
 "database.user": "debezium",
 "database.password": "cambiar_esto",
 "database.dbname": "rag_db",
 "topic.prefix": "rag",
 "table.include.list": "public.documents",
 "plugin.name": "pgoutput",
 "slot.name": "debezium_rag",
 "publication.autocreate.mode": "filtered",
 "snapshot.mode": "initial",
 "tombstones.on.delete": "true",
 "transforms": "unwrap",
 "transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState",
 "transforms.unwrap.drop.tombstones": "false",
 "transforms.unwrap.delete.handling.mode": "rewrite"
 }
}

curl -X POST http://kafka-connect:8083/connectors \
 -H 'Content-Type: application/json' \
 -d @connector-config.json

Check its status:

curl http://kafka-connect:8083/connectors/postgres-debezium/status

Minimal consumer in Python

from confluent_kafka import Consumer
from qdrant_client import QdrantClient
from qdrant_client.models import Filter, FieldCondition, MatchValue
import json

consumer = Consumer({
 "bootstrap.servers": "kafka:9092",
 "group.id": "qdrant-sync",
 "auto.offset.reset": "earliest",
 "enable.auto.commit": False,
})
consumer.subscribe(["rag.public.documents"])
qdrant = QdrantClient("qdrant", port=6333)

while True:
 msg = consumer.poll(timeout=0.1)
 if msg is None:
 continue
 event = json.loads(msg.value())
 op = event.get("op")
 if op in ("c", "u"): # INSERT or UPDATE
 doc = event["after"]
 # ... vectorise and upsert into Qdrant
 elif op == "d": # DELETE
 doc_id = event["before"]["id"]
 qdrant.delete(
 collection_name="documents",
 points_selector=Filter(
 must=[FieldCondition(key="doc_id", match=MatchValue(value=doc_id))]
 )
 )
 consumer.commit()

9. On-premise deployment

The Debezium stack does not compete for GPU. On a node with 4×H100 SXM (320 GB, NVLink) serving the inference LLM, the CDC pipeline runs entirely on general-purpose (CPU-only) nodes:

Component	Recommended resources	Role
Kafka Connect + Debezium	2-4 cores, 4-8 GB RAM	Read the WAL, emit events
Kafka brokers (×3)	4 cores, 32 GB RAM each	High availability, retention
Qdrant-sync consumer	2 cores, 4 GB RAM	Vectorise plus upsert/delete
Qdrant	8 cores, 64 GB RAM	Vector store

The Debezium connector is notably light: in production at 10,000 events/second, the connector typically uses less than 1 core and 2 GB of RAM. JVM memory (Kafka Connect runs on the JVM) must be capped explicitly with -Xmx4g to stop the GC from causing pauses.

For high availability, Kafka Connect supports distributed mode with multiple workers. If a worker dies, the connector is reassigned automatically to another worker within seconds, and the replication slot guarantees no events are lost during the switchover.

What we have not covered

Debezium with MySQL, MongoDB and Oracle: each connector uses the native log mechanism (binlog on MySQL, oplog on MongoDB, LogMiner on Oracle). The resulting event API is similar, but the configuration details and limitations differ.
Debezium Server: standalone mode without Kafka Connect, with direct sinks to HTTP, S3, Redis Streams or NATS. Useful when the Kafka infrastructure is too complex for the use case.
Schema Registry: how Avro with Confluent Schema Registry or Apicurio manages event schema evolution (adding columns, changing types) without breaking existing consumers.
Exactly-once semantics: why at-least-once is enough for most RAG cases (an idempotent upsert into Qdrant with the same vector does no harm) and when exactly-once is needed (financial counters, inventory deductions).
Outbox pattern plus Debezium combined: Debezium reading the outbox table instead of the business table’s WAL directly, the Transactional Outbox + CDC pattern that combines the best of both worlds.

References

Debezium Documentation — PostgreSQL Connector. debezium.io/documentation/reference/stable/connectors/postgresql.html
PostgreSQL Documentation — Logical Replication. postgresql.org/docs/current/logical-replication.html
PostgreSQL Documentation — Write-Ahead Logging. postgresql.org/docs/current/wal-intro.html
PostgreSQL Documentation — Replication Slots. postgresql.org/docs/current/logicaldecoding-explanation.html
Confluent — Kafka Performance Benchmarks (2023). confluent.io/blog/kafka-fastest-messaging-system
Gunnar Morling — Outbox Pattern. morling.dev/blog/sending-messages-as-part-of-database-transactions
Debezium — SMT documentation. debezium.io/documentation/reference/stable/transformations
Qdrant Documentation — Filtering. qdrant.tech/documentation/concepts/filtering

Function calling and tool-augmented retrieval: the detective who knows which file to ask for

Thu, 04 Jun 2026 10:00:00 +0200

This post goes deeper into the agentic reasoning mechanism that extends the RAG described in RAG with reranker and hybrid retrieval. The retriever invoked when the LLM chooses vector_search is exactly the pipeline in that article. The JSON Schema that defines each tool call is structured output applied to the tool interface. And the agent’s requests pass through the L7 inference gateway before reaching the model.

TL;DR

A naive RAG always queries the same source. Function calling gives the LLM the ability to decide which tool to invoke (vector store, SQL, web search) according to what the query actually needs. The ReAct pattern chains those invocations in a reasoned loop until there is enough evidence. A 3-iteration pipeline with Llama-3.1-70B on on-premise hardware takes ≈ 1.1 s against the ≈ 300 ms of a single-step RAG; the gain is not in speed but in queries that a naive RAG simply cannot answer. The critical eval metric is tool selection accuracy: the percentage of turns in which the model picks the right tool, measured over a synthetic eval set.

The analogy: the detective who knows which file to ask for

A detective in a novel does not go to the same filing cabinet regardless of the case that lands on his desk. When he takes a case, he reasons first: what kind of evidence do I need? If there are witnesses, he asks for the statements (vector search over unstructured documents). If there are financial transactions, he asks the bank for the accounting records (SQL over the structured database). If the suspect has recent activity that the company cannot possibly have indexed, he goes to the press archive (web search). He does not consult all three sources at once on every case: he picks the one the evidence requires, receives the result, reasons again if he needs more, and only writes the report when he has enough evidence.

A bad detective always goes to the same filing cabinet. A naive RAG is that bad detective: it vectorises the query, goes to the vector store, and returns whatever it finds even if the question was “how many contracts?”, something no PDF chunk can answer better than a COUNT(*) in SQL.

Function calling is giving the LLM the ability to reason about which source to ask for, and to invoke it in a structured way. The analogy has three edges worth keeping in mind:

The detective does not improvise the file he asks for: there is a catalogue of available sources with a description of what each one contains. The tool description in the system prompt does that job.
The detective can ask for several pieces of evidence at once if they are independent (parallel tool calling).
The detective knows when to stop: if after N rounds he reaches no conclusion, he declares that he does not have enough evidence. The agent has an iteration limit for the same reason.

What function calling is: the anatomy of a tool call

Function calling, also called tool use, is a mechanism by which the LLM, instead of generating free text as its answer, generates a structured JSON object representing a tool invocation. The system intercepts that JSON, executes the real tool, and returns the result as a message with the tool role in the conversation.

Defining tools in the system prompt

Each tool is defined through a JSON Schema that specifies name, description and parameters. This JSON Schema is exactly the same mechanism described in structured output, applied here to the tool interface:

{
 "tools": [
 {
 "name": "vector_search",
 "description": "Search internal company documents about policies, contracts and procedures. Use when the query requires unstructured text, document context or semantic similarity.",
 "parameters": {
 "type": "object",
 "properties": {
 "query": {"type": "string", "description": "Natural language search query"},
 "top_k": {"type": "integer", "default": 5}
 },
 "required": ["query"]
 }
 },
 {
 "name": "sql_query",
 "description": "Query the SQL database for structured metrics, counts, aggregations and financial data. Use when the query requires exact numbers, filters, sums or joins over structured records.",
 "parameters": {
 "type": "object",
 "properties": {
 "query": {"type": "string", "description": "Parameterized SQL query with $1, $2 placeholders"},
 "params": {"type": "array", "items": {}, "description": "Parameter values for the placeholders"}
 },
 "required": ["query"]
 }
 },
 {
 "name": "web_search",
 "description": "Search public web for real-time information, recent news or current prices. Use only when data is public and not covered by internal sources.",
 "parameters": {
 "type": "object",
 "properties": {
 "query": {"type": "string"}
 },
 "required": ["query"]
 }
 }
 ]
}

The life cycle of a tool call

When the LLM decides to invoke a tool, the message it generates in place of free text has this structure (OpenAI-compatible format, the same one vLLM supports):

{
 "role": "assistant",
 "tool_calls": [
 {
 "id": "call_01",
 "type": "function",
 "function": {
 "name": "sql_query",
 "arguments": "{\"query\": \"SELECT COUNT(*), SUM(amount) FROM contracts WHERE amount > $1 AND year = $2 AND provider_region = $3\", \"params\": [100000, 2025, \"EU\"]}"
 }
 }
 ]
}

The system executes the tool and returns:

{"role": "tool", "tool_call_id": "call_01", "content": "{\"count\": 47, \"total\": 8300000}"}

The LLM receives that message as a continuation of the conversation and decides whether it needs more information or can generate the final answer.

Support in OSS models

In 2026, native function calling support (not emulated via system prompt) is available in:

Llama 3.1 / 3.3+: native tool call format, supported in vLLM with --enable-auto-tool-choice --tool-call-parser llama3_json
Qwen 2.5+: native support with --tool-call-parser hermes
Mistral NeMo / Mistral 7B Instruct v0.3+: native format with --tool-call-parser mistral
Qwen3 (2025+): extended native support with parallel tool calling

Without --enable-auto-tool-choice, vLLM can force the tool call format via guided decoding (JSON Schema), but the ability to decide when to invoke a tool versus generating free text requires specific model training.

The ReAct pattern: Reasoning + Acting

ReAct (Yao et al., 2022) is the loop that turns an LLM with tools into an agent with reasoning. The central idea is that the model interleaves thoughts (explicit reasoning about what to do) with actions (tool invocations) and observations (results received). This loop repeats until the model reckons it has enough information to answer, or until the iteration limit is reached.

Thought → Action → Observation → Thought → Action → Observation → ... → Response

A full ReAct trace: a real production query

Query: “How many contracts above 100k€ did we sign in 2025 with EU providers, and which special conditions recur?”

This query requires two different sources: the exact numbers live in SQL, and the context of special conditions lives in the contract PDFs indexed in the vector store.

Iteration 1

Thought: The query asks for aggregated data (count and sum) with exact filters over structured fields (amount, year, region). This is a SQL query, not a semantic search.

Action: sql_query

SELECT COUNT(*) AS num_contratos, SUM(amount) AS total_eur
FROM contracts
WHERE amount > $1 AND year = $2 AND provider_region = $3

params: [100000, 2025, "EU"]

Observation: {"num_contratos": 47, "total_eur": 8300000}

Iteration 2

Thought: I have the numbers. Now I need the special conditions that recur in these contracts. That kind of information is in the free text of the PDFs, not in the structured table. I go to the vector store.

Action: vector_search

"EU provider contracts 2025 amount above 100k special conditions recurring clauses"

top_k: 5

Observation:

[chunk_1] Provider A contract — clause 8.3: late-delivery penalty of 2% per month on the invoiced amount...
[chunk_2] Provider C contract — clause 7.1: price revision linked to the euro area CPI capped at 4% per year...
[chunk_3] Provider B contract — clause 8.3: late-delivery penalty of 2% per month...

Thought: I have enough data from both sources. Clause 8.3 on late-delivery penalties appears in at least two of the retrieved contracts. I can synthesise the answer.

Response: “In 2025 you signed 47 contracts with EU providers above 100,000 €, for a cumulative total of 8.3 M€. The special condition that appears most often in the higher-value contracts is the late-delivery penalty clause (8.3 in the contract templates of Provider A and B): 2% per month on the invoiced amount. The price revision linked to the euro area CPI, capped at 4% per year (clause 7.1), also recurs.”

This trace has 2 iterations. A more complex query (comparing with the previous year, cross-referencing with payment status) would add further iterations. Common practice is to set a limit of 5–10 iterations maximum to avoid loops.

The three sources and when to use each one

The choice of source is neither arbitrary nor left solely to the LLM: the architecture defines which tools exist and how they are described. The table below summarises the selection criteria:

Characteristic	Vector store (Qdrant)	SQL / structured (PostgreSQL)	Web search
Data type	Free text, documents, PDFs	Tables with a fixed schema	Public pages, news
Structure	Unstructured	Highly structured	Semi-structured
Temporality	Static index (periodic refresh)	Real time (transactional)	Real time (crawl)
Typical latency	5–50 ms	10–200 ms	200–2,000 ms
Privacy	Internal data, full sovereignty	Internal data, full sovereignty	Public data only
Natural query	Yes (natural language → embedding)	No (parameterised SQL)	Yes
Exact aggregations	No	Yes (`COUNT`, `SUM`, `GROUP BY`)	No
When to use	Document context, semantics, PDFs	Metrics, counts, exact filters, joins	Data that does not exist internally and is public

The most important rule of thumb: if the question contains words like “how many”, “sum”, “total”, “more than X”, “in year Y” and the data is in a structured table, the right answer is sql_query. If the question asks for context, explanations, clauses, procedures or examples from documents, the answer is vector_search. If it asks for the current price of something or recent news about a third party, web_search, but only if no data sovereignty is involved.

Tool routing: how the LLM picks the right tool

The description of each tool in the system prompt is the most critical factor for routing accuracy. An LLM with good function calling capabilities can still choose badly if the descriptions are ambiguous or overlap.

Descriptions that work versus those that do not

Weak description (leads the LLM to use the wrong tool):

"search_docs" — Searches for information in the available sources.
"query_data" — Gets data from the system.

Strong description (delimits precisely when to use each one):

"vector_search" — Search internal company documents about policies, contracts and procedures.
Use when the query requires unstructured text, document context or semantic
similarity. NOT for counts, sums or exact filters.
"sql_query" — Query the SQL database for structured metrics, counts, aggregations and
financial data. Use when the query requires exact numbers, filters, sums or
joins over structured records. NOT for finding document context.

The difference lies in two elements: (1) positive use-case examples, and (2) explicit exclusions with NOT for. Both reduce the semantic overlap between tools and improve tool selection accuracy.

Parallel tool calling

When two tools are independent of each other, that is, when the result of one does not affect the query of the other, the LLM can invoke them simultaneously in the same turn:

{
 "tool_calls": [
 {"id": "call_01", "function": {"name": "sql_query", "arguments": "..."}},
 {"id": "call_02", "function": {"name": "vector_search", "arguments": "..."}}
 ]
}

The system executes both in parallel and returns both observations before the LLM’s next turn. This cuts total latency when the queries are independent: instead of 2 sequential iterations (2 × tool_latency), you pay 1 × max(sql_latency, vector_latency). For the detective example: if he needs both the accounting records and the witness statements to answer, he can ask for them at the same time.

Tool selection accuracy: the eval metric

Tool selection accuracy is the percentage of turns in which the LLM picks the right tool given a set of evaluated queries:

\[ \text{TSA} = \frac{\text{turns with correct tool chosen}}{\text{total turns with expected tool call}} \]

It is measured over a synthetic eval set built with (query, expected_tool, expected_args) triples. An example of a minimal eval set:

Query	Expected tool	Failure indicator
“How many orders in March?”	`sql_query`	LLM uses `vector_search`
“What does the returns policy say?”	`vector_search`	LLM uses `sql_query`
“What is the price of copper today?”	`web_search`	LLM uses `vector_search`
“Sum the Q3 contracts”	`sql_query`	LLM uses `vector_search`

A TSA < 0.85 in a production agent is a sign that the tool descriptions need revising before the model does. For more detail on how to build these evals, see LLM evals.

Error handling in tool calls

SQL injection via prompt

The most serious risk in tool-augmented retrieval is that the LLM generates malicious SQL, either because a user induced it via prompt injection, or because the model hallucinated a destructive query. This attack vector is covered in detail in LLM Guard, but the minimum rules on the tool side are:

Rule 1: parameterised queries always, never direct interpolation.

# NEVER this:
cursor.execute(f"SELECT * FROM contracts WHERE provider = '{llm_output}'")

# Always this:
cursor.execute("SELECT * FROM contracts WHERE provider = $1", [llm_output])

Rule 2: DB user with minimum permissions. The user the agent runs SQL as must have SELECT on the necessary tables and nothing else. No DROP, INSERT, UPDATE or DELETE.

Rule 3: table allowlist. The system validates that the query generated by the LLM only references tables on an allowlist before executing it.

Rule 4: per-query timeout. Queries that block for too long, potentially induced to DoS the database, are cancelled with a configured timeout.

Rate limits, timeouts and tool errors

When a tool fails, the error is returned to the LLM as an observation:

{"role": "tool", "tool_call_id": "call_01", "content": "{\"error\": \"timeout after 5s\", \"tool\": \"web_search\"}"}

The system prompt must instruct the LLM on what to do in this case:

If a tool returns an error or is unavailable, acknowledge the limitation in your response.
Do not retry more than once. If web_search is unavailable, state that real-time data
is not accessible at this moment and answer with available internal sources only.

This stops the agent from entering retry loops and handles graceful degradation: if web_search is unavailable, it answers with what it has in the internal sources.

Diagram of the ReAct loop with the three sources

Latency maths of the ReAct pipeline

Each iteration of the ReAct loop has three latency components:

\[ T_{\text{iter}} = \text{TTFT}_{\text{LLM}} + T_{\text{tool}} + \Delta_{\text{context}} \]

where:

TTFT_LLM: time to the LLM’s first token (dominated by the prefill of the accumulated context)
T_tool: tool execution time
Δ_context: overhead of the growing context window (each iteration adds the previous output to the context)

Reference values: Llama-3.1-70B on 4×H100 SXM (320 GB, NVLink)

With Llama-3.1-70B in FP8 on a node with 4×H100 SXM (320 GB HBM3, NVLink 900 GB/s), the typical values in production are:

Component	Value
TTFT with context < 4k tokens	≈ 150 ms
TTFT with context 8k tokens	≈ 220 ms
`sql_query` (simple query, indexed)	≈ 50 ms
`vector_search` (top-5, Qdrant in RAM)	≈ 20 ms
`web_search` (external API)	≈ 600 ms

Total latency by number of iterations

1-iteration pipeline (simple query, a single tool):

\[ T_1 = 150 + 50 = 200 \text{ ms} + \text{final synthesis} \approx 200 + 300 = 500 \text{ ms} \]

2-iteration pipeline (SQL + vector_search in sequence):

\[ T_2 = (150 + 50) + (180 + 20) + 400 = 800 \text{ ms} \]

The context in the second iteration already includes the result of the first, so TTFT rises slightly to ≈ 180 ms.

3-iteration pipeline (the most common case in complex queries):

\[ T_3 = (150 + 50) + (180 + 20) + (200 + 50) + 450 \approx 1,100 \text{ ms} \]

Parallel tool calling (SQL + vector_search in parallel, a single iteration):

\[ T_{\text{parallel}} = 150 + \max(50, 20) + 400 = 600 \text{ ms} \]

When the two queries are independent, parallel tool calling trims latency from ≈ 800 ms to ≈ 600 ms: a 25% improvement over the 2 sequential iterations case.

Comparison with a naive RAG

Configuration	Latency	Queries it can answer
Naive RAG (1 retriever, 1 step)	≈ 300 ms	Document context queries
ReAct 1 iteration (SQL)	≈ 500 ms	Structured aggregation queries
ReAct 2 iterations (SQL + vector)	≈ 800 ms	Hybrid numeric + context queries
ReAct 3 iterations	≈ 1,100 ms	Complex multi-source queries
ReAct with web_search	≈ 1,500 ms	Queries requiring real-time data

The practical conclusion: multi-hop ReAct is 3–5× slower than a single-step naive RAG. The gain is not in speed but in query coverage: a naive RAG cannot answer “how many contracts?” because that answer is in no text chunk at all. For applications with a strict latency SLO (< 500 ms), you have to work out whether the use case really needs ReAct or whether a well-configured RAG with hybrid retrieval covers 90% of the queries.

On-premise hardware for ReAct agents

A ReAct agent with Llama-3.1-70B in production has different requirements from a naive RAG, because the context grows with each iteration and prefill throughput is more critical.

Recommended configuration: 4×H100 SXM (320 GB HBM3, NVLink 900 GB/s)

Llama-3.1-70B in FP8: fits in 2×H100 (70B params × 1 byte/param ≈ 70 GB + KV cache). With 4×H100 you can serve it with tensor parallelism TP=4, cutting prefill TTFT by ≈ 2×.
Qdrant instance: it can sit on the same node (if the collection fits in RAM) or on a dedicated node. For collections < 50M vectors of 768 dims: ≈ 150 GB, which fits in the RAM of a dual-socket server.
PostgreSQL: separate node or managed instance. The agent adds no unusual load to SQL; the queries are simple and bounded by timeout.
vLLM with --enable-auto-tool-choice --tool-call-parser llama3_json --max-model-len 16384: a 16k token context comfortably covers the 5–10 turns of a ReAct pipeline.

Minimum viable configuration: 2×H100 SXM (160 GB)

Llama-3.1-70B in FP8 with TP=2. TTFT ≈ 250–300 ms for 4k token contexts (a 60–100% increase over TP=4).
Good enough for internal workloads with < 20 concurrent requests.
Not advisable for an SLO < 1 s with more than 5 concurrent users and a long context.

What we have not covered

Agentic retrieval loops with planning. ReAct is the simplest agent pattern. When a query requires decomposition into sub-tasks with dependencies, you need orchestration frameworks such as LangGraph (state graphs), smolagents (Hugging Face, agents with Python code as actions) or llama-index Agents (planning + retrieval pipeline). These introduce a planning step before the execution loop.

MCP (Model Context Protocol). Anthropic’s emerging standard, with OSS implementations, for defining tools portably across frameworks and hosts. Instead of defining each tool’s JSON Schema separately in every application, MCP centralises those definitions in an MCP server that any compatible client can discover and invoke. Adoption in 2025–2026 is fast among OSS frameworks (LangChain, smolagents, OpenWebUI).

Tool caching. If the same tool call (same arguments, same tool) is going to be invoked several times within the same context or in very similar contexts, the result can be cached. The mechanism is analogous to the semantic cache described for RAG: before executing the tool, the hash of the arguments (or its embedding, for semantic matching) is compared against a cache with a TTL. Especially valuable for sql_query with frequent queries and data that changes little.

Multi-agent. When an orchestrating agent delegates sub-tasks to specialised agents, one for SQL, one for document retrieval, one for code generation, you enter multi-agent system territory. Each sub-agent can have its own tool set and its own LLM (possibly smaller and more specialised). Coordination between agents introduces additional tracing and observability complexity.

References

Yao, S. et al. (2022). ReAct: Synergizing Reasoning and Acting in Language Models. arXiv:2210.03629. https://arxiv.org/abs/2210.03629
vLLM documentation. Tool calling. https://docs.vllm.ai/en/stable/features/tool_calling.html
Qdrant documentation. Search. https://qdrant.tech/documentation/concepts/search/
OpenAI. Function calling. https://platform.openai.com/docs/guides/function-calling
Meta AI. Llama 3.1 Model Card. https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/MODEL_CARD.md
Qwen Team (Alibaba). Qwen2.5 Technical Report. arXiv:2412.15115. https://arxiv.org/abs/2412.15115
Anthropic. Model Context Protocol. https://modelcontextprotocol.io
OWASP. LLM Top 10 for Large Language Model Applications. LLM01: Prompt Injection. https://owasp.org/www-project-top-10-for-large-language-model-applications/

PostgreSQL + Qdrant in RAG ingestion: the postman who keeps two worlds in sync

Thu, 04 Jun 2026 07:00:00 +0200

TL;DR — In a production RAG system, PostgreSQL holds the official truth about documents and Qdrant holds the vectors for search. Keeping them in sync is not trivial: if you delete a document from Postgres and do not invalidate its chunks in Qdrant, the system returns answers from ghost documents. There are two patterns to avoid this: the outbox pattern (atomic transaction plus asynchronous worker, at-least-once) and CDC with Debezium (reading the Postgres WAL directly, low latency, higher complexity). This article explains when to use each one, how to orchestrate them as microservices and what numbers to expect with bge-m3 on on-premise hardware.

The postman and the civil registry

Picture a town with two complementary offices.

The first is the civil registry: it keeps the official census. Every time someone is born, dies or moves house, the registry is the first to know. It is slow, structured, transactional. If the registry says someone exists, they exist. If it says they died, they are dead. PostgreSQL is the civil registry of your documents.

The second is the postman’s notebook: a copy optimised for finding any resident in seconds, organised by area, phonetic names and habitual routes. The postman cannot update the registry, but can search at speeds the registry would never reach. Qdrant is the postman’s notebook.

The problem is synchronisation. If the registry records a death but nobody tells the postman, he will keep trying to deliver letters to an address that no longer exists. In RAG, that translates into indexed chunks of documents that have already been deleted, edited or replaced: ghost documents that pollute the results.

How does the registry tell the postman?

Outbox pattern: every time the registry updates its ledger, it notes the change on an outbox sheet. A messenger clerk reads that sheet periodically and updates the postman’s notebook. Guaranteed, asynchronous, fault tolerant.
CDC with Debezium: the postman has a direct phone line to the registry. Every time the clerk writes something new, the phone rings and the postman updates his notebook in near real time.

The consistency gap problem

In a naive RAG architecture, the flow is:

The user uploads a document → it is inserted into Postgres with metadata.
A worker splits it into chunks, generates embeddings and upserts into Qdrant.
Retrieval uses Qdrant to find relevant chunks and Postgres to hydrate metadata.

So far so good. The problem shows up in updates and deletes:

The user edits a document → Postgres updates the row, but the vectors of the old chunks are still in Qdrant. Retrieval returns stale context.
The user deletes a document → Postgres removes the row, but the chunks remain in Qdrant. Retrieval returns chunks from a document that should no longer exist.
The permission system revokes a tenant’s access → Qdrant has no way of knowing unless there is explicit synchronisation.

This is not a theoretical problem. In living corpora (corporate wikis, knowledge bases updated daily), the consistency gap accumulates noise progressively. An internal study on production pipelines shows that without active reconciliation, 3-8% of indexed chunks correspond to documents that no longer exist in the source of truth after 30 days of operation.

The answer is not “reindex everything every night”. With 10M chunks and a non-trivial embedding model, that costs hours of compute and causes unavailability windows. The answer is change propagation with guarantees.

Outbox pattern: the outbox sheet

Mechanism

The outbox pattern solves the problem of “writing to two systems in the same operation” without distributed transactions (which are expensive and fragile).

The idea is simple: PostgreSQL is the single coordinator. When the ingestion microservice processes a document, it performs two writes in the same local transaction:

It inserts or updates the document in the documents table.
It inserts an event into the outbox_events table.

If the transaction fails, both writes are rolled back. If it succeeds, both are committed atomically. There is no inconsistent intermediate state.

-- Relevant tables
CREATE TABLE documents (
 id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
 tenant_id TEXT NOT NULL,
 title TEXT NOT NULL,
 content TEXT NOT NULL,
 checksum TEXT NOT NULL,
 updated_at TIMESTAMPTZ DEFAULT now()
);

CREATE TABLE outbox_events (
 id BIGSERIAL PRIMARY KEY,
 aggregate_id UUID NOT NULL, -- document id
 event_type TEXT NOT NULL, -- 'document.created' | 'document.updated' | 'document.deleted'
 payload JSONB NOT NULL,
 created_at TIMESTAMPTZ DEFAULT now(),
 processed_at TIMESTAMPTZ -- NULL = pending
);

-- Example of an atomic insert
BEGIN;
 INSERT INTO documents (tenant_id, title, content, checksum)
 VALUES ($1, $2, $3, $4)
 RETURNING id INTO _doc_id;

 INSERT INTO outbox_events (aggregate_id, event_type, payload)
 VALUES (_doc_id, 'document.created', jsonb_build_object(
 'tenant_id', $1,
 'title', $2,
 'checksum', $4
 ));
COMMIT;

The outbox worker

A separate process (the outbox worker) polls outbox_events where processed_at IS NULL, processes each event (chunking, embedding, upsert into Qdrant) and marks the row as processed:

UPDATE outbox_events
 SET processed_at = now()
 WHERE id = $1;

Guarantee: at-least-once. If the worker fails between the Qdrant upsert and the UPDATE, the event will be reprocessed. Qdrant tolerates idempotent upserts (same point id = overwrite), so reprocessing does not create duplicates.

Latency: it depends on the polling interval. With polling every 500ms, p50 latency is ~250ms; p99, ~500ms. Acceptable for most RAG cases, where the user does not expect to see a document indexed in under a second.

CDC with Debezium: the direct phone line

Mechanism

Change Data Capture (CDC) reads PostgreSQL’s Write-Ahead Log (WAL) directly. Postgres writes every change to the WAL before applying it to the tables; it is the mechanism it uses for replication and recovery. Debezium subscribes to a logical replication slot and turns those events into structured messages.

-- Enable logical replication in postgresql.conf
-- wal_level = logical

-- Create a replication slot for Debezium
SELECT pg_create_logical_replication_slot('debezium_slot', 'pgoutput');

The full flow:

Postgres WAL → Debezium connector → Kafka/NATS → Indexer consumer → Qdrant

Debezium emits events with the record’s before/after structure:

{
 "op": "d",
 "before": {
 "id": "550e8400-e29b-41d4-a716-446655440000",
 "tenant_id": "acme",
 "checksum": "sha256:abc123"
 },
 "after": null
}

With "op": "d" (delete), the consumer knows it must delete every point in Qdrant whose payload contains that document_id.

# Consumer: delete by payload filter
qdrant_client.delete(
 collection_name="corpus",
 points_selector=FilterSelector(
 filter=Filter(
 must=[
 FieldCondition(
 key="document_id",
 match=MatchValue(value=event["before"]["id"])
 )
 ]
 )
 )
)

Upsides and downsides

CDC removes polling and cuts latency to tens of milliseconds (WAL propagation time plus consumer processing). But it adds operational complexity: you need to manage the replication slot (unconsumed slots retain WAL indefinitely, which can fill the disk), the message broker and the consumer offset state.

Comparison: outbox vs CDC

Criterion	Outbox pattern	CDC with Debezium
Typical latency	250ms – 2s	20ms – 200ms
Delivery guarantee	At-least-once	At-least-once
Operational complexity	Low (Postgres only)	High (Debezium + broker)
WAL retention risk	None	High if the slot stalls
Idempotency required	Yes (in the indexer)	Yes (in the consumer)
Multi-table support	Manual	Automatic (any table)
Backpressure	Natural (polling)	Needs explicit design
When to choose it	Corpus < 100k docs/day, small team	Corpus > 1M docs/day, low latency critical

Rule of thumb: start with outbox. Move to CDC when the change volume goes past ~50k events/hour or when latency measured in seconds becomes unacceptable for the use case (for example, real-time news indexing).

Microservice architecture

The ingestion pipeline is made up of three microservices with cleanly separated responsibilities:

Microservice 1: Ingestor

Responsibilities: receive documents, split them into chunks and request embeddings. It does not write to Qdrant directly.

# ingestor/main.py (simplified)
from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
 chunk_size=512, # tokens, not characters
 chunk_overlap=64,
 length_function=token_count,
)

def ingest_document(doc: Document, db: Session) -> None:
 chunks = splitter.split_text(doc.content)
 with db.begin():
 db.execute(
 "UPDATE documents SET checksum=$1 WHERE id=$2",
 [doc.checksum, doc.id]
 )
 db.execute(
 """INSERT INTO outbox_events (aggregate_id, event_type, payload)
 VALUES ($1, 'document.updated', $2)""",
 [doc.id, {"chunks": chunks, "tenant_id": doc.tenant_id,
 "model": "bge-m3", "model_version": "1.0.0"}]
 )

Microservice 2: Indexer

It reads the outbox, generates embeddings by calling the inference server (vLLM or TEI) and upserts into Qdrant.

# indexer/worker.py
import asyncio
from qdrant_client import QdrantClient
from qdrant_client.models import PointStruct, VectorParams, Distance

qdrant = QdrantClient(host="qdrant-service", port=6333)

async def process_event(event: dict) -> None:
 payload = event["payload"]
 chunks = payload["chunks"]
 doc_id = str(event["aggregate_id"])

 # Batch embedding on TEI (Text Embeddings Inference)
 embeddings = await embed_batch(chunks, model="bge-m3")

 points = [
 PointStruct(
 id=f"{doc_id}_{i}",
 vector=emb,
 payload={
 "document_id": doc_id,
 "tenant_id": payload["tenant_id"],
 "chunk_index": i,
 "text": chunks[i],
 "model_version": payload["model_version"],
 }
 )
 for i, emb in enumerate(embeddings)
 ]

 # If it is an update, delete the previous chunks first
 if event["event_type"] in ("document.updated", "document.deleted"):
 qdrant.delete(
 collection_name="corpus",
 points_selector=filter_by_doc_id(doc_id)
 )

 if event["event_type"] != "document.deleted":
 qdrant.upsert(collection_name="corpus", points=points)

Microservice 3: Reconciler

The reconciler is the safety net. Periodically (every hour, for example) it compares the set of document_id values in Postgres with the set of document_id values in Qdrant. IDs present in Qdrant but absent from Postgres are ghosts: they get deleted.

# reconciler/diff.py
async def reconcile(tenant_id: str) -> int:
 pg_ids = set(await fetch_all_doc_ids(tenant_id))
 qdrant_ids = set(await scroll_all_doc_ids(tenant_id)) # paginated scroll

 orphans = qdrant_ids - pg_ids
 if orphans:
 logger.warning("Orphan chunks for %d documents", len(orphans))
 for doc_id in orphans:
 qdrant.delete("corpus", filter_by_doc_id(doc_id))

 return len(orphans)

Throughput and ingestion maths

Embedding throughput

The bge-m3 model (1024 dimensions, dense + sparse + colbert support) on a node with 4×H100 SXM (320 GB NVLink) running via vLLM or HuggingFace TEI reaches roughly 2,000 chunks/second with batch size = 256 and 512-token sequences.

$$\text{throughput} = 4 \times 500 \text{ chunks/s/GPU} = 2{,}000 \text{ chunks/s}$$

The figure of 500 chunks/s per GPU comes from public TEI benchmarks with bge-m3 on H100 SXM5, batch=256, seq_len=512 ¹.

Total re-ingestion time

For a corpus of 10M chunks:

$$t = \frac{10{,}000{,}000 \text{ chunks}}{2{,}000 \text{ chunks/s}} = 5{,}000 \text{ s} \approx 83 \text{ minutes}$$

That is pure embedding time. Adding Qdrant write latency (~0.5ms per upsert batch of 100 points):

$$t_{\text{qdrant}} = \frac{10{,}000{,}000}{100} \times 0.5\text{ ms} = 50{,}000 \text{ ms} = 50 \text{ s}$$

Estimated total for a full re-ingestion: ~85-90 minutes on a 4×H100 node.

Storage cost in Qdrant

Each bge-m3 vector has 1024 dimensions in float32 (4 bytes):

$$\text{size per vector} = 1{,}024 \times 4 \text{ B} = 4{,}096 \text{ B} = 4 \text{ KB}$$

For 10M chunks (dense vectors only):

$$\text{total vectors} = 10^7 \times 4{,}096 \text{ B} = 40.96 \text{ GB}$$

Adding the JSON payload (estimated at ~500 bytes/chunk):

$$\text{payload} = 10^7 \times 500 \text{ B} = 5 \text{ GB}$$

HNSW index (roughly 1.2× the vector size for $m=16$):

$$\text{HNSW} \approx 40.96 \text{ GB} \times 1.2 = 49.15 \text{ GB}$$

Estimated total on disk: ~95 GB for 10M chunks with dense bge-m3.

With scalar quantization (int8), the vector size shrinks 4×:

$$\text{with int8 quantization} \approx \frac{40.96}{4} + 5 + \frac{49.15}{4} \approx 27.5 \text{ GB}$$

Configuration	Vectors	HNSW	Payload	Total
float32, no quantization	40.96 GB	49.15 GB	5 GB	~95 GB
int8 scalar quantization	10.24 GB	12.29 GB	5 GB	~28 GB
binary quantization	1.28 GB	1.54 GB	5 GB	~8 GB

Binary quantization loses recall precision (~2-5% in NDCG@10), but it lets you host much larger corpora in RAM. For production where recall is critical, int8 is the usual balance point.

Recommended on-premise hardware

For a continuous ingestion pipeline in production:

Embedding node: 4×H100 SXM (320 GB, NVLink), 2× 64-core CPU (EPYC 9654), 1 TB DDR5 RAM, 100 GbE. Runs vLLM or TEI serving bge-m3. Sustained throughput: ~2,000 chunks/s with an asynchronous batch pipeline.

Qdrant node: 32-core CPU, 256 GB RAM (to keep the HNSW index in memory with 10M chunks and no quantization), 2 TB NVMe (Qdrant snapshot and WAL writes). Qdrant recommends that the HNSW index fit in RAM for p99 latency < 5ms.

PostgreSQL node: 16-core CPU, 128 GB RAM, 4 TB NVMe for the WAL (especially relevant if you use CDC with a logical replication slot; the slot retains WAL until Debezium consumes it).

Broker (if CDC): 3-broker Kafka with 500 GB NVMe per node, or NATS JetStream with 3 nodes for more modest loads.

Kubernetes manifests

Indexer Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
 name: rag-indexer
 namespace: rag-pipeline
spec:
 replicas: 3
 selector:
 matchLabels:
 app: rag-indexer
 template:
 metadata:
 labels:
 app: rag-indexer
 spec:
 containers:
 - name: indexer
 image: registry.example.com/rag-indexer:1.0.0
 env:
 - name: POSTGRES_DSN
 valueFrom:
 secretKeyRef:
 name: pg-credentials
 key: dsn
 - name: QDRANT_HOST
 value: "qdrant-service.qdrant.svc.cluster.local"
 - name: EMBEDDING_ENDPOINT
 value: "http://tei-service.embeddings.svc.cluster.local:8080"
 - name: POLL_INTERVAL_MS
 value: "500"
 resources:
 requests:
 memory: "512Mi"
 cpu: "500m"
 limits:
 memory: "2Gi"
 cpu: "2"

Reconciler CronJob

apiVersion: batch/v1
kind: CronJob
metadata:
 name: rag-reconciler
 namespace: rag-pipeline
spec:
 schedule: "0 * * * *" # hourly
 concurrencyPolicy: Forbid
 jobTemplate:
 spec:
 template:
 spec:
 restartPolicy: OnFailure
 containers:
 - name: reconciler
 image: registry.example.com/rag-reconciler:1.0.0
 env:
 - name: POSTGRES_DSN
 valueFrom:
 secretKeyRef:
 name: pg-credentials
 key: dsn
 - name: QDRANT_HOST
 value: "qdrant-service.qdrant.svc.cluster.local"
 - name: SCROLL_PAGE_SIZE
 value: "1000"
 resources:
 requests:
 memory: "256Mi"
 cpu: "250m"
 limits:
 memory: "1Gi"
 cpu: "1"

Production gotchas

Reindexing when the embedding model changes

This is the most painful problem. If you move from bge-m3 to nomic-embed-text-v2, the vectors are incompatible: they live in different embedding spaces and the cosine distances between them are meaningless.

The answer is dual-index aliasing:

Create a new collection in Qdrant: corpus_v2.
Re-embed the whole corpus with the new model and load corpus_v2.
When the new collection is complete and validated (recall test), switch the corpus_prod alias from corpus_v1 to corpus_v2.
Delete corpus_v1 once traffic has migrated.

During the migration the two indexes coexist. The retriever uses the alias, not the collection’s direct name.

Index versioning

Store model_version in the payload of every point in Qdrant. This lets you:

Filter by version during retrieval (useful in model A/B testing).
Have the reconciler detect points with an old version and reprocess them selectively.
Audit: know which model generated each embedding.

# Filter by model version at retrieval time
results = qdrant.search(
 collection_name="corpus",
 query_vector=query_embedding,
 query_filter=Filter(
 must=[
 FieldCondition(key="model_version", match=MatchValue(value="1.0.0")),
 FieldCondition(key="tenant_id", match=MatchValue(value=tenant_id)),
 ]
 ),
 limit=10
)

Namespace per tenant (multi-tenancy)

There are two strategies in Qdrant:

Strategy	Pros	Cons
Collection per tenant	Total isolation, no extra filter	N collections = N HNSW indexes in RAM
Payload filter per tenant	A single collection, less RAM	The filter adds ~10-15% search latency

For fewer than 100 tenants with large corpora (> 1M chunks/tenant), use a collection per tenant. For hundreds or thousands of tenants with small corpora, use a payload filter with tenant_id indexed:

qdrant.create_payload_index(
 collection_name="corpus",
 field_name="tenant_id",
 field_schema=PayloadSchemaType.KEYWORD
)

What we have not covered

Streaming corpus updates with near-real-time CDC: selective chunk invalidation when only one section of a document changes (incremental chunking based on content diff, not full re-chunking).
Multi-tenant corpus isolation with per-chunk ACLs: going beyond the tenant_id filter towards group, role or even individual document permissions applied at retrieval time.
Federated corpus: corpora distributed across cross-border silos where regulation (GDPR, CCPA) prevents centralising the embeddings; federated search patterns without moving data.
Zero-downtime incremental reindexing using dual-index aliasing: the full model migration protocol with rollback, recall regression testing and progressive traffic splitting.

References

HuggingFace Text Embeddings Inference — benchmarks oficiales con modelos de la familia bge en hardware A100/H100. https://github.com/huggingface/text-embeddings-inference ↩︎

Continuous fine-tuning in production: from real traffic to the deployed adapter

Thu, 21 May 2026 10:00:00 +0200

TL;DR

Continuous fine-tuning is not “training the model every so often”. It is a closed loop where real production traffic generates the datasets, a short pipeline trains a LoRA adapter, a battery of evaluations decides whether it is promoted, and vLLM loads it without restarting. The state of the art in May 2026 has fragmented the stack: it is no longer DPO against everything, but a choice between SFT, DPO, KTO, ORPO and SimPO according to the type of signal your product captures. What has consolidated the pattern is the combination of PostgreSQL 18 + pgvector 0.8 as the nervous system of the pipeline, traffic capture, dataset versioning, eval results, adapter registry, together with vLLM multi-LoRA hot-swap, which turns deployment into an HTTP call. This article takes the cycle apart with concrete schemas, real queries, and the numbers it costs on an RTX 4090 against a 4×H100 cluster.

You are here: Tune + Retrain

This post crosses two stages of the six-stage LLMOps pipeline: the decision to train a new adapter (the Tune stage) is triggered by the signals from Observe that travel through the Retrain stage until the loop closes. The post takes apart the complete circuit between the two boxes.

Picture a neighbourhood restaurant with a signature dish that works, but the chef knows it can be tuned. Every night things happen:

Some diners leave part of the dish: a weak signal that something did not quite land.
Others ask for another version (“could you put less salt in it?”): an explicit, directional signal.
Others finish the dish and come back the following week: the only signal that really matters, but it arrives late.
And a select group offers an opinion without being asked, usually a negative one.

The chef does not redo the menu every night. He does something more interesting: he writes down in a notebook the dishes served, the returns, the changes requested, the tips. Every so often, he reads the whole notebook, decides on minimal adjustments to a recipe, tries the new version at a private table with his staff, and only if they take to it does he add it to the next day’s menu. Sometimes he even serves two different versions of the dish to different tables for a week, measures what happens, and picks one.

That is continuous fine-tuning. The notebook is Postgres. The dish is the base model. The notes are feedback signals, explicit and implicit. The “minimal adjustment” is a 30 MB LoRA adapter. The private table is the battery of automated evaluations. The next day’s menu is vLLM with multi-LoRA hot-swap, which loads the new adapter without restarting the service. Serving two versions to different tables is A/B testing with real traffic.

The analogy is exact on one critical point: the chef does not throw away the original recipe. He keeps the base recipe and holds a separate notebook with the “modifications that work well for the neighbourhood regulars”. That notebook is the LoRA adapter: on top of the base model, not in its place.

The cycle, taken apart

Before going into components, it helps to fix the complete flow. These seven steps are what any serious team replicates with variations:

The cycle lasts between 1 and 4 weeks in real production. What changes between teams is the tempo (faster in assistant chat, slower in regulated banking) and the details of each step. The structure is the same.

Why continuous fine-tuning (and why it is not RAG)

Before going deeper, a distinction that people still confuse. Fine-tuning is for form, not for facts. If your problem is that the model does not know the customer’s rates or the updated catalogue, do not fine-tune: use RAG. If your problem is that the model responds in a tone that does not fit, does not respect your JSON format, refuses legitimate cases or invents structure, then fine-tuning is the answer.

In 2026 the boundary is well established by community practice:

Observed problem	Solution
The model does not know X (X changes weekly)	RAG
The model knows X but responds badly in tone or format	SFT fine-tuning
There are two ways of responding and I prefer one over the other	Fine-tuning with preferences (DPO/KTO/ORPO/SimPO)
The model reasons badly in a verifiable domain (code, maths)	RL with verifiable reward (GRPO/DAPO)
The model is competent, it just needs a memory of facts	RAG, not fine-tuning

Continuous fine-tuning is the disciplined version of the second and third cases. The key word is continuous: it is not a one-off “we aligned the model” event, it is a process that runs every time the traffic distribution drifts far enough, or new use cases appear.

The four techniques according to the signal you capture

The most important change of the last 12 months has been the end of DPO’s monopoly. In 2024 every team doing alignment used DPO with (chosen, rejected) pairs. In 2026 the choice is finer and depends on what the signal you collect in your product looks like:

Real signal in the product	Recommended technique	Why
Labelled correct examples (input → expected output)	SFT + LoRA	Still the baseline. 500-5,000 examples are enough for style.
Explicit `(chosen, rejected)` pairs	DPO or SimPO	SimPO removes the reference model → 50 % less VRAM in training.
Loose 👍 / 👎 on responses	KTO	The method that fits real telemetry most naturally.
SFT and preferences in a single pass	ORPO	A single model in memory, avoids drift between phases.
Verifiable reward (tests, solutions)	GRPO / DAPO	Reasoning, not chat. A different world.

The practical rule: design the feedback capture in the product thinking about which method you will be able to use afterwards. If your UI only has 👍/👎, you force the path to KTO. If you add a “regenerate response” button, you unlock DPO from regenerate-as-rejected (we will see it below). If you add an “edit response” button, the edited response becomes high-quality direct SFT.

There is a cost detail that gets little publicity. DPO needs to keep two models in memory: the one you are training and the reference one. SimPO removes that second model. ORPO does too. For a Llama 3 8B in BF16 this is the difference between needing ~32 GB of active VRAM during training (DPO) or ~16 GB (SimPO/ORPO). It is the difference between the training fitting on an RTX 4090 with aggressive QLoRA, or not fitting without offload.

Postgres as the nervous system of the pipeline

Here is the strong technical opinion of this article, and it is one worth defending with data: Postgres 18 + pgvector 0.8 + an S3/MinIO bucket for the weights is enough for the whole pipeline. You do not need MLflow, you do not need lakeFS, you do not need DVC.

This is not ideological minimalism. It is about three concrete advantages that no alternative stack matches in the on-premise scenario with compliance:

A single source of truth, a single authorisation model. The ACLs you already have for Postgres cover the training data, the eval results, the adapter registry and the audit log. You do not multiply control planes.
SQL as the pipeline’s universal language. The query that generates the dataset, the eval gate predicate, the A/B traffic assignment, the promotion decision: it is all SQL. Your team already knows SQL.
Audit and cryptographic reproducibility for free. The pg_audit and pgcrypto extensions, combined with set_hash over the dataset, give you cryptographic traceability with no additional code. It is a subject that would fill an article of its own.

A concrete schema

We start with the traffic table, partitioned by week so that DROP PARTITION is cheap:

CREATE TABLE obs.inference_log (
 id BIGSERIAL,
 request_id UUID NOT NULL,
 tenant_id INT NOT NULL,
 user_hash BYTEA, -- GDPR pseudonymisation
 adapter_id TEXT NOT NULL, -- e.g. "support-es-v4.1"
 experiment TEXT, -- e.g. "rerank-v2-canary"
 variant CHAR(1), -- 'A' | 'B' | NULL
 messages JSONB NOT NULL,
 completion TEXT,
 ttft_ms INT,
 tokens_in INT,
 tokens_out INT,
 -- feedback signals
 fb_explicit SMALLINT, -- -1/0/+1 (KTO-ready)
 fb_regen BOOLEAN DEFAULT false, -- user regenerated -> DPO-rejected
 fb_edited BOOLEAN DEFAULT false, -- user edited -> SFT golden
 parent_id BIGINT, -- self-reference for regenerate
 -- vector and meta
 embedding HALFVEC(1024), -- pgvector 0.8, half the RAM
 pii_flags SMALLINT DEFAULT 0, -- bitmask
 created_at TIMESTAMPTZ DEFAULT now()
) PARTITION BY RANGE (created_at);

CREATE TABLE obs.inference_log_2026w21 PARTITION OF obs.inference_log
 FOR VALUES FROM ('2026-05-18') TO ('2026-05-25');

CREATE INDEX ON obs.inference_log_2026w21
 USING hnsw (embedding halfvec_cosine_ops);
CREATE INDEX ON obs.inference_log_2026w21
 (tenant_id, adapter_id, created_at);

Three decisions deserve a note:

HALFVEC(1024). Native FP16 vectors in pgvector 0.8. Half the RAM and disk, with a loss of precision that is irrelevant for semantic deduplication. This alone, at a scale of millions of rows, saves between 4 and 8 GB.

Weekly partitioning by time range. At 90 days, DROP TABLE obs.inference_log_2026wXX frees space in milliseconds without a prolonged lock. Autovacuum never touches frozen partitions again.

Self-referencing parent_id. The user regenerates the response → a new row is inserted with parent_id pointing at the previous one. That will give us a DPO dataset without touching the UX.

The adapter registry

CREATE TABLE serve.adapter (
 id TEXT PRIMARY KEY,
 base_model TEXT NOT NULL,
 rank INT, alpha INT,
 target_modules JSONB,
 method TEXT, -- 'sft'|'dpo'|'kto'|'orpo'|'simpo'
 training_run_id UUID,
 dataset_snapshot_id UUID,
 weights_uri TEXT, -- s3://.../v4.2.safetensors
 eval_summary JSONB,
 status TEXT NOT NULL, -- 'training'|'canary'|'prod'|'retired'
 traffic_pct NUMERIC(5,2) DEFAULT 0,
 promoted_at TIMESTAMPTZ
);

The vLLM router reads this table with a TTL of a few seconds. An UPDATE serve.adapter SET status='prod', traffic_pct=100 WHERE id='v4.2' is a promotion. An UPDATE ... SET status='retired' is a rollback. The audit of who did what and when comes from pg_audit without writing a single additional line of code.

Generating DPO and KTO datasets from real traffic

This is where the elegance of the schema pays off. The dataset is not a static file: it is a materialised view built with SQL over obs.inference_log.

KTO dataset from 👍/👎

KTO is the method that best fits the signal any decent chat product captures. The query:

CREATE MATERIALIZED VIEW train.kto_v3_candidate AS
SELECT
 messages AS prompt,
 completion AS response,
 CASE WHEN fb_explicit > 0 THEN true ELSE false END AS label,
 adapter_id, created_at
FROM obs.inference_log
WHERE fb_explicit != 0
 AND created_at > now() - interval '60 days'
 AND pii_flags = 0
 AND tenant_id IN (SELECT id FROM tenant WHERE consent_training);

Simple. Every row with explicit feedback becomes an example (prompt, response, desirable_yes_no). KTO trains directly on this signal, with no need to build pairs.

DPO dataset from “regenerate”

The trick that is worth this article on its own. When the user presses “regenerate response”, they are giving an extraordinarily strong signal: the first response was no good to them. If the second is neither regenerated nor rated negatively, we assume it was. That is a DPO pair without a single extra click in the UI:

CREATE MATERIALIZED VIEW train.dpo_v3_candidate AS
SELECT
 rej.messages AS prompt,
 cho.completion AS chosen,
 rej.completion AS rejected,
 rej.adapter_id
FROM obs.inference_log rej
JOIN obs.inference_log cho ON cho.parent_id = rej.id
WHERE rej.fb_regen = true
 AND cho.fb_explicit >= 0
 -- length bias mitigation in classic DPO
 AND cho.tokens_out BETWEEN rej.tokens_out * 0.5
 AND rej.tokens_out * 2.5;

The clause on lengths is the cheap cure for the length bias documented in DPO. Without it, the model learns that “longer = better” because the responses the user accepts tend to be slightly longer. With SimPO or ORPO this filter is optional; with classic DPO it is necessary.

Semantic deduplication with pgvector

Before training, dedup. Two almost identical prompts in the dataset are noise that biases the model:

WITH ranked AS (
 SELECT id, embedding,
 row_number() OVER (
 PARTITION BY hashtext(messages::text)
 ORDER BY fb_explicit DESC, created_at DESC
 ) AS rn
 FROM obs.inference_log
 WHERE created_at > now() - interval '60 days'
)
DELETE FROM train.kto_v3_candidate kto
USING ranked r
WHERE r.rn > 1 AND kto.id = r.id;

And for the semantic duplicates (paraphrases) we use pgvector 0.8 directly with an iterative index scan:

-- Find near-duplicates of any given example
SELECT id, messages, embedding <=> $1 AS dist
FROM obs.inference_log
WHERE created_at > now() - interval '60 days'
 AND embedding <=> $1 < 0.05
ORDER BY embedding <=> $1
LIMIT 50;

The iterative scan is a key improvement in pgvector 0.8: before, the HNSW index could return fewer results than requested when there were additional filters (WHERE); now it iterates until it meets the limit. Without that improvement, curation queries over datasets of millions of rows were unworkable without a brutal pre-filter.

Eval gates: three stages, all SQL

The most common mistake when implementing continuous fine-tuning is skipping or watering down the eval gates. That turns the cycle into a roulette wheel. The pattern that works in 2026 is three stages, each with a different latency/coverage trade-off:

And this is where Postgres shines again: the promotion gate is expressed as an SQL predicate. Nothing more:

CREATE TABLE train.eval_result (
 adapter_id TEXT REFERENCES serve.adapter(id),
 suite_id TEXT, -- 'safety-es', 'support-helpfulness'
 metric TEXT,
 score NUMERIC,
 judge_model TEXT,
 judged_at TIMESTAMPTZ DEFAULT now(),
 PRIMARY KEY (adapter_id, suite_id, metric)
);

CREATE OR REPLACE FUNCTION serve.can_promote(candidate TEXT, current TEXT)
RETURNS BOOLEAN AS $$
 SELECT NOT EXISTS (
 SELECT 1
 FROM train.eval_result c
 JOIN train.eval_result p USING (suite_id, metric)
 WHERE c.adapter_id = candidate
 AND p.adapter_id = current
 AND suite_id IN ('safety-es','support-helpfulness','refusal-rate')
 AND c.score < p.score * 0.98 -- 2 % tolerance
 );
$$ LANGUAGE sql STABLE;

An SQL function as the gate. Usable from CI with psql -c "SELECT serve.can_promote('v4.2','v4.1')" and an exit code of 0/1. No orchestrator needed, no specific UI needed. The audit stays in the Postgres log.

vLLM multi-LoRA: the deploy is an HTTP POST

Two years ago, deploying a new fine-tune meant rotating inference pods. Today it is an HTTP call. vLLM 0.7+ supports loading and unloading LoRA adapters hot, keeping several resident in VRAM and choosing the right one per request.

Server configuration:

vllm serve meta-llama/Llama-3.1-8B-Instruct \
 --enable-lora \
 --max-loras 4 \
 --max-lora-rank 64 \
 --env VLLM_ALLOW_RUNTIME_LORA_UPDATING=True

Deploying a new adapter:

curl -X POST http://localhost:8000/v1/load_lora_adapter \
 -H "Content-Type: application/json" \
 -d '{
 "lora_name": "support-es-v4.2",
 "lora_path": "/mnt/adapters/support-es-v4.2"
 }'

From that moment on, requests including "model": "support-es-v4.2" are served with that adapter applied on top of the base model. Switching between adapters has negligible latency (the most recent research on Activated LoRA takes this to levels where the switching cost is invisible).

This changes operations substantially. Deploying a new fine-tune stops being an infrastructure event and becomes a state change in Postgres. The router queries the serve.adapter table, sees that v4.2 is in canary with traffic_pct=5, and directs 5 % of requests to the new adapter. The exact route of that 5 % is decided with deterministic hashing of the user_id so that the same user always sees the same variant (sticky):

-- No assignment table, no additional state
-- the variant is computed in-place in SQL or in the router:
SELECT
 CASE WHEN hashtext($user_id || $experiment) % 100
 < (SELECT traffic_pct FROM serve.adapter WHERE id = $candidate)
 THEN $candidate ELSE $current END AS adapter_id;

A/B with real traffic: measure or live deceived

The eval gates measure against fixed benchmarks. That is necessary but insufficient. Reality is only measured with real traffic. Once the adapter is in canary, what matters are the online metrics measured over obs.inference_log for each variant:

SELECT
 adapter_id,
 COUNT(*) AS n,
 AVG(fb_explicit) AS mean_score,
 STDDEV(fb_explicit) / SQRT(COUNT(*)) AS sem,
 AVG(ttft_ms) AS ttft_avg,
 percentile_cont(0.5) WITHIN GROUP
 (ORDER BY ttft_ms) AS ttft_p50,
 percentile_cont(0.95) WITHIN GROUP
 (ORDER BY ttft_ms) AS ttft_p95,
 AVG(CASE WHEN fb_regen THEN 1 ELSE 0 END) AS regen_rate
FROM obs.inference_log
WHERE experiment = $1
 AND created_at > now() - interval '7 days'
GROUP BY adapter_id;

What to look at: explicit feedback, latency (TTFT, p50, p95), regeneration rate. An adapter that raises the mean feedback but also raises the regeneration rate is suspicious, probably it is responding in a flashier but less useful way. An adapter that lowers latency but lowers feedback may be worth studying: it may be being more concise than it should.

Promotion to prod happens when, after 24-72 hours in canary, the candidate adapter beats the current one on at least one key metric without degrading the others. Again: it is an UPDATE in Postgres.

Applied to typical on-premise hardware

Let us come down to two representative configurations, one for iteration and one for production.

Case 1 — RTX 4090 (24 GB) for development iteration

An RTX 4090 with 4-bit QLoRA can train adapters on an 8B model without trouble. The VRAM budget combines four components; the KV cache during the intermediate evaluations is not negligible and deserves an explicit margin:

8B base model in 4-bit: ~5 GB
Activations + gradients: ~8 GB (depends on batch and context)
Optimizer state (LoRA r=16): ~0.5 GB
KV cache during eval: ~2 GB
Safety margin: ~8 GB

Typical times (an estimate based on community benchmarks; measure in the lab before promising anything):

Dataset	Technique	Adapter rank	Approximate time
1,000 SFT examples	LoRA r=16	16	20-40 min
5,000 SFT examples	LoRA r=32	32	2-4 h
2,000 DPO pairs	LoRA r=16	16	1-2 h
5,000 KTO examples	LoRA r=32	32	3-5 h

This puts the iteration cycle, a change in the dataset, retrain, eval, look at the number, in the range of a working day. Enough to validate a hypothesis before moving anything to the production cluster.

Case 2 — 4×H100 SXM cluster (320 GB, NVLink) for production

With a cluster of this order the whole picture changes. You can:

Train LoRA on a 70B in BF16 with no quantisation with tensor parallel = 4.
Do full DPO with a resident reference model when the reference is quantised to FP8, or move to SimPO / ORPO, which remove that intermediate model and simplify VRAM planning (see the table of techniques above).
Support multi-tenant fine-tuning: several customer adapters training in parallel in separate logical pipelines, each isolated in a different Postgres partition with its own ACLs.
Serve multi-LoRA with --max-loras 8 on top of the base model without concurrency lowering throughput perceptibly.

The practical budgeting rule: over a 12-month horizon, a team with this cluster can run ~150-200 continuous fine-tuning cycles (training + eval + canary + promotion or discard) if the discipline around the dataset and the eval gates is strict. If it is not, it will run twice as many but with half the usefulness.

Position within the architecture: what this article covers and what it does not

To place the scope: the cycle drawn at the start has seven boxes, all of them covered here in their mechanics. Three cross-cutting layers are deliberately left out, and they are the ones that end up separating a pipeline that works technically from one that survives an audit:

Cryptographic provenance and traceability. We have mentioned dataset_snapshot and pg_audit, but the complete mechanics, the set_hash over the examples, the integration with the EU AI Act, the frozen query_sql as proof of what trained the model, is enough for a whole analysis.
Judge calibration. We have assumed LLM-as-judge works. It needs calibrating against a human rubric on at least 100 cases per critical suite before trusting it. Without that calibration, the eval gates are theatre.
The forgetting problem. What happens if a user exercises their GDPR right to be forgotten and their interactions were part of the dataset of an adapter already in production? There is no clean solution. There are options, incremental retrain, sample-level machine unlearning, negative LoRA, and it is worth knowing them before a customer asks.

What we have not covered (upcoming articles)

Cryptographic provenance on Postgres: how set_hash and a frozen query_sql compose an auditable chain of custody under the EU AI Act.
Honest judge calibration: why score > 0.85 means nothing without a human baseline, and how to build that baseline without it costing a month of work.
The forgetting problem in adapters: sample-level machine unlearning, incremental retrain and other techniques for answering GDPR without throwing the adapter away.
Online DPO and on-policy continual learning: the state of 2026 research (Fast-Slow Chasing, RLOO, iterative on-policy) and why it is not production yet.

References

Hu et al., LoRA: Low-Rank Adaptation of Large Language Models (ICLR 2022).
Dettmers et al., QLoRA: Efficient Finetuning of Quantized LLMs (NeurIPS 2023).
Rafailov et al., Direct Preference Optimization: Your Language Model is Secretly a Reward Model (NeurIPS 2023).
Meng, Xia, Chen, SimPO: Simple Preference Optimization with a Reference-Free Reward (NeurIPS 2024).
Hong et al., ORPO: Monolithic Preference Optimization without Reference Model (2024).
Ethayarajh et al., KTO: Model Alignment as Prospect Theoretic Optimization (2024).
Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP 2023) — original vLLM.
Official vLLM Multi-LoRA documentation: https://docs.vllm.ai/en/stable/features/lora/.
Official pgvector 0.8 documentation: https://github.com/pgvector/pgvector.
TRL (HuggingFace) docs: https://huggingface.co/docs/trl.
EU AI Act, consolidated text and application timeline: https://artificialintelligenceact.eu/.

Postgresql on lo0 — Tech Blog

LiteLLM on day 2: one worker per pod, six thousand seconds of timeout and the other five things to change before opening the traffic

TL;DR

You are here: the gateway layer, the day after

The analogy: the switchboard and its notebook

The process model: one worker per pod

Postgres: the arithmetic that blows up when you scale

Redis: when it stops being optional

Background jobs register per worker

Probes: three endpoints and only two are usable

Retries, cooldowns and fallbacks: the real numbers

The six thousand second timeout

Overhead: the project’s two figures

The latency gap the metrics cannot see

The failure mode to put in the runbook

Upgrading without cutting traffic

Go-live checklist

Traps and things that are not what they look like

Closing

See also

Sources

Debezium and CDC: the notary who hears changes before anyone asks

TL;DR

The master analogy: the land registry notary

1. The problem CDC solves

2. What the PostgreSQL WAL is

The operations journal

Physical vs logical replication

The replication slot: the notary’s bookmark

The pgoutput plugin

3. Debezium architecture

The connector as a Kafka Connect plugin

Structure of a Debezium event

Initial snapshot

SMT transformations (Single Message Transforms)

4. Debezium vs the Outbox pattern

5. The maths

Throughput

End-to-end latency

WAL disk blowup risk

6. Use cases in LLMOps / RAG

RAG synchronisation with real deletion

Event sourcing for fine-tuning datasets

Immutable audit log

7. Architecture diagram

8. Minimum configuration

PostgreSQL: enable logical replication

Debezium connector (Kafka Connect REST API)

Minimal consumer in Python

9. On-premise deployment

What we have not covered

See also

References

Function calling and tool-augmented retrieval: the detective who knows which file to ask for

TL;DR

The analogy: the detective who knows which file to ask for

What function calling is: the anatomy of a tool call

Defining tools in the system prompt

The life cycle of a tool call

Support in OSS models

The ReAct pattern: Reasoning + Acting

A full ReAct trace: a real production query

The three sources and when to use each one

Tool routing: how the LLM picks the right tool

Descriptions that work versus those that do not

Parallel tool calling

Tool selection accuracy: the eval metric

Error handling in tool calls

SQL injection via prompt

Rate limits, timeouts and tool errors

Diagram of the ReAct loop with the three sources

Latency maths of the ReAct pipeline

Reference values: Llama-3.1-70B on 4×H100 SXM (320 GB, NVLink)

Total latency by number of iterations

Comparison with a naive RAG

On-premise hardware for ReAct agents

What we have not covered

See also

References

PostgreSQL + Qdrant in RAG ingestion: the postman who keeps two worlds in sync