GitOps for the inference stack with Flux: operating the assistant as code

Contents

This post is part of the operational series on how to squeeze a generic on-premise 4×H100 SXM 80 GB LLM cluster. The sibling pieces: the document ingestion pipeline for RAG that fills the vector store we deploy here, the embeddings and reranker service with TEI that is one of the services GitOps manages, and the secrets hardening of the sovereign stack, which goes deep into the secrets problem we only state here. The complete end-to-end assistant (LibreChat + LiteLLM + RAG) that orchestrates all of this will get its own post.

TL;DR

An LLM assistant in production is a distributed system, not a binary. At a minimum it has an inference engine (vLLM or similar), an L7 gateway that routes per model and applies rate limiting, a chat front end, a vector store for RAG, an embeddings/reranker service, and an observability stack. Six substantial services, spread across three or four namespaces, with start-up dependencies (the front end is useless if the gateway is not there, the gateway is useless if the engine has not loaded the model, RAG does not answer if the vector store is empty). Operating it by hand, a kubectl apply here and a helm upgrade there, produces a cluster whose real state nobody can reconstruct: there is no record of what was applied, or when, or why. GitOps solves this with four principles (OpenGitOps): the desired state is declarative, it is versioned and immutable in git, it is applied automatically by an agent (nobody SSHes in to deploy) and it is continuously reconciled by comparing the actual state against the declared one. Flux is that agent: six controllers (source, kustomize, helm, notification, image-reflector, image-automation) that clone the repo, render Kustomize/Helm, apply to the cluster, correct drift and, optionally, write back to git to bump the tag of a new image. The unit of work is the Kustomization or the HelmRelease, with interval (how often it reconciles), prune (deletes what is no longer in git), dependsOn (orders the start-up) and health checks. The interval sets the MTTR of drift: with interval=1m, a manual edit to the cluster is reverted in ≤ 1 min on average. Secrets cannot sit in the clear in git, a chicken-and-egg problem, and it is solved with SOPS/age, sealed-secrets or External Secrets + Vault. Rollback is a git revert. GitOps is not free: the learning curve is real, and debugging why a Kustomization is not reconciling is a new skill.

The analogy: the master plan and the foreman who does not negotiate

Picture a large building site with a signed and filed master plan, and a foreman who has a single order: the site must be identical to the plan, at all times. The foreman does not improvise. Every so often he walks the site with the plan in hand and compares: if a wall is where the plan says, he leaves it; if someone moved a partition overnight without updating the plan, he puts it back; if the plan says there is a column and it does not exist, he builds it; if a column exists but no longer appears on the plan, he knocks it down. The plan is the single source of truth: to change the site you do not touch the site, you change the plan, and the foreman takes care of the rest on his next round.

That is exactly the mechanics of GitOps. Git is the master plan: declarative (it describes the end state, not the steps), versioned (every change is signed into the history, with author and reason), immutable (a commit is not rewritten). Flux is the foreman: every interval it walks the cluster, compares against git and converges. And here is the lesson that separates GitOps from “keeping the YAML in a repo”: the foreman undoes manual changes. If an operator comes in with kubectl edit and raises the inference engine replicas from 2 to 4 at three in the morning to put out a fire, on the next round Flux takes it back to 2, because the plan says 2. This enrages anyone coming from the imperative world, and it is precisely the point: if you want 4 replicas permanently, you update the plan. The cluster stops being a system with its own opaque memory and becomes a reproducible projection of git. Delete the entire cluster, point a fresh Flux at the same repo, and the site is rebuilt identically.

The analogy also marks the boundary: the plan describes the building, not who holds the keys to the materials store. Secrets (passwords, tokens, keys) cannot go into the public plan. That is the chicken-and-egg problem we cover further down and that the hardening piece develops in full.

The problem: an assistant is six services, not one

Before Flux, it is worth fixing the size of the problem. A minimally serious sovereign LLM assistant, deployed on the generic reference cluster, has this service topology:

Service	Function	Typical namespace	State
Inference engine (vLLM)	Serves the tokens of the general LLM and the code one	`llm-serving`	Stateless (model from object store)
Embeddings + reranker (TEI)	Vectorises queries and reorders candidates for RAG	`llm-serving`	Stateless
Vector store (Qdrant)	Stores and searches document embeddings	`data`	Stateful (PVC)
L7 gateway (Envoy AI Gateway / LiteLLM)	Per-model routing, rate limiting, auth	`gateway`	Semi-stateless
Chat front end	Assistant UI	`apps`	Stateless
Observability (Langfuse, OTel Collector, dashboards)	LLM-aware traces, metrics and logs	`observability`	Stateful (Langfuse Postgres)

Six services, four or five namespaces, two components with persistent state, and a real dependency graph: the front end calls the gateway, the gateway calls the engine and the embeddings service, RAG depends on the vector store being full, and everything emits traces to observability. Deploying this by hand means remembering the order, the values of each helm install, the ConfigMaps, the Secrets, the nodeSelectors that pin the GPU pods to the right nodes. Do it twice (a staging environment and a production one) and the two diverge within days. That is exactly what GitOps eliminates.

The six Flux controllers

Flux is not a monolithic binary but a set of cooperating controllers, the GitOps Toolkit. A default installation brings four; the two image automation ones are added with --components-extra (Flux installation):

source-controller: clones and keeps the sources up to date, whether git repos (GitRepository), Helm repos (HelmRepository), buckets or OCI artifacts. It is the one “holding the plan”: it exposes the repo contents as an internal artifact that the others consume.
kustomize-controller: takes a source artifact, renders Kustomize (bases + overlays) and applies the result to the cluster. It is responsible for prune, dependsOn and the health checks of the Kustomizations.
helm-controller: reconciles HelmRelease objects, installing and updating Helm charts declaratively, without anybody running helm from a terminal (helm-controller).
notification-controller: the bridge to the outside world in both directions. It receives webhooks (to reconcile instantly on every push, instead of waiting for the interval) and emits events and alerts to Slack, an internal chat or an incident system.
image-reflector-controller: scans container registries and stores the tags it finds in an internal database. It is the “eyes” of image automation.
image-automation-controller: uses what the eyes see to write back to git, committing the new image tag into the manifests when a version appears that satisfies the policy.

A cluster without image automation does not need the last two. But for an inference stack where the model server is updated fairly often, they are what automates version promotion without touching anything by hand.

The diagram has three paths. The grey one is the main flow: git → source-controller → kustomize/helm-controller → cluster. The dashed red one is the reconciliation loop that runs every interval: it compares actual state against desired, fixes drift and prunes what is left over. The purple one is image automation: the reflector sees a new tag in the registry, the policy decides whether it qualifies, and the automation-controller writes back to git. That last path is the one that closes the circle and turns git into a system that updates itself.

Repo structure: clusters, infrastructure, apps

The most widespread convention separates three levels of responsibility. It is not the only one, but it ages well:

gitops-repo/
├── clusters/
│   └── prod/
│       ├── infrastructure.yaml   # Kustomization → ./infrastructure
│       └── apps.yaml             # Kustomization → ./apps (dependsOn infrastructure)
├── infrastructure/
│   ├── controllers/              # ingress, cert-manager, GPU operator, KEDA...
│   └── configs/                  # ClusterIssuer, RuntimeClass, StorageClass...
└── apps/
    ├── base/                     # manifests common to all environments
    │   ├── llm-engine/           # vLLM HelmRelease
    │   ├── gateway/              # L7 gateway HelmRelease
    │   ├── chat-front/           # Deployment + Service + Ingress
    │   ├── vector-store/         # Qdrant HelmRelease (+ PVC)
    │   └── observability/        # Langfuse + OTel HelmRelease
    ├── staging/                  # overlay: low replicas, small model
    └── prod/                     # overlay: high replicas, large model, MIG

The key piece is Kustomize’s base / overlay separation. The base/ describes the service once; each overlay (staging/, prod/) applies a patch with the environment’s differences: number of replicas, model size, MIG profile, nodeSelector, --gpu-memory-utilization. That avoids duplicating complete manifests per environment. The clusters/ folder is the entry point Flux reconciles first: it holds the root Kustomizations that point to infrastructure/ and apps/, with a dependsOn that guarantees the infrastructure (CRDs, operators, storage classes) is ready before the applications.

How many manifests: the calculation that justifies overlays

This is where the numbers make the argument. Without overlays, each service needs a complete set of manifests per environment. With $N$ services and $M$ environments, the cost in files to maintain is:

$$ \text{files}_{\text{naïve}} = N \times M \times k $$

where $k$ is the average number of manifests per service (Deployment/HelmRelease + Service + ConfigMap + PVC + Ingress ≈ 5). For our assistant, $N=6$ services and $M=3$ environments (dev, staging, prod), with $k=5$:

$$ 6 \times 3 \times 5 = 90 \text{ complete files, each maintained separately.} $$

With the base/overlay structure, the base is written once and each overlay contains only the patch with the differences (typically 1 short file per service per environment):

$$ \text{files}_{\text{overlay}} = \underbrace{N \times k}_{\text{base}} + \underbrace{N \times M}_{\text{patches}} = 6 \times 5 + 6 \times 3 = 30 + 18 = 48 $$

It is not just that there are fewer files (48 against 90): it is that the bulk of a change happens in a single place. Changing the engine’s memory limit for every environment is one edit in base/, not three synchronised edits. The saving factor grows with $M$: for 5 environments, the naïve version is 150 files and the overlay one is 60. Duplication is the enemy of auditability, and overlays attack it at the root.

The reconciliation loop: desired vs actual

The heart of Flux is a control loop that never ends: every interval it reads the desired state (git), reads the actual state (cluster), computes the difference and applies it. It is the same principle as a thermostat: it reads the target temperature (git), reads the actual temperature (cluster) and switches the boiler on or off until they match. It does not “deploy once”; it converges forever.

A Kustomization manifest for the applications, with the pieces that matter (Kustomization reference):

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: apps
  namespace: flux-system
spec:
  interval: 1m              # reconciles every minute: sets the drift MTTR
  path: ./apps/prod         # production overlay
  prune: true               # deletes from the cluster whatever is removed from git
  sourceRef:
    kind: GitRepository
    name: gitops-repo
  dependsOn:
    - name: infrastructure  # does not apply apps until infra is Ready
  wait: true                # waits for the resources to be healthy
  timeout: 5m               # fails the reconciliation if it does not converge in 5 min
  healthChecks:
    - apiVersion: apps/v1
      kind: Deployment
      name: gateway
      namespace: gateway

Four fields do the heavy lifting:

interval defines how often the foreman’s round runs. It is the direct determinant of the drift MTTR (next section).
prune: true switches on garbage collection: objects that were applied before but are no longer in the current git revision are deleted from the cluster automatically. Without prune, a service withdrawn from the repo keeps running, and the plan and the site diverge silently.
dependsOn orders the graph: Flux does not apply the manifests of a Kustomization until every referenced one is in state Ready: True. It is the mechanism that guarantees data → gateway → front end.
wait + healthChecks: with wait: true, Flux monitors every applied resource and waits for them to be ready before marking the reconciliation as successful; healthChecks lets you tune exactly which resources to watch. That is what makes a dependsOn mean something: the dependency is not satisfied until it is healthy, not merely applied.

MTTR and drift detection as a function of the interval

The interval is the only parameter the operator chooses to govern the speed of the loop, and it translates directly into operational metrics. If a deviation (drift) happens at a random instant within the reconciliation period $T$, the waiting time until Flux detects it is uniformly distributed over $[0, T]$, with mean:

$$ \mathbb{E}[t_{\text{detection}}] = \frac{T}{2} $$

To that you add the correction time $t_c$ (render, diff, apply), normally a few seconds. The drift MTTR comes out as:

$$ \text{MTTR}_{\text{drift}} = \frac{T}{2} + t_c $$

With interval=1m and $t_c \approx 10\text{s}$, drift is corrected in $\frac{60}{2} + 10 = 40$ s on average, with a worst case of $60 + 10 = 70$ s. With interval=10m, the mean rises to $5\text{min}\;10\text{s}$ and the worst case to more than 10 min: a window in which a manual change stays active. The temptation is to set interval=10s and forget about it, but there is a cost: every reconciliation consumes CPU and, above all, makes requests to the cluster API and to the registry. With dozens of Kustomizations reconciling every 10 s, the kube-apiserver and the image-reflector start to feel the pressure. The practical rule: a short interval (1m) for critical applications whose drift hurts, a long interval (10–30m) for stable infrastructure that almost never changes, and notification-controller webhooks for instant reconciliation on every push, so that the interval governs only drift and not deployment latency.

Helm as code: the inference engine HelmRelease

For the engine, the gateway and the front end, the usual approach is to package them as Helm charts and declare them with HelmRelease. The helm-controller reconciles them without anyone ever running helm:

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: llm-engine
  namespace: llm-serving
spec:
  interval: 10m
  chart:
    spec:
      chart: vllm
      sourceRef:
        kind: HelmRepository
        name: vllm-charts
  values:
    image:
      tag: "0.8.4"          # managed by image automation (see below)
    replicaCount: 2
    nodeSelector:
      nvidia.com/gpu.product: H100-SXM
    resources:
      limits:
        nvidia.com/gpu: 1
    extraArgs:
      - "--gpu-memory-utilization=0.90"
      - "--tensor-parallel-size=1"

Everything that in the imperative world would be a vllm serve flag, --gpu-memory-utilization or --tensor-parallel-size, now lives as versioned data. Changing the fraction of VRAM the engine reserves is a commit, reviewable in a PR, with history. The relationship with sharing a GPU is direct: the MIG profile and the --gpu-memory-utilization are deployment parameters, and here they are code.

Image automation: bumping the tag without touching anything by hand

The inference engine is updated fairly often (vLLM patches, security fixes). Without automation, every new version requires somebody to edit the tag in the HelmRelease. Flux image automation does it on its own, with three objects (automate image updates):

apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImageRepository
metadata:
  name: vllm
  namespace: flux-system
spec:
  image: registry.example.local/inference/vllm
  interval: 5m              # scans the registry tags every 5 min
---
apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImagePolicy
metadata:
  name: vllm
  namespace: flux-system
spec:
  imageRepositoryRef:
    name: vllm
  policy:
    semver:
      range: ">=0.8.0 <0.9.0"   # only patches and minors within 0.8.x–0.8.x
---
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageUpdateAutomation
metadata:
  name: vllm
  namespace: flux-system
spec:
  interval: 5m
  sourceRef:
    kind: GitRepository
    name: gitops-repo
  git:
    commit:
      author:
        name: fluxbot
        email: fluxbot@example.local
      messageTemplate: "auto: bump vllm to {{ .NewTag }}"

The division of roles: ImageRepository scans all the tags of the image repository and stores them in the reflector’s internal database. ImagePolicy reads those tags and picks the “latest” according to the policy; the policy field is mandatory and defines how the selection is made (ImagePolicy). ImageUpdateAutomation takes the chosen tag, edits the manifest in the repo (wherever there is a # {"$imagepolicy": "flux-system:vllm"} marker) and commits (ImageUpdateAutomation).

The choice of policy matters:

semver: interprets tags as semantic versions and picks the highest one that satisfies the range (range: ">=0.8.0 <0.9.0"). By default it excludes prereleases (0.8.0-rc.1 does not qualify) unless they are asked for explicitly. It is the right option for production: you stay inside a tested version range and do not jump to a new major by accident.
regex / alphabetical / numerical: for tag schemes that are not pure semver, for example main-<sha>-<timestamp> from an internal pipeline. More flexible, but it forces you to trust that tag order reflects version order, which is fragile.

A safety guideline: instead of having ImageUpdateAutomation push directly to main, configure it to write to a branch and open a PR. That way the version bump goes through human review or a validation pipeline before reaching the cluster. Automation proposes; the human (or an automatic gate) disposes. And for production, the version bump is not the deployment: it is the trigger for the progressive rollout described in canary, blue-green and shadow.

Secrets in GitOps: the chicken-and-egg problem

GitOps demands that all the desired state is in git. But secrets, the Langfuse Postgres password, the registry token, the gateway API keys, cannot go into a repo in the clear, not even a private one: git history is immutable, and a secret committed once stays there forever. This is the fundamental tension: GitOps wants everything in git; security forbids secrets in git. There are three families of solution, all with the same underlying idea, that only the encrypted secret goes into git and the cluster holds the key to decrypt it:

SOPS + age/KMS: you encrypt the sensitive values of a manifest with SOPS (it leaves the keys readable and encrypts only the values). The encrypted YAML goes to git; the kustomize-controller carries an age key or reaches a KMS to decrypt it at apply time. Simple, with no extra components in the cluster.
sealed-secrets: a controller in the cluster holds a private key. You encrypt the secret against its public key (with kubeseal), the encrypted SealedSecret goes to git, and the controller decrypts it into a normal Secret inside the cluster. The private key never leaves the cluster.
External Secrets + Vault: not even the encrypted secret goes into git, only a reference (ExternalSecret) saying “the value of this key is in Vault, at this path”. The External Secrets operator resolves it at apply time. It is the cleanest pattern for many secrets and frequent rotation, at the cost of operating a Vault.

The choice depends on the volume of secrets and on whether you already have a central manager. For a small stack, SOPS+age is enough and has no dependencies. The sibling hardening piece goes into the detail of each option, key rotation and the threat model.

Promotion and rollback: git revert is the panic button

The most elegant consequence of having the cluster as a projection of git is that rollback is a git revert. If a deployment breaks production, the new engine gives worse latency or the gateway starts returning 5xx, there is no need to remember which version was there before or to rebuild the state by hand. You revert the commit that introduced the change, and on the next reconciliation (≤ interval, or instantly with a webhook) Flux returns the cluster to the previous state. The git history is, literally, the deployment history: every commit is a restore point with author, date and reason.

This fits with the promotion strategies of the canary, blue-green and shadow post: Flux manages what is declared, and the progressive rollout tool manages how traffic moves between the old version and the new one. The git revert is the last-resort rollback (it returns everything to a known state); the canary is the mechanism that avoids needing it. For services where the version change is delicate, the right approach is for image automation to do the bump on a branch, for the canary to validate the regression gates, and only then to promote the commit to main.

The seams: GitOps is not free

It would be dishonest to sell GitOps as magic. It has real costs that any team pays:

Learning curve. The declarative model is a change of mindset. The operator who has spent years fixing incidents with kubectl edit has to unlearn the reflex: now the cluster reverts their changes and that, at first, feels like Flux “fighting them”. Understanding that the change is made in git, not in the cluster, takes weeks of discomfort.

Debugging the reconciler. When a Kustomization does not apply, the error is not in the pod; it is in the reconciliation chain. Did the source-controller clone the right revision? Did the kustomize-controller render the overlay properly? Did a dependsOn health check fail? Did the timeout fire before the large model finished loading? Diagnosing this needs flux get, flux logs, reading the status.conditions of the Flux objects and understanding which link got stuck. It is a new skill, different from debugging plain Kubernetes.

Timeouts and slow loads. The timeout of a Kustomization with wait: true has to be longer than the start-up time of the slowest service. An inference engine that takes 4 minutes to load a large model from the object store will fail a Kustomization with timeout: 2m, even though everything is fine. Calibrating timeouts per service is fine-grained work.

The drift you did want. Sometimes the operator needs an urgent temporary change and Flux reverts it. The correct answer, suspending reconciliation of that Kustomization with flux suspend, making the change, then reflecting it in git and resuming, is discipline that has to be built. Without that discipline, people end up fighting the foreman at 3 in the morning.

None of these costs cancels the benefit. But a team adopting GitOps expecting “everything to be easier from day one” gets frustrated. It is easier from month two onwards, once auditability, reproducibility and trivial rollback have paid off the curve.

Applied to the generic 4×H100 cluster

On the reference cluster (4×H100 SXM 80 GB per GPU node, NVLink, plus CPU and control nodes), the assistant stack is managed entirely by GitOps. Hardware decisions become data in the overlays.

nodeSelectors and MIG profiles as code. The prod overlay pins each service to the right node and declares the GPU profile. The general engine uses a whole GPU; embeddings and the small LLM fit in MIG slices, exactly the split from the sharing a GPU post, but now versioned:

# apps/prod/llm-engine-patch.yaml
spec:
  values:
    nodeSelector:
      nvidia.com/gpu.product: H100-SXM
    extraArgs:
      - "--gpu-memory-utilization=0.92"
      - "--tensor-parallel-size=1"
---
# apps/prod/embeddings-patch.yaml — on a MIG slice
spec:
  values:
    nodeSelector:
      nvidia.com/mig.config: "3g.40gb"
    resources:
      limits:
        nvidia.com/mig-3g.40gb: 1

The MIG profile (3g.40gb), the VRAM fraction (0.92), the tensor-parallel setting: all of it is text in a PR. Changing the GPU split between services is a reviewable commit, not a hand-run nvidia-smi mig session that nobody records.

Start-up order with dependsOn. The assistant’s dependency graph is encoded with chained dependsOn. The right order is data → embeddings/engine → gateway → front end:

# apps/prod/gateway.yaml
spec:
  dependsOn:
    - name: llm-engine        # the gateway is useless without the engine
    - name: embeddings
  # ...
---
# apps/prod/chat-front.yaml
spec:
  dependsOn:
    - name: gateway           # the front end is useless without the gateway

With wait: true on each Kustomization, Flux does not mark llm-engine as Ready until the engine pod passes its health check, which in the case of vLLM means model loaded and endpoint responding, not just pod started. Only then does it begin applying the gateway. This avoids the classic cascade of failures of manual deployment: bring up the front end first, see a 502 because the gateway is not there, bring up the gateway, see a 503 because the engine is still loading the model. With dependsOn + wait, the order is guaranteed by the reconciler, not by the operator’s memory.

The result: deleting the assistant’s entire namespace and letting Flux rebuild it from git produces exactly the same stack, in the same order, with the same GPU configuration. The 4×H100 cluster stops having a configuration known only to whoever built it, and becomes a reproducible projection of a repo that anyone can audit.

Conclusion

GitOps is not a tool, it is an inversion of the direction of control: instead of pushing changes to the cluster, you declare the state in git and let an agent pull the cluster towards it. For an LLM assistant (six services, several namespaces, start-up dependencies, delicate GPU configuration) that inversion turns a fragile and opaque system into a reproducible and auditable one. Flux is the foreman who keeps the site identical to the plan and undoes any change that has not gone through the plan first. The price is a real learning curve and a new debugging skill. The prize is that the cluster stops keeping secrets about itself: everything it is, is written down, signed and reproducible in git.

References

OpenGitOps — GitOps principles — opengitops.dev
Flux installation — fluxcd.io/flux/installation
Flux Kustomization — fluxcd.io/flux/components/kustomize/kustomizations
helm-controller — github.com/fluxcd/helm-controller
Flux Image Policies — fluxcd.io/flux/components/image/imagepolicies
Flux Image Update Automations — fluxcd.io/flux/components/image/imageupdateautomations
Automate image updates to Git — fluxcd.io/flux/guides/image-update
SOPS — github.com/getsops/sops