Mlflow on lo0 — Tech Blog

Model chain of trust (2/4): where the bytes live — registry, OCI artefacts and distribution to the GPU node

Sun, 26 Jul 2026 19:00:00 +0200

In the first article in the series we described the control plane: who declares that an inference endpoint exists, who reconciles it and what contract the clients speak. But an InferenceService is no more than a promise until somebody puts 140 GB of weights on the filesystem the container sees. This second article is about exactly that: where those bytes come from and by what route they reach the GPU’s memory.

It is a subject the blog has touched around the edges, in the cold start series, in data versioning with DVC and lakeFS, in the catalogue of OSS LLMOps tools, but never head on. It is time to, because almost every on-premise inference platform that gets audited shares the same original defect, and it is conceptual before it is about performance: they conflate three things that live in different places and fail in different ways.

TL;DR

Three planes, not one. Model registry (metadata, versions, lineage, promotion) ≠ artefact repository (the bytes, addressed by digest) ≠ deployment storage (what the pod mounts at /mnt/models). Mixing them is what breaks reproducibility.
The anti-pattern: a Hugging Face token in the pod and a download from the internet on every start-up. No immutability, no cache, no audit trail, no air-gap and cold starts measured in minutes.
The OCI artefact is today’s pragmatic answer: OCI Image Spec 1.1 (March 2024) brought artifactType, subject and the Referrers API; ORAS v1.3.0 (October 2025) added backup/restore for air-gap; ModelPack (CNCF sandbox since June 2025) standardises the mediaTypes for weights.
Kubernetes already knows how to mount OCI artefacts: the image volume source (KEP-4639) reached GA in v1.36 (22 April 2026). It is the most operationally relevant change of the year for model distribution.
The real bottleneck is almost never the GPU. 140 GB over a saturated 10 Gb/s link is about 112 s; over Gen4 NVMe about 20 s; over PCIe Gen5 about 2-3 s. Optimising the loader without fixing the network is optimising the wrong link.
A model registry (MLflow, Kubeflow Hub) does not store the bytes: it stores the record card and a pointer. If you were expecting it to solve your distribution, you bought the wrong tool.

The analogy

The simile is a serious historical archive, with three things a distracted visitor constantly confuses.

The catalogue: record cards with a shelfmark, author, date, provenance, conservation state and consultation restrictions. The card weighs grams and describes something that weighs kilos. The repository: the compact shelving in the basement where the physical boxes are, with access control, temperature control and an inventory that balances to the gram; it knows nothing about why a document matters, but it guarantees that box 4711 still contains exactly what it contained. And the reading room table: where the researcher has the document open in front of them. A temporary and expensive place, from which the document goes back to the repository when the session ends.

The model registry is the catalogue. The artefact repository is the repository. The GPU node with the model in HBM is the reading room table. Anyone who says “we have set up a model registry” and what they have set up is MLflow has set up the catalogue and left the repository unbuilt: the boxes are still in somebody’s car boot. Putting the model inside the container image is the equivalent of putting the document inside the table: every time the table changes, the document moves. And anyone who downloads from the internet on every start-up has no repository: they order the document by courier from another city every time a researcher sits down.

We will come back to the archive later, with the weeding that deletes boxes that are still referenced, the trolley that carries things between the basement and the reading room, and the certified reproduction sent to the isolated site.

Three planes that are not the same

Before discussing tools you have to fix the vocabulary, because a good part of the architecture discussions at this layer are really discussions about nomenclature.

Plane	What it stores	Unit	Question it answers	OSS examples
Model registry	Metadata, logical versions, lineage, promotion state, evaluation metrics	The model’s record card (KB)	Which version is approved for production and where did it come from?	MLflow, Kubeflow Hub, ClearML
Artefact repository	The bytes, addressed by digest, immutable	The blob (GB-TB)	What exactly are the bytes of that version?	Harbor, `distribution`, Zot, MinIO, lakeFS
Deployment storage	The copy the pod sees mounted	The volume (`/mnt/models`)	What is the inference process reading right now?	PVC, image volume, emptyDir + NVMe cache

The three layers have different life cycles. A record card lives forever (audit); a blob, for as long as some version references it (retention); a volume, for as long as the pod lives. Implemented as a single thing, it inherits the worst of the three: if the registry is also the deployment storage, deleting an old version takes down a pod in production; if the deployment storage is also the canonical reference, there is no way to answer what was served last Tuesday.

Hence the operational rule that gets written on the wall: the source of truth for the bytes is a digest, not a tag and not a path. Everything else (tags, aliases, PVC paths) is a mutable pointer that resolves to that digest at a given moment. Article 3/4 builds all of its provenance verification on that premise.

The anti-pattern we start from

This is, by a distance, the most widespread pattern in on-premise deployments that started as a pilot and stayed that way:

apiVersion: apps/v1
kind: Deployment
metadata:
 name: vllm-anti-patron
spec:
 template:
 spec:
 containers:
 - name: vllm
 image: vllm/vllm-openai:v0.11.0
 args: ["--model", "meta-llama/Llama-3.3-70B-Instruct"]
 env:
 - name: HF_TOKEN
 valueFrom:
 secretKeyRef:
 name: hf-token
 key: token
 resources:
 limits:
 nvidia.com/gpu: "4"

It works and it serves tokens. And it is unacceptable on a sovereign platform for five independent reasons, any one of which would be enough:

An external dependency on the critical start-up path. A Deployment that cannot recover from a failure without outbound internet access carries a third-party dependency that nobody has declared in the SLA.
Zero immutability. meta-llama/Llama-3.3-70B-Instruct is a mutable reference, and a repository’s default branch changes (tokenizer fixes, chat templates). Two replicas of the same Deployment started a week apart can serve different bytes, and nothing in the cluster says so.
Zero coordinated cache and zero audit trail. Every pod downloads its own copy: eight replicas of a 140 GB model is 1.1 TB of outbound traffic per rolling update. And no internal record remains of what was downloaded, with which digest or who authorised it; faced with an ENS or ISO 42001 auditor, the answer is “we got it off the internet”.
Incompatible with air-gap. The whole design collapses when the platform has to be replicated in an isolated environment, which is the defence and healthcare case dealt with in regulatory compliance infrastructure.
Cold start measured in minutes. Downloading 140 GB over a 1 Gb/s outbound link (125 MB/s effective, being generous) is about 19 minutes of transfer alone, before touching the disk or the GPU; with 10 Gb/s dedicated and saturated, about 112 seconds. In a load-driven scaling event, that is service latency.

The blog’s cold start series explores how to trim the disk-to-HBM leg. What this article adds is that if the first leg is the internet, the rest of the optimisation is irrelevant.

The model as an OCI artefact

The pragmatic answer is boring, and that is why it works: the model is stored in the same registry where the containers are already stored. There is already access control, cross-site replication, scanning, quotas, audit and a team that knows how to operate it.

What OCI 1.1 made possible

For years, putting things that were not container images into an OCI registry was a bodge. The OCI Image Spec 1.1 and Distribution Spec 1.1 specifications, published on 13 March 2024, turned it into a first-class use case with three pieces:

artifactType in the manifest: it explicitly declares “this is not a runnable image, it is an artefact of type X”. Previously this was encoded by abusing the config mediaType, which is why even today there are registries that show a model as a broken image.
subject: a manifest can declare that it refers to another manifest. That is what lets you hang a signature, an SBOM or an attestation off a model without modifying it and without changing its digest.
Referrers API (GET /v2/<repo>/referrers/<digest>): the reverse query, “give me everything that points at this digest”, with a tag-based fallback for registries that do not implement it yet.

Article 3/4 builds signing, provenance and AIBOM on top of these three pieces. Without them, verifying a model forces you to invent a convention of your own.

ORAS: the generic client

ORAS (OCI Registry As Storage) is the Swiss army knife for getting arbitrary artefacts in and out of an OCI registry. Version v1.3.0 was published on 6 October 2025, conformant with distribution-spec v1.1.1, and added three things relevant to models: backup/restore to a directory or tarball (key for air-gap), management of multi-platform indexes and --format for structured output.

One data point to calibrate maturity: ORAS entered the CNCF sandbox on 13 July 2021 and as of this article is still in sandbox. Five years at the entry level does not invalidate the tool, which is stable and the de facto reference, but it says something about the project’s governance that is worth weighing before turning it into a critical dependency with no plan B.

Pushing a model with ORAS, with explicit content types:

oras push registro.interno/modelos/llama-3.3-70b:2026.07.1 \
 --artifact-type application/vnd.cncf.model.manifest.v1+json \
 --annotation "org.opencontainers.image.created=2026-07-20T09:00:00Z" \
 --annotation "es.ejemplo.modelo.precision=bf16" \
 --annotation "es.ejemplo.modelo.origen=hf:meta-llama/Llama-3.3-70B-Instruct" \
 config.json:application/vnd.cncf.model.weight.config.v1.raw \
 tokenizer.json:application/vnd.cncf.model.weight.config.v1.raw \
 model-00001-of-00030.safetensors:application/vnd.cncf.model.weight.v1.raw \
 model-00002-of-00030.safetensors:application/vnd.cncf.model.weight.v1.raw

# Retrieve it, with a local blob cache so nothing already present is re-downloaded
export ORAS_CACHE=/var/cache/oras
oras pull registro.interno/modelos/llama-3.3-70b:2026.07.1 --output /srv/modelos/llama-70b
oras manifest fetch registro.interno/modelos/llama-3.3-70b:2026.07.1 --descriptor

ModelPack: the missing convention

ORAS lets you invent the mediaTypes, with the predictable result that every organisation invents its own and nothing interoperates. ModelPack is the attempt to standardise them, accepted into the CNCF sandbox in June 2025. On top of OCI Image Manifest 1.1 it defines:

artifactType: application/vnd.cncf.model.manifest.v1+json
config: application/vnd.cncf.model.config.v1+json
weight layers: application/vnd.cncf.model.weight.v1.raw (and the .tar, .tar+gzip, .tar+zstd variants)
weight configuration, documentation, code and dataset layers following the same pattern

Its reference CLI is modctl, which works with a declarative Modelfile:

NAME llama-3.3-70b-instruct
ARCH transformer
FAMILY llama
FORMAT safetensors
PARAMSIZE 70
PRECISION bf16
CONFIG config.json
CONFIG tokenizer.json
MODEL *.safetensors
DOC *.md

modctl build -t registro.interno/modelos/llama-3.3-70b:2026.07.1 -f Modelfile .
modctl push registro.interno/modelos/llama-3.3-70b:2026.07.1
modctl pull registro.interno/modelos/llama-3.3-70b:2026.07.1 \
 --extract-dir /srv/modelos/llama-70b --extract-from-remote

The alternative with more traction among users is KitOps (ModelKit), with Docker-like ergonomics and native integration with Hugging Face. As of this article there is no convergence: ModelPack aims at the enterprise case and at integration with runtimes and registries; KitOps and Docker Model Runner, at developer ergonomics. Picking one today is a bet; using artifactType and explicit annotations instead of home-grown conventions is not.

How a 140 GB model is sliced up

A large model is 20-40 safetensors shards plus half a dozen small configuration files, and the layering strategy matters:

One layer per shard, uncompressed. Weights in bf16 or fp8 are incompressible in practice: gzip over safetensors burns CPU to save 2-3 %, and forces decompression on the node, adding a full pass through CPU and disk on the critical path. The right type is .raw.
The small files in their own layer. config.json, tokenizer.json and the templates change far more often than the weights. If they go in the same layer as a 5 GB shard, every tokenizer fix invalidates 5 GB of cache on every node.
Layer size between 2 and 8 GB. Large enough not to pay per-blob overhead, small enough to parallelise and so that a network failure does not force you to retry 40 GB.

In practice three things hurt. The upload uses the distribution-spec’s chunked upload, with historically uneven interoperability between implementations, so pushing 140 GB is an operation measured in tens of minutes that demands a decent link and retries. Deduplication is by layer digest, so two versions that differ in one shard share the rest only if those blobs are byte-for-byte identical, and any repackaging that alters metadata breaks it. And the default values of reverse proxies (timeouts, maximum body size, upload session lifetime) are calibrated for 1 GB images, not for 8 GB blobs.

Harbor as the on-premise repository

Harbor is the reference OSS registry for on-premise: a CNCF graduated project since June 2020, with a healthy release cycle (the 2.15 series came out on 20 March 2026 and patch 2.15.2 on 2 July 2026) and support for the last three minors for about nine months each. What matters for models:

Cross-site replication. Push or pull rules, filtered by repository, tag and resource type, with manual, scheduled or event-driven triggering. For models, scheduled and within a window, with a label marking which versions get replicated (see traps).

Proxy cache. Here you have to be precise, because a lot of misinformation circulates: Harbor’s proxy cache works against OCI registries (Harbor, Docker Hub, a distribution registry, ECR, ACR, GCR, Quay and GHCR) and not against Hugging Face, which does not expose an OCI distribution API. If the idea was “I put Harbor in front of Hugging Face and that is it”, it does not work that way. The real options are (a) an ingestion task that pulls from Hugging Face in a staging environment and pushes to Harbor as an OCI artefact, the recommended pattern because it introduces an explicit control point, or (b) a P2P accelerator with native support for model repositories, such as Dragonfly. What the proxy cache does solve is mirroring the runtime base images (vLLM, TGI, TEI), which is no small thing: it creates a 7-day retention by default and the project is read-only.

Per-project quotas. In bytes, via the UI or the API:

curl -u admin:REDACTED -X PUT "https://registro.interno/api/v2.0/quotas/7" \
 -H "Content-Type: application/json" \
 -d '{"hard": {"storage": 21990232555520}}'

Those 21990232555520 bytes are 20 TiB, tight sizing, not generous, for a dozen models with three live versions each. Two warnings: the quota is checked when the manifest arrives, that is after uploading the blobs, so a push that exceeds it has already consumed disk and network; and the accounting does not always reflect deduplication of blobs shared between projects.

Immutable tags. Per-project rules that prevent overwriting a tag matching a pattern. For models it is not optional: a rule making everything under modelos/** with tag pattern 2* (date-based versions) immutable eliminates the entire class of “the tag changed underneath us” incidents.

Retention and GC. Per-project retention rules (last N versions, or those from the last N days) that mark artefacts for deletion, and a separate garbage collector that frees the unreferenced blobs. They are two different things, and deleting in the UI does not free disk: the weeding runs separately. Running GC with a concurrent push in flight is the classic recipe for deleting blobs that were about to be referenced.

Scanning. Trivy is integrated, with one caveat: scanning a weights artefact detects nothing relevant about the model. It is useful for the runtime images, which do execute code. Detecting dangerous formats, a PyTorch pickle with arbitrary code instead of safetensors, requires specific tools, and that is the subject of article 3/4.

Model registries proper

Now the catalogue, with the uncomfortable claim up front: a model registry does not solve distribution.

MLflow

MLflow (LF AI & Data) is the de facto catalogue standard, with a mature 3.x series; 3.13.0 was published in June 2026, focusing on RBAC, an administration UI and trace retention. Its data model: registered model (logical name) → model version (incrementing integer) → aliases and tags.

The important conceptual change: stages (Staging, Production, Archived) have been deprecated since MLflow 2.9.0 and are replaced by aliases and tags, because a fixed enumeration is too inflexible to express real MLOps flows. An alias is a named mutable pointer:

from mlflow import MlflowClient

client = MlflowClient()
client.set_registered_model_alias("llama-3.3-70b-soberano", "champion", 7)
client.set_model_version_tag("llama-3.3-70b-soberano", "7",
 "estado_validacion", "aprobado_ens_medio")
client.set_model_version_tag("llama-3.3-70b-soberano", "7",
 "oci_digest", "sha256:9f2c...")

Consumption happens through the alias URI: models:/llama-3.3-70b-soberano@champion.

What MLflow gives you: lineage back to the run that produced the model, evaluation metrics attached to the version, a shared promotion language and an API to hang approvals off. What it does not give you: it is not a CDN and it does not manage distribution to nodes, and its artifact store (S3/MinIO/NFS) provides neither immutability by digest, nor Referrers, nor cross-site replication. The healthy pattern is the one in the snippet: the catalogue stores the OCI digest as a tag on the version and points at the repository, instead of trying to be the repository.

Kubeflow Hub (formerly Model Registry)

The Kubeflow Model Registry was renamed Kubeflow Hub and unified two functions: the registry itself and a federated Model Catalog that discovers external models (YAML, Hugging Face), with lineage across data, code and models, metadata and state-based promotion.

The honest assessment: as of this article the component is still in the 0.3.x series and marked Alpha under Kubeflow’s versioning policy, with limited support. Interesting if you already run the full Kubeflow and want a single console; it is not yet the piece on which to build model governance for a sovereign platform in production. The rename in the middle of 0.3.x is, moreover, a sign that the API is still moving.

Comparison table by function

Function	OCI registry (Harbor + ORAS/ModelPack)	MLflow Registry	Kubeflow Hub	lakeFS / DVC
Stores the bytes	Yes, by digest, immutable	No (pointer to an artifact store)	No (pointer)	Yes (data and versions)
Lineage to data/code	Only via annotations	Yes, to the run	Yes, explicit	Yes, for datasets
Rich metadata	Annotations and JSON config	Yes, typed	Yes, typed	Partial
Promotion / approval	By convention (tags, labels)	Aliases and tags	States	No
Signing and attestations	Yes, native (`subject` + Referrers)	No	No	No
Cross-site replication	Yes, native	No	No	Partial
Standard API	OCI Distribution 1.1	Its own REST	Its own REST	S3/Git-like
Air-gap	Yes (`oras backup`/`restore`, `skopeo`)	Manual	Manual	Manual
Maintainer	CNCF (Harbor graduated; ORAS and ModelPack sandbox)	LF AI & Data	Kubeflow (CNCF)	Commercial OSS
Real maturity	High (registry), medium (model convention)	High	Alpha	High

The thesis of the table: the columns are complementary, not alternatives. The OCI registry is the repository; MLflow, the catalogue; lakeFS or DVC version the dataset that everything originates from, as detailed in data versioning with DVC and lakeFS.

The hot path: from the registry to HBM

Image volumes: the change of the year

The image volume source (KEP-4639) mounts an OCI image or artefact directly as a read-only volume in a pod: no initContainer, no copying, no PVC. Its track record: alpha in v1.31 (August 2024), beta in v1.33 (April 2025) with subPath support, beta by default in v1.35 and GA in v1.36, released on 22 April 2026.

apiVersion: v1
kind: Pod
metadata:
 name: vllm-modelo-oci
spec:
 containers:
 - name: vllm
 image: registro.interno/runtime/vllm:v0.11.0
 args: ["--model", "/mnt/models", "--served-model-name", "llama-3.3-70b"]
 volumeMounts:
 - name: pesos
 mountPath: /mnt/models
 readOnly: true
 resources:
 limits:
 nvidia.com/gpu: "4"
 volumes:
 - name: pesos
 image:
 reference: registro.interno/modelos/llama-3.3-70b@sha256:9f2c...
 pullPolicy: IfNotPresent

Two details that change operations: the reference is by digest, so immutability stops depending on anyone’s discipline; and pullPolicy: IfNotPresent makes the node’s runtime cache the blobs in its local store, so the second replica on that node starts without touching the network. It is the coordinated cache the anti-pattern was missing.

KServe: `storageUri` and modelcars

In KServe there are two routes: the classic one, storageUri with a storage initializer that copies from S3, PVC, HTTP or GCS into an emptyDir; and the modelcar, with the oci:// scheme:

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
 name: llama-70b-soberano
spec:
 predictor:
 model:
 modelFormat:
 name: huggingface
 storageUri: oci://registro.interno/modelos/llama-3.3-70b:2026.07.1
 resources:
 limits:
 nvidia.com/gpu: "4"

You have to understand the modelcar’s internal mechanism before depending on it: KServe sets shareProcessNamespace: true on the pod, starts a sidecar container with the model image and creates a symbolic link from /mnt/models to that process’s root filesystem through /proc. The runtime reads the weights without copying them, which is the saving being sought. Two warnings: it is not enabled by default (you have to turn on enableModelcar in the inferenceservice-config ConfigMap) and the latest tag forces pullPolicy: Always, cancelling out the cache. With image volumes now GA, the modelcar remains the route for clusters that are not yet on 1.36.

Loading fast: where each technique attacks

You have to separate two problems that get confused all the time: getting the bytes to the node (network) and getting them into HBM (disk → CPU → PCIe → GPU).

For the first, Dragonfly is the mature piece: it graduated in CNCF on 14 January 2026, distributes weights at the scale of hundreds of terabytes to hundreds of nodes “in minutes”, cuts the bandwidth consumed at the origin by up to 90 %, and makes use of between 70 % and 80 % of each node’s bandwidth. Its v2.5.0 (30 June 2026) added direct download from model repositories (dfget hf://...), P2P acceleration of Git LFS and a dragonfly-injector that injects P2P capability via a webhook without rebuilding images. Its subproject Nydus supplies the lazy-loading image format. There is a very common misplaced expectation here: for model weights lazy loading helps little, because an inference engine reads all the weights almost immediately; the real benefit is for fat runtimes and for models where only part is read.

For the second, the numbers published by the NVIDIA Run:ai Model Streamer on Llama-3-8B (15 GB, safetensors, a node with one A10G) are the cleanest reference available:

Source	Safetensors loader	Run:ai Model Streamer	Tensorizer
SSD gp3	47.99 s	14.34 s (concurrency 16)	16.11 s (16 workers)
SSD io2	47.0 s	7.53 s (concurrency 8)	10.36 s (8 workers)
S3 object storage	not supported	4.88 s (concurrency 32)	37.36 s (16 workers)

And the total time until vLLM is ready to serve: 66.13 s with the standard loader over gp3 versus 35.08 s with the streamer. Two critical readings. First: these are the project’s own benchmarks, not an independent measurement, on a 15 GB model on a modest GPU; extrapolating linearly to 140 GB on 4×H100 is a leap nobody has published. Second: the gain comes from saturating the storage medium with concurrency. If the bottleneck is an NFS at 1 Gb/s, no loader fixes it. The detail of this leg is in speeding up cold start with tensorizer and in from disk to HBM.

As the diagram shows, on a reference node of 4×H100 SXM 80 GB with NVLink each leg is between 5 and 10 times faster than the previous one. The conclusion is boring: the money is in having the model already on the node’s NVMe before the pod starts.

LoRA adapters: when the artefact weighs megabytes

Everything above assumes artefacts of tens or hundreds of gigabytes. A LoRA adapter breaks that assumption: for a 70B base, a rank-16 adapter over the attention projections weighs tens or a few hundred megabytes, three orders of magnitude less, and that changes the whole strategy.

With artefacts like that, the local cache and P2P stop mattering: the download is instant at any reasonable bandwidth. What starts to matter is cardinality and rate of change. A multi-tenant platform like the one described in multi-LoRA serving can have hundreds of adapters, each with its owner and its access policy, over a handful of bases. The practical consequences:

The adapter is an OCI artefact in its own right, with its own repository and an immutable tag. But the registry then has thousands of small repositories instead of tens of large ones, which stresses metadata, listings and granular RBAC, not disk.
It must declare its base. An adapter with no immutable reference to the digest of the base model it was trained on is a time bomb: applied over another version, it degrades quality silently. It is the natural use case for subject: the adapter refers to the base, and the Referrers API answers “which adapters exist for this digest”.
Loading is dynamic, not at start-up. vLLM allows hot loading of adapters with VLLM_ALLOW_RUNTIME_LORA_UPDATING and the /v1/load_lora_adapter endpoint, or declaratively with the LoRA resolver plugins (VLLM_PLUGIN_LORA_RESOLVERS), which resolve an adapter by name from a directory. The clean pattern on Kubernetes is to mount an image volume with the adapter and let the filesystem resolver discover it.
Versioning becomes the dominant problem. With 300 live adapters, knowing which adapter serves tenant X, over which base and trained with which dataset, only has an answer if the catalogue exists. Here a model registry does beat convention over tags.

Air-gap and sovereignty: promotion without losing traceability

The strict sovereign case: a connected environment where you ingest and validate, and an isolated environment where you serve, with no network route between them. Promotion crosses a data diode or a physical medium, and traceability has to survive the journey. It is the archive’s certified reproduction: the document, its record card and the seal certifying that the copy is faithful all travel together.

# 1. In the connected environment: export the artefact with EVERYTHING that hangs off it
oras backup \
 --output /transfer/llama-3.3-70b-2026.07.1.tar \
 --include-referrers \
 registro-dmz.interno/modelos/llama-3.3-70b:2026.07.1

# 2. Verify the exact digest before crossing
oras manifest fetch --descriptor \
 registro-dmz.interno/modelos/llama-3.3-70b:2026.07.1 > /transfer/descriptor.json
sha256sum /transfer/llama-3.3-70b-2026.07.1.tar > /transfer/SHA256SUMS

# 3. Physical crossing or data diode

# 4. In the isolated environment: restore preserving digests and referrers
oras restore \
 --input /transfer/llama-3.3-70b-2026.07.1.tar \
 registro-aislado.interno/modelos/llama-3.3-70b

The --include-referrers flag is what makes this worth anything: without it the weights cross, but the signature, the AIBOM and the attestations stay behind, and the isolated environment receives a model indistinguishable from one downloaded by hand. oras backup is marked experimental in v1.3.0, which has to be accepted explicitly before turning it into a procedure; the consolidated alternative for pure copying is skopeo copy with the oci-archive format, at the cost of managing the referrers separately.

What must travel attached to the artefact so that article 3/4 can verify it: the digest of the artefact and of its blobs; the provenance of the origin (the repository and the specific revision, not the model’s name, plus who ingested it and when); the identity of the process that packaged it (pipeline, commit, run), the raw material for an SLSA attestation; the cross-references to the evaluation dataset and to the base model if it is a derivative; and the promotion state with its approver, which in an isolated environment cannot be queried against the connected environment’s MLflow. All of it fits in OCI annotations and in artefacts referred to via subject. The rule: the isolated environment must be able to verify without calling anyone; if verification needs an external query, it is not air-gap.

Decision map

An OCI registry and a convention are enough when you have fewer than twenty models or so, a single team decides what gets promoted, lineage back to training is not a regulatory requirement (typical if you consume third-party models) and you already operate Harbor. Here a model registry adds one more console to maintain and no answer that immutable tags and annotations were not already giving. It is the majority case in pure inference.

A model registry is needed if you train or fine-tune and need auditable lineage from model to dataset and to code, the continuous fine-tuning cycle; if several teams compete to promote versions and an explicit approval flow is needed; if you have to answer an ISO 42001 or EU AI Act auditor about which version was in production on a given date; or if you manage tens of adapters with different owners.

The registry becomes valueless bureaucracy when the model is registered after being deployed (the catalogue as an administrative act, not as a gate); when the promotion state is not connected to any technical control and an unapproved version can be deployed; or when a model registry and an OCI registry coexist with two truths and no digest joining them. The last one is the most common failure and the most expensive: two versioning systems that diverge in silence.

Operational traps

Mutable tags and latest in production. Beyond style, a mutable tag turns any audit into guesswork and lets two replicas of the same Deployment serve different bytes. On top of that, in KServe latest forces pullPolicy: Always and cancels the node cache, so you pay in cold start what you lose in traceability. Rule: immutable tags by project policy and deployment by digest.

GC that deletes referenced layers. The archive’s weeding. The garbage collector identifies blobs that no manifest references, and the window between “I have uploaded the blobs” and “I have uploaded the manifest” is exactly where a concurrent GC can take them away. In Harbor this has historically shown up in _uploads and in discrepancies between the space the UI claims to have freed and the real figure. Mitigation: GC in a window with pushes blocked, and do not trust the first pass.

The registry’s disk and the cost of versioning. Brutal arithmetic: 140 GB per version with three live versions is 420 GB; twelve models like that, 5 TB; with six-version retention and two replicated sites, 20 TB. And it is expensive-class storage if the registry lives on replicated block. Deduplication helps little, because the weights change completely between versions. The retention policy has to be designed before filling the registry, and models must be separated from runtime images by project, because their retentions have nothing to do with each other.

Replication that saturates the inter-site link. An event-driven rule on a model repository moves 140 GB every time someone pushes a version: on a shared 1 Gb/s link, almost twenty minutes flat out in the middle of the working day. Schedule it in a window, limit bandwidth, filter by label which versions get replicated and use P2P inside each site.

Putting the model inside the runtime image. Tempting because “that way there is only one artefact”. The result is a 145 GB image that has to be rebuilt and redistributed every time a runtime CVE is patched, with the security life cycle coupled to the model’s. Separated: a small, patchable runtime and a large, stable model.

For an inference factory

Three actionable things for anyone running an on-premise inference factory, not a training one.

First: cut the internet dependency at start-up, this week. You do not need a platform project: an ingestion task that pulls each model once, pushes it to Harbor as an OCI artefact with an immutable tag and provenance annotations, plus switching the deployments to reference by digest, is enough. The HF_TOKEN disappears from the pod and moves to the ingestion pipeline, which is where it makes sense and where it fits with secrets hardening.

Second: decide consciously whether you need a catalogue. If you only consume third-party models and do not train, probably not: the OCI registry with a convention gives 90 % of the value at 10 % of the operational cost. If you train, fine-tune or manage per-tenant adapters, set up MLflow and store the OCI digest as a tag on each version, so that catalogue and repository cannot diverge.

Third: move the bottleneck to the right place. Before investing in load accelerators, measure the three legs. If the model arrives over the network on every start-up, the work is a local NVMe cache, image volumes with pullPolicy: IfNotPresent and, with many nodes, P2P. Only when the model is already on the node’s disk does it make sense to wrestle with loader concurrency.

So much for where the bytes come from and by what route they arrive. What remains is what we have taken for granted all along: why trust those bytes. A digest guarantees integrity, that nobody has changed them, but not provenance: it does not say who produced them, with what data or whether somebody with authority certified that they could be served. The third article in the series builds that answer with Sigstore, SLSA, in-toto and AIBOM, hanging precisely off the subject and the Referrers API that we have left ready here.

Sources

Open Container Initiative, OCI Image and Distribution Specs v1.1 Releases (13 March 2024) — https://opencontainers.org/posts/blog/2024-03-13-image-and-distribution-1-1/
CNCF, Announcing ORAS v1.3.0: Elevating artifact and registry management workflows (6 October 2025) — https://www.cncf.io/blog/2025/10/06/announcing-oras-v1-3-0-elevating-artifact-and-registry-management-workflows/
CNCF, ORAS project page (sandbox since 13 July 2021) — https://www.cncf.io/projects/oras/
ORAS, oras backup (experimental) — https://oras.land/docs/commands/oras_backup
ModelPack, CNCF ModelPack Specification — model spec (artifactType and mediaTypes) — https://github.com/modelpack/model-spec
ModelPack, modctl — getting started (Modelfile and commands) — https://github.com/modelpack/modctl/blob/main/docs/getting-started.md
CNCF, Cloud Native Computing Foundation announces Harbor graduation (23 June 2020) — https://www.cncf.io/announcements/2020/06/23/cloud-native-computing-foundation-announces-harbor-graduation/
Harbor, Configure proxy cache (supported registries) — https://goharbor.io/docs/2.6.0/administration/configure-proxy-cache/
OneUptime, How to configure Harbor project quotas for storage limits (9 February 2026) — https://oneuptime.com/blog/post/2026-02-09-harbor-project-quotas-storage/view
endoflife.date, Harbor releases (2.15.2 on 2 July 2026; 2.15 on 20 March 2026) — https://endoflife.date/harbor
VMware Cloud Foundation Blog, Using Harbor as an AI Model Registry (3 March 2026) — https://blogs.vmware.com/cloud-foundation/2026/03/03/using-harbor-as-an-ai-model-registry/
MLflow, Model Registry workflows — stages deprecated since MLflow 2.9.0 — https://mlflow.org/docs/latest/ml/model-registry/workflow/
MLflow, Release 3.13.0 highlights — https://mlflow.org/releases/3.13.0/
Kubeflow, Hub (formerly Model Registry) — overview — https://www.kubeflow.org/docs/components/hub/overview/
kubernetes/enhancements, KEP-4639: OCI VolumeSource (alpha 1.31, beta 1.33/1.35, GA 1.36) — https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/4639-oci-volume-source/README.md
PerfectScale, Kubernetes v1.36 release: OCI artifact volume support reaches GA (22 April 2026) — https://www.perfectscale.io/blog/kubernetes-v1-36-release
KServe, OCI storage (modelcar) — https://kserve.github.io/website/docs/model-serving/storage/providers/oci
CNCF, CNCF announces Dragonfly’s graduation (14 January 2026) — https://www.cncf.io/announcements/2026/01/14/cloud-native-computing-foundation-announces-dragonflys-graduation/
CNCF, Dragonfly v2.5.0 is released (30 June 2026) — https://www.cncf.io/blog/2026/06/30/dragonfly-v2-5-0-is-released/
NVIDIA Run:ai, Model Streamer benchmarks — https://github.com/run-ai/runai-model-streamer/blob/master/docs/src/benchmarks.md
vLLM, LoRA adapters (dynamic loading and resolver plugins) — https://docs.vllm.ai/en/stable/features/lora/

Prompt versioning: the contract that stops a five-word change sinking your system

Fri, 22 May 2026 07:30:00 +0200

TL;DR

In a classic software system, the most dangerous line a team can change is a SQL migration. In an LLM system, it is a line of prompt. The prompt determines the output as much as the model or more, it does not show up in unit tests, it does not appear in the logs by default, and if it is changed without leaving a trail there is no way to know which version generated which response. Prompt versioning is the discipline that turns the prompt into a first-class artefact: with a unique identifier, history, deployment labels, an associated eval suite, and per-request traceability. The field has settled on three primitives (immutable version, mutable label, read cache) and two dominant tools (Langfuse OSS with a built-in UI, MLflow Prompts integrated into the registry since MLflow 3.10). This article covers the pattern at first level: why it matters, how it is implemented, which tool to choose, and how it fits with Eval, Deploy and Observe.

You are here: cross-cutting (touches Data, Tune, Eval, Deploy and Observe)

Prompt versioning does not live in one stage, it cuts across five. It appears as a cross-cutting component in the master map of the six-stage LLMOps pipeline precisely for that reason: the prompt version is necessary metadata at every stage, not the responsibility of a single one.

The master analogy: the prompt is an invisible SQL migration

A serious backend team would never accept someone modifying a column directly in production without going through a versioned migration. Even if the change “works” at the time, without a migration there is no way to:

Reproduce the previous state if something fails.
Know who applied the change and when.
Apply the same change in staging before prod.
Test the new version against an automated suite before promoting.
Know, two months later, why the table has the shape it has.

The LLM prompt occupies exactly that position in an inference system. Changing "You are a helpful assistant." to "You are a helpful, concise assistant. Answer in fewer than 3 sentences." can:

Cut the average cost per response by 30 % (the responses are shorter).
Or degrade quality in a segment where concision breaks necessary nuance.
Or change the distribution of tools the agent decides to invoke.
Or alter the behaviour of the downstream LLM judge that assumes a certain length.

And most importantly: if the change is made by editing a constant in the app code and deploying, when two weeks later someone asks “why did the complaint rate go up in the finance segment?”, there is no way to know which prompt was being served at each moment. The logs store the response and, with luck, the model invoked; the prompt is rarely stored explicitly.

Prompt versioning solves the same problem Flyway/Liquibase/Alembic solved for SQL: turning an invisible change into an auditable artefact.

The three primitives of the pattern

Whatever the tool, the systems that work in 2026 share three operational primitives worth fixing before looking at products.

1. Immutable version

Every time the prompt content changes (template, system message, available variables, recommended model parameters such as temperature), a new version is generated with a unique identifier. The version is immutable: once created, it is not overwritten; if something needs changing, v+1 is created.

prompt_id: customer_support_v3
versions:
v1 (2026-03-12): "Eres un asistente de soporte..."
v2 (2026-04-08): "Eres un asistente de soporte... formato JSON..."
v3 (2026-05-21): "Eres un asistente de soporte... formato JSON... 3 frases máx..."

Immutability is what allows a trace from two months ago to be reproduced: if the trace says “customer_support_v3@v2 was served”, version v2 exists literally and can be reloaded.

2. Mutable label (deployment alias)

Versions are immutable, but which version is in production changes. That decision is materialised in labels: pointers with a semantic name (production, staging, canary) that point at a specific version and can be re-pointed.

prompt_id: customer_support_v3
labels:
production → v2 (servida al 100% del tráfico)
canary → v3 (servida al 5% del tráfico via gateway)
staging → v3

Promoting a version is moving a label, not editing the prompt. Rollback is moving the label back, not copying text. The operation reduces to an atomic mutation of a (label, version) tuple.

3. Read cache

The prompt is read on every request to the model. If every read calls the prompt registry service, you add latency and a dependency. The standard solution is a local cache in the client (TTL on the order of minutes) that invalidates when the label changes or when the TTL expires.

Langfuse implements a native client cache with configurable TTL and lazy invalidation; MLflow Prompts leaves the responsibility to the client or to a gateway layer. In both cases, in production the client serves the prompt from memory with negligible overhead (<1 ms), and only goes to the registry when it refreshes.

┌──────────────────┐
│ Client (app) │
│ - local cache TTL=60s
│ - lookup label "production"
│ - gets template
│ - renders variables
│ - sends to LLM
└─────────┬────────┘
│ (when the TTL expires or a change event arrives)
▼
┌──────────────────┐
│ Prompt registry │
│ - Langfuse / MLflow
│ - GET label="production"
│ - response: version_id + template
└──────────────────┘

With these three primitives, any reasonable tool is equivalent in the essentials. What distinguishes one from another is UI, integrations, RBAC, eval integration, and so on.

The two dominant tools in 2026

The field has converged on two main options. Any serious production deployment uses one of the two (sometimes both, for different teams).

Langfuse (OSS, prompt-management UI built in)

Langfuse is the prompt-first system: it was born for tracing and observability, and prompt management is one of its core layers. Key features for versioning:

Built-in UI to create, edit and version prompts. Versions are generated automatically on save; the history is visible and diffable.
Arbitrary labels beyond the usual ones (production, latest). You can define eu-prod, internal-only, customer-a for fine-grained routing.
Native client cache in the official SDKs (Python, JS), with configurable TTL, event-driven invalidation and fallback to the last-known-good if the registry is down.
Native tracing integration: when you record a call to the LLM, Langfuse automatically associates the prompt_id@version that served it. In the UI you see: this trace, this span, this prompt version X.
Eval integration: Langfuse lets you register eval suites that fire when a new prompt version is created. The results stay linked to the prompt_id@version and are the natural gating for promoting staging → production.
Self-hosted or cloud: the core is OSS (MIT), runs on Docker compose or Helm; the cloud version adds SLA, SSO and support.

When Langfuse suits:

Teams that want a rich UI so product/PM/analyst can manage prompts without touching code.
OSS-first deployments where control of the runtime and of persistence is a requirement (on-premise, ENS).
When LLM observability is already in Langfuse: prompt management is marginal in setup.

MLflow Prompts (included in MLflow 3.10, March 2026)

MLflow Prompts is the classic MLOps ecosystem’s answer for LLMs. Features:

Integrated into the MLflow Model Registry: prompts are first-class registry artefacts, with the same stage semantics (Staging, Production, Archived) MLOps teams already know.
API consistent with the rest of MLflow: mlflow.register_prompt(), mlflow.load_prompt(name, stage="Production"). The learning curve for teams already using MLflow for models is nil.
Automatic versioning with a numeric version_id (1, 2, 3, …) and optional comments on promotion.
No built-in UI dedicated to prompts (the MLflow UI does the job, but it is designed for models; the flow is less polished than in Langfuse).
No native GenAI-aware tracing (MLflow Tracing in the 3.10 GenAI dashboard provides it, but the trace↔prompt integration is more manual than in Langfuse).
Compatible with any model registry backend MLflow supports (filesystem, Postgres, MySQL, S3, GCS, Azure Blob).

When MLflow Prompts suits:

Teams already operating MLflow for classic ML who want to extend the same discipline to LLMs without adding vendors.
Deployments where the centre of gravity is the model registry and the prompt is one more artefact.
CI/CD pipelines that already speak MLflow (CLI, REST API).

Comparison

Feature	Langfuse	MLflow Prompts
Core licence	MIT (OSS)	Apache 2.0 (OSS)
Prompt-first UI	✅	⚠️ via Model Registry
Immutable versioning	✅	✅
Mutable labels	✅ (arbitrary)	✅ (Staging/Production/Archived)
Native client cache	✅	❌ (DIY)
Integrated tracing	✅ native	⚠️ via MLflow Tracing
Eval gating on promotion	✅	⚠️ DIY with MLflow Recipes
Easy self-hosting	✅ Docker/Helm	✅ standard MLflow
Curve if you come from MLOps	medium	nil
Curve if you come from DevOps	nil	medium

In May 2026, the most widespread hybrid pattern is to use MLflow for the models+adapters registry and Langfuse for prompts+tracing, connected by trace_id and prompt_id travelling in the OpenTelemetry span attributes. Covered in evals and MCP observability.

Minimum schema of a versioned prompt

Whatever the tool, what the registry stores in each version has a reasonable minimum schema:

# prompt_id: customer_support_v3, version: 3
template:
 system: |
 Eres un asistente de soporte de {{company_name}}.
 Responde en español neutral, máximo 3 frases.
 Formato de respuesta: JSON {"answer": "...", "needs_human": bool}
 user: |
 Pregunta del cliente: {{user_message}}
 Contexto del ticket: {{ticket_context}}

variables:
 required: [company_name, user_message, ticket_context]
 defaults: {}

recommended_params:
 model: "llama-3-70b-instruct"
 temperature: 0.3
 max_tokens: 300
 response_format: "json_object"

metadata:
 author: "mlops@empresa.com"
 created_at: "2026-05-21T14:23:00Z"
 commit_message: "Añade límite de 3 frases tras feedback ticket #1842"
 eval_suite: "customer_support_v3_evals"
 related_traces: ["trace_id_x", "trace_id_y"]

This is the minimum contract. What sets a serious deployment apart:

variables.required is validated in the client before sending to the model. A missing variable blows up at client time, not in a confusing model response.
recommended_params.model ties the prompt version to a model. Changing model opens a debate (does the new version work with Llama 3 70B and with GPT-4o?). If it is not tied, the model is one more variable that wrecks reproducibility.
metadata.eval_suite is what the eval suites hook into: on creating v3, MLflow/Langfuse fires customer_support_v3_evals automatically.

Integration with eval gates: governed promotion

The real value of prompt versioning appears when it is integrated with eval. The canonical pattern:

Developer edits the prompt in the UI (Langfuse) or via the API (MLflow). v4 is created.
Automatic trigger: the prompt_created event fires the associated eval suite (the eval_suite in the metadata).
The suite runs against the golden dataset (questions+answers labelled by a human). Covered at first level in the evals post.
Results are attached to the version: v4 now has eval_score: 0.84, regression_vs_v3: -0.03.
Promotion gate: if eval_score >= threshold and regression < tolerance, the staging label moves to v4 automatically. If not, it alerts the developer.
Manual promotion to production: with the eval passed, someone with permission moves production from v3 to v4. Atomic, auditable, reversible.

Developer edits prompt → v4 created
│
▼
[eval suite trigger]
│
▼
Golden dataset 200 examples
│
▼
score = 0.84 (vs 0.87 for v3)
│
├── If it passes the threshold → label staging → v4
│ └── Manual promotion to production after review
└── If it does not pass → block + alert the developer

This flow turns the prompt change from “someone touched the code and we prayed” into “a prompt change is a PR that passes CI”. It is the same discipline classic MLOps applied to models.

Per-request traceability: which version served each response

The last piece is operational traceability: given a model response in production, which prompt version generated it?

The pattern is to propagate the version as a span attribute in OpenTelemetry, following the gen_ai.* semantic conventions we covered in MCP observability:

# In the client (common pseudo-code)
prompt = registry.load("customer_support_v3", label="production") # v3 → v_id=14

with tracer.start_as_current_span("llm_call") as span:
 span.set_attribute("gen_ai.prompt.id", "customer_support_v3")
 span.set_attribute("gen_ai.prompt.version", "14")
 span.set_attribute("gen_ai.prompt.label", "production")
 span.set_attribute("gen_ai.request.model", prompt.params.model)

 response = llm.complete(prompt.render(user_message=msg), **prompt.params)

 span.set_attribute("gen_ai.usage.input_tokens", response.usage.input)
 span.set_attribute("gen_ai.usage.output_tokens", response.usage.output)

In any trace (Langfuse, Phoenix, Jaeger, Honeycomb) you can see which exact version served that response. In an incident (“customer X received this on 22 May”) you reproduce literally the version and the model that generated the output.

Without that traceability, the incident stays an anecdote; with it, it is debuggable.

Applied to typical on-premise hardware

Prompt versioning is a computationally light layer compared with the inference engine or the fine-tuning pipeline. Its requirements:

Storage: the prompt registry typically weighs megabytes (hundreds to thousands of prompts with their versions). Postgres with a prompts(id, version, template, params jsonb, metadata jsonb, created_at) schema is more than enough. Langfuse uses Postgres by default; MLflow uses it for metadata (the blobs go to object storage or the filesystem).
Registry compute: a small instance (1-2 vCPU, 2 GB RAM) serves tens of thousands of reads per minute if the client cache is enabled. Without a cache, it scales linearly with QPS but is still trivial.
Triggered eval compute: here there is a real cost. Every time a new version is created, the eval suite runs. If the suite does LLM-as-judge over 200 examples and each eval costs 4 K tokens, one promotion costs on the order of 1 M tokens, minutes on a decent cluster, seconds if the suite already has its hot prefix cache.

For an RTX 4090 serving Llama 3 8B with a self-hosted prompt registry (Langfuse or MLflow): the registry runs on the same node in a sidecar container, the local app caches in RAM, the eval triggers run against the same inference engine at low priority. Full setup in a morning.

For a 4×H100 SXM cluster serving a large model to several tenants: registry in a dedicated K8s pod with replicated Postgres, eval suites running in pods with the spot priority class (covered in the cluster as a platform), OTel tracing propagating prompt_id+version to a central Langfuse.

Traps and things that are not what they look like

Prompts hardcoded in the app code. The most common antipattern. The prompt lives in a prompts.py file or templates/customer.txt that gets deployed with the app. There is no real versioning (git history is not a substitute: it does not tie commit ↔ production trace operationally). Migrating to a registry is 1-2 sprints of work; it is worth every hour.

Badly calibrated cache. A TTL of hours with a mutable label means a rollback takes time to propagate. A TTL of seconds overloads the registry. The reasonable default is 60-300 seconds with event-driven invalidation (the registry emits a message to Kafka/Redis when a label changes, and clients invalidate immediately).

Unvalidated variables. The template uses {{user_name}} but the app passes {{username}}. The render produces a prompt with a literal {{user_name}}. The model answers something bizarre and nobody knows why. Validating required variables in the client before sending to the model is the minimum discipline.

Prompts inside chains evaluated at runtime. If your stack uses LangChain, LlamaIndex or similar with chains that compose prompts at runtime, the final prompt the model sees may not be in the registry because it was composed from several fragments. Solutions: either register the chains as artefacts, or log the effective composed prompt in every trace.

Eval suite not hooked to the prompt_id. Without that link, a prompt change gets promoted without passing evals. The integration has to be a field in the prompt metadata (eval_suite: ...) that the system reads and fires automatically. If it depends on the developer “remembering”, the pattern will fail.

Non-existent RBAC roles. Anyone with access to the UI can move production to any version. Without separating editor (creates versions) from releaser (moves production labels), a junior developer can break production with an accidental promotion. Langfuse Enterprise has granular RBAC; MLflow has it via the server backend with per-experiment/registry permissions.

Prompts with sensitive data inline. The prompt template includes few-shot examples with real names, addresses, customer IDs. The registry stores that indefinitely. Under GDPR, the right to erasure applies to the registry too. Good practice: variables for sensitive data, not inline; periodic audit of the registry content.

Recommended operational pattern: the cycle on one screen

A serious team with prompt versioning properly set up has the following cycle, repeatable and cheap:

Developer opens a PR in the repo: changes the app code if necessary, but does not touch the prompt there.
Edits the prompt in the Langfuse/MLflow UI: creates v_new. Adds a commit message (“adds a 3-sentence limit after feedback on ticket #1842”).
The eval suite fires automatically: it runs against the golden dataset, results appear in the UI within minutes.
If it passes eval: the staging label moves to v_new automatically. The developer can test staging with controlled traffic.
Human review (1-2 people, optional depending on severity): approval.
Promotion to production: move the label, atomic. The client caches for 60-300 s, then serves the new version.
Observe: in Langfuse/Phoenix, production metrics and eval scores are segmented by prompt version. If the score drops with v_new, alert.
If there is a serious regression: rollback is moving the label back. A 5-second operation.

Every step is audited, every decision leaves a trail, every rollback is an atomic operation. This is what separates a GenAI system from “demos that worked once” and makes it operable for years.

What we have not covered (upcoming posts)

Automatic prompt optimisation: techniques such as DSPy, TextGrad, PromptBreeder that generate prompt candidates and optimise them against a measurable objective. The extension of versioning where the “developer” can be an optimiser.
Prompt injection and red teaming: integrating versioning with the adversarial evaluation flow. Partially covered in guardrails.
Different versions per tenant: when the same prompt_id needs variants per customer (i18n, branding, domain). Fork + override pattern.

References

Langfuse documentation, Prompt Management: https://langfuse.com/docs/prompts/get-started.
MLflow 3.10 release notes, Prompts in Model Registry (marzo 2026): https://mlflow.org/releases/3.10.
OpenTelemetry, Semantic Conventions for Generative AI (estables desde 1.36 de OTel): https://opentelemetry.io/docs/specs/semconv/gen-ai/.
Google Cloud, Prompt Management: Best Practices for Production LLM Systems (2025).
Chip Huyen, Designing Machine Learning Systems — capítulo sobre model registry y prompt-as-artifact (2ª edición, marzo 2026).
Eugene Yan, Prompt Engineering as Software Engineering (blog, 2025).

Mlflow on lo0 — Tech Blog

Model chain of trust (2/4): where the bytes live — registry, OCI artefacts and distribution to the GPU node

TL;DR

The analogy

Three planes that are not the same

The anti-pattern we start from

The model as an OCI artefact

What OCI 1.1 made possible

ORAS: the generic client

ModelPack: the missing convention

How a 140 GB model is sliced up

Harbor as the on-premise repository

Model registries proper

MLflow

Kubeflow Hub (formerly Model Registry)

Comparison table by function

The hot path: from the registry to HBM

Image volumes: the change of the year

KServe: storageUri and modelcars

Loading fast: where each technique attacks

LoRA adapters: when the artefact weighs megabytes

Air-gap and sovereignty: promotion without losing traceability

Decision map

Operational traps

For an inference factory

See also

Sources

Prompt versioning: the contract that stops a five-word change sinking your system

TL;DR

You are here: cross-cutting (touches Data, Tune, Eval, Deploy and Observe)

The master analogy: the prompt is an invisible SQL migration

The three primitives of the pattern

1. Immutable version

2. Mutable label (deployment alias)

3. Read cache

The two dominant tools in 2026

Langfuse (OSS, prompt-management UI built in)

MLflow Prompts (included in MLflow 3.10, March 2026)

Comparison

Minimum schema of a versioned prompt

Integration with eval gates: governed promotion

Per-request traceability: which version served each response

Applied to typical on-premise hardware

Traps and things that are not what they look like

Recommended operational pattern: the cycle on one screen

What we have not covered (upcoming posts)

See also

References

KServe: `storageUri` and modelcars