Model chain of trust (2/4): where the bytes live — registry, OCI artefacts and distribution to the GPU node

In the first article in the series we described the control plane: who declares that an inference endpoint exists, who reconciles it and what contract the clients speak. But an InferenceService is no more than a promise until somebody puts 140 GB of weights on the filesystem the container sees. This second article is about exactly that: where those bytes come from and by what route they reach the GPU’s memory.

It is a subject the blog has touched around the edges, in the cold start series, in data versioning with DVC and lakeFS, in the catalogue of OSS LLMOps tools, but never head on. It is time to, because almost every on-premise inference platform that gets audited shares the same original defect, and it is conceptual before it is about performance: they conflate three things that live in different places and fail in different ways.

TL;DR

• Three planes, not one. Model registry (metadata, versions, lineage, promotion) ≠ artefact repository (the bytes, addressed by digest) ≠ deployment storage (what the pod mounts at /mnt/models). Mixing them is what breaks reproducibility.
• The anti-pattern: a Hugging Face token in the pod and a download from the internet on every start-up. No immutability, no cache, no audit trail, no air-gap and cold starts measured in minutes.
• The OCI artefact is today’s pragmatic answer: OCI Image Spec 1.1 (March 2024) brought artifactType, subject and the Referrers API; ORAS v1.3.0 (October 2025) added backup/restore for air-gap; ModelPack (CNCF sandbox since June 2025) standardises the mediaTypes for weights.
• Kubernetes already knows how to mount OCI artefacts: the image volume source (KEP-4639) reached GA in v1.36 (22 April 2026). It is the most operationally relevant change of the year for model distribution.
• The real bottleneck is almost never the GPU. 140 GB over a saturated 10 Gb/s link is about 112 s; over Gen4 NVMe about 20 s; over PCIe Gen5 about 2-3 s. Optimising the loader without fixing the network is optimising the wrong link.
• A model registry (MLflow, Kubeflow Hub) does not store the bytes: it stores the record card and a pointer. If you were expecting it to solve your distribution, you bought the wrong tool.

The analogy

The simile is a serious historical archive, with three things a distracted visitor constantly confuses.

The catalogue: record cards with a shelfmark, author, date, provenance, conservation state and consultation restrictions. The card weighs grams and describes something that weighs kilos. The repository: the compact shelving in the basement where the physical boxes are, with access control, temperature control and an inventory that balances to the gram; it knows nothing about why a document matters, but it guarantees that box 4711 still contains exactly what it contained. And the reading room table: where the researcher has the document open in front of them. A temporary and expensive place, from which the document goes back to the repository when the session ends.

The model registry is the catalogue. The artefact repository is the repository. The GPU node with the model in HBM is the reading room table. Anyone who says “we have set up a model registry” and what they have set up is MLflow has set up the catalogue and left the repository unbuilt: the boxes are still in somebody’s car boot. Putting the model inside the container image is the equivalent of putting the document inside the table: every time the table changes, the document moves. And anyone who downloads from the internet on every start-up has no repository: they order the document by courier from another city every time a researcher sits down.

We will come back to the archive later, with the weeding that deletes boxes that are still referenced, the trolley that carries things between the basement and the reading room, and the certified reproduction sent to the isolated site.

Three planes that are not the same

Before discussing tools you have to fix the vocabulary, because a good part of the architecture discussions at this layer are really discussions about nomenclature.

The three layers have different life cycles. A record card lives forever (audit); a blob, for as long as some version references it (retention); a volume, for as long as the pod lives. Implemented as a single thing, it inherits the worst of the three: if the registry is also the deployment storage, deleting an old version takes down a pod in production; if the deployment storage is also the canonical reference, there is no way to answer what was served last Tuesday.

Hence the operational rule that gets written on the wall: the source of truth for the bytes is a digest, not a tag and not a path. Everything else (tags, aliases, PVC paths) is a mutable pointer that resolves to that digest at a given moment. Article 3/4 builds all of its provenance verification on that premise.

The anti-pattern we start from

This is, by a distance, the most widespread pattern in on-premise deployments that started as a pilot and stayed that way:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-anti-patron
spec:
  template:
    spec:
      containers:
        - name: vllm
          image: vllm/vllm-openai:v0.11.0
          args: ["--model", "meta-llama/Llama-3.3-70B-Instruct"]
          env:
            - name: HF_TOKEN
              valueFrom:
                secretKeyRef:
                  name: hf-token
                  key: token
          resources:
            limits:
              nvidia.com/gpu: "4"

It works and it serves tokens. And it is unacceptable on a sovereign platform for five independent reasons, any one of which would be enough:

1. An external dependency on the critical start-up path. A Deployment that cannot recover from a failure without outbound internet access carries a third-party dependency that nobody has declared in the SLA.
2. Zero immutability. meta-llama/Llama-3.3-70B-Instruct is a mutable reference, and a repository’s default branch changes (tokenizer fixes, chat templates). Two replicas of the same Deployment started a week apart can serve different bytes, and nothing in the cluster says so.
3. Zero coordinated cache and zero audit trail. Every pod downloads its own copy: eight replicas of a 140 GB model is 1.1 TB of outbound traffic per rolling update. And no internal record remains of what was downloaded, with which digest or who authorised it; faced with an ENS or ISO 42001 auditor, the answer is “we got it off the internet”.
4. Incompatible with air-gap. The whole design collapses when the platform has to be replicated in an isolated environment, which is the defence and healthcare case dealt with in regulatory compliance infrastructure.
5. Cold start measured in minutes. Downloading 140 GB over a 1 Gb/s outbound link (125 MB/s effective, being generous) is about 19 minutes of transfer alone, before touching the disk or the GPU; with 10 Gb/s dedicated and saturated, about 112 seconds. In a load-driven scaling event, that is service latency.

The blog’s cold start series explores how to trim the disk-to-HBM leg. What this article adds is that if the first leg is the internet, the rest of the optimisation is irrelevant.

The model as an OCI artefact

The pragmatic answer is boring, and that is why it works: the model is stored in the same registry where the containers are already stored. There is already access control, cross-site replication, scanning, quotas, audit and a team that knows how to operate it.

What OCI 1.1 made possible

For years, putting things that were not container images into an OCI registry was a bodge. The OCI Image Spec 1.1 and Distribution Spec 1.1 specifications, published on 13 March 2024, turned it into a first-class use case with three pieces:

• artifactType in the manifest: it explicitly declares “this is not a runnable image, it is an artefact of type X”. Previously this was encoded by abusing the config mediaType, which is why even today there are registries that show a model as a broken image.
• subject: a manifest can declare that it refers to another manifest. That is what lets you hang a signature, an SBOM or an attestation off a model without modifying it and without changing its digest.
• Referrers API (GET /v2/<repo>/referrers/<digest>): the reverse query, “give me everything that points at this digest”, with a tag-based fallback for registries that do not implement it yet.

Article 3/4 builds signing, provenance and AIBOM on top of these three pieces. Without them, verifying a model forces you to invent a convention of your own.

ORAS: the generic client

ORAS (OCI Registry As Storage) is the Swiss army knife for getting arbitrary artefacts in and out of an OCI registry. Version v1.3.0 was published on 6 October 2025, conformant with distribution-spec v1.1.1, and added three things relevant to models: backup/restore to a directory or tarball (key for air-gap), management of multi-platform indexes and --format for structured output.

One data point to calibrate maturity: ORAS entered the CNCF sandbox on 13 July 2021 and as of this article is still in sandbox. Five years at the entry level does not invalidate the tool, which is stable and the de facto reference, but it says something about the project’s governance that is worth weighing before turning it into a critical dependency with no plan B.

Pushing a model with ORAS, with explicit content types:

oras push registro.interno/modelos/llama-3.3-70b:2026.07.1 \
  --artifact-type application/vnd.cncf.model.manifest.v1+json \
  --annotation "org.opencontainers.image.created=2026-07-20T09:00:00Z" \
  --annotation "es.ejemplo.modelo.precision=bf16" \
  --annotation "es.ejemplo.modelo.origen=hf:meta-llama/Llama-3.3-70B-Instruct" \
  config.json:application/vnd.cncf.model.weight.config.v1.raw \
  tokenizer.json:application/vnd.cncf.model.weight.config.v1.raw \
  model-00001-of-00030.safetensors:application/vnd.cncf.model.weight.v1.raw \
  model-00002-of-00030.safetensors:application/vnd.cncf.model.weight.v1.raw

# Retrieve it, with a local blob cache so nothing already present is re-downloaded
export ORAS_CACHE=/var/cache/oras
oras pull registro.interno/modelos/llama-3.3-70b:2026.07.1 --output /srv/modelos/llama-70b
oras manifest fetch registro.interno/modelos/llama-3.3-70b:2026.07.1 --descriptor

ModelPack: the missing convention

ORAS lets you invent the mediaTypes, with the predictable result that every organisation invents its own and nothing interoperates. ModelPack is the attempt to standardise them, accepted into the CNCF sandbox in June 2025. On top of OCI Image Manifest 1.1 it defines:

• artifactType: application/vnd.cncf.model.manifest.v1+json
• config: application/vnd.cncf.model.config.v1+json
• weight layers: application/vnd.cncf.model.weight.v1.raw (and the .tar, .tar+gzip, .tar+zstd variants)
• weight configuration, documentation, code and dataset layers following the same pattern

Its reference CLI is modctl, which works with a declarative Modelfile:

NAME llama-3.3-70b-instruct
ARCH transformer
FAMILY llama
FORMAT safetensors
PARAMSIZE 70
PRECISION bf16
CONFIG config.json
CONFIG tokenizer.json
MODEL *.safetensors
DOC *.md

modctl build -t registro.interno/modelos/llama-3.3-70b:2026.07.1 -f Modelfile .
modctl push registro.interno/modelos/llama-3.3-70b:2026.07.1
modctl pull registro.interno/modelos/llama-3.3-70b:2026.07.1 \
  --extract-dir /srv/modelos/llama-70b --extract-from-remote

The alternative with more traction among users is KitOps (ModelKit), with Docker-like ergonomics and native integration with Hugging Face. As of this article there is no convergence: ModelPack aims at the enterprise case and at integration with runtimes and registries; KitOps and Docker Model Runner, at developer ergonomics. Picking one today is a bet; using artifactType and explicit annotations instead of home-grown conventions is not.

How a 140 GB model is sliced up

A large model is 20-40 safetensors shards plus half a dozen small configuration files, and the layering strategy matters:

• One layer per shard, uncompressed. Weights in bf16 or fp8 are incompressible in practice: gzip over safetensors burns CPU to save 2-3 %, and forces decompression on the node, adding a full pass through CPU and disk on the critical path. The right type is .raw.
• The small files in their own layer. config.json, tokenizer.json and the templates change far more often than the weights. If they go in the same layer as a 5 GB shard, every tokenizer fix invalidates 5 GB of cache on every node.
• Layer size between 2 and 8 GB. Large enough not to pay per-blob overhead, small enough to parallelise and so that a network failure does not force you to retry 40 GB.

In practice three things hurt. The upload uses the distribution-spec’s chunked upload, with historically uneven interoperability between implementations, so pushing 140 GB is an operation measured in tens of minutes that demands a decent link and retries. Deduplication is by layer digest, so two versions that differ in one shard share the rest only if those blobs are byte-for-byte identical, and any repackaging that alters metadata breaks it. And the default values of reverse proxies (timeouts, maximum body size, upload session lifetime) are calibrated for 1 GB images, not for 8 GB blobs.

Harbor as the on-premise repository

Harbor is the reference OSS registry for on-premise: a CNCF graduated project since June 2020, with a healthy release cycle (the 2.15 series came out on 20 March 2026 and patch 2.15.2 on 2 July 2026) and support for the last three minors for about nine months each. What matters for models:

Cross-site replication. Push or pull rules, filtered by repository, tag and resource type, with manual, scheduled or event-driven triggering. For models, scheduled and within a window, with a label marking which versions get replicated (see traps).

Proxy cache. Here you have to be precise, because a lot of misinformation circulates: Harbor’s proxy cache works against OCI registries (Harbor, Docker Hub, a distribution registry, ECR, ACR, GCR, Quay and GHCR) and not against Hugging Face, which does not expose an OCI distribution API. If the idea was “I put Harbor in front of Hugging Face and that is it”, it does not work that way. The real options are (a) an ingestion task that pulls from Hugging Face in a staging environment and pushes to Harbor as an OCI artefact, the recommended pattern because it introduces an explicit control point, or (b) a P2P accelerator with native support for model repositories, such as Dragonfly. What the proxy cache does solve is mirroring the runtime base images (vLLM, TGI, TEI), which is no small thing: it creates a 7-day retention by default and the project is read-only.

Per-project quotas. In bytes, via the UI or the API:

curl -u admin:REDACTED -X PUT "https://registro.interno/api/v2.0/quotas/7" \
  -H "Content-Type: application/json" \
  -d '{"hard": {"storage": 21990232555520}}'

Those 21990232555520 bytes are 20 TiB, tight sizing, not generous, for a dozen models with three live versions each. Two warnings: the quota is checked when the manifest arrives, that is after uploading the blobs, so a push that exceeds it has already consumed disk and network; and the accounting does not always reflect deduplication of blobs shared between projects.

Immutable tags. Per-project rules that prevent overwriting a tag matching a pattern. For models it is not optional: a rule making everything under modelos/** with tag pattern 2* (date-based versions) immutable eliminates the entire class of “the tag changed underneath us” incidents.

Retention and GC. Per-project retention rules (last N versions, or those from the last N days) that mark artefacts for deletion, and a separate garbage collector that frees the unreferenced blobs. They are two different things, and deleting in the UI does not free disk: the weeding runs separately. Running GC with a concurrent push in flight is the classic recipe for deleting blobs that were about to be referenced.

Scanning. Trivy is integrated, with one caveat: scanning a weights artefact detects nothing relevant about the model. It is useful for the runtime images, which do execute code. Detecting dangerous formats, a PyTorch pickle with arbitrary code instead of safetensors, requires specific tools, and that is the subject of article 3/4.

Model registries proper

Now the catalogue, with the uncomfortable claim up front: a model registry does not solve distribution.

MLflow

MLflow (LF AI & Data) is the de facto catalogue standard, with a mature 3.x series; 3.13.0 was published in June 2026, focusing on RBAC, an administration UI and trace retention. Its data model: registered model (logical name) → model version (incrementing integer) → aliases and tags.

The important conceptual change: stages (Staging, Production, Archived) have been deprecated since MLflow 2.9.0 and are replaced by aliases and tags, because a fixed enumeration is too inflexible to express real MLOps flows. An alias is a named mutable pointer:

from mlflow import MlflowClient

client = MlflowClient()
client.set_registered_model_alias("llama-3.3-70b-soberano", "champion", 7)
client.set_model_version_tag("llama-3.3-70b-soberano", "7",
                             "estado_validacion", "aprobado_ens_medio")
client.set_model_version_tag("llama-3.3-70b-soberano", "7",
                             "oci_digest", "sha256:9f2c...")

Consumption happens through the alias URI: models:/llama-3.3-70b-soberano@champion.

What MLflow gives you: lineage back to the run that produced the model, evaluation metrics attached to the version, a shared promotion language and an API to hang approvals off. What it does not give you: it is not a CDN and it does not manage distribution to nodes, and its artifact store (S3/MinIO/NFS) provides neither immutability by digest, nor Referrers, nor cross-site replication. The healthy pattern is the one in the snippet: the catalogue stores the OCI digest as a tag on the version and points at the repository, instead of trying to be the repository.

Kubeflow Hub (formerly Model Registry)

The Kubeflow Model Registry was renamed Kubeflow Hub and unified two functions: the registry itself and a federated Model Catalog that discovers external models (YAML, Hugging Face), with lineage across data, code and models, metadata and state-based promotion.

The honest assessment: as of this article the component is still in the 0.3.x series and marked Alpha under Kubeflow’s versioning policy, with limited support. Interesting if you already run the full Kubeflow and want a single console; it is not yet the piece on which to build model governance for a sovereign platform in production. The rename in the middle of 0.3.x is, moreover, a sign that the API is still moving.

Comparison table by function

The thesis of the table: the columns are complementary, not alternatives. The OCI registry is the repository; MLflow, the catalogue; lakeFS or DVC version the dataset that everything originates from, as detailed in data versioning with DVC and lakeFS.

The hot path: from the registry to HBM

Image volumes: the change of the year

The image volume source (KEP-4639) mounts an OCI image or artefact directly as a read-only volume in a pod: no initContainer, no copying, no PVC. Its track record: alpha in v1.31 (August 2024), beta in v1.33 (April 2025) with subPath support, beta by default in v1.35 and GA in v1.36, released on 22 April 2026.

apiVersion: v1
kind: Pod
metadata:
  name: vllm-modelo-oci
spec:
  containers:
    - name: vllm
      image: registro.interno/runtime/vllm:v0.11.0
      args: ["--model", "/mnt/models", "--served-model-name", "llama-3.3-70b"]
      volumeMounts:
        - name: pesos
          mountPath: /mnt/models
          readOnly: true
      resources:
        limits:
          nvidia.com/gpu: "4"
  volumes:
    - name: pesos
      image:
        reference: registro.interno/modelos/llama-3.3-70b@sha256:9f2c...
        pullPolicy: IfNotPresent

Two details that change operations: the reference is by digest, so immutability stops depending on anyone’s discipline; and pullPolicy: IfNotPresent makes the node’s runtime cache the blobs in its local store, so the second replica on that node starts without touching the network. It is the coordinated cache the anti-pattern was missing.

KServe: storageUri and modelcars

In KServe there are two routes: the classic one, storageUri with a storage initializer that copies from S3, PVC, HTTP or GCS into an emptyDir; and the modelcar, with the oci:// scheme:

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: llama-70b-soberano
spec:
  predictor:
    model:
      modelFormat:
        name: huggingface
      storageUri: oci://registro.interno/modelos/llama-3.3-70b:2026.07.1
      resources:
        limits:
          nvidia.com/gpu: "4"

You have to understand the modelcar’s internal mechanism before depending on it: KServe sets shareProcessNamespace: true on the pod, starts a sidecar container with the model image and creates a symbolic link from /mnt/models to that process’s root filesystem through /proc. The runtime reads the weights without copying them, which is the saving being sought. Two warnings: it is not enabled by default (you have to turn on enableModelcar in the inferenceservice-config ConfigMap) and the latest tag forces pullPolicy: Always, cancelling out the cache. With image volumes now GA, the modelcar remains the route for clusters that are not yet on 1.36.

Loading fast: where each technique attacks

You have to separate two problems that get confused all the time: getting the bytes to the node (network) and getting them into HBM (disk → CPU → PCIe → GPU).

For the first, Dragonfly is the mature piece: it graduated in CNCF on 14 January 2026, distributes weights at the scale of hundreds of terabytes to hundreds of nodes “in minutes”, cuts the bandwidth consumed at the origin by up to 90 %, and makes use of between 70 % and 80 % of each node’s bandwidth. Its v2.5.0 (30 June 2026) added direct download from model repositories (dfget hf://...), P2P acceleration of Git LFS and a dragonfly-injector that injects P2P capability via a webhook without rebuilding images. Its subproject Nydus supplies the lazy-loading image format. There is a very common misplaced expectation here: for model weights lazy loading helps little, because an inference engine reads all the weights almost immediately; the real benefit is for fat runtimes and for models where only part is read.

For the second, the numbers published by the NVIDIA Run:ai Model Streamer on Llama-3-8B (15 GB, safetensors, a node with one A10G) are the cleanest reference available:

And the total time until vLLM is ready to serve: 66.13 s with the standard loader over gp3 versus 35.08 s with the streamer. Two critical readings. First: these are the project’s own benchmarks, not an independent measurement, on a 15 GB model on a modest GPU; extrapolating linearly to 140 GB on 4×H100 is a leap nobody has published. Second: the gain comes from saturating the storage medium with concurrency. If the bottleneck is an NFS at 1 Gb/s, no loader fixes it. The detail of this leg is in speeding up cold start with tensorizer and in from disk to HBM.

As the diagram shows, on a reference node of 4×H100 SXM 80 GB with NVLink each leg is between 5 and 10 times faster than the previous one. The conclusion is boring: the money is in having the model already on the node’s NVMe before the pod starts.

LoRA adapters: when the artefact weighs megabytes

Everything above assumes artefacts of tens or hundreds of gigabytes. A LoRA adapter breaks that assumption: for a 70B base, a rank-16 adapter over the attention projections weighs tens or a few hundred megabytes, three orders of magnitude less, and that changes the whole strategy.

With artefacts like that, the local cache and P2P stop mattering: the download is instant at any reasonable bandwidth. What starts to matter is cardinality and rate of change. A multi-tenant platform like the one described in multi-LoRA serving can have hundreds of adapters, each with its owner and its access policy, over a handful of bases. The practical consequences:

• The adapter is an OCI artefact in its own right, with its own repository and an immutable tag. But the registry then has thousands of small repositories instead of tens of large ones, which stresses metadata, listings and granular RBAC, not disk.
• It must declare its base. An adapter with no immutable reference to the digest of the base model it was trained on is a time bomb: applied over another version, it degrades quality silently. It is the natural use case for subject: the adapter refers to the base, and the Referrers API answers “which adapters exist for this digest”.
• Loading is dynamic, not at start-up. vLLM allows hot loading of adapters with VLLM_ALLOW_RUNTIME_LORA_UPDATING and the /v1/load_lora_adapter endpoint, or declaratively with the LoRA resolver plugins (VLLM_PLUGIN_LORA_RESOLVERS), which resolve an adapter by name from a directory. The clean pattern on Kubernetes is to mount an image volume with the adapter and let the filesystem resolver discover it.
• Versioning becomes the dominant problem. With 300 live adapters, knowing which adapter serves tenant X, over which base and trained with which dataset, only has an answer if the catalogue exists. Here a model registry does beat convention over tags.

Air-gap and sovereignty: promotion without losing traceability

The strict sovereign case: a connected environment where you ingest and validate, and an isolated environment where you serve, with no network route between them. Promotion crosses a data diode or a physical medium, and traceability has to survive the journey. It is the archive’s certified reproduction: the document, its record card and the seal certifying that the copy is faithful all travel together.

# 1. In the connected environment: export the artefact with EVERYTHING that hangs off it
oras backup \
  --output /transfer/llama-3.3-70b-2026.07.1.tar \
  --include-referrers \
  registro-dmz.interno/modelos/llama-3.3-70b:2026.07.1

# 2. Verify the exact digest before crossing
oras manifest fetch --descriptor \
  registro-dmz.interno/modelos/llama-3.3-70b:2026.07.1 > /transfer/descriptor.json
sha256sum /transfer/llama-3.3-70b-2026.07.1.tar > /transfer/SHA256SUMS

# 3. Physical crossing or data diode

# 4. In the isolated environment: restore preserving digests and referrers
oras restore \
  --input /transfer/llama-3.3-70b-2026.07.1.tar \
  registro-aislado.interno/modelos/llama-3.3-70b

The --include-referrers flag is what makes this worth anything: without it the weights cross, but the signature, the AIBOM and the attestations stay behind, and the isolated environment receives a model indistinguishable from one downloaded by hand. oras backup is marked experimental in v1.3.0, which has to be accepted explicitly before turning it into a procedure; the consolidated alternative for pure copying is skopeo copy with the oci-archive format, at the cost of managing the referrers separately.

What must travel attached to the artefact so that article 3/4 can verify it: the digest of the artefact and of its blobs; the provenance of the origin (the repository and the specific revision, not the model’s name, plus who ingested it and when); the identity of the process that packaged it (pipeline, commit, run), the raw material for an SLSA attestation; the cross-references to the evaluation dataset and to the base model if it is a derivative; and the promotion state with its approver, which in an isolated environment cannot be queried against the connected environment’s MLflow. All of it fits in OCI annotations and in artefacts referred to via subject. The rule: the isolated environment must be able to verify without calling anyone; if verification needs an external query, it is not air-gap.

Decision map

An OCI registry and a convention are enough when you have fewer than twenty models or so, a single team decides what gets promoted, lineage back to training is not a regulatory requirement (typical if you consume third-party models) and you already operate Harbor. Here a model registry adds one more console to maintain and no answer that immutable tags and annotations were not already giving. It is the majority case in pure inference.

A model registry is needed if you train or fine-tune and need auditable lineage from model to dataset and to code, the continuous fine-tuning cycle; if several teams compete to promote versions and an explicit approval flow is needed; if you have to answer an ISO 42001 or EU AI Act auditor about which version was in production on a given date; or if you manage tens of adapters with different owners.

The registry becomes valueless bureaucracy when the model is registered after being deployed (the catalogue as an administrative act, not as a gate); when the promotion state is not connected to any technical control and an unapproved version can be deployed; or when a model registry and an OCI registry coexist with two truths and no digest joining them. The last one is the most common failure and the most expensive: two versioning systems that diverge in silence.

Operational traps

Mutable tags and latest in production. Beyond style, a mutable tag turns any audit into guesswork and lets two replicas of the same Deployment serve different bytes. On top of that, in KServe latest forces pullPolicy: Always and cancels the node cache, so you pay in cold start what you lose in traceability. Rule: immutable tags by project policy and deployment by digest.

GC that deletes referenced layers. The archive’s weeding. The garbage collector identifies blobs that no manifest references, and the window between “I have uploaded the blobs” and “I have uploaded the manifest” is exactly where a concurrent GC can take them away. In Harbor this has historically shown up in _uploads and in discrepancies between the space the UI claims to have freed and the real figure. Mitigation: GC in a window with pushes blocked, and do not trust the first pass.

The registry’s disk and the cost of versioning. Brutal arithmetic: 140 GB per version with three live versions is 420 GB; twelve models like that, 5 TB; with six-version retention and two replicated sites, 20 TB. And it is expensive-class storage if the registry lives on replicated block. Deduplication helps little, because the weights change completely between versions. The retention policy has to be designed before filling the registry, and models must be separated from runtime images by project, because their retentions have nothing to do with each other.

Replication that saturates the inter-site link. An event-driven rule on a model repository moves 140 GB every time someone pushes a version: on a shared 1 Gb/s link, almost twenty minutes flat out in the middle of the working day. Schedule it in a window, limit bandwidth, filter by label which versions get replicated and use P2P inside each site.

Putting the model inside the runtime image. Tempting because “that way there is only one artefact”. The result is a 145 GB image that has to be rebuilt and redistributed every time a runtime CVE is patched, with the security life cycle coupled to the model’s. Separated: a small, patchable runtime and a large, stable model.

For an inference factory

Three actionable things for anyone running an on-premise inference factory, not a training one.

First: cut the internet dependency at start-up, this week. You do not need a platform project: an ingestion task that pulls each model once, pushes it to Harbor as an OCI artefact with an immutable tag and provenance annotations, plus switching the deployments to reference by digest, is enough. The HF_TOKEN disappears from the pod and moves to the ingestion pipeline, which is where it makes sense and where it fits with secrets hardening.

Second: decide consciously whether you need a catalogue. If you only consume third-party models and do not train, probably not: the OCI registry with a convention gives 90 % of the value at 10 % of the operational cost. If you train, fine-tune or manage per-tenant adapters, set up MLflow and store the OCI digest as a tag on each version, so that catalogue and repository cannot diverge.

Third: move the bottleneck to the right place. Before investing in load accelerators, measure the three legs. If the model arrives over the network on every start-up, the work is a local NVMe cache, image volumes with pullPolicy: IfNotPresent and, with many nodes, P2P. Only when the model is already on the node’s disk does it make sense to wrestle with loader concurrency.

So much for where the bytes come from and by what route they arrive. What remains is what we have taken for granted all along: why trust those bytes. A digest guarantees integrity, that nobody has changed them, but not provenance: it does not say who produced them, with what data or whether somebody with authority certified that they could be served. The third article in the series builds that answer with Sigstore, SLSA, in-toto and AIBOM, hanging precisely off the subject and the Referrers API that we have left ready here.

See also

• Model chain of trust (1/4): KServe and the Open Inference Protocol — the control plane that consumes the storageUri described here.
• Model chain of trust (3/4): signing, provenance and AIBOM — what hangs off the digest via subject and the Referrers API.
• Speeding up cold start: model loading with tensorizer — the disk-to-HBM leg in detail.
• From disk to HBM: anatomy of the cold start — where the time goes inside the node.
• Data versioning with DVC and lakeFS — the equivalent plane for datasets, which feeds the catalogue’s lineage.
• Catalogue of OSS LLMOps tools — where each piece of this article fits in the complete stack.
• Multi-LoRA serving: fundamentals — the case where the artefact weighs megabytes and the strategy changes.

Sources

• Open Container Initiative, OCI Image and Distribution Specs v1.1 Releases (13 March 2024) — https://opencontainers.org/posts/blog/2024-03-13-image-and-distribution-1-1/
• CNCF, Announcing ORAS v1.3.0: Elevating artifact and registry management workflows (6 October 2025) — https://www.cncf.io/blog/2025/10/06/announcing-oras-v1-3-0-elevating-artifact-and-registry-management-workflows/
• CNCF, ORAS project page (sandbox since 13 July 2021) — https://www.cncf.io/projects/oras/
• ORAS, oras backup (experimental) — https://oras.land/docs/commands/oras_backup
• ModelPack, CNCF ModelPack Specification — model spec (artifactType and mediaTypes) — https://github.com/modelpack/model-spec
• ModelPack, modctl — getting started (Modelfile and commands) — https://github.com/modelpack/modctl/blob/main/docs/getting-started.md
• CNCF, Cloud Native Computing Foundation announces Harbor graduation (23 June 2020) — https://www.cncf.io/announcements/2020/06/23/cloud-native-computing-foundation-announces-harbor-graduation/
• Harbor, Configure proxy cache (supported registries) — https://goharbor.io/docs/2.6.0/administration/configure-proxy-cache/
• OneUptime, How to configure Harbor project quotas for storage limits (9 February 2026) — https://oneuptime.com/blog/post/2026-02-09-harbor-project-quotas-storage/view
• endoflife.date, Harbor releases (2.15.2 on 2 July 2026; 2.15 on 20 March 2026) — https://endoflife.date/harbor
• VMware Cloud Foundation Blog, Using Harbor as an AI Model Registry (3 March 2026) — https://blogs.vmware.com/cloud-foundation/2026/03/03/using-harbor-as-an-ai-model-registry/
• MLflow, Model Registry workflows — stages deprecated since MLflow 2.9.0 — https://mlflow.org/docs/latest/ml/model-registry/workflow/
• MLflow, Release 3.13.0 highlights — https://mlflow.org/releases/3.13.0/
• Kubeflow, Hub (formerly Model Registry) — overview — https://www.kubeflow.org/docs/components/hub/overview/
• kubernetes/enhancements, KEP-4639: OCI VolumeSource (alpha 1.31, beta 1.33/1.35, GA 1.36) — https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/4639-oci-volume-source/README.md
• PerfectScale, Kubernetes v1.36 release: OCI artifact volume support reaches GA (22 April 2026) — https://www.perfectscale.io/blog/kubernetes-v1-36-release
• KServe, OCI storage (modelcar) — https://kserve.github.io/website/docs/model-serving/storage/providers/oci
• CNCF, CNCF announces Dragonfly’s graduation (14 January 2026) — https://www.cncf.io/announcements/2026/01/14/cloud-native-computing-foundation-announces-dragonflys-graduation/
• CNCF, Dragonfly v2.5.0 is released (30 June 2026) — https://www.cncf.io/blog/2026/06/30/dragonfly-v2-5-0-is-released/
• NVIDIA Run:ai, Model Streamer benchmarks — https://github.com/run-ai/runai-model-streamer/blob/master/docs/src/benchmarks.md
• vLLM, LoRA adapters (dynamic loading and resolver plugins) — https://docs.vllm.ai/en/stable/features/lora/