Model chain of trust (3/4): signature, provenance and AIBOM — why you trust those bytes

· By José Manuel Román
Contents

In the first article of this series we built the serving control plane; in the second, where the bytes come from. At the end of that route there is an initContainer that runs oras pull on a 140 GB artefact and leaves it on a volume vLLM is going to read. The uncomfortable question is the one that gives this third article its title: why you trust those bytes.

The usual answer (“because they come from our registry”) is the one the supply-chain discipline has spent a decade dismantling: a registry is a warehouse, not an authority; it keeps whatever is pushed to it, and whoever can push can poison. Here we cover the four pieces that turn that sentence into something a machine verifies before the pod starts: signing (Sigstore), model signing (OpenSSF Model Signing), provenance (SLSA and in-toto) and a bill of materials (AIBOM). And the part that almost never gets told: what each one proves.

TL;DR

  • Model artefact: third-party code with access to your data. pickle executes code in torch.load; safetensors removes that class of attack, but says nothing about whether the weights are poisoned.
  • Sigstore is the mature piece: cosign 3.1.1 (9 June 2026), Rekor v2 GA since 10 October 2025. In air-gap, your own PKI or a private Sigstore; the new bundle allows offline verification.
  • OpenSSF Model Signing (OMS) v1 (June 2025) signs a manifest of hashes for the whole directory, because a servable model is a directory and not a file; the reference library, model_signing 1.1.1, is in the OpenSSF sandbox.
  • SLSA v1.2 (24 November 2025) adds the Source track; Build L3 is the realistic target in fine-tuning. in-toto graduated in the CNCF on 23 April 2025.
  • AIBOM: SPDX 3.0.1 and CycloneDX 1.7 (ECMA-424 2nd ed.) have the fields; the tooling that fills them in automatically barely exists. It is emitted from the pipeline, not from a scanner.
  • Regulation: Art. 53 of the AI Act binds GPAI providers from 2 August 2025, and Regulation (EU) 2026/1744 pushed high risk back to December 2027 and August 2028 without touching GPAI. The CRA requires an SBOM, not an AIBOM, with full application from 11 December 2027.
  • Limit: none of this says the model is good. Ultralytics produced valid attestations of artefacts with a cryptominer inside.

The analogy: batch traceability

A batch of ham arriving at a supermarket’s loading bay brings four things that are easy to confuse. A batch number, which identifies it and no other: the sha256. A seal from the abattoir, which if broken gives away tampering in transit: the signature. A health register saying which abattoir it came from, on what date and under whose inspection: the provenance attestation. And an ingredients and allergens label: the AIBOM. At the bay, the supervisor does not accept the pallet if the seal is broken or the register is missing, and that is admission control.

The analogy holds down to the detail: the seal travels stuck on the outside and not inside the ham, because it is a detached signature; the register is issued by the abattoir and not by the carrier, so its value depends on the inspection being real. What lets you recall product when a problem shows up is the batch number, not the seal.

But it serves above all for what it does not promise: a ham with an intact seal and an impeccable label can be off. Traceability answers “where did it come from and did anybody touch it”; it does not answer “is it good”. This article is about the first question; the second is the job of evaluation and of guardrails, and confusing the two is what leads to treating as safe a model you only know the origin of.

PipelineQLoRA / mergeephemeral runnerSealinghash manifestsignature (OMS / cosign)SLSA provenanceAIBOMOCI registryartefact + referrersimmutable by digestLoading bayadmission controlverifies signaturerequires provenancefail-closedGPUre-verifiesbefore loadingThe seal is applied once, at the source; it is checked twice: at the bay and before loading.Traceability does not say the model is good. It says where it came from and whether anyone touched it.

The threat model, without generalities

pickle, and what safetensors solves

The problem with pickle is not a bug, it is its semantics: an object can define __reduce__ and on deserialisation Python executes whatever that method returns. A torch.save checkpoint is a program that on loading can open a reverse shell. This is not theoretical: in February 2024 JFrog documented around a hundred malicious models on Hugging Face with a real payload (PyTorch with an injected __reduce__, Keras abusing the Lambda layer), and in February 2025 ReversingLabs described nullifAI, two models with deliberately corrupted pickle files that evaded Picklescan by exploiting the fact that the payload runs before the file fails. The response came late: PyTorch 2.6 flipped the default value of weights_only to True, restricting deserialisation to an allowlist of types. It broke a lot of code (there are open issues in nnUNet, accelerate and half a dozen more projects) and it is not a formal guarantee: that allowlist has had documented escapes.

safetensors does solve the whole class: pure data, a JSON header with offsets plus a blob of tensors, with no execution possible. Trail of Bits’ audit for EleutherAI and Hugging Face, on 23 May 2023, found no critical flaw leading to arbitrary code execution, and as a bonus the direct memory mapping gives loads on the order of a hundred times faster on CPU, something we already exploited in the route from disk to HBM.

The nuance that is systematically forgotten: safetensors guarantees that loading the file does not execute code, and guarantees nothing about the numbers inside. A model with a trained backdoor, normal except in the face of a specific trigger, is distributed in safetensors and sails past the scanners and the signature. It is the sealed ham that is off.

LoRA adapters, mutable tags and Friday’s model

An adapter weighs megabytes, gets shared casually and in a multi-LoRA stack dozens are hot-loaded on the same base: it is the cheapest vector for the attacker and the one with the least governance. The 2026 literature is dense on detection: arXiv 2602.15195, revised in April 2026, reports a ROC-AUC of 1.00 classifying adapters by spectral statistics of the attention projections. But with a cool head these are adapters poisoned by the authors themselves with a known method: live research, not deployable defence. The realistic operational answer is still where the adapter came from and who signed it.

And two threats with no attacker. A tag is mutable: mi-registro/llama-70b:produccion can point to one digest today and another tomorrow without a line of your manifests changing, and one well-meaning oras push is enough; the mitigation is identical to the one for images in stack hardening, pinning by digest and never by tag, with the detail that the digest is what the signature covers. And the classic: an engineer converts a checkpoint on their laptop, runs oras push with their credentials and on Monday there is an InferenceService serving it, with no signature, provenance or AIBOM. It is not malicious; it is ungovernable. Here the signature helps because it makes the shortcut unworkable.

Ultralytics: valid signature, poisoned artefact

On 4 and 5 December 2024, versions 8.3.41 and 8.3.42 of ultralytics (YOLO) were published on PyPI with a cryptominer, through poisoning of the GitHub Actions cache in the publishing workflow; days later a second round arrived (8.3.45 and 8.3.46) with API tokens that had not been rotated when migrating to Trusted Publishing. All four malicious versions carried valid attestations, because the workflow itself generated them. The signature was correct and so was the provenance; what was compromised was the build environment. As the researcher who uncovered it summed up, an attestation guarantees the relationship between an artefact and a build job plus a commit, not between the developer’s intentions and the final artefact.

Sigstore: who signed, and how you check without calling anybody

Sigstore is three pieces and a client. Fulcio is a CA that, in exchange for a valid OIDC token, issues a very short-lived X.509 certificate bound to that identity. Rekor is an append-only transparency log where the signature is recorded with its timestamp. cosign orchestrates it. The result is keyless signing: instead of a long-lived private key to look after, there is an OIDC identity (the CI workflow) and an immutable record that this identity signed that digest.

Status as of July 2026: cosign 3.1.1, from 9 June 2026. The 3.x branch, since 8 October 2025, enabled the new bundle format by default, along with --trusted-root and --use-signing-config; the practical consequence is that the bundle carries the verification material inside and allows verification without calling Rekor or Fulcio. Rekor v2 has been GA since 10 October 2025: a reimplementation over tiles that makes running your own log far cheaper, because Trillian log server and log signer are switched off and reads are cached in a CDN, at the price of cutting down to two entry types (hashedrekord and dsse) and going without a search index. It requires cosign 3.0.1+ or 2.6.0+; v1 continues in parallel and its freeze will be announced a year in advance.

Signing and verifying a model artefact in OCI

On the artefact from the second article, the operation is identical to that for an image: for the registry, a model artefact is an OCI object with its digest.

# Resolve the tag to a digest ONCE, in the pipeline. Sign the digest, never the tag.
oras resolve registro.interno/modelos/llama-70b-fin:v7
#  -> sha256:9f2a4c1e77b0d3a8...

# Keyless signature from CI (the OIDC token comes from the runner)
cosign sign --yes registro.interno/modelos/llama-70b-fin@sha256:9f2a4c1e...

# Verification: exact workflow identity, exact issuer
cosign verify \
  --certificate-identity-regexp '^https://git.interno/plataforma/modelos/.*@refs/heads/main' \
  --certificate-oidc-issuer 'https://git.interno' \
  registro.interno/modelos/llama-70b-fin@sha256:9f2a4c1e...

The most frequent mistake is omitting --certificate-identity and --certificate-oidc-issuer, or setting them so loosely that they accept any identity from the provider. Without those two flags narrowed down, verification checks that somebody signed, not that the right party signed: the difference between “it has a seal” and “it has our abattoir’s seal”.

The air-gap problem

Keyless needs a reachable OIDC when signing and, in the classic model, a reachable Rekor when verifying. In a disconnected datacenter, three strategies:

StrategyWhat you gainWhat it costs
Own key / PKI (cosign sign --key)Works without a network; fits HSM and corporate PKICustody and rotation of long-lived keys comes back; without a transparency log there is no detection of unauthorised signing
Private Sigstore (internal Fulcio + Rekor v2 + OIDC)Full keyless and real transparency inside the perimeterThree more services to operate, with their TUF root and their rotation cycle
New bundle + offline verificationVerification without a network using --offline and a local --trusted-rootTransporting and keeping the trust root up to date by sneakernet
cosign verify --offline=true \
  --trusted-root /etc/sigstore/trusted_root.json \
  --certificate-identity-regexp '^https://git.interno/plataforma/modelos/.*' \
  --certificate-oidc-issuer 'https://git.interno' \
  registro.interno/modelos/llama-70b-fin@sha256:9f2a4c1e...

Two warnings. --insecure-ignore-tlog exists and gets used a lot in air-gap, but its name does not lie: it disables the inclusion check against the transparency log, which is exactly what distinguishes Sigstore from a conventional PKI; if you use it, you are doing certificate signing, not Sigstore. And there is documented friction: cosign issue 4550 describes 3.0.2 trying to reach the TUF CDN despite having a local key and access only to an internal Nexus. Air-gap works, but it is not the happy path.

Signing a model is not signing a file

A servable model is a directory: several safetensors shards, config.json, tokenizer, perhaps a chat_template.jinja. Changing config.json (the context length, the rope_scaling) alters behaviour without touching a single weight. Signing only the weights leaves the door open.

The answer is the OpenSSF Model Signing (OMS) specification, published in June 2025 with contributions from Google, HiddenLayer, NVIDIA, Red Hat, Intel, Meta, IBM and Microsoft. Its design is a detached signature, which neither modifies nor repackages the content, over a manifest listing every file by its hash (SHA-256 by default, BLAKE2b as an alternative), with one signature covering the whole manifest. It is deliberately PKI-agnostic: it accepts bare keys, self-signed certificates, corporate PKI or keyless Sigstore. The reference implementation is model_signing, from sigstore/model-transparency, version 1.1.1 of 10 October 2025; 1.1.0 added PKCS#11 (HSM), private Sigstore instances, BLAKE3 and OpenTelemetry traces.

# Sign the whole directory: weights, config and tokenizer
model_signing sign /modelos/llama-70b-fin --signature /modelos/llama-70b-fin/model.sig

# Verification with a narrowed OIDC identity
model_signing verify /modelos/llama-70b-fin \
  --signature /modelos/llama-70b-fin/model.sig \
  --identity 'https://git.interno/plataforma/modelos/.github/workflows/publicar.yml@refs/heads/main' \
  --identity-provider 'https://git.interno'

The payload is a DSSE envelope with an in-toto statement whose subjects are the path-digest pairs of each file and whose predicateType is https://model_signing/signature/v1.0; support for file shards allows hashing in chunks. Maturity without embellishment: the project is in the OpenSSF sandbox. The specification is solid and has the vendors that matter behind it, with NVIDIA signing models in NGC with it, but it is the first formal step and there are few independent verifiers beyond the reference CLI.

The arithmetic of hashing, which turns out not to be the problem

What remains is how long it takes to hash a model of hundreds of gigabytes before every start. Let \( S \) be the total size, \( p \) the threads running in parallel over different shards, \( r_{\text{cpu}} \) the hashing throughput per thread and \( r_{\text{io}} \) that of reading from storage:

$$t_{\text{hash}} = \frac{S}{\min\left(p \cdot r_{\text{cpu}},\ r_{\text{io}}\right)}$$

With a dense 70B model in bf16, some 140 GB across thirty-odd shards, SHA-256 accelerated by SHA-NI runs at around 1.5-2 GB/s per thread. With eight threads the CPU term is around 14 GB/s, well above what a Gen4 NVMe delivers, on the order of 6 GB/s. The minimum is set by the disk, not by the hash: about 23 seconds. And that is the same throughput the loader is going to consume anyway to move the weights to HBM: if verification happens in the same initContainer that already reads the artefact, the marginal cost is CPU, not I/O. What is expensive is having put verification in a separate step that reads the disk twice (the rest of the startup budget is in the cold start post). What does have a real cost is incremental verification: if only one shard changes, the manifest allows checking only that one, but only if the distribution layer does incremental pulls. With an OCI artefact and granular layers it works; with a monolithic tarball, it does not.

Provenance: SLSA and in-toto over a QLoRA pipeline

The signature says who published; provenance says how it was produced. The framework is SLSA, whose v1.2 was published on 24 November 2025, backwards compatible with v1.1. Its novelty is the Source track; the Build Environment track and the Dependency track are still in development, and the first is exactly the one that would have helped with Ultralytics.

LevelWhat SLSA requiresWhat it means in a QLoRA pipeline
Build L0NothingThe adapter somebody trained on their workstation
Build L1Automatically generated provenance: who built it, with what process and with what top-level inputs. It may be unsignedThe job emits provenance with commit, dataset and hyperparameters. It catches mistakes, not attacks
Build L2The above plus a build on a hosted platform that generates and signs the provenance, verifiable by the consumerThe CI runner signs with its identity. Now there is something an admission controller can verify
Build L3The above plus isolation between runs and signing material inaccessible from user-defined stepsAn ephemeral runner per job, with no signing secrets within reach of the training script. This is the realistic target

in-toto is the format in which all of this is expressed: it graduated in the CNCF on 23 April 2025, which in practice makes it the common substrate of SLSA, of cosign attest and of OMS itself. Over the flow from the QLoRA runbook and from continuous fine-tuning, the useful attestation is the one that allows reproduction and auditing:

{
  "_type": "https://in-toto.io/Statement/v1",
  "subject": [{ "name": "adapter-soporte-v7",
                "digest": { "sha256": "e3b0c44298fc1c149afbf4c8..." } }],
  "predicateType": "https://slsa.dev/provenance/v1",
  "predicate": {
    "buildDefinition": {
      "buildType": "https://interno/build-types/qlora-finetune@v2",
      "externalParameters": {
        "modeloBase": "registro.interno/modelos/llama-70b@sha256:9f2a...",
        "dataset": "lakefs://corpus-soporte@commit-4c1f9e",
        "hiperparametros": { "rank": 16, "alpha": 32, "lr": 0.0002,
                             "epochs": 3, "quant": "nf4" }
      },
      "internalParameters": {
        "gpu": "4xH100-SXM-80GB", "cuda": "12.6", "torch": "2.8.0",
        "imagenEntrenamiento": "registro.interno/ci/qlora@sha256:71ca..."
      },
      "resolvedDependencies": [
        { "uri": "git+https://git.interno/plataforma/finetune@a91c3f",
          "digest": { "gitCommit": "a91c3f..." } }
      ]
    },
    "runDetails": {
      "builder": { "id": "https://git.interno/plataforma/runners/gpu-efimero" },
      "metadata": { "invocationId": "run-2026-07-19-0041",
                    "startedOn": "2026-07-19T02:14:33Z" }
    }
  }
}

The five fields you need and that are almost never complete: hash of the base model, immutable identifier of the dataset (here a lakeFS commit, in line with data versioning), hyperparameters, hardware and stack versions, and the code commit. Without the first the chain back to the original model cannot be rebuilt; without the second, the retrain loop is left without reproducible evidence. And the obligatory warning: SLSA says nothing about the quality of the dataset; Build L3 guarantees that nobody tampered with the build, not that the corpus was clean.

AIBOM: SPDX 3.0.1 versus CycloneDX 1.7

If the manifest is the seal and provenance the health register, the AIBOM is the ingredients label. There are two formats, and the honest comparison is less flattering than the vendor literature suggests. SPDX 3.0 (April 2024), with patch 3.0.1 on 17 December 2024, reorganised the specification into profiles, two of them relevant here: AI and Dataset. The AI profile defines on AIPackage fields such as typeOfModel, hyperparameter, informationAboutTraining, metric, safetyRiskAssessment and limitation, plus four energy-consumption fields, unique to it, broken down into training, fine-tuning and inference. v3.1 has been in RC since January 2025 without GA, and 3.0 is going through ISO as ISO/IEC DIS 5962 (the standard in force is still 2.2.1). CycloneDX 1.7 was published on 21 October 2025 and ratified as ECMA-424, 2nd edition, in December 2025; its ML-BOM rests on the machine-learning-model type and the modelCard object.

CriterionSPDX 3.0.1CycloneDX 1.7
Formal standardisationISO/IEC DIS 5962 in progress; the standard in force is 2.2.1ECMA-424 2nd ed. (December 2025)
AI modelAI profile over AIPackagemachine-learning-model + modelCard
DatasetDedicated Dataset profiledata and modelParameters.datasets
EnergyFour broken-down fields (unique)Not natively covered
Intended use and ethicslimitation, safetyRiskAssessmentconsiderations (closer to Model Cards)
ErgonomicsRich model, verbose, steep curveCompact JSON, more adoption in tooling
Fit with VEXVia the Security profileNative, aligned with CSAF VEX 2.0
{
  "bomFormat": "CycloneDX", "specVersion": "1.7", "version": 1,
  "components": [{
    "type": "machine-learning-model",
    "bom-ref": "modelo/adapter-soporte-v7",
    "name": "adapter-soporte", "version": "7.0.0",
    "hashes": [{ "alg": "SHA-256", "content": "e3b0c44298fc1c14..." }],
    "licenses": [{ "license": { "id": "Apache-2.0" } }],
    "modelCard": {
      "modelParameters": {
        "task": "text-generation",
        "modelArchitecture": "llama-70b + LoRA r=16",
        "datasets": [{ "type": "dataset", "name": "corpus-soporte",
                       "contents": { "url": "lakefs://corpus-soporte@commit-4c1f9e" } }]
      },
      "quantitativeAnalysis": {
        "performanceMetrics": [
          { "type": "exactitud-eval-interna", "value": "0.83", "slice": "soporte-es" }
        ]
      },
      "considerations": {
        "useCases": ["asistencia interna a agentes de soporte"],
        "technicalLimitations": ["no evaluado fuera de castellano"]
      }
    }
  }]
}

What gets filled in on its own is little. Automated tooling gives you name, version, hashes, licences and dependencies of the inference environment: that is a classic SBOM and Trivy or Syft do it unaided. From the pipeline, because no scanner infers it from the bytes, must come dataset, hyperparameters, metrics, base model and energy. And from a person: limitations, intended uses and ethical considerations. The conclusion is uncomfortable: an AIBOM generated after the fact by a tool that looks at the artefact is, for the most part, an empty document with a valid structure. The useful AIBOM is emitted by the training job, the only one that knows those fields, and is signed along with the artefact.

Regulation: what binds, and from exactly when

The dates call for precision, because they changed in 2026 and much of the documentation in circulation is out of date.

EU AI Act

The obligations on providers of general-purpose AI models (GPAI) apply from 2 August 2025. Article 53 requires technical documentation in line with Annex XI, information for downstream providers in line with Annex XII, a copyright policy and a public summary of the training content. Annex XI, section 1, explicitly asks for the specifications of the training process, information on the data used for training, testing and validation, including its type and provenance, the computational resources employed (for example, floating-point operations), the training time and the known or estimated energy consumption. Anyone who had placed a model on the market before that date has until 2 August 2027.

The 2026 change: Regulation (EU) 2026/1744, the AI “Digital Omnibus”, was published in the Official Journal on 24 July 2026 and came into force on 27 July 2026. Chapter III for Annex III high risk moves from 2 August 2026 to 2 December 2027; Annex I high risk (AI as a safety component of regulated products), to 2 August 2028; on 2 December 2026 new prohibitions under Article 5 and machine-readable marking for GPAI come into force. The GPAI obligations of Articles 51 to 56 are not delayed.

The reading for an architect: if you fine-tune a model and place it on the market, Annex XI is enforceable now, and its fields are almost literally those of a SLSA attestation plus an AIBOM. If your case is high risk you have a year and a half more headroom than you thought, but that headroom is for the harmonised standards, not for starting late (detail in the AI Act post).

Cyber Resilience Act

The CRA came into force on 10 December 2024; the notification obligations of Article 14 apply from 11 September 2026 and full application from 11 December 2027. What it requires, in Annex I, Part II, point 1, is a software bill of materials in a commonly used, machine-readable format “covering at the very least the top-level dependencies”.

Three nuances that get cited badly. “At least the top-level dependencies” is a very low floor: it does not require the transitive tree. The SBOM does not have to be public, and recital 77 says so expressly: it is internal documentation that the authorities may require. And the CRA names no format; the Commission reserves the right to specify one by implementing act, and the most precise guidance today is BSI TR-03183-2, which accepts CycloneDX 1.6+ or SPDX 3.0.1+. The CRA talks about SBOM and not AIBOM; that the model artefact falls under its umbrella is interpretation, not text.

Fit with ENS and ISO/IEC 42001

None of this is voluntary hygiene. The signature and its verification at admission implement op.exp.6 and op.ext.3 of the ENS (supply chain) and control A.10 of ISO/IEC 42001; the provenance attestation covers op.exp.2 and A.6 (AI system life cycle); the AIBOM with dataset and hyperparameters is op.exp.1 (inventory) and A.7 (data); and the transparency log, op.exp.8. The evidence an ISO/IEC 42001 auditor asks for on the life cycle is, to a large extent, the same JSON the pipeline emits; the control-by-control breakdown is in the technical controls post.

Verification in the cluster: the receiving bay

None of the above is worth anything if nobody checks the seal at the door. Kyverno graduated in the CNCF on 16 March 2026; its 1.17 (February 2026) promoted CEL policies to v1, including ImageValidatingPolicy, and marked ClusterPolicy as deprecated, and 1.18 (24 April 2026) polished image verification. The alternative, Sigstore’s policy-controller, is at v0.15.1 (26 March 2026), which moved from cosign v2 to v3 and migrated to go-tuf v2, relevant for anyone running a private Sigstore with delegated roles.

apiVersion: policies.kyverno.io/v1
kind: ImageValidatingPolicy
metadata:
  name: modelos-firmados-y-con-provenance
spec:
  validationActions: [Deny]        # fail-closed: it does not admit, it does not audit
  failurePolicy: Fail              # if the webhook fails, it is denied
  webhookConfiguration:
    timeoutSeconds: 20             # pulling the signature can take a while
  matchConstraints:
    resourceRules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        resources: ["pods"]
        operations: ["CREATE", "UPDATE"]
  matchImageReferences:
    - glob: "registro.interno/modelos/*"
  attestors:
    - name: ci-plataforma
      cosign:
        keyless:
          identities:
            - issuer: "https://git.interno"
              subject: "https://git.interno/plataforma/modelos/.github/workflows/publicar.yml@refs/heads/main"
        ctlog:
          url: "https://rekor.interno"
  attestations:
    - name: slsaProvenance
      intoto:
        type: "https://slsa.dev/provenance/v1"
  validations:
    - expression: >-
        images.containers.map(image,
          verifyImageSignatures(image, [attestors.ci_plataforma])).all(e, e > 0)        
      message: "Artefacto de modelo sin firma valida de la CI de plataforma"
    - expression: >-
        images.containers.map(image,
          verifyAttestationSignatures(image, attestations.slsaProvenance,
            [attestors.ci_plataforma])).all(e, e > 0)        
      message: "Falta atestacion SLSA de procedencia"

Verify the model, not just the image

That policy covers the serving container’s image. The model is downloaded afterwards, in the initContainer, where the admission controller no longer reaches. The pattern that works is to verify there, before writing to the shared volume:

initContainers:
  - name: traer-y-verificar-modelo
    image: registro.interno/plataforma/oras-cosign@sha256:4b7e...
    command: ["/bin/sh", "-ec"]
    args:
      - |
        oras pull registro.interno/modelos/llama-70b-fin@sha256:9f2a... -o /modelos
        cosign verify --offline=true \
          --trusted-root /etc/sigstore/trusted_root.json \
          --certificate-oidc-issuer https://git.interno \
          --certificate-identity-regexp '^https://git.interno/plataforma/modelos/.*' \
          registro.interno/modelos/llama-70b-fin@sha256:9f2a...
        model_signing verify /modelos --signature /modelos/model.sig \
          --identity-provider https://git.interno \
          --identity 'https://git.interno/plataforma/modelos/.github/workflows/publicar.yml@refs/heads/main'        
    volumeMounts:
      - { name: modelos, mountPath: /modelos }
      - { name: sigstore-root, mountPath: /etc/sigstore, readOnly: true }

The double verification is not redundant: cosign verify covers the OCI artefact as it stands in the registry, and model_signing verify covers the unpacked content as the engine is going to read it. If somebody mounts a ConfigMap that overwrites config.json after the pull, the first passes and the second does not.

Fail-closed versus fail-open, and what it costs

failurePolicy: Fail with validationActions: [Deny] means that if the webhook does not answer nothing starts: the right call from a security standpoint, and exactly what takes a cluster down on a Sunday morning. The defensible posture is fail-closed in the production namespace, with the controller in high availability and its own namespace excluded from its policies so it can be recovered; Audit during adoption, the same “observe first, block later” rule we applied with Tetragon in hardening; and a written exception procedure with an expiry date, because on the day of the 3 a.m. hotfix somebody is going to bypass the policy and it is better that they do it down an audited path.

On latency, the cost is at startup and not per request: verification happens once per pod. A third-party benchmark from March 2026 on policy-controller v0.15 reports 92 ms at p50 and 184 ms at p99 for internally signed images, with peaks of up to 4 seconds in mass deployments. It is a third-party measurement with no published independent replication, but the order of magnitude is coherent: hundreds of milliseconds against the minutes a pod takes to load 140 GB into HBM.

As a second net, Harbor stores cosign signatures as referrer artefacts and can prevent the deployment of anything unsigned by project policy. And in GitOps with Flux, what closes the circle is always referencing by digest and verifying the OCI artefact’s signature at reconciliation: two independent checks at different moments.

Decision map: what to put in, and in what order

By decreasing return on effort:

  1. Pin by digest throughout GitOps. An afternoon; without this, the rest is decorative.
  2. Image signing with keyless cosign from CI and ImageValidatingPolicy in Audit. Days. It reveals how much of your cluster is unsigned, which is usually a surprise.
  3. Signing the model artefact with model_signing in the job that publishes it. Days. It kills the model uploaded by hand on a Friday.
  4. Moving to Deny in the inference namespace, with audited exceptions and the controller in HA.
  5. SLSA attestation, targeting Build L2 and then L3. Weeks: it requires ephemeral runners and taking the signing material out of the training script’s reach.
  6. An AIBOM emitted from the pipeline, with the five fields only it knows. Weeks, and above all process work.
  7. Private Sigstore, only with a real air-gap requirement and with the previous six done.

Between Kyverno and policy-controller: Kyverno if there is or will be a broader policy programme (Pod Security, labels, quotas), for unification and for its graduated status; policy-controller if the organisation is Sigstore-centric and does not want more policy engine than strictly necessary. Between keyless and a key: keyless if CI can reach an OIDC, even an internal one; a key with an HSM in strict air-gap where you are not going to run your own Fulcio.

Operational traps

  • Verifying without narrowing the identity. cosign verify without --certificate-identity or --certificate-oidc-issuer, or with a regexp that accepts any repository from the provider, is security theatre. Trap number one and the quietest.
  • Fail-closed bugs in the verifier itself. Kyverno issue 16435, opened on 2 July 2026 against 1.18.0, describes two simultaneous regressions in ImageValidatingPolicy with key and certificate attestors: a null-pointer SIGSEGV and a tlog failure with the message not enough verified log entries from transparency log: 0 < 1. Both deny legitimate images or bring the controller down. Fixed for 1.19, with the underlying lesson intact: the verifier is a critical component and its regressions are unavailability.
  • The trust root that expires in air-gap. It was copied by hand nine months ago, nobody keeps the calendar, and one day verification fails without anyone having touched anything.
  • The AIBOM nobody regenerates. It is emitted at the initial deployment, the model is fine-tuned three times and it still describes version one. An out-of-date inventory is worse than none: it creates unjustified confidence.
  • Verifying the image and forgetting the model. The most common of all: the vLLM container impeccably signed, and the 140 GB it loads coming from an unverified bucket.

What this does NOT prove

A signature says who published, not that the model is good or safe. It is a claim of authorship, not of quality. Ultralytics documents it: four releases with a cryptominer and valid attestations. If the build environment is compromised, the signature faithfully certifies the compromised artefact.

A signed model can have a backdoor. Nothing in Sigstore, OMS or SLSA examines the weights. A backdoor trained by someone with legitimate access to the pipeline passes through the whole chain without triggering an alarm; the defence is adversarial evaluation and canary or shadow, not cryptography. And SLSA says nothing about dataset quality: a formally perfect pipeline over poisoned data produces a poisoned model with impeccable provenance.

The AIBOM is only as good as the process that generates it. The fields that matter are not inferred by any tool. And the operational cost is recurring: rotation of keys and TUF roots, verification incidents in the small hours, policy noise during adoption, and the continuous work of maintaining an inventory nobody reads until there is an audit.

The maturity split as of July 2026. Mature and deployable: Sigstore and cosign over OCI artefacts, Kyverno and policy-controller as admission control, in-toto as a CNCF-graduated format, pinning by digest. Usable with judgement: OMS and model_signing (a solid specification with strong industry backing, but a sandbox project), SLSA Build L2-L3 in fine-tuning, CycloneDX ML-BOM. Still work in progress: SLSA’s Build Environment and Dependency tracks (precisely the ones that would cover the Ultralytics scenario), the tooling that fills in an AIBOM automatically, and all backdoor detection in weights, which today is literature and not product.

For an inference factory

First: the initContainer is your receiving bay, and today it probably checks nothing. The admission controller looks at the image; the model comes in afterwards through a side door. Adding cosign verify --offline and model_signing verify there costs a couple of afternoons, fits inside the cold start budget (23 seconds of hashing over a disk throughput you are already paying for) and closes the largest hole in most deployments.

Second: pin by digest today, sign tomorrow. If only one thing fits this quarter, it is removing mutable tags from GitOps: it is cheap, it breaks nothing and it turns “I trust the registry” into “I trust these specific bytes”. The signature adds the issuer’s identity on top, but without a digest it has nothing to hold on to.

Third: require the attestation from whoever hands you the model, even if it is the team next door. An inference factory consumes what somebody else produces, and that contract should be explicit: a signature from a known CI identity, an attestation with the base model’s hash and the dataset’s commit, and an AIBOM emitted from the pipeline. Annex XI of the AI Act already requires that, and it is in force, and it is what lets you answer in twenty minutes, rather than in two weeks, when somebody asks what exactly is behind the endpoint.

The chain now has three links: you know how the model is served, where its bytes come from and why you trust them. The last question remains: what that pod is relative to the rest of the system when it starts with the verified model, and how far you can trust the machine it runs on. The fourth article goes into workload identity with SPIFFE/SPIRE and isolation with Confidential Containers: a verified model, served by an unidentified process on a host you cannot attest, leaves the chain open right at the last link.

See also

Sources