Kueue on lo0 — Tech Blog

Volcano and Kueue: gang scheduling, queues and GPU quotas for distributed workloads on Kubernetes

Tue, 16 Jun 2026 13:00:00 +0200

TL;DR

Volcano (volcano-sh, CNCF incubating) is a complete batch scheduler that replaces or complements the kube-scheduler: it places pods with gang semantics (all-or-nothing via PodGroup/minMember), manages queues with priority, DRF fair-share and preemption between queues, and understands network and NUMA topology.

Kueue (kubernetes-sigs/kueue) is a queue and quota manager at the Job level: it does NOT place pods (it delegates to the kube-scheduler or to Volcano), but it decides when a workload can be admitted according to available quota (ClusterQueue/LocalQueue/Cohort), with fair sharing, borrowing between teams and preemption by priority. It natively integrates Job, JobSet, RayJob, all the Kubeflow operators and more.

The winning combination in production for multi-tenant GPU workloads is: Kueue for quota and queues + Volcano (or the sig-scheduler coscheduling plugin) for the job’s gang.

The analogy

Imagine a club with limited capacity and a dance floor inside.

Kueue is the doorman and the booking manager: it checks whether the team’s quota (its reserved capacity) allows the group in, applies the fair waiting list, borrows capacity from other teams if any are idle, and takes the space back when the owner needs it. But the doorman does not decide where each person sits inside the venue.

Volcano is the head waiter: once the group has permission to come in, he decides which tables they sit at, makes sure the whole group sits down at once or nobody comes in (gang), picks the tables according to topology (who needs to talk to whom) and throws out lower-priority groups to make room if necessary.

Without a doorman (Kueue), the head waiter does not know how many groups he can take at once or whether a team is exceeding its capacity. Without a head waiter (Volcano), the doorman lets the group in but its members scatter across the available tables on their own, and the party of 8 that needs to sit together never manages it.

The problem neither solves by default: the kube-scheduler

Kubernetes’ kube-scheduler is a pod scheduler, not a job scheduler. It assigns pods one by one to the most suitable node according to available resources and affinity constraints. For a distributed training workload that needs, say, 8 pods with 4 GPUs each (32 GPUs in total across 8 nodes of 4×H100), the standard scheduler does the following:

It looks for a node with 4 available GPUs. It finds one. It schedules pod 1.
It looks for another node with 4 GPUs. It finds one. It schedules pod 2.
It carries on until it reaches pod 6 and it turns out there are no longer any nodes with 4 free GPUs: the cluster has exactly 32 GPUs and other workloads are using some of them.
Pods 1–5 are Running. Pods 6–8 are Pending.
Pods 1–5 can do nothing without the others: a distributed PyTorch job needs all the workers to start before the torchrun process can begin. It waits with the resources occupied. Deadlock.

This is not a bug, it is the design: the kube-scheduler has no concept of “schedule this group of pods only if you can schedule all of them”. As a result:

The resources of pods 1–5 are locked up without producing work.
Other jobs that could run with the partial resources also wait.
If several jobs are in this situation, the cluster can end up with fragmented resources, no job running and everyone in circular deadlock.

On top of that, the kube-scheduler has no notion of:

Queues per team or project with relative priority.
Quotas of resources per team with the ability to borrow from idle teams.
Fair-share: if team A has spent weeks using 80 % of the cluster, it should wait longer than team B, which has spent weeks idle.
Inter-queue preemption: evicting a lower-priority job from another team to make room for this team’s urgent job.

Solving any of these problems requires adding a layer on top of the scheduler. Volcano and Kueue are the two dominant OSS solutions in 2026, with complementary architectural approaches.

Volcano: the batch scheduler

What it is and what it replaces

Volcano (volcano-sh) is a Kubernetes-native batch scheduler accepted by the CNCF as its first and only official container batch scheduling project (volcano.sh/en/docs). At version v1.15.x as of June 2026, with CNCF incubating status.

Volcano is not an addon to the kube-scheduler: it is an alternative scheduler (or a complementary one, depending on the configuration) that places pods. It is installed as a deployment, and jobs that want to benefit from its capabilities must use the volcano scheduler class in their pod spec (schedulerName: volcano) or use the VolcanoJob CRD.

Volcano’s central value proposition is that it treats groups of pods as atomic scheduling units, not individual pods. That is what makes it possible to solve the deadlock described above.

Gang scheduling via PodGroup

Volcano’s central mechanism is the PodGroup: a CRD that groups a job’s pods and defines how many must be schedulable before Volcano starts any of them.

# PodGroup for a distributed PyTorch training job: 8 workers, minimum 8
apiVersion: scheduling.volcano.sh/v1beta1
kind: PodGroup
metadata:
 name: pytorch-train-pg
 namespace: ml-training
spec:
 minMember: 8 # all-or-nothing: if there is no room for 8, none starts
 minResources:
 nvidia.com/gpu: "32" # 8 pods × 4 GPUs = 32 GPUs minimum in the cluster
 queue: team-datos  # which Volcano queue this job is assigned to
 priorityClassName: high-priority

The minMember parameter implements all-or-nothing semantics: Volcano only assigns nodes to the group’s pods when it can assign at least minMember pods simultaneously. If the cluster does not have capacity for 8 GPU pods at this moment, no pod in the group moves out of Pending. Nothing is blocked, nothing is fragmented.

minMember can be lower than the job’s total pod count: this allows elastic gang scheduling, where the job can start with fewer workers and scale up, useful for jobs that tolerate a reduced worker count.

For a job’s pods to be associated with the PodGroup, they carry the annotation:

# Pod spec of the PyTorch worker
metadata:
 annotations:
 scheduling.volcano.sh/pod-group-name: pytorch-train-pg
spec:
 schedulerName: volcano
 containers:
 - name: trainer
 image: pytorch/pytorch:2.5-cuda12.4
 resources:
 limits:
 nvidia.com/gpu: "4"

Queue: queues with quotas and priority

Volcano introduces the Queue CRD to manage multiple tenants with independent quotas:

apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
 name: team-datos
spec:
 weight: 4 # relative weight for fair-share between queues (proportion plugin)
 capability: # absolute ceiling of resources this queue can use
 nvidia.com/gpu: "16"
 guarantee: # guaranteed resource, never lent to other queues
 resource:
 nvidia.com/gpu: "8"
 reclaimable: true # if true, others can reclaim the resources it lends when they need them

apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
 name: team-ia
spec:
 weight: 2
 capability:
 nvidia.com/gpu: "8"
 guarantee:
 resource:
 nvidia.com/gpu: "4"
 reclaimable: true

The weight field feeds the proportion plugin: queues compete for the cluster’s available resources in proportion to their weight. A cluster with 32 GPUs and two queues of weight 4 and 2 splits the GPUs in a 4:2 ratio (≈21 and 11 GPUs respectively) when both are saturated.

Scheduler plugins: DRF, binpack, topology-aware

Volcano implements its scheduling logic as a pipeline of actions and plugins:

Actions (what the scheduler does in each cycle):

enqueue: moves jobs from the waiting queue to schedulable when quota is available.
allocate: assigns nodes to schedulable pods.
preempt: evicts lower-priority pods to make room for higher-priority ones within the same queue.
reclaim: evicts pods from other queues that are using more than their guarantee to return resources to the owner.
backfill: fills idle resources with best-effort jobs that do not interfere with the rest.

Plugins relevant to GPU workloads:

Plugin	What it does
`gang`	Implements the PodGroup’s all-or-nothing semantics
`proportion`	Fair-share by queue weight (proportional quota)
`capacity`	Quotas with `guarantee`/`capability` and reclaim; a more expressive alternative to proportion
`drf`	Dominant Resource Fairness: multi-dimensional fair-share (CPU, memory, GPU)
`binpack`	Packs pods onto the fullest nodes; reduces GPU fragmentation
`priority`	Orders jobs by priority within the same queue
`nodeorder`	Node scoring according to multiple criteria (affinity, resources, spread)
`task-topology`	Affinity between pods of the same job (inter-GPU communication)
`numa-aware`	NUMA affinity: aligns pods with the node’s NUMA socket to reduce memory latency

Network and NUMA topology (v1.11+)

Volcano v1.11 (February 2025) introduced Network Topology Aware Scheduling as a first-class feature (CNCF blog, March 2025). Distributed training jobs in a datacenter with a hierarchical network structure (spine/leaf, blocks of nodes with NVSwitch) can declare topology constraints:

# VolcanoJob with a network topology constraint
apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
 name: llm-pretrain
 namespace: ml-training
spec:
 minAvailable: 8
 schedulerName: volcano
 queue: team-datos
 plugins:
 ssh: []
 env: []
 svc: []
 networkTopology:
 mode: hard # hard: the job MUST satisfy the constraint
 highestTierAllowed: block  # pods cannot span beyond a network "block"
 tasks:
 - replicas: 8
 name: worker
 template:
 spec:
 containers:
 - name: trainer
 image: nvcr.io/nvidia/pytorch:25.01-py3
 resources:
 limits:
 nvidia.com/gpu: "4"

The highestTierAllowed: block semantics instruct Volcano to place the 8 pods within the same network block (for example, all the nodes under the same access switch), minimising the inter-block traffic that degrades a distributed all-reduce.

NUMA awareness works in a similar way: with the numa-aware plugin, pods request a NUMA policy (single-numa-node, restricted, best-effort) and Volcano selects nodes where the requested CPU, memory and GPU resources are in the same NUMA domain, avoiding the remote memory access overhead (NUMA-crossing) that can degrade training throughput by 15-30 % on multi-socket nodes.

GPU virtualization in v1.11+

Volcano v1.11 also introduces support for dynamic MIG and vCUDA: instead of declaring nvidia.com/gpu: 1 for a whole GPU, workloads can declare nvidia.com/gpu-memory: 20Gi and Volcano (with the corresponding device plugin) dynamically provisions the MIG instance or the vCUDA partition. This is [project marketing with no independent benchmarks published as of June 2026], but the feature’s architecture is documented in the code.

What it replaces or adds to the default scheduler

Capability	kube-scheduler	Volcano
Placing pods on nodes	Yes	Yes (it replaces it for workloads marked with `schedulerName: volcano`)
Gang scheduling (all-or-nothing)	No	Yes (PodGroup + minMember)
Queues with priority	No	Yes (Queue CRD)
Inter-queue fair-share	No	Yes (DRF, proportion, capacity plugins)
Inter-queue preemption	No	Yes (reclaim action)
Topology-aware (NUMA, network)	Partial (node affinity)	Yes (dedicated plugins)
Elastic gang (minMember < total)	No	Yes
Best-effort backfill	No	Yes
Framework integrations	Partial	MPI, PyTorch, Ray, TensorFlow, Spark, Flink, Horovod

Kueue: the queue and quota manager

What it is and what it does NOT do

Kueue (kubernetes-sigs/kueue) is a Kubernetes-native system that manages quotas and how jobs consume them (kueue.sigs.k8s.io). Kueue decides when a job must wait, when it should be admitted (pods can be created) and when it should be evicted (active pods must be deleted).

Kueue’s central design principle is explicit in its documentation: avoid duplicating mature functionality of Kubernetes components. Autoscaling is the cluster-autoscaler’s responsibility. Pod-to-node scheduling is the kube-scheduler’s responsibility. Job lifecycle management is the kube-controller-manager’s responsibility. Kueue replaces none of them: it sits on top as a layer of admission control and quota management at the Job level.

This is the fundamental distinction: Kueue does not place pods on nodes. When Kueue admits a workload, it simply allows the corresponding job controller to create the pods, and those pods are scheduled by the kube-scheduler (or by Volcano, if it is configured as the scheduler).

The four core objects

ResourceFlavor: maps abstract resources to concrete groups of physical nodes.

apiVersion: kueue.x-k8s.io/v1beta1
kind: ResourceFlavor
metadata:
 name: h100-sxm
spec:
 nodeLabels:
 accelerator: h100-sxm
 node-pool: gpu-training
 tolerations:
 - key: "nvidia.com/gpu"
 operator: "Exists"
 effect: "NoSchedule"

ClusterQueue: defines the resource quota per flavor for a tenant. A cluster-scoped object.

apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
 name: cq-team-datos
spec:
 cohort: llm-platform  # cohort it belongs to (can lend/borrow)
 queueingStrategy: BestEffortFIFO
 namespaceSelector:
 matchLabels:
 kubernetes.io/metadata.name: ns-datos
 resourceGroups:
 - coveredResources: ["nvidia.com/gpu", "cpu", "memory"]
 flavors:
 - name: h100-sxm
 resources:
 - name: "nvidia.com/gpu"
 nominalQuota: 16 # GPUs guaranteed for this team
 borrowingLimit: 8 # can take up to 8 additional GPUs from the cohort
 lendingLimit: 8 # can lend up to 8 of its 16 nominal GPUs
 - name: "cpu"
 nominalQuota: "128"
 - name: "memory"
 nominalQuota: "512Gi"
 preemption:
 reclaimWithinCohort: LowerPriority  # reclaims lent quota by evicting lower-priority jobs
 borrowWithinCohort:
 policy: LowerPriority
 withinClusterQueue: LowerPriority

LocalQueue: the namespace-scoped entry point for a team’s workloads. Jobs point to their LocalQueue; Kueue maps them to the corresponding ClusterQueue.

apiVersion: kueue.x-k8s.io/v1beta1
kind: LocalQueue
metadata:
 name: lq-datos
 namespace: ns-datos
spec:
 clusterQueue: cq-team-datos

Cohort: groups ClusterQueues that can lend quota to each other. It is not a standalone CRD; it is declared as a field in the ClusterQueue (spec.cohort: name). Kueue aggregates the available quota of all the ClusterQueues in the cohort and lets any of them borrow what the others are not using, respecting the borrowingLimit and lendingLimit.

Kueue implements Fair Sharing as the ordering policy for the queue of pending workloads (kueue.sigs.k8s.io/docs/concepts/fair_sharing): when several workloads compete for quota in the cohort, those belonging to ClusterQueues with higher accumulated historical usage have lower admission priority. This implements equitable sharing without permanently blocking any team.

Preemption in Kueue operates along two dimensions:

reclaimWithinCohort: the ClusterQueue that lends quota reclaims it by evicting workloads that are using it on loan, according to a priority policy.
withinClusterQueue: within the same ClusterQueue, lower-priority workloads are evicted to make way for higher-priority workloads from the same team.

Gang semantics in Kueue: all-or-nothing with ready Pods

Kueue provides gang admission at the Job level: it admits the complete workload only when all the necessary quota is available. If a RayJob needs 8 GPUs (1 head + 7 workers), Kueue does not admit the workload until 8 GPUs are available in the ClusterQueue (or borrowed from the cohort). waitForPodsReady with a timeout adds a second guarantee: if the created pods do not become Ready within the configured time, Kueue re-queues the workload and releases the quota (kueue.sigs.k8s.io/docs/tasks/manage/setup_wait_for_pods_ready).

This is gang semantics at the admission level, not at the pod placement level. It guarantees that the quota is available before creating the pods, but it does not guarantee that the kube-scheduler can place them all on concrete nodes at the same time. For that second guarantee you need Volcano or the coscheduling plugin.

Topology-Aware Scheduling (TAS)

Kueue v0.10+ introduces Topology-Aware Scheduling (kueue.sigs.k8s.io/docs/concepts/topology_aware_scheduling): it allows node topologies (blocks, subblocks, hosts) to be defined and workloads to request co-location levels. Kueue only admits the workload when it can satisfy the topological constraint, and it adds node selectors and taints at admission time so that the scheduler places the pods in the right topology.

TAS is configured with the Topology CRD:

apiVersion: kueue.x-k8s.io/v1beta1
kind: Topology
metadata:
 name: datacenter-topology
spec:
 levels:
 - nodeLabel: "topology.kubernetes.io/block"
 - nodeLabel: "topology.kubernetes.io/rack"
 - nodeLabel: "kubernetes.io/hostname"

And the ResourceFlavor references the topology:

apiVersion: kueue.x-k8s.io/v1beta1
kind: ResourceFlavor
metadata:
 name: h100-sxm
spec:
 nodeLabels:
 accelerator: h100-sxm
 topologyName: datacenter-topology

Framework integrations

Kueue has built-in integration (no additional code) for the following workload types, activated with an annotation on the job:

metadata:
 labels:
 kueue.x-k8s.io/queue-name: lq-datos  # points to the team's LocalQueue

The supported workload types include: batch/Job, JobSet, RayJob, RayCluster, PytorchJob, TFJob, MPIJob, JAXJob, PaddleJob, XGBoostJob, TrainJob, AppWrapper, LeaderWorkerSet, Deployment, StatefulSet, and plain Pod/PodGroup.

For LLM workloads, the directly relevant cases:

RayJob (distributed training with Ray Train): Kueue admits the RayJob when there is quota for the whole Ray cluster (head + workers). Documented at docs.ray.io.
PyTorchJob (Kubeflow Training Operator): gang admission of the complete job.
JobSet: for coordinated multi-replica jobs (LWS, multi-step pipelines).
Deployment/StatefulSet: for continuous inference, allowing inference GPU quota to be managed the same way as training quota.

The key distinction: Volcano places, Kueue admits

This table sums up the fundamental architectural difference:

Dimension	Volcano	Kueue
Main role	Scheduler (places pods on nodes)	Admission controller + quota manager (decides when to create pods)
Gang scheduling	Yes, at the placement level (PodGroup/minMember)	Yes, at the admission level (all-or-nothing on quota)
Multi-tenant quotas	Yes (Queue with capability/guarantee)	Yes (ClusterQueue with nominalQuota/borrowingLimit)
Cohorts / borrowing	Partial (reclaimable between queues)	Yes (Cohort with explicit lendingLimit/borrowingLimit)
Fair sharing	Yes (DRF plugin)	Yes (Fair Sharing based on historical usage)
Preemption	Yes (preempt + reclaim actions)	Yes (reclaimWithinCohort, withinClusterQueue)
Topology/NUMA	Yes (dedicated plugins, network topology, NUMA-aware)	Yes (TAS, topology levels in ResourceFlavor)
Framework integrations	Volcano Job (MPI, PyTorch, Ray, TF, Spark, Flink)	Native: Job, JobSet, RayJob, Kubeflow, LWS, AppWrapper, Deployment, StatefulSet
Who places the pods	Volcano	kube-scheduler (or Volcano if configured)
Installation footprint	Medium-high (its own scheduler, CRDs, webhook, metrics)	Light (controller, CRDs, webhook; does not replace the scheduler)
Maturity / status	CNCF incubating; v1.15 (June 2026); production at Huawei, Baidu, DiDi	kubernetes-sigs; API v1beta2; production at Google GKE, Red Hat OpenShift 4.20, Runway ML
Adoption curve	Steeper (requires changing schedulerName or using the VolcanoJob CRD)	Gentler (adds labels to existing jobs; does not change the scheduler)

How they coexist: the production pattern

In production, Kueue and Volcano are complementary, not mutually exclusive. The most common pattern in 2026 for multi-tenant GPU clusters is:

Kueue manages the global quota: which team can use how many GPUs, how much it can borrow, when a job goes into a queue versus being admitted.
Volcano does the gang scheduling at the pod level: once Kueue admits the job (the quota is available), Volcano places the pods, making sure they are all placed simultaneously on compatible nodes.

The integration is configured by specifying schedulerName: volcano in the pod specs of the workloads managed by Kueue. Kueue sees the Job/RayJob/PyTorchJob and manages its quota; when it admits it, the pods are created and Volcano places them with gang semantics. Volcano’s PodGroups are created automatically by the Volcano Job controller or by the Kubeflow Training Operator itself when it detects that the scheduler is Volcano.

# PyTorchJob managed by Kueue (quota) + Volcano (gang placement)
apiVersion: kubeflow.org/v1
kind: PyTorchJob
metadata:
 name: llm-finetune-70b
 namespace: ns-datos
 labels:
 kueue.x-k8s.io/queue-name: lq-datos  # Kueue manages the quota
 annotations:
 scheduling.volcano.sh/queue-name: team-datos  # Volcano uses its own Queue
spec:
 pytorchReplicaSpecs:
 Master:
 replicas: 1
 restartPolicy: OnFailure
 template:
 spec:
 schedulerName: volcano  # Volcano does the placement
 containers:
 - name: pytorch
 image: nvcr.io/nvidia/pytorch:25.01-py3
 resources:
 limits:
 nvidia.com/gpu: "4"
 Worker:
 replicas: 7
 restartPolicy: OnFailure
 template:
 spec:
 schedulerName: volcano
 containers:
 - name: pytorch
 image: nvcr.io/nvidia/pytorch:25.01-py3
 resources:
 limits:
 nvidia.com/gpu: "4"

The third way: the sig-scheduler coscheduling plugin

If you do not want to deploy a complete alternative scheduler but you need gang scheduling, there is the coscheduling plugin from kubernetes-sigs/scheduler-plugins. This plugin extends the kube-scheduler with a PodGroup mechanism similar to Volcano’s, implemented as a scheduling framework plugin (permit plugin). The advantage is that it does not replace the scheduler; the disadvantage is that it has less functionality than Volcano (no DRF, no Queue/fair-share, no network topology). It is the right option for simple clusters that only need gang and do not want Volcano’s operational complexity. Kueue can also work alongside this plugin.

Complete YAML examples

Volcano: Queue + PodGroup + VolcanoJob

# 1. Queue for the data team (16 nominal GPUs, ceiling at 24)
apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
 name: team-datos
spec:
 weight: 4
 capability:
 nvidia.com/gpu: "24"
 cpu: "192"
 memory: "768Gi"
 guarantee:
 resource:
 nvidia.com/gpu: "16"
 reclaimable: true
---
# 2. Queue for the AI team (8 nominal GPUs, ceiling at 16)
apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
 name: team-ia
spec:
 weight: 2
 capability:
 nvidia.com/gpu: "16"
 cpu: "64"
 memory: "256Gi"
 guarantee:
 resource:
 nvidia.com/gpu: "8"
 reclaimable: true
---
# 3. PodGroup: 70B fine-tuning job, 8 workers × 4 GPUs = 32 GPUs
apiVersion: scheduling.volcano.sh/v1beta1
kind: PodGroup
metadata:
 name: finetune-70b-pg
 namespace: ns-datos
spec:
 minMember: 8
 minResources:
 nvidia.com/gpu: "32"
 queue: team-datos
 priorityClassName: training-high
---
# 4. VolcanoJob (wrapper that Volcano understands natively)
apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
 name: finetune-70b
 namespace: ns-datos
spec:
 minAvailable: 8
 schedulerName: volcano
 queue: team-datos
 priorityClassName: training-high
 plugins:
 env: []
 svc: []
 policies:
 - event: PodEvicted
 action: RestartJob
 tasks:
 - replicas: 8
 name: worker
 policies:
 - event: TaskCompleted
 action: CompleteJob
 template:
 metadata:
 annotations:
 scheduling.volcano.sh/pod-group-name: finetune-70b-pg
 spec:
 schedulerName: volcano
 containers:
 - name: trainer
 image: nvcr.io/nvidia/pytorch:25.01-py3
 command: ["torchrun", "--nproc_per_node=4", "--nnodes=8",
 "--node_rank=$(RANK)", "--master_addr=$(MASTER_ADDR)",
 "--master_port=23456", "train.py"]
 env:
 - name: NCCL_DEBUG
 value: "INFO"
 resources:
 requests:
 nvidia.com/gpu: "4"
 cpu: "16"
 memory: "64Gi"
 limits:
 nvidia.com/gpu: "4"
 cpu: "16"
 memory: "64Gi"
 restartPolicy: Never

Kueue: ResourceFlavor + ClusterQueue + LocalQueue + annotated Job

# 1. ResourceFlavor: nodes with H100 SXM
apiVersion: kueue.x-k8s.io/v1beta1
kind: ResourceFlavor
metadata:
 name: h100-sxm
spec:
 nodeLabels:
 accelerator: h100-sxm
 tolerations:
 - key: "nvidia.com/gpu"
 operator: "Exists"
 effect: "NoSchedule"
---
# 2. ClusterQueue for the data team: 16 nominal GPUs, can borrow 8
apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
 name: cq-team-datos
spec:
 cohort: llm-platform
 queueingStrategy: BestEffortFIFO
 namespaceSelector:
 matchLabels:
 kubernetes.io/metadata.name: ns-datos
 resourceGroups:
 - coveredResources: ["nvidia.com/gpu", "cpu", "memory"]
 flavors:
 - name: h100-sxm
 resources:
 - name: "nvidia.com/gpu"
 nominalQuota: 16
 borrowingLimit: 8
 lendingLimit: 8
 - name: "cpu"
 nominalQuota: "128"
 - name: "memory"
 nominalQuota: "512Gi"
 preemption:
 reclaimWithinCohort: LowerPriority
 borrowWithinCohort:
 policy: LowerPriority
 withinClusterQueue: LowerPriority
---
# 3. ClusterQueue for the AI team: 8 nominal GPUs
apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
 name: cq-team-ia
spec:
 cohort: llm-platform
 queueingStrategy: BestEffortFIFO
 namespaceSelector:
 matchLabels:
 kubernetes.io/metadata.name: ns-ia
 resourceGroups:
 - coveredResources: ["nvidia.com/gpu", "cpu", "memory"]
 flavors:
 - name: h100-sxm
 resources:
 - name: "nvidia.com/gpu"
 nominalQuota: 8
 borrowingLimit: 8
 lendingLimit: 4
 - name: "cpu"
 nominalQuota: "64"
 - name: "memory"
 nominalQuota: "256Gi"
 preemption:
 reclaimWithinCohort: LowerPriority
 withinClusterQueue: LowerPriority
---
# 4. LocalQueue in the data team's namespace
apiVersion: kueue.x-k8s.io/v1beta1
kind: LocalQueue
metadata:
 name: lq-datos
 namespace: ns-datos
spec:
 clusterQueue: cq-team-datos
---
# 5. LocalQueue in the AI team's namespace
apiVersion: kueue.x-k8s.io/v1beta1
kind: LocalQueue
metadata:
 name: lq-ia
 namespace: ns-ia
spec:
 clusterQueue: cq-team-ia
---
# 6. Batch inference RayJob managed by Kueue
apiVersion: ray.io/v1
kind: RayJob
metadata:
 name: batch-eval-llama70b
 namespace: ns-datos
 labels:
 kueue.x-k8s.io/queue-name: lq-datos  # Kueue manages the admission
spec:
 entrypoint: "python batch_eval.py --model /models/llama-70b"
 rayClusterSpec:
 headGroupSpec:
 rayStartParams:
 num-gpus: "4"
 template:
 spec:
 containers:
 - name: ray-head
 image: rayproject/ray-ml:2.40.0-gpu
 resources:
 limits:
 nvidia.com/gpu: "4"
 cpu: "16"
 memory: "64Gi"
 workerGroupSpecs:
 - replicas: 3
 minReplicas: 3 # gang: Kueue does not admit if there is no quota for 3 workers
 maxReplicas: 3
 groupName: gpu-worker
 rayStartParams:
 num-gpus: "4"
 template:
 spec:
 containers:
 - name: ray-worker
 image: rayproject/ray-ml:2.40.0-gpu
 resources:
 limits:
 nvidia.com/gpu: "4"
 cpu: "16"
 memory: "64Gi"

Kueue + Volcano together: PyTorchJob with Kueue quota and Volcano gang placement

# PyTorchJob: Kueue controls the quota, Volcano does the gang placement
apiVersion: kubeflow.org/v1
kind: PyTorchJob
metadata:
 name: distributed-finetune
 namespace: ns-datos
 labels:
 kueue.x-k8s.io/queue-name: lq-datos # Kueue: quota and admission
 annotations:
 # Volcano automatically creates the PodGroup when schedulerName=volcano
 scheduling.volcano.sh/queue-name: team-datos
spec:
 pytorchReplicaSpecs:
 Master:
 replicas: 1
 restartPolicy: OnFailure
 template:
 spec:
 schedulerName: volcano # Volcano: gang placement
 containers:
 - name: pytorch
 image: nvcr.io/nvidia/pytorch:25.01-py3
 resources:
 limits:
 nvidia.com/gpu: "4"
 cpu: "16"
 memory: "64Gi"
 Worker:
 replicas: 7
 restartPolicy: OnFailure
 template:
 spec:
 schedulerName: volcano
 containers:
 - name: pytorch
 image: nvcr.io/nvidia/pytorch:25.01-py3
 resources:
 limits:
 nvidia.com/gpu: "4"
 cpu: "16"
 memory: "64Gi"

For LLM workloads: training, fine-tuning and batch inference

Distributed multi-GPU training and fine-tuning

Gang scheduling is essential for any distributed training job that uses NCCL all-reduce (PyTorch DDP, FSDP, DeepSpeed ZeRO). If a single worker in the group does not start, the torchrun coordinator waits indefinitely; with the standard kube-scheduler this scenario happens every time the cluster is under contention.

In a generic cluster of 4 nodes of 4×H100 SXM (16 GPUs in total), a fine-tuning job for a 70B model typically requires 8 GPUs in tensor-parallel 8 (TP=8) or 16 GPUs in TP=4 plus data-parallel 4. With Volcano, the PodGroup with minMember: 8 guarantees that either the 8 pods are placed at once or none blocks resources. With Kueue on top, the quota guarantees that the team does not exceed its 16 nominal GPUs and that other teams with available quota are not blocked by a waiting job.

The cross-link with inference capacity planning is direct: the model’s VRAM budget (weights plus KV-cache) determines the minimum TP and therefore the PodGroup’s minMember.

Batch inference and evaluations (evals)

Batch inference jobs, generating responses for an evaluation dataset, processing embeddings in bulk, offline re-ranking, are naturally parallel workloads that do not necessarily require strict gang scheduling (each request is independent), but they do benefit from quota and fair-share.

For these workloads, Kueue alone is enough: a batch/Job with multiple independent pods is managed with the ClusterQueue’s quota with no need for Volcano. If several teams are submitting evaluation jobs simultaneously, Kueue orders the admission by fair-share and priority, and with cohort borrowing the jobs of teams with free quota do not have to wait for the quota of busy teams.

The chargeback of these workloads connects directly with what is described in GPU chargeback and showback: the ClusterQueue’s nominalQuota is the expression of the GPU budget in Kubernetes, and OpenCost can attribute the cost per namespace or label for the monthly report.

Multi-tenant GPU quota and chargeback

The alignment between Kueue and the chargeback system is direct:

FinOps concept	Kueue mechanism
Guaranteed GPU budget	`nominalQuota` per ClusterQueue
Maximum spending ceiling	`nominalQuota + borrowingLimit`
Lending idle capacity	Cohort + `lendingLimit`
Reclaiming your own quota	`preemption.reclaimWithinCohort: LowerPriority`
Fair-share between teams	Fair Sharing policy in the ClusterQueue
Chargeback of the loan	borrowed GPU hours × cost per GPU-hour (OpenCost)

For the utilisation dimension as a FinOps lever, see GPU utilisation as a FinOps lever: Kueue and Volcano together make it possible to maximise utilisation without sacrificing the quota guarantees, which is exactly the FinOps objective.

Managing MIG partitions within this system (declaring nvidia.com/mig-4g.40gb as a resource in the ClusterQueue) integrates naturally: the ResourceFlavor can map to nodes with a specific MIG profile, as explained in sharing a GPU: time-slicing, MPS and MIG.

Diagram: the flow of a training job with Kueue + Volcano

Full comparison table

Criterion	Volcano	Kueue	Coscheduling plugin
Role	Scheduler (placement)	Admission + quota (no placement)	kube-scheduler plugin (placement)
Gang scheduling	Yes, pod level (PodGroup/minMember)	Yes, admission level (quota gang)	Yes, pod level (PodGroup)
Multi-tenant quotas	Yes (Queue capability/guarantee)	Yes (ClusterQueue nominalQuota)	No
Cohorts / borrowing	Limited (reclaimable)	Yes (Cohort with borrowingLimit/lendingLimit)	No
Fair-share	Yes (DRF, proportion)	Yes (Fair Sharing by historical usage)	No
Inter-queue preemption	Yes (reclaim action)	Yes (reclaimWithinCohort)	No
Network / NUMA topology	Yes (v1.11+, dedicated plugins)	Yes (TAS, Topology CRD)	No
Native integrations	MPI, PyTorch, Ray, TF, Spark, Flink, Horovod	Job, JobSet, RayJob, Kubeflow, LWS, AppWrapper, Deployment, StatefulSet	Any job with a PodGroup
Who places the pods	Volcano	kube-scheduler (or Volcano)	kube-scheduler
Elastic gang	Yes (minMember < total replicas)	Partial (partial admission in batch/Job)	Limited
Footprint	Medium-high (its own scheduler)	Light (an additional controller)	Minimal (a scheduler plugin)
Compatibility with Kueue	Yes (as the scheduler under Kueue)	—	Yes (a complementary option)
CNCF status / maturity	CNCF incubating, v1.15	kubernetes-sigs, v1beta2, adopted in GKE/OpenShift	kubernetes-sigs/scheduler-plugins, experimental
When to choose it	HPC-like distributed training, NUMA, network topology, MPI	Multi-tenancy with flexible quota, heterogeneous workloads, inference + batch together	Simple clusters that only need gang without their own scheduler

Operational pitfalls and honest scepticism

1. Deadlock from a badly configured gang

The most frequent scenario: minMember set equal to the total replica count in a cluster where several jobs compete for the same resources. If two jobs of 8 pods each try to use a cluster with 8 GPU nodes and job A has 4 pods placed (not 8, so Volcano holds them pending), and job B also has 4 pods held, nobody moves forward. Volcano is doing its job properly: it places none of them until there is room for all 8. But if the queues’ nominalQuota values are badly sized against the cluster’s real capacity, this produces indefinite waits.

Solution: size the queue quotas so that the sum of guarantee does not exceed the real capacity, and so that the minMember values of the active jobs fit inside the available quota. Node autoscaling with ProvisioningRequest (Kueue + cluster-autoscaler) helps, but it introduces provisioning latency that has to be taken into account in the job’s SLA.

2. Quota versus real capacity: the silent drift

Kueue’s nominalQuota and Volcano’s guarantee are administrative declarations. They do not guarantee that the nodes with those GPUs are available, healthy or that the device plugin has registered them correctly. A node in NotReady with 4 GPUs reduces the real capacity without Kueue knowing: the ClusterQueue will carry on admitting workloads that then cannot be placed.

Recommended monitoring: cross Kueue’s metrics (kueue_admitted_workloads_total, kueue_pending_workloads) with the cluster’s real capacity metrics (GPUs registered in the device plugin) to detect the drift. Kueue exposes native Prometheus metrics; so does Volcano.

3. GPU resource naming: MIG, time-slicing and ResourceFlavor

If the cluster uses MIG, the resources in the pod specs change from nvidia.com/gpu to nvidia.com/mig-Xg.Ygb (for example, nvidia.com/mig-3g.40gb). Kueue’s ResourceFlavors and Volcano’s Queues must declare the right resource, or the quota will not match the pods. With time-slicing, the resource is still nvidia.com/gpu but the device plugin advertises more instances than there are physical GPUs; the quota is expressed in virtual replicas, which can lead to over-admission if the VRAM budget is not taken into account (see sharing a GPU: time-slicing, MPS and MIG).

4. Preemption in production: the evicted workload loses progress

When Kueue or Volcano evicts a training job that has been running for hours, that job loses its progress if it has no checkpointing configured. The preemption is correct from the quota point of view, but it destroys work if the job is not prepared for it. Before enabling aggressive preemption, verify that all training jobs have periodic checkpointing with automatic restoration. PyTorch plus Torchrun have native support; so does DeepSpeed. Stateless batch inference jobs do not have this problem.

5. Volcano as the sole scheduler versus coexistence with the kube-scheduler

Volcano can be configured as the cluster’s default scheduler (all pods go through it) or as an alternative scheduler (only pods with schedulerName: volcano). The first option simplifies the configuration but breaks system pods that assume kube-scheduler behaviour. The second, the recommended one, requires ML jobs to explicitly set schedulerName: volcano, which can be a non-trivial operator or chart change for existing workloads. Kueue solves this more transparently: it only requires a label on the job, with no change of scheduler.

6. Real operational complexity in 2026

Running Kueue plus Volcano plus the Training Operator plus the GPU Operator in production means four components with their own CRDs, webhooks, versions and release cycles. A Kubernetes upgrade may require updating all four in sequence. The operational debt is real. For a small team without the capacity to maintain this stack, a managed Kubernetes provider (GKE with native Kueue, OpenShift with the Red Hat build of Kueue) may be more pragmatic than assembling the full stack from scratch.

The sig-scheduler coscheduling plugin is a deliberately simpler option when all you need is gang: fewer features, less complexity, fewer things to maintain.

Sources

Volcano — Introducción y features (CNCF incubating, v1.15) — https://volcano.sh/en/docs/
Volcano — Volcano v1.11 Released: A New Era of Cloud-Native Scheduling (CNCF blog, marzo 2025) — https://www.cncf.io/blog/2025/03/05/volcano-v1-11-released-a-new-era-of-cloud-native-scheduling-for-ai-and-big-data/
Volcano — Release v1.11.0 (GitHub) — https://github.com/volcano-sh/volcano/releases/tag/v1.11.0
Volcano — Network Topology Aware Scheduling design doc — https://github.com/volcano-sh/volcano/blob/master/docs/design/Network%20Topology%20Aware%20Scheduling.md
Volcano — NUMA-aware scheduling design — https://github.com/volcano-sh/volcano/blob/master/docs/design/numa-aware.md
Volcano — Unified Scheduling (docs oficiales) — https://volcano.sh/en/docs/unified_scheduling/
Volcano — Capacity scheduling design — https://github.com/volcano-sh/volcano/blob/master/docs/design/capacity-scheduling.md
NVIDIA Technical Blog — Practical Tips for Preventing GPU Fragmentation for Volcano Scheduler — https://developer.nvidia.com/blog/practical-tips-for-preventing-gpu-fragmentation-for-volcano-scheduler/
Kueue — Overview (kueue.sigs.k8s.io, actualizado febrero 2026) — https://kueue.sigs.k8s.io/docs/overview/
Kueue — Cluster Queue (nominalQuota, borrowingLimit, lendingLimit, preemption) — https://kueue.sigs.k8s.io/docs/concepts/cluster_queue/
Kueue — Cohort — https://kueue.sigs.k8s.io/docs/concepts/cohort/
Kueue — Fair Sharing — https://kueue.sigs.k8s.io/docs/concepts/fair_sharing/
Kueue — Topology Aware Scheduling — https://kueue.sigs.k8s.io/docs/concepts/topology_aware_scheduling/
Kueue — Preemption — https://kueue.sigs.k8s.io/docs/concepts/preemption/
Kueue — Setup All-or-nothing with ready Pods — https://kueue.sigs.k8s.io/docs/tasks/manage/setup_wait_for_pods_ready/
Kueue — GitHub (kubernetes-sigs/kueue) — https://github.com/kubernetes-sigs/kueue
Ray — Gang Scheduling with RayJob and Kueue — https://docs.ray.io/en/latest/cluster/kubernetes/examples/rayjob-kueue-gang-scheduling.html
Kubeflow — Volcano scheduler integration — https://www.kubeflow.org/docs/components/trainer/operator-guides/job-scheduling/volcano/
kubernetes-sigs/scheduler-plugins — Coscheduling plugin README — https://github.com/kubernetes-sigs/scheduler-plugins/blob/master/pkg/coscheduling/README.md
kubernetes/enhancements — KEP-583 Coscheduling (sig-scheduling) — https://github.com/kubernetes/enhancements/tree/master/keps/sig-scheduling/583-coscheduling
InfraCloud — Batch Scheduling on Kubernetes: Comparing YuniKorn, Volcano and Kueue — https://www.infracloud.io/blogs/batch-scheduling-on-kubernetes/
Red Hat — Red Hat build of Kueue (OpenShift 4.20) — https://docs.redhat.com/en/documentation/openshift_container_platform/4.20/html/ai_workloads/red-hat-build-of-kueue
AceCloud — Multi GPU Orchestration in Kubernetes 2026: Kueue, Volcano, DRA — https://acecloud.ai/blog/multi-gpu-orchestration-kubernetes/
CloudOptimo — Kubernetes AI Infrastructure in 2026: GPU Scheduling and Production Realities — https://www.cloudoptimo.com/blog/kubernetes-ai-infrastructure-in-2026-gpu-scheduling-and-production-realities/

GPU chargeback and showback in multi-tenancy: how to split cluster cost between teams

Mon, 15 Jun 2026 04:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The dollar sign is not used (on this site it is a formula delimiter).

TL;DR

A cluster of 4×H100 SXM at ~1.40 €/GPU-hour (depreciated capex plus energy) shared between three teams with different occupancy produces a monthly chargeback report of 3 rows and an additional idle row that nobody claimed. Without tooling, that idle cost vanishes, diluted into the total. With OpenCost + LiteLLM + Kueue, attribution operates on three orthogonal planes: the iron (OpenCost, €/GPU-hour per namespace/label), token consumption (LiteLLM, €/token per key/team/model) and the scheduler quota (Kueue, GPUs reserved per ClusterQueue). Crossing the three produces the number that goes to finance: team B consumed X million tokens at Y €/1M tok, the GPU cost it Z €, and it still has 2 GPUs borrowed from the cohort that will cost it W € if it holds on to them next month.

Showback vs chargeback: the FinOps Foundation definition

The distinction is not one of technology but of accounting formality (FinOps Foundation — Invoicing & Chargeback, FinOps Foundation — Data Analysis and Showback):

Concept	Definition	Moves money	Requires
Showback	visibility of consumption and its cost per team; the report arrives, the budget does not change	No	metrics + attribution
Chargeback	the cost is transferred to the team’s or product’s P&L as real expenditure	Yes	showback + accounting policy + financial system

Two points from the framework worth pinning down:

Showback is a requirement of any FinOps practice; chargeback is optional and depends on the organisation’s accounting policy supporting transfers between cost centres.
Neither is “more mature” than the other. The narrative that chargeback is the “grown-up” version is false according to the framework itself. The natural sequence is: showback → trust in the data → chargeback if policy allows it.

When to use each

Situation	Recommended mode
First attribution cycle; teams do not trust the data yet	Showback
Teams with their own budget in a financial system	Chargeback
Shared cluster with no P&L separation per team	Showback with visible idle cost
Teams with a committed GPU availability SLA	Chargeback (real reservation)
Research/experimentation workloads with no formal budget	Showback + threshold alert

GPU cost attribution with OpenCost

Allocation API: attribution parameters

The OpenCost /allocation API is the piece that turns Prometheus metrics into a cost report per dimension (OpenCost — API):

Parameter	Values	Use in multi-tenancy
`window`	`today`, `7d`, `lastmonth`, RFC3339 range	window of the monthly report
`aggregate`	`namespace`, `label:LABEL`, `annotation:NAME`, `pod`, `controller`	attribution dimension
`includeIdle`	`true` / `false`	adds an `__idle__` row to the report
`shareIdle`	`true` / `false`	distributes idle among the non-idle allocations (proportional to non-idle cost)
`idleByNode`	`true` / `false`	computes idle per node instead of per cluster
`filter`	`namespace:"llm-prod"`, `label:equipo:"datos"`	restrict to one specific team

Query for a monthly report per team, with idle visible as a separate row:

curl -G http://localhost:9003/allocation \
 -d window=lastmonth \
 -d aggregate=label:equipo \
 -d includeIdle=true \
 -d shareIdle=false \
 -d resolution=10m

With shareIdle=false (the default), OpenCost returns idle as a separate __idle__ entry, which makes visible how much was paid for unused capacity. With shareIdle=true, that idle is distributed among the tenants in proportion to their non-idle cost: each absorbs its share of the waste, which is the right behaviour for a chargeback that penalises whoever is responsible for the idle.

Labelling inference pods

For aggregate=label:equipo to work, the inference pods need the right label. Two methods:

Label in the pod spec (the most direct):

# vLLM Deployment for the datos team
metadata:
 labels:
 equipo: datos
 producto: rag-prod
 modelo: llama-70b
 entorno: prod

Annotation (when the label is already taken by another convention):

# aggregate as an annotation
aggregate=annotation:finops.io/equipo

OpenCost supports aggregate=label:KEY and aggregate=annotation:KEY with the same syntax; the choice between the two depends on the cluster’s labelling convention.

Shared costs and idle: the three modes

API mode	Behaviour	When
`includeIdle=false` (default)	Ignores idle; total cost looks lower than it is	Never in production
`includeIdle=true, shareIdle=false`	Idle in a separate `__idle__` row	Showback: visibility of waste
`includeIdle=true, shareIdle=true`	Idle distributed among tenants proportionally	Chargeback: the team pays its share of idle
`shareIdle=true, idleByNode=true`	Idle distributed per node (finer-grained if there are dedicated nodes)	Chargeback with heterogeneous nodes

For infrastructure with nodes dedicated per team, idleByNode=true is fairer: the idle of a node dedicated to team A is not transferred to team B.

LiteLLM as the attribution point for token consumption

Virtual keys and teams

LiteLLM materialises token chargeback with four mechanisms (LiteLLM — Virtual Keys, LiteLLM — Setting Team Budgets, LiteLLM — Spend Tracking):

Mechanism	What it does
Virtual key with `max_budget`	monthly budget per key; the key is blocked once it is exhausted
`budget_duration`	reset window: `30d`, `7d`, `24h`; the proxy runs a daily cron that resets according to the duration
`team_id`	groups keys; spend accumulates in `LiteLLM_TeamTable` per team
`tags`	tag each request; they allow budgets per tag (cost centre)
Spend logs	one row per request with `team_id`, `model`, `prompt_tokens`, `completion_tokens`, `response_cost`

Configuring teams with a monthly budget

# litellm-config.yaml — per-team budgets with the declared on-prem cost
model_list:
 - model_name: llama-3-70b-onprem
 litellm_params:
 model: openai/llama-3-70b
 api_base: http://vllm-svc:8000/v1
 input_cost_per_token: 0.00000140 # 1.40 €/1M tok (€/GPU-hour ÷ throughput)
 output_cost_per_token: 0.00000140
 - model_name: llama-3-8b-onprem
 litellm_params:
 model: openai/llama-3-8b
 api_base: http://vllm-8b-svc:8000/v1
 input_cost_per_token: 0.00000035 # 0.35 €/1M tok (1 GPU, higher throughput)
 output_cost_per_token: 0.00000035

litellm_settings:
 success_callback: ["langfuse"] # or any external logger

# Teams (via the /team/new API or config)
# POST /team/new
# {
# "team_alias": "equipo-datos",
# "max_budget": 800, # 800 € per month
# "budget_duration": "30d",
# "tpm_limit": 500000, # max tokens/min
# "rpm_limit": 1000 # max requests/min
# }

The input_cost_per_token key comes from the identity in article A4: the €/GPU-hour from OpenCost divided by the benchmark throughput. With that value, the response_cost of each request reflects the real cost of the on-prem iron for that model.

Endpoints for querying spend per team

# Team spend accumulated over the period
GET /team/info?team_id=equipo-datos

# Team spend per model (aggregated spend_logs table)
GET /global/spend/logs?team_id=equipo-datos&start_date=2026-06-01&end_date=2026-06-30

# Total spend across all teams
GET /global/spend/teams

The response includes spend (accumulated cost in the configured currency), total_tokens, prompt_tokens, completion_tokens and a per-model breakdown, the direct fields for the monthly chargeback report.

Joining €/GPU-hour (iron) with tokens per team

The two attribution planes, OpenCost (iron) and LiteLLM (tokens), are crossed with one key: the model_name and the namespace. The join happens outside the tools (in a BI pipeline or a SQL query over the LiteLLM database and the data exported from OpenCost):

coste_por_token_real = OpenCost.GPU_cost_namespace / LiteLLM.total_tokens_team

When the on-prem cost is properly declared in input_cost_per_token, LiteLLM already does this calculation internally and the response_cost is correct. The explicit join serves to validate: if the sum of LiteLLM’s response_cost per team does not match the GPU cost OpenCost assigns to the namespace, there is drift in input_cost_per_token that has to be corrected (changed throughput, a new optimisation, more replicas).

Kueue as the GPU budget/quota mechanism

ClusterQueue, LocalQueue and cohorts

Kueue introduces the quota-based scheduling layer on top of Kubernetes (Kueue — Cluster Queue, Kueue — Cohort, Kueue — Fair Sharing):

Object	Scope	What it does
`ResourceFlavor`	cluster	maps resources to a group of nodes (e.g. H100 nodes)
`ClusterQueue`	cluster	defines `nominalQuota`, `borrowingLimit`, `lendingLimit` per resource/flavor
`LocalQueue`	namespace	entry point for the team’s workloads; points to a `ClusterQueue`
`Cohort`	cluster	groups ClusterQueues that can lend quota to each other

Quota concepts

nominalQuota: GPUs guaranteed to the ClusterQueue at all times.
borrowingLimit: maximum additional GPUs it can borrow from the cohort when others are not using them.
lendingLimit: GPUs from its nominalQuota that it allows to be lent to others (if not specified, it can lend all unused ones).
Fair Sharing: a mechanism that orders pending workloads by the historical resource usage of their LocalQueue, giving preference to whoever has consumed less. Compatible with hierarchical cohorts since Kueue v0.11.

Example YAML: 3 teams in the llm-platform cohort

# ResourceFlavor mapping to the H100 SXM nodes
apiVersion: kueue.x-k8s.io/v1beta2
kind: ResourceFlavor
metadata:
 name: h100-sxm
spec:
 nodeLabels:
 accelerator: h100-sxm
---
# ClusterQueue for the datos team — 2 guaranteed GPUs, can borrow up to 2
apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata:
 name: cq-datos
spec:
 cohort: llm-platform
 namespaceSelector:
 matchLabels:
 kubernetes.io/metadata.name: ns-datos
 queueingStrategy: BestEffortFIFO
 resourceGroups:
 - coveredResources: ["nvidia.com/gpu"]
 flavors:
 - name: h100-sxm
 resources:
 - name: "nvidia.com/gpu"
 nominalQuota: 2
 borrowingLimit: 2 # can use up to 4 GPUs in total
 preemption:
 reclaimWithinCohort: LowerPriority
 withinClusterQueue: LowerPriority
---
# ClusterQueue for the ia team — 1 guaranteed GPU, lends what it does not use
apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata:
 name: cq-ia
spec:
 cohort: llm-platform
 namespaceSelector:
 matchLabels:
 kubernetes.io/metadata.name: ns-ia
 resourceGroups:
 - coveredResources: ["nvidia.com/gpu"]
 flavors:
 - name: h100-sxm
 resources:
 - name: "nvidia.com/gpu"
 nominalQuota: 1
 lendingLimit: 1 # lends its GPU when it is not using it
---
# ClusterQueue for the plataforma team — 1 guaranteed GPU, low priority
apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata:
 name: cq-plataforma
spec:
 cohort: llm-platform
 namespaceSelector:
 matchLabels:
 kubernetes.io/metadata.name: ns-plataforma
 resourceGroups:
 - coveredResources: ["nvidia.com/gpu"]
 flavors:
 - name: h100-sxm
 resources:
 - name: "nvidia.com/gpu"
 nominalQuota: 1
 borrowingLimit: 3 # can use all 4 if the rest are free
---
# LocalQueue in each namespace — entry point for the workloads
apiVersion: kueue.x-k8s.io/v1beta2
kind: LocalQueue
metadata:
 name: lq-datos
 namespace: ns-datos
spec:
 clusterQueue: cq-datos

How Kueue materialises the “GPU budget”

FinOps concept	Kueue mechanism
Guaranteed budget	`nominalQuota`: the team always has these GPUs available
Maximum spend limit	`nominalQuota + borrowingLimit`: absolute ceiling of admissible GPUs
Lending idle capacity	cohort + `lendingLimit`: others take what this one does not use
Fair sharing between teams	`Fair Sharing` + `WorkloadPriorityClass`: those who have consumed most wait longest
Reclaiming your own quota	`preemption.reclaimWithinCohort: LowerPriority`: the owner of the lent GPU reclaims it by evicting lower-priority workloads

The nominalQuota is the expression of the budget in GPUs: if the team has 2 nominal GPUs and the price is 1.40 €/GPU-hour, its maximum guaranteed spend is

\[ \text{maximum nominal spend} = 2 \times 1.40 \times 720 = 2{,}016 \text{ €/month} \]

The borrowingLimit sets the potential overspend if it borrows from the cohort (and that borrowing can be charged with the same formula, multiplying the borrowed usage hours by 1.40 €/GPU-hour).

Cost model: an example with 4×H100 SXM and 3 teams

Node price

Generic hardware: a server with 4×H100 SXM 80 GB. 2026 market prices: a single H100 SXM costs between 27,000 and 40,000 € depending on the source (GMI Cloud — H100 GPU Pricing 2026); a 4×H100 server sits in the 120,000–180,000 € range including chassis, power supplies and NVLink. We use 140,000 € as a generic assumption for the complete server (4 GPUs plus infrastructure).

Component	Calculation	€/hour node
Depreciated capex (140,000 €, 4 years, 90 % availability)	140,000 ÷ (4 × 8,760 × 0.9)	~4.43
Energy (4 × 700 W TDP × PUE 1.4 × 0.12 €/kWh)	4 × 0.7 × 1.4 × 0.12	~0.47
Operations / networking / maintenance	estimate	~0.70
Total 4×H100 node		~5.60 €/h
Per GPU	5.60 ÷ 4	~1.40 €/GPU-hour

Cost-per-token formula for the 70B model (sustained throughput ~2,000 tok/s at TP=4):

$$\text{€/1M tokens} = \frac{5.60 \text{ €/h}}{2{,}000 \text{ tok/s} \times 3{,}600 \text{ s/h}} \times 10^6 \approx 0.78 \text{ €/1M tokens}$$

Monthly scenario: 3 teams, heterogeneous utilisation

Reference month (720 hours). The 4 GPUs are nominally assigned: 2 to Datos, 1 to IA, 1 to Plataforma. Real occupancy varies:

Team	Nominal GPUs	Average occupancy	GPU-hours used	Idle GPU-hours	GPU cost (€)
Datos	2	75 %	1,080	360	~1,512
IA	1	60 %	432	288	~605
Plataforma	1	35 %	252	468	~353
Cluster idle	—	—	—	1,116	~1,562
Node total	4	—	1,764	1,116	~4,032

Total node cost per month: 5.60 €/h × 720 h = 4,032 €. Cost per GPU-hour used: 5.60 ÷ 4 = 1.40 €.

With shareIdle=false (showback), the 1,562 € of idle appears as a separate row in the report and nobody pays for it directly. With shareIdle=true (proportional chargeback), it is split among the three in proportion to their non-idle cost: Datos absorbs ~857 €, IA ~333 €, Plataforma ~192 € of additional idle.

Token consumption per team (LiteLLM spend logs)

Team	Model	Tokens/month	Cost/token	Token cost (€)
Datos	llama-70b (TP=4)	380 M	0.78 €/1M	~296
IA	llama-8b (1 GPU)	210 M	0.35 €/1M	~74
Plataforma	llama-8b + batch	80 M	0.35 €/1M	~28

Monthly chargeback report (showback + tokens)

Team	Iron GPU cost (€)	Assigned idle cost (€)	LiteLLM token cost (€)	Total chargeable (€)
Datos	1,512	857	296	2,665
IA	605	333	74	1,012
Plataforma	353	192	28	573
Idle (without shareIdle)	1,562	—	—	visible, not charged

Attribution dimensions × policy table

Attribution dimension	Tool	Showback	Chargeback
namespace	OpenCost `aggregate=namespace`	`/allocation?aggregate=namespace&includeIdle=true`	`shareIdle=true` + transfer to P&L
pod label	OpenCost `aggregate=label:equipo`	report per label	same with shareIdle
virtual key / team	LiteLLM `/global/spend/teams`	token dashboard	hard `max_budget` per key
ClusterQueue (reserved GPU)	Kueue `nominalQuota`	observe usage vs quota	committed budget in GPUs
request tag	LiteLLM `tag_budgets`	per cost centre	budget per tag
pod annotation	OpenCost `aggregate=annotation:KEY`	report per annotation	same with shareIdle

End-to-end data flow

When the Datos team has exhausted its 2 nominal GPUs and borrows those of Plataforma (which is not using them), Kueue records that loan. As soon as Plataforma launches a new workload:

Kueue detects that cq-plataforma is below its nominalQuota.
With preemption.reclaimWithinCohort: LowerPriority, Kueue evicts the Datos workload that was on the borrowed GPU, if its priority is lower.
The evicted workload returns to the cq-datos queue and is readmitted once quota frees up.

Fair Sharing orders the pending queue by the LocalQueue’s accumulated historical usage: if Datos has consumed more GPU-hours than IA in the recent period, IA’s next workloads have admission priority over new ones from Datos. This implements an equitable split without blocking anyone permanently.

Configuring OpenCost for the node price

For the numbers in the previous section to be reproducible, the node price has to be declared explicitly (the default value underestimates on-prem GPU, issue #3781):

# OpenCost Helm values.yaml — on-prem 4×H100 node, 5.60 €/h
opencost:
 customPricing:
 enabled: true
 provider: custom
 costModel:
 description: "On-prem 4xH100 SXM node"
 CPU: "0.025" # €/CPU-hour (minor component)
 RAM: "0.003" # €/GB-hour
 GPU: "1.40" # €/GPU-hour ← the one that drives the split
 storage: "0.0002" # €/GB-hour

Verify after configuring:

# Check the price resolved per node
curl http://localhost:9003/allNodePricing

# Attribution query per team label, last month, idle separate
curl -G http://localhost:9003/allocation \
 -d window=lastmonth \
 -d aggregate=label:equipo \
 -d includeIdle=true \
 -d shareIdle=false \
 -d accumulate=true

Reference PromQL for chargeback dashboards

# GPU cost assigned per team (label:equipo) — €/hour
sum by (label_equipo) (
 container_gpu_allocation
 * on(node) group_left node_gpu_hourly_cost
)

# Idle GPU-hours per node (idle > 15 min)
sum by (node) (
 (1 - avg_over_time(DCGM_FI_DEV_GPU_UTIL[15m]) / 100)
 * on(node) group_left node_gpu_hourly_cost
)

# Accumulated monthly cost per team (sum over window)
sum_over_time(
 sum by (label_equipo) (
 container_gpu_allocation * on(node) group_left node_gpu_hourly_cost
 )[30d:1h]
)

Sources

FinOps Foundation — Invoicing & Chargeback Capability — https://www.finops.org/framework/capabilities/invoicing-chargeback/
FinOps Foundation — Data Analysis and Showback — https://www.finops.org/framework/previous-capabilities/analysis-showback/
FinOps Foundation — Allocation Capability — https://www.finops.org/framework/capabilities/allocation/
OpenCost — API (Allocation API, parámetros window/aggregate/shareIdle/idleByNode) — https://opencost.io/docs/integrations/api/
OpenCost — API Examples — https://opencost.io/docs/integrations/api-examples/
OpenCost — GitHub (issue #3781, infra-precio GPU on-prem por defecto) — https://github.com/opencost/opencost/issues/3781
LiteLLM — Virtual Keys (max_budget, budget_duration, team_id) — https://docs.litellm.ai/docs/proxy/virtual_keys
LiteLLM — Setting Team Budgets — https://docs.litellm.ai/docs/proxy/team_budgets
LiteLLM — Spend Tracking — https://docs.litellm.ai/docs/proxy/cost_tracking
LiteLLM — Budgets & Rate Limits — https://docs.litellm.ai/docs/proxy/users
LiteLLM — Setting Tag Budgets — https://docs.litellm.ai/docs/proxy/tag_budgets
LiteLLM — Budget Reset Times — https://docs.litellm.ai/docs/proxy/budget_reset_and_tz
Kueue — Cluster Queue (nominalQuota, borrowingLimit, lendingLimit, preemption) — https://kueue.sigs.k8s.io/docs/concepts/cluster_queue/
Kueue — Cohort — https://kueue.sigs.k8s.io/docs/concepts/cohort/
Kueue — Fair Sharing — https://kueue.sigs.k8s.io/docs/concepts/fair_sharing/
Kueue — Administer Cluster Quotas — https://kueue.sigs.k8s.io/docs/tasks/manage/administer_cluster_quotas/
GMI Cloud — NVIDIA H100 GPU Pricing 2026 — https://www.gmicloud.ai/en/blog/nvidia-h100-gpu-pricing-2026-rent-vs-buy-cost-analysis
IntuitionLabs — NVIDIA AI GPU Prices H100 Cost Guide — https://intuitionlabs.ai/articles/nvidia-ai-gpu-pricing-guide

Kueue on lo0 — Tech Blog

Volcano and Kueue: gang scheduling, queues and GPU quotas for distributed workloads on Kubernetes

TL;DR

The analogy

The problem neither solves by default: the kube-scheduler

Volcano: the batch scheduler

What it is and what it replaces

Gang scheduling via PodGroup

Queue: queues with quotas and priority

Scheduler plugins: DRF, binpack, topology-aware

Network and NUMA topology (v1.11+)

GPU virtualization in v1.11+

What it replaces or adds to the default scheduler

Kueue: the queue and quota manager

What it is and what it does NOT do

The four core objects

Fair sharing and preemption

Gang semantics in Kueue: all-or-nothing with ready Pods

Topology-Aware Scheduling (TAS)

Framework integrations

The key distinction: Volcano places, Kueue admits

How they coexist: the production pattern

The third way: the sig-scheduler coscheduling plugin

Complete YAML examples

Volcano: Queue + PodGroup + VolcanoJob

Kueue: ResourceFlavor + ClusterQueue + LocalQueue + annotated Job

Kueue + Volcano together: PyTorchJob with Kueue quota and Volcano gang placement

For LLM workloads: training, fine-tuning and batch inference

Distributed multi-GPU training and fine-tuning

Batch inference and evaluations (evals)

Multi-tenant GPU quota and chargeback

Diagram: the flow of a training job with Kueue + Volcano

Full comparison table

Operational pitfalls and honest scepticism

1. Deadlock from a badly configured gang

2. Quota versus real capacity: the silent drift

3. GPU resource naming: MIG, time-slicing and ResourceFlavor

4. Preemption in production: the evicted workload loses progress

5. Volcano as the sole scheduler versus coexistence with the kube-scheduler

6. Real operational complexity in 2026

See also

Sources

GPU chargeback and showback in multi-tenancy: how to split cluster cost between teams

TL;DR

Showback vs chargeback: the FinOps Foundation definition

When to use each

GPU cost attribution with OpenCost

Allocation API: attribution parameters

Labelling inference pods

Shared costs and idle: the three modes

LiteLLM as the attribution point for token consumption

Virtual keys and teams

Configuring teams with a monthly budget

Endpoints for querying spend per team

Joining €/GPU-hour (iron) with tokens per team

Kueue as the GPU budget/quota mechanism

ClusterQueue, LocalQueue and cohorts

Quota concepts

Example YAML: 3 teams in the llm-platform cohort

How Kueue materialises the “GPU budget”

Cost model: an example with 4×H100 SXM and 3 teams

Node price

Monthly scenario: 3 teams, heterogeneous utilisation

Token consumption per team (LiteLLM spend logs)

Monthly chargeback report (showback + tokens)

Attribution dimensions × policy table

End-to-end data flow

Fair sharing and preemption: how Kueue reclaims quota

Configuring OpenCost for the node price

Reference PromQL for chargeback dashboards

Sources

See also