Volcano and Kueue: gang scheduling, queues and GPU quotas for distributed workloads on Kubernetes

Tue, 16 Jun 2026 13:00:00 +0200

TL;DR

Volcano (volcano-sh, CNCF incubating) is a complete batch scheduler that replaces or complements the kube-scheduler: it places pods with gang semantics (all-or-nothing via PodGroup/minMember), manages queues with priority, DRF fair-share and preemption between queues, and understands network and NUMA topology.

Kueue (kubernetes-sigs/kueue) is a queue and quota manager at the Job level: it does NOT place pods (it delegates to the kube-scheduler or to Volcano), but it decides when a workload can be admitted according to available quota (ClusterQueue/LocalQueue/Cohort), with fair sharing, borrowing between teams and preemption by priority. It natively integrates Job, JobSet, RayJob, all the Kubeflow operators and more.

The winning combination in production for multi-tenant GPU workloads is: Kueue for quota and queues + Volcano (or the sig-scheduler coscheduling plugin) for the job’s gang.

The analogy

Imagine a club with limited capacity and a dance floor inside.

Kueue is the doorman and the booking manager: it checks whether the team’s quota (its reserved capacity) allows the group in, applies the fair waiting list, borrows capacity from other teams if any are idle, and takes the space back when the owner needs it. But the doorman does not decide where each person sits inside the venue.

Volcano is the head waiter: once the group has permission to come in, he decides which tables they sit at, makes sure the whole group sits down at once or nobody comes in (gang), picks the tables according to topology (who needs to talk to whom) and throws out lower-priority groups to make room if necessary.

Without a doorman (Kueue), the head waiter does not know how many groups he can take at once or whether a team is exceeding its capacity. Without a head waiter (Volcano), the doorman lets the group in but its members scatter across the available tables on their own, and the party of 8 that needs to sit together never manages it.

The problem neither solves by default: the kube-scheduler

Kubernetes’ kube-scheduler is a pod scheduler, not a job scheduler. It assigns pods one by one to the most suitable node according to available resources and affinity constraints. For a distributed training workload that needs, say, 8 pods with 4 GPUs each (32 GPUs in total across 8 nodes of 4×H100), the standard scheduler does the following:

It looks for a node with 4 available GPUs. It finds one. It schedules pod 1.
It looks for another node with 4 GPUs. It finds one. It schedules pod 2.
It carries on until it reaches pod 6 and it turns out there are no longer any nodes with 4 free GPUs: the cluster has exactly 32 GPUs and other workloads are using some of them.
Pods 1–5 are Running. Pods 6–8 are Pending.
Pods 1–5 can do nothing without the others: a distributed PyTorch job needs all the workers to start before the torchrun process can begin. It waits with the resources occupied. Deadlock.

This is not a bug, it is the design: the kube-scheduler has no concept of “schedule this group of pods only if you can schedule all of them”. As a result:

The resources of pods 1–5 are locked up without producing work.
Other jobs that could run with the partial resources also wait.
If several jobs are in this situation, the cluster can end up with fragmented resources, no job running and everyone in circular deadlock.

On top of that, the kube-scheduler has no notion of:

Queues per team or project with relative priority.
Quotas of resources per team with the ability to borrow from idle teams.
Fair-share: if team A has spent weeks using 80 % of the cluster, it should wait longer than team B, which has spent weeks idle.
Inter-queue preemption: evicting a lower-priority job from another team to make room for this team’s urgent job.

Solving any of these problems requires adding a layer on top of the scheduler. Volcano and Kueue are the two dominant OSS solutions in 2026, with complementary architectural approaches.

Volcano: the batch scheduler

What it is and what it replaces

Volcano (volcano-sh) is a Kubernetes-native batch scheduler accepted by the CNCF as its first and only official container batch scheduling project (volcano.sh/en/docs). At version v1.15.x as of June 2026, with CNCF incubating status.

Volcano is not an addon to the kube-scheduler: it is an alternative scheduler (or a complementary one, depending on the configuration) that places pods. It is installed as a deployment, and jobs that want to benefit from its capabilities must use the volcano scheduler class in their pod spec (schedulerName: volcano) or use the VolcanoJob CRD.

Volcano’s central value proposition is that it treats groups of pods as atomic scheduling units, not individual pods. That is what makes it possible to solve the deadlock described above.

Gang scheduling via PodGroup

Volcano’s central mechanism is the PodGroup: a CRD that groups a job’s pods and defines how many must be schedulable before Volcano starts any of them.

# PodGroup for a distributed PyTorch training job: 8 workers, minimum 8
apiVersion: scheduling.volcano.sh/v1beta1
kind: PodGroup
metadata:
 name: pytorch-train-pg
 namespace: ml-training
spec:
 minMember: 8 # all-or-nothing: if there is no room for 8, none starts
 minResources:
 nvidia.com/gpu: "32" # 8 pods × 4 GPUs = 32 GPUs minimum in the cluster
 queue: team-datos  # which Volcano queue this job is assigned to
 priorityClassName: high-priority

The minMember parameter implements all-or-nothing semantics: Volcano only assigns nodes to the group’s pods when it can assign at least minMember pods simultaneously. If the cluster does not have capacity for 8 GPU pods at this moment, no pod in the group moves out of Pending. Nothing is blocked, nothing is fragmented.

minMember can be lower than the job’s total pod count: this allows elastic gang scheduling, where the job can start with fewer workers and scale up, useful for jobs that tolerate a reduced worker count.

For a job’s pods to be associated with the PodGroup, they carry the annotation:

# Pod spec of the PyTorch worker
metadata:
 annotations:
 scheduling.volcano.sh/pod-group-name: pytorch-train-pg
spec:
 schedulerName: volcano
 containers:
 - name: trainer
 image: pytorch/pytorch:2.5-cuda12.4
 resources:
 limits:
 nvidia.com/gpu: "4"

Queue: queues with quotas and priority

Volcano introduces the Queue CRD to manage multiple tenants with independent quotas:

apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
 name: team-datos
spec:
 weight: 4 # relative weight for fair-share between queues (proportion plugin)
 capability: # absolute ceiling of resources this queue can use
 nvidia.com/gpu: "16"
 guarantee: # guaranteed resource, never lent to other queues
 resource:
 nvidia.com/gpu: "8"
 reclaimable: true # if true, others can reclaim the resources it lends when they need them

apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
 name: team-ia
spec:
 weight: 2
 capability:
 nvidia.com/gpu: "8"
 guarantee:
 resource:
 nvidia.com/gpu: "4"
 reclaimable: true

The weight field feeds the proportion plugin: queues compete for the cluster’s available resources in proportion to their weight. A cluster with 32 GPUs and two queues of weight 4 and 2 splits the GPUs in a 4:2 ratio (≈21 and 11 GPUs respectively) when both are saturated.

Scheduler plugins: DRF, binpack, topology-aware

Volcano implements its scheduling logic as a pipeline of actions and plugins:

Actions (what the scheduler does in each cycle):

enqueue: moves jobs from the waiting queue to schedulable when quota is available.
allocate: assigns nodes to schedulable pods.
preempt: evicts lower-priority pods to make room for higher-priority ones within the same queue.
reclaim: evicts pods from other queues that are using more than their guarantee to return resources to the owner.
backfill: fills idle resources with best-effort jobs that do not interfere with the rest.

Plugins relevant to GPU workloads:

Plugin	What it does
`gang`	Implements the PodGroup’s all-or-nothing semantics
`proportion`	Fair-share by queue weight (proportional quota)
`capacity`	Quotas with `guarantee`/`capability` and reclaim; a more expressive alternative to proportion
`drf`	Dominant Resource Fairness: multi-dimensional fair-share (CPU, memory, GPU)
`binpack`	Packs pods onto the fullest nodes; reduces GPU fragmentation
`priority`	Orders jobs by priority within the same queue
`nodeorder`	Node scoring according to multiple criteria (affinity, resources, spread)
`task-topology`	Affinity between pods of the same job (inter-GPU communication)
`numa-aware`	NUMA affinity: aligns pods with the node’s NUMA socket to reduce memory latency

Network and NUMA topology (v1.11+)

Volcano v1.11 (February 2025) introduced Network Topology Aware Scheduling as a first-class feature (CNCF blog, March 2025). Distributed training jobs in a datacenter with a hierarchical network structure (spine/leaf, blocks of nodes with NVSwitch) can declare topology constraints:

# VolcanoJob with a network topology constraint
apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
 name: llm-pretrain
 namespace: ml-training
spec:
 minAvailable: 8
 schedulerName: volcano
 queue: team-datos
 plugins:
 ssh: []
 env: []
 svc: []
 networkTopology:
 mode: hard # hard: the job MUST satisfy the constraint
 highestTierAllowed: block  # pods cannot span beyond a network "block"
 tasks:
 - replicas: 8
 name: worker
 template:
 spec:
 containers:
 - name: trainer
 image: nvcr.io/nvidia/pytorch:25.01-py3
 resources:
 limits:
 nvidia.com/gpu: "4"

The highestTierAllowed: block semantics instruct Volcano to place the 8 pods within the same network block (for example, all the nodes under the same access switch), minimising the inter-block traffic that degrades a distributed all-reduce.

NUMA awareness works in a similar way: with the numa-aware plugin, pods request a NUMA policy (single-numa-node, restricted, best-effort) and Volcano selects nodes where the requested CPU, memory and GPU resources are in the same NUMA domain, avoiding the remote memory access overhead (NUMA-crossing) that can degrade training throughput by 15-30 % on multi-socket nodes.

GPU virtualization in v1.11+

Volcano v1.11 also introduces support for dynamic MIG and vCUDA: instead of declaring nvidia.com/gpu: 1 for a whole GPU, workloads can declare nvidia.com/gpu-memory: 20Gi and Volcano (with the corresponding device plugin) dynamically provisions the MIG instance or the vCUDA partition. This is [project marketing with no independent benchmarks published as of June 2026], but the feature’s architecture is documented in the code.

What it replaces or adds to the default scheduler

Capability	kube-scheduler	Volcano
Placing pods on nodes	Yes	Yes (it replaces it for workloads marked with `schedulerName: volcano`)
Gang scheduling (all-or-nothing)	No	Yes (PodGroup + minMember)
Queues with priority	No	Yes (Queue CRD)
Inter-queue fair-share	No	Yes (DRF, proportion, capacity plugins)
Inter-queue preemption	No	Yes (reclaim action)
Topology-aware (NUMA, network)	Partial (node affinity)	Yes (dedicated plugins)
Elastic gang (minMember < total)	No	Yes
Best-effort backfill	No	Yes
Framework integrations	Partial	MPI, PyTorch, Ray, TensorFlow, Spark, Flink, Horovod

Kueue: the queue and quota manager

What it is and what it does NOT do

Kueue (kubernetes-sigs/kueue) is a Kubernetes-native system that manages quotas and how jobs consume them (kueue.sigs.k8s.io). Kueue decides when a job must wait, when it should be admitted (pods can be created) and when it should be evicted (active pods must be deleted).

Kueue’s central design principle is explicit in its documentation: avoid duplicating mature functionality of Kubernetes components. Autoscaling is the cluster-autoscaler’s responsibility. Pod-to-node scheduling is the kube-scheduler’s responsibility. Job lifecycle management is the kube-controller-manager’s responsibility. Kueue replaces none of them: it sits on top as a layer of admission control and quota management at the Job level.

This is the fundamental distinction: Kueue does not place pods on nodes. When Kueue admits a workload, it simply allows the corresponding job controller to create the pods, and those pods are scheduled by the kube-scheduler (or by Volcano, if it is configured as the scheduler).

The four core objects

ResourceFlavor: maps abstract resources to concrete groups of physical nodes.

apiVersion: kueue.x-k8s.io/v1beta1
kind: ResourceFlavor
metadata:
 name: h100-sxm
spec:
 nodeLabels:
 accelerator: h100-sxm
 node-pool: gpu-training
 tolerations:
 - key: "nvidia.com/gpu"
 operator: "Exists"
 effect: "NoSchedule"

ClusterQueue: defines the resource quota per flavor for a tenant. A cluster-scoped object.

apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
 name: cq-team-datos
spec:
 cohort: llm-platform  # cohort it belongs to (can lend/borrow)
 queueingStrategy: BestEffortFIFO
 namespaceSelector:
 matchLabels:
 kubernetes.io/metadata.name: ns-datos
 resourceGroups:
 - coveredResources: ["nvidia.com/gpu", "cpu", "memory"]
 flavors:
 - name: h100-sxm
 resources:
 - name: "nvidia.com/gpu"
 nominalQuota: 16 # GPUs guaranteed for this team
 borrowingLimit: 8 # can take up to 8 additional GPUs from the cohort
 lendingLimit: 8 # can lend up to 8 of its 16 nominal GPUs
 - name: "cpu"
 nominalQuota: "128"
 - name: "memory"
 nominalQuota: "512Gi"
 preemption:
 reclaimWithinCohort: LowerPriority  # reclaims lent quota by evicting lower-priority jobs
 borrowWithinCohort:
 policy: LowerPriority
 withinClusterQueue: LowerPriority

LocalQueue: the namespace-scoped entry point for a team’s workloads. Jobs point to their LocalQueue; Kueue maps them to the corresponding ClusterQueue.

apiVersion: kueue.x-k8s.io/v1beta1
kind: LocalQueue
metadata:
 name: lq-datos
 namespace: ns-datos
spec:
 clusterQueue: cq-team-datos

Cohort: groups ClusterQueues that can lend quota to each other. It is not a standalone CRD; it is declared as a field in the ClusterQueue (spec.cohort: name). Kueue aggregates the available quota of all the ClusterQueues in the cohort and lets any of them borrow what the others are not using, respecting the borrowingLimit and lendingLimit.

Kueue implements Fair Sharing as the ordering policy for the queue of pending workloads (kueue.sigs.k8s.io/docs/concepts/fair_sharing): when several workloads compete for quota in the cohort, those belonging to ClusterQueues with higher accumulated historical usage have lower admission priority. This implements equitable sharing without permanently blocking any team.

Preemption in Kueue operates along two dimensions:

reclaimWithinCohort: the ClusterQueue that lends quota reclaims it by evicting workloads that are using it on loan, according to a priority policy.
withinClusterQueue: within the same ClusterQueue, lower-priority workloads are evicted to make way for higher-priority workloads from the same team.

Gang semantics in Kueue: all-or-nothing with ready Pods

Kueue provides gang admission at the Job level: it admits the complete workload only when all the necessary quota is available. If a RayJob needs 8 GPUs (1 head + 7 workers), Kueue does not admit the workload until 8 GPUs are available in the ClusterQueue (or borrowed from the cohort). waitForPodsReady with a timeout adds a second guarantee: if the created pods do not become Ready within the configured time, Kueue re-queues the workload and releases the quota (kueue.sigs.k8s.io/docs/tasks/manage/setup_wait_for_pods_ready).

This is gang semantics at the admission level, not at the pod placement level. It guarantees that the quota is available before creating the pods, but it does not guarantee that the kube-scheduler can place them all on concrete nodes at the same time. For that second guarantee you need Volcano or the coscheduling plugin.

Topology-Aware Scheduling (TAS)

Kueue v0.10+ introduces Topology-Aware Scheduling (kueue.sigs.k8s.io/docs/concepts/topology_aware_scheduling): it allows node topologies (blocks, subblocks, hosts) to be defined and workloads to request co-location levels. Kueue only admits the workload when it can satisfy the topological constraint, and it adds node selectors and taints at admission time so that the scheduler places the pods in the right topology.

TAS is configured with the Topology CRD:

apiVersion: kueue.x-k8s.io/v1beta1
kind: Topology
metadata:
 name: datacenter-topology
spec:
 levels:
 - nodeLabel: "topology.kubernetes.io/block"
 - nodeLabel: "topology.kubernetes.io/rack"
 - nodeLabel: "kubernetes.io/hostname"

And the ResourceFlavor references the topology:

apiVersion: kueue.x-k8s.io/v1beta1
kind: ResourceFlavor
metadata:
 name: h100-sxm
spec:
 nodeLabels:
 accelerator: h100-sxm
 topologyName: datacenter-topology

Framework integrations

Kueue has built-in integration (no additional code) for the following workload types, activated with an annotation on the job:

metadata:
 labels:
 kueue.x-k8s.io/queue-name: lq-datos  # points to the team's LocalQueue

The supported workload types include: batch/Job, JobSet, RayJob, RayCluster, PytorchJob, TFJob, MPIJob, JAXJob, PaddleJob, XGBoostJob, TrainJob, AppWrapper, LeaderWorkerSet, Deployment, StatefulSet, and plain Pod/PodGroup.

For LLM workloads, the directly relevant cases:

RayJob (distributed training with Ray Train): Kueue admits the RayJob when there is quota for the whole Ray cluster (head + workers). Documented at docs.ray.io.
PyTorchJob (Kubeflow Training Operator): gang admission of the complete job.
JobSet: for coordinated multi-replica jobs (LWS, multi-step pipelines).
Deployment/StatefulSet: for continuous inference, allowing inference GPU quota to be managed the same way as training quota.

The key distinction: Volcano places, Kueue admits

This table sums up the fundamental architectural difference:

Dimension	Volcano	Kueue
Main role	Scheduler (places pods on nodes)	Admission controller + quota manager (decides when to create pods)
Gang scheduling	Yes, at the placement level (PodGroup/minMember)	Yes, at the admission level (all-or-nothing on quota)
Multi-tenant quotas	Yes (Queue with capability/guarantee)	Yes (ClusterQueue with nominalQuota/borrowingLimit)
Cohorts / borrowing	Partial (reclaimable between queues)	Yes (Cohort with explicit lendingLimit/borrowingLimit)
Fair sharing	Yes (DRF plugin)	Yes (Fair Sharing based on historical usage)
Preemption	Yes (preempt + reclaim actions)	Yes (reclaimWithinCohort, withinClusterQueue)
Topology/NUMA	Yes (dedicated plugins, network topology, NUMA-aware)	Yes (TAS, topology levels in ResourceFlavor)
Framework integrations	Volcano Job (MPI, PyTorch, Ray, TF, Spark, Flink)	Native: Job, JobSet, RayJob, Kubeflow, LWS, AppWrapper, Deployment, StatefulSet
Who places the pods	Volcano	kube-scheduler (or Volcano if configured)
Installation footprint	Medium-high (its own scheduler, CRDs, webhook, metrics)	Light (controller, CRDs, webhook; does not replace the scheduler)
Maturity / status	CNCF incubating; v1.15 (June 2026); production at Huawei, Baidu, DiDi	kubernetes-sigs; API v1beta2; production at Google GKE, Red Hat OpenShift 4.20, Runway ML
Adoption curve	Steeper (requires changing schedulerName or using the VolcanoJob CRD)	Gentler (adds labels to existing jobs; does not change the scheduler)

How they coexist: the production pattern

In production, Kueue and Volcano are complementary, not mutually exclusive. The most common pattern in 2026 for multi-tenant GPU clusters is:

Kueue manages the global quota: which team can use how many GPUs, how much it can borrow, when a job goes into a queue versus being admitted.
Volcano does the gang scheduling at the pod level: once Kueue admits the job (the quota is available), Volcano places the pods, making sure they are all placed simultaneously on compatible nodes.

The integration is configured by specifying schedulerName: volcano in the pod specs of the workloads managed by Kueue. Kueue sees the Job/RayJob/PyTorchJob and manages its quota; when it admits it, the pods are created and Volcano places them with gang semantics. Volcano’s PodGroups are created automatically by the Volcano Job controller or by the Kubeflow Training Operator itself when it detects that the scheduler is Volcano.

# PyTorchJob managed by Kueue (quota) + Volcano (gang placement)
apiVersion: kubeflow.org/v1
kind: PyTorchJob
metadata:
 name: llm-finetune-70b
 namespace: ns-datos
 labels:
 kueue.x-k8s.io/queue-name: lq-datos  # Kueue manages the quota
 annotations:
 scheduling.volcano.sh/queue-name: team-datos  # Volcano uses its own Queue
spec:
 pytorchReplicaSpecs:
 Master:
 replicas: 1
 restartPolicy: OnFailure
 template:
 spec:
 schedulerName: volcano  # Volcano does the placement
 containers:
 - name: pytorch
 image: nvcr.io/nvidia/pytorch:25.01-py3
 resources:
 limits:
 nvidia.com/gpu: "4"
 Worker:
 replicas: 7
 restartPolicy: OnFailure
 template:
 spec:
 schedulerName: volcano
 containers:
 - name: pytorch
 image: nvcr.io/nvidia/pytorch:25.01-py3
 resources:
 limits:
 nvidia.com/gpu: "4"

The third way: the sig-scheduler coscheduling plugin

If you do not want to deploy a complete alternative scheduler but you need gang scheduling, there is the coscheduling plugin from kubernetes-sigs/scheduler-plugins. This plugin extends the kube-scheduler with a PodGroup mechanism similar to Volcano’s, implemented as a scheduling framework plugin (permit plugin). The advantage is that it does not replace the scheduler; the disadvantage is that it has less functionality than Volcano (no DRF, no Queue/fair-share, no network topology). It is the right option for simple clusters that only need gang and do not want Volcano’s operational complexity. Kueue can also work alongside this plugin.

Complete YAML examples

Volcano: Queue + PodGroup + VolcanoJob

# 1. Queue for the data team (16 nominal GPUs, ceiling at 24)
apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
 name: team-datos
spec:
 weight: 4
 capability:
 nvidia.com/gpu: "24"
 cpu: "192"
 memory: "768Gi"
 guarantee:
 resource:
 nvidia.com/gpu: "16"
 reclaimable: true
---
# 2. Queue for the AI team (8 nominal GPUs, ceiling at 16)
apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
 name: team-ia
spec:
 weight: 2
 capability:
 nvidia.com/gpu: "16"
 cpu: "64"
 memory: "256Gi"
 guarantee:
 resource:
 nvidia.com/gpu: "8"
 reclaimable: true
---
# 3. PodGroup: 70B fine-tuning job, 8 workers × 4 GPUs = 32 GPUs
apiVersion: scheduling.volcano.sh/v1beta1
kind: PodGroup
metadata:
 name: finetune-70b-pg
 namespace: ns-datos
spec:
 minMember: 8
 minResources:
 nvidia.com/gpu: "32"
 queue: team-datos
 priorityClassName: training-high
---
# 4. VolcanoJob (wrapper that Volcano understands natively)
apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
 name: finetune-70b
 namespace: ns-datos
spec:
 minAvailable: 8
 schedulerName: volcano
 queue: team-datos
 priorityClassName: training-high
 plugins:
 env: []
 svc: []
 policies:
 - event: PodEvicted
 action: RestartJob
 tasks:
 - replicas: 8
 name: worker
 policies:
 - event: TaskCompleted
 action: CompleteJob
 template:
 metadata:
 annotations:
 scheduling.volcano.sh/pod-group-name: finetune-70b-pg
 spec:
 schedulerName: volcano
 containers:
 - name: trainer
 image: nvcr.io/nvidia/pytorch:25.01-py3
 command: ["torchrun", "--nproc_per_node=4", "--nnodes=8",
 "--node_rank=$(RANK)", "--master_addr=$(MASTER_ADDR)",
 "--master_port=23456", "train.py"]
 env:
 - name: NCCL_DEBUG
 value: "INFO"
 resources:
 requests:
 nvidia.com/gpu: "4"
 cpu: "16"
 memory: "64Gi"
 limits:
 nvidia.com/gpu: "4"
 cpu: "16"
 memory: "64Gi"
 restartPolicy: Never

Kueue: ResourceFlavor + ClusterQueue + LocalQueue + annotated Job

# 1. ResourceFlavor: nodes with H100 SXM
apiVersion: kueue.x-k8s.io/v1beta1
kind: ResourceFlavor
metadata:
 name: h100-sxm
spec:
 nodeLabels:
 accelerator: h100-sxm
 tolerations:
 - key: "nvidia.com/gpu"
 operator: "Exists"
 effect: "NoSchedule"
---
# 2. ClusterQueue for the data team: 16 nominal GPUs, can borrow 8
apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
 name: cq-team-datos
spec:
 cohort: llm-platform
 queueingStrategy: BestEffortFIFO
 namespaceSelector:
 matchLabels:
 kubernetes.io/metadata.name: ns-datos
 resourceGroups:
 - coveredResources: ["nvidia.com/gpu", "cpu", "memory"]
 flavors:
 - name: h100-sxm
 resources:
 - name: "nvidia.com/gpu"
 nominalQuota: 16
 borrowingLimit: 8
 lendingLimit: 8
 - name: "cpu"
 nominalQuota: "128"
 - name: "memory"
 nominalQuota: "512Gi"
 preemption:
 reclaimWithinCohort: LowerPriority
 borrowWithinCohort:
 policy: LowerPriority
 withinClusterQueue: LowerPriority
---
# 3. ClusterQueue for the AI team: 8 nominal GPUs
apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
 name: cq-team-ia
spec:
 cohort: llm-platform
 queueingStrategy: BestEffortFIFO
 namespaceSelector:
 matchLabels:
 kubernetes.io/metadata.name: ns-ia
 resourceGroups:
 - coveredResources: ["nvidia.com/gpu", "cpu", "memory"]
 flavors:
 - name: h100-sxm
 resources:
 - name: "nvidia.com/gpu"
 nominalQuota: 8
 borrowingLimit: 8
 lendingLimit: 4
 - name: "cpu"
 nominalQuota: "64"
 - name: "memory"
 nominalQuota: "256Gi"
 preemption:
 reclaimWithinCohort: LowerPriority
 withinClusterQueue: LowerPriority
---
# 4. LocalQueue in the data team's namespace
apiVersion: kueue.x-k8s.io/v1beta1
kind: LocalQueue
metadata:
 name: lq-datos
 namespace: ns-datos
spec:
 clusterQueue: cq-team-datos
---
# 5. LocalQueue in the AI team's namespace
apiVersion: kueue.x-k8s.io/v1beta1
kind: LocalQueue
metadata:
 name: lq-ia
 namespace: ns-ia
spec:
 clusterQueue: cq-team-ia
---
# 6. Batch inference RayJob managed by Kueue
apiVersion: ray.io/v1
kind: RayJob
metadata:
 name: batch-eval-llama70b
 namespace: ns-datos
 labels:
 kueue.x-k8s.io/queue-name: lq-datos  # Kueue manages the admission
spec:
 entrypoint: "python batch_eval.py --model /models/llama-70b"
 rayClusterSpec:
 headGroupSpec:
 rayStartParams:
 num-gpus: "4"
 template:
 spec:
 containers:
 - name: ray-head
 image: rayproject/ray-ml:2.40.0-gpu
 resources:
 limits:
 nvidia.com/gpu: "4"
 cpu: "16"
 memory: "64Gi"
 workerGroupSpecs:
 - replicas: 3
 minReplicas: 3 # gang: Kueue does not admit if there is no quota for 3 workers
 maxReplicas: 3
 groupName: gpu-worker
 rayStartParams:
 num-gpus: "4"
 template:
 spec:
 containers:
 - name: ray-worker
 image: rayproject/ray-ml:2.40.0-gpu
 resources:
 limits:
 nvidia.com/gpu: "4"
 cpu: "16"
 memory: "64Gi"

Kueue + Volcano together: PyTorchJob with Kueue quota and Volcano gang placement

# PyTorchJob: Kueue controls the quota, Volcano does the gang placement
apiVersion: kubeflow.org/v1
kind: PyTorchJob
metadata:
 name: distributed-finetune
 namespace: ns-datos
 labels:
 kueue.x-k8s.io/queue-name: lq-datos # Kueue: quota and admission
 annotations:
 # Volcano automatically creates the PodGroup when schedulerName=volcano
 scheduling.volcano.sh/queue-name: team-datos
spec:
 pytorchReplicaSpecs:
 Master:
 replicas: 1
 restartPolicy: OnFailure
 template:
 spec:
 schedulerName: volcano # Volcano: gang placement
 containers:
 - name: pytorch
 image: nvcr.io/nvidia/pytorch:25.01-py3
 resources:
 limits:
 nvidia.com/gpu: "4"
 cpu: "16"
 memory: "64Gi"
 Worker:
 replicas: 7
 restartPolicy: OnFailure
 template:
 spec:
 schedulerName: volcano
 containers:
 - name: pytorch
 image: nvcr.io/nvidia/pytorch:25.01-py3
 resources:
 limits:
 nvidia.com/gpu: "4"
 cpu: "16"
 memory: "64Gi"

For LLM workloads: training, fine-tuning and batch inference

Distributed multi-GPU training and fine-tuning

Gang scheduling is essential for any distributed training job that uses NCCL all-reduce (PyTorch DDP, FSDP, DeepSpeed ZeRO). If a single worker in the group does not start, the torchrun coordinator waits indefinitely; with the standard kube-scheduler this scenario happens every time the cluster is under contention.

In a generic cluster of 4 nodes of 4×H100 SXM (16 GPUs in total), a fine-tuning job for a 70B model typically requires 8 GPUs in tensor-parallel 8 (TP=8) or 16 GPUs in TP=4 plus data-parallel 4. With Volcano, the PodGroup with minMember: 8 guarantees that either the 8 pods are placed at once or none blocks resources. With Kueue on top, the quota guarantees that the team does not exceed its 16 nominal GPUs and that other teams with available quota are not blocked by a waiting job.

The cross-link with inference capacity planning is direct: the model’s VRAM budget (weights plus KV-cache) determines the minimum TP and therefore the PodGroup’s minMember.

Batch inference and evaluations (evals)

Batch inference jobs, generating responses for an evaluation dataset, processing embeddings in bulk, offline re-ranking, are naturally parallel workloads that do not necessarily require strict gang scheduling (each request is independent), but they do benefit from quota and fair-share.

For these workloads, Kueue alone is enough: a batch/Job with multiple independent pods is managed with the ClusterQueue’s quota with no need for Volcano. If several teams are submitting evaluation jobs simultaneously, Kueue orders the admission by fair-share and priority, and with cohort borrowing the jobs of teams with free quota do not have to wait for the quota of busy teams.

The chargeback of these workloads connects directly with what is described in GPU chargeback and showback: the ClusterQueue’s nominalQuota is the expression of the GPU budget in Kubernetes, and OpenCost can attribute the cost per namespace or label for the monthly report.

Multi-tenant GPU quota and chargeback

The alignment between Kueue and the chargeback system is direct:

FinOps concept	Kueue mechanism
Guaranteed GPU budget	`nominalQuota` per ClusterQueue
Maximum spending ceiling	`nominalQuota + borrowingLimit`
Lending idle capacity	Cohort + `lendingLimit`
Reclaiming your own quota	`preemption.reclaimWithinCohort: LowerPriority`
Fair-share between teams	Fair Sharing policy in the ClusterQueue
Chargeback of the loan	borrowed GPU hours × cost per GPU-hour (OpenCost)

For the utilisation dimension as a FinOps lever, see GPU utilisation as a FinOps lever: Kueue and Volcano together make it possible to maximise utilisation without sacrificing the quota guarantees, which is exactly the FinOps objective.

Managing MIG partitions within this system (declaring nvidia.com/mig-4g.40gb as a resource in the ClusterQueue) integrates naturally: the ResourceFlavor can map to nodes with a specific MIG profile, as explained in sharing a GPU: time-slicing, MPS and MIG.

Diagram: the flow of a training job with Kueue + Volcano

Full comparison table

Criterion	Volcano	Kueue	Coscheduling plugin
Role	Scheduler (placement)	Admission + quota (no placement)	kube-scheduler plugin (placement)
Gang scheduling	Yes, pod level (PodGroup/minMember)	Yes, admission level (quota gang)	Yes, pod level (PodGroup)
Multi-tenant quotas	Yes (Queue capability/guarantee)	Yes (ClusterQueue nominalQuota)	No
Cohorts / borrowing	Limited (reclaimable)	Yes (Cohort with borrowingLimit/lendingLimit)	No
Fair-share	Yes (DRF, proportion)	Yes (Fair Sharing by historical usage)	No
Inter-queue preemption	Yes (reclaim action)	Yes (reclaimWithinCohort)	No
Network / NUMA topology	Yes (v1.11+, dedicated plugins)	Yes (TAS, Topology CRD)	No
Native integrations	MPI, PyTorch, Ray, TF, Spark, Flink, Horovod	Job, JobSet, RayJob, Kubeflow, LWS, AppWrapper, Deployment, StatefulSet	Any job with a PodGroup
Who places the pods	Volcano	kube-scheduler (or Volcano)	kube-scheduler
Elastic gang	Yes (minMember < total replicas)	Partial (partial admission in batch/Job)	Limited
Footprint	Medium-high (its own scheduler)	Light (an additional controller)	Minimal (a scheduler plugin)
Compatibility with Kueue	Yes (as the scheduler under Kueue)	—	Yes (a complementary option)
CNCF status / maturity	CNCF incubating, v1.15	kubernetes-sigs, v1beta2, adopted in GKE/OpenShift	kubernetes-sigs/scheduler-plugins, experimental
When to choose it	HPC-like distributed training, NUMA, network topology, MPI	Multi-tenancy with flexible quota, heterogeneous workloads, inference + batch together	Simple clusters that only need gang without their own scheduler

Operational pitfalls and honest scepticism

1. Deadlock from a badly configured gang

The most frequent scenario: minMember set equal to the total replica count in a cluster where several jobs compete for the same resources. If two jobs of 8 pods each try to use a cluster with 8 GPU nodes and job A has 4 pods placed (not 8, so Volcano holds them pending), and job B also has 4 pods held, nobody moves forward. Volcano is doing its job properly: it places none of them until there is room for all 8. But if the queues’ nominalQuota values are badly sized against the cluster’s real capacity, this produces indefinite waits.

Solution: size the queue quotas so that the sum of guarantee does not exceed the real capacity, and so that the minMember values of the active jobs fit inside the available quota. Node autoscaling with ProvisioningRequest (Kueue + cluster-autoscaler) helps, but it introduces provisioning latency that has to be taken into account in the job’s SLA.

2. Quota versus real capacity: the silent drift

Kueue’s nominalQuota and Volcano’s guarantee are administrative declarations. They do not guarantee that the nodes with those GPUs are available, healthy or that the device plugin has registered them correctly. A node in NotReady with 4 GPUs reduces the real capacity without Kueue knowing: the ClusterQueue will carry on admitting workloads that then cannot be placed.

Recommended monitoring: cross Kueue’s metrics (kueue_admitted_workloads_total, kueue_pending_workloads) with the cluster’s real capacity metrics (GPUs registered in the device plugin) to detect the drift. Kueue exposes native Prometheus metrics; so does Volcano.

3. GPU resource naming: MIG, time-slicing and ResourceFlavor

If the cluster uses MIG, the resources in the pod specs change from nvidia.com/gpu to nvidia.com/mig-Xg.Ygb (for example, nvidia.com/mig-3g.40gb). Kueue’s ResourceFlavors and Volcano’s Queues must declare the right resource, or the quota will not match the pods. With time-slicing, the resource is still nvidia.com/gpu but the device plugin advertises more instances than there are physical GPUs; the quota is expressed in virtual replicas, which can lead to over-admission if the VRAM budget is not taken into account (see sharing a GPU: time-slicing, MPS and MIG).

4. Preemption in production: the evicted workload loses progress

When Kueue or Volcano evicts a training job that has been running for hours, that job loses its progress if it has no checkpointing configured. The preemption is correct from the quota point of view, but it destroys work if the job is not prepared for it. Before enabling aggressive preemption, verify that all training jobs have periodic checkpointing with automatic restoration. PyTorch plus Torchrun have native support; so does DeepSpeed. Stateless batch inference jobs do not have this problem.

5. Volcano as the sole scheduler versus coexistence with the kube-scheduler

Volcano can be configured as the cluster’s default scheduler (all pods go through it) or as an alternative scheduler (only pods with schedulerName: volcano). The first option simplifies the configuration but breaks system pods that assume kube-scheduler behaviour. The second, the recommended one, requires ML jobs to explicitly set schedulerName: volcano, which can be a non-trivial operator or chart change for existing workloads. Kueue solves this more transparently: it only requires a label on the job, with no change of scheduler.

6. Real operational complexity in 2026

Running Kueue plus Volcano plus the Training Operator plus the GPU Operator in production means four components with their own CRDs, webhooks, versions and release cycles. A Kubernetes upgrade may require updating all four in sequence. The operational debt is real. For a small team without the capacity to maintain this stack, a managed Kubernetes provider (GKE with native Kueue, OpenShift with the Red Hat build of Kueue) may be more pragmatic than assembling the full stack from scratch.

The sig-scheduler coscheduling plugin is a deliberately simpler option when all you need is gang: fewer features, less complexity, fewer things to maintain.

Sources

Volcano — Introducción y features (CNCF incubating, v1.15) — https://volcano.sh/en/docs/
Volcano — Volcano v1.11 Released: A New Era of Cloud-Native Scheduling (CNCF blog, marzo 2025) — https://www.cncf.io/blog/2025/03/05/volcano-v1-11-released-a-new-era-of-cloud-native-scheduling-for-ai-and-big-data/
Volcano — Release v1.11.0 (GitHub) — https://github.com/volcano-sh/volcano/releases/tag/v1.11.0
Volcano — Network Topology Aware Scheduling design doc — https://github.com/volcano-sh/volcano/blob/master/docs/design/Network%20Topology%20Aware%20Scheduling.md
Volcano — NUMA-aware scheduling design — https://github.com/volcano-sh/volcano/blob/master/docs/design/numa-aware.md
Volcano — Unified Scheduling (docs oficiales) — https://volcano.sh/en/docs/unified_scheduling/
Volcano — Capacity scheduling design — https://github.com/volcano-sh/volcano/blob/master/docs/design/capacity-scheduling.md
NVIDIA Technical Blog — Practical Tips for Preventing GPU Fragmentation for Volcano Scheduler — https://developer.nvidia.com/blog/practical-tips-for-preventing-gpu-fragmentation-for-volcano-scheduler/
Kueue — Overview (kueue.sigs.k8s.io, actualizado febrero 2026) — https://kueue.sigs.k8s.io/docs/overview/
Kueue — Cluster Queue (nominalQuota, borrowingLimit, lendingLimit, preemption) — https://kueue.sigs.k8s.io/docs/concepts/cluster_queue/
Kueue — Cohort — https://kueue.sigs.k8s.io/docs/concepts/cohort/
Kueue — Fair Sharing — https://kueue.sigs.k8s.io/docs/concepts/fair_sharing/
Kueue — Topology Aware Scheduling — https://kueue.sigs.k8s.io/docs/concepts/topology_aware_scheduling/
Kueue — Preemption — https://kueue.sigs.k8s.io/docs/concepts/preemption/
Kueue — Setup All-or-nothing with ready Pods — https://kueue.sigs.k8s.io/docs/tasks/manage/setup_wait_for_pods_ready/
Kueue — GitHub (kubernetes-sigs/kueue) — https://github.com/kubernetes-sigs/kueue
Ray — Gang Scheduling with RayJob and Kueue — https://docs.ray.io/en/latest/cluster/kubernetes/examples/rayjob-kueue-gang-scheduling.html
Kubeflow — Volcano scheduler integration — https://www.kubeflow.org/docs/components/trainer/operator-guides/job-scheduling/volcano/
kubernetes-sigs/scheduler-plugins — Coscheduling plugin README — https://github.com/kubernetes-sigs/scheduler-plugins/blob/master/pkg/coscheduling/README.md
kubernetes/enhancements — KEP-583 Coscheduling (sig-scheduling) — https://github.com/kubernetes/enhancements/tree/master/keps/sig-scheduling/583-coscheduling
InfraCloud — Batch Scheduling on Kubernetes: Comparing YuniKorn, Volcano and Kueue — https://www.infracloud.io/blogs/batch-scheduling-on-kubernetes/
Red Hat — Red Hat build of Kueue (OpenShift 4.20) — https://docs.redhat.com/en/documentation/openshift_container_platform/4.20/html/ai_workloads/red-hat-build-of-kueue
AceCloud — Multi GPU Orchestration in Kubernetes 2026: Kueue, Volcano, DRA — https://acecloud.ai/blog/multi-gpu-orchestration-kubernetes/
CloudOptimo — Kubernetes AI Infrastructure in 2026: GPU Scheduling and Production Realities — https://www.cloudoptimo.com/blog/kubernetes-ai-infrastructure-in-2026-gpu-scheduling-and-production-realities/

Batch on lo0 — Tech Blog