Why AI workloads queue while GPUs appear idle

A low GPU utilisation number does not necessarily mean that the device is available for a new job. A model may perform little computation most of the time while still holding the entire GPU and much of its memory. The cluster scheduler therefore sees no allocatable device, and the next workload remains queued.

Utilisation is not availability

The common GPU utilisation metric usually measures the percentage of streaming multiprocessors active during a sampling window. It does not show whether the device has already been assigned to a pod, how much VRAM is occupied, or whether a new workload satisfies the available hardware and scheduling constraints. A GPU reporting 5 percent compute activity can consequently remain fully allocated from Kubernetes' perspective.

The article cites a 5 percent average compute-utilisation figure from Cast AI's 2026 Kubernetes Optimisation Report. That figure is evidence of underuse in the studied fleet, but it does not mean that 95 percent of capacity is immediately free or reclaimable. Each cluster still requires its own diagnosis.

Four causes of queues beside idle-looking GPUs

• Exclusive device allocation: By default, the NVIDIA device plugin assigns a whole GPU to one pod. A lightly loaded service can therefore prevent another pod from using the same device.
• Occupied memory: Model weights and caches may fill VRAM even while compute units are mostly idle. Time-slicing does not solve a memory-capacity problem.
• Placement constraints: GPU model selectors, affinity rules, taints, and topology requirements may stop a workload from landing on otherwise unused hardware.
• Non-GPU bottlenecks: CPU preprocessing, data loading, networking, storage, or PCIe transfers can leave the GPU waiting without making it truly available to another workload.

A practical diagnostic order

Start by checking device allocation across nodes, then inspect VRAM occupancy, review Pending pod events, and finally observe compute activity over time. Events reporting insufficient resources, node-selector mismatches, or unmet taints can reveal a placement issue. A workload that repeatedly jumps from zero to high GPU activity may instead be gated by CPU or I/O.

GPU-sharing options

Time-slicing works across a broad range of NVIDIA GPUs and alternates device access among workloads, but it provides no memory or fault isolation. MIG, available on Ampere and newer supported devices, creates hardware-isolated instances with separate memory allocations and can better suit multi-tenant workloads, although its profiles are fixed. MPS allows CUDA processes to execute concurrently and can reduce context-switch overhead, but its basic configuration does not provide complete memory isolation.

No method fits every workload. Inference servers such as vLLM may also reserve a large share of VRAM at startup. Teams should therefore measure combined memory needs, realistic request overlap, and tail latency such as p95 and p99 before co-locating services.

When more GPUs are justified

Additional hardware is justified after allocation, memory, placement, and application bottlenecks have been ruled out and the devices are genuinely at their effective capacity, or when sharing produces unacceptable latency. Otherwise, buying GPUs may expand the same configuration problem rather than clear the queue.

This summary is based on an AI News article. Examples and some performance claims in the source draw on Cast AI reports and products and should be independently tested in each organisation's environment.