Why did it crash if PyTorch was still holding 8 gigabytes it wasn't using?

1. The Problem: Allocator Fragmentation

head_dim: if absent, use hidden_size / num_attention_heads ; if present, trust it. Qwen3-32B gives 5120 / 64 = 80 but declares 128.
Tensor parallelism: each GPU holds num_key_value_heads / tp KV heads; vLLM replicates them once TP exceeds that count.
Sliding windows: Mistral-7B-v0.1 sets sliding_window to 4096, capping per-sequence KV in those layers.

To keep latency low, PyTorch manages its own heap via CUDACachingAllocator instead of calling cudaMalloc() on every tensor. When you run dynamic batching with varying sequence lengths, memory chunks are constantly split and merged. Eventually, those 8 GB of cached-but-unused space are broken into hundreds of tiny 10 MB and 20 MB slices. When your code requests a single contiguous 2 GB tensor, the allocator fails because no single slice is large enough.

2. Inside the Caching Allocator

Fragmentation follows from a few rules in c10/cuda/CUDACachingAllocator.cpp, the same in the PyTorch 2.0 and 2.15 sources:

Pools: requests are rounded up to a multiple of 512 bytes. Up to 1 MiB they share 2 MiB small-pool segments; below 10 MiB, new segments are 20 MiB; anything larger gets a cudaMalloc of its own size rounded up to 2 MiB.
Best fit, per stream: the smallest cached free block that fits, owned by the requesting stream.
Split: a large-pool block more than 1 MiB bigger than the request is split; the remainder stays free inside that segment.
Merge and release: freed blocks merge only with neighbours in the same segment, and a segment returns to the driver only when entirely free.

When nothing fits and cudaMalloc fails, the allocator synchronizes, frees every completely idle segment and retries before it raises. So the error's