70GB of Weights, Four GPUs: What’s Left for the KV Cache at 80, 96, 141 and 288GB

Take one model, a 70-billion-parameter LLM in FP8, and load it on four GPUs with 80GB, 96GB, 141GB and 288GB of memory. The weights take 70GB in every case. What differs is what is left: about 10GB, 26GB, 71GB and 218GB. That remainder is where the KV cache lives, so it sets how many sequences the GPU can serve at once and how long they can be. This post works through the 70B model GPU memory requirements at each size, using published architecture figures and no throughput claims.

70GB of weights, four different remainders

Weight memory is parameter count times bytes per parameter: 2 bytes in FP16, 1 in FP8, 0.5 in FP4. For a 70B model that is 140GB, 70GB and 35GB. NVIDIA lists the H100 SXM at 80GB, the RTX PRO 6000 Blackwell at 96GB of GDDR7 with ECC, the H200 at 141GB of HBM3e, and the Blackwell Ultra GPU in the B300 at 288GB of HBM3e.

FP16 weights of 140GB do not fit on an 80GB or a 96GB GPU. On a 141GB GPU they fit with about 1GB to spare, which is not a usable amount. On a 288GB GPU they fit with 148GB left. In practice, then, the FP8 case is the one that all four sizes can run, and it is the one the rest of this post uses. Dropping to FP8 or FP4 trades some accuracy for memory, and how much depends on the model and the quantization method, so test it on your own evaluation set before committing to a size.