8 x 288GB: What Fits on a B300 Node

A 405-billion-parameter model needs about 203GB for its weights in FP4. An NVIDIA B300 has 288GB of HBM3e per GPU. That means the weights load on one GPU with about 85GB left for the KV cache, the memory that grows with every token of context. In FP8 the same model needs about 405GB and takes two GPUs, and in FP16 it needs about 810GB. The B300 moves the line between single-GPU and multi-GPU inference, and this post works out where it sits for a B300 node: weights, KV cache and headroom, built on published architecture figures rather than vendor throughput claims.

810GB, 405GB, 203GB: what a 405B model's weights cost at each precision

Weight memory is parameter count times bytes per parameter: 2 bytes in FP16, 1 in FP8, 0.5 in FP4. NVIDIA lists the Blackwell Ultra GPU at 288GB of HBM3e, 3.6 times the on-package memory of an H100, which makes the weight budgets below the first thing to check.

A 70B model fits on one B300 at every precision, including FP16, which is more than a 96GB card can say. A 405B model fits on one GPU only at FP4. At FP8 it needs two GPUs, and at FP16 at least three, with almost nothing left over for anything else.

Lower precision isn't free. Moving from FP16 to FP8 or FP4 trades some accuracy for memory, and how much depends on the model and the quantization method, so it's worth testing on your own evaluation set before committing to a node size. The arithmetic shows which options exist, not which one to pick.

16.9GB per 32K-token sequence: the KV cache at 405B scale

Weights are a fixed cost. The KV cache is the variable one: every token in every active sequence stores a key and a value vector at each layer, and that memory stays allocated while the sequence is live. Meta's Llama 3 paper gives the 405B architecture: 126 layers, 128 attention heads across a 16,384 model dimension (128 per head), and 8 key/value heads. At 16-bit precision that's 2 x 126 x 8 x 128 x 2 bytes, about 0.52MB per token. A 32,768-token context holds roughly 16.9GB. An 8,192-token context holds roughly 4.2GB, and a 131,072-token context roughly 67.7GB.

Both the 70B and the 405B use grouped-query attention with 8 KV heads, which is why the cache is smaller than the size of the model would suggest. The 70B model, with 80 layers, needs about 0.33MB per token.

Next to the weight budgets, for the 405B model:

These are upper bounds. They ignore activation memory, the CUDA context and framework overhead, and they assume a 16-bit KV cache and an even split of the weights across GPUs: quantizing the cache stretches them, and allocator inefficiency shrinks them. At FP4, one B300 holds a 405B model plus a working set of long sequences. At FP8, two GPUs hold the model and a larger one.

About 20 sequences at 32K: a 70B model on a single B300

The 70B model shows what 288GB means for the more common case. At FP8 the weights take 70GB and leave about 218GB. With a 0.33MB KV cache per token, that room holds about 81 concurrent 8K sequences or about 20 concurrent 32K sequences on a single GPU.

For comparison, the same FP8 model on a 96GB RTX Pro 6000 Blackwell leaves about 26GB, enough for roughly 9 sequences at 8K and 2 at 32K, as we worked out in our previous post. The difference isn't that one is better, it's that the two cards sit at different points on the same sizing curve.

8TB/s per GPU: the ceiling memory bandwidth puts on decode speed

NVIDIA lists the B300 at 8TB/s of memory bandwidth per GPU, 2.4 times the 3.35TB/s of an H100. For a dense model, generating each token means reading the weights from memory, so at batch size 1 the decode speed of a single sequence is bounded by bandwidth divided by the bytes read per token. That gives a ceiling of about 39 tokens per second for the 405B model at FP4 (8,000GB/s divided by 203GB), about 114 for the 70B model at FP8 and about 229 for the 70B model at FP4.

Those are ceilings, not predictions. They ignore the KV cache reads, and real speed depends on the serving stack. They also explain why batching matters: with many sequences in flight, one read of the weights serves all of them, so total tokens per second goes up until the KV cache capacity from the tables above runs out. Capacity and bandwidth together set the operating range, and the only way to know where a model lands in it is to benchmark it on your own workload.

Where one GPU stops: a 405B model at FP8 needs two, and the interconnect joins the sizing

Once a model's weights don't fit on one GPU, it has to be split across several, and every token then involves communication between them. NVIDIA lists NVLink at 1.8TB/s per Blackwell Ultra GPU. PCIe Gen5, as we noted in our post on foundation model training bottlenecks, tops out around 128GB/s, roughly 14 times less. How well a split model performs depends on which of those paths carries the traffic between its GPUs.

That's a different sizing question from the one in this post, and it's the reason the tables above separate the single-GPU cases from the two-GPU ones. If weights, KV cache and headroom fit on one GPU, the interconnect isn't part of the answer.

Eight pools of 288GB, 2,304GB per node

A 1Legion B300 server has 8 GPUs and 2,304GB of GPU memory in total. The useful way to read that number is as eight 288GB pools. Per-GPU workloads spread across them without needing GPU-to-GPU traffic: eight replicas of a 405B FP4 model behind a load balancer, eight replicas of a 70B FP8 model each serving about 20 sequences at 32K, or four two-GPU replicas of a 405B FP8 model. Mixed layouts work too, for example one two-GPU 405B replica next to six single-GPU 70B replicas.

1Legion rents the B300 as a full 8-GPU server on dedicated bare metal, with a 36-month minimum term and a rate fixed for the term. Single-GPU rental isn't offered.

The sizing rule: weights + KV cache + headroom must fit in 288GB

To check a model against one B300 GPU, pick the precision, multiply parameters by bytes per parameter, then estimate the KV cache as 2 x layers x KV heads x head dimension x bytes per value, multiplied by your target context length and concurrency. Leave room for activations and framework overhead. If the total fits in 288GB, the model is a per-GPU workload, and an 8-GPU node gives you eight of them to schedule. If it doesn't, the next question is how the model gets split across GPUs, and that depends on the interconnect as much as on memory.

Talk to an engineer about sizing an 8-GPU B300 node for your models.

‍