The hourly rate on a cloud GPU pricing page describes almost nothing about what a production workload actually costs. It leaves out egress, the storage tier your checkpoints have quietly accumulated in, what happens to the bill when a run overruns its estimate, and a handful of other charges that only become visible once someone sits down with twelve months of invoices instead of one. That review rarely happens on a schedule. It happens when finance asks why the infrastructure line grew faster than output did.

The questions below are the ones worth running against your own billing data this week, not against a vendor's claims.

1. What is your GPU utilization over the last 30 days, measured against what you paid for?

Cast AI's 2026 State of Kubernetes Optimization Report, drawn from tens of thousands of clusters, put average GPU utilization at 5%. At that level, the effective cost per utilized GPU-hour on an on-demand H100 instance works out to roughly 20 times the nominal rate, since the vast majority of provisioned capacity sits idle. Even the more forgiving industry estimates put typical real-world utilization between 10% and 40%, which still means paying for two to ten GPUs to get the output of one. Pull your own utilization metrics from your monitoring stack, not your billing dashboard, and compare them to the compute hours on the invoice.

2. What did egress cost you last quarter, broken out as its own line?

Public egress on AWS, Azure, and GCP runs $0.05 to $0.12 per gigabyte depending on volume tier. Industry benchmarks put egress at 6% to 12% of a typical cloud bill, but for data-intensive AI workloads, teams moving large training sets or model weights regularly, that share can climb to 20-30% of total infrastructure cost. Most teams don't isolate it as its own line, which is exactly why it's worth pulling separately.

3. Does your contract cap the cost of a training run that exceeds its estimate?

On usage-based billing, a run that takes 40% longer than planned costs 40% more, with no ceiling unless it's been specifically negotiated. This matters more for long training and fine-tuning jobs than short inference workloads, since the variance compounds over days or weeks. Ask whether an overrun triggers an alert before it triggers a charge.

4. How much is checkpoint and dataset sprawl actually costing you?

A single 70B-parameter model saving hourly checkpoints generates roughly 6.7TB of storage per day. Multiply that across every experiment, retrain, and abandoned branch, and the volume compounds fast, especially since cloud providers typically charge four to six times more to retrieve data than to store it. Old checkpoints and superseded dataset versions rarely have an owner assigned to clean them up, which is exactly why they're usually the easiest line item to cut without touching production.

5. What's your effective rate after reservations, spot pricing, and discounts?

List price and effective price diverge fast. GCP preemptible A100 instances run up to 91% below on-demand, and Azure spot pricing on H100 saves up to 60%, but both come with interruption risk that doesn't show up in the discount headline. If your team can't produce the blended effective rate without pulling in finance, that's a sign the true cost isn't visible to the people making architecture decisions.

6. Is support bundled into your rate, or billed as a percentage of spend?

Several major providers price premium support tiers as a percentage of monthly usage, which means the cost of getting a fast response scales with how much you're already paying them, independent of how much support you actually need. Confirm what tier you're on and whether its response-time SLA has ever been tested against a real incident.

7. What are your non-production environments costing while idle?

Cast AI's 2026 report also found CPU overprovisioning jumped from 40% to 69% year over year across Kubernetes clusters, with memory overprovisioning sitting at 79%. Dev, staging, and experimentation environments are usually the biggest contributor, provisioned once for a project and left running long after nobody owns the decommissioning step. A utilization audit on non-prod instances is typically the fastest way to find savings with zero architectural change.

8. How much does your total GPU spend vary month over month, and can you explain the variance?

Recent industry surveys put the share of teams exceeding their 2026 cloud budget at 62%, driven largely by fees that weren't part of the original forecast. Line up your last three invoices. If the swings aren't tied to a deliberate scaling decision, that's not a pricing model, it's a forecasting liability finance absorbs every cycle.

9. What does contention on shared capacity actually cost you in wasted GPU time?

A May 2026 study of 608 production H100 training jobs found mean Model FLOPs Utilization of just 25.1%, with real-world execution-idle time (checkpointing, initialization, queue delays, debugging) accounting for roughly 20% of total in-execution time in a separate large-scale cluster analysis. None of that shows up as its own line item. It shows up as jobs that take longer than they should.

10. What would your trailing 12-month cloud GPU spend total on a fixed-rate, dedicated contract instead?

At sustained utilization above roughly 70%, owned or dedicated GPU infrastructure runs an effective all-in cost of $2.50 to $4.00 per GPU-hour, factoring hardware, power, and colocation, comparable to spot pricing but without the interruption risk. Add your base compute, egress, storage, support, and overrun charges from the last 12 months, then compare that total to a fixed monthly rate over the same period. For sustained workloads rather than short bursts, the gap is frequently large enough to change the decision entirely.

Individually, none of these questions are hard to answer. Together, they tend to point to the same conclusion: the real cost of cloud GPU infrastructure was never the number on the pricing page. It's the accumulation of egress, idle capacity, storage sprawl, and support fees that each look small in isolation and add up to a materially different number over a full contract term.

Dedicated bare metal infrastructure is built around removing that accumulation. A single server, a fixed monthly rate, unmetered bandwidth, and no line items that surface only after the fact. It isn't the right architecture for every workload, but for teams running sustained AI or media production at scale, the math above is worth running before the next renewal is signed.

Talk to an engineer about what a fixed-cost, dedicated GPU setup would look like for your workload.

‍