Optimizing GPU Costs for LLM Inference on Kubernetes

Most Kubernetes clusters running LLM inference use only 15% to 25% of the GPU capacity they pay for. A single NVIDIA H100 runs $30 to $50 an hour. Do the math on twenty of them sitting mostly idle, and you get roughly $200,000 a year in wasted spend on one cluster alone. That's the number most engineering leaders don't see until the AWS invoice lands.

If you searched for this topic, you're probably staring at a GPU bill that doesn't match your traffic, and you're trying to figure out whether to fix it yourself or bring in a tool. This blog walks through both paths honestly, with real numbers from teams who've already done it.

Why does GPU spend for LLM inference spiral out of control on Kubernetes?#

Kubernetes was built for stateless web services. A pod needs some CPU, some memory, it starts fast, it dies fast, you scale it with a Horizontal Pod Autoscaler watching CPU usage. GPUs, and LLM inference in particular, break almost every assumption baked into that model.

Start with allocation. Kubernetes hands out GPUs as whole units. If your inference workload needs 20% of an A100's compute, it still reserves the entire card, because the default scheduler has no concept of a "slice" of a GPU. Then add the workload itself.

An LLM serving process has two very different phases: prefill, where it reads and tokenizes your prompt (this is compute-heavy), and decode, where it generates output tokens one at a time (this is memory-heavy, bottlenecked on the GPU's memory bandwidth, not its compute cores).

A single autoscaling metric can't capture both, which is why teams who try to scale inference pods with plain CPU-based HPA rules end up either overprovisioned or throttled.

Then there's the fear factor. Nobody wants to be the engineer who under-provisioned GPU memory and caused an outage during a launch, so the default move is to round up. Reserve the 80GB card even though the model only touches 8GB. Keep a warm pool running through the night because a cold start on a 70B parameter model can take two to five minutes. (Confirmed in a live Kubernetes GPU demo by Abdel Sghiouar, Devoxx 2026, who timed a 1B parameter model taking roughly two minutes just to download and load).

That instinct is rational for any individual engineer protecting uptime. Multiplied across a fleet, it's how a $10M compute bill ends up with $5M going to memory, CPU, and GPU capacity nobody used.

What is actually eating your GPU budget?#

Break the waste down by workload type, because each one wastes money differently, and a single fix won't cover all of them.

Training jobs left running after they finish. A researcher kicks off a fine-tuning run, walks away, and the GPU sits allocated long after the job completes. This is the single most common waste pattern DevZero sees when scanning new clusters.

Warm inference pools sized for peak traffic. If your chatbot gets hammered at 9am and goes quiet at 2am, but you provision for the 9am load 24 hours a day, you're paying peak price around the clock. This is the most common pattern in production LLM serving, because nobody wants a cold-start delay when a real user is waiting.

Interactive notebooks nobody remembers to shut down. A data scientist spins up a GPU-backed Jupyter environment for an afternoon of experimentation and it's still running a week later, at 5% utilization, because closing the browser tab doesn't terminate the pod.

Cold starts that force you to keep capacity warm just in case. This is the sneaky one. If reloading an 80GB model into GPU memory takes 30 seconds to several minutes, engineers respond by never letting the pod scale to zero, which defeats the entire point of elastic infrastructure.

One more pattern worth naming separately, because it's specific to LLMs and most generic Kubernetes cost tools miss it entirely: KV cache over-allocation.

Serving engines like vLLM reserve a chunk of GPU memory, often around 30%, purely for key-value cache used during the decode phase.

Set that fraction too high for your actual traffic pattern and you're burning memory that never gets touched.

Set it too low and your throughput collapses under concurrent requests. Getting this number right requires knowing your real request-size distribution, not a guess copied from a tutorial.

How do you actually measure GPU utilization?#

Here's the trap almost every team falls into: they check nvidia-smi, see the GPU listed as "in use," and assume it's healthy. nvidia-smi gives you a point-in-time snapshot on one node. It can't tell you whether that GPU is doing productive work, thrashing on oversized batches, or just holding memory it isn't touching.

NVIDIA's Data Center GPU Manager (DCGM) is the tool built for this, and it exposes metrics that actually separate the signal from the noise:

• SM utilization (DCGM_FI_DEV_GPU_UTIL) tells you if the compute cores are busy.
• Memory bandwidth utilization tells you if you're bottlenecked moving data, not computing.
• Tensor core utilization (DCGM_FI_PROF_PIPE_TENSOR_ACTIVE) tells you if your AI workload is even using the GPU's most valuable hardware. A GPU can show 90% SM utilization while its tensor cores sit idle, meaning the workload isn't using the capability you paid the premium for.

Deploy the DCGM Exporter as a DaemonSet, feed it into Prometheus, and you finally have per-namespace, per-workload, time-series visibility instead of a snapshot.

Independent research backs the same conclusion from a different angle: in unoptimized production deployments, inference workloads commonly run at 5% to 20% GPU utilization while the bill reflects 100% of allocated capacity, and teams that fix this properly report reaching 70% to 80% utilization.

Two independent sources, two different numbers, same story: the gap between what you're paying for and what you're using is enormous, and it's invisible until you measure it correctly.

What technical levers actually reduce the bill?#

Once you can see the waste, here's what moves the needle, roughly in order of effort required.

Right-size GPU memory to the model, not the biggest card available#

A model that needs 8GB doesn't belong on an 80GB A100. Match instance type to actual memory footprint: A10G (24GB) or T4 (16GB) for smaller models, A100 40GB for mid-size single-model inference, and reserve the 80GB cards for genuinely large models or multi-model serving.

Use NVIDIA Multi-Instance GPU (MIG) for inference#

MIG partitions a single A100 into up to seven isolated instances, each with dedicated compute and memory, running true parallel workloads rather than time-sliced access. For inference, where a single request rarely needs a whole GPU, this is often the single highest-leverage change you can make.

Checkpoint and restore, paired with spot instances#

Application-level checkpointing saves training or model state to persistent storage at intervals. Combine that with spot or preemptible GPU capacity and you can run at 60% to 70% below on-demand pricing, because an interruption no longer means starting from zero. CRIU-GPU extends this specifically to serialize what's already loaded in GPU memory, which also shrinks cold-start time on inference pod restarts, since you're not re-downloading and re-loading model weights from scratch.

Split prefill and decode onto separate, purpose-matched hardware#

This is a newer pattern, and it's worth understanding even if you're not ready to implement it. Because prefill is compute-bound and decode is memory-bound, running them on the same GPU means one phase always fights the other for resources.

Disaggregated serving, the approach behind the open source LLM-D project, runs dedicated prefill pods and dedicated decode pods, sometimes even on different accelerator types, and uses NVIDIA's NIXL or GPU-to-GPU transfer to move the KV cache between them.

Benchmarks on Llama 2 70B showed measurably better throughput when prefill ran on one accelerator generation and decode on another, instead of forcing both phases onto identical hardware.

Tune attention mechanisms at the model-serving layer#

Paged attention manages KV cache the way an operating system manages virtual memory, cutting fragmentation and letting you handle longer inputs without running out of GPU memory. Flash attention reduces the data transfer between GPU RAM and cache during token generation, which cuts idle time on the compute cores. Both are documented as standard practice in Google Cloud's own GKE inference optimization guide.

Schedule around predictable patterns#

Training often runs overnight. Inference traffic often follows business hours. Time-based autoscaling that shrinks GPU node pools during known-quiet windows is a low-effort, high-return move most teams skip simply because nobody owns it.

Should you build this yourself, or buy a platform?#

This is the real question behind the search, so let's be direct about it instead of vague.

What building it yourself actually takes#

You can absolutely build a GPU cost optimization stack in-house, and plenty of platform teams do. Here's the honest shape of that project, not the marketing version:

You'll deploy the DCGM Exporter DaemonSet across every GPU node and wire it into Prometheus. You'll build Grafana dashboards that break utilization down by namespace and workload, because the default cluster-wide view hides which team is actually responsible for the waste. Since standard HPA only understands CPU and memory, you'll need a custom metrics adapter so autoscaling can react to GPU-specific signals like queue depth and KV cache saturation instead.

For fair GPU allocation across teams sharing a cluster, you'll likely adopt Kueue, the open source Kubernetes-native job queueing project, to enforce quotas instead of trusting everyone to self-report an honest priority level. If you're running production inference at meaningful scale, you'll want the Gateway API Inference Extension, which adds an "endpoint picker" component that reads live GPU memory and queue-depth metrics to route each request to the least-loaded pod, since a plain round-robin load balancer has no idea one pod's KV cache is already full while another sits empty.

That's a real, workable stack. It's also five or six separate open source projects that each need someone to own upgrades, watch for breaking changes, and debug at 2am when the metrics pipeline silently stops reporting.

Teams like DataBahn evaluated this route directly, alongside cloud-native tooling, before deciding the engineering time cost outweighed the control they'd gain (more on their numbers below).

Build makes sense when you have a dedicated platform team, a single cloud provider, and genuinely unusual workload patterns that off-the-shelf tools don't model well.

It stops making sense the moment "keep the optimization stack running" becomes its own full-time job, competing directly with the product work that actually generates revenue.

What to look for if you're buying#

If you decide to buy rather than build, the tool has to solve the specific problems above, not just show you a dashboard. A few things actually matter for this use case:

It needs to rightsize without restarting pods. A tool that requires a redeploy to apply a recommendation isn't solving the "fear of disruption" problem that caused the overprovisioning in the first place; it's just moving the risk to a different moment.

It needs to handle GPU and CPU/memory optimization together, because most real clusters mix GPU-backed inference with CPU-backed application services, and optimizing one in isolation misses cross-workload bin-packing opportunities.

It needs to work across clouds, not just one. If your GPU strategy involves AWS today and Azure or GCP tomorrow (increasingly common given GPU supply constraints), a single-cloud tool means retooling and rebuilding expertise every time you expand.

And separately from Kubernetes-level GPU rightsizing, if you're also paying for LLM API traffic (OpenAI, Anthropic, Bedrock, and similar), that's a distinct cost surface worth checking.

DevZero's inference platform sits in front of that traffic as a self-hosted gateway, running a shadow cache that measures exactly how much you'd save from caching repeat prompts before you flip anything on, plus an eval lab that replays your real traffic against cheaper models so a swap decision comes with an actual cost-and-quality number attached.

If part of your GPU cost problem is really an "are we calling the most expensive model for every request" problem, that's worth checking before you spend engineering time on the Kubernetes layer alone.

What does this look like when a real team does it?#

Numbers in the abstract are easy to wave off. Here's what happened at DataBahn, a Series A AI data infrastructure company running multi-cloud on AWS and Azure.

Their multi-cloud approach was a deliberate strategic choice, giving customers cloud flexibility and giving DataBahn negotiating leverage on committed spend.

The problem was execution: AWS cross-AZ data transfer costs crept up unnoticed, Azure's pod CIDR limits and default premium SSD tiers weren't visible without specialist knowledge, and engineers were spending real hours chasing cost surprises that surfaced weeks after they happened, instead of building the product.

After deploying DevZero, DataBahn saw up to 75% cost reduction on a single AWS cluster and roughly 60% on Azure, with measurable savings inside the first ten hours of deployment. Engineers reclaimed 40 to 60-plus hours per week that had been going into manual infrastructure firefighting.

"Our engineers can finally focus on building product features that matter to customers. DevZero gave us back the time to innovate," said Mihir Nair, Head of Architecture at DataBahn.

Poornima Verghese, DataBahn's Director of Operations, pointed to something less obvious than the raw savings number: "That level of visibility helped us build customer-specific pricing models. We can now attribute costs accurately and give customers real choices."

Once you can see cost broken down by workload and by customer, you can price your product differently, which is a business outcome most GPU cost conversations never reach.

What just changed that makes this more urgent right now?#

If you've been putting this off, the ground shifted under you in January 2026. On Saturday, January 4th, AWS raised prices on EC2 Capacity Blocks for NVIDIA H200 GPUs by 15%, with no formal announcement, just an updated pricing page.

The p5e.48xlarge instance (eight H200 accelerators) went from $34.61 to $39.80 an hour. In US West, the same instance jumped from $43.26 to $49.75 an hour. For a team running that instance continuously, that's an extra $3,700-plus a month, on top of whatever they were already paying.

This wasn't a supply-and-demand blip. It followed AWS publicly touting "up to 45% price reductions" on GPU instances just seven months earlier, in June 2025, for a different pricing category (On-Demand and Savings Plans, not Capacity Blocks).

The underlying pressure is real: NVIDIA reportedly has orders for 2 million H200 chips for 2026 against inventory of roughly 700,000 units, and HBM3E memory prices climbed around 20%.

Whether you're in the US, UAE, India, the UK, or elsewhere in Asia, this matters the same way: GPU supply constraints are global, and every major cloud provider is pricing accordingly.

The takeaway: every percentage point of GPU utilization now matters more than it did six months ago, because the per-hour price it's multiplied against just went up, and nothing suggests that's the last increase this category will see.

What should you actually do this week?#

If you take one thing from this article, make it this: don't start with a tool decision.

Start by measuring. Deploy DCGM if you haven't, or run a read-only scan against your existing clusters, and get an honest number for your current utilization before you decide whether to build or buy. Most teams are surprised by how bad the number is, and that number is what makes the rest of the decision obvious.

From there, the sequence that works: identify your two or three most expensive workloads (training and production inference, almost always), establish a utilization baseline for each, apply the cheap fixes first (memory right-sizing, MIG partitioning where applicable, killing idle notebooks), and only then decide whether the ongoing operational cost of a DIY monitoring and scheduling stack is worth it compared to a platform built to do this continuously.

DevZero's free assessment installs a read-only operator in under 45 seconds, requires no upfront cost, and surfaces your first savings insights within 24 hours, so you can see the real number for your own cluster before committing engineering time either direction.

Conclusion#

GPU waste on Kubernetes isn't a mystery once you measure it correctly. It's coarse-grained scheduling, fear-driven overprovisioning, cold starts nobody wants to risk, and a KV cache setting nobody's touched since the tutorial they copied it from.

Every lever in this blog, from MIG partitioning to prefill/decode disaggregation to checkpoint/restore on spot instances, is proven and documented.

The only real decision left is whether your team builds and maintains that stack indefinitely, or points a platform at the problem and gets the engineering time back.

Teams like DataBahn chose to get the time back, and used the cost visibility that came with it to change how they price their own product. If your GPU bill doesn't match your traffic, book a 15-minute walkthrough with DevZero and see what your own cluster is actually running.

FAQs#

Why are Kubernetes GPU costs for LLM inference usually so high?#

Most Kubernetes clusters run at only 15% to 25% GPU capacity because the default scheduler allocates entire GPUs, even if a workload only needs a fraction of the compute. This structural waste is multiplied by teams keeping warm inference pools sized for peak traffic 24/7 to avoid cold starts, leaving training jobs running post-completion, and overallocating the KV cache.

Why shouldn't I use nvidia-smi to measure Kubernetes GPU utilization?#

nvidia-smi only provides a point-in-time snapshot for a single node. It can show a GPU as "in use" even if it is simply holding memory without doing any productive computing. For accurate, time-series visibility, you must deploy NVIDIA's Data Center GPU Manager (DCGM) exporter to track actual Streaming Multiprocessor (SM) utilization, memory bandwidth, and tensor core activity.

Does standard K8s HPA (Horizontal Pod Autoscaler) work for LLM workloads?#

No. Standard HPA scales based on basic CPU and memory metrics, which fail to capture the dual nature of LLM serving: prefill (which is heavily compute-bound) and decode (which is memory-bandwidth-bound). To scale effectively without throttling or overprovisioning, you need custom metrics adapters that react to leading indicators like queue depth or KV cache saturation.

How can I reduce cold-start delays for LLM inference?#

Downloading and reloading massive model weights (like a 70B parameter model) from scratch can take several minutes. You can dramatically reduce this time by utilizing application-level checkpointing paired with CRIU-GPU to snapshot the in-memory state. This approach bypasses the need to rebuild from scratch, making it much safer to scale down or utilize cheaper spot instances.

What is disaggregated serving, and should I separate prefill and decode?#

Disaggregated serving is the practice of running the prefill phase and the decode phase of LLM inference on separate, purpose-matched hardware. Because prefill fights for compute and decode fights for memory, splitting them across different pods, and transferring the KV cache between them, prevents resource contention and measurably increases throughput.

What is the true engineering cost of building a custom GPU optimization stack?#

Building a GPU cost-optimization stack in-house means deploying and actively maintaining multiple open-source tools: DCGM Exporter, Prometheus, custom metrics adapters, Kueue for job quotas, and Gateway API extensions for intelligent routing. Maintaining this complex pipeline quickly becomes a full-time job, pulling platform engineers away from revenue-generating product work.

How does DevZero optimize GPU costs differently than native K8s tools?#

Rather than relying on reactive scaling or manual redeployments, DevZero provides autonomous, live rightsizing without disrupting active training or inference jobs. By analyzing real-time GPU allocation, enforcing policy-driven controls, and leveraging technologies like checkpoint/restore, DevZero reclaims unused capacity and typically reduces cloud bills by 40% to 70%.

How fast can I see ROI if I implement DevZero for my AI workloads?#

DevZero is designed for immediate visibility and rapid time-to-value. The read-only assessment operator installs in under 45 seconds without modifying your workloads, and surfaces actionable savings insights within 24 hours. For example, DataBahn achieved up to a 75% AWS cost reduction within their first 10 hours of deployment, reclaiming dozens of engineering hours per week.