A single B200 went from costing about 11 cents per million tokens at launch to 2 cents two months later, with no hardware change. Same silicon, same rack, same power draw. The only thing that moved was the serving stack. Meanwhile the big four hyperscalers are on track to spend north of $650 billion on data center infrastructure this year, and roughly 7 GW of announced AI capacity sits stranded because there is nowhere to plug it in. Two facts, one lesson: in 2026 the cost of AI compute is not the chip. It is the power feeding it and the software running on it.

I have spent the last year watching platform teams budget AI on the wrong axis. They negotiate the GPU contract on dollars per hour, put peak FLOPS on a slide in the architecture review, then act surprised when a shipped feature comes back underwater on unit economics. Three numbers actually decide whether your inference is solvent: tokens per watt, the cost per million tokens that falls out of it, and whether you can get the megawatts at all. None of them are printed on the spec sheet.

The axis flipped from FLOPS to tokens per watt

Training owned the narrative for three years. It is over as the cost center. Inference now runs roughly 80 to 90 percent of AI compute spend, and inference behaves like a power problem long before it behaves like a silicon problem.

Here is the part that bites teams: almost every serious data center is power-capped, not space-capped or budget-capped in the short term. You have a fixed megawatt allocation from the utility, and the grid queue to get more is measured in years. Once you accept that, the GPU question inverts. You are not buying the fastest chip. You are buying the chip that converts a fixed megawatt into the most sellable tokens. Throughput per megawatt is the product. Everything else is vanity.

The benchmarks finally admit this. NVIDIA's October 2025 InferenceMAX results, run on the open-source SemiAnalysis harness, lead with cost per token and throughput per megawatt instead of TFLOPS. When the vendor stops bragging about FLOPS and starts reporting the metric your CFO already tracks, the framing war is over.

Most of the 5x is the stack, not the silicon

Here is the number that should bother anyone who just signed a Blackwell purchase order. An engineer drove a 96-GPU B200 cluster to over 1.1 million tokens per second serving Qwen 3.5 27B in FP8 on vLLM, and the win that mattered most was not the silicon. The cleanest evidence is on the Google Cloud community blog: 12 nodes, 96 GPUs, and a cost of $0.30 per million tokens self-hosted on one-year committed pricing, against $0.67 for a comparable hosted API. Self-hosting on a tuned open engine came in at less than half the hosted price.

The author is blunt about why. Multi-token prediction (MTP) was the single largest throughput lever, hitting a 90 percent acceptance rate at about 1.9 tokens per decode step. Turn it off and a third of the throughput vanishes, which means a third of your cost-per-token advantage vanishes with it. Same chips, same model, same cluster. A third of the performance lived in a config flag.

So when someone tells you Blackwell cut inference cost 5x, the correct response is: Blackwell running what? The stack is where the money is, and it comes in four moves you can make on hardware you already own.

Speculative decoding and its MTP cousin. Both guess several tokens cheaply, then verify them in one pass of the big model. Accepted guesses are free throughput. AWS published P-EAGLE on March 13, 2026: parallel speculative decoding in vLLM v0.16.0 and later, up to 1.69x over vanilla EAGLE-3 on a single B200 serving GPT-OSS 20B at low concurrency. The catch operators keep walking into: that 1.69x compresses to 1.05 to 1.25x at concurrency 64, because at high batch sizes the GPU has no idle compute left to spend on verification. Measure it where you run, not at a single stream. Larger models flip this in your favor: Ege Erdil's "Inference Economics of Language Models" (arXiv:2506.04645) models an 80 percent acceptance rate yielding a 66 percent gain on Llama 3 70B and a doubling on Llama 3.1 405B at fixed cost per token.

Prefix caching, the free win nobody benchmarks. SGLang's RadixAttention reuses the KV cache for shared prompt prefixes. A chat product sends the same system prompt every turn. A RAG pipeline reuses the same retrieved context across a conversation. A naive engine recomputes all of it every time. On prefix-heavy pipelines the throughput delta over a cold engine runs several-fold, for the cost of enabling a flag. Teams miss it because synthetic benchmarks fire unique prompts, so prefix caching shows zero benefit on the test and a large benefit in production. This is the cheapest hour of work in the entire stack. Do it first.

Prefill and decode disaggregation. Inference has two phases with opposite appetites. Prefill is compute-bound; decode is memory-bandwidth-bound. Put them on the same GPU and a big prefill stalls the decode stream, time to first token spikes, and tail latency falls apart under load. LMSYS's January 12, 2026 EPD writeup shows disaggregation roughly doubling throughput at higher request rates and cutting time to first token 6 to 8x under load. The TTFT number is the tell: that is a latency rescue, not a throughput trick. If your p99 first-token latency degrades the moment traffic climbs while decode stays healthy, prefill is starving decode, and you can buy 2x before adding a single GPU.

The engine you pick matters less than the features you turn on. On H100 at moderate concurrency SGLang leads vLLM by about 29 percent on standard workloads, with TensorRT-LLM marginally ahead at high concurrency. Twenty-nine percent is real money, but hold it next to MTP's one-third and disaggregation's 2x. The gap between two engines is smaller than the gap between one engine with the right features on and the same engine with them off. Picking SGLang over vLLM and then serving with defaults is optimizing the wrong variable.

Precision is the cheapest lever you own

Before anyone buys a GPU, the largest free win is numerical precision. Introl's unit-economics breakdown puts Llama 3.1 70B on an 8x H100 node at about $1.90 per million tokens in FP16. Move that same model to FP8 and it drops to roughly $0.95 to $1.10. You roughly halved cost per token without touching hardware or renegotiating power. On Blackwell, native FP4 roughly doubles throughput again over FP8 where model quality holds up.

The caveat is real and I will not paper over it. FP4 is not free quality. On some workloads the accuracy hit shows up in eval and you back it out. But testing FP8 and FP4 on your actual traffic costs an afternoon. Buying more GPUs costs a quarter and a power allocation you may not have.

Rubin's 10x is real, on a clock you do not control

Now the number loose in every 2026 budget deck: up to 10x lower cost per token than Blackwell. That is NVIDIA's headline for the Vera Rubin NVL72, launched at CES in January and detailed at GTC in March. The same rack promises up to 5x greater inference performance and a 4x cut in the GPUs needed to train a mixture-of-experts model. The figure is real. It is also surrounded by footnotes that decide whether it ever lands in your P&L.

First, two clocks that do not line up. The marketing clock started in January 2026. The deployment clock, by NVIDIA's own guidance, starts shipping in the second half of 2026 and widens toward broad availability in 2027. Cut your Blackwell order today on the strength of a January slide and you open a capacity hole in the exact window demand climbs fastest.

Second, the 10x is a rack number on a named workload. Per Tom's Hardware's CES coverage, it is benchmarked on the Kimi-K2-Thinking MoE model at 32K input and 8K output. A dense model does not see that multiplier. A short-context workload does not. A single node pulled out of the 72-GPU fabric does not. If your production traffic is dense models at 4K context, the honest planning number is a fraction of the headline.

Third, the cost win lives in NVFP4, which means it lives in your quantization backlog. NVIDIA's developer blog quotes 50 PFLOPS of NVFP4 inference per Rubin GPU, framed as 5x Blackwell. NVFP4 is four-bit. If you serve FP8 or BF16 today and have not validated four-bit accuracy on your own models and eval set, the 10x is not yours. The hardware exposes cheaper tokens; your engineering has to go claim them.

And Rubin is denser and hotter, not lighter. A Vera Rubin NVL72 rack packs 72 Rubin GPUs and 36 Vera CPUs, with a per-chip TDP reported around 2,000W. The per-token cost falls while the per-rack power and cooling burden climbs. The cheapest token in the world is stranded if your facility cannot land a high-density liquid-cooled rack, and a lot of existing space cannot without a capital project that takes longer than the GPUs do to arrive.

One figure to handle carefully: analyst write-ups have floated roughly $0.02 to $0.03 per million tokens for dense inference on Rubin. That is a third-party extrapolation with its own utilization and quantization assumptions baked in. It is not an NVIDIA list price and it does not belong pasted into a P&L.

The binding constraint is the plug, not the chip

For two years the bottleneck was silicon. That story is over. Of roughly 12 GW of AI capacity announced across about 140 U.S. projects for 2026, only 5 GW is actually under construction. Sightline Climate puts the other 7 GW in limbo, not killed by regulators or stalled for capital, but stalled because there is nowhere to plug them in.

The core problem is a calendar mismatch no capex fixes. Wiring a large facility into the high-voltage grid in the U.S. takes four to ten years. A data center gets designed, built, and commissioned in two to three. High-voltage transformers that used to ship in 24 to 30 months now run up to five years. Gas turbines are queued through 2029 and 2030. The manufacturing base was tuned for slow, incremental load growth, and the industry decided to roughly double its industrial power draw overnight.

The demand side explains the panic. A single AI reasoning task can pull up to 1,000 times the electricity of a plain web search, and 2026 is the year workloads moved to reasoning models and multi-step agents that grind through dozens of inference calls per request. U.S. data centers drew about 176 TWh in 2023; EPRI now projects 383 to 793 TWh by 2030, attributed almost entirely to AI. Gartner's number is the one I would pin to the wall: by 2027, 40 percent of AI data centers will face active power restrictions. If you are committing to a multi-year cloud migration, that is four in ten facilities your provider is counting on.

Watch where capital moves and you see operators pricing power as the scarce input. Microsoft put $15.2 billion into the UAE; Meta dropped more than $10 billion on a Louisiana campus. Those are power-availability bets, not market-size bets. The sharper move is going behind the meter: on-site natural gas turbines and Bloom Energy solid oxide fuel cells that skip the high-voltage transformer entirely, turning a five-year wait into a non-issue. Above 50 MW that is no longer a science experiment, and past 100 MW it is becoming the default plan.

Cooling is part of the GPU decision now

Every watt you spend on chillers and fans is a watt you are not spending producing tokens, so fold it directly into the tokens-per-watt number. A liquid-cooled cluster at PUE 1.10 carries about a 17 percent tokens-per-watt advantage over an air-cooled cluster at PUE 1.55 with identical GPUs. Under a power cap, that 17 percent is product you can ship or product you cannot. It is also stopping being a choice: rack density is averaging about 27 kW in 2026 and heading toward 45 to 100 kW by 2027. At Blackwell and Rubin densities, air cooling is not a worse option, it is not an option. If your facility is air-only and you are planning a 2027 refresh, the cooling retrofit is on the critical path for the GPU decision, not after it.

The counterpoint: Blackwell is not standing still

The strongest objection to all of this is real. Rubin being months out is only half the comparison. The other half is that Blackwell keeps getting faster while you wait, through software, via TensorRT-LLM and Dynamo serving gains, not new hardware. The marginal cost per token on B200 and B300 in mid-2026 is not frozen at last year's figure. The decision is not "expensive Blackwell now versus cheap Rubin later." It is "improving Blackwell I can deploy this quarter versus a bigger step I cannot rack until 2027." Framed that way, waiting looks a lot less obvious. And NVIDIA's own ROI math, a $5 million GB200 NVL72 generating $75 million in token revenue, is a 15x return only under near-perfect utilization. I have seen utilization sit in the 30s for months on strategic GPU buys, and at that point the cooling bill did not get the memo.

What to re-run this week

Each step ties to a number above, and the order is deliberate: cheapest and safest first.

  • Re-derive your real cost per million tokens before Friday. Take the node's all-in hourly cost, divide by measured tokens per second at your actual batch size and precision, not the spec sheet. If you priced chargeback before FP8 was standard, expect to be off by 2x or more (the $1.90 to $0.95 swing). You are comparing against $0.67 hosted and a tuned $0.30 self-hosted.
  • Turn on prefix caching and continuous batching today. For chat or RAG with shared system prompts, RadixAttention is several-fold throughput for zero cost. Validate it on replayed production traffic, never a synthetic benchmark, or it will read as zero benefit.
  • Enable MTP or speculative decoding next, and watch acceptance rate, not headline speedup. Target above 70 percent; below that your draft model is wrong for your domain. Validate at your real batch size, because P-EAGLE's 1.69x at low load collapses toward 1.05x at concurrency 64.
  • Reach for prefill-decode disaggregation when TTFT degrades under load while decode stays fine. That signature means prefill is starving decode. Expect roughly 2x and 6-to-8x better TTFT before adding a GPU.
  • Move to FP8 now, then pilot FP4 on Blackwell where eval quality holds. FP8 roughly halves cost per token; back FP4 out where accuracy slips.
  • Score every GPU option on throughput per megawatt under your cap, not dollars per hour. Put PUE inside that number: the 17 percent liquid-versus-air gap is free output, and above 45 kW per rack air cooling is off the menu.
  • Treat power as a first-class procurement variable. Push for SLAs on capacity availability, not just uptime, and take behind-the-meter generation seriously while it is still a differentiator. With 40 percent of AI data centers facing power restrictions by 2027, the grid is the constraint, not the GPU.
  • Do not sign multi-year Hopper or Blackwell without modeling the Rubin curve. Build refresh flexibility into the contract, budget at 2x to 3x rather than the 10x headline, and stand up an FP4 validation track this quarter so the next-gen cost win is not gated on work you have not started.

The chip is necessary and nowhere near sufficient. In 2026 the cheapest token belongs to whoever has the power to run it and the stack tuned to serve it. Audit both before you sign for the next GPU.

Sources