Last updated: 2026-06-22

Inference energy cost explained

Inference energy cost is the electricity expense, and the associated carbon emissions, of serving a volume of language-model inference requests. It is derived from a per-query energy figure scaled by the number of queries, then priced at an electricity rate and converted to carbon using the local grid's emission intensity. Unlike training, which is a one-time event, inference energy recurs with every request a deployed model answers, so at scale it dominates the operational footprint of a served model.

The formulas

Let $N$ be the number of queries and $E_q$ the energy per query. Total energy is their product, the electricity cost multiplies energy by the price $p_e$ per kilowatt-hour, and emissions multiply energy by the grid intensity $I_g$ in grams of carbon dioxide per kilowatt-hour, divided by 1000 to yield kilograms:

Energy per query is small — a short completion on a well-utilized accelerator is often a fraction of a watt-hour — so the figures are converted to kilowatt-hours before pricing and emissions are applied.

Worked example

Consider N = 1{,}000{,}000 queries at $E_q = 0.5$ watt-hours each. Total energy is

At an electricity price of p_e = \0.15$ per kilowatt-hour, the cost is

At a grid intensity of $I_g = 400$ grams of carbon dioxide per kilowatt-hour — a mid-range mixed grid — the emissions are

A million served queries thus cost about $75 in electricity and release roughly 200 kg of carbon dioxide, before cooling and idle overhead beyond what $E_q$ already includes.

Variations and application

The three inputs each shift the result independently. Per-query energy depends on model size, sequence length, hardware, and batching; a long generation from a large model on under-utilized hardware draws far more than a short one. Electricity price varies by region and contract, and grid intensity ranges from under one hundred grams per kilowatt-hour on hydro- or nuclear-heavy grids to over seven hundred on coal-heavy ones, so the same workload's emissions can differ several-fold by location of the compute. The energy and cost feed infrastructure choices such as the Self-Host vs API Break-even Calculator analysis, where electricity is part of the self-host variable cost. Training energy is excluded: it is a separate, one-time cost amortized across every query the model ever answers, so a full lifecycle figure must account for it separately. The result is an order-of-magnitude estimate, not a metered measurement.