Last updated: 2026-06-22

Transformer Parameter Count

The size of a large language model is usually quoted as a single headline number — seven billion, seventy billion — but that figure is determined entirely by a handful of architectural choices. Given the number of layers, the hidden size, the feed-forward expansion, and the vocabulary, the total parameter count follows from a short formula. This calculator applies that formula to give a first-order estimate, the same back-of-the-envelope reasoning used to size a model before any weights exist.

Where the parameters live

Almost every weight in a decoder-only transformer sits in one of three places. The attention sublayer of each block holds four square weight matrices — the query, key, value, and output projections — each of size hidden by hidden. The feed-forward sublayer holds two matrices that expand the hidden stream to a wider inner dimension and project it back. And separate from the stack of layers, the token embedding table stores one hidden-size vector for every entry in the vocabulary.

Counting these gives the architecture estimate. The attention block contributes about $4d^2$ values per layer, where $d$ is the hidden size. The feed-forward block, expanding to an inner width of $m \cdot d$, contributes about $2 m d^2$. The embedding table contributes the vocabulary size $V$ times $d$.

The formula

With $L$ layers, hidden size $d$, feed-forward multiplier $m$, and vocabulary size $V$, the total parameter count is

The first term collects the per-layer attention and feed-forward weights across all layers; the second is the embedding table. Because the per-layer term scales with the square of the hidden size, the width of the model matters far more than the depth: doubling the hidden size roughly quadruples the count, while doubling the layers only doubles it.

Worked example

Take a model with 32 layers, a hidden size of 4096, a feed-forward expansion of four, and a vocabulary of 32,000 tokens. The per-layer factor is $4 + 2 \times 4 = 12$, so

The layer stack accounts for almost all of it; the embedding table adds only about 131 million, a little over two percent. With the common expansion of four, roughly two thirds of each layer sits in the feed-forward weights and one third in attention.

What the estimate leaves out

This is a first-order count. It captures the matrices that dominate the total but omits the small terms: biases, the scale and shift values of layer normalization, and any learned position embeddings. Together these are typically well under one percent of a large model, so they do not move the headline figure. The estimate also assumes the input embedding and the output projection share one matrix — tied embeddings — so the table is counted once; a model with untied embeddings carries that table twice.

Once the count is known, it drives the practical questions of deployment. The memory needed to hold the weights depends on the count and the numeric precision, explored in the Model Size from Quantization Calculator. The memory required to train the same model, which adds optimizer state and activations on top of the weights, is covered in the LLM Training VRAM Calculator.