Learning Rate Warmup Calculator
Inputs
| Peak learning rate | 0.0003 |
|---|---|
| Minimum learning rate | 3e-5 |
| Warmup steps | 2,000 |
| Total steps | 100,000 |
| Current step | 1,000 |
Learning Rate Warmup Calculator
Compute the learning rate at any training step for the standard transformer schedule: a linear warmup to a peak followed by cosine decay toward a floor, plus the warmup fraction.
Inputs
Schedule
Results
Enter a value to see results.
Details
Learning Rate Warmup
The learning rate is the single most consequential hyperparameter in neural-network training, and modern transformers rarely hold it constant. The standard recipe ramps it up from zero to a peak over the first steps, then decays it along a cosine curve toward a small floor for the rest of the run. This calculator returns the learning rate at any chosen step under that schedule, along with the fraction of training spent in warmup.
Why the rate changes over time
At the start of training the weights are random and the optimizer's running estimates of gradient statistics are unreliable, so large updates can send the loss diverging. A short warmup, during which the rate climbs linearly from zero, keeps early updates small while the optimizer stabilizes. Once warmup ends, a high rate accelerates progress, but holding it high to the end leaves the model bouncing around a minimum rather than settling. Decaying the rate — here along a cosine curve — lets the model take large steps early and progressively finer steps as it converges.
The schedule
Two phases govern the rate. During warmup, while the current step is at most the warmup length , the rate rises linearly to the peak . After warmup, it follows a cosine decay from the peak toward the floor across the remaining steps up to the total .
ηη=ηmaxws=ηmin+21(ηmax−ηmin)(1+cosS−wπ(s−w))(s≤w)(s>w)The warmup fraction is simply . At the midpoint of decay the cosine term is zero, so the rate sits exactly halfway between the peak and the floor.
Worked example
Take a peak of 0.0003, a floor of 0.00003, 2,000 warmup steps, and 100,000 total steps. At step 1,000 — still inside warmup — and at step 51,000 — the midpoint of decay — the rates are
η1000η51000=0.0003×20001000=0.00015=0.00003+21(0.0003−0.00003)(1+cos2π)=0.000165Halfway through warmup the rate is half the peak. At the decay midpoint, where the cosine equals zero, it lands exactly between peak and floor. Warmup here spans 2,000 of 100,000 steps, or 2% of the run.
Choosing the values
The peak rate dominates the schedule and is usually found empirically, often with a range test that raises the rate until the loss stops improving. Published peaks for large transformers commonly fall between roughly 1e-4 and 6e-4, and the peak tends to scale with batch size, so changing the effective batch size generally calls for re-tuning it. Warmup of a few hundred to a few thousand steps, often a small percentage of the run, is typical. The floor is frequently set to about a tenth of the peak, or to zero when full decay is wanted.
Limits
Frameworks differ in details this model does not capture: some measure the schedule in tokens rather than steps, some parameterize the floor as a ratio of the peak, and some add a final constant or linear tail. The cosine-with-warmup form here is the common default, but the right hyperparameters are problem-specific and should be confirmed by experiment. The compute that a given step count represents is estimated by the Training Time and Cost Calculator, and the hardware bill behind it by the GPU Cloud Cost Calculator.
Frequently Asked Questions (FAQ)
Why warm the learning rate up instead of starting at the peak?
Early in training the optimizer has little reliable information: adaptive methods have noisy running estimates of gradient statistics, and the randomly initialized weights produce large, erratic updates. Starting at the full peak rate in that state can destabilize or diverge the run.
Ramping the rate up over the first steps lets the optimizer settle while updates are small, after which the peak rate can be applied safely. Warmup is especially important for large batch sizes and deep transformers, where the early instability is more pronounced.
What schedule does this calculator model?
It models the schedule used by most modern transformer training: a linear warmup from zero to the peak over the warmup steps, followed by a cosine decay from the peak toward the minimum over the remaining steps. The cosine shape spends more time near the high rate early in decay and eases gently into the floor at the end, which empirically trains well.
Other schedules exist — constant, linear decay, inverse square root, and step decay — but linear warmup with cosine decay is the common default that this tool reproduces.
How is the peak learning rate chosen?
The peak rate is the most important hyperparameter of the schedule and is usually found empirically, often with a short range test that increases the rate until the loss stops improving and then backs off.
Published values for large transformers commonly fall between roughly 1e-4 and 6e-4 for the largest models, with smaller models tolerating somewhat higher rates. The peak typically scales with batch size, so a change in effective batch size usually calls for re-tuning it.
Disclaimer
This reproduces the common linear-warmup-then-cosine-decay schedule; actual frameworks vary in details such as whether decay is measured in steps or tokens and how the minimum is parameterized. Optimal hyperparameters are problem-specific and should be confirmed empirically.