Last updated: 2026-06-29

Floating-Point Precision

Computers store real numbers in a finite number of bits, so most values are kept only approximately. This calculator reports the precision of the IEEE 754 binary formats — half, single, double, and quadruple — by computing the machine epsilon, the number of reliable decimal digits, and the spacing between representable numbers (the ULP) at a value you choose. Together these explain why floating-point arithmetic is precise but not exact.

How a float stores a number

An IEEE 754 number is written as a sign, a significand (the significant digits), and an exponent that scales it by a power of two — much like scientific notation in base 2. Each format fixes how many bits the significand gets: half precision has 11 bits of significand, single has 24, double has 53, and quadruple has 113 (each count includes one implicit leading bit). The more significand bits, the finer the resolution.

Machine epsilon

The cleanest measure of resolution is machine epsilon, the gap between 1.0 and the next representable number above it:

$\varepsilon = 2^{-(p-1)}$

where $p$ is the number of significand bits. For double precision, $\varepsilon = 2^{-52} \approx 2.22 \times 10^{-16}$. This is a relative bound: any real number is stored with a relative error of at most $\varepsilon / 2$. The reliable number of significant decimal digits follows directly, as $(p-1)\log_{10} 2$ — about 15.7 for double precision.

ULP: spacing that grows with magnitude

Machine epsilon describes the spacing near 1.0, but floating-point numbers are not evenly spaced. The gap between consecutive representable values, the unit in the last place (ULP), scales with magnitude:

$\text{ULP}(x) = 2^{\lfloor \log_2 |x| \rfloor - (p-1)}$

Near 1 the spacing is machine epsilon; near 1,000 it is about a thousand times larger. This is why floating-point keeps constant relative precision but loses absolute precision as numbers grow.

Worked example

Take double precision and the value 1000. The exponent is $\lfloor \log_2 1000 \rfloor = 9$, so:

The largest rounding error near 1000 is half of that, about $5.7 \times 10^{-14}$. So a double can distinguish 1000 from 1000.0000000000001 but not from a value differing in the sixteenth significant digit.

Why 0.1 is not exact

Binary floating-point can only represent sums of powers of two, and 0.1 has no finite binary expansion — just as one third has none in decimal. It is therefore rounded to the nearest representable value, which is why 0.1 + 0.2 does not exactly equal 0.3 in most languages. The discrepancy is on the order of machine epsilon and usually harmless, but it accumulates over long computations and is the reason money is often stored as integers or in a decimal type rather than as a float. For how integers themselves are encoded in binary, see the Two's Complement Calculator.