What Is Mixed Precision Training in TensorFlow?

Mixed precision is a training technique that combines 16-bit floating-point operations (float16 or bfloat16) with 32-bit operations (float32) to speed up deep learning training while preserving numerical stability. According to TensorFlow’s own mixed precision guide, the technique improves performance by more than 3x on modern NVIDIA GPUs with compute capability 7.0 or higher, by around 60% on Cloud TPUs, and by more than 2x on recent Intel CPUs — by running most operations in float16 (or bfloat16 on TPUs) while keeping the loss and output layer in float32.

When you are training large Machine Learning models, you want to minimise the training time. In TensorFlow, you can enable mixed precision model training, which gives a real performance improvement because it runs most operations at 16 bits (float16) instead of full single precision (float32). Google TPUs and NVIDIA GPUs have dedicated hardware — Tensor Cores, on the NVIDIA side — built to run 16-bit operations faster than 32-bit ones, see Mixed precision. For the official API reference covering tf.keras.mixed_precision.set_global_policy and the full list of supported policies, see TensorFlow’s mixed_precision documentation. The performance gain comes from two things: float16 values take half the memory of float32, so more data fits through memory bandwidth per second, and Tensor Cores execute float16 matrix multiplications and convolutions directly in hardware. In this post, I will outline the relevant data types and show you the main steps for setting up mixed-precision training in TensorFlow.

Computer Data Types: Bits, Bytes, and Numeric Ranges

Computers store and process data as bits, each one set to 0 (no signal) or 1 (signal present) — that on/off state is what flows through the circuitry, and it is called binary data representation. You are used to thinking in decimal (powers of 10), so we group bits into bytes of 8, and from there into larger formats built for different kinds of numbers. The more bits a format uses, the more values it can store. There are other numerical bases too — octal, hexadecimal — and converting between them is straightforward, see Computer number format.

Decimal Values	Binary Values
0	0000
1	0001
2	0010
3	0011
4	0100
5	0101
6	0110
7	0111
8	1000
9	1001
10	1010

Table 1. Values in Decimal and Binary Bases

In one byte, we can store 2^8 possible values. In 4 bytes, we can store 2^32 values. You can also represent fractions using fixed-point numbers, where you set aside a fixed number of bits for the integer part and a fixed number for the fractional part. That works, but you cannot represent exact fractions such as 1/3 this way — the recurring tail of 3s gets truncated once you run out of fractional bits. The range of stored values is limited by the number of bits used, and that limit is exactly why precision loss matters whenever a computation needs a specific numeric range.

Floating-point Format: float16 vs float32 vs float64

The floating-point format is a numeric representation that stores a sign, an exponent, and a significand (mantissa) so the decimal point can “float” to any position, letting a fixed number of bits encode both very large and very small numbers. This works like scientific notation: a significand multiplied by an exponent, which lets you represent both very large and very tiny values — exactly what you need when computing neural networks. Table 2 shows the layout for the three precision levels defined by the IEEE 754 standard.

Mixed Precision in TensorFlow: Setup and the mixed_float16 Policy

🔒 Subscribe to keep reading.

Final Thoughts: When to Use Mixed Precision Training

🔒 Subscribe to keep reading.

References

🔒 Subscribe to keep reading.