Int8 Quantization for Microcontrollers: How It Works

The float model is 92% accurate. After conversion to int8 for the microcontroller it’s 71%, and the team concludes that the model “doesn’t quantize well”. Usually it does. Int8 quantization for microcontrollers is a mapping from real numbers to 256 integer levels, and the accuracy it keeps depends mostly on choosing the right range for each tensor. This article explains the arithmetic (scales, zero points, per-channel weights and requantization), shows with worked numbers where accuracy is lost, and summarizes what the research says about post-training quantization versus quantization-aware training.

Key Takeaways

• Each int8 tensor stores real values as (q − zero point) × scale, so 256 levels have to cover the tensor’s range. The range choice decides the precision.
• Int8 weights are four times smaller than float32 and run on integer SIMD kernels, which is what makes microcontroller inference practical.
• Per-channel weight scales matter: with one scale per tensor, a channel whose weights are small can end up using only a handful of the 255 levels.
• Calibration data sets activation ranges. A single outlier can stretch the range and leave typical values with very few levels.
• With per-channel weights, post-training quantization stayed within about 2% of float accuracy across the CNNs in one Google study, and quantization-aware training within about 1%.

The problem: “it lost 20 points when we quantized it”

Large accuracy drops after quantization almost always trace back to a handful of causes. The calibration dataset was a few random images, or synthetic data, so activation ranges don’t match real inputs. A layer has a few extreme activations that stretch its range. Weights were quantized with one scale for the whole tensor while some channels have much smaller weights than others. The input preprocessing on the device differs from the one used during calibration. Or an operation that’s sensitive to precision, such as a final softmax or a residual addition with mismatched scales, was quantized carelessly.

Each of these is a range problem, and each can be found by comparing float and int8 outputs layer by layer.

How int8 quantization works

Scale and zero point

The scheme most embedded runtimes use was described by Jacob and colleagues in their 2018 CVPR paper on integer-arithmetic-only inference. A real value r is represented by an integer q through an affine mapping: r = S × (q − Z), where S is a positive real scale and Z is the integer zero point, chosen so that the real value 0 is exactly representable. Google’s LiteRT (formerly TensorFlow Lite) 8-bit quantization specification states the same formula and the conventions current tools follow: activations use int8 in [−128, 127] with an arbitrary zero point, one scale per tensor; weights use int8 in [−127, 127] with zero point 0 (symmetric), and can have one scale per output channel; biases are int32 with a scale equal to the input scale times the weight scale.

A worked example. Suppose calibration shows that a layer’s activations range from −2.5 to 6.0. The scale is (6.0 − (−2.5)) / 255 ≈ 0.0333, and the zero point that puts −2.5 at −128 is Z = −53. Check: 6.0 maps to 6.0 / 0.0333 − 53 = 127, and 0.0 maps exactly to −53. The worst-case rounding error inside the range is half a step, about 0.017.

Why per-channel weight scales matter

A convolution’s output channels can have very different weight magnitudes. With one symmetric scale for the whole tensor, the scale is set by the largest weight. As an illustrative example, if the largest weight in a layer is 0.42, the per-tensor scale is 0.42 / 127 ≈ 0.0033. A channel whose weights never exceed 0.02 then uses only about ±6 of the available ±127 levels, which is barely more than 3 bits of precision. Giving each channel its own scale restores the full range for every channel.

Raghuraman Krishnamoorthi’s 2018 Google white paper on quantizing convolutional networks measured the effect: post-training quantization with per-channel weights and per-layer activations stayed within about 2% of floating-point accuracy across a range of CNN architectures, and it recommends exactly that combination for hardware.

Calibration and outliers

Weights are known after training, but activation ranges depend on the data. Post-training quantization runs a calibration set through the float model and records each tensor’s range. If that set doesn’t look like real inputs, the ranges are wrong.

Outliers are the other trap. Illustratively, if a layer’s activations are usually between 0 and 3 but a single calibration sample reaches 30, a min-max range gives a scale of 30 / 255 ≈ 0.118, and typical values between 0 and 3 are represented by only about 25 levels. Calibration methods that clip to a percentile of the observed distribution, or that minimize the quantization error, trade a little clipping of rare outliers for much better precision on common values.

Requantization: integer-only from layer to layer

Inside a quantized layer, int8 inputs and int8 weights are multiplied and summed in an int32 accumulator. The result has scale S_in × S_w, and it has to be converted to the output tensor’s scale S_out before the next layer. Jacob and colleagues showed how to do this without floating point: the ratio M = S_in × S_w / S_out is a real number between 0 and 1, represented as a normalized fixed-point integer multiplier and a bit shift. In the running example, with an input scale of 0.0333, a weight scale of 0.0033 and an output scale of 0.05, M ≈ 0.0022, which is 0.564 × 2⁻⁸, so it’s stored as 0.564 in 32-bit fixed point plus a right shift of 8 bits. Every layer runs on the integer pipeline that microcontroller SIMD instructions accelerate.

Post-training quantization or quantization-aware training

Nagel and colleagues’ 2021 white paper on neural network quantization summarizes the trade-off: post-training quantization (PTQ) needs no retraining and no labelled data, and in most cases it’s sufficient for 8-bit quantization with close to floating-point accuracy. Quantization-aware training (QAT) simulates quantization during fine-tuning so the network learns to tolerate it; it needs the training pipeline and labelled data but recovers more accuracy and enables lower bit widths. Krishnamoorthi’s numbers make the same point: QAT narrowed the 8-bit gap to about 1%, and 4-bit weights with QAT lost between 2% and 10%, more for smaller networks.

The memory and speed gains come from the bit width either way. Krishnamoorthi’s paper notes that 8-bit weights cut model size by four even on hardware without 8-bit arithmetic, and on microcontrollers the integer kernels are what make inference fast enough to be useful.

Int8 quantization choices compared

Choice	Option A	Option B	Effect on accuracy	When to choose B
Weight scale	Per-tensor	Per-channel	B avoids starving small channels	Almost always for convolutions
Weight symmetry	Asymmetric	Symmetric (Z = 0)	Small; B is simpler and faster	Default in current specs
Activation range	Min-max	Percentile or error-minimizing	B is better with outliers	When activations have long tails
Calibration data	Random or synthetic	Representative real inputs	B gives correct ranges	Always
Method	Post-training (PTQ)	Quantization-aware training (QAT)	B recovers more, about 1% gap in one study	When PTQ loses more than you can accept
Bit width	8-bit	4-bit or mixed	B loses more (2–10% with QAT in one study)	Only when memory forces it

What to actually do

Compare float and int8 layer by layer

Run the same inputs through the float and quantized models and compare each layer’s output. The first layer where the error jumps is where the range or scheme is wrong.

Use per-channel weights and real calibration data

Convert with per-channel weight quantization and a few hundred representative real inputs, preprocessed exactly as the device will preprocess them.

Handle outliers deliberately

Inspect activation histograms for long tails. Use percentile calibration, or fix the cause in the model (an unbounded activation can often be replaced by a bounded one before training).

Escalate to QAT only when needed

If PTQ with good calibration still misses the accuracy target, fine-tune with quantization-aware training rather than abandoning int8.

Verify on the device

Run the quantized model on the microcontroller with on-device features and compare its outputs with the host’s quantized model. Bit-exact agreement isn’t guaranteed across implementations, but differences should be tiny.

What not to do

Don’t judge a model’s “quantizability” from a conversion with random calibration data. And don’t move to 4-bit weights to save memory before checking where the memory is actually going; activations and buffers often dominate RAM.

How to put it to work

If you have an hour

Quantize your model twice, with per-tensor and per-channel weights, using the same real calibration set, and compare accuracy. If per-channel recovers most of the loss, you’ve found your problem.

If you have a sprint

Build a layer-by-layer comparison between the float and int8 models on a few hundred real inputs, plot the error per layer and fix the worst layers’ ranges. Then run QAT only if the target still isn’t met.

If you want it to last the product’s lifetime

Store the calibration dataset, the quantization settings and the float-versus-int8 accuracy with each model release, and refresh the calibration set when the sensor, the installation or the preprocessing changes.

The question to ask yourself: “When my model lost accuracy in int8, did I find out which layer’s range caused it, or did I blame the bit width?”

Frequently asked questions

What is int8 quantization?

Int8 quantization represents a neural network’s weights and activations as 8-bit integers, each tensor with a scale and a zero point that map integers back to real values: r = S × (q − Z). It lets the network run with integer arithmetic in a quarter of the memory of 32-bit floats.

How much accuracy does int8 quantization lose?

Often very little. In one Google study, post-training quantization with per-channel weights stayed within about 2% of float accuracy across several CNNs, and quantization-aware training within about 1%. Larger losses usually come from poor calibration, outliers or per-tensor weight scales.

What is the difference between per-tensor and per-channel quantization?

Per-tensor quantization uses one scale for a whole weight tensor; per-channel uses one scale for each output channel. Per-channel prevents channels with small weights from being represented by only a few integer levels, and current specifications support it for convolution weights.

What is calibration in post-training quantization?

Calibration runs a set of representative inputs through the float model to measure each activation tensor’s range, which sets its scale and zero point. Calibration data that doesn’t match real inputs gives wrong ranges and lower accuracy.

When should I use quantization-aware training?

When post-training quantization with per-channel weights and good calibration still misses your accuracy target, or when you need fewer than 8 bits. QAT simulates quantization during fine-tuning so the network adapts to it.

Related reading: running inference on a Cortex-M, from sensor input to embedded inference and how to measure an embedded AI demo.

Sources

1. Jacob, B., Kligys, S., Chen, B., et al. (2018). “Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference.” 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2704–2713. https://doi.org/10.1109/CVPR.2018.00286
2. Krishnamoorthi, R. (2018). “Quantizing Deep Convolutional Networks for Efficient Inference: A Whitepaper.” arXiv:1806.08342. https://arxiv.org/abs/1806.08342
3. Nagel, M., Fournarakis, M., Amjad, R. A., Bondarenko, Y., van Baalen, M., & Blankevoort, T. (2021). “A White Paper on Neural Network Quantization.” arXiv:2106.08295. https://arxiv.org/abs/2106.08295
4. Google AI Edge. “LiteRT 8-bit Quantization Specification.” https://developers.google.com/edge/litert/conversion/tensorflow/quantization/quantization_spec
5. Lai, L., Suda, N., & Chandra, V. (2018). “CMSIS-NN: Efficient Neural Network Kernels for Arm Cortex-M CPUs.” arXiv:1801.06601. https://arxiv.org/abs/1801.06601