Running Inference on a Cortex-M: What Limits the Speed

A keyword-spotting model runs in 20 ms on one development board and 400 ms on another, both labelled “Arm Cortex-M”. The model and the runtime are identical. The difference is the core’s instruction set, the memory the activations live in and whether the optimized kernels were linked in. Running inference on a Cortex-M is a hardware question as much as a model question, and a few numbers (multiply-accumulates per inference, peak activation memory and the core’s MAC throughput) predict most of the outcome. This article explains what determines speed and fit on Cortex-M microcontrollers, using the published results behind the main kernels and runtimes.

Key Takeaways

• Count multiply-accumulates (MACs) per inference and peak activation memory before anything else; they predict latency and fit.
• Cortex-M cores differ a lot. Cores with DSP extensions can do two 16-bit MACs in one instruction (SMLAD); Helium cores (Cortex-M55 and later) have 128-bit vectors with sixteen 8-bit lanes.
• Optimized kernels matter: CMSIS-NN reported 4.6× throughput and 4.9× energy efficiency over a baseline on a Cortex-M7.
• RAM is usually the first limit. In the CMSIS-NN CIFAR-10 example, partial im2col cut peak memory from about 332 KB to about 133 KB.
• Whole-network memory scheduling can cut memory further: MCUNet’s TinyEngine reported 4.8× lower memory use than TF Lite Micro and CMSIS-NN.

The problem: “Cortex-M” is a family, not a performance figure

Cortex-M spans cores with very different capabilities. Some are designed for minimum area and power, with no DSP instructions and no floating-point unit. Others add DSP SIMD instructions and a floating-point unit. The newest add vector processing. On top of the core, each microcontroller adds its own memory system: flash with wait states, SRAM split into banks, sometimes tightly coupled memory or caches.

An engineer choosing a part for an AI feature is usually asking two questions: will the model fit, and will it run in time? Neither is answered by the core’s name or clock frequency alone.

What determines speed and fit

Multiply-accumulates are the workload

Most of the work in a convolutional or fully connected network is multiply-accumulate operations. Count them per layer: for a convolution, output height × output width × kernel height × kernel width × input channels × output channels.

An illustrative layer: a 3 × 3 convolution with 32 input and 32 output channels on a 25 × 5 feature map needs 25 × 5 × 3 × 3 × 32 × 32 = 1,152,000 MACs. At an idealized two MACs per cycle, that’s 576,000 cycles, about 7.2 ms at 80 MHz, before any overhead for loading data, requantizing outputs and loop control. Real kernels don’t reach the ideal, but the count tells you the order of magnitude and which layers dominate.

SIMD and vector instructions set the MAC rate

Lai, Suda and Chandra’s CMSIS-NN paper explains how Arm’s optimized kernels use the DSP instructions available on Cortex-M cores with SIMD support. Int8 inputs are expanded to 16 bits and fed to SMLAD, which performs two 16-bit multiply-accumulates in one instruction, with 32-bit accumulators. Without those instructions, each MAC costs at least one multiply and one add.

Arm’s Helium technology (the M-Profile Vector Extension) goes further. Arm’s Introduction to Helium guide describes 128-bit vector registers that can hold sixteen 8-bit elements, and names the Cortex-M55 as the first processor to support it. Vector instructions on int8 data are what make larger models practical on recent Cortex-M parts, and some microcontrollers pair the core with a dedicated neural processing unit for further gains.

Kernels and runtimes turn instructions into throughput

Having the instructions isn’t enough; the software has to use them. CMSIS-NN’s benchmark ran a small CIFAR-10 network (three convolution layers and one fully connected layer, quantized to 8 bits) on a Cortex-M7 at 216 MHz. It took about 99 ms per image, about 10 images per second, with 79.9% accuracy after quantization against 80.3% before. The paper reports that the optimized kernels gave 4.6× higher throughput and 4.9× higher energy efficiency than a baseline built from generic DSP functions.

Runtimes such as TensorFlow Lite Micro sit above the kernels. Its authors describe an interpreter designed for systems without dynamic memory allocation, with operator implementations that can be replaced by optimized versions for a given core. Whether those optimized versions are linked into your build is one of the most common reasons two “identical” deployments run at very different speeds.

Memory is usually the first limit

Weights live in flash; activations live in RAM, and the RAM needed is set by the largest pair of input and output tensors that must exist at the same time, plus any scratch buffers. In the CMSIS-NN CIFAR-10 example, the weights took about 87 KB and the activations about 55 KB. A full im2col transformation, which reorganizes input patches into a matrix for fast multiplication, would have pushed peak memory to about 332 KB, too large for the board. Expanding only a couple of columns at a time (partial im2col) kept peak memory to about 133 KB while keeping most of the speed.

Lin and colleagues’ MCUNet work at NeurIPS 2020 attacked the same limit from both sides. They noted that microcontroller memory is two to three orders of magnitude smaller than a phone’s, designed network architectures to fit specific memory budgets (TinyNAS) and built an inference engine that plans memory across the whole network instead of layer by layer (TinyEngine). They reported 4.8× lower memory use and 1.7–3.3× faster inference than TF Lite Micro and CMSIS-NN, and the first ImageNet top-1 accuracy above 70% on an off-the-shelf microcontroller.

Quantization makes it possible

All of this assumes integer arithmetic. Jacob and colleagues’ integer-only inference scheme, which stores weights and activations as 8-bit integers with scales and zero points, is what most current runtimes use. CMSIS-NN’s original paper used a simpler power-of-two scaling so that no floating-point dequantization was needed between layers, which matters on cores without a floating-point unit. Either way, int8 weights take a quarter of the flash of float32 and run on integer SIMD instructions.

Where the data lives changes the speed

Code and weights executing from flash may stall on wait states at high clock frequencies, depending on the part’s flash accelerator or cache. Activations in tightly coupled or zero-wait-state SRAM run faster than in slower banks. These effects are specific to each microcontroller, so read the part’s reference manual and measure; the same core at the same clock can run a model at noticeably different speeds on two parts.

Cortex-M inference factors compared

Factor	What it affects	Typical impact	How to check
MAC count per inference	Latency, energy	Sets the workload; dominant layers stand out	Per-layer count from the model
Core instruction set (no SIMD / DSP SIMD / Helium)	MACs per cycle	Large; SIMD gives 2 × 16-bit MACs per SMLAD, Helium has 16 int8 lanes per vector	Core’s technical reference manual
Optimized kernels linked	Throughput, energy	CMSIS-NN reported 4.6× throughput, 4.9× energy efficiency	Build configuration, profiling
Peak activation memory	Whether it fits	Often the binding constraint; partial im2col 332 → 133 KB in one example	Runtime’s arena report, per-layer tensor sizes
Memory scheduling across the network	Peak RAM	TinyEngine reported 4.8× lower memory use	Runtime documentation, measurement
Flash wait states, caches, TCM placement	Real speed at a given clock	Part-specific	Reference manual; measure at shipping clock
Quantization scheme	Accuracy, need for FPU	Int8 is 4× smaller than float32	Accuracy of quantized model on test set

What to actually do

Count MACs and peak memory first

Before choosing a part, compute MACs per layer and the peak activation memory for the model you plan to run. Add sample and feature buffers to the RAM figure.

Match the core to the workload

Divide the MAC count by a realistic MAC rate for the core class and clock, then compare with your deadline. If it doesn’t fit with margin, change the model, the part, or the deadline, in that order of cost.

Make sure the optimized kernels are used

Check the build: the runtime’s optimized operator implementations for your core should be enabled and linked. Profile per layer; a layer running far slower than its MAC count suggests is often falling back to a reference kernel.

Place memory deliberately

Put the tensor arena in the fastest RAM available and check flash wait states at your clock. Measure before and after.

What not to do

Don’t compare Cortex-M parts by clock frequency alone. And don’t assume a model that fits in flash will run; RAM for activations is usually the first wall.

How to put it to work

If you have an hour

Take your model, list each layer’s MACs and output tensor size, and find the layer with the most MACs and the point of peak memory. Those two layers are where optimization pays.

If you have a sprint

Profile the model on the target board layer by layer, with and without the optimized kernels, and record latency per layer. Compare with the MAC counts to find layers that underperform.

If you want it to last the product’s lifetime

Keep a per-layer profile as part of each model release, on the shipping clock and memory configuration, so a model update or a part change shows its cost before it reaches the field.

The question to ask yourself: “Do I know how many MACs and how much RAM my model needs, and what my chosen core can actually deliver at the clock I’ll ship?”

Frequently asked questions

Can a Cortex-M run a neural network?

Yes. Small quantized networks for tasks such as keyword spotting, gesture recognition and anomaly detection run on many Cortex-M microcontrollers using runtimes such as TensorFlow Lite Micro and kernels such as CMSIS-NN. The limits are RAM for activations and the core’s multiply-accumulate throughput.

Which Cortex-M is best for machine learning?

Cores with DSP SIMD instructions run int8 inference much faster than cores without them, and cores with Helium vector instructions, starting with the Cortex-M55, are faster again. The best choice depends on the model’s MAC count, its memory needs and the power budget.

What is CMSIS-NN?

CMSIS-NN is Arm’s library of optimized neural network kernels for Cortex-M processors. Its original paper reported 4.6× higher throughput and 4.9× better energy efficiency than a baseline implementation on a Cortex-M7, by using SIMD multiply-accumulate instructions and memory-saving techniques such as partial im2col.

How do I know if my model will fit on a microcontroller?

Check flash for code, runtime and int8 weights, and RAM for the peak activation memory plus sample and feature buffers. The runtime’s tensor arena requirement, measured on the device, is the reliable figure for activations.

Why is my model slower on one Cortex-M board than another?

Common reasons are a different core (with or without SIMD or Helium), optimized kernels not linked into one build, a different clock or flash wait states, and the tensor arena placed in slower memory.

Related reading: from sensor input to embedded inference, int8 quantization for microcontrollers and how to measure an embedded AI demo.

Sources

1. Lai, L., Suda, N., & Chandra, V. (2018). “CMSIS-NN: Efficient Neural Network Kernels for Arm Cortex-M CPUs.” arXiv:1801.06601. https://arxiv.org/abs/1801.06601
2. Lin, J., Chen, W.-M., Lin, Y., Cohn, J., Gan, C., & Han, S. (2020). “MCUNet: Tiny Deep Learning on IoT Devices.” Advances in Neural Information Processing Systems 33 (NeurIPS 2020). arXiv:2007.10319. https://arxiv.org/abs/2007.10319
3. David, R., Duke, J., Jain, A., Reddi, V. J., et al. (2021). “TensorFlow Lite Micro: Embedded Machine Learning on TinyML Systems.” Proceedings of Machine Learning and Systems (MLSys). arXiv:2010.08678. https://arxiv.org/abs/2010.08678
4. Jacob, B., Kligys, S., Chen, B., et al. (2018). “Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference.” 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2704–2713. https://doi.org/10.1109/CVPR.2018.00286
5. Arm Ltd. (2024). Introduction to Helium, Version 1.3 (Document 102102). https://documentation-service.arm.com/static/65969ed3159ca73387227541