From Sensor Input to Embedded Inference: The Full Pipeline

The model scored 96% in the notebook. On the microcontroller it misses events, fires on noise and drains the battery in two weeks. The network is the same; what changed is everything around it. Embedded inference on a sensor is a pipeline that starts at the analog front end and ends at a decision, and the model is only one stage. This article walks through every stage between sensor input and the output of a tiny neural network (sampling, buffering, feature extraction, the quantized model, post-processing) and shows how to budget time, memory and energy across them, with the research that shaped the current tools.

Key Takeaways

• Treat inference as a pipeline with a deadline: features and inference must finish before the next window arrives, or samples are lost.
• Features must be computed on the device exactly as they were in training. A different window, scale or filter is a different input to the model.
• Embedded processors have roughly 100 to 1,000 times less compute, memory and power than mobile ones, according to the TensorFlow Lite Micro authors, so models and runtime must fit in kilobytes.
• Int8 quantization cuts weight storage by four relative to float32 and enables integer-only kernels; optimized Cortex-M kernels reported 4.6× throughput and 4.9× energy-efficiency gains.
• Accuracy alone is not a result. MLPerf Tiny reports accuracy, latency and energy together, and so should you.

The problem: a good model in a bad pipeline

Most failed edge-AI demos fail outside the network. Samples arrive with jitter because the ADC is triggered from a busy loop. The window on the device is 1,000 samples while training used 1,024. The training data were normalized with statistics the firmware doesn’t have. The model outputs a probability every 100 ms and the application treats every single frame above 0.5 as an event. The microcontroller never sleeps because inference runs continuously.

None of these show up in a notebook accuracy score. They show up as missed detections, false alarms and battery life, which are the things a customer actually notices.

What happens at each stage

Stage 1: Sampling on a clock, not in a loop

Everything downstream assumes evenly spaced samples. On a microcontroller that means triggering the ADC (or reading a digital sensor’s FIFO) from a hardware timer and moving the data with DMA, so that the CPU’s other work can’t shift the sample instants. The anti-aliasing filter and sample rate follow the same rules as any data acquisition system; a model can’t recover information that aliasing destroyed.

The data rate is easy to underestimate. An illustrative example: a three-axis accelerometer at 3,200 samples per second with 16-bit samples produces 19,200 bytes per second.

Stage 2: Buffering into windows

Models consume windows, not samples. The firmware collects a window (say 1 s), hands it to feature extraction, and keeps collecting the next one. With a double buffer (ping-pong), the DMA fills one half while the CPU processes the other. In the accelerometer example, a 1 s window is 19,200 bytes and the double buffer is 38,400 bytes of RAM before any feature or model memory.

The hop between windows sets the deadline. With 50% overlap, a new window arrives every 0.5 s, so feature extraction plus inference must finish in well under 500 ms, and ideally much less so the processor can sleep.

Stage 3: Feature extraction, the stage that must match training

Raw samples are rarely the model input. Audio models usually take spectral features such as MFCCs or log-mel energies; vibration models often take FFT magnitudes or band energies; slow sensors may take simple statistics over the window. Zhang and colleagues’ Hello Edge study of keyword spotting on microcontrollers is a useful reference here: they compared published network architectures under microcontroller memory and compute limits and found that a depthwise-separable CNN reached 95.4% accuracy on their keyword task, about 10 points higher than a fully connected network with a similar number of parameters. Their results depend on the spectral features computed before the network, and so do yours.

The practical rule is strict. Window length, overlap, window function, FFT size, filter coefficients, scaling and normalization constants must be identical in training and on the device. The safest way to ensure it is to generate both from the same code or the same specification, and to test the device’s features against the training pipeline’s features on the same recorded input.

Stage 4: The quantized model

Two things make neural networks fit on a microcontroller: small architectures and integer arithmetic. Jacob and colleagues’ 2018 CVPR paper on integer-arithmetic-only inference described the quantization scheme that most embedded runtimes now use: weights and activations stored as 8-bit integers with a scale and zero point, accumulation in 32-bit integers, and a training procedure that keeps accuracy close to floating point. Storage drops by four: an illustrative model with 50,000 parameters needs about 200 KB as float32 and about 50 KB as int8.

The runtime matters too. The TensorFlow Lite Micro paper by David and colleagues describes the constraints it was designed for: embedded processors with at least 100 to 1,000 times less compute, memory and power than mobile devices, models and runtime that must fit in a few kilobytes, no dynamic memory allocation or virtual memory, and a fragmented landscape of instruction sets with and without floating-point units. Its answer is an interpreter with a statically planned memory arena. Underneath, optimized kernels do the arithmetic; Lai, Suda and Chandra reported that their CMSIS-NN kernels for Arm Cortex-M improved neural network throughput by 4.6× and energy efficiency by 4.9×.

Stage 5: Post-processing into a decision

A model emits scores, frame by frame. The application needs events. Between them sit smoothing (averaging scores over several frames), thresholds with hysteresis (so a score hovering around the threshold doesn’t toggle the output), minimum durations and refractory periods. These choices change the false-alarm and missed-detection rates as much as the model does, and they’re cheap to tune with recorded data.

Stage 6: Budgets for time, memory and energy

Time: features plus inference plus post-processing must fit in the hop interval, with margin for interrupts and communication.

Memory: flash holds the code, the model weights and constant tables; RAM holds sample buffers, feature buffers and the runtime’s tensor arena. The sample buffers alone can be larger than the arena.

Energy: average current is set by the duty cycle. An illustrative calculation: if the processor draws 10 mA while active and 5 µA asleep, and is active 40 ms out of every second, the average is about 0.40 mA and a 1,000 mAh battery lasts about 100 days. Active for 200 ms per second, the same battery lasts about 21 days. Shortening inference, or running it only when a cheap trigger fires, often matters more than any model improvement.

Measuring it like a benchmark

Banbury and colleagues’ MLPerf Tiny benchmark, developed by more than 50 organizations, evaluates tiny ML systems on four tasks (keyword spotting, visual wake words, image classification and anomaly detection) and reports accuracy, latency and energy of inference together. The lesson for any product is the same: a result is a triple, and a change that improves one number while hiding another isn’t an improvement.

Embedded inference pipeline stages compared

Stage	What it does	Typical failure	Budget it consumes	How to check it
Sampling	Timer-triggered ADC or sensor FIFO via DMA	Jitter, aliasing	DMA channel, interrupt load	Timestamp samples; check spacing
Buffering	Collects windows (double buffer)	Overrun when processing is late	RAM (often the largest block)	Overrun counter in firmware
Feature extraction	FFT, MFCC, band energy, statistics	Mismatch with training features	CPU time, RAM, constant tables	Compare device and training features on the same recording
Quantized model	Int8 inference via runtime and kernels	Accuracy loss from quantization; arena too small	Flash (weights), RAM (arena), CPU time	Run the quantized model on recorded data before flashing
Post-processing	Smoothing, thresholds, hysteresis	False alarms, chattering outputs	Small	Replay recordings, count false alarms and misses
Power management	Sleep between windows, wake triggers	Processor never sleeps	Average current	Measure current over a full cycle

What to actually do

Write the deadline first

From the sample rate, window and overlap, calculate the hop interval. That’s the time available for features, inference and post-processing.

Freeze the feature specification

Write down every parameter of feature extraction and generate the training features and the firmware features from the same definition. Test them against each other on recorded data.

Quantize early and test on recorded data

Convert the model to int8 as soon as the architecture is stable and evaluate the quantized model on the same test set as the float model. Don’t wait for the board to discover the accuracy loss.

Budget RAM including the buffers

Add the sample buffers, feature buffers and tensor arena. If they don’t fit, reduce the window or the overlap before shrinking the model.

Tune post-processing on real recordings

Replay hours of real sensor data through the full pipeline and tune smoothing and thresholds for the false-alarm rate the application can tolerate.

What not to do

Don’t report the notebook accuracy as the device’s accuracy. And don’t measure power with inference running continuously if the product will duty-cycle; measure the real cycle.

How to put it to work

If you have an hour

Calculate your hop interval, your RAM budget (buffers plus arena) and your duty cycle at the current inference time. One of the three is probably the real constraint.

If you have a sprint

Record a representative dataset on the target hardware, run it through the device’s feature code and the training code, and compare the features. Then run the quantized model on the device features and compare accuracy with the notebook.

If you want it to last the product’s lifetime

Keep the feature specification, the quantized model, the post-processing parameters and the measured accuracy, latency and energy together, versioned as one release. Re-measure all three when any part changes, including the sensor.

The question to ask yourself: “If I changed one stage of the pipeline tomorrow, would I know whether accuracy, latency and battery life all still hold?”

Frequently asked questions

What is embedded inference?

Embedded inference is running a trained machine learning model directly on a microcontroller or small embedded processor, next to the sensor, instead of sending data to a server. The device acquires the data, computes features, runs the model and acts on the result locally.

Why does my model work in Python but not on the microcontroller?

The most common reasons are features computed differently on the device than in training, accuracy loss from quantization that wasn’t tested, sampling problems such as jitter or aliasing, and post-processing that treats single frames as events. Testing each stage on the same recorded data finds the difference.

How much memory does a TinyML model need?

It depends on the model, but runtimes such as TensorFlow Lite Micro target models and runtimes that fit in kilobytes. Int8 weights take one byte per parameter, so 50,000 parameters need about 50 KB of flash, plus a RAM arena for activations and, often more significant, the sample and feature buffers.

What is int8 quantization and why is it used?

Int8 quantization stores weights and activations as 8-bit integers with a scale and zero point, so inference uses integer arithmetic. It reduces weight storage by four compared with 32-bit floats and runs efficiently on microcontrollers without floating-point units, usually with a small accuracy loss.

How do you benchmark an embedded AI system?

Measure accuracy, latency and energy per inference together on the target hardware, as the MLPerf Tiny benchmark does. Report the conditions (clock speed, memory, runtime, model version) so results are comparable.

Related reading: from sensor to useful data: the signal chain, measuring an embedded AI demo and int8 quantization for microcontrollers.

Sources

1. Banbury, C., Reddi, V. J., Torelli, P., Holleman, J., et al. (2021). “MLPerf Tiny Benchmark.” Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks. arXiv:2106.07597. https://arxiv.org/abs/2106.07597
2. David, R., Duke, J., Jain, A., Reddi, V. J., et al. (2021). “TensorFlow Lite Micro: Embedded Machine Learning on TinyML Systems.” Proceedings of Machine Learning and Systems (MLSys). arXiv:2010.08678. https://arxiv.org/abs/2010.08678
3. Jacob, B., Kligys, S., Chen, B., et al. (2018). “Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference.” 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2704–2713. https://doi.org/10.1109/CVPR.2018.00286
4. Lai, L., Suda, N., & Chandra, V. (2018). “CMSIS-NN: Efficient Neural Network Kernels for Arm Cortex-M CPUs.” arXiv:1801.06601. https://arxiv.org/abs/1801.06601
5. Zhang, Y., Suda, N., Lai, L., & Chandra, V. (2017). “Hello Edge: Keyword Spotting on Microcontrollers.” arXiv:1711.07128. https://arxiv.org/abs/1711.07128