Join the early list

Quantisation: making models fit

Storing weights as 8-bit integers instead of 32-bit floats makes a model about four times smaller. Quantising the activations too lets integer-only NPUs run it.

Deep dive

Read firstMachine learning on a microcontroller
Quantising a model from 32-bit float to 8-bit integers A 32-bit floating-point weight becomes an 8-bit integer, making the model about 4 times smaller. Quantising the activations too lets the whole model run in integer arithmetic on an integer-only NPU, with unsupported operations falling back to the CPU. Quantisation is done after training or simulated during training. One weight Each square is one bit 32-bit float 8-bit integer about 4× smaller model Full integer quantisation activations quantised too, so the whole model runs in fast integer arithmetic Integer-only NPU such as Arm's Ethos-U CPU Usually a small loss of accuracy Any operation the NPU cannot run falls back to the CPU. Many embedded NPUs only accelerate quantised integer models, so a float model is quantised first. Post-training quantisation converts an already-trained model Quantisation-aware training simulates low precision during training; usually recovers more accuracy
Storing each weight in 8 bits instead of 32 makes a model about 4× smaller, and quantising the activations too lets an integer-only NPU run it.

Key points

  1. Converting 32-bit floating-point weights to 8-bit integers cuts model size by about 4×. Quantising the activations too (full integer quantisation) lets the whole model run in fast integer arithmetic, usually with a small loss of accuracy.1
  2. Post-training quantisation converts an already-trained model; quantisation-aware training simulates low precision during training and usually recovers more accuracy.1
  3. Many embedded NPUs, including Arm's Ethos-U, only accelerate quantised integer models, so a float model has to be quantised first. Any operation the NPU cannot run falls back to the CPU.21

Where to go next

Sources

  1. LiteRT: Post-training quantization · Google · Project docs
  2. Vela: compiler for Arm Ethos-U NPUs (README and supported operators) · Arm · Project docs