Join the early list

GPUs, DSPs, ISPs, NPUs and what TOPS means

Accelerators do one kind of work far more efficiently than a CPU. TOPS is the headline number, and it is hard to compare between chips.

Core

Read firstMicrocontroller or microprocessor?
What GPUs, DSPs, ISPs and NPUs accelerate, what TOPS means, and how unsupported layers fall back to the CPU A GPU accelerates graphics and parallel arithmetic, a DSP signal-processing loops, an ISP raw camera data, and an NPU neural-network operations. TOPS is tera operations per second, usually quoted for INT8 and sometimes assuming sparse weights, so it is not directly comparable between vendors. A model layer the NPU's toolchain does not support falls back to the CPU, which can dominate the total time. What each one accelerates GPU graphics and parallel arithmetic DSP signal-processing loops, such as audio filters ISP raw camera-sensor data into usable images NPU neural-network operations, usually at reduced precision TOPS tera (1012) operations per second Usually quoted for INT8 arithmetic Some assume sparse (partly zero) weights Not directly comparable between vendors One model, layer by layer Not to scale Layer 1 Layer 2 Layer 3: unsupported operator Layer 4 Layer 5 NPU NPU falls back to the CPU NPU NPU time A model only runs at NPU speed if every layer is supported by the NPU's toolchain. The CPU fallback can dominate the total inference time.
Each accelerator speeds up one kind of work; TOPS depends on precision and sparsity, and a layer the NPU cannot run falls back to the CPU, which can dominate the time.

Key points

  1. A GPU accelerates graphics and parallel arithmetic; a DSP accelerates signal-processing loops such as audio filters; an ISP turns raw camera-sensor data into usable images; an NPU accelerates neural-network operations, usually at reduced precision.123
  2. TOPS means tera (10^12) operations per second. Vendors usually quote it for INT8 arithmetic, and some quote figures that assume sparse (partly zero) weights, so TOPS from different vendors are not directly comparable.45
  3. A model only runs at NPU speed if every layer is supported by the NPU's toolchain; unsupported operators fall back to the CPU, which can dominate the total inference time.367

Where to go next

Sources

  1. i.MX 8M Plus · NXP · Vendor page
  2. Ethos-U55 microNPU · Arm · Vendor page
  3. Vela: compiler for Arm Ethos-U NPUs (README and supported operators) · Arm · Project docs
  4. Jetson Orin Nano Super Developer Kit · NVIDIA · Vendor page
  5. NVIDIA Jetson Orin Nano Developer Kit Gets a 'Super' Boost · NVIDIA · Vendor page
  6. GPU delegates for LiteRT · Google · Project docs
  7. LiteRT Delegates · Google · Project docs