A technique for compressing large language model weights to ultra-low precision (2-4 bits) using small per-group non-uniform codebooks learned to match the non-uniform distribution of the weights, paired with Arm CPU kernels that use an interleaved group data layout and amortized weight-unpacking to minimize non-multiply instructions. Students learn how codebook-based (non-uniform) quantization differs from uniform group quantization, and how data layout and instruction-level optimization let sub-4-bit LLMs run efficiently on commodity CPUs while preserving accuracy.
This paper introduces highly optimized compute kernels together with a group-wise non-uniform codebook quantization scheme for running large language model (LLM) inference on commodity Arm CPUs. The …