Browse Teach Curate

Block-wise Mixed-Format FP4 Quantization for LLM Inference

A quantization strategy for large language models in which each fine-grained block of weights or activations is assigned its own optimal low-bit floating-point format from a shared library of format variants, instead of imposing a single format across all blocks. By matching the format to each block's data distribution and keeping the formats compatible with integer arithmetic, the approach handles activation outliers while preserving the energy and memory savings of 4-bit inference.

Data Text

BlockDialect: Block-wise Fine-grained Mixed Format Quantization for Energy-Efficient LLM Inference

BlockDialect is a post-training quantization method that lowers the memory and energy cost of large-language-model inference. Rather than forcing every fine-grained block of weights or activations in…