Grouped-Query and Multi-Query Attention

sharing key/value heads across query groups cuts the KV cache several-fold at close to full multi-head quality

jeremy Video

Multi-query attention and grouped query attention variants in Transformer architecture

In the domain of Transformer architectures within deep learning theory, Multi-query attention (MQA), Grouped query attention (GQA), and standard multi-head attention (MHA) represent variants designed…