FlashInfer: A Customizable Block-Sparse Attention Engine for LLM Serving
FlashInfer is a code-generation-based GPU attention engine for LLM inference serving that unifies diverse KV-cache layouts (paged attention, radix trees, sparse masks) into a single block-sparse and composable format with adjustable block sizes for efficient memory access. It exposes a customizable attention template compiled Just-In-Time so new attention variants can be specialized per GPU architecture, and a load-balanced runtime scheduler that adapts to dynamic request lengths while remaining CUDAGraph-compatible. Integrated into SGLang, vLLM, and MLC-Engine, it delivers large inter-token-latency and long-context latency reductions over state-of-the-art serving kernels.
FLASHINFER: EFFICIENT AND CUSTOMIZABLE ATTENTION ENGINE FOR LLM INFERENCE SERVING Zihao Ye * 1 2
FlashInfer is a customizable, high-performance GPU attention engine for large-language-model inference serving, built to handle the workload diversity (prefill, batched decode, prefix reuse, tree/spe…