A systems study benchmarking Intel Gaudi-2 NPUs against NVIDIA A100 GPUs for AI model serving: microbenchmarks of primitive compute, memory-bandwidth, and communication operations plus end-to-end AI workloads, together with a programmability assessment that develops software-level optimizations for key operators (FBGEMM) and the vLLM serving engine on Gaudi. It argues that with sufficient software optimization non-NVIDIA accelerators can rival GPUs for LLM serving, challenging the assumption that the CUDA/NVIDIA stack is strictly necessary.
This computer-systems paper is a comprehensive empirical evaluation of Intel Gaudi NPUs as an alternative to NVIDIA GPUs (the de facto standard) for AI model serving. The authors build a suite of mic…