Hardware-Agnostic Models in vLLM – PyTorch

vLLM is introducing hardware-agnostic layers to reconcile two competing pressures: frontier models increasingly need bespoke, hardware-specific 'flat' model definitions that break compatibility with torch.compile, while out-of-tree accelerator plugins (like IBM Spyre), older GPUs, and the transformers backend still depend on compilable, extensible shared layers. The new HW-agnostic layers, built with native PyTorch or portable DSLs like Triton and Helion, aim to be compilable, extensible via CustomOp/PluggableLayer, isolated from hardware-specific paths, and portable across accelerators. Initial support has landed for the transformers backend (enabled via USE_HW_AGNOSTIC=1) and was validated on Spyre with models like Gemma 4, Qwen3, and Granite 4.2. On NVIDIA H100 GPUs, these portable layers achieve throughput within 3.4% of native CUDA-optimized implementations (geometric mean across three recent models), with a full flat-model implementation (e.g., DeepSeek V4) still under review.

Source: https://pytorch.org/blog/hardware-agnostic-models-in-vllm. 8 Sync News only summarizes and links out; content copyright belongs to the authors and original sources.