Low Precision Flash Attention 4: End-to-End Block-Scaled Attention for Blackwell – PyTorch

Meta engineers extend FlashAttention-4 with end-to-end MXFP8 (microscaling FP8) support for both forward and backward passes on NVIDIA Blackwell GPUs, targeting production Ads recommendation model (GEM) training. The work covers TMEM allocation strategies to fit scale factors alongside MMA accumulators, online transpose-invariant square block-scale quantization for dS using Blackwell's redux.sync.max.abs.f32 instruction, fused RMSNorm+Quantize and GEMM+Quantize kernels that eliminate separate quantization passes, and a zero-gather jagged module that keeps FP8 data compact while only scale factors get padded and permuted for TMA. Results show up to 1.6x forward and 1.52x backward speedups over BF16 on internal shapes, reaching 2.85 PF/s forward and up to 1.98 PF/s backward on LLM shapes, competitive with cuDNN 9.24. The code is open-sourced in Meta's Ads Model Kernel Library on GitHub.

Source: https://pytorch.org/blog/low-precision-flash-attention-4-end-to-end-block-scaled-attention-for-blackwell. 8 Sync News only summarizes and links out; content copyright belongs to the authors and original sources.