FP8 Training on AMD GPUs with TorchTitan and TorchAO: Upstreaming Performance Improvements – PyTorch

AMD and Meta/PyTorch engineers upstreamed FP8 training optimizations for AMD Instinct GPUs directly into TorchAO and TorchTitan, giving competitive FP8 performance out of the box without any AMD-specific installs. On dense models, rowwise FP8 delivers a 13.4% throughput gain over BF16 on Llama3-8B. For MoE architectures like DeepSeek-V3 671B, FP8 quantization initially added heavy overhead due to memory-bound kernel launches and HBM round-trips; fused Triton kernels recovered 89% of that overhead, with the colwise scales fix alone delivering a 6.2x speedup per MoE layer. Key work included adding hardware auto-detection for AMD's e4m3fnuz FP8 format (fixing silent numerical corruption from mismatched max values), enabling FP8 grouped GEMM on ROCm via Composable Kernel, and a three-level Triton fusion pipeline reducing kernel launches, improving memory coalescing, and relaxing unnecessary atomic memory fences on AMD hardware. An autotuning search-space expansion was tried and reverted after showing no benefit. Work continues on MXFP8 grouped GEMM for MI355X GPUs.

Source: https://pytorch.org/blog/fp8-training-on-amd-gpus-with-torchtitan-and-torchao-upstreaming-performance-improvements. 8 Sync News only summarizes and links out; content copyright belongs to the authors and original sources.