ParallelKernelBench: Frontier LLMs can't write fast multi-GPU kernels (yet)

ParallelKernelBench (PKB) is a new benchmark testing whether frontier LLMs can write optimized multi-GPU CUDA kernels across 87 real-world workloads. Results show significant gaps: the best model (zero-shot) solves only 28 of 87 problems correctly, with just 22 beating the PyTorch+NCCL baseline. Models struggle with rank coordination, data partitioning, and choosing optimal GPU-to-GPU transfer mechanisms like TMA and NVLS. An agentic feedback loop with Gemini 3 Pro improved results modestly (35 correct, 26 faster-than-baseline) but plateaued after ~20 refinement steps. Despite poor overall performance, a few generated kernels — for NeMo-RL GRPO, Hyena context parallelism, and SAM 3 mask suppression — outperformed any publicly available implementation. PKB is released as an open benchmark to drive further research into LLM-driven distributed kernel optimization.

Source: https://www.together.ai/blog/parallelkernelbench. 8 Sync News only summarizes and links out; content copyright belongs to the authors and original sources.