Don’t Just “Throw Adam at It”: Misunderstanding Adam Will Cost You

Blindly using Adam optimizer with default hyperparameters (β₁=0.9, β₂=0.999, ε=1e-8) is a common but costly mistake. The author shares a personal story of burning thousands of GPU hours on architectural changes before discovering the real culprit was misconfigured Adam betas. The post explains how Adam's first and second moment estimates work mathematically, documents proven failure modes (convergence to suboptimal solutions, loss spikes in large models, instability in reinforcement learning), and provides practical guidance on when and how to modify defaults — such as reducing β₂ to 0.95, increasing ε dramatically in RL settings, and using gradient clipping. A PyTorch gotcha around differing weight_decay defaults between Adam and AdamW is also highlighted.

Nguồn: https://towardsdatascience.com/dont-just-throw-adam-at-it-misunderstanding-adam-will-cost-you. 8sync News chỉ tóm tắt và dẫn link; bản quyền nội dung thuộc tác giả và nguồn gốc.