https://www.ai-gakkai.or.jp/about/award/jsai_award-conf
◼︎ Bibliographic Information
Shohei TANIGUCHI, Keno HARADA, Gouki MINEGISHI, Yuta OSHIMA, Seong Cheol JEONG, Go NAGAHARA, Tomoshi IIYAMA, Masahiro SUZUKI, Yusuke IWASAWA, Yutaka MATSUO, ADOPT: an Adaptive Gradient Method with the Optimal Convergence Rate with Any Hyperparameters, Proceedings of the Annual Conference of JSAI, 2024, Volume JSAI2024, 38th (2024), Session ID 4D3-GS-2-01, Pages 4D3GS201.
◼︎Overview
Adaptive gradient methods, such as Adam, are widely used for deep learning. However, it is known that they do not converge unless choosing hyperparameters in a problem-dependent manner. There have been many attempts to fix their convergence (e.g., AMSGrad), but they require an impractical assumption that the gradient noise is uniformly bounded. In this paper, we propose a new adaptive gradient method named ADOPT, which achieves the optimal convergence rate of O(1/√T) with any hyperparameter choice without the bounded noise assumption. ADOPT addresses the non-convergence issue of Adam by removing the current gradient from the second moment estimate and changing the order of the momentum calculation and the scaling operation by the second moment estimate. We also conduct intensive numerical experiments, and verify that our ADOPT achieves competitive or even better results compared to Adam and its variants across a wide range of tasks, including image classification, generative modeling, language modeling, and deep reinforcement learning.
