심층신경망으로 가는 통계 여행, 네 번째 여행: 확률적경사하강법 탐험하기

초록

Stochastic Gradient Descent (SGD) has been fundamental to the success of deep neural networks. This paper provides a comprehensive review of SGD and its extensions. We begin by surveying a wide range of function optimization techniques from numerical analysis and statistics to reveal how modern SGD methods relate to traditional optimization methods. We then explore the characteristics of various SGD-based algorithms to present a systematic comparison of the major SGD optimizers used in deep learning within a unified theoretical framework.

키워드