Find Paper, Faster
Example：10.1021/acsami.1c06204 or Chem. Rev., 2007, 107, 2411-2502
Search Direction Correction with Normalized Gradient Makes First-Order Methods Faster
SIAM Journal on Scientific Computing  (IF2.373),  Pub Date : 2021-09-16, DOI: 10.1137/20m1335480
Yifei Wang, Zeyu Jia, Zaiwen Wen

SIAM Journal on Scientific Computing, Volume 43, Issue 5, Page A3184-A3211, January 2021.
The so-called fast inertial relaxation engine is a first-order method for unconstrained smooth optimization problems. It updates the search direction by a linear combination of the past search direction, the current gradient, and the normalized gradient direction. We explore more general combination rules and call this generalized technique the search direction correction (SDC). SDC is extended to composite and stochastic optimization problems as well. Deriving from a second-order ODE, we propose a fast inertial search direction correction (FISC) algorithm as an example of methods with SDC. We prove the $\mathcal{O}(k^{-2})$ convergence rate of FISC for convex optimization problems. Numerical results on sparse optimization, logistic regression, as well as deep learning demonstrate that our proposed methods are quite competitive to other state-of-the-art first-order algorithms.