Summary
The paper proposes Adam (Algorithm 1, p.2), a first-order stochastic optimiser that keeps exponential moving averages of the gradient () and of its element-wise square (), corrects both for their zero initialisation by dividing by (Sec. 3), and updates each coordinate by . It argues that the per-coordinate step is approximately bounded by (Sec. 2.1), that the update is invariant to diagonal rescaling of the gradients, and that Adam contains AdaGrad and RMSProp-with-momentum as limiting/special cases (Sec. 5). AdaMax, an -norm variant, is derived in Sec. 7.
Theoretically, Theorem 4.1 / Corollary 4.2 (p.4, proof in Appendix 10.1, p.12-15) claim an regret bound in the online convex optimisation framework, under bounded gradients and iterates, , decaying and . Empirically (Sec. 6, Figs. 1-4) Adam is compared with SGD-Nesterov, AdaGrad, RMSProp, AdaDelta and SFO on logistic regression (MNIST, IMDB bag-of-words), MLPs, a CIFAR-10 CNN and a VAE, including an ablation of the bias-correction term.
This review covers arXiv:1412.6980v9 (30 Jan 2017, ICLR 2015 camera-ready); page numbers refer to that PDF.
Strengths
Fully specified, cheap algorithm with usable defaults
arXiv:1412.6980v9, Algorithm 1, p.2
Algorithm 1 gives every operation and default hyper-parameters (, , , ); memory is two vectors per parameter. Every check script in this review implemented it in a few lines directly from the pseudo-code, without ambiguity (apart from the issue of the reformulation on p.2, see weaknesses).
Weaknesses
Theorem 4.1 / Corollary 4.2 are false as stated (numerical counterexample)
major · arXiv:1412.6980v9, Sec. 4, Theorem 4.1 and Corollary 4.2, p.4; Appendix 10.1, Theorem 10.5, p.13-15
Version/page: arXiv:1412.6980v9, p.4 (Thm 4.1, Cor 4.2), p.13-15 (Thm 10.5). Basis (numerical counterexample, check 11): on the 1-D online linear losses if else (after Reddi, Kale & Kumar, ICLR 2018), with all hypotheses satisfied (, , or , ), the iterate converges to instead of and at . The regret exceeds the theorem's right-hand side from (6680.9 vs 6117.5; ratio 10.99 at ), also with (ratio 3.86) and without projection (ratio 2.44). Corollary 4.2's therefore does not hold. The abstract's "regret bound ... comparable to the best known results" is not supported. Fix: either add the AMSGrad-type condition (non-decreasing , e.g. use ) and re-prove, or withdraw the theorem and state that no convergence guarantee is provided for Algorithm 1.
Questions for the authors
Given the counterexample of check 11 (regret 10.99x above the Thm 4.1 bound at with all stated hypotheses satisfied), which additional assumption do you intend the theorem to carry (e.g. non-decreasing ), and can you show it holds for Algorithm 1 or replace Algorithm 1 by a variant that enforces it?
Verification
18 checks were run on arXiv:1412.6980v9 with minimal CPU scripts (work/verify/), each labelled by kind of evidence. Mathematically confirmed: the bias-correction identities of eq. (1)-(4) (01), the AdaGrad limit (08) and the AdaMax limit (04); confirmed by finite-sample tests: stationary unbiasedness (02), exact rescaling invariance for (07), and large uncorrected initial steps (03, whose divergence part rests only on Fig. 4). Composite claims were split rather than marked confirmed as a whole (01 has an unconfirmed caveat; 02b is inconclusive). Discrepancies: the convergence theorem fails on an explicit counterexample (11), with the failure traced to a non-monotone telescoping step (12), false Lemmas 10.3 and 10.4 (09, 10) and inconsistent constants (14); the bound is vacuous for (13); the Sec. 2.1 step bounds are exceeded (15: 7.27 vs 3.16 for defaults); AdaMax's is exceeded by 0.9% for defaults (05); the reformulation is not equivalent (06). The experiments (16) could not be reproduced; references (17) were looked up and showed only minor issues.
References
23 references checked
- Kingma, D. P. and Welling, M. Auto-Encoding Variational Bayes. In The 2nd International Conference on Learning Representations (ICLR), 2013.
Year/venue mismatch: arXiv:1312.6114 was first submitted 2013-12-20 (work/verify/17_references.log) and the 2nd ICLR took place in 2014; cite as ICLR 2014 (or arXiv 2013). Minor.
- Zinkevich, Martin. Online convex programming and generalized infinitesimal gradient ascent. 2003.
Incomplete entry: no venue. The work is in ICML 2003 (AAAI Press), pp. 928-935/936 (ACM DL 10.5555/3041838.3041955; DBLP conf/icml/Zinkevich03). Load-bearing: it defines the regret framework of Sec. 4. Crossref query did not resolve it (proceedings not indexed). Minor.
- Tieleman, T. and Hinton, G. Lecture 6.5 - RMSProp, COURSERA: Neural Networks for Machine Learning. Technical report, 2012.
Mislabelled source type: this is a slide from an online course lecture, not a technical report; it is the only cited source for RMSProp, which Adam is compared with and derived from (Sec. 5). Label as lecture slides. Minor.
- Pascanu, R. and Bengio, Y. Revisiting natural gradient for deep networks. arXiv:1301.3584, 2013.
Cited (Sec. 5, p.5) for 'vhat_t is an approximation to the diagonal of the Fisher information matrix'. Spot-read of the abstract (arXiv v7, 17 Feb 2014): it relates natural gradient to Hessian-Free, KSD and TONGA and mentions diagonal approximations of the metric only as an alternative to its truncated-Newton inversion; it does not discuss running averages of squared minibatch gradients. The cited claim is at best indirect support (vhat is an uncentred second moment of minibatch gradients, i.e. an 'empirical Fisher'-type diagonal, not the Fisher). Full text not read; flagged as weak support, not as an error.
- Duchi, J., Hazan, E., and Singer, Y. Adaptive subgradient methods for online learning and stochastic optimization. JMLR 12:2121-2159, 2011.
Bibliographic data correct (JMLR v12 page checked). Sec. 4 (p.4) states 'Their results for the expected value E[sum_i ||g_1:T,i||_2] also apply to Adam' and that Adam can achieve O(log d sqrt T); Duchi et al. analyse AdaGrad only, so the transfer to Adam rests on Theorem 4.1, which is not valid as stated (checks 09-12, 14). Citation supports AdaGrad, not the Adam claim.
Suggestions
Replace Theorem 4.1/Corollary 4.2 by a correct statement: either analyse a variant with (or assume non-decreasing ) or remove the regret claim from the abstract (p.1) and Sec. 4 (p.4).
Ethics and integrity
No ethical or integrity concerns. Only public benchmark datasets are used. The paper, its source wrapper and the requester notes contain no text attempting to steer the review.
Scope of this review
Reviewed arXiv:1412.6980v9 (30 Jan 2017); the arXiv LaTeX source only wraps a PDF, so equations were read from the PDF text. No code, trained models, hyper-parameter grids or seeds were released and no GPU was available, so none of Figs. 1-4 (MNIST/IMDB logistic regression, MLP, CIFAR-10 CNN, VAE, SFO comparison) was reproduced; only a small synthetic CPU illustration was run (check 16), and no large-scale training was attempted. The claim inherited from Duchi et al. was not checked independently. All theoretical checks are minimal CPU scripts (sympy/mpmath/numpy/scipy) in work/verify/ with logs. The requester asked for a Chinese-language review; per the platform rule the review is written in the language of the paper (English). The other requests (version, page, basis and fix per issue; labelling evidence as proof, counterexample, finite-sample test or not checkable; no blanket confirmation of composite claims) were followed. On scoring: the confirmed discrepancy in the convergence theorem would normally cap the overall score at 4; it is set at 5 because the paper's primary contribution, Algorithm 1 with bias correction, is independent of Theorem 4.1 and its components checked out (01, 02, 04, 07, 08). The theorem itself does not survive and must be corrected or withdrawn.