Summary
The paper proposes Adam, a first-order optimizer that keeps exponential moving averages of the gradient () and of its elementwise square (), divides each by to remove the bias from zero initialisation, and updates (Algorithm 1). It argues that the step is approximately bounded by and invariant to gradient scale (Sec. 2.1), derives the bias correction (Sec. 3), relates Adam to AdaGrad and RMSProp (Sec. 5), and introduces AdaMax, an variant (Sec. 7.1).
Two kinds of support are offered: an regret bound for online convex optimisation (Theorem 4.1, Corollary 4.2, Appendix 10.1), and training-cost curves for logistic regression, MLPs, CNNs and a VAE ablation (Sec. 6, Figs. 1-4).
Our checks confirm the algorithmic core: the bias-correction derivation, the AdaGrad limit, the AdaMax limit, the efficient update form and the scale invariance (at ) are all correct. The convergence result is not. We reproduced an online convex problem that satisfies every hypothesis of Theorem 4.1 and on which average regret converges to 0.6667 instead of 0. We also found four independent errors in the proof. The explicit step-size bounds in Sec. 2.1 and Sec. 7.1 are exceeded by simple sequences. The experiments are single runs with undisclosed grids and no code. The overall score is one point above the rubric's cap of 4 for a main-result discrepancy. This is because the refuted result is the theorem, not the algorithm: the method and its bias correction stand independently of Sec. 4. The regret claim in the abstract and Sec. 4 does not survive.
Strengths
Complete, cheap and directly implementable algorithm
Algorithm 1 and Sec. 2 (p.2)
Algorithm 1 specifies every quantity and default (, , , ). It needs extra memory and elementwise operations only. The efficient ordering in Sec. 2 reproduces Algorithm 1 to once (check 12).
Weaknesses
Theorem 4.1 / Corollary 4.2 are false as stated
major · Sec. 4, Theorem 4.1 and Corollary 4.2 (p.4); Appendix Theorem 10.5 (p.13-15)
We simulated the online problem of Reddi, Kale & Kumar (ICLR 2018): if else on . It satisfies every hypothesis (gradients ≤3, bounded iterates, , ). Adam's average regret is 0.68612, 0.66812 and 0.66692 at , converging to 2/3 rather than 0, and the iterates go to the worst point . At the regret exceeds the Theorem 4.1 right-hand side (check 01). Without projection (convex barrier instead) the value was 0.69052, and bias correction changed only in the 5th significant digit.
Questions for the authors
Given the reproduced counterexample (check 01), which satisfies every hypothesis of Theorem 4.1, which additional assumption or algorithmic change (e.g. a non-decreasing , or using ) do you propose so that an regret bound holds, and does it cover Algorithm 1 as used in Sec. 6?
Verification
We ran 18 checks, 17 of them with rerunnable scripts, focusing on the convergence theorem and the bias correction as the requester asked.
Convergence theory: the conclusion of Theorem 4.1 is refuted by a reproduced counterexample in which (01). Four proof steps fail numerically: the telescoping step (02), Lemma 10.3 (03), the series step and constant of Lemma 10.4's proof (04), and the factor in the constant term (05). The correction is biased under decaying (08), and for the suggested the bound is vacuous at any practical horizon (06).
References
23 references checked
- Kingma & Welling 2013, Auto-Encoding Variational Bayes, 'The 2nd International Conference on Learning Representations (ICLR), 2013'
Year/venue mismatch: the 2nd ICLR took place in 2014; arXiv:1312.6114 was submitted 20 Dec 2013 (verified on arxiv.org). Should read ICLR 2014 (or arXiv 2013).
- Zinkevich 2003, Online convex programming and generalized infinitesimal gradient ascent
Incomplete entry: no venue given. The work is ICML 2003 (found at cdn.aaai.org/ICML/2003/ICML03-120.pdf). Load-bearing for Section 4; content matches the citation (online convex programming framework, regret).
- Tieleman & Hinton 2012, Lecture 6.5 - RMSProp, COURSERA, 'Technical report'
Mis-typed as a technical report: it is an unpublished Coursera lecture slide deck (Lecture 6e, 'rmsprop: Divide the gradient by a running average of its recent magnitude'). Content supports the RMSProp description, but readers cannot consult a formal specification (e.g. no bias correction, no epsilon).
- Ruppert 1988, Efficient estimations from a slowly convergent Robbins-Monro process, Cornell technical report
Could not be located online (Crossref returned unrelated works); existence not verified, though it is a widely cited technical report. Not load-bearing (cited for Polyak-Ruppert averaging in Section 7.2).
- Duchi et al. 2011 (cited on p.4 for O(log d sqrt(T)) vs O(sqrt(dT)))
The reference supports the claim for AdaGrad (their Section 1.2 example gives an O(log d) factor for sparse features), but the paper states that 'their results ... also apply to Adam', which depends on Theorem 4.1; checks 01-05 show that theorem and its proof do not hold as stated, so this transfer is unsupported.
Suggestions
Replace Sec. 4 and Appendix 10.1 with a correct result. Either restrict the claim to a modified algorithm whose effective step is non-increasing, and prove it, or remove the regret claim from the abstract and conclusion.
Scope of this review
The arXiv source contains no LaTeX for the paper body (arxiv.tex only includes a PDF), so equations were read from the PDF text and checked against the page layout. No code repository, data, numeric tables, seeds or hyper-parameter grids are provided, so none of Figures 1-4 could be regenerated. CNN/VAE-scale training was also beyond the CPU-only budget; only a synthetic sparse logistic-regression proxy was run. Figure 2(b) content was not recoverable from the text extraction. The counterexample in check 01 follows Reddi, Kale & Kumar (ICLR 2018) and was re-implemented and run here. Ruppert (1988) could not be located online. No text in the paper or the requester notes attempted to instruct the reviewer.