How a paper is reviewed
A structured review in the OpenReview style: a summary of the paper's claims, strengths and weaknesses with their locations, questions for the authors, concrete suggestions, and scores for soundness, presentation, contribution, and overall — plus the reviewer's confidence.
- Key claims picked out, each verified by a small script that actually runs
- Tables recomputed, formulas re-derived, p-values recalculated, public code rerun
- Every result links to its script — anyone can rerun it
arXiv:1412.6980 · Kingma & Ba
2.1 Adam's update rule
An important property of Adam's update rule is its careful choice of stepsizes. Assuming ε = 0, the effective step taken in parameter space at timestep t is Δt = α · m̂t / √v̂t. The effective stepsize has two upper bounds: |Δt| ≤ α · (1 − β1) / √(1 − β2)1 in the case (1 − β1) > √(1 − β2), and |Δt| ≤ α otherwise.
3 Initialization bias correction
Taking expectations of the exponential moving average vt relates it to the true second moment:
𝔼[vt] = 𝔼[gt2] · (1 − β2t) + ζ2(4)
4 Convergence analysis
Theorem 4.1. Assume that the function ft has bounded gradients and the distance between any θt generated by Adam is bounded. Let γ = β12/√β2 < 1. Then Adam achieves R(T)/T → 03 for all T ≥ 1.
