A landmark, simple and broadly applicable method whose headline test-server deltas all recompute exactly from the paper's own tables, but whose appendix pre-training budget is internally inconsistent (~40 vs ~13 epochs) and whose secondary ablation claims (NSP 'significantly', RND-only 'much worse', LARGE gains 'especially' on small data) outrun single-run evidence with unexplained cross-table inconsistencies.
Soundness
3/4
Presentation
3/4
Contribution
4/4
Confidence
3/5
✓ 10 confirmed✕ 6 discrepancy◐ 2 inconclusive○ 2 not checkable
Summary
The paper introduces BERT, a Transformer encoder pre-trained on BooksCorpus and English Wikipedia (3.3B words) with two self-supervised objectives: a masked language model (15% of WordPiece tokens selected, replaced 80/10/10 by [MASK]/random/unchanged) that lets every layer condition on both left and right context, and a binary next-sentence-prediction (NSP) task. The same pre-trained weights are then fine-tuned end to end with a single task-specific output layer for sentence classification, sentence-pair classification, span extraction and multiple choice (Sec. 3, Fig. 4).
Two sizes are reported (BASE: 12 layers, 768 hidden, 110M parameters; LARGE: 24 layers, 1024 hidden, 340M). The paper claims state-of-the-art results on eleven tasks (eight GLUE tasks, SQuAD v1.1 and v2.0, SWAG; Tables 1-4), with test-server numbers such as GLUE 80.5, MNLI 86.7, SQuAD v1.1 F1 93.2 and SQuAD v2.0 F1 83.1. Ablations (Sec. 5, Tables 5-8, Fig. 5) attribute the gains mainly to deep bidirectionality, report that NSP helps QNLI/MNLI/SQuAD, that larger models help even on small datasets, that BERT features work nearly as well as fine-tuning for NER, and that the 80/10/10 masking recipe is a reasonable default.
Strengths
One architecture and one set of weights for four task families
Sec. 3.2, Fig. 4, Secs. 4.2-4.4
Sec. 3.2 and Fig. 4 show that single-sentence classification, sentence pairs, span extraction (start/end vectors S,E with score S⋅Ti+E⋅, Sec. 4.2) and multiple choice (Sec. 4.4) are all handled by adding one output layer to the same encoder. SQuAD v2.0 is handled by a single null-span score at [CLS] with a tuned threshold (Sec. 4.3). This removes the task-specific architectures that dominated the comparators in Tables 2-4 and is the main reason the work matters.
Weaknesses
Pre-training budget in App. A.2 is internally inconsistent
major · App. A.2 (p. 4183); App. A.4
App. A.2 states 256 sequences x 512 tokens = 128,000 tokens/batch for 1M steps, approximately 40 epochs over 3.3B words, but later says 90% of steps use sequence length 128 and only 10% use 512. Recomputing (check 08): all steps at 512 gives 131.1B tokens = 39.7 epochs (matches ~40; note 256x512 = 131,072 not 128,000), whereas the stated 90/10 schedule gives 42.6B tokens = 12.9 epochs, a factor 3.08 lower, unless the batch size changes between phases (not stated). This propagates to App. A.4, where the 4x batch-size advantage over GPT (128,000 vs 32,000 words) shrinks to about 1.33x under the 90/10 schedule. The compute budget is central to reproducing and to interpreting BERT vs GPT. Fix: state the batch size and number of steps in each phase and the resulting tokens seen.
Unexplained differences between repeated BERT_BASE Dev numbers
major · Tables 2, 5, 6, 7, 8; footnote 7
Five of seven quantities reported more than once for the nominally same BERT_BASE configuration disagree (check 14): MNLI-m 84.4 (Tables 5, 6) vs 84.2 (Table 8, 80/10/10); SST-2 92.7 (Table 5) vs 92.9 (Table 6); NER fine-tune Dev 96.4 (Table 7) vs 95.4 (Table 8); NER concat-last-four 96.1 (Table 7) vs 94.9 (Table 8); SQuAD Dev F1 90.9 (Table 2) vs 91.0 (footnote 7). The NER gaps (1.0 and 1.2 F1) exceed every effect Table 8 is used to interpret (max spread 0.9), so the masking-strategy conclusions cannot be read against the other tables. Fix: state which checkpoint/pre-training budget and which restart-aggregation (best vs mean) each table uses.
Ablation and analysis claims stronger than single-run evidence
major · Sec. 4.1 (p. 4176); Sec. 5.1, Table 5; App. C.2, Table 8
No seeds, variances or tests accompany Tables 5, 6 and 8, yet the text uses strong language. (i) Removing NSP hurts 'significantly' on QNLI, MNLI and SQuAD (Sec. 5.1): QNLI -3.5 is beyond sampling noise (z = 5.4), but MNLI -0.5 (z = 0.96) and SQuAD F1 -0.6 (z = 1.35) are not (check 12, inconclusive). (ii) '+BiLSTM ... hurts performance on the GLUE tasks': deltas are -0.2/-1.8/-0.5 and 0.0, all z < 0.7. (iii) RND-only 'performs much worse' (App. C.2): -0.6/-0.5/-0.3, smaller than the unexplained 1.0-1.2 differences above (check 15). (iv) BERT_LARGE beats BASE 'significantly ... especially those with very little training data' (Sec. 4.1): Spearman rho(train size, gain) = +0.048 (8 tasks), and MRPC (+0.4) and STS-B (+0.7) show the smallest gains (check 16); LARGE also alone received Dev-selected random restarts. Claim (i) directly underpins the decision to include NSP in the method. report mean and std over several fine-tuning seeds and, where feasible, pre-training runs, and soften claims not backed by them.
Questions for the authors
In App. A.2, what batch size was used in the length-128 phase (90% of steps) and the length-512 phase, and how many tokens did BERT see in total? Does the BERT vs GPT batch-size comparison in App. A.4 still hold?
Which pre-training checkpoint and which restart aggregation produced each of Tables 5-8? In particular, why is BERT_BASE NER Dev F1 96.4 in Table 7 but 95.4 in Table 8 for the same 80/10/10 recipe?
What is the seed-to-seed standard deviation of fine-tuning BERT_BASE on MNLI, SQuAD v1.1 and MRPC? Are the No NSP drops of 0.5 (MNLI) and 0.6 (SQuAD F1) larger than it?
Is the GPT GLUE score 72.8 quoted in Sec. 4.1 computed on the same GLUE version and metrics as BERT's 80.5, given that the Table 1 GPT row (e.g. QNLI 87.4) differs from the GPT report (QNLI 88.1)?
Were BERT_BASE results also selected over random restarts on small GLUE tasks? If not, how much of the BERT_LARGE vs BASE gap on CoLA and RTE comes from restart selection?
What is the Test F1 on SQuAD v1.1 without TriviaQA data, and which single leaderboard system is the +1.3 F1 measured against?
Verification
Only the proceedings PDF was supplied, so no number could be reproduced by rerunning models; all checks test internal arithmetic, consistency between sections, and agreement with cited sources. Of 20 checks, 10 are confirmed, 6 found discrepancies, 2 are inconclusive and 2 are not checkable. The headline results hold up: all four abstract deltas, the Table 1 averages (under an undocumented 9-entry convention), the eleven-task SOTA tally, SWAG and NER deltas, parameter counts, masking rates and literature comparator values reproduce. The discrepancies are in the appendix and secondary analyses: the pre-training budget (~40 vs ~12.9 epochs), unexplained differences between repeated BERT_BASE Dev numbers (up to 1.2 F1), and textual claims about significance, robustness and model-size effects that the tabled numbers do not support. None changes a Table 1-4 test-server number.
Incomplete citation: no venue, URL or report number; not resolvable on Crossref or arXiv (work/refs_checked.json). The dataset cited for QQP cannot be located from the bibliography entry alone.
Radford et al. 2018. Improving language understanding with unsupervised learning (OpenAI technical report)
No URL given; not indexed on Crossref/arXiv. Retrieved from the OpenAI CDN: GLUE 72.8, MNLI 82.1/81.4, CoLA 45.4, SST-2 91.3, MRPC 82.3, QQP 70.3 match, but QNLI is 88.1 (vs 87.4 in BERT Table 1) and STS-B is 82.0 Pearson (vs 80.0 Spearman in Table 1). The Table 1 GPT row therefore comes from the GLUE leaderboard while the 72.8 in Sec. 4.1, described as the official leaderboard score, is the GPT paper's own figure (work/verify/18_refs.log).
Peters et al. 2018a. Deep contextualized word representations (NAACL)
Cited for the ELMo row of Table 7. Test F1 92.2 matches (92.22 +/- 0.10), but the Dev F1 95.7 attributed to ELMo does not appear in the cited paper; its source is not given.
Clark et al. 2018. Semi-supervised sequence modeling with cross-view training (EMNLP)
The 92.6 NER Test F1 in Table 7 is the 'CVT + Multi-task + Large' variant of that paper (single-task CVT is 92.3); Table 7 labels it simply 'CVT'. Minor attribution precision issue.
Akbik, Blythe, Vollgraf 2018 (COLING); Brown et al. 1992; Dolan & Brockett 2005; Mnih & Hinton 2009; Turian et al. 2010
Not resolved by the automatic Crossref/arXiv lookup (fuzzy query matched unrelated works). Akbik 2018 was retrieved directly from ACL Anthology C18-1139 and reports 93.09 F1, matching Table 7. The others are not load-bearing for any number and were not spot-checked further; no evidence that they are wrong.
Suggestions
Rewrite App. A.2 with per-phase batch size, sequence length and step count, and the total number of WordPiece tokens seen; update the batch comparison in App. A.4 accordingly.
Add a short table stating, for every Dev result, the pre-training checkpoint and whether it is the best of restarts or a mean, and reconcile the BERT_BASE values that differ across Tables 2, 5, 6, 7, 8 and footnote 7.
Report mean and standard deviation over at least five fine-tuning seeds for Tables 5, 6 and 8, and replace 'significantly' and 'much worse' with the measured effect sizes where the differences are within seed noise.
State in the Table 1 caption that the Average counts MNLI-m and MNLI-mm separately, and report the official GLUE score of each baseline computed under the same GLUE version.
Add the top single leaderboard system to Table 2 and a Test score without TriviaQA to back the statements in Sec. 4.2.
Add an NSP ablation that keeps single long contiguous segments, to separate the NSP objective from the segment-packing effect.
Fix 'four datasets' in Sec. 5.2, the conclusion sentence about 'deep unidirectional architectures', the listed typos, and complete the Chen et al. (2018) and ELMo Dev F1 citations.
Scope of this review
Only the NAACL-HLT 2019 proceedings PDF was supplied (inputs/request.json); there was no LaTeX source, code, checkpoints, pre-training corpus, fine-tuning data or logs. Checks supported by the paper itself (arithmetic, cross-table consistency, parameter counts, masking rates, Figure 5 digitisation) were run as scripts in work/verify/. Claims requiring unavailable experimental artifacts (test-server scores in Tables 1-4, Dev results in Tables 5-8, NSP accuracy 97-98%, one-hour fine-tuning on a Cloud TPU, 4-day pre-training on 16-64 TPU chips) were not reproduced and are marked not_checkable (check 19). All 56 references were looked up on Crossref/arXiv and the cited sources of the comparator numbers in Tables 1, 4 and 7 were retrieved and searched; the released code at github.com/google-research/bert was not examined because it was not part of the supplied material. No text in the paper or requester notes attempted to instruct the reviewer.
Headline numbers are arithmetically sound and largely on blind test servers
Abstract; Tables 1-4; Sec. 4.1 footnote on GLUE submissions
All four abstract deltas recompute exactly from the paper's tables (80.5-72.8=+7.7, 86.7-82.1=+4.6, 93.2-91.7=+1.5, 83.1-78.0=+5.1; check 01). Every BERT entry beats every prior entry in all nine Table 1 columns and the eleven-task SOTA tally holds, with CoNLL NER correctly excluded (check 03). SWAG gains +27.1/+8.3 are exact (check 11). Tables 1-4 are test-server numbers, and the GLUE caption notes one submission per model, which limits test-set overfitting.
Controlled LTR ablation isolates bidirectionality
Sec. 5.1, Table 5; App. C.1, Fig. 5
Table 5 trains a left-to-right, no-NSP model on the same data, size and fine-tuning recipe, and also tries to strengthen it with a BiLSTM. The resulting drops (MRPC -9.0, SQuAD F1 -10.1 vs No NSP) are far beyond sampling noise (z = 3.4 and 19.7 with standard Dev sizes; check 12), so the paper's central claim that deep bidirectional pre-training matters is well supported. Fig. 5 confirms MLM overtakes LTR from about 100k steps on (check 17).
Model-specification details are complete and correct
Sec. 3, footnote 3; Sec. 3.1; App. A.1-A.3
The architecture is specified fully enough to count parameters: 109.1M for BASE and 334.6M for LARGE vs stated 110M/340M (within 1.6%; check 05). The masking generator in Sec. 3.1/App. A.1 is unambiguous and its 1.5% random-replacement figure is exact (check 09). App. A.2-A.3 list optimiser, warm-up, dropout, activation and fine-tuning grids, and literature comparators were found verbatim in the cited works (check 18).
Careful scope on NER and feature-based use
Sec. 5.3, Table 7
Sec. 5.3 reports that BERT does not beat CSE on CoNLL-2003 (92.8 vs 93.1) and describes it as competitive rather than SOTA; the best feature-based variant is 0.3 F1 behind fine-tuning on Dev (96.1 vs 96.4), exactly as claimed (check 13). This analysis makes BERT useful also when fine-tuning is impractical.
Honest discussion of confounds with OpenAI GPT
App. A.4
App. A.4 lists the differences between BERT and GPT beyond the objective (training data, [CLS]/[SEP] at pre-training, batch size, per-task learning rate) instead of attributing the whole gap to bidirectionality, and points to the Sec. 5.1 ablation as the controlled comparison.
Fix:
GLUE averaging and leaderboard comparison under-documented
minor · Table 1 caption; Sec. 4.1; App. B.1
The Table 1 Average reproduces only if MNLI-m and MNLI-mm are counted as two of nine entries (so MNLI carries 2/9 of the weight) and WNLI is dropped; the caption mentions only WNLI (check 02). The gain is called 'accuracy' though QQP/MRPC are F1 and STS-B is Spearman. Adding WNLI = 65.1 explains the 82.1 vs 80.5 gap for BERT_LARGE (80.38, within 0.15) but not GPT's 72.8 (check 04, inconclusive). 72.8 is the GPT paper's own figure, while the Table 1 GPT row uses leaderboard values (QNLI 87.4 vs 88.1 in Radford et al.; STS-B 80.0 Spearman vs 82.0 Pearson; check 18), so the +7.7 headline may compare scores from different GLUE versions. The conclusion that BERT leads by a wide margin is not affected.
SQuAD v1.1 textual claims partly rely on Dev numbers or absent comparators
minor · Sec. 4.2; Table 2; footnote 7
The '+1.3 F1 as a single system' over the top leaderboard single system cannot be checked because that system is not in Table 2 (check 10b). The 'only 0.1-0.4 F1' loss without TriviaQA is 0.2-0.4 in F1; the 0.1 appears only in EM, and it is computed on Dev only, while the text says the model still beats all existing systems, whose numbers are Test (check 10). The single model beats the top ensemble by +0.1 F1 but trails it by 0.9 EM (85.1 vs 86.0). Fix: add the top single leaderboard entry and a Test score without TriviaQA.
Table 6 text and design details
minor · Sec. 5.2; Table 6
The text claims a strict improvement 'across all four datasets', but Table 6 reports three downstream datasets (MNLI-m, MRPC, SST-2); MRPC is said to have 3,600 training examples vs 3.5k in Table 1 (check 07). The monotonicity itself holds. Rows (6,768,3) and (6,768,12) have identical parameter counts (66.6M, check 05), so their 1.3-point MNLI gap is a head-count effect rather than a size effect, and some steps (MRPC +0.2, about one Dev example) are below any plausible seed noise.
Dev-set selection conventions are mixed and not always stated
minor · Sec. 4.1; App. A.3; Tables 5-8
Sec. 4.1 selects learning rate on Dev and, for BERT_LARGE only, the best of several random restarts on Dev; Table 6 averages five restarts; other Dev tables do not say which convention they use. Dev numbers in Tables 2, 5, 7 and 8 are therefore selection-biased by unknown amounts, which matters for the sub-point ablation differences above. Also, Sec. 4.1 lists 4e-5 among candidate learning rates while the App. A.3 grid omits it.
Figure 5 '500k steps' claim and missing error bars
minor · App. C.1, Fig. 5
App. C.1 states almost 1.0% additional MNLI accuracy at 1M vs 500k steps, but no 500k checkpoint is plotted (checkpoints 400k, 600k). Digitising the vector plot gives 83.2 (400k), 84.0 (600k), 84.4 (1M); the interpolated gain is 0.8 (check 17). Acceptable as 'almost', but reporting the actual checkpoint values would be clearer.
NSP's role is asserted rather than analysed
minor · Sec. 3.1, footnote 5; Sec. 5.1, Table 5
Sec. 3.1 motivates NSP by QA/NLI needing sentence relationships, and footnote 5 reports 97-98% NSP accuracy, but the ablation does not separate the NSP objective from the side effect of packing two contiguous segments per example. With only QNLI showing a clear effect (check 12), an ablation with single long contiguous segments and no NSP would make the design choice better grounded.
Typos, a confusing conclusion sentence and citation precision
minor · Sec. 6; Sec. 4.2; Sec. 5.1; App. A, A.4, C.2; References; Table 7
Sec. 6 says the results 'enable even low-resource tasks to benefit from deep unidirectional architectures', which seems to contradict the paper's thesis. Typos: 'befor' (Sec. 4.2), 'langauge' and 'pretraing' (App. A), 'Here we studies' and 'ELMo ,and' (App. A.4), 'this it is strictly less powerful' (Sec. 5.1), 'The right part of the paper' (App. C.2, meaning the table). References: 'Chen et al. 2018, Quora question pairs' has no venue or URL; the Table 7 ELMo Dev F1 95.7 does not appear in Peters et al. (2018a); the CVT 92.6 is the multi-task large variant.
SQuAD v1.1: +1.5 F1 over top leaderboard ensemble; single BERT beats top ensemble in F1; without TriviaQA only 0.1-0.4 F1 lost; footnote 7 Dev F1 91.0
Minor. Confirmed: 93.2 - 91.7 = +1.5 F1 (Test, vs nlnet); single Sgl.+TriviaQA 91.8 > top ensemble 91.7 by +0.1 F1 (but -0.9 EM: 85.1 vs 86.0). The "0.1-0.4 F1" loss without TriviaQA is computable only on Dev: F1 losses are 0.2 (single 91.1 -> 90.9) and 0.4 (ensemble 92.2 -> 91.8); the 0.1 appears only in EM (84.2 -> 84.1). No Test score without TriviaQA is reported, so "still outperforming all existing systems by a wide margin" compares own Dev with others Test. Footnote 7 Dev F1 91.0 vs Table 2 single Dev F1 90.9 (0.1 apart, unexplained).
Recomputed deltas from Table 2
SWAG: BERT_LARGE outperforms ESIM+ELMo by +27.1% and OpenAI GPT by 8.3%
86.3 - 59.2 = +27.1 and 86.3 - 78.0 = +8.3, exact. These are absolute percentage points (relative gains 45.8% and 10.6%); the % sign in the text is ambiguous. BERT_LARGE Test 86.3 also exceeds the expert human figure 85.0 (100 samples).
Recomputed from Table 4 Test column; tolerance 0.05
Table 5: removing NSP hurts significantly on QNLI, MNLI, SQuAD 1.1; LTR & No NSP worse than No NSP on all tasks with large drops on MRPC and SQuAD; BiLSTM improves SQuAD but hurts GLUE
Deltas reproduce the direction of every statement except one. NSP removal: QNLI -3.5 (z=5.4, clearly beyond sampling noise), MNLI -0.5 (z=0.96, p=0.34), SQuAD F1 -0.6 (z=1.35, p=0.18), MRPC -0.2, SST-2 -0.1; so "significantly" is supported for QNLI only on sampling noise grounds, and no seed variance is reported to support it for MNLI/SQuAD. LTR & No NSP < No NSP on all 5 tasks (confirmed), with MRPC -9.0 (z=3.4) and SQuAD -10.1 (z=19.7) the large drops. +BiLSTM: SQuAD +7.1 (z=13.3) confirmed; on GLUE it lowers QNLI/MRPC/SST-2 by 0.2/1.8/0.5 (all z<0.7) and leaves MNLI unchanged (82.1 vs 82.1), so "hurts performance on the GLUE tasks" is not supported beyond noise. Significance cannot be settled without seed/restart variance; single pre-training run per variant.
Feature-based NER: concat last four layers only 0.3 F1 behind fine-tuning; BERT_LARGE competitive with SOTA
Fine-tune BERT_BASE Dev 96.4 - Concat Last Four Dev 96.1 = 0.3, and Concat Last Four is the best feature-based variant. BERT_LARGE Test 92.8 vs ELMo 92.2 (+0.6), CVT 92.6 (+0.2), CSE 93.1 (-0.3): "competitively" is accurate and NER is (correctly) not claimed as SOTA. Feature-based results are Dev-only, so their standing vs SOTA on Test is unknown.
Recomputed from Table 7; tolerance 0.05
Repeated BERT_BASE / BERT_LARGE Dev numbers are consistent across tables
5 of 7 repeated quantities differ without explanation: MNLI-m 84.4 (T5, T6) vs 84.2 (T8 80/10/10); SST-2 92.7 (T5) vs 92.9 (T6); NER fine-tune Dev 96.4 (T7) vs 95.4 (T8); NER concat-last-4 Dev 96.1 (T7) vs 94.9 (T8); SQuAD Dev F1 90.9 (T2) vs 91.0 (fn. 7). MRPC 86.7 and SQuAD BASE 88.5 agree. The NER differences (1.0 and 1.2 F1) exceed every effect in Table 8 (max spread 0.9). Possible explanations (different pre-training checkpoints/budgets for the ablation, best-of vs average of restarts) are not stated.
Tabulated every nominally identical configuration reported more than once
Table 8: fine-tuning robust to masking strategy; MASK-only problematic for feature-based NER; RND-only performs much worse than the 80/10/10 strategy
Robustness confirmed: MNLI spread 0.8 (83.6-84.4), NER fine-tune 0.6. MASK-only feature-based deficit -0.9 (94.0 vs 94.9), the largest in the table, consistent with "problematic". RND-only deficits are -0.6 (MNLI 83.6 vs 84.2), -0.5 (NER fine-tune), -0.3 (NER feature); "much worse" is not supported by these magnitudes, which are below the 1.0-1.2 pt unexplained differences for the same 80/10/10 configuration between Table 7 and Table 8 (check 14). Single runs, no variance; 80/20/0 actually has the best MNLI (84.4).
Deltas vs 80/10/10 per column; compared with unexplained same-configuration spreads from check 14; reviewer threshold for "much worse" = 1.0 pt
BERT_LARGE significantly outperforms BERT_BASE across all tasks, especially those with very little training data
LARGE > BASE on all 9 entries, but gains are small on MRPC (+0.4, 3.5k examples), STS-B (+0.7, 5.7k) and QQP (+0.9). Spearman rho(train size, gain) = +0.048 (8 tasks, exact p for rho <= observed 0.559) and +0.100 (9 entries, p 0.608): no tendency for larger gains on smaller datasets; the pattern rests on CoLA (+8.4) and RTE (+3.7) alone while the two other small tasks show the smallest gains. "Significantly" is not backed by any test; LARGE additionally benefited from Dev-selected random restarts on small datasets (Sec. 4.1), which BASE did not.
Per-task LARGE-BASE Test gains vs training-set size (Table 1 header); Spearman rank correlation with exact permutation p-value
Figure 5: BERT_BASE gains almost 1.0% MNLI accuracy at 1M vs 500k pre-training steps; MLM converges slightly slower than LTR but outperforms it almost immediately
Checkpoints plotted: 30k, 50k, 100k, 200k, 400k, 600k, 800k, 1M; there is no 500k point. MLM: 83.2 (400k), 84.0 (600k), 84.4 (1M); interpolated 500k = 83.6, so the gain is 0.80 (1M minus 600k is only 0.40; 1M minus 400k is 1.20). "Almost 1.0%" is therefore at the generous end. MLM - LTR: -0.8 (30k), -0.2 (50k), +0.2 (100k), +1.2 (200k), then +1.5 to +2.2, so MLM is behind for the first two checkpoints and ahead from 100k on, consistent with the text. MLM at 1M (84.4) equals Table 5 BERT_BASE MNLI-m 84.4. Single runs, no error bars.
Digitised the vector polylines of Figure 5 from the PDF content stream (qpdf), calibrated to the labelled grid lines; legend colours identify the series; 500k interpolated between 400k and 600k; accepted range for "almost 1.0" set to 0.70-1.05
Bibliography resolves; comparator numbers taken from cited works (CSE, ELMo, CVT NER; ESIM+ELMo/GloVe SWAG; OpenAI GPT GLUE) match those works
49/56 references resolve automatically; the 7 unresolved (Akbik 2018 COLING, Brown 1992, Chen 2018, Dolan 2005, Mnih 2009, Radford 2018, Turian 2010) are venue-indexing misses except Chen 2018 "Quora question pairs", which gives no venue or URL. All 8 comparator values found verbatim: CSE 93.09, ELMo 92.22 +/- 0.10, CVT 92.6 (the CVT + Multi-task + Large variant), ESIM+ELMo 59.1/59.2, ESIM+GloVe 51.9/52.7, GPT GLUE 72.8, GPT MNLI 82.1/81.4, GPT CoLA/SST-2/MRPC/QQP 45.4/91.3/82.3/70.3. Notes: GPT QNLI is 88.1 in the GPT paper vs 87.4 in BERT Table 1, and GPT STS-B is 82.0 (Pearson) in the GPT paper vs 80.0 (Spearman) in Table 1, so the GPT row mixes leaderboard values while 72.8 is the GPT paper own score; the ELMo Dev F1 95.7 in Table 7 does not appear in Peters et al. (2018a).
tools/check_refs.py (Crossref + arXiv) summarised by title-word overlap; downloaded cited PDFs (ACL Anthology, OpenAI) and grepped for the quoted values
Fine-tuning results replicable in at most 1 hour on a single Cloud TPU; NSP accuracy 97-98%; test-server scores of Tables 1-4
Only inputs/paper.pdf and request.json were supplied: no code, checkpoints, pre-training corpus, fine-tuning data or logs. Reproducing any experimental number would require the released checkpoints and fine-tuning scripts (github.com/google-research/bert, not supplied), GLUE/SQuAD/SWAG/CoNLL data, TPU or GPU compute, and GLUE/SQuAD server access for Test numbers. Pre-training itself (4 days on 16-64 TPU chips) is out of reach for this review.
Inventory of supplied inputs
Abstract headline deltas: GLUE 80.5 (+7.7), MultiNLI 86.7 (+4.6), SQuAD v1.1 Test F1 93.2 (+1.5), SQuAD v2.0 Test F1 83.1 (+5.1)
All four deltas reproduce exactly (tolerance 0.05): 80.5-72.8=+7.7 (GPT official GLUE score); 86.7-82.1=+4.6 (MNLI-m vs GPT, Table 1); 93.2-91.7=+1.5 (vs #1 Ensemble nlnet, Table 2); 83.1-78.0=+5.1 (vs #1 Single MIR-MRC, Table 3). Comparators are leaderboard entries; vs best published SQuAD 2.0 system (unet 74.9) the gap is +8.2. Underlying test-server scores themselves are not reproducible from the PDF.
Recomputed each delta from comparator values printed in Sec. 4.1 and Tables 1-3
Table 1 Average column (74.0, 71.0, 75.1, 79.6, 82.1) and +4.5 / +7.0 average improvement over prior state of the art
Only the 9-number convention (MNLI-m and MNLI-mm counted as separate entries, WNLI excluded) reproduces all rows: 74.000, 70.944, 75.133, 79.600, 82.078 vs printed 74.0, 71.0, 75.1, 79.6, 82.1 (BiLSTM row off by 0.056, within double-rounding tolerance 0.10). 8-task conventions give e.g. 81.60 / 81.55 for BERT_LARGE and fail. Deltas vs best prior row (GPT 75.1): 79.6-75.1=+4.5, 82.1-75.1=+7.0, matching the text. Reporting point: the caption only says WNLI is excluded; it does not say MNLI is double-weighted (2/9), and the gain is described as accuracy although QQP/MRPC are F1 and STS-B is Spearman.
Recomputed row means under three conventions; tolerance 0.10 to allow for double rounding
BERT_BASE and BERT_LARGE outperform all systems on all Table 1 tasks by a substantial margin; SOTA on eleven tasks
Every BERT entry exceeds every prior entry in all 9 Table 1 columns. Margins over best prior: BERT_BASE min +0.3 (SST-2 93.5 vs 93.2), QQP +0.9; BERT_LARGE min +1.7 (SST-2). Eleven = 8 GLUE tasks + SQuAD v1.1 + SQuAD v2.0 + SWAG, all above the comparators in the paper tables. CoNLL-2003 NER is not SOTA (92.8 vs CSE 93.1, Table 7) and is correctly not counted. Caveat: "substantial margin" is generous for BERT_BASE on SST-2 (+0.3) and QQP (+0.9).
Column-wise comparison of BERT rows vs every non-BERT row of Table 1; tally of SOTA tasks over Tables 1-4 and 7
Official GLUE score 80.5 (BERT_LARGE) vs Table 1 Average 82.1, difference attributed to excluding WNLI (majority class 65.1)
BERT_LARGE: 9-entry + WNLI = 80.38 vs official 80.5 (gap +0.12, within 0.15), so WNLI exclusion explains most of the 82.1 vs 80.5 gap. OpenAI GPT (assuming WNLI 65.1, an assumption): 74.13 (9-entry) or 73.28 (8-task) vs official 72.8 quoted in Sec. 4.1, gaps -1.33 / -0.48 not explained by WNLI. The official score also uses secondary metrics (QQP/MRPC accuracy, STS-B Pearson) and the diagnostic set conventions that are not reported in the paper, so the residual cannot be settled from the PDF. Check 18 shows that 72.8 is the score self-reported in the GPT paper (old GLUE version), whereas the Table 1 GPT row uses leaderboard values (QNLI 87.4 vs 88.1 in the GPT paper), so the 80.5 vs 72.8 comparison may mix GLUE versions; this is a plausible but unverified source of the GPT residual.
Added WNLI=65.1 to Table 1 rows under the 9-entry convention (check 02) and an 8-task convention
BERT_BASE (12/768/12) has 110M parameters, BERT_LARGE (24/1024/16) 340M; BASE same size as OpenAI GPT
BERT_BASE 109.08M (V=30,000) / 109.48M (V=30,522) vs claimed 110M (-0.8%/-0.5%); BERT_LARGE 334.61M / 335.14M vs claimed 340M (-1.6%/-1.4%); tolerance 3%. OpenAI GPT 116.53M, BERT_BASE/GPT = 0.936 (same size to within ~6%). Head count A does not change parameter count, so Table 6 rows (6,768,3) and (6,768,12) have identical size (relevant to check 07).
Exact parameter count of the described encoder (token/position/segment embeddings, L layers, pooler) with vocab 30,000 (paper) and 30,522 (released vocab, external); GPT with 40,478 BPE vocab
Vaswani et al. largest Transformer (L=6,H=1024,A=16) has 100M parameters for the encoder; Al-Rfou et al. (L=64,H=512,A=2) has 235M
Transformer-big encoder: 75.6M without embeddings, 108.3M with a 32k embedding (+8.3%), 113.5M with 37k (+13.5%); sanity check: full model with shared 37k embedding = 214.2M vs Vaswani Table 3 213M. So 100M is a rough figure, matched within 10% only under the 32k-vocab accounting. Al-Rfou T64: 201.8M transformer layers, 218.5M with per-layer positional embeddings vs quoted 235M (-7%); the remainder plausibly comes from auxiliary heads in that paper. Both quotes are order-of-magnitude correct and serve the argument (BERT_LARGE 340M is larger).
Parameter count from stated shapes with external assumptions (FFN 4096 / 2048, BPE vocab 32k-37k, byte vocab, per-layer positional embeddings for Al-Rfou); tolerance 10%
Table 6: larger models give a strict accuracy improvement across all four datasets, even MRPC with only 3,600 training examples; LM perplexity decreases with size
Substantive claim holds: MNLI-m, MRPC, SST-2 all strictly increase and LM ppl strictly decreases down Table 6 (sizes 45.3M, 66.6M, 66.6M, 109.1M, 183.5M, 334.6M). Discrepancies are textual: Table 6 reports three downstream datasets, not four; MRPC is 3,600 examples in Sec. 5.2 vs 3.5k in Table 1. Two caveats for the review: rows (6,768,3) and (6,768,12) have identical parameter counts (66.55M), so their gap (MNLI 80.6 vs 81.9) is a head-count effect, not size; and some steps are tiny (MRPC 86.7 -> 86.9 = 0.2 pts, about 0.8 of the 408 Dev examples, Dev size external) with no variance reported despite averaging 5 restarts.
Parameter count per configuration, strict monotonicity test of each column in size order, count of datasets
Pre-training: 256 sequences x 512 tokens = 128,000 tokens/batch, 1,000,000 steps, approximately 40 epochs over the 3.3B-word corpus; but 90% of steps at sequence length 128 and 10% at 512
256x512 = 131,072 (paper: 128,000, a rounding). All steps at 512: 131.1B tokens = 39.7 epochs, matching ~40. With the stated 90%/10% schedule at 128/512: 42,598 tokens/step on average, 42.6B tokens = 12.9 epochs, a factor 3.08 lower. The two statements in A.2 cannot both hold unless the batch size changes between phases, which is not stated. This also affects App. A.4: BERT 128,000 words/batch vs GPT 32,000 (4x) becomes ~1.33x under the 90/10 schedule. Tokens are WordPiece, corpus is in words, so true epochs would be lower still.
Arithmetic of tokens/step x steps / corpus size under both stated schedules (batch assumed 256 sequences in both phases)
MLM masking: 15% of tokens chosen; 80% [MASK], 10% random, 10% unchanged; random replacement affects 1.5% of all tokens
MC: chosen 0.15001 (SE 0.00011), [MASK] 0.11996, random 0.01503 (SE 0.00004), unchanged 0.01502; all |z| < 0.7. 10% of 15% = 1.5% exactly. The chance that a random replacement equals the original is 1/V (3.3e-5), negligible. Total corrupted input positions = 13.5% of tokens.
Closed form plus fixed-seed Monte Carlo (10M tokens, V=30,000) of the described generator; tolerance 4 SE
SQuAD v1.1: best single system beats the top leaderboard single system by +1.3 F1
Table 2 lists only leaderboard ensembles (nlnet 91.7, QANet 90.5) and published systems; the top leaderboard single system and its F1 (implicitly 90.5 if +1.3 from 91.8) are not given. Would need the Dec 10, 2018 SQuAD leaderboard snapshot. For reference, vs the best published single system (BiDAF+ELMo 85.8) the gain is +6.0.
Searched Table 2 for a top-leaderboard single entry (script 10_squad11.py, part b)