Summary
The paper asks whether the gap in publication outcomes between authors affiliated in non-English-speaking countries and authors in six core English-speaking countries changed after ChatGPT's release (30 Nov 2022). From a 1% hash sample of OpenAlex author identifiers (2026-10-14 snapshot) it builds a panel of 74,627 authors (56,883 treated, 17,744 comparison) for 2017–2025 and estimates a two-way fixed-effects difference-in-differences (author FE, field-by-year FE; PPML for output) comparing 2024–2025 with 2017–2022 on four primary outcomes: output, top-quartile journal share, log days from receipt to acceptance (PubMed), and international collaboration. Inference uses a wild cluster restricted bootstrap over 172 country clusters (only 6 in the comparison group, 68% of comparison authors in the US), with Holm correction across outcomes.
The pooled estimates are output +0.2033 (22.5%), top-quartile share +2.8 pp, log days −0.039 and collaboration −0.015; none survives Holm, and top-quartile share and log days fail the pre-registered pre-trend rule. The event study shows convergence before 2022, and a clearly labelled post hoc linear-trend model is reported. The paper concludes that the data do not show that ChatGPT levelled the playing field in its first two years.
The analysis plan was frozen with a self-held SHA-256 lock (2026-11-09T20:58:04Z), not lodged with a registry. The paper was produced end to end by an automated research system. Neither the code package, the lock file nor the frozen plan was supplied for this review.
Strengths
Complete, honest reporting of pre-specified estimates
Sec. 3.9, Table 2, Table 3, Table 10, Figure 2
Every primary, secondary and breakdown estimate is reported, nulls included, with both bootstrap and CR1 inference side by side (Table 3). The deviation log (Table 2) lists eight deviations together with their consequences. The post hoc trend model is labelled as such in the Status column of Table 10 and as a dashed line in Figure 2. Readers can therefore see exactly where conclusions depend on choices made after the plan was frozen.
Tables are arithmetically clean
Tables 3–10; Figure 1; Secs. 4.2–4.8
Our recomputations found no transcription errors. The checks covered the Holm adjustments (check 01), the BH q-values for all 16 outcome × breakdown families (check 07), CR1 CI/p consistency for every row with a p-value, CI midpoints for 95 Table 9 cells, and bootstrap p/CI agreement for 24 rows (check 05). They also covered relative-magnitude bands for all 16 endpoints (check 04), the trend extrapolation (check 03), the pre-trend rule classification (check 06) and 41 text-versus-table numbers (check 15). The pre-period means read off Figure 1 match Table 3 (check 11).
Serious treatment of few-cluster inference
Sec. 2.4, Sec. 3.6, Table 3
The paper recognises that six comparison clusters, with 68% of comparison authors in one country, make conventional clustered SEs unreliable. It simulates the pipeline before touching real data and uses a WCR score bootstrap with Webb weights and test-inversion CIs. The contrast between CR1 p = 0.011 and bootstrap p = 0.198 for output (Table 3) shows readers how much the inference choice matters.
Pre-trends handled beyond a single joint test
Sec. 3.7, Sec. 4.3–4.4, Tables 4–5, Figure 2
The paper combines a joint lead test, two fake-date placebos and an explicit pass/fail rule with relative-magnitude bands, and it states that non-rejection is weak evidence. The documented pre-period convergence (Table 4, Figure 2) is a substantive finding in its own right, and it warns against reading the raw post-2022 gap as an LLM effect.
Detector-free, intent-to-treat design and candid limitations
Sec. 1, Sec. 2.2–2.3, Sec. 5.3
Defining exposure by affiliation country and release date avoids the selection and misclassification problems of adopter designs that rely on LLM detectors (Secs. 1, 2.2). Sec. 2.3 states explicitly that the contrast 'estimates a change in group gaps, not the effect of any individual's LLM use'. Sec. 5.3 lists the main threats: the proxy, six clusters, the short post-period and concurrent shocks.
Data provenance details check out where testable
Sec. 3.2, Table 1, Table 2 no. 1
The 2026-10-14 OpenAlex release exists, and its works manifest lists exactly 1,898 Parquet files (464,765,485 records, 744 GB), as stated in deviation no. 1 (check 12). The sample-flow arithmetic in Table 1 closes exactly: 86,612 − 3,637 = 82,975 = 56,883 + 17,744 + 8,348 (check 08).
Weaknesses
Abstract, conclusion and plain-language summary misdescribe the log-days and collaboration results
major · Abstract; Sec. 5.1; Sec. 6; Plain-language summary; Tables 3, 4, 10
The abstract says log days to acceptance 'fell by 0.039' and that 'a post hoc trend adjustment leaves little change'. The plain-language summary says papers 'were accepted slightly faster' and 'little change remains'. Sec. 6 says time to acceptance 'shortened'. The paper's own tables say otherwise. The 2024/2025 event-study coefficients relative to 2022 are +0.026/+0.030, so the gap in acceptance times widened after the last pre-year, and the negative pooled estimate comes from pre-period coefficients averaging +0.078 (check 02). Observed log days lie +0.048/+0.074 above the linear pre-trend (check 03). The trend-adjusted estimate is +0.057, CR1 CI [0.023, 0.090], bootstrap p = 0.062: opposite in sign and 1.48× the pooled magnitude. Collaboration likewise flips from −0.015 to +0.012 (CR1 CI [0.002, 0.022]). 'Leaves little change' is true only for output (0.002) and top-quartile share (−0.008). Fix: report the sign reversals in the abstract and Sec. 5.1/6, and drop 'accepted slightly faster'.
'Output rose by 22.5%' is a decline in the comparison group
major · Abstract; Sec. 4.1–4.2; Figure 1a; Plain-language summary
Digitising Figure 1a (check 11) reproduces the raw log-ratio DiD (+0.207 vs PPML 0.2033) and shows its source. Non-English output was essentially flat (1.928 → 1.909, −1.0%), while English-country output fell by 19.5% (2.034 → 1.638, from a 2020 peak of 2.20). The abstract's 'Output rose by 22.5% relative to English-country authors' and the plain-language 'published somewhat more' therefore suggest a gain for the treated group that the data do not show. More importantly, a ~20% fall in the comparison group needs its own explanation before it can be read as relative gains abroad. Candidates include differential attrition or retirement, indexing lag for 2025 works, the COVID-era peak, and OpenAlex author-profile splits and merges. The paper does not examine any of these. Fix: report group-specific trajectories in the text and analyse attrition and indexing completeness by group.
Headline output null is uninformative: realised sensitivity far below the power analysis
major · Sec. 3.8 (power); Sec. 5.3; Table 3; Abstract
The pre-registered MDE for log output was 0.011–0.054, with 'realistic sensitivity between the two' (Sec. 3.8). The realised bootstrap CI for output [−0.162, 0.436] implies an MDE of 0.427, about 53% on the count scale and 7.9× the upper planned bound; even the CR1 MDE (0.221) is 4.1× (check 10). The international-share MDE is 1.5× its upper bound. The non-rejection for output therefore cannot distinguish 'no levelling' from effects as large as those reported by adopter studies (36–72%). The abstract's closing caveat that 'small group effects cannot be excluded' understates this. Fix: report realised MDEs or equivalence bounds next to each primary estimate, and frame the output result as inconclusive rather than null.
Simulation evidence for the primary inference method is internally inconsistent
major · Sec. 3.6; Sec. 2.4
Sec. 3.6 reports that the bootstrap rejected 7.1% of 336 true-null tests (24/336; Clopper–Pearson 95% CI 4.6–10.4%). It also calls the bootstrap 'conservative, with a smallest null p-value of 0.065'. A 7.1% rejection rate at the 5% level is over-rejection, not conservativeness. And 24 rejections require 24 null p-values below 0.05, which contradicts a smallest null p of 0.065 unless the two statements refer to different test sets, which the paper does not say (check 09). The simulations also used 38 countries rather than the real design (172 clusters, 6 control, 68% in the US). MacKinnon & Webb (2018) show that the WCR bootstrap can be extremely conservative when control clusters are this few and unbalanced. Fix: clarify the two statistics, rerun the size and power simulation on the realised cluster structure, and cite the relevant few-treated/few-control literature.
Pre-registration cannot be verified, and 'Pre-Registered' in the title overstates it
major · Title; Sec. 3.1; Table 2; Data and code availability
The plan was locked by a self-held SHA-256 hash at 2026-11-09T20:58:04Z, not lodged with a registry (Sec. 3.1). A 186-author pilot was inspected first. The lock file, frozen plan, code package and deviation log are said to 'accompany this article' but were not supplied (Data and code availability). The PDF was created 2026-11-10 00:11:35 UTC, 3.23 h after the lock (check 12). In that window the pipeline would have scanned 744 GB of works (≥64 MB/s sustained), retrieved 595,873 PubMed histories (≥51 records/s), run all 9,999-rep bootstraps and written 21 pages. This is feasible for an automated system, but it means that adherence to the plan, including whether the pre-trend rule, the RM-band convention (check 04 shows the bands reproduce only when the 2021→2022 step is included) and the eligibility rules were pre-specified, cannot be assessed by anyone outside the system. Fix: deposit the plan and lock with a public timestamping service or registry, release the code and logs, and qualify 'pre-registered' in the title and abstract (e.g. 'with a hash-locked analysis plan').
Causal attribution to ChatGPT in the title and framing
major · Title; Abstract; Sec. 3.3; Sec. 5.2
The estimand is the change in a country-group gap around a calendar date, and the paper says so itself (Sec. 2.3, 4.4: 'not the causal effect of the release'). The title 'ChatGPT and the English-Language Gap', the abstract's 'The data do not show that ChatGPT levelled the playing field', and the plain-language framing 'After ChatGPT…' nonetheless attribute the contrast to ChatGPT. The release coincides with the end of China's zero-COVID policy (China is 22% of treated authors and drives the output estimate, Sec. 5.2, Table 9), post-pandemic shifts in journal output, and research-evaluation reforms. Fix: retitle around the 2022 break in group gaps, and state in the abstract that the design cannot separate ChatGPT from concurrent shocks.
Identification threats specific to an OpenAlex author-ID panel are not addressed
major · Sec. 3.2; Table 1; Sec. 5.3
The OpenAlex release notes for 2025–2026 document large author-profile clean-ups, e.g. '2.6M splinter profiles merged… affecting ~5M works' (check 12). Disambiguation quality is known to differ by name origin, and East Asian names matter here because China is the largest treated country. Revisions of this kind can create group-specific output trends in an author-ID panel. The paper also does not discuss indexing completeness of 2025 works in an October 2026 snapshot by group. The eligibility rule (≥1 article in 2020–2022) conditions on recent activity and mechanically shapes the lead profile. 'Movers' are defined from post-period affiliations, a small post-treatment selection (1,998 authors). The sampling frame is also unclear: 966,412 sampled IDs is 0.956% of the 101.1M-record authors table, 44 SE below a 1% hash sample, which fits only if the frame is the author IDs occurring in works (check 13, inconclusive). Fix: add these to Sec. 5.3, define the frame, and show robustness to excluding 2025 and to Chinese-name authors.
Robustness section overstates stability
minor · Sec. 4.7; Table 9
Sec. 4.7 says 'the remaining checks leave all four estimates close to their primary values'. In Table 9, leaving out Europe & Central Asia moves output to 0.294 (+45%), top-quartile share to 0.035 (+26%), log days to −0.054 (−40%) and collaboration to −0.025 (−67%). Leaving out GB moves log days to −0.025 and collaboration to −0.009 (check 14). Signs are preserved, but the heterogeneity is substantial and should be described as such.
Output 'passes' the pre-trend rule only because the test has low power
minor · Sec. 4.3; Table 3 (last row); Tables 4–5
The output leads rise monotonically (−0.217 → −0.077), the CR1 joint Wald p is < 0.001, and the fake-2021 placebo estimate (0.138) is 68% of the primary estimate (check 06). Under the same rule applied to CR1 p-values, all four outcomes fail. Sec. 4.3 says that non-rejection is weak evidence, but Table 3's 'Passes' label invites over-reading. Similarly, Sec. 4.3 says 'collaboration leads are close to zero', although the 2017 lead is 0.028 (SE 0.014), 1.9× the primary estimate (check 15).
Small reporting inconsistencies
minor · Tables 2, 7, 8, 10; Figure 4; Sec. 2.3, 4.8
(a) The four Table 7 domain sizes sum to 56,875, 8 short of 56,883, although Table 8 fields sum to 56,883 (check 08). (b) Table 2 no. 2 calls the 13-author 'Unclassified' region too small to estimate, yet Table 7 reports it, and it enlarges the BH family to m = 8 (check 07). (c) The imputation estimate in Table 10 is reported without SE or CI, and Sec. 4.8 calls the OLS output change 'imprecise' although its CR1 p is 0.023. (d) Figure 4 omits the 'Unknown' sex group that Sec. 3.8 says is always reported. (e) The '74 records in our review' (Sec. 2.3) refers to a literature review whose search strategy is not described.
Citation problems
minor · Secs. 1, 2.2, 5.2; References [10], [11], [13], [18], [60]
[13] is cited twice for the failure of detectors on polished LLM text, but its abstract does not address detection. [11]'s 72% is the 2025-specific estimate of arXiv v2 (52–88% across specifications) and is given without that qualifier. [18] merges two author names. [60] is a blog post without a locator. The published critique of [10] (Renault, Bergeaud & Bosquet, PNAS 2026), which argues that adopter designs mechanically inflate gains, bears directly on Sec. 5.2 and is not cited (check 16).
Questions for the authors
Do the 7.1% bootstrap rejection rate and the 'smallest null p-value of 0.065' in Sec. 3.6 refer to the same 336 true-null tests? If so, how can both hold?
What explains the ~19.5% fall in mean output of English-country authors from 2022 to 2024–2025 (Figure 1a)? How much of it is author attrition, incomplete indexing of 2025 works, or OpenAlex profile splits and merges, and does the output DiD survive excluding 2025?
Can you release the lock file, frozen plan and code with an external timestamp, so that readers can confirm the pre-trend rule, the RM-band convention (including the 2021→2022 step) and the eligibility rules were fixed before the data scan?
Given that the trend-adjusted log-days estimate is +0.057 (CR1 CI [0.023, 0.090]), why do the abstract and conclusion describe time to acceptance as having shortened, with 'little change' after adjustment?
Was the 1% hash sample drawn from the OpenAlex authors table or from author IDs appearing in works authorships? 966,412 IDs is 0.956% of the 101.1M authors table.
What are the size and power of the WCR bootstrap when simulated on the realised structure (6 control clusters, 68% of comparison authors in one cluster) rather than on 38 countries?
Verification
Only the PDF was available: no code, data, lock file or plan, so no estimate could be re-estimated from microdata. We ran 16 scripted checks. Seven confirmed internal consistency: headline transformation and Holm (01), trend extrapolation (03), RM bands (04), CR1 and bootstrap CI/p consistency across Tables 3–10 (05), pre-trend rule classification (06), BH q-values (07) and 41 text-versus-table numbers (15). Two more confirmed facts with interpretive consequences: digitised Figure 1 reproduces Table 3's pre-period means and shows the output DiD comes from a fall in the comparison group (11), and the OpenAlex 2026-10-14 snapshot has exactly 1,898 works files (12). Six found discrepancies. The abstract and conclusion misdescribe the log-days and collaboration results under the paper's own trend adjustment (02). Table 7 domain counts are 8 authors short (08). The simulation size statements are mutually inconsistent (09). The realised MDE for output is 7.9× the pre-registered upper bound (10). The robustness text overstates stability (14). Several reference issues were found (16). The sampling-frame check (13) was inconclusive. None of the discrepancies is a numerical error in a reported estimate; they concern how the results are described and how informative they are.
Output DiD 0.2033 = 22.5%; Holm-adjusted p 0.396/0.140/0.146/0.403 from bootstrap p 0.198/0.035/0.049/0.403; abstract numbers match Table 3ConfirmedAbstract; Sec. 4.2; Table 3
22.5% claimed vs 22.544% obtained (tol 0.05 pp). Holm: 0.396 vs 0.3960, 0.140 vs 0.1400, 0.146 vs 0.1470 (range 0.1455-0.1485 given rounding of p=0.049; consistent), 0.403 vs 0.4030. Abstract 2.8 pp = 0.0277, -0.039 = -0.0385 rounded.
script: 01_headline_holm.pyRecomputed 100(exp(b)-1) and Holm step-down from the reported bootstrap p-values, with interval arithmetic for 3-dp rounding
Abstract/conclusion: log days to acceptance 'fell by 0.039' / 'shortened' and 'a post hoc trend adjustment leaves little change'DiscrepancyAbstract; Sec. 5.1; Sec. 6; Plain-language summary; Tables 3, 4, 10
Approximate pooled DiD from Table 4 reproduces signs (output 0.216 vs 0.2033; log days -0.050 vs -0.0385), so the negative log-days estimate comes from pre-period coefficients averaging +0.078, while 2024/2025 coefficients relative to 2022 are +0.026/+0.030 (mean +0.028), i.e. the acceptance-time gap widened relative to the last pre-year. Trend-adjusted log days is +0.0569, CR1 CI [0.023, 0.090], bootstrap p 0.062: opposite sign and 1.48x the pooled magnitude. Collaboration trend-adjusted +0.0119, CR1 CI [0.002, 0.022] vs pooled -0.0150 (sign flip). 'Leaves little change' holds for output (0.0021) and top-quartile share (-0.0075) only. The abstract/conclusion/plain-language summary ('accepted slightly faster', 'little change remains') therefore misdescribe two of four outcomes under the paper's own post hoc analysis.
script: 02_abstract_vs_results.pyTabulated pooled (Table 3), event-study (Table 4) and trend-adjusted (Table 10) estimates; approximated the pooled DiD as mean(post coefs) - mean(pre coefs)
Linear extrapolation of 2017-2022 coefficients predicts output 0.059/0.102 and top-quartile 0.010/0.018 for 2024/2025; observed log days lie 0.047/0.074 above the extrapolationConfirmedSec. 4.4; Figure 2; Table 4
Output predicted 0.0592/0.1018 (claimed 0.059/0.102); TQ 0.0107/0.0182 (claimed 0.010/0.018); log days observed minus predicted +0.0480/+0.0743 (claimed 0.047/0.074); tolerance 0.005. The log-days slope is -0.0222/yr, so post-2022 values are above trend, i.e. relative lengthening of acceptance times. Collaboration also lies +0.010/+0.018 above its (downward) trend.
script: 03_trend_extrapolation.pyOLS line through the six 2017-2022 Table 4 coefficients (2022=0), extrapolated to 2024-2025
Relative-magnitude bands equal the CR1 CI widened by M x max|Delta_pre| x (t-2022), M in {0.5,1}; 2025 M=0.5 bands include zero for all outcomesConfirmedSec. 3.7; Sec. 4.3; Table 4 (bottom rows)
All 16 band endpoints reproduced within 0.0018 (tolerance 0.004; SEs are rounded to 3 dp), e.g. output M=0.5 2025: claimed [-0.097, 0.297], obtained [-0.096, 0.296]. The max change includes the 2021->2022 step (output 0.077, log days 0.055); excluding it would give bands up to 0.067 narrower, so the bands are reproducible only with that convention. All four 2025 M=0.5 bands contain zero, as stated.
script: 04_rm_bands.pyRecomputed CR1 CIs from Table 4 coefficients and SEs with t_{0.975,G-1} (G from Table 3), widened using the largest consecutive pre-period change including the 2021->2022 step
CR1 intervals and p-values are mutually consistent; bootstrap p<0.05 iff bootstrap CI excludes zeroConfirmedTables 3, 5, 6, 9, 10
All 10 CR1 rows with p-values in Tables 3 and 10 reproduce within rounding (e.g. Table 3 output CR1 p claimed 0.011, obtained 0.0107; log days 0.033 vs 0.0330). 0/95 Table 9 cells deviate from the CI midpoint by >0.0015. All 24 bootstrap rows in Tables 3, 5, 6 have p<0.05 exactly when the CI excludes zero and the estimate lies inside the CI. No transcription errors detected.
script: 05_ci_p_consistency.pyBacked out SE from CI half-width with t_{G-1}, recomputed p with interval arithmetic for rounding; checked estimate = CI midpoint for 95 Table 9 cells; checked p/CI agreement and estimate-within-CI for 24 bootstrap rows
Pre-registered pre-trend rule applied correctly: output and collaboration pass; top-quartile share and log days failConfirmedSec. 3.7; Sec. 4.3; Table 3 (last row); Tables 4, 5
Rule reproduces the reported classification: output passes (joint 0.433, fake-2021 0.245, fake-2019 0.106), TQ fails (fake-2021 0.022), log days fails (fake-2021 0.017, fake-2019 0.042), collaboration passes (0.123/0.339/0.082). Bootstrap joint p range 0.123-0.491 and 'CR1 rejects all four' match the text. Context: under CR1 p-values all four outcomes would fail, and the output fake-2021 placebo estimate (0.1377) is 68% of the primary estimate (0.2033), so output's 'pass' rests on the low power of the bootstrap rather than on a small placebo.
script: 06_pretrend_rule.pyApplied the stated rule (bootstrap joint lead test or either fake-date placebo rejects at 5%) to Table 4/5 p-values; also applied the same rule to CR1 p-values
BH q-values within each outcome x breakdown family follow from the bootstrap p-values; five estimates have q < 0.05ConfirmedSec. 3.6; Sec. 4.6; Table 7
All 16 families reproduce within rounding when the region family has m=8 (including the 13-author 'Unclassified' row) and the sex family m=3 (including 'Unknown'); e.g. China output p 0.020 -> q 0.080 (reported 0.079), SSA output 0.002 -> 0.016 (reported 0.018, within rounding). Excluding Unclassified (m=7) or Unknown (m=2) does not reproduce the table. Exactly five reported q < 0.05, as stated. Minor inconsistency: Table 2 (no. 2) calls the Unclassified row 'too small to estimate', yet Table 7 reports estimates for it and it enlarges the BH family.
script: 07_bh_qvalues.pyRecomputed BH step-up q from reported 3-dp p-values with interval arithmetic for rounding, for all 16 outcome x family combinations; tested alternative family sizes
Sample-flow and subgroup counts are internally consistent (Table 1 flow, 74,627 authors, subgroup sizes, sex shares and coverage, observation counts)DiscrepancyTable 1; Sec. 3.3, 3.8; Tables 3, 4, 7, 8
Closes: 86,612 - 3,637 exclusions = 82,975 = 56,883 + 17,744 + 8,348; 56,883 + 17,744 = 74,627; career, region, strict sex, lenient sex, sex x career cells and the 26 Table 8 fields all sum to 56,883; strict sex shares 0.225/0.283/0.492 match Table 1 (0.23/0.28/0.49); coverage 50.8% (claimed 51%); event-study minus pooled output observations = 74,627 exactly (one 2023 row per author). Does not close: the four Table 7 domain sizes sum to 56,875, 8 authors short of 56,883, although every author has a field (Table 8 fields sum to 56,883) and domains are aggregates of fields. Minor; unexplained in the paper.
script: 08_sample_counts.pyInteger arithmetic on all reported counts
Simulation evidence: CR1 rejected 13.1% of 336 true-null tests vs 7.1% for the bootstrap; CR1 pre-trend test rejected in 32-57% of runs vs none; 'the bootstrap is conservative, with a smallest null p-value of 0.065'DiscrepancySec. 3.6
Percentages map to integers (44/336, 24/336, 9/28, 16/28). CR1 rate 13.1% (95% CI 9.7-17.2%) clearly over-rejects. Bootstrap 7.1% (CI 4.6-10.4%; binomial p = 0.079 vs 5%) is not shown to be conservative: it rejects above nominal. 24 rejections at 5% require 24 null p-values below 0.05, which contradicts 'a smallest null p-value of 0.065' if both statements refer to the same true-null tests; the paper does not say they refer to different test sets (e.g. joint pre-trend tests). The characterisation 'conservative' and the size evidence for the primary inference method are therefore internally inconsistent as written. Simulations used 38 countries, not the real 6-control / 172-cluster design.
script: 09_simulation_claims.pyMapped percentages to integer counts; Clopper-Pearson CIs and binomial tests vs 5%; logical compatibility check
Pre-registered MDEs (80% power): log output 0.011-0.054, international share 0.009-0.042, log days 0.021-0.107; 'realistic sensitivity lies between the two'DiscrepancySec. 3.8 (power paragraph); Sec. 5.3; Table 3
Log output: realised bootstrap MDE 0.427 (about 53% on the count scale), 7.9x the planned upper bound 0.054; even the CR1 MDE 0.221 is 4.1x the upper bound. International share: bootstrap MDE 0.064, 1.5x the upper bound 0.042. Log days: 0.064, inside the planned range. So for the headline outcome the realised sensitivity is far outside, not 'between', the pre-registered values; the null result for output could not have detected effects smaller than ~50%. Sec. 5.3 acknowledges that MDEs 'exceed some plausible effect sizes' but understates the gap.
script: 10_realized_mde.pyRealised SE from Table 3 bootstrap CI half-width/1.96 and CR1 half-width/t_{G-1}; realised MDE = (z_0.975 + z_0.8) x SE
Abstract 'Output rose by 22.5% relative to English-country authors'; pre-period means (Table 3) match Figure 1; what drives the output DiDConfirmedAbstract; Sec. 4.1; Figure 1a-d; Table 3
Digitised pre-period means agree with Table 3 within weighting/digitisation tolerance (output non-English 1.928 vs 1.944, English 2.034 vs 2.037; TQ 0.615 vs 0.621, 0.722 vs 0.724; log days 4.457 vs 4.433, 4.554 vs 4.546; intl 0.271 vs 0.269, 0.342 vs 0.343). Raw output log-ratio DiD = +0.207 (PPML 0.2033), but non-English output changed by -1.0% (1.928 -> 1.909) while English-country output fell by 19.5% (2.034 -> 1.638; from a 2020 peak of 2.204 to 1.629 in 2025): the entire 'rise' is a fall in the comparison group. Log days rose for both groups after 2022 (non-English +0.184 vs English +0.093 from 2022 to 2024-25, raw means), consistent with check 02. The abstract wording 'Output rose' invites a reading the data do not support; and the comparison-group decline raises attrition/indexing questions the paper does not examine.
script: 11_fig1_digitize.pyDigitised the embedded Figure 1 raster (axis calibration from detected tick marks, marker centroids by colour) and computed unweighted group pre/post means and the raw log-ratio DiD
OpenAlex snapshot of 14 Oct 2026 with 1,898 works Parquet files (Table 2 no. 1); plan frozen 2026-11-09T20:58:04Z before the data scanConfirmedSec. 3.1-3.2; Table 2 no. 1; Data and code availability; PDF metadata
Release 2026-10-14 exists; works manifest dated 2026-10-14 lists exactly 1,898 files (464,765,485 records, 744.0 GB), matching deviation no. 1. PDF CreationDate is 2026-11-10 00:11:35 UTC, 3.23 h after the stated lock: scanning 744 GB (>= 64 MB/s sustained), retrieving 595,873 PubMed histories (>= 51 records/s), running all bootstrap models and writing the paper all fit in that window. Feasible for an automated pipeline but tight, and the lock is self-held: the lock file, plan and code are not supplied, so 'frozen before the scan' cannot be verified. Release notes also document large author-profile split/merge clean-ups during 2025-2026 (e.g. '2.6M splinter profiles merged ... affecting ~5M works'), relevant to an author-ID panel and not discussed in the paper.
script: 12_snapshot_provenance.pyFetched OpenAlex RELEASE_NOTES.txt and data/parquet/works/manifest.json from the public S3 bucket; counted files/bytes; read PDF CreationDate with pdfinfo; computed the lock-to-PDF window
Sampling frame: 966,412 sampled author ids from a 1% hash sample of OpenAlex author identifiers; 8,442,229 works with a sampled authorInconclusiveSec. 3.2; Table 1
The authors table has 101,066,311 records; a 1% Bernoulli/hash sample would give 1,010,663 +/- 1,000, whereas 966,412 is 0.956% (-44 SE). This is consistent only if the frame is the ~96.6M distinct author ids occurring in works authorships (the paper scanned the works table), which the paper does not state. Reported works share 8,442,229/464,765,485 = 1.82%; one partition (19,563 works, 39.6% without author ids) gives 0.84-1.18% under 1% hashes, same order but not representative of the whole table. Cannot be settled without scanning the 744 GB works table or the released code.
script: 13_sampling_frame.pyRead the 2026-10-14 authors manifest record count; estimated the share of works with >=1 sampled author from one downloaded works Parquet partition under three arbitrary 1% hashes
Sec. 4.7: leaving out China roughly halves output (0.099); leaving out US nearly doubles log-days reduction (-0.070); intermediate-as-English output 0.158 CI includes zero; remaining checks leave all four estimates close to primary valuesDiscrepancySec. 4.7; Table 9; Table 3
The named statements hold: 0.099/0.2033 = 0.49; -0.070/-0.0385 = 1.82; CI [-0.015, 0.331] contains 0. But 'the remaining checks leave all four estimates close to their primary values' does not hold for several rows. Leaving out Europe & Central Asia gives output 0.294 (+45%), top-quartile 0.035 (+26%), log days -0.054 (-40%, i.e. larger reduction) and collaboration -0.025 (-67%). Leaving out GB gives log days -0.025 (35% smaller reduction, smaller than the China leave-out's -0.026) and collaboration -0.009 (+40%). The ECA leave-out moves output by more in absolute terms (+0.091) than the US leave-out moves log days. Severity minor-to-moderate: point estimates keep their sign, but the text understates the heterogeneity.
script: 14_robustness_text.pyRatios of Table 9 estimates to Table 3 primary estimates; relative deviation of every 'remaining' row for each outcome
Numbers quoted in the text of Secs. 4.3, 4.4, 4.5, 4.6 and 4.8 match Tables 4, 5, 6, 7 and 10 (e.g. fake-2021 TQ 0.025 'almost as large as' 0.0277; no secondary bootstrap p<0.05; five q<0.05)ConfirmedSecs. 4.3-4.6, 4.8; Tables 4, 5, 6, 7, 10
All 41 quoted values match their tables after rounding. Two match within one unit in the last digit (TQ extrapolation 2024: text 0.010 vs 0.0107; log-days gap 2024: text 0.047 vs 0.048), but these are compared with values recomputed in check 03 from rounded Table 4 coefficients. Fake-2021 TQ / primary = 0.92. The minimum secondary bootstrap p is 0.066 (log team size). Exactly 5 of 72 Table 7 cells have q<0.05, and they are the five named in Sec. 4.6. Minor wording points: the 'collaboration leads are close to zero' (Sec. 4.3) although the 2017 lead is 0.028 (SE 0.014, t=2.0), 1.9x the magnitude of the primary estimate -0.0150; Sec. 4.8 calls the OLS output change 'imprecise' while its CR1 p is 0.023 (bootstrap p 0.207), and the imputation estimate has no uncertainty reported.
script: 15_text_table_numbers.py41 text-vs-table comparisons with rounding to the printed decimals; Table 7 parsed from the extracted text (72 cells) to count q<0.05; qualitative statements checked against table p-values
References exist and support the load-bearing claims ([7] 13.5% of 2024 PubMed abstracts; [10] 36-60% preprint gains; [11] 72%; [13] detection; [14]; [15]; [16] trend-dependent gains; [83] ~3/4 affiliation countries correct)DiscrepancySecs. 1, 2.2, 2.3, 5.3; References
All 83 references resolve to existing works; the 23 weak automatic matches were markup or title-version issues and resolved on targeted queries, except [60], a blog post with no locator. The abstracts support [7] ('at least 13.5% of 2024 abstracts', varying by country), [8], [14] (no post-ChatGPT change in non-native share or top-quartile share), [15] (~8% output gain; NES researchers produced more than NNES), [16] (advantage 'disappears after accounting for a differential trend already visible before 2022'), [41] (-36% NES collaboration) and [83] (country-level accuracy ~0.75 in OpenAlex, from 100 authors). [11]'s 72% is the 2025-specific estimate of arXiv v2 (52-88% across specifications), presented in the paper without the year. [13] is cited twice for detection failure on polished LLM text, but its abstract contains nothing on detection (unsupported as far as the abstract shows). [18] merges two author names. [10]'s 36-60% matches per-platform figures reported by web sources (not script-verified; the arXiv abstract says 23.7-89.3%). A directly relevant published critique of [10] (Renault et al., PNAS 2026) is not cited. Issues are listed in work/ref_issues.json.
script: 16_refs.pytools/check_refs.py on all 83 references (work/refs_checked.json); targeted Crossref/arXiv re-queries of the 23 weak matches; abstracts fetched from arXiv, OpenAlex, Semantic Scholar and PubMed and searched for the attributed quantities (work/verify/16_refs.py, cached responses in work/verify/refs/)
References
83 references checked
- [13] Lin, Zhao, Tian & Li, 'Large language models reshape the language of science', arXiv 2504.12317 (2025)
Cited in Sec. 1 ('polished LLM text frequently escapes detection') and Sec. 2.2 (detectors 'miss most polished LLM text') as evidence on detection failure, but its abstract reports lexical and syntactic shifts in 21.36M abstracts and polishing experiments that reproduce them. It says nothing about detectors missing polished text. Sec. 2.3 also says the abstracts are 'in OpenAlex', which the abstract does not state. The detection claim appears unsupported by this source (checked against the abstract only; work/verify/16_refs.log).
- [10] Kusumegi et al., 'Scientific production in the era of large language models', Science 2025 (10.1126/science.adw3000)
The '36-60%' figure matches the per-platform averages reported for the Science version (36.2% arXiv, 52.9% bioRxiv, 59.8% SSRN; from web sources, since the PubMed abstract gives no numbers). The arXiv abstract gives 23.7-89.3%. The paper uses [10] as the main adopter-design contrast in Secs. 1, 2.3 and 5 but does not cite the published critique by Renault, Bergeaud & Bosquet (PNAS 2026, 10.1073/pnas.2618638123; verified on Crossref). That critique argues that dating adoption from the first flagged paper mechanically produces such gains, which bears directly on the paper's comparison.
- [11] Filimonovic, Rutzer & Wunsch, 'GenAI-Assisted Writing and Academic Performance...', arXiv 2510.02408
Exists (v2 carries this title; the SSRN/v1 version is titled 'Can GenAI Improve Academic Performance?'). The 72% figure is the 2025-specific estimate of the revised version (range 52-88% across specifications; earlier version: 15% in 2023, 36% in 2024). The paper presents it as a general gain ('increase their journal output by about 72%') without the year qualifier or the version.
- [18] Man, Weinkauf, Tsang & Sin, European Journal of Epidemiology 2004 (10.1023/B:EJEP.0000036571.00320.b8)
The work exists and the title matches, but the author list prints 'James Hogg Don D. Sin', which appears to merge two authors (James Hogg; Don D. Sin) into one name.
- [60] Himmelstein, 'The history of publishing delays', 2015
A non-archival blog post given without URL or DOI. Crossref resolves to a different Himmelstein item ('Licensing neo4j'). It is cited for the publication-delay data rule in Sec. 3.4 and cannot be located from the reference as given.
Suggestions
Rewrite the abstract, Sec. 5.1, Sec. 6 and the plain-language summary so that they report the post-2022 relative lengthening of acceptance times and the sign reversals of log days and collaboration under trend adjustment.
Replace 'Output rose by 22.5%' with a description of both group trajectories, and add an analysis of comparison-group attrition, 2025 indexing completeness and OpenAlex profile revisions by group.
Report realised MDEs or equivalence tests for each primary outcome, and describe the output result as inconclusive given an MDE of ~0.43 log points.
Rerun the inference simulation on the realised cluster structure (172 clusters, 6 control, US 68%), clarify the 7.1% versus 'smallest null p 0.065' statements, and cite MacKinnon & Webb (2018) on few-control-cluster wild bootstrap.
Deposit the frozen plan, lock file and code with an external timestamp (e.g. OSF, Zenodo, or an RFC 3161 timestamp), and qualify 'Pre-Registered' in the title until it is verifiable.
Retitle around the change in the English-language gap after 2022 rather than 'ChatGPT', and name the concurrent shocks (end of zero-COVID in China, post-pandemic publishing shifts, evaluation reforms) in the abstract.
State the sampling frame precisely (authors table vs author IDs in works) and reconcile 966,412 with 1% of the authors table.
Add robustness checks that exclude 2025, exclude China, and use alternative reference years (2019 or 2021) to test sensitivity to the COVID-affected 2022 base year.
Fix the minor inconsistencies: Table 7 domain sizes, the Unclassified-region treatment, the Unknown sex group in Figure 4, inference for the imputation estimator, [13], [11], [18] and [60].
Ethics and integrity
No embedded instructions aimed at reviewers were found in the paper, and none were present in the requester's notes. The study uses public bibliometric data. Sex is inferred from first names with gender-guesser, a practice with known misclassification and non-binary exclusion issues, which the paper acknowledges (Sec. 3.8, 5.3). Integrity note: the manuscript was produced end to end by an automated research system. The claimed pre-registration is a self-held hash lock created about 3.2 h before the PDF, and the materials said to accompany the article were not supplied, so provenance and plan adherence rest on the system's own account. We found no sign of fabricated numbers: every table is internally consistent and the snapshot details match public OpenAlex metadata.
Scope of this review
Only the PDF was supplied (no LaTeX source, code, frozen plan, lock file, deviation log or derived data, although the paper says they accompany it). We could not re-estimate any model, rerun the bootstrap, verify the pre-registration timestamp or hashes, or inspect the eligibility and sampling code. Figure 1 values were obtained by digitising the embedded raster and are approximate. Reference support was checked against abstracts only. Scanning the 744 GB OpenAlex works table to rebuild the sample was outside the time budget. The overall score of 5 exceeds the cap of 4 for a confirmed discrepancy because none of the discrepancies overturns the paper's central conclusion of no detectable change in group gaps: they weaken how it is described and how informative it is.