A careful, honestly non-causal matched-cohort design whose exposure data I could largely reproduce. However, Table 3's unadjusted estimates cannot be reconciled with Table 2, the claim of 'no deviation' from the preregistration overstates adherence, and the E-value and the comparison with prior work overstate how robust and typical the 3x association is.
Soundness
2/4
Presentation
3/4
Contribution
2/4
Confidence
4/5
✓ 12 confirmed✕ 6 discrepancy◐ 1 inconclusive○ 1 not checkable
Summary
The paper asks whether arXiv papers listed on Hugging Face Daily Papers go on to receive more citations than comparable papers that were never listed. It matches each listed paper (listed 0 to 30 days after v1) to up to three never-listed papers from the same primary category and v1 week, using a propensity score built from arXiv-observable features. It then regresses log(1+cit) at 6, 12 and 24 months on listing status, with stratum fixed effects and pre-v1 author-prominence and institution covariates (H1). Among listed papers it relates citations to day-7 upvotes (H2), and it planned a rank-cutoff regression discontinuity at top-1/top-3 (H3).
Headline results: the adjusted geometric mean of citations+1 is 203.3% higher (ratio 3.03) at 24 months (N = 11,401) and 107.1% higher at 6 months. The H2 elasticity is 0.623 at 24 months. H3 was infeasible under the paper's own rule, and descriptive top-rank contrasts show no advantage. The paper describes itself as preregistered with "no deviation from the plan". Its interpretation is explicitly associational: the listing is "a strong early marker" and the design cannot separate attention from selection.
Strengths
Well-chosen comparison group and outcome clock
Sec 3.2-3.4
Controls come from the same primary category and v1 week (Sec 3.2), which holds fixed the release moment and the competing supply of papers. Citations are counted over fixed windows anchored at v1, with a 3-month indexing buffer and horizon-specific cohorts (Sec 3.3). My calendar check (check 09) confirms that every stated cut-off gives a fully elapsed window before the 10 Nov 2026 reference date, with 10 days to spare. Covariates are measured before v1 (Sec 3.4), which avoids the post-treatment author-metric problem the paper correctly identifies in [5].
Disciplined, non-causal language throughout
Title; Abstract; Sec 5.3-5.4; Sec 6
The title is associational, and the abstract closes by saying the design "cannot separate an attention effect of the listing from selection". A scripted scan (check 16) found that every causal-vocabulary hit in the body is hedged, negated or refers to prior work. All 10 numbers in the abstract appear in Tables 3/5 or the results text. The requester specifically asked about causal language, and on this point the paper is exemplary.
Exposure data are real and reproducible
Sec 3.1; Table 1; Table 6; Table 4 Panel B
I rebuilt the exposure side from the public sources named in Sec 3.1 (check 14, check 18). The Daily Papers API's first content date is 2023-05-04, as stated. The community dataset [11] exists under CC BY 4.0 and starts on 12 Mar 2024. All 12 Table 6 feasibility cells reproduce within 6 eligible days and 2 gap-1 days. S7 N is 322 vs 324 reported. Author-participation flags are 61.5% vs 62.3%. Listed papers in frame number 12,187 vs 12,325.
Internally consistent sample flow and H1/H2 arithmetic
Table 1; Table 3; Table 5; Table 9; Sec 4.1
Every sum in Table 1 closes exactly (12,251; 11,020; 31,523; 43,774), and the H1 cohort counts equal the Table 3 Papers column (check 05). All Table 3 and Table 5 CIs are symmetric on the log scale, and the implied z-values support Holm p< (checks 01, 12). The ratio 3.03 and the 15% attenuation from adjustment reproduce. The Table 9 subgroup estimates pool back to the overall estimates (check 11). Small checks pass GRIM and SMD recomputation (check 13).
Weaknesses
Table 3 unadjusted estimates are inconsistent with Table 2 under the stated weighting
Sec 3.2 gives featured papers weight 1 and controls weight 1/k, so treated and control weights are equal within every stratum. With equal weights, a stratum-fixed-effects WLS coefficient equals the weighted raw contrast exactly (I verified this to 10−9 in simulation). The unweighted Table 2 contrast can then differ from it only through the 970 matched sets with fewer than 3 controls. Yet the unweighted log geometric-mean ratios from Table 2 are 1.169 / 1.208 / 1.364 (6/12/24 m), while the Table 3 unadjusted estimates (135.2%, 187.8%, 266.4%) are 0.855 / 1.057 / 1.299 on the log scale. That is a gap of 0.31 log points at 6 months. Closing it would require controls in small sets to have a mean about 7 units higher than all other controls, whose overall mean is 0.445 (check 02). So either a table is wrong, or the unadjusted model is not the one described in Sec 3.5.1, and this raises doubt about the adjusted column computed by the same pipeline. The direction and the large size of the association are not in question, since both tables show listed papers ahead. But the 15% 'attenuation from adjustment' and the exact headline magnitudes cannot be trusted until this is resolved. Fix: report the weighted group geometric means beside Table 2, state exactly which sample and weights each column uses, and release the code.
Questions for the authors
Exactly what sample and weights produce the Table 3 'Unadjusted' column, and can you reproduce Table 2's geometric-mean ratio (e.g. log(5.02/1.56)=1.169 at 6 m) from the same model? If Table 3 is corrected, how do the adjusted estimates change?
Can you deposit S1 (full SHA-256 and plan text) and S2 with an independent timestamp, and list each post-lock operational definition next to the original plan wording, so readers can judge whether any of them is a deviation?
When the plan was written after inspecting the exposure data, was it already known that no H3 cell could reach 300 close-vote days? If so, why was H3 framed as confirmatory?
What is the H1 estimate separately for curator-listed and author-submitted papers? If the association is similar in both, what does that imply for the selection explanation?
Why does the H2 elasticity roughly halve when purged-vote papers are excluded, and why do 74% of 24-month listed papers have purged votes?
Why does Table 7 use 6 one-vote days for day-7 top-1 at 6 months when Table 6 (and the public data) give 8, and where are the 12- and 24-month day-7 top-1 rows?
Verification
Only the PDF was available, so I checked the paper's internal arithmetic, its consistency across tables, its references and its public exposure data; I could not rerun its pipeline. I ran 20 scripted checks: 12 confirmed, 6 discrepancies, 1 inconclusive, 1 not checkable. Confirmed: the sample flow, the arithmetic behind the headline H1 and H2 numbers (CIs, ratios, Holm significance, E-values under the paper's own formula), the calendar windows, and the Table 9 pooling. I also confirmed the existence and licence of the data sources, and reproduced the exposure-side counts (Table 6, S7, author flags) from the public Hugging Face data to within a few percent. Discrepancies: the Table 2 vs Table 3 unadjusted inconsistency (up to 0.31 log points); the 'similar factor' comparison with [5] (1.71 vs 3.91); the E-value conversion (about 3.5 vs 5.52); a Table 6/7 day-count mismatch; two Table 7 CI/p conflicts; and the 'no deviation' preregistration claim. Citation outcomes, covariates and the model fits themselves could not be recomputed.
[5] Weissburg et al., Position: AI/ML Influencers Have a Place in the Academic Process (ICML 2024)
Exists and does report median 24 vs 14 citations for papers shared by AK, but Sec 5.2 uses it to claim the 24-month medians here (43 vs 11) 'differ by a similar factor'; 24/14 = 1.71 vs 43/11 = 3.91 (work/verify/03_weissburg_ratio.py). The citation is accurate but the comparison drawn from it is not.
[1] Adina Yakefu. Exploring the Daily Papers Page on Hugging Face, 2024.
Grey-literature blog post with no URL; Crossref returns an unrelated item and a guessed huggingface.co/blog URL returns 404. It is the only source given for how the page is curated and ordered by upvotes (Sec 1). Needs a URL or archived link.
[11] tardellirs. Paper Pulse: daily upvote histories for Hugging Face Daily Papers (dataset tardellirs/paper-pulse-data), 2026.
Exists and is CC BY 4.0 as stated, but it was created on 2026-10-04 and is rebuilt daily (last modified 2026-11-11). It is itself derived from @hysts' daily-papers-stats (CC0) and has two collection gaps. The reference gives no commit/revision or download date, so the exposure data version is not pinned. Counts recomputed from the current revision match Tables 5-6 within a few percent (work/verify/18_exposure_reproduction.py).
[2] Du, Tuan, Huang, Ng. Paper Espresso: From Paper Overload to Research Insight. arXiv preprint, 2026.
Cited as an arXiv preprint; Crossref finds an ACM-published version (doi 10.1145/3799682.3840141). Minor bibliographic update. The claim attributed to it (right-skewed upvote distribution) was not spot-checked.
[25] Feenberg, Ganguli, Gaule, Gruber. It's Good to Be First. Review of Economics and Statistics 99(1), 2017.
Crossref's best match is the NBER working-paper version (2015, doi 10.3386/w21141). The journal version exists; no substantive issue.
Suggestions
Reconcile Table 2 and Table 3: add weighted group geometric means, state the sample and weights for every column, and rerun the unadjusted and adjusted models from the released code.
Replace "No deviation from the plan occurred" with an explicit deviations table (post-lock definitions, unrun RD diagnostics, S5, S7-for-H3, missing Table 7 rows), and deposit S1/S2 with an independent registry timestamp.
Recompute S10 with the continuous-outcome E-value conversion of [49], or on a binary outcome, and temper the robustness language in Sec 4.3 and 5.3 accordingly.
Correct the comparison with [5] in Sec 5.2 (1.7x vs 3.9x) and discuss why the association here is larger despite tighter controls.
In the abstract, describe the horizon trend using the within-24-month-cohort trajectory, and mention the S7 sensitivity of H2 next to the 0.623 elasticity.
Estimate H1 separately for curator-listed and author-submitted papers. This is the most direct available lever on the selection-versus-visibility question.
State the CI method for Table 7 and use permutation-based (exact) intervals for cells with 5-8 days. Mention that all top-1 contrasts are negative.
Pin the data: give the revision hash or download date for [11], the Semantic Scholar/OpenAlex snapshot dates, and a URL for [1]. Release the frozen dataset and code promised in the Data availability section.
Ethics and integrity
No human or consent issues: all data are public bibliometric and platform metadata, and [11] is CC BY 4.0. Two integrity points. First, the sole listed author is an AI system (IterSCI) and no accountable human author or contact is named. Readers should know who is responsible for the plan lock, the data and the corrections requested above. Second, the preregistration is self-attested: plan lock, outcome retrieval, analysis and paper date all fall on 10 Nov 2026, and only a truncated hash is printed. I found no sign of fabricated data, because the exposure-side counts reproduce from public sources. No instructions aimed at reviewers were found in the paper or the requester notes.
Scope of this review
Only the PDF was supplied: no code, LaTeX source, frozen dataset, or Supplementary Files S1 (plan and hash) and S2 (deviations log). I therefore could not rerun the matching or regressions, recompute citation outcomes from Semantic Scholar/OpenAlex, or verify the plan hash and lock timestamp. The exposure side was partly reproduced from the live public Hugging Face API and dataset [11], which is rebuilt daily, so exact matches were not expected. The E-value recomputation (check 04) relies on SDs inferred from Table 2 under a lognormal approximation. Figures 2-3 could not be read numerically.
Infeasible confirmatory analysis reported rather than buried
Sec 3.5.3; Sec 3.6; Tables 6-8; Table 10
The paper reports that the preregistered RD failed its feasibility rule (Table 6, maximum 102 of 300 required days; confirmed in check 06). It labels the local-polynomial RD as exploratory (Table 8) and gives a descriptive local-randomisation analysis instead of quietly substituting a confirmatory estimate. Data-quality problems are tabulated openly (Table 10: unresolved records, pre-v1 citing works, missing affiliations).
Load-bearing literature cited accurately
Sec 2; References
All 49 references resolve except a blog post [1] that has no URL. Spot-checks of [5], [8], [9], [10] and [24] show that each supports the specific claim it is cited for (check 17). Related work fairly presents the evidence pointing both ways: observational promotion studies versus RCTs and self-selection in listing position.
log(1+cit)
"No deviation from the plan occurred" overstates preregistration adherence
The same section (Sec 3.6) lists several departures from the plan. (i) A post-lock record "fixes the operational definitions" for matching weights after cohort restriction, per-paper cohort membership, the highly-cited indicator, tie averaging and all-zero strata in S1. (ii) The RD density test, balance checks and covariate-adjusted RD were not run. (iii) S5 was not computed because citing-author records were never retrieved. (iv) S7 was not estimated for H3. In addition, the day-7 top-1 rows at 12 and 24 m are missing from Table 7 even though cells below the feasibility threshold were to be reported descriptively (check 07). The plan was written after the exposure data had been inspected (Sec 3), and the H3 feasibility rule depends only on those data. The 24-month cells could never reach 300 days (at most 172 calendar days of snapshots), and the 6/12-month cells would have needed 61-85% close-vote days against 4-31% observed (check 06). So the confirmatory H3 had no realistic chance at lock time. Only a truncated hash is printed, S1/S2 were not supplied, and there is no third-party registry timestamp. Lock, reference date and paper date are all 10 Nov 2026 (check 09). The fair description is "a planned analysis with documented post-lock clarifications and several unrun components". Fix: replace the blanket statement with a deviations table, and deposit the plan with an independent timestamp (OSF/AsPredicted).
E-value treats a ratio of geometric means as a risk ratio
S10 converts eβ from a log(1+cit) model into an E-value as if it were a risk ratio (Sec 3.5.5). Sec 4.3 and 5.3 then read 5.52 as evidence that confounding "would need" to be strong. The paper applies its own formula correctly (check 01). However, the continuous-outcome conversion recommended by the cited source [49], RR≈exp(0.91d), gives about 3.5 (limit 3.3) at 24 m, with the SD of log(1+cit) estimated at about 1.4 from Table 2 (check 04). Even an implausibly small SD of 1.0 gives 4.93. The adjustment set cannot capture paper quality or the curator's own judgement, and the measured covariates already move the estimate by 15%. So an E-value near 3.5 sets a materially lower bar for an unmeasured 'quality' confounder than the paper suggests. Fix: use the standardized-difference conversion, or dichotomise the outcome (e.g. the S2 top-decile indicator) and report the E-value for that risk ratio.
Comparison with prior work misstates the effect size
major · Sec 5.2 para 1; Table 2; Ref [5]
Sec 5.2 says the 24-month medians here (43 vs 11, Table 2) "differ by a similar factor" to the 24 vs 14 reported in [5]. [5] does report 24 vs 14 (check 03), but 24/14 = 1.71 while 43/11 = 3.91, and even the adjusted ratio is 3.03. The association found here is roughly twice as large as the closest prior estimate. This is substantive, because the paper uses the agreement to place its result in context. A much larger association than in [5], despite tighter controls, is itself a finding that needs explanation (e.g. the stronger selection implied by author self-submission, or different citation sources). Fix: correct the sentence and discuss the gap.
Horizon trend in the abstract and intro mixes horizon with release cohort
The intro (p.2) and Sec 4.3 say the difference "was smaller at shorter windows", and the abstract pairs 107.1% (6 m) with 203.3% (24 m). Those estimates come from different release cohorts. Pooling Table 9 (check 11), the 24-month cohort itself shows 160.3% at 6 m and 210.5% at 12 m. Later releases (2024H2-2026H1) show only about 92% at 6 m. Only about 45% of the 6-to-24-month log gap is a within-cohort change over time; the rest reflects later releases having smaller differences. Sec 5.4 acknowledges this, but the abstract does not. Fix: report the within-24-month-cohort trajectory (Fig 3b) as the horizon comparison in the abstract, and present the pooled 6-month estimate as a different population.
The abstract reports an elasticity of 0.623 at 24 m. Table 4 Panel B shows that S7, which excludes papers with purged votes, gives 0.333 [0.020, 0.647]. S7 keeps only 324 of the 1,255 papers, so 74% of the 24-month H2 sample had purged votes (check 12). The difference from the primary estimate is not significant (rough z=1.67), but the text calls the other versions similar and does not discuss S7. Given that purging indicates vote manipulation or cleaning, the primary estimate may partly reflect manipulated upvotes. Fix: mention the S7 estimate next to the headline and discuss why most of the early cohort was affected.
Table 7 day counts and CI/p reporting are inconsistent
Day-7 top-1 at 6 m uses 6 days in Table 7 but Table 6 counts 8 one-vote days. The public data also give 8 (checks 07, 18). Two cells (12 and 24 m) are omitted without comment. In two rows the 95% CI excludes 0 while the permutation p exceeds 0.05: day-1 top-1 12 m, −0.99 [−1.96, −0.02], p=0.070; and 24 m, −1.95 [−3.00, −0.90], p=0.065 on 5 days (check 08). The CIs appear to be large-sample normal intervals, which are invalid with 5-6 days, and the CI method is not stated. Separately, "no consistent rank advantage" (Abstract) is literally true, but every top-1 contrast in Tables 7 and 8 is negative (e.g. Table 8 top-1 24 m: −2.23 [−3.31, −1.16]). That pattern deserves mention.
S3 shows most listings are author-submitted; its precision is unexplained
minor · Table 4 Panel A (S3); Sec 4.4; Sec 1
S3 flags 7,638 featured papers (62% of eligible listings) as author-submitted or author-participating. That sits uneasily with the "curated feed" framing (Sec 1), because most listings are self-selected, much as [9,10] found for arXiv position. At least 6,407 of 11,020 matched featured papers are flagged, so the SE should grow by at least 1.55x. Observed SE ratios are 1.38/1.32/1.19 (check 10, inconclusive). The S3 sample sizes are not reported. Fix: report S3 Ns, and give results separately for curator-only and author-submitted listings, since this is the most informative contrast for the selection-versus-visibility question.
Data provenance not pinned; code and supplements not supplied
minor · Sec 3.1; Data and code availability; Refs [1], [11]
The upvote dataset [11] was created on 2026-10-04 and is rebuilt daily (last modified 2026-11-11). It is derived from another CC0 dataset and has two collection gaps (check 14). The reference gives no revision or download date, so the exposure data cannot be pinned, and my reproduction differs by a few percent (check 18). The Data availability section promises code, a frozen dataset, S1 and S2, but none were provided, so the outcome side (Semantic Scholar/OpenAlex citations, covariates) could not be examined (check 20). Ref [1], the only source for how the page is curated and ordered, has no URL.
(a) The plain-language summary sets 11,020 vs 31,523 papers beside 2-year medians that come from the 2,981/8,420 subcohort (Sec 7). (b) The title says "AI preprints", but the strata include non-AI "Other categories" (Table 9). (c) 10.3% (featured) and 15.6% (control) of 6-month citing works are dated on or before v1 (Table 10). This biases the 6-month ratio toward the null by about 6% (check 19) and deserves a sentence. (d) Table 5 H2b counts are 1-3 papers below the Table 1 featured counts without explanation (check 05). (e) The reference population for the S2 top-decile percentile is not stated.
At least 6,407 of 11,020 matched featured papers carry a flag (7,638 - 1,231 unmatched), so at most 41.9% remain and SEs should grow by >= 1.55x if residual variance is unchanged. Observed SE ratios are 1.38 / 1.32 / 1.19 (6/12/24 m), implying effective retained shares of 52%/57%/71%. This needs the retained subset to have 10-23% lower residual SD, or flags to be concentrated in later cohorts (possible at 24 m). It cannot be settled without the S3 sample sizes, which are not reported. Separately, 62% of eligible listings being author-submitted or author-participating qualifies the curated-feed framing.
Backed out log-scale SEs of primary and S3 rows and compared the SE ratio with the ratio expected from the share of matched featured papers that can remain after exclusion
The difference was smaller at shorter windows (107.1% at 6 m, 143.1% at 12 m, 203.3% at 24 m) and, within the 24-month cohort, largely established by 12 months; Table 9 subgroups
Table 9 is coherent with the overall estimates: pooled half-years 106.7/144.8/192.4 and pooled categories 105.2/140.9/204.5 vs overall 107.1/143.1/203.3. Within the 24-month cohort the estimate is 160.3% at 6 m, 210.5% at 12 m and 208.7% at 24 m, which supports 'largely established by 12 months'. However, only 45% of the log-scale gap between the 6-month (107.1%) and 24-month (203.3%) headline estimates appears within the same releases. The rest reflects later releases having smaller differences (2024H2-2026H1 pooled at 6 m: 92.3%). The abstract and intro wording ('smaller at shorter windows') therefore mixes horizon with release cohort. Sec 5.4 acknowledges this.
Inverse-variance pooling of Table 9 subgroup estimates on the log scale (SE from CIs): all half-years and all categories vs overall; 2023H1-2024H1 (the 24-month cohort) followed across horizons
H2 elasticities 0.355/0.436/0.623 (CIs), unadjusted 0.504/0.608/0.770, decile description (1.06 at median 2 upvotes to 2.68 at median 60 upvotes), S7 0.333 [0.020, 0.647] on 324 papers at 24 m
All H2 CIs symmetric (|asym| <= 0.001) with z >= 9 for primary rows (Holm p<0.001 holds). Decile endpoints imply a slope of 0.538, close to the unadjusted 0.504. Decile means 1.06/2.68 are consistent with ln(1+median cit) = 1.10/2.64. Robustness caveat: S7 keeps only 324 of 1,255 papers at 24 m (74% of the 24-month H2 sample had purged votes). The estimate falls to 0.333 (0.53x the primary), z = 2.1 on its own, and the difference from the primary is not significant (rough z = 1.67). The abstract reports 0.623 without this caveat, and the text calls only the NB, complete-case and OpenAlex versions similar.
CI symmetry and z for every H2 row; implied between-decile slope from the decile description; log(1+median) vs mean log; approximate z for primary vs S7
35.2% vs weighted 18.9% large-lab affiliation, SMD 0.37; winners vs runners-up 0.21 vs 0.50 on 14 days; -1.02 log difference = -64%; S2 24.0% vs 4.3%, adjusted 17.3 pp
SMD = 0.373 (claimed 0.37). 0.21 = 3/14 and 0.50 = 7/14 pass GRIM. exp(-1.02)-1 = -63.9% (claimed -64%). The raw S2 difference of 19.7 pp shrinking to an adjusted 17.3 pp is plausible.
Binary SMD with pooled SD; GRIM test for n=14; exponentiation; raw vs adjusted S2 difference
Data provenance: Daily Papers API has content from 2023-05-04; community dataset [11] (tardellirs/paper-pulse-data, CC BY 4.0) gives day-0/1/3/7 upvote snapshots from 12 Mar 2024, within-day ranks and purge flags
API returns 0 items for 2023-05-02/03 and 4 items for 2023-05-04, so the first date with content is as stated. The dataset exists, is licensed cc-by-4.0, and its README describes upvotes on day 0/1/3/7, within-day ranks, a purged flag, purge events, and history starting 2024-03-12. Provenance notes the paper omits: the dataset was created on 2026-10-04 and is rebuilt daily (last modified 2026-11-11). It is derived from @hysts' CC0 daily-papers-stats revisions and has two collection gaps (13 and 7 days). Purge events listed in meta.json: 2025-09-12, 2025-12-23 and seven in Sep-Nov 2026. The API also exposes isAuthorParticipating and submittedBy fields, consistent with the S3 flags.
Queried the public Hugging Face Daily Papers API for dates around 2023-05-04 and the dataset API/README/meta.json with a generic user agent
Preregistration adherence: plan locked 2026-11-10 05:14:48 UTC before outcome/covariate retrieval, SHA-256 ce0c87691dac..., 'No deviation from the plan occurred', every pre-registered estimate reported
Four strong adherence statements ('No deviation from the plan occurred', 'every pre-registered estimate is reported', locked before any citation/author/affiliation data, 'results could not steer the choice of method') sit alongside six items that qualify them. (1) A post-lock record fixes operational definitions (matching weights after cohort restriction, per-paper cohort membership, highly-cited indicator, tie averaging, all-zero strata in S1). (2) The RD density test, balance and covariate-adjusted estimates were not run. (3) S5 was not computed. (4) S7 was not estimated for H3. (5) The plan was written after the exposure data had been inspected. (6) The H3 feasibility rule depends only on those exposure data, and check 06 shows it could not be met. Further, day-7 top-1 descriptive rows at 12/24 m are missing (check 07), and plan lock, reference date and paper date are the same day (check 09). The hash and the plan itself cannot be verified: S1/S2 were not provided, only a truncated hash is printed, and no third-party registry (OSF, AsPredicted) is cited. The fair reading is 'planned analysis with documented post-lock clarifications and unrun components', not 'no deviation'.
Scripted text audit of work/paper.txt extracting adherence statements, post-lock records, unrun analyses and timing facts; cross-referenced with checks 06, 07 and 09
Abstract and title accurately summarise the results and avoid causal claims
All 10 abstract numbers appear in Tables 3/5 and the results text. All 10 causal-vocabulary hits are hedged, negated or describe prior work (e.g. 'cannot separate an attention effect', 'Neither should be read as evidence that the listing causes those citations'). The title is associational. Framing gaps: (i) the abstract reports H2 = 0.623 without noting that S7 (324 of 1,255 papers) gives 0.333; (ii) 'no consistent rank advantage' is literally true, but every top-1 estimate is negative (Table 7: -1.02, -0.99, -1.95, -1.45; Table 8: -0.30, -0.39, -2.23); (iii) the 6-month vs 24-month contrast mixes horizon and cohort (check 11); (iv) the plain-language summary sets the 11,020 vs 31,523 sample beside 2-year medians that come from the 2,981/8,420 subcohort; (v) 'AI preprints' in the title while the strata include non-AI 'Other categories' (Table 9).
Scripted comparison of every abstract number with the results tables; regex scan of the body for causal vocabulary; listing of framing gaps
Load-bearing references exist and support their uses: [24] about twice the citations at the top of astro-ph; [9,10] self-promotion explains much of the advantage while accidental top placement still helps; [8] no detectable citation gain, power calculation implies several hundred papers needed; [5] 24 vs 14 medians
All 49 references resolve to real works (Crossref or arXiv) except [1] (HF blog post with no URL; not located) and [11] (a dataset, verified in check 14). [24] abstract: 'two times more citations for papers at the top'. [9] reports a superposition of self-promotion and visibility bias. [10] says the correlation is 'due primarily to intentional self-promotion', with a measurable visibility boost for accidental placement. [8] reports tweeted papers downloaded 2.6-3.9x more with a non-significant citation increase, and '>= 770 tweeted articles would be needed'. All four uses are supported. The only misuse found is the 'similar factor' comparison with [5] (check 03).
tools/check_refs.py over all 49 references, arXiv look-ups for entries that were rate-limited or unmatched, and abstract/full-text spot-checks of [24], [9], [10], [8] (plus [5] in check 03)
Exposure-side counts (Table 6 feasibility counts; Table 1 listed/eligible counts; Table 5 H2 sample; S7 N=324; S3 7,638 flagged) are reproducible from the public data sources named in Sec 3.1
All 12 Table 6 cells are reproduced within 6 eligible days, 2 gap-1 days and 3 gap-1-2 days (e.g. day-1 top-1 6 m: 488/14/29 vs paper 482/14/29; day-7 top-1 6 m gap-1 = 8, matching Table 6 and not the 6 days used in Table 7). S7 24 m: 322 papers without purged votes vs 324 reported. H2 24 m: 1,197 papers on 99 days vs 1,255/107. Listed with v1 in frame: 12,187 vs 12,325. Author-participating flags: 7,470 (61.5%) vs 7,638 (62.3%). Listed >30 days after v1: 32 vs 74 (larger relative gap, likely because the paper combines two listing sources and uses arXiv v1 dates). Overall, the exposure data are real and the reported counts are reproducible to within a few percent. The dataset is rebuilt daily, so exact matches are not expected.
Downloaded papers.parquet from ref [11] and recomputed day-level rank gaps at the top-1/top-3 cutoffs (day-1 and day-7 votes), excluding purge days and days with missing snapshots, for each horizon cohort (v1 approximated by HF published_at); recomputed cohort counts and flags
Citing works dated on or before v1 (10.3% featured vs 15.6% controls at 6 m) are a dating artefact counted as planned and do not threaten the comparison
Ratio of means 3.61 -> 3.83 (+6.3%) at 6 m, +2.4% at 12 m, +0.8% at 24 m. The artefact is larger for controls, so counting it biases the contrast slightly toward the null. It does not threaten the direction of H1, but it adds to the horizon/cohort comparability issue at 6 m. The approximation uses means because per-paper data are not available.
Removed the Table 10 pre-v1 shares from the Table 2 mean counts and recomputed the featured/control ratio of means at each horizon
The analysis plan with hash (S1), deviations log (S2), analysis code and frozen dataset accompany the article, and every table and figure regenerates offline with a single command
Only the PDF was supplied: no code URL, LaTeX source, frozen dataset, S1 or S2. So the single-command reproduction, the full SHA-256 hash (only ce0c87691dac... is printed) and the timestamped post-lock records cannot be checked. Checking them needs the S1/S2 files and the code/data archive, ideally with a third-party timestamp (e.g. an OSF registration). The exposure-side inputs were partly reproduced from public data instead (check 18).
Inspected the supplied inputs (inputs/ directory and request manifest)
H1 adjusted 107.1/143.1/203.3% with CIs, ratio 3.03, Holm p<0.001, 15% log-scale reduction vs unadjusted 266.4%, E-values 3.56/3.43, 4.30/4.09, 5.52/5.14
All six CIs symmetric on log scale (|asymmetry| <= 0.0002); z = 33.9-51.2, so p<0.001 holds after Holm. Ratio 3.033 vs claimed 3.03. Reduction 14.6% (claimed 15%; 14.9% and 16.0% at 6/12 m). All six E-values reproduce to 2 dp (e.g. 24 m: 5.52/5.14). Internal arithmetic is correct given the paper treats a ratio of geometric means as a risk ratio (see check 04 for that choice).
Converted percentages to log ratios; checked CI symmetry on log scale; backed out SE, z, p; recomputed log-scale reduction and VanderWeele-Ding E-values RR+sqrt(RR(RR-1)) treating exp(beta) as RR (as the paper states)
Unadjusted stratum-FE differences 135.2/187.8/266.4% (Table 3) are consistent with unweighted geometric means 5.02 vs 1.56, 13.32 vs 3.98, 41.52 vs 10.61 (Table 2)
Unweighted log-GM ratios 1.169/1.208/1.364 vs Table 3 unadjusted log ratios 0.855/1.057/1.299 (gaps +0.313/+0.151/+0.066). Reconciling the 6 m gap would need controls in 1:1/1:2 sets to have mean log(1+cit) 7.05 higher than other controls (overall control mean 0.445), which is implausible; 3.39 at 12 m. Small drops for unresolved records (14 featured, 96 controls) cannot close this. Either Table 2 or the Table 3 unadjusted column (or the stated weighting) is wrong, or the unadjusted model is not what Sec 3.5.1 describes. Tolerance used: gap > 0.05 log points.
Algebra plus numpy simulation: with featured weight 1 and control weight 1/k, treated and control weights are equal in every stratum, so the stratum-FE WLS coefficient equals the 1/k-weighted raw contrast exactly (verified to 1e-9 on synthetic data of Table 1 size). The weighted and unweighted contrasts can differ only through the 970 sets with <3 controls (weight leverage 0.0445).
Sec 5.2: listing association comparable to [5] (median 24 vs 14 citations) and the 24-month medians here differ by a similar factor (Table 2)
[5] does report median 24 (95% CI 23-25) vs 14 (13-15) for papers shared by AK, so the citation is accurate. But 24/14 = 1.71, whereas the 24-month medians here are 43/11 = 3.91 (GM ratio 3.91; adjusted ratio 3.03). The factor here is 2.3 times larger, so it is not a similar factor. [5] abstract summarises its result as medians 2-3 times higher (its other curator: 31 vs a lower control median).
Downloaded arXiv 2401.13782v3 (ref [5]), confirmed the 24 vs 14 medians in its text, compared ratios
S10 E-value 5.52 (5.14 for CI limit) at 24 m, computed with exp(beta) of a log(1+cit) model as an approximate risk ratio, read as confounding having to be strong
Estimated SD of log(1+cit) is about 1.4 in both groups. With it, E = 3.49 (limit 3.32) at 24 m, 3.01/2.90 at 12 m and 2.59/2.53 at 6 m, versus the paper 5.52/5.14, 4.30/4.09, 3.56/3.43. Even an implausibly small SD = 1.0 gives 4.93 < 5.52. The paper computes its own formula correctly (check 01), but treating a ratio of geometric means of (cit+1) as a risk ratio overstates robustness to unmeasured confounding by about 2 E-value units at 24 m. SD is approximate (lognormal assumption), so the exact values are indicative.
Recomputed the E-value with the continuous-outcome conversion of VanderWeele & Ding (ref [49]): d = beta/SD, RR ~ exp(0.91 d), SD of log(1+cit) estimated from the Table 2 mean and GM under a lognormal model, with SD sensitivity 1.0-2.0
Sample flow in Table 1 and its consistency with Tables 3, 5 and 10
Every sum in Table 1 closes exactly (12,251; 11,020; 31,523; 43,774), and H1 featured+controls equal Table 3 Papers at all horizons. 6-month losses (14 featured, 96 controls) match those expected from 0.14%/0.19% unresolved plus cascading set drops (~15 and ~104). Minor unexplained differences: Table 5 H2b Papers (12,231/8,633/3,340) are 3/2/1 fewer than Table 1 all-featured counts (12,234/8,635/3,341). Listings cover 1.77% of the arXiv frame, consistent with a few percent.
Recomputed all sums and products; compared 6-month cohort losses with losses implied by the Table 10 unresolved shares (including controls dropped with their featured paper); compared H2 counts across tables
No H3 cell met the pre-registered feasibility rule (>=300 eligible days with a 1-2 vote boundary gap); maximum 102; rule evaluated from exposure data that had been inspected before the plan was written
Table 6 is internally consistent (gap1 <= gap1-2 <= eligible, nested across horizons) and its maximum is 102, as stated. Infeasibility is confirmed, but it was foreseeable: the 24-month cells have at most 172 calendar days of snapshots (95/99 eligible), so 300 days is impossible by construction; the 12- and 6-month cells would need 84-85% and 61-62% of eligible days to have a 1-2 vote gap, whereas the observed shares are 3.9-30.5%. Because the rule uses exposure data only and the plan was written after those data were inspected (Sec 3), the confirmatory H3 test had no realistic chance of being run when the plan was locked.
Calendar arithmetic from the snapshot start (12 Mar 2024) to each cohort cut-off + 30 days; required vs observed share of gap-1-2 days from Table 6; Table 6 internal consistency and nesting
Table 7 local-randomisation comparisons are on the one-vote days counted in Table 6 (Gap 1), for every cutoff-horizon cell
10 of 12 Table 6 cells have a Table 7 row; 9 match exactly. Day-7 top-1 at 6 months: Table 6 gives 8 one-vote days, Table 7 uses 6. Day-7 top-1 rows at 12 months (6 days in Table 6) and 24 months (2 days) are missing from Table 7 without explanation, although the plan says cells below threshold are to be reported descriptively. Table 8 days are all <= eligible days (consistent).
Cell-by-cell comparison of Table 7 Days with Table 6 Gap-1 counts; Table 8 Days vs Table 6 eligible days
Table 7 local-randomisation estimates with 95% CIs and two-sided sign-flip permutation p-values (10,000 draws)
No reported p is below its attainable minimum (e.g. day-1 top-1 24 m: n=5, minimum 0.0625, reported 0.065; day-7 top-1 6 m: n=6, minimum 0.031, reported 0.032, i.e. all days negative). In 19 of 22 rows the CI-implied p agrees with the permutation p. But in two rows the CI excludes 0 while p > 0.05 (day-1 top-1 12 m: -0.99 [-1.96, -0.02], p=0.070; day-1 top-1 24 m: -1.95 [-3.00, -0.90], p=0.065, normal-theory p 0.0003). The CIs appear to be large-sample normal intervals, which are not valid with 5-6 days. The CI method is not stated. This is a reporting inconsistency, not a change to the descriptive conclusion.
For each of 22 rows: back out SE from the CI, compute the normal-theory p, compare with the reported permutation p and with the minimum attainable two-sided sign-flip p (2/2^n)
Cohort cut-offs (24 m: v1 <= 31 Jul 2024; 12 m: <= 31 Jul 2025; 6 m: frame to 31 Jan 2026) give fully elapsed windows with a 3-month buffer before 10 Nov 2026; H2 listing days 107/373/505 follow from snapshots starting 12 Mar 2024
Latest usable citation date is 10 Aug 2026; latest admissible v1 dates are 10 Aug 2024 / 10 Aug 2025 / 10 Feb 2026, so all three stated cut-offs give fully elapsed windows (10 days of slack). Table 5 days 107/373/505 are within the 124/384/515 weekdays available (86%/97%/98%). Note: plan lock (05:14 UTC), reference date and paper date are all 10 Nov 2026, so all outcome retrieval and analysis took place on the day of locking.
Calendar arithmetic with datetime: latest admissible v1 per horizon; available listing days from 12 Mar 2024 to cut-off + 30 days