Hugging Face Daily Papers Listing and Later Citations of AI Preprints: A Pre-Registered Matched Cohort Study

Summary

Strengths

1. Well-chosen comparison group and outcome clock

   Sec 3.2-3.4

   Controls come from the same primary category and v1 week (Sec 3.2), which holds fixed the release moment and the competing supply of papers. Citations are counted over fixed windows anchored at v1, with a 3-month indexing buffer and horizon-specific cohorts (Sec 3.3). My calendar check (check 09) confirms that every stated cut-off gives a fully elapsed window before the 10 Nov 2026 reference date, with 10 days to spare. Covariates are measured before v1 (Sec 3.4), which avoids the post-treatment author-metric problem the paper correctly identifies in [5].

2. Disciplined, non-causal language throughout

   Title; Abstract; Sec 5.3-5.4; Sec 6

   The title is associational, and the abstract closes by saying the design "cannot separate an attention effect of the listing from selection". A scripted scan (check 16) found that every causal-vocabulary hit in the body is hedged, negated or refers to prior work. All 10 numbers in the abstract appear in Tables 3/5 or the results text. The requester specifically asked about causal language, and on this point the paper is exemplary.

3. Exposure data are real and reproducible

   Sec 3.1; Table 1; Table 6; Table 4 Panel B

   I rebuilt the exposure side from the public sources named in Sec 3.1 (check 14, check 18). The Daily Papers API's first content date is 2023-05-04, as stated. The community dataset [11] exists under CC BY 4.0 and starts on 12 Mar 2024. All 12 Table 6 feasibility cells reproduce within 6 eligible days and 2 gap-1 days. S7 N is 322 vs 324 reported. Author-participation flags are 61.5% vs 62.3%. Listed papers in frame number 12,187 vs 12,325.

4. Internally consistent sample flow and H1/H2 arithmetic

   Table 1; Table 3; Table 5; Table 9; Sec 4.1

   Every sum in Table 1 closes exactly (12,251; 11,020; 31,523; 43,774), and the H1 cohort counts equal the Table 3 Papers column (check 05). All Table 3 and Table 5 CIs are symmetric on the log scale, and the implied z-values support Holm $p<0.001$ (checks 01, 12). The ratio 3.03 and the 15% attenuation from adjustment reproduce. The Table 9 subgroup estimates pool back to the overall estimates (check 11). Small checks pass GRIM and SMD recomputation (check 13).

5. Infeasible confirmatory analysis reported rather than buried

   Sec 3.5.3; Sec 3.6; Tables 6-8; Table 10

   The paper reports that the preregistered RD failed its feasibility rule (Table 6, maximum 102 of 300 required days; confirmed in check 06). It labels the local-polynomial RD as exploratory (Table 8) and gives a descriptive local-randomisation analysis instead of quietly substituting a confirmatory estimate. Data-quality problems are tabulated openly (Table 10: unresolved records, pre-v1 citing works, missing affiliations).

6. Load-bearing literature cited accurately

   Sec 2; References

   All 49 references resolve except a blog post [1] that has no URL. Spot-checks of [5], [8], [9], [10] and [24] show that each supports the specific claim it is cited for (check 17). Related work fairly presents the evidence pointing both ways: observational promotion studies versus RCTs and self-selection in listing position.

Weaknesses

1. Table 3 unadjusted estimates are inconsistent with Table 2 under the stated weighting

   major · Table 2; Table 3; Sec 3.2; Sec 3.5.1; Sec 4.3

   Sec 3.2 gives featured papers weight 1 and controls weight $1/k$, so treated and control weights are equal within every stratum. With equal weights, a stratum-fixed-effects WLS coefficient equals the weighted raw contrast exactly (I verified this to $10^{-9}$ in simulation). The unweighted Table 2 contrast can then differ from it only through the 970 matched sets with fewer than 3 controls. Yet the unweighted log geometric-mean ratios from Table 2 are 1.169 / 1.208 / 1.364 (6/12/24 m), while the Table 3 unadjusted estimates (135.2%, 187.8%, 266.4%) are 0.855 / 1.057 / 1.299 on the log scale. That is a gap of 0.31 log points at 6 months. Closing it would require controls in small sets to have a mean $\log(1+\text{cit})$ about 7 units higher than all other controls, whose overall mean is 0.445 (check 02). So either a table is wrong, or the unadjusted model is not the one described in Sec 3.5.1, and this raises doubt about the adjusted column computed by the same pipeline. The direction and the large size of the association are not in question, since both tables show listed papers ahead. But the 15% 'attenuation from adjustment' and the exact headline magnitudes cannot be trusted until this is resolved. Fix: report the weighted group geometric means beside Table 2, state exactly which sample and weights each column uses, and release the code.

2. "No deviation from the plan occurred" overstates preregistration adherence

   major · Sec 1 (p.1-2); Sec 3 (p.3); Sec 3.5.3; Sec 3.6; Table 6; Table 7

   The same section (Sec 3.6) lists several departures from the plan. (i) A post-lock record "fixes the operational definitions" for matching weights after cohort restriction, per-paper cohort membership, the highly-cited indicator, tie averaging and all-zero strata in S1. (ii) The RD density test, balance checks and covariate-adjusted RD were not run. (iii) S5 was not computed because citing-author records were never retrieved. (iv) S7 was not estimated for H3. In addition, the day-7 top-1 rows at 12 and 24 m are missing from Table 7 even though cells below the feasibility threshold were to be reported descriptively (check 07). The plan was written after the exposure data had been inspected (Sec 3), and the H3 feasibility rule depends only on those data. The 24-month cells could never reach 300 days (at most 172 calendar days of snapshots), and the 6/12-month cells would have needed 61-85% close-vote days against 4-31% observed (check 06). So the confirmatory H3 had no realistic chance at lock time. Only a truncated hash is printed, S1/S2 were not supplied, and there is no third-party registry timestamp. Lock, reference date and paper date are all 10 Nov 2026 (check 09). The fair description is "a planned analysis with documented post-lock clarifications and several unrun components". Fix: replace the blanket statement with a deviations table, and deposit the plan with an independent timestamp (OSF/AsPredicted).

3. E-value treats a ratio of geometric means as a risk ratio

   major · Sec 3.5.5 (S10); Sec 4.3; Sec 5.3; Table 3

   S10 converts $e^{\beta}$ from a $\log(1+\text{cit})$ model into an E-value as if it were a risk ratio (Sec 3.5.5). Sec 4.3 and 5.3 then read 5.52 as evidence that confounding "would need" to be strong. The paper applies its own formula correctly (check 01). However, the continuous-outcome conversion recommended by the cited source [49], $RR\approx\exp(0.91\,d)$, gives about 3.5 (limit 3.3) at 24 m, with the SD of $\log(1+\text{cit})$ estimated at about 1.4 from Table 2 (check 04). Even an implausibly small SD of 1.0 gives 4.93. The adjustment set cannot capture paper quality or the curator's own judgement, and the measured covariates already move the estimate by 15%. So an E-value near 3.5 sets a materially lower bar for an unmeasured 'quality' confounder than the paper suggests. Fix: use the standardized-difference conversion, or dichotomise the outcome (e.g. the S2 top-decile indicator) and report the E-value for that risk ratio.

4. Comparison with prior work misstates the effect size

   major · Sec 5.2 para 1; Table 2; Ref [5]

   Sec 5.2 says the 24-month medians here (43 vs 11, Table 2) "differ by a similar factor" to the 24 vs 14 reported in [5]. [5] does report 24 vs 14 (check 03), but 24/14 = 1.71 while 43/11 = 3.91, and even the adjusted ratio is 3.03. The association found here is roughly twice as large as the closest prior estimate. This is substantive, because the paper uses the agreement to place its result in context. A much larger association than in [5], despite tighter controls, is itself a finding that needs explanation (e.g. the stronger selection implied by author self-submission, or different citation sources). Fix: correct the sentence and discuss the gap.

5. Horizon trend in the abstract and intro mixes horizon with release cohort

   minor · Abstract; Sec 1 (p.2); Sec 4.3; Fig 3b; Table 9

   The intro (p.2) and Sec 4.3 say the difference "was smaller at shorter windows", and the abstract pairs 107.1% (6 m) with 203.3% (24 m). Those estimates come from different release cohorts. Pooling Table 9 (check 11), the 24-month cohort itself shows 160.3% at 6 m and 210.5% at 12 m. Later releases (2024H2-2026H1) show only about 92% at 6 m. Only about 45% of the 6-to-24-month log gap is a within-cohort change over time; the rest reflects later releases having smaller differences. Sec 5.4 acknowledges this, but the abstract does not. Fix: report the within-24-month-cohort trajectory (Fig 3b) as the horizon comparison in the abstract, and present the pooled 6-month estimate as a different population.

6. H2 headline omits its fragility to purged votes

   minor · Abstract; Sec 4.4-4.5; Table 4 Panel B; Table 5

   The abstract reports an elasticity of 0.623 at 24 m. Table 4 Panel B shows that S7, which excludes papers with purged votes, gives 0.333 [0.020, 0.647]. S7 keeps only 324 of the 1,255 papers, so 74% of the 24-month H2 sample had purged votes (check 12). The difference from the primary estimate is not significant (rough $z=1.67$), but the text calls the other versions similar and does not discuss S7. Given that purging indicates vote manipulation or cleaning, the primary estimate may partly reflect manipulated upvotes. Fix: mention the S7 estimate next to the headline and discuss why most of the early cohort was affected.

7. Table 7 day counts and CI/p reporting are inconsistent

   minor · Table 6; Table 7; Table 8; Sec 4.6; Abstract

   Day-7 top-1 at 6 m uses 6 days in Table 7 but Table 6 counts 8 one-vote days. The public data also give 8 (checks 07, 18). Two cells (12 and 24 m) are omitted without comment. In two rows the 95% CI excludes 0 while the permutation p exceeds 0.05: day-1 top-1 12 m, $-0.99$ [$-1.96$, $-0.02$], $p=0.070$; and 24 m, $-1.95$ [$-3.00$, $-0.90$], $p=0.065$ on 5 days (check 08). The CIs appear to be large-sample normal intervals, which are invalid with 5-6 days, and the CI method is not stated. Separately, "no consistent rank advantage" (Abstract) is literally true, but every top-1 contrast in Tables 7 and 8 is negative (e.g. Table 8 top-1 24 m: $-2.23$ [$-3.31$, $-1.16$]). That pattern deserves mention.

8. S3 shows most listings are author-submitted; its precision is unexplained

   minor · Table 4 Panel A (S3); Sec 4.4; Sec 1

   S3 flags 7,638 featured papers (62% of eligible listings) as author-submitted or author-participating. That sits uneasily with the "curated feed" framing (Sec 1), because most listings are self-selected, much as [9,10] found for arXiv position. At least 6,407 of 11,020 matched featured papers are flagged, so the SE should grow by at least 1.55x. Observed SE ratios are 1.38/1.32/1.19 (check 10, inconclusive). The S3 sample sizes are not reported. Fix: report S3 Ns, and give results separately for curator-only and author-submitted listings, since this is the most informative contrast for the selection-versus-visibility question.

9. Data provenance not pinned; code and supplements not supplied

   minor · Sec 3.1; Data and code availability; Refs [1], [11]

   The upvote dataset [11] was created on 2026-10-04 and is rebuilt daily (last modified 2026-11-11). It is derived from another CC0 dataset and has two collection gaps (check 14). The reference gives no revision or download date, so the exposure data cannot be pinned, and my reproduction differs by a few percent (check 18). The Data availability section promises code, a frozen dataset, S1 and S2, but none were provided, so the outcome side (Semantic Scholar/OpenAlex citations, covariates) could not be examined (check 20). Ref [1], the only source for how the page is curated and ordered, has no URL.

10. Smaller framing issues

    minor · Sec 7; Title; Table 9; Table 10; Table 5; Table 4 (S2)

    (a) The plain-language summary sets 11,020 vs 31,523 papers beside 2-year medians that come from the 2,981/8,420 subcohort (Sec 7). (b) The title says "AI preprints", but the strata include non-AI "Other categories" (Table 9). (c) 10.3% (featured) and 15.6% (control) of 6-month citing works are dated on or before v1 (Table 10). This biases the 6-month ratio toward the null by about 6% (check 19) and deserves a sentence. (d) Table 5 H2b counts are 1-3 papers below the Table 1 featured counts without explanation (check 05). (e) The reference population for the S2 top-decile percentile is not stated.

Questions for the authors

1. Exactly what sample and weights produce the Table 3 'Unadjusted' column, and can you reproduce Table 2's geometric-mean ratio (e.g. $\log(5.02/1.56)=1.169$ at 6 m) from the same model? If Table 3 is corrected, how do the adjusted estimates change?

2. Can you deposit S1 (full SHA-256 and plan text) and S2 with an independent timestamp, and list each post-lock operational definition next to the original plan wording, so readers can judge whether any of them is a deviation?

3. When the plan was written after inspecting the exposure data, was it already known that no H3 cell could reach 300 close-vote days? If so, why was H3 framed as confirmatory?

4. What is the H1 estimate separately for curator-listed and author-submitted papers? If the association is similar in both, what does that imply for the selection explanation?

5. Why does the H2 elasticity roughly halve when purged-vote papers are excluded, and why do 74% of 24-month listed papers have purged votes?

6. Why does Table 7 use 6 one-vote days for day-7 top-1 at 6 months when Table 6 (and the public data) give 8, and where are the 12- and 24-month day-7 top-1 rows?

Verification

References

49 references checked

• [5] Weissburg et al., Position: AI/ML Influencers Have a Place in the Academic Process (ICML 2024)

  Exists and does report median 24 vs 14 citations for papers shared by AK, but Sec 5.2 uses it to claim the 24-month medians here (43 vs 11) 'differ by a similar factor'; 24/14 = 1.71 vs 43/11 = 3.91 (work/verify/03_weissburg_ratio.py). The citation is accurate but the comparison drawn from it is not.

• [1] Adina Yakefu. Exploring the Daily Papers Page on Hugging Face, 2024.

  Grey-literature blog post with no URL; Crossref returns an unrelated item and a guessed huggingface.co/blog URL returns 404. It is the only source given for how the page is curated and ordered by upvotes (Sec 1). Needs a URL or archived link.

• [11] tardellirs. Paper Pulse: daily upvote histories for Hugging Face Daily Papers (dataset tardellirs/paper-pulse-data), 2026.

  Exists and is CC BY 4.0 as stated, but it was created on 2026-10-04 and is rebuilt daily (last modified 2026-11-11). It is itself derived from @hysts' daily-papers-stats (CC0) and has two collection gaps. The reference gives no commit/revision or download date, so the exposure data version is not pinned. Counts recomputed from the current revision match Tables 5-6 within a few percent (work/verify/18_exposure_reproduction.py).

• [2] Du, Tuan, Huang, Ng. Paper Espresso: From Paper Overload to Research Insight. arXiv preprint, 2026.

  Cited as an arXiv preprint; Crossref finds an ACM-published version (doi 10.1145/3799682.3840141). Minor bibliographic update. The claim attributed to it (right-skewed upvote distribution) was not spot-checked.

• [25] Feenberg, Ganguli, Gaule, Gruber. It's Good to Be First. Review of Economics and Statistics 99(1), 2017.

  Crossref's best match is the NBER working-paper version (2015, doi 10.3386/w21141). The journal version exists; no substantive issue.

Suggestions

• Reconcile Table 2 and Table 3: add weighted group geometric means, state the sample and weights for every column, and rerun the unadjusted and adjusted models from the released code.

• Replace "No deviation from the plan occurred" with an explicit deviations table (post-lock definitions, unrun RD diagnostics, S5, S7-for-H3, missing Table 7 rows), and deposit S1/S2 with an independent registry timestamp.

• Recompute S10 with the continuous-outcome E-value conversion of [49], or on a binary outcome, and temper the robustness language in Sec 4.3 and 5.3 accordingly.

• Correct the comparison with [5] in Sec 5.2 (1.7x vs 3.9x) and discuss why the association here is larger despite tighter controls.

• In the abstract, describe the horizon trend using the within-24-month-cohort trajectory, and mention the S7 sensitivity of H2 next to the 0.623 elasticity.

• Estimate H1 separately for curator-listed and author-submitted papers. This is the most direct available lever on the selection-versus-visibility question.

• State the CI method for Table 7 and use permutation-based (exact) intervals for cells with 5-8 days. Mention that all top-1 contrasts are negative.

• Pin the data: give the revision hash or download date for [11], the Semantic Scholar/OpenAlex snapshot dates, and a URL for [1]. Release the frozen dataset and code promised in the Data availability section.

Ethics and integrity

Scope of this review