The numbers are right: our independent rerun on the public files matches virtually every replicated value. But the pre-registration cannot be verified, an undisclosed choice between two public data files produces all five 'approximate' regression verdicts (which Section 6.2 mis-explains), and the new pre-registered hypotheses are close to implied by the known 0.91 outcome correlation.
Soundness
3/4
Presentation
3/4
Contribution
2/4
Confidence
4/5
✓ 14 confirmed✕ 4 discrepancy◐ 1 inconclusive
Summary
The paper is a computational check of Chetty, Hendren, Kline and Saez (2014, "Where is the Land of Opportunity?"). Using the authors' public commuting-zone (CZ) files, it recomputes the CZ-level descriptive statistics (Table 4), the values and rankings of the 50 largest CZs (Table 5, original Table III), the 35 x 6 standardised univariate correlate regressions of Online Appendix Table VIII (Tables 6, 8) and the five-factor model of Table VI (Table 7). Each comparison is graded against tolerances and a verdict rule that the paper says were fixed in a time-stamped plan before any outcome was computed (Section 3, Table 1). Of 479 graded quantities, 471 are reported as reproduced, 8 as approximately reproduced and none as failed.
As an extension, the paper estimates the same correlate regressions for the second headline outcome, the probability that a bottom-quintile child reaches the top quintile, P(Q5|Q1). It reports three pre-registered hypotheses as supported (19 of 19 signs agree; Spearman 0.985 between coefficient vectors; coefficients about 11% smaller in median absolute value) and documents where the two outcomes diverge (social capital in the five-factor model; much weaker associations under population weights, Table 9). The paper frames all relationships as cross-place associations.
Strengths
Every reported replicated number holds up under independent recomputation
Tables 4-9; Sections 5.1-5.7
We re-coded the authors' replicate.do estimators in Python and ran them on the public Dataverse files (doi:10.7910/DVN/NALG3E). The paper's replicated values match ours within 0.0005 in every graded case we checked (the only exception is the ungraded national AUM 41.485, which is within 0.008 of our closest rank convention; check 13): Table 4 scalars (SD 5.681, weighted SD 3.342, P90 52.016, urban/rural 41.713/45.833; checks 02, 05), all 150 Table 5 values and ranks (check 04), all 20 Table 6 coefficients and clustered SEs (check 06), the five-factor model (0.302, -0.009, -0.147, 0.169, -0.487, R2 = 0.757; check 08), all 125 new Table 9 cells (check 09) and the descriptive intervals in Table 4 (check 14). The Pub. columns transcribe CHKS Tables III, VI and VIII without error. We found no sign of fabricated or mistyped results.
Explicit tolerances, verdict rule and deviation log
Section 3.2, Table 1; Section 3.3, Table 2
Table 1 sets a tolerance for each type of quantity, tied to the printed precision (e.g. 0.0005 rounding bound, 0.005 band for coefficients), and Section 3.2 turns them into a four-part headline rule (a)-(d). Table 2 lists eight deviations with reasons. This is good practice for reproductions: readers can see exactly what 'reproduced' means and re-grade if they disagree.
Full side-by-side reporting, including unflattering cells
Tables 3, 5, 8; Section 5.7
Every Table VIII cell (Table 8), every Table III value (Table 5) and the count table (Table 3) are printed with published and replicated values next to each other. The tax-rate cells that fall outside the narrow band, and the significance reversals they produce in specs 2-3, are reported rather than hidden (Section 5.7). Table 3's bookkeeping is arithmetically exact (column sums 471/8/0/1/1/135/616; check 01).
Weaknesses
Undisclosed choice between two public data files drives every 'approximate' regression verdict, and Section 6.2 mis-explains it
major · Section 4.1 (p.4); Section 5.7 and Table 8 Local Tax Rate row (p.12); Section 6.2 (p.14); Table 2
The cited Dataverse deposit contains two different CZ workbooks named online_data_tables.xls: a top-level file (also distributed as onlinedata1-8.tab) and replicate_2014-1/replicate/online_data_tables.xls, the file that the authors' replicate.do actually reads. The two differ in taxrate for all 740 non-missing CZs (max 0.034) and in prob_p1_k5 by up to 0.68 pp (check 16). With the top-level file we reproduce the paper's tax-rate values (0.3189, 0.1394, 0.2017, 0.2170, 0.1865, -0.3232). With the replicate-folder file we obtain 0.3247, 0.1352, 0.1552, 0.1825, 0.2068, -0.3280, which match every published value within 0.0005, including the published non-significance in specs 2-3 (check 07).
So the five 'A' verdicts and the significance reversals reported in Section 5.7 are artefacts of file choice. Section 6.2's statement that 'the public material does not allow that explanation to be verified' is contradicted by the public material itself. The paper never says which file it used. The same choice moves St. Louis's P(Q5|Q1) to |d| = 0.0508 > 0.05 (check 04) and shifts the new Table 9 estimates by up to 0.0031 (check 09).
Fix: name the file (path and checksum), rerun with the file read by replicate.do, report both, and rewrite Section 6.2 and the Table 2 tax-rate note.
Pre-registration cannot be verified, so the 'pre-registered' claim in the title rests on self-attestation
Section 3.1 says the plan was locked at 10:08:58 UTC on 9 November 2026 with a SHA-256 hash. The digest is not printed anywhere (0 64-hex strings in the text). No external registry or third-party timestamp is named. The lock time falls on the date of the paper itself, and the analysis/ folder that Section 8 says contains the locked plan was not supplied (check 17). Adherence to the plan therefore cannot be audited.
The design limits what pre-registration can do in any case. For RQ1-RQ3 every target value was public, and had been transcribed before the lock, so the plan protects only against post-hoc tolerance choices. An AI system may also have prior exposure to the published tables. Internally, Table 1 already lists a graded 'Table VI ... N = 709' tolerance, yet Table 2 records 'Table VI and N added as graded rows' and 'the plan had no tolerance for counts' as deviations.
Questions for the authors
Which of the two online_data_tables.xls workbooks in the Dataverse deposit did the analysis read (path and SHA-256)? Rerunning with the replicate_2014-1/replicate/ file reproduces all six published tax-rate cells within 0.0005. Would you then grade them as reproduced and revise Section 6.2?
Can you supply the locked plan, print its SHA-256 digest, and point to any third-party time-stamp? Without these, adherence to the plan cannot be checked by readers.
Did the plan fix the percentile definition and the R2 definition used for the variance shares before the lock? The published 37.4 and 53%/36% are matched by a linear percentile and by adjusted R2 respectively.
How do H4a-H4c compare with the benchmark in which P(Q5|Q1) is AUM rescaled plus covariate-independent noise? Given corr = 0.91, what result would have rejected the hypotheses with reasonable probability?
Verification
We ran 19 checks. From the authors' public Dataverse deposit we independently recomputed nearly every number in the paper, re-coding replicate.do in Python, and compared the Pub. columns against NBER w19843 and the authors' own Stata log. We also scripted audits of the pre-registration, the provenance statements, the paper's language and the references.
15 checks confirmed the paper's numbers. Its replicated values match ours within 0.0005 throughout Tables 4-9, and its Table 3 bookkeeping is exact. Four checks found discrepancies:
the tax-rate 'approximate' verdicts come from an undisclosed file choice that the public material does explain (07);
the provenance statement and file identity are inaccurate or unspecified (16);
Section 9's 'matched to the last printed digit' is false as stated, and its causal sentence conflicts with Sections 2.1 and 6.4 (18);
two citations are misapplied (19).
The pre-registration audit (17) is inconclusive because the plan and its hash were not supplied. None of the discrepancies overturns a reproduced number. In fact, the file-choice finding shows that the original tables reproduce better than the paper reports, so the core conclusion survives and the score is not capped. The weaknesses concern disclosure, framing and the evidentiary value of the extension.
[16] Michael A. Clemens. The Meaning of Failed Replications: A Review and Proposal. Journal of Economic Surveys, 31(1):326-342, 2017.
Cited for a claim it does not support. Section 6.1 calls the study 'a computational reproduction in the sense of Clemens [16]: the same data and the same specification'. In Clemens's Table 1, 'reproduction' means the same specification and population but a different sample (resampling the same population). Re-running the same specification on the original data is a 'verification' test (same specification, population and sample). Section 2.3 ('replication test of the narrowest kind') is consistent with Clemens; Section 6.1 is not. Evidence: work/verify/19_refs.log.
[5] Raj Chetty, Nathaniel Hendren, Patrick Kline, and Emmanuel Saez. Replication Data for: Where is the Land of Opportunity? ... Harvard Dataverse, 2022.
Exists (DataCite doi:10.7910/DVN/NALG3E; submitted 2022-01-28, available 2022-02-23), but it does not support the text it is cited for. Section 4.1 says the data were 'released on 17 January 2014 together with the code that produced the published tables [5]'. The cited deposit is a 2022 deposit and documents no 2014 release date. It also contains two different online_data_tables.xls workbooks (top-level vs replicate_2014-1/replicate/), which differ in taxrate and prob_p1_k5. The citation gives no version, DOI or file, so readers cannot tell which data were used (checks 07, 16). The DOI is not printed in the reference list.
Suggestions
State the exact data file (path, version, checksum) and rerun the tax-rate cells and the Table 9 grid with the file read by replicate.do; report both files' results.
Deposit the pre-registration on a public registry with an independent time-stamp, print the hash, and include the plan as supplementary material.
Break the 479 graded quantities down into looked-up values, derived ranks and computed statistics in the abstract and Table 3; remove the three blank cells from the 'new estimates'.
Report a null benchmark for H4a-H4c (rescaled AUM plus independent noise) and the AUM-residualised P(Q5|Q1) associations, so the extension's information content is clear.
Mention the population-weighted weakening (7/20 significant, two sign changes) and the five-factor social-capital result in the abstract.
Replace 'matters a great deal' and 'matched to the last printed digit' in Section 9 with associational wording and an accurate description of the tolerances.
Explain the P10 and variance-share misses via percentile and adjusted-R2 definitions, and report the alternative values.
Use Clemens's 'verification' consistently and consider 'computational reproduction' or 'verification' in the title.
Correct the provenance date in Section 4.1 and give the DOI and version of reference [5].
Report Kish-effective-sample intervals for weighted statistics and state the rank convention used for national AUM.
Ethics and integrity
No text in the paper attempts to instruct reviewers. The byline (IterSCI) and the requester note identify the paper as produced end to end by an AI system. Two integrity-relevant points follow from this. First, the claim that the plan was locked before any outcome was computed is self-attested and cannot currently be verified (no hash value, no registry, plan not supplied). Second, a system with possible prior exposure to the published tables makes an independent time-stamp especially important. We found no sign of fabricated results: every replicated value we recomputed matched. The data are public, aggregate CZ statistics, so no consent issues arise.
Scope of this review
Only the PDF was supplied. The replication package described in Section 8 (code/run_all.py, data manifest with checksums, locked pre-registration, 616-row comparison table) was not available, so the paper's own code could not be rerun and adherence to the plan could not be audited. Instead we used the authors' public Harvard Dataverse deposit (doi:10.7910/DVN/NALG3E), including both CZ workbooks, replicate.do and its Stata log, and NBER WP 19843 for published values. We did not examine the Census 2010 cartographic boundary files or render the maps; map claims were checked only through legend breaks, missing counts and regional summaries. The confidential tax-record stage of the original study is outside the scope of both the paper and this review.
State-clustered CR1 standard errors with t(G−1) critical values (2.009 for 51 clusters), the areg degrees-of-freedom convention for the state fixed-effects column, and tie-block handling for rank comparisons are all stated (Sections 4.4-4.5) and reproduce exactly (checks 06, 04). The spec-2 SEs match the authors' Stata log only under the areg convention, which confirms that the paper implemented it correctly.
Informative divergences in the P(Q5|Q1) extension
Table 7 last column; Table 9; Sections 5.6-5.7
The new estimates add more than sign agreement. In the five-factor model the social capital association with P(Q5|Q1) vanishes (0.030 (0.063), versus 0.169 for AUM). Under population weights only 7 of 20 covariates remain significant, and two change sign (Table 9; check 12). Both findings are reproduced exactly (checks 11, 12) and are useful to researchers who use the bottom-to-top probability as an outcome.
Candid limitations and mostly careful causal framing
Abstract; Sections 6.1, 6.4, 7
The abstract ends with 'associations across places, not causal effects'. Section 6.4 and Section 7 state that the tax-record stage cannot be checked, that tolerances are judgements, that map comparisons are visual and that the descriptive intervals ignore spatial correlation. Section 6.1 says the study 'does not test whether the findings hold in other data'.
R2
R2
Fix: deposit the plan with a public time-stamp (OSF, AEA registry, Zenodo), print the digest, supply the plan, and reconcile Table 1 with Table 2.
The new pre-registered hypotheses are close to implied by the 0.91 outcome correlation
major · Abstract; Section 3.1 (H4a-H4c); Section 5.6; Section 6.3
Section 3.1 sets the H4b threshold (Spearman >= 0.90) because corr(AUM, P(Q5|Q1)) = 0.91. If P(Q5|Q1) were simply AUM rescaled plus noise unrelated to the covariates, the expected result would be identical signs (H4a), Spearman about 1 (H4b) and a coefficient ratio of about 0.91, i.e. 'about one-tenth smaller' (H4c) (check 10). We reproduce the paper's figures exactly (19/19 signs, Spearman 0.9850, median ratio 0.8938). However, after removing the part of P(Q5|Q1) explained by AUM, only 1 of 20 covariates (violent crime, 0.169 (0.058)) remains significantly associated.
The abstract's 'same correlate pattern ... coefficients about one-tenth smaller' is therefore largely a restatement of the known correlation, not independent evidence. The informative findings are the divergences (strength 5), and these are absent from the abstract.
Fix: report the null benchmark and residualised associations, and present H4a-c as weak tests.
The headline '471 of 479 reproduced' overstates the evidentiary weight
250 of the 479 graded rows (52.2%) are the 150 Table III values and 100 ranks that the Table 5 caption says are 'copied from the public file'. They test the release, not a computation (check 01). Fifty of these ranks (the P(Q5|Q1) ranks) are not printed in CHKS Table III at all; the replication derives them from rounded published values, so they are not 'published quantities' (check 04).
The three 'New' Table VIII cells counted among the 135 new estimates are the three dashes that Section 5.7 says have 'no estimate here either' (check 01). Only 229 graded rows involve a computation, and outside the tax rate (weakness 1) all of these reproduce within 0.0005.
The result is reassuring, but the count should be broken down by type in the abstract and Section 1.
The other 'approximate' verdicts have identifiable definitional causes that the paper does not investigate
minor · Table 4; Section 4.5; Section 6.2 (p.14)
P10: the replication's Stata-default value 37.347 does not round to the published 37.4. However, the 71st and 72nd sorted values are 37.347 and 37.422, and other common percentile definitions give 37.407 (linear) or 37.422 (check 02). The miss is a convention difference, not the rounding artefact that Section 6.2 describes ('visible only because the narrow tolerance is set at half a printed unit').
Variance shares: 1 minus adjusted R2 gives 52.740% and 36.473%, which round exactly to the published 53% and 36%. The replication's plain R2 gives 52.144% and 33.897% (check 03).
Thus all eight approximate verdicts have documentable causes, which would strengthen the paper's conclusion if reported.
Plain-language summary contains a causal reading and a factual overstatement
minor · Section 9 (pp.15-16)
Section 9 says the 2014 study asked 'whether the place where a poor child grows up matters' and 'found that it matters a great deal'. This is a causal reading of results that Section 2.1 and Section 6.4 describe as descriptive. The summary's own closing caution ('patterns, not proof') partly offsets it.
Section 9 also says '471 matched to the last printed digit'. The coefficient tolerance is 'ten times the rounding bound' (Section 3.2), and at least one R-graded cell differs by 5 units in the last digit (Local Tax Rate, spec 6: -0.328 vs -0.323; check 18). 'The maps looked the same' rests on visual inspection only (Section 7).
Fix: rephrase to associational language and state the tolerances accurately.
Abstract omits the robustness weakening and slightly overstates the map description
The abstract reports that P(Q5|Q1) has 'the same correlate pattern'. It does not mention that under population weights only 7 of 20 associations remain significant with two sign changes, or that social capital loses its conditional association in the five-factor model (Tables 7, 9; checks 11, 12).
The abstract lists 'the industrial Midwest' among low-mobility areas. The East North Central division has mean AUM 42.57 against 43.94 overall: 61% of its CZs are below the median but only 4% are in the bottom decile, versus 32% for the Southeast (check 18). The region is below average rather than 'low' in the sense the Southeast is.
Section 1 (p.1) also gives '4.4% in Charlotte to 12.9% in San Jose' without the 'among the 50 largest' qualifier; across all CZs the paper's own range is 2.2-47.0% (Section 5.3).
Data provenance statement is inaccurate and the cited deposit is not pinned
minor · Section 4.1 (p.4); Section 8 (p.15); Reference [5]
Section 4.1 says the inputs are the online data tables 'released on 17 January 2014 together with the code that produced the published tables [5]'. Reference [5] is a Harvard Dataverse deposit submitted 2022-01-28 and published 2022-02-23 (version 1.0). Nothing in it documents a January 2014 release, and the reference list itself dates [5] to 2022 (checks 16, 19).
The reference gives no DOI, version or file. The manifest with source addresses and checksums described in Section 8 was not supplied. The parts we could verify are consistent: 3,143 counties map to 741 CZs, identical across both CZ files.
Clemens's taxonomy is misapplied
minor · Title; Section 2.3; Section 6.1 (p.14); Reference [16]
Section 6.1 calls the study 'a computational reproduction in the sense of Clemens [16]: the same data and the same specification'. In Clemens's Table 1, 'reproduction' means the same specification and population but a new sample. Same data with the same specification is a verification test (check 19). Section 2.3's 'replication test of the narrowest kind' is consistent with Clemens.
The title's 'Replication' combined with Section 6.1's 'reproduction' may lead title-only readers to expect new data. Section 6.1's claim that the specification 'can be recovered ... without ambiguity' also sits uneasily with weakness 1.
Sample inconsistencies between the P(Q5|Q1) analyses
The Table 7 note says the P(Q5|Q1) sample is 'those with a non-missing value, also 709'. However, 729 CZs have a non-missing P(Q5|Q1); the 709 arises from the n250 restriction (check 11). The univariate P(Q5|Q1) estimates in Table 6 use up to 729 CZs while the AUM estimates use 709, so the H4c ratios compare coefficients from different samples. Section 4.4 mentions an n250 sensitivity, but the headline ratio does not use it.
N also varies by covariate (705 for test scores, 702 for crime; check 09), while captions say only 'up to 729'.
Some descriptive uncertainty and national figures are under-specified
minor · Table 4 (p.6); Section 4.5
The population-weighted SD interval [3.177, 3.526] treats the 709 CZs as equally informative. With the Kish effective sample size (86.2) the interval is [2.907, 3.932], about three times wider (check 14).
The national AUM 41.485 is not reproduced exactly by any of six rank conventions we tried (41.323-41.993; closest |d| = 0.008), and the convention is not stated (check 13). This is ungraded (F).
Table 4 prints the published correlations as 0.910 and 0.920, although the original reports 0.91 and 0.92 (check 05).
Small textual errors
minor · Section 2.1 (p.2); Section 6.3 (p.14)
Section 2.1 uses 'jointly' twice in one sentence ('Jointly, one representative variable ... explains a large share of the variance jointly'). Section 6.3's 'about a fifth to a quarter weaker' covers test scores (23%), dropout (20%) and social capital (20%), but crime is 28% weaker. It also omits household income (33% weaker, n.s.) (check 18).
Why are the three blank Table VIII cells counted among the 135 new estimates, and why are the 50 P(Q5|Q1) ranks, which CHKS Table III does not print, counted as published quantities?
Which rank convention produces the national AUM of 41.485 from the public 100x100 matrix?
Pre-registered H4a (19 of 19 significant AUM covariates have same-sign P(Q5|Q1) coefficients), H4b (Spearman 0.985 >= 0.90) and H4c (median |ratio| 0.894, "about one-tenth smaller") all supported; 18 of 20 P coefficients significant, violent crime p = 0.077; group ratios as listed
Reproduced exactly: 19 of 20 published AUM coefficients significant (household income is not); H4a 19 of 19 same sign; H4b Spearman 0.9850; H4c median ratio 0.8938; ratios 0.961-0.976 for segregation and commuting, 1.013/1.007 for spending and pupil-teacher, 0.676 household income, 0.722 crime, 0.768 tests, 0.794 SCI, 0.799 dropout; 18 of 20 significant, violent crime p = 0.077, household income p = 0.561. Interpretive caveat from the same script: corr(AUM, P) = 0.9112, so if P(Q5|Q1) were just rescaled AUM plus noise unrelated to the covariates the expected outcome would be the same signs, Spearman 1.0 and ratio 0.911. After removing the part of P(Q5|Q1) explained by AUM, only 1 of 20 covariates (violent crime, 0.169 (0.058)) remains significantly associated. H4a/H4b are therefore close to guaranteed by the outcome correlation (Section 3.1 says the 0.90 threshold was chosen because of it) and give little independent information; "about one-tenth smaller" in the abstract matches the attenuation expected from r = 0.91 alone.
Recomputed the 20 P(Q5|Q1) baseline coefficients (check 09), used the published AUM coefficients/SEs from the authors log, t(50) critical value, Spearman, ratios and p-values; added a benchmark of what the statistics would be if P(Q5|Q1) were AUM rescaled plus noise unrelated to covariates, and regressed the AUM-residualised P(Q5|Q1) on each covariate
P(Q5|Q1) five-factor model: 0.344 (0.061), -0.044 (0.060), -0.114 (0.068), 0.030 (0.063), -0.449 (0.052), R2 = 0.598, N = 709; social capital not significant; dropout CI includes zero
Online Data Tables release: 0.3441 (0.0611), -0.0438 (0.0600), -0.1137 (0.0683) CI [-0.251, 0.023], 0.0301 (0.0628) CI [-0.096, 0.156], -0.4491 (0.0523); R2 = 0.5980; N = 709; max |ours - paper| = 0.0003. The replicate-folder release gives values up to 0.0035 away (R2 0.5987). Minor: the Table 7 note says the P(Q5|Q1) sample is "those with a non-missing value, also 709", but 729 CZs have a non-missing P(Q5|Q1); the 709 arises from the n250 restriction, so this model uses a different sample from the 729-CZ univariate estimates in Table 6, which the text does not acknowledge.
Same re-coded replicate.do five-factor estimator as check 08 with prob_p1_k5 x 100 as outcome on the n250 sample; t(50) CIs; both public releases
Table 9 robustness: significant P(Q5|Q1) coefficients among 20 main covariates: n250 18 (no sign change), state FE 18, controls 16, urban 14 (pupil-teacher changes sign, not significant), population-weighted 7 with sign changes for top 1% (0.192, p = 0.056) and pupil-teacher (0.173, p = 0.19)
Both routes give identical counts: base 18, n250 18, state FE 18, pop-weighted 7, urban 14, controls 16. Sign changes: urban only pupil-teacher (0.070, p = 0.552); pop-weighted top 1% share (0.192, p = 0.056) and pupil-teacher (0.173, p = 0.192); none for n250, FE or controls. All statements in Section 5.7 hold.
Counted two-sided 5% significance with t(G-1) from our recomputed Table 9 grid (check 09, online release) and, separately, from the coefficients/SEs printed in Table 9 with t(50); sign changes relative to column (1) Base
National statistics from the public 100x100 transition matrix: P(Q5|Q1) 1980-82 = 7.542% (pub. 7.5), rank-rank slope 0.3406 (pub. 0.341), national AUM 41.485 (pub. ~41.5, F); 1980-85 national 7.8% not assessable
P(Q5|Q1) = 7.5420% exactly as printed (published 7.5). Rank-rank slope = 0.34053 with bin-index or bin-midpoint ranks (paper 0.3406, |diff| 0.00007 < 0.0005), 0.33954 if the merged zero-income child bins (2,3,5,6 absent from the file) get range-midpoint ranks. National AUM depends on the rank convention: 41.493 (midpoint ranks), 41.323-41.993 across six conventions, non-parametric parent-bin-25 mean 41.572; the paper 41.485 is close to but not identical with any convention we tried (closest |diff| 0.008), and the paper does not state the convention. Since the published comparator is only read off a figure (F), this does not affect a graded verdict. The 1980-85 national figure (7.8) is correctly marked X: the public matrix covers only the 1980-82 core sample.
Computed from onlinedata1.tab (Online Data Table 1, 96 child bins x 100 parent bins, columns sum to 1): quintile transition probability, weighted OLS of child rank on parent rank, fitted value at parent rank 25, under six rank conventions
Every printed SE and interval is reproduced to 3 decimals: SD 5.6808 (0.1510) [5.400, 5.993]; weighted SD 3.3421 (0.0888) [3.177, 3.526]; urban N = 325, 41.713 (0.217) [41.286, 42.139]; rural N = 384, 45.833 (0.318) [45.207, 46.459]; P90 order statistics 622/654 give [51.025, 52.803]; P10 56/88 give [36.776, 37.837]. Caveat: the weighted-SD interval treats the 709 CZs as equally informative; with population weights the Kish effective size is 86.2, giving SE 0.256 and CI [2.907, 3.932], so the printed interval understates its width by about 3x. All these intervals treat the 709 CZs as a random sample and ignore the sampling error in each CZ AUM estimate; the table caption says only that they are "descriptive sampling summaries".
Recomputed from the 709 n250 CZs: SE s/sqrt(2(n-1)) and chi-square CI for SDs, t(n-1) CIs for urban/rural means, binomial order-statistic CIs for P10/P90; added a Kish effective-sample-size version for the weighted SD
Section 5.3 descriptive statistics and map legends: AUM 26.7-64.0, mean 43.94, median 43.34, weighted mean 41.64, weighted P90 44.84; P(Q5|Q1) 2.2-47.0%, mean 10.04, median 8.95, weighted mean 8.10; decile breaks of Figures 3-4; 32 and 12 CZs without data; Great Plains high, Southeast low
AUM: N = 709 (32 missing), 26.672-64.019, mean 43.944, median 43.345, weighted mean 41.636, weighted P90 44.843, P90 52.017. P(Q5|Q1): N = 729 (12 missing), 2.210-46.970, mean 10.042, median 8.951, weighted mean 8.105. All 11 legend breaks of each map match after rounding to 0.1 (e.g. AUM 37.347, 39.180, ..., 52.016; P 4.807, 6.121, ..., 16.867). Top AUM decile CZs are mostly in NE 13, ND 12, KS 11, MN 9, SD 8, IA 7; bottom decile in GA 20, NC 12, MS 10, SC 7, consistent with the text. Note that the two maps use different samples (709 vs 729 CZs; the 20 extra P(Q5|Q1) CZs have fewer than 250 children), which the captions state.
Recomputed range, mean, median, population-weighted mean and P90, Stata-default decile breaks, missing counts and the states of top/bottom-decile CZs
Data provenance: inputs are the authors online data tables released on 17 January 2014 together with the code that produced the published tables (Harvard Dataverse [5]); 3,143 counties in the 2010 boundary file match the crosswalk after excluding 78 Puerto Rico polygons; 741 CZ polygons
Confirmed parts: the crosswalk county_2010.tab has 3,143 counties (51 state codes, no Puerto Rico rows) mapping to 741 distinct CZs, identical to the 741 CZ ids in both CZ files, consistent with the stated counts (the Census boundary file itself was not examined). Discrepancies: (i) the cited source [5] is a Dataverse deposit created 2022-01-28 and published 2022-02-23 (version 1.0); nothing in it documents a 17 January 2014 release, and the reference list itself dates [5] to 2022; (ii) the deposit contains two different CZ workbooks named online_data_tables.xls: a top-level one (2,887,168 bytes, also as 8 numbered copies and onlinedata1-8.tab) and replicate_2014-1/replicate/online_data_tables.xls (2,910,720 bytes), which is the file read by replicate.do, the code that produced the published tables. Checks 04, 07 and 09 show that the paper used the top-level file, whose taxrate (all 740 non-missing CZs, up to 0.034) and prob_p1_k5 (up to 0.68 pp) differ from the code-package file. The statement that the data were released "together with the code that produced the published tables" therefore hides a file choice that drives the only approximate regression verdicts, and the paper does not say which file (or checksum) it used. The analysis/ folder with checksums and the comparison CSV mentioned in Section 8 were not supplied for review.
Read the Dataverse API metadata of doi:10.7910/DVN/NALG3E (deposit dates, file list, file sizes); counted counties and CZs in the authors 2010 county crosswalk; compared CZ identifiers across files; combined with the file-version comparison of checks 04 and 07
Pre-registration adherence and verifiability: plan locked 10:08:58 UTC 9 Nov 2026 with a SHA-256 hash; analyses follow the plan; deviations (Table 2) do not change pre-specified estimates.
Adherence cannot be tested. The locked plan, its hash and the analysis/ folder (Section 8) were not supplied. The paper prints no SHA-256 digest (0 64-hex strings) and names no external registry or third-party timestamp. The lock time (10:08:58 UTC, 9 Nov 2026) falls on the paper's own date, so the 'locked before any outcome statistic' claim rests on self-attestation only. What the script could verify: Table 3 is arithmetically consistent (column sums 471/8/0/1/1/135/616; graded 479, published 481, new 135, all as stated), and the verdict-rule conditions (a)-(d) follow from the values confirmed in checks 04, 05, 06 and 08. Internal inconsistencies: (i) Table 1 already specifies a graded Table VI R2 tolerance 'including N = 709', yet Table 2 lists 'Table VI R2 and N added as graded rows' and 'The plan had no tolerance for counts' as deviations. (ii) The Table III family grades 250 rows, but condition (a) refers to the 150 value rows; the remaining 100 are ranks, and 50 of these (P(Q5|Q1) ranks) are not printed in CHKS Table III (check 04), so they are derived quantities rather than published ones. (iii) Table 2 records an 'interpretation note on the local tax rate' but no data-file choice. The paper never says which of the two public online_data_tables.xls workbooks was used, and check 07 shows this choice alone produces the 5 'approximately reproduced' tax-rate verdicts.
Scripted text audit of the paper: lock and hash statements, registry mentions, Table 1 vs Table 2 consistency, Table 3 arithmetic, and coverage of the deviation list, cross-referenced with checks 04 and 07.
Abstract, title and plain-language summary match the results, and causal language is avoided ('associations across places, not causal effects').
Mostly faithful, with one factual overstatement. (1) Causal language: the abstract, Section 6.3 ('Characteristics associated with raising the average...') and Section 6.4 are careful, and Section 9 closes with 'patterns, not proof'. However, Section 9 opens by saying the 2014 study 'found that it [where a child grows up] matters a great deal', which gives a causal reading of a paper that Section 2 itself describes as descriptive. (2) Abstract numbers match the body: 479/471/8/none (Table 3 arithmetic verified in check 17); SD 5.68 vs 5.681; R2 = 0.757; 19 of 19 signs and rank correlation 0.985 (check 10); 'about one-tenth smaller' vs 'on median 11% smaller' (Section 1), against the ratio median 0.894 in check 10. (3) Discrepancy: Section 9 says '471 matched to the last printed digit'. Section 3.2, however, sets the coefficient tolerance at 0.005, 'ten times the rounding bound', and the script finds a Table 8 cell graded R that differs in the last printed digit by 5 units: Local Tax Rate, spec 6, Pub. -0.328 vs Repl. -0.323 (|d| = 0.005). Other R rows sit at tolerances equal to half a printed unit, so most do match to the printed digit, but the sentence is not true as stated. (4) Section 6.3's 'about a fifth to a quarter weaker' (test scores, dropout, social capital, crime): recomputed spec-1 ratios are 0.768, 0.799, 0.795 and 0.722, i.e. 23%, 20%, 20% and 28% weaker. Crime falls slightly outside 'a quarter', and household income p.c. (33% weaker, non-significant) is not mentioned. (5) Map claims: the Plains (West North Central, mean 49.76) and the Southeast (South Atlantic + East South Central, mean 38.72; 32% of CZs in the bottom decile) are clear. The 'industrial Midwest' (East North Central, mean 42.57 vs 43.94 overall; 61% below the median, but only 4% in the bottom decile) is below average rather than among the lowest. (6) Framing: the title says 'Replication', Section 1 says 'computational replication' and Section 6.1 'computational reproduction'. Readers of the title alone would not learn that only same-data reproduction was done. Section 6.1's 'specification can be recovered ... without ambiguity' sits uneasily with the undisclosed choice between two public data files (check 07). Section 5.2's 'differ from the printed rank' refers to P(Q5|Q1) ranks that CHKS Table III does not print (check 04).
Scripted scan of paper.txt for causal/directional vocabulary with context; abstract numbers matched to body values; Table 8 R-graded cells compared with the Section 9 'matched to the last printed digit' wording; Section 6.3 coefficient ratios recomputed from the Dataverse files; regional AUM means for the abstract's map claims; title vs framing terms.
References exist and are cited accurately (incl. [12] Opportunity Atlas as AER 116(1), 2026; [16] Clemens taxonomy used for 'replication test of the narrowest kind' / 'computational reproduction').
All 22 references exist. 18 resolve on Crossref with matching titles; year offsets for [11] (2019 online vs 2020), [15] (2023 vs 2024) and [16] (2015 online, 2017 print) are online-first dates. [5] is a DataCite DOI (404 on Crossref; DataCite confirms the title, submission 2022-01-28 and availability 2022-02-23). [6], [18], [20] and [21] are grey literature that Crossref matches poorly; there is no indication they are fabricated, and [20] matches a MetaROR review of the same handbook. [12] is confirmed as AER 116(1):1-51, 2026 (doi:10.1257/aer.20200108). Discrepancies: (i) Section 6.1 calls the study 'a computational reproduction in the sense of Clemens [16]: the same data and the same specification'. Clemens's Table 1 rows are Verification = same specification/population/sample 'Yes Yes Yes' and Reproduction = 'Yes Yes No'; his text defines reproduction as 'resampling precisely the same population'. In Clemens's terms the study is a verification test, so the citation is misapplied (Section 2.3's wording is fine). (ii) [5] is cited for data 'released on 17 January 2014' (Section 4.1), but the cited deposit dates from 2022 and documents no 2014 release (see check 16). Both issues are written to work/ref_issues.json.
tools/check_refs.py on all 22 references (log 19_check_refs_raw.log, output work/refs_checked.json). Direct lookups: Crossref for [12] and [16], DataCite for [5]. The Clemens working-paper text was retrieved and its definitions and Table 1 extracted (19_refs.py).
616 compared quantities = 481 with a published value (479 gradable: 471 reproduced + 8 approximately, 0 failed) + 135 new (132 for P(Q5|Q1)); 207 Table VIII coefficients with a published value; 98% within narrow tolerance
Table 3 arithmetic is internally consistent: column sums (471, 8, 0, 1, 1, 135, 616) equal the printed totals; 471/479 = 98.3% (claimed 98%); Table 8 has 202 R + 5 A + 3 New = 210 cells, so 207 have a published value. 132 new P(Q5|Q1) quantities = 125 Table 9 cells + 7 Table 7 rows. Caveats found by the same script: (i) the 3 Table VIII New cells are the three dashes (Fraction Black, spec 5; State EITC and tax progressivity, spec 2), which Section 5.7 says have no estimate, yet they are counted among the 135 new estimates; (ii) 250 of the 479 graded rows (52.2%) are the 150 Table III values and 100 ranks that the Table 5 caption says are copied from the public file, so only 229 graded rows involve any computation.
Recomputed Table 3 row and column sums; parsed all 210 verdict cells of Table 8 from the paper text; counted Table 9 cells (21x6-1=125) plus Table 7 P(Q5|Q1) rows (5 coefficients + R2 + N = 7); counted graded rows that are look-ups of released CZ values
Obtained 741 / 709 / 325; SD 5.6808 and weighted SD 3.3421; P10 37.3473 and P90 52.0165 (Stata default); urban 41.7126 (N=325), rural 45.8331 (N=384) - all equal to the replicated values within half a unit of the third printed decimal, and the published values within the paper own tolerances except P10. On P10: the 71st and 72nd sorted values are 37.347 and 37.422, so the published 37.4 is exactly what other common percentile conventions give (numpy linear 37.407, higher/nearest 37.422 - i.e. the first value of the second decile class). The A verdict therefore reflects a percentile-definition choice, not rounding; the paper explanation in Section 6.2 (that the miss is visible only because the narrow band is half a printed unit) does not identify this, and 37.347 does not round to 37.4.
Independent recomputation from Dataverse analysis.tab with Stata _pctile default and eight alternative percentile definitions
Variance of AUM within Census divisions 52.144% and within states 33.897% (published 53% and 36%), approximately reproduced
Replicated values reproduced exactly (52.144% / 33.897%, tolerance 0.005 pp). New finding: 1 minus the adjusted R2 gives 52.740% and 36.473%, which round to exactly the published 53% and 36%. The published shares are therefore almost certainly based on adjusted R2 (Stata areg/reg reports it alongside R2), so both approximate verdicts reflect a definitional difference the replication did not investigate. Section 6.2 instead treats them as rounding of a qualitative statement. Other variants: population-weighted 57.6% / 30.9%; 1980-85 cohorts 53.4% / 36.8%.
Recomputed 1-R2 of AUM on division and state dummies over the 709 n250 CZs; tried adjusted R2, population weights and the 1980-85 cohort measure
Table 5: 50 largest CZs; all 150 AUM, P(Q5|Q1) and slope values within narrow tolerance of Chetty et al. Table III (max differences 0.0497, 0.048, 0.0005); AUM ranks identical, Spearman 0.9997; 12 P(Q5|Q1) rank differences all within ties
Published (Pub.) columns transcribe CHKS Table III with 0 mismatches in 50 rows. Using onlinedata5.tab: max|dAUM| = 0.0497 (claimed 0.0497), max|dP| = 0.0477 (claimed 0.048), max|dslope| = 0.00049 (claimed 0.0005), 150/150 within the narrow tolerance (0.05/0.05/0.0005); paper Repl. values equal the file to printed precision; AUM ranks identical; Spearman 0.9997; 12 P ranks differ, all inside tie blocks. Caveats: (i) the Dataverse package contains two releases of the CZ file; with the replicate-folder workbook (analysis.tab, the one used by the authors own replicate.do log) prob_p1_k5 differs by up to 0.68 pp and St. Louis gives |dP| = 0.0508 > 0.05 and Manchester differs from the paper Repl. value by 0.0135, so the paper should say which release it used; (ii) CHKS Table III prints no P(Q5|Q1) rank column, so the 50 "published" P ranks graded in Table 3 are derived by the replication from rounded published values, not published quantities; (iii) the slope margin 0.00049 vs tolerance 0.0005 is at rounding level.
Parsed all 50 rows of paper Table 5 and of CHKS (NBER w19843) Table III; matched the 50 largest CZs by pop2000 in the authors public Dataverse files (doi:10.7910/DVN/NALG3E); recomputed differences, ranks, Spearman and tie blocks; ran once with onlinedata5/8.tab (top-level Online Data Tables) and once with analysis.tab (replicate/ workbook)
Correlation of AUM and P(Q5|Q1): 0.911 unweighted (0.006) [0.898, 0.923], 0.922 population-weighted (0.006) [0.910, 0.932]; published 0.91/0.92
Online Data Tables release: N = 709 (P(Q5|Q1) exists only for the n250 CZs), r = 0.9112 [0.898, 0.923], SE 0.0064; weighted r = 0.9221 [0.910, 0.932], SE 0.0056; all agree with the paper to 3 decimals. The replicate-folder release gives 0.9122 [0.899, 0.924] and 0.9223, still within the 0.005 tolerance. Minor: Table 4 prints the published values as 0.910 and 0.920, but the original reports two digits (0.91, 0.92), so the trailing zero suggests false precision.
Pearson and pop2000-weighted correlation of e_rank_b and prob_p1_k5 from the public Dataverse files, Fisher-z 95% CI and delta-method SE, for both public releases
Twenty main AUM univariate coefficients and state-clustered SEs (Table 6; App. Table VIII col. 1) reproduced, largest difference 0.0005; Spearman 1.0 over 35; same top-5; 202 of 207 Table VIII cells reproduced with largest difference 0.0048
All 20 Table 6 coefficients and SEs agree to the 3rd decimal (max |ours - paper| 0.0005 for coefficients and SEs, e.g. single-mother -0.7637 (0.0745) vs -0.764 (0.074)). The paper Pub. values equal the authors log in all 207 cells. Our 207 cells equal the paper Repl. values to within 0.001 (incl. the 5 tax-rate cells, see check 07); outside the tax rate, max |ours - authors log| = 0.0005 for both coefficients and SEs, identical under both public releases. Spearman over 35 = 1.0; top-5 by |coef| (single-mother -0.764, middle class 0.679, Gini-99 -0.647, social capital 0.641, teen LFP 0.631) identical. 19 of 20 significant with t(50) = 2.0086. The spec-2 SEs match the log only if absorbed state FE are counted in K (max SE diff 0.0005 vs 0.0053 otherwise), confirming the areg convention. The Section 5.7 figure 'largest absolute difference of 0.0048' among the 202 reproduced cells is the tax-rate spec-6 cell (-0.3232 vs -0.328; see check 07), which is graded R only because 0.0048 falls just inside the 0.005 band; all other cells differ by at most 0.0005.
Re-coded the replicate.do Table VIII loop (lines 414-505) in numpy (standardise on the regression sample, OLS, CR1 state clusters; areg for spec 2; aweights for spec 3; residualise on cs_race_bla and incgrowth0010 for spec 5; s_rank outcome for spec 6); compared all 35x6 cells with paper Pub./Repl. rows and with the beta/se matrices printed in the authors Stata log tables.smcl
Local tax rate is the only covariate not exactly reproduced (Repl. 0.319, 0.139, 0.202, 0.217, 0.187 vs Pub. 0.325, 0.135, 0.155, 0.182, 0.207); plausibly the variable was revised between the analysis file and the public release, which the public material does not allow to be verified
The Dataverse package contains two releases of the CZ covariates whose taxrate differs in 740 of 740 CZs (max |diff| 0.034, r = 0.967); all other covariates agree up to rounding. With the top-level Online Data Tables file we obtain 0.3189, 0.1394, 0.2017, 0.2170, 0.1865, -0.3232, i.e. the paper Repl. values (max diff 0.0005). With replicate_online_data_tables.xls (the file read by the authors replicate.do, whose log prints the published matrix) we obtain 0.3247, 0.1352, 0.1552, 0.1825, 0.2068, -0.3280, equal to every published value within 0.0005, with published-style non-significance in specs 2 and 3 (t = 1.84, 1.68 < 2.009). The approximate verdicts therefore come from the replication choosing a different public file, not from an unverifiable revision. The Section 6.2 statement that the public material does not allow the explanation to be verified is contradicted by the public material, and the five A verdicts (and the significance reversals in specs 2-3 reported in Section 5.7) are artefacts of file choice. The same file choice explains the spec-6 tax-rate cell (0.0048 off, graded R) and the P(Q5|Q1) deviations in check 04.
Recomputed the six Table VIII specifications for taxrate with each of the two CZ covariate files in the authors public Dataverse package (onlinedata8.tab / online_data_tables.xls and replicate_online_data_tables.xls, which replicate.do reads), and compared both with the paper and with the authors Stata log
Five-factor model (Table VI col. 1) reproduced: 0.302 (0.065), -0.009 (0.053), -0.147 (0.055), 0.169 (0.047), -0.487 (0.062), R2 = 0.757, N = 709, missing indicators for dropout (116 CZs) and social capital (16 CZs)
Both public releases give 0.3022 (0.0646), -0.0087 (0.0526), -0.1474 (0.0553), 0.1688 (0.0465), -0.4869 (0.0618), R2 = 0.7570, N = 709, 51 clusters, 116 CZs missing dropout and 16 missing the social capital index. Max |ours - paper| over coefficients and SEs = 0.0005 (rounding); paper Pub. column equals CHKS Table VI col. (1) (0.302 (0.065) ... R2 0.757, 709 obs).
Re-coded replicate.do lines 305-327 (standardise over n250, zero-fill + missing dummies, OLS with CR1 state clusters) in numpy; published values checked against CHKS w19843 Table VI col. (1)
New P(Q5|Q1) univariate coefficients and state-clustered SEs for the 20 main covariates on CZs with non-missing P(Q5|Q1) (N up to 729), e.g. single-mother -0.686 (0.079), test scores 0.451 (0.099), violent crime -0.274 (0.152); same for all Table 9 columns
With the Online Data Tables release (onlinedata5/8.tab) all 125 Table 9 cells are reproduced: max |coef diff| <= 0.0005 and max |SE diff| <= 0.0005 in every column, e.g. single-mother -0.6858 (0.0793, N = 729), test scores 0.4515 (0.0986, N = 705), violent crime -0.2743 (0.1518, N = 702). Table 6 last column equals Table 9 col. (1) for all 20 rows. With the replicate-folder release the cells move by up to 0.0031 (school spending, FE), so the new estimates depend slightly on which of the two public files is used; the paper does not state which. Note that N varies by covariate (705 for test scores, 702 for crime), which the captions only describe as "up to 729".
Applied the replicate.do Table VIII estimator (re-coded in numpy) with prob_p1_k5 x 100 as outcome on the 729 CZs with an observed value (and on n250), for all 21 rows x 6 columns of Table 9, using both public CZ releases