Summary
The paper re-analyses Saint-Maurice et al. (JAMA 2020), who reported that 8000 vs 4000 steps/day was associated with an all-cause mortality HR of 0.49 (95% CI 0.44-0.55) in NHANES 2003-2006. Because 2003-04 step counts were never released and the 2015 public mortality file has been withdrawn, the replication uses only NHANES 2005-06 (n = 2378, 484 deaths) linked to the 2019 public-use mortality file, with censoring at 31 Dec 2015 reconstructed by a documented rule. It fits survey-weighted Cox models with a 3-knot restricted cubic spline in median daily steps, following the original covariate models.
The main result is a Model 3 HR of 0.43 (0.35-0.52) for 8000 vs 4000 and 0.30 (0.19-0.46) for 12 000 vs 4000. Under a stated pre-registered verdict rule (same direction, dose-response, overlapping CIs), the paper calls the original 'replicated'. It also attributes the somewhat steeper replicated curve to sampling variation and to an association concentrated early in follow-up, rather than to analytic choices.
The requester asked us to check adherence to the pre-registration, data provenance, causal language, and whether the title and abstract match the results. We rebuilt the cohort and the main models from the public NHANES files ourselves.
Strengths
Main estimates and cohort reproduce from public data
Figure 1; Sec. 2.3-2.5, 2.9; Sec. 3.1-3.2; Table 1 Models 1-3
We rebuilt the cohort from DEMO_D, PAXRAW_D and the 2019 LMF without author code. The Figure 1 flow matched exactly through 2481; we got 2376 vs 2378 participants with at least one valid day, and 483 vs 484 deaths. Our weighted Cox fits gave Model 1 0.202 (paper 0.20), Model 2 0.350 (0.35) and Model 3 without diet 0.428 (0.43). All 27 descriptive statistics checked were within tolerance (checks 01-03). This is the core of a credible replication, and the paper describes its methods precisely enough to make that possible.
Honest account of why exact repetition is impossible
Sec. 1 para 3; Sec. 2.2, 2.5
The Introduction names three structural barriers: unreleased 2003-04 step counts, the withdrawn 2015 LMF, and leading-cause-only coding. We verified each one against NCHS files: PAXRAW_C has no PAXSTEP variable, the 2015 file for 2005-06 returns HTTP 404, and ucod_leading has codes 1/2/5 (check 10). The 48-month / MVU-median rule for reconstructing 2015 censoring is a sensible, documented workaround, and it reproduced the death counts to within 1 (check 02).
Agreement criteria stated in advance and reported in full
Sec. 2.8; Table 1; Discussion para 1
Sec. 2.8 separates the verdict criteria (C1, C2, C4) from descriptive closeness measures (C3, C5), and Table 1 reports all criteria for every published estimate. Every printed flag matches the printed numbers (17/17 C1 and C4, 11/17 C3, 14/17 C5; check 04). The Discussion openly says that '2 of the 3 primary-model point estimates fell outside the published CIs'.
Published comparison values are transcribed accurately
Table 1 Published column; Table 2; Sec. 2.4, 2.6, 2.9
All 17 published HR (CI) triples in Table 1, the published Model 1 and Model 2 values, and the Table 2 counts and rates appear verbatim in the original article (check 08). The design choices that the paper says match the original also check out against the original text, including median rather than mean steps, at least 1 valid day of at least 10 h, weighted 5/50/95th-percentile knots, and the covariate models (check 09).
Conservative handling of the validation gate and careful causal language
Sec. 2.7; Abstract Conclusions; Sec. 4, 4.1; Plain-Language Summary (p. 12)
The paper did not run the pooled 2003-06 analysis. Its stricter R^2 definition gave 0.779, below the 0.80 threshold, even though the alternative definition (0.805) would have passed (Sec. 2.7, deviation 2). That is a disclosed choice against the analyst's convenience. Causal language is careful throughout. The abstract's final sentence, the end of the Discussion and the Plain-Language Summary all state that the estimates describe an association, not the effect of walking more, and the Limitations name reverse causation explicitly.
Weaknesses
Pre-registration cannot be verified
major · Sec. 2.1; Sec. 2.7; Data Sharing Statement (p. 9); Abstract Design
The title-level framing ('pre-registered replication', 'locked before mortality data were opened') rests only on an internal claim. No registry entry exists: we searched OSF and ClinicalTrials.gov and found 0 hits, and the paper gives no identifier. The plan, the deviation log and the code are said to be in a 'replication package (experiment/code)', but none was supplied. The printed SHA-256 shows only 15 of 64 hex characters, so even a supplied plan could not be matched to it. The PDF was created 2 h 03 min after the stated lock time (check 11). As a result, adherence to the plan, the claim that mortality data were not read before locking, and the completeness of the three logged deviations cannot be audited, which is exactly what the requester asked about. Fix: deposit the plan with a timestamped public registration (e.g. OSF), give the full checksum, and release the code and deviation log. Until then, describe the study as 'following an internal analysis plan'.
Abstract reports the sensitivity range selectively; Conclusions not supported by Table 3
major · Abstract Results/Conclusions; Sec. 3.5; Sec. 4 para 2; Sec. 5; Supplement Table 3
The Abstract says 'Alternative analytic choices gave estimates of 0.42 to 0.44'. The 9 analytic-choice rows of Table 3 actually span 0.38-0.47: unweighted 0.47, at least 4 valid days 0.46, and without self-rated health/mobility 0.38 (check 12). Sec. 3.5 limits the 0.42-0.44 statement to the pre-registered rows, where it holds, but the Abstract generalises it. The Conclusions add that the sensitivity analyses point to sampling variation and early follow-up 'rather than to analytic choices'. Yet three analytic choices move log-HR by 0.07-0.12, more than the follow-up-matched analysis (0.44, a shift of +0.023), which is the most direct test of the timing explanation. Fix: report the full analytic-choice range in the Abstract and remove 'rather than to analytic choices'.
Ratio-of-HR CIs assume independence between overlapping samples
major · Table 1 C5 column and footnote; Sec. 3.3; Discussion para 1; Abstract Conclusions
The C5 ratio CIs 'treat the two estimates as independent'. However, NHANES 2005-06 is roughly half of the original 2003-06 cohort, and 484 of the 1165 original deaths are shared. Positive correlation narrows the CI of the ratio. With a correlation approximated as , the ratio CIs exclude 1 for Model 3 2000 vs 4000 (1.093, 1.001-1.193) and for age 65 and older (1.184, 1.014-1.382). The primary 8000 vs 4000 ratio becomes 0.878 (0.754-1.022) (check 05). The statement 'the 95% CI of every ratio of HRs included 1', and the abstract's 'compatible with sampling variation', therefore depend on an assumption that is anti-conservative toward agreement. Three printed C5 upper limits (Mexican American 1.50, cancer 1.43, cadence 2.21) are also narrower than any rounding-consistent value. account for the overlap, for example by comparing the 2005-06 estimate with the original estimate on its complement (2003-04), or bootstrap, or at least report the sensitivity of these CIs to .
Questions for the authors
Can you release the locked plan v1.0, its full SHA-256 checksum, the deviation log and the analysis code? Is there any third-party timestamp, such as a registry or a git commit on a public host, showing that the plan predates access to the mortality file?
How do the ratio-of-HR CIs change once the overlap between the 2005-06 sample and the original 2003-06 sample is accounted for? Do the 2000 vs 4000 and age 65+ ratios still include 1?
Were any Table 3 analyses, or the definitions of the C1-C5 criteria, added or changed after the primary result was seen beyond the three logged deviations? Why does the Abstract's 0.42-0.44 range exclude the unweighted, at-least-4-valid-days and reduced-covariate variants?
What is the operational definition of C2, and how was it evaluated for the primary verdict?
Were the CIs computed with z or with multipliers?
Verification
We ran 15 checks. We rebuilt the analytic cohort from the public NHANES 2005-06 files and the 2019 LMF ourselves, and refitted survey-weighted Cox models with our own code (no author code was available). The cohort flow, deaths, 27 descriptive statistics and the Model 1, Model 2 and Model 3 (without diet) HRs all reproduced within tight tolerances, so data provenance and the headline numbers are sound. Internal arithmetic in Tables 1-3 is consistent, and the published comparison values are transcribed exactly. Two checks found discrepancies in interpretation: the ratio-of-HR CIs rely on an independence assumption that does not hold for overlapping samples (check 05), and the Abstract's sensitivity range and the Conclusions' 'rather than to analytic choices' are not supported by Table 3 (check 12). A further check shows the C4 verdict criterion is lenient (check 15). The pre-registration could not be checked because no plan, checksum or registry entry exists (check 11).
ConfirmedAbstract; Sec. 3.2; Table 1 Models 1-3; Supplement Table 3
ConfirmedFigure 1; Sec. 2.3-2.5; Sec. 3.1; Table 3 S1
ConfirmedSec. 3.1; Sec. 2.9; Table 2; Sec. 3.3
ConfirmedKey Points; Abstract; Sec. 2.8; Sec. 3.2; Table 1
DiscrepancyTable 1 C5 column; Sec. 3.3; Discussion para 1; Abstract Conclusions
ConfirmedTables 1-3; Sec. 2.9; Sec. 3.4
ConfirmedTable 2; Sec. 3.1
ConfirmedTable 1 Published column; Table 2; Sec. 1; Sec. 3.1
ConfirmedSec. 2.3-2.6, 2.9
ConfirmedSec. 1 para 3; Sec. 2.2; Sec. 2.5
Not checkableSec. 2.1; Sec. 2.7; Data Sharing Statement (p. 9)
DiscrepancyAbstract; Sec. 3.5; Sec. 4 para 2; Sec. 5; Supplement Table 3
ConfirmedSec. 2.9
ConfirmedDiscussion para 2; Sec. 1 para 3; References [1]-[18]
ConfirmedSec. 2.8; Table 1 primary rows
References
18 references checked
- [18] Tarp J, et al. Accelerometer-measured physical activity and sedentary time in a cohort of US adults followed for up to 13 years. IJBNPA 17(1):39, 2020
Metadata correct (DOI 10.1186/s12966-020-00945-4; automated lookup failed, manual check passed). The study used NHANES 2003-2006 activity counts (n = 3542), not step counts, so the paper's 'the same NHANES accelerometry data' overstates the overlap. This matters because [18] is the main external support for the early-follow-up explanation.
- [10] National Center for Health Statistics. Codebook for the NCHS 2019 public-use linked mortality files. NCHS, 2022.
Grey literature with no Crossref record and no URL, although it is the sole source for the LMF variable definitions and the perturbation statement; exact title not verifiable (NCHS pages returned HTTP 403). Add URL or report number.
- [12] NCHS. NHANES 2005-2006 data documentation, codebook, and frequencies: Physical activity monitor (PAXRAW_D). 2008.
No URL. The document exists at wwwn.cdc.gov/Nchs/Data/Nhanes/Public/2005/DataFiles/PAXRAW_D.htm; add URL and access date.
Suggestions
Register the analysis plan publicly with a third-party timestamp, print the full SHA-256 checksum, and release the plan, deviation log and code. Otherwise, replace 'pre-registered' with 'following an internal analysis plan' in the title-level framing.
Rewrite the Key Points finding as: 'both primary estimates met the pre-specified verdict criteria (same direction, dose-response, overlapping CIs); the 8000 vs 4000 estimate lay outside the published CI'.
Report the full range of analytic-choice estimates in Table 3 (0.38-0.47) in the Abstract, and drop 'rather than to analytic choices' from the Conclusions.
Account for the sample overlap when comparing estimates. Contrast the 2005-06 estimate with an estimate from 2003-04 alone (if obtainable), or report the ratio CIs as a function of the between-estimate correlation.
Discuss the power of C4. Consider an equivalence-style criterion (e.g. ratio CI within pre-set bounds) or a prediction interval for the replication estimate given the original estimate.
Remove the non-converged age 40-49 and 'other race' estimates from Table 1 and from the 17/11/14 tallies.
Define C2 operationally and add it as a column in Table 1. Describe the cadence analysis in Methods, define 'paper knots', and state the CI multiplier (z or t15) and the Model 3 covariate category levels.
Label the early-follow-up explanation as exploratory in the Abstract and the Conclusions, and note that [18] analysed activity counts rather than steps.
Correct the Plain-Language Summary: under non-proportional hazards, 0.43 is an average over follow-up, not a ratio that holds 'at any time'.
Name a responsible human contact and state the extent of human oversight of the AI-generated analysis.
Ethics and integrity
No ethical concerns with the data: NHANES is public, de-identified and consented, and the paper's ethics statement is adequate. On integrity, the study's main methodological credential, pre-registration before outcome access, is asserted without any externally verifiable record (check 11). The paper says the whole pipeline from plan lock to PDF took about 2 hours. We found no sign of fabricated results: the main estimates were independently reproduced from public data (checks 01-03). No text in the paper or the requester's notes attempted to instruct the reviewer. The paper names no accountable human author.
Scope of this review
Only the PDF was supplied. No LaTeX source, code, analysis plan, checksum or deviation log was provided, so adherence to the pre-registration could not be audited, and Model 3 was re-fitted without the HEI-2005 diet score, using our own covariate coding (compared against the paper's S4). The original article's supplement (e.g. the age 50-64 estimates) was not examined. The NCHS perturbation statement could not be retrieved because CDC pages returned HTTP 403. Our correction of the ratio CIs for sample overlap uses an approximate correlation and is indicative only.