Review: Short Sleep, Depression and Their Association Among US Adults Before and After the COVID-19 Pandemic: A Pre-registered Repeated Cross-sectional Analysis of NHANES 2005–2023 · CheckPaper
Review by CheckPaper · Reviewed 10 Nov 2026
Short Sleep, Depression and Their Association Among US Adults Before and After the COVID-19 Pandemic: A Pre-registered Repeated Cross-sectional Analysis of NHANES 2005–2023
The numbers are right: every headline estimate reproduces exactly from public NHANES files. But the 'before and after the pandemic' framing, an unverifiable pre-registration, a fragile H1 and a mis-sourced mode-effect argument need fixing before the title and abstract match the evidence.
Soundness
3/4
Presentation
3/4
Contribution
2/4
Confidence
4/5
✓ 12 confirmed✕ 5 discrepancy○ 1 not checkable
Summary
The paper analyses eight public NHANES cycles (2005–2006 to August 2021–August 2023; 42,222 MEC-examined adults aged 20+) with design-based methods. It estimates per-cycle prevalences of short workday sleep and of depression (PHQ-9 ≥ 10). Short sleep is kept as two series because the sleep item changed in 2015. The paper also estimates adjusted prevalence ratios (PRs) of depression for short and long sleep. Three pre-registered primary hypotheses compare cycle L (2021–2023) with cycle P (2017–March 2020), with Holm control.
Main claims: age-adjusted depression was 12.6% in L, 4.1 pp (2.2, 6.0) above P, the highest of the eight cycles, and the gap was largest at ages 20–39 (H2). Age-adjusted short sleep was 3.2 pp lower (−5.2, −1.3), at 20.9%, mainly among men (H1). Short sleep was associated with depression in every cycle (adjusted PR 1.41 in P, 1.71 in L), but the ratio of PRs, 1.21 (0.89, 1.63), is too imprecise to show a change (H3). The paper says that the mode change of the depression screener, the fall in response and the cross-sectional design mean the contrasts describe populations rather than pandemic effects.
We re-analysed all 24 public NHANES files independently. Every per-cycle prevalence, every primary and secondary contrast, the PR models, the trend tests, the multiplicity adjustments and the sensitivity analyses reproduce to the printed precision. The weaknesses lie in framing and interpretation, not in arithmetic. The title, opening sentence, conclusion and lay summary call a cycle that was mostly fielded during the declared emergency 'after the pandemic' or 'post-pandemic'. A reader cannot verify the pre-registration. The cited evidence for the mode-effect caveat to H2 does not concern self-administration, and the H1 headline loses statistical significance under a pre-specified sensitivity analysis that the abstract does not mention.
Strengths
Fully reproducible estimates from public data
Tables 2–7, 9, 10; Sec. 3
From the 24 public XPT files and our own Taylor-linearisation code we reproduced all 48 depression and 48 short-sleep cells of Table 2 (max diff 0.00 pp), H1 −3.23 (−5.20, −1.26) and H2 4.08 (2.17, 5.99), every adjusted PR in Table 4 (max diff 0.010), the H3 ratio 1.207 (0.892, 1.635) p = 0.216, all six Table 6 trend rows, all 31 BH q values and all 10 Table 7 sensitivity contrasts (checks 01, 02, 04, 06–09, 15). For a fully automated analysis this level of numerical agreement is unusual and makes the paper's numbers trustworthy.
Correct handling of the 2015 sleep-item break
Sec. 2.3; Table 2; Table 6; Fig. 1
Series B (SLD010H, whole hours, D–H) and series A (SLD012, half-hours, I–L) are never joined. Short sleep is defined appropriately for each item (≤6 h vs <7 h), and trend tests are run separately (Table 6). This avoids the ~20 pp artefactual jump that a naive joined series would show, a common error in NHANES sleep trend papers.
Sound design-based inference throughout
Sec. 2.4–2.5; Eq. 1; Table 6
Strata/PSU linearisation, domain estimation for subgroups, Korn–Graubard intervals for crude proportions, direct age standardisation with the 2000 US standard (Eq. 1; weights 0.3966/0.3718/0.2316), and contrast df equal to the sum of cycle design df (D–L sums 134/119/79/55 confirmed from the DEMO files, check 09). Check 05 also shows that Table 2's age-adjusted values follow from the Table 9 age-specific values.
Pre-specified primary family, complete reporting of null and weakening results
Sec. 2.5; Sec. 3.5; Tables 3, 5, 7
H1–H3 are Holm-adjusted and the secondary families are BH-adjusted with stated family sizes (14/9/8, confirmed in check 08). The inconclusive H3 is reported as 'too imprecise to establish or exclude a change' rather than as no change. Sensitivity analyses that weaken H1 (S4) and H2 (S2, PHQ-8: 2.7 pp) are printed and discussed in the text.
Unusually candid deviation log
Sec. 2.6; Sec. 3.4
Sec. 2.6 discloses that the pre-registered dry run exposed approximate cycle-level prevalences before H1/H2 were tested, and that a PD variance bug was found and fixed. The fix can be checked: in a sleep-only model our SEs are 1.0631 (P) and 1.5662 (L) vs the stated 1.063 and 1.566, and the corrected L−P PD difference 4.02 (0.09, 7.95) p = 0.045 matches 4.0 (0.1, 7.9) p = 0.045 (check 10).
External validation and accurate literature anchors
Sec. 1; Sec. 2.5 (validation); Sec. 4.2–4.3
Weaknesses
'Before and after the pandemic' mislabels the comparison period
major · Title; Abstract (sentence 1); Sec. 1 (p. 1–2); Sec. 3.3; Sec. 4.1–4.2; Sec. 5; Plain-language summary
Cycle L (Aug 2021–Aug 2023) was fielded mostly during the pandemic emergency. 642 of 761 fieldwork days (84.4%) precede the end of the WHO PHEIC on 5 May 2023, and 85.2% precede the end of the US PHE (check 14). The label nonetheless appears on 14 lines, including the title, the abstract's first sentence ('Depression among US adults rose after the COVID-19 pandemic'), the conclusion ('the post-pandemic rise in depression') and the lay summary ('After the pandemic this link may have become somewhat stronger'). The abstract's last sentence and Secs. 4.3 and 5 then say that the contrasts are not effects of the pandemic. The title and opening sentence therefore carry a temporal and causal framing that the paper itself disowns, and they do not match the results. Fix: retitle to e.g. '… in 2017–March 2020 and August 2021–August 2023' or 'pre-pandemic vs during/late-pandemic', and replace 'rose after the pandemic' and 'post-pandemic rise' with period-neutral wording.
Pre-registration cannot be verified, and the dry run already exposed the primary estimands
major · Title; Sec. 1 (p. 2); Sec. 2.1; Sec. 2.6 (deviation 3); Sec. 4.4; Data and code availability; Plain-language summary
The protocol is said to have been frozen on 9 Nov 2026 at 13:26:43 UTC, with a SHA-256 hash recorded internally. No public registry ID, URL or third-party timestamp is given, and neither the protocol, the hash, the deviation log nor the code was supplied (Data and code availability says they are 'in the experiment directory'). The PDF was created 4 h 57 min after the stated freeze (check 10), so the whole analysis and manuscript fall in a window that no outside party witnessed. Deviation 3 also shows that approximate cycle-level prevalences, the very quantities compared in H1/H2, were seen before H1/H2 were tested. The NCHS validation computed cycle-L age-specific depression (16.6/10.8/8.7%), and its timing relative to the freeze is not given. 'Pre-registered' in the title, and the lay-summary claim 'We wrote down our questions and methods before looking at the answers', are therefore stronger than the evidence a reader can check. Fix: deposit the frozen protocol on a public time-stamped registry (OSF) or attach it with its hash; state when the validation ran; soften the title/lay wording, e.g. to 'protocol-specified'.
H2 interpretation: the cited mode-effect evidence is mismatched and the age argument is unsupported
major · Sec. 4.3 (paragraphs 2–3); ref. [36]; Abstract
Questions for the authors
Can you provide the frozen protocol v1.0 and a third-party timestamp (e.g. an OSF registration) that predates the analysis? Was the NCHS validation, which computed cycle-L depression by age, run before or after the freeze?
What evidence supports the claim that a CAPI→ACASI mode effect on PHQ-9 reporting would be uniform across age groups? Is there any NHANES or other evidence on self-administered vs interviewer PHQ-9 reporting by age?
Why were Table 2 and Table 6 computed on per-outcome samples rather than the analytic sample defined in Sec. 2.2, and was the SLD012 = 14.5 h response in cycle I deliberately excluded?
Does the H2 difference at ages 20–39 change when estimated with the L-cycle age-group oversampling in mind (e.g. checking weight stability or design effects in that domain)?
Would you accept retitling to name the actual periods (2017–March 2020 vs August 2021–August 2023), given that ~85% of L fieldwork preceded the end of the public-health emergency?
Verification
We ran 18 checks with scripts. Using only the public NHANES files (24 XPT files, cycles D–L), we rebuilt the full analysis with our own design-based code. All main numerical results reproduce to the printed precision: Table 2 prevalences, the H1/H2/H3 contrasts and Holm p values, the Table 4 PRs and Table 5/10 ratios, the Table 6 trends, the BH q values, the Table 7 sensitivity analyses, the power statement and the PD-variance correction (checks 01, 02, 04–09, 15, 16). No discrepancy affects a reported estimate. The discrepancies are about provenance and framing. 'Post-pandemic' labels a cycle that was ~85% within the emergency period (14). The mode-effect citation does not concern self-administration (13). An undocumented 14-h cap and per-outcome samples were used (03, 02, 09). Smaller items are a Table 1 column sum (17) and citation type errors (18). Adherence to the pre-registration could not be checked because the protocol was not supplied (10).
12Confirmed
5Discrepancy
1Not checkable
✓ConfirmedAbstract; Sec. 3.2-3.3; Table 2; Table 3; Table 9
[36] C Li, ES Ford, G Zhao, J Tsai, LS Balluz. A comparison of depression prevalence estimates measured by the Patient Health Questionnaire with two administration modes: CATI versus CAPI. Int J Public Health 57(1):225-233, 2012 (PMID 21523617)
Cited for a claim it does not support. Sec. 4.3 cites it for 'Self-administration and the absence of an interviewer in the room can increase the reporting of depressive symptoms'. The study compares PHQ-8 prevalence between two different surveys, BRFSS by telephone (CATI) and NHANES face to face (CAPI). Both modes are interviewer-administered and it has no self-administered (ACASI) arm. The quoted 9.16% vs 6.28% is correct but is a cross-survey CATI-vs-CAPI contrast, not evidence about the CAPI-to-ACASI change in NHANES 2021-2023 (verify/13_mode_reference.log).
[25] LJ Akinbami, TC Chen, O Davy, et al. National Health and Nutrition Examination Survey, 2017-March 2020 prepandemic file: Sample design, estimation, and analytic guidelines
Wrong publication type and series. It is cited as 'NCHS Data Brief 190', but the work is Vital and Health Statistics, Series 1, No. 190 (2022), pp. 1-36 (PMID 35593699) (verify/18_references.log).
[28] JD Parker, M Talih, DJ Malec, et al. National Center for Health Statistics data presentation standards for proportions
Wrong publication type and series. It is cited as 'NCHS Data Brief 175', but the work is Vital and Health Statistics, Series 2, No. 175 (2017), pp. 1-22 (PMID 30248016) (verify/18_references.log).
[10] B Yu, X Zhang, C Wang, et al. Trends in depression among adults in the United States, NHANES 2005-2016. J Affect Disord 263:609-620, 2020 (PMID 31744739)
Only partial support. It is cited (with [11]) for 'the series has risen since 2005' and for a positive linear trend in PHQ-9 >= 10 consistent with 2005-2016 analyses. The abstract reports increasing trends only for severe depression among adults (driven by ages 65+) and for moderate depression at ages 20-39; it does not report a trend for PHQ-9 >= 10 overall. [11] (He et al. 2026) does report an overall increasing trend for 2005-2023. Checked against the abstract only.
[22] National Center for Health Statistics. Plan and operations of the National Health and Nutrition Examination Survey, August 2021-August 2023. Vital Health Stat 1(66), 2024
Minor attribution issue. The report exists (PMID 38768042, Vital Health Stat 1 2024;(66):1-21) but has named authors (Terry AL, Chiappa MM, McAllister J, et al.), whereas it is cited with NCHS as a corporate author and without page numbers.
[17] J Liu, W Sun, Q Zhu, et al. Age-specific determinants of depression prevalence: Insights from pre- and post-COVID-19 pandemic analysis. Research Square preprint, 2025
Suggestions
Retitle and reword the abstract, conclusion and lay summary to name the periods compared (2017–March 2020 vs August 2021–August 2023) instead of 'before/after' or 'post-pandemic'.
Register the frozen protocol publicly (e.g. OSF with its SHA-256 hash) or attach it as a supplement, and state the timing of the dry run and the NCHS validation relative to the freeze.
Report the S4 (−1.7 pp, CI crossing 0) and S2 PHQ-8 (2.7 pp) results in the abstract next to H1 and H2.
Replace or supplement ref. [36] with evidence on interviewer-administered vs self-administered (ACASI/web) PHQ reporting, and either support or remove the age-uniform mode-effect argument.
Add a quantitative bias analysis (e.g. how large a mode effect or differential nonresponse would explain the 4.1 pp) to bound the H2 interpretation.
Describe the actual per-outcome samples used in Tables 2 and 6 and the 14-h upper bound on SLD012 in Sec. 2.2, and mention the age-group oversampling added in cycle L.
Release the code, estimate file, deviation log and imputation specification (S9) with the paper.
Correct references [25] and [28] (Vital and Health Statistics series reports, not Data Briefs) and the authorship of [22]; add an ethics statement noting NCHS ERB approval and consent for NHANES.
Ethics and integrity
No integrity red flags in the results: every estimate we recomputed matches the public data, and we found no text attempting to instruct the reviewer. Two accountability points. (1) The byline is 'IterSCI', an AI research system per the requester. No human author, affiliation, funding, competing-interest statement or person accountable for the work is named, and most venues require one. (2) The paper has no ethics statement. NHANES is de-identified public-use data collected under NCHS Ethics Review Board approval with informed consent, and this should be stated. The pre-registration claim is not independently verifiable (see weaknesses).
Scope of this review
Only the PDF was supplied: no LaTeX, code, frozen protocol, hash, deviation log or estimate file, although the paper says these accompany it. We therefore re-derived the analysis from the public NHANES files. Adherence to the pre-registration and the original PD-variance error could not be examined, and S9 (multiple imputation) could not be refitted because its model and seed are not reported. Some references were checked against abstracts only (e.g. the specific claims attributed to [14]). Because www.cdc.gov blocks scripted access, NCHS data-brief tables were taken from saved web extracts.
Every quoted literature figure we checked appears in its source: Zhai RR 1.31/1.42; NHIS 30.5%; BRFSS 33.2%; DB527 13.1% vs 8.3% and 16.6/10.8/8.7%; Fan 8.45%→12.20%; DB303 8.1/10.4/5.5%. Our raw recomputation of DB303 and DB527 (8.06/10.43/5.55; 16.56/10.85/8.73) supports the 'within 0.1 pp' validation claim (check 12). Design facts used in the Discussion are also correct: CAPI→ACASI, telephone household interview in L, 46.9% examined response in P, 15 design df in L (check 11).
In L the PHQ-9 moved from interviewer CAPI to self-administered ACASI (confirmed from DPQ_L documentation, check 11). This change coincides exactly with the H2 contrast. Sec. 4.3 supports the mode caveat with Li et al. [36], 9.16% by telephone vs 6.28% face-to-face. The numbers are quoted correctly, but that study compares PHQ-8 prevalence across two different surveys (BRFSS CATI vs NHANES CAPI). Both modes are interviewer-administered, so it says nothing about self-administration (check 13). The paper then argues that a 'uniform mode effect would … raise reported symptoms in all age groups by similar amounts' and uses the concentration at ages 20–39 to discount it. No evidence is offered for an age-uniform effect, and disclosure effects for sensitive items plausibly differ by age. Response also fell from 46.9% to 25.6–25.7%. The +4.1 pp difference is reproducible, but how much of it is real cannot be judged. Fix: cite evidence on interviewer vs self-administered PHQ/sensitive-item reporting; drop or support the age-uniformity argument; consider a bounding or quantitative bias analysis for mode and nonresponse; give the abstract's caveat equal weight with the 'highest since 2005–2006' claim.
H1 headline is not robust, and the abstract omits this
major · Abstract; Sec. 3.5; Table 7 (S2, S4); Sec. 4.1
The abstract reports short sleep 3.2 pp lower (−5.2, −1.3). Under the pre-specified S4 (adults at exactly 6.5 h not counted as short), the difference halves to −1.7 (−3.6, 0.3), p = 0.088. We reproduced this as −1.67 (−3.59, 0.26), p = 0.0876 (checks 02, 15). Sec. 4.1 notes that heaping at the cut-off and telephone administration of the sleep item in L may contribute. Because the 'opposite movements' finding is one of the paper's two headline messages, the abstract should state that about half of it depends on how 6.5 h reports are classified. Similarly, the PHQ-8 analysis (S2) reduces the H2 difference from 4.1 to 2.7 pp, which bears on the sleep-item overlap and also belongs in the abstract.
Code, protocol and estimate files not provided
minor · Data and code availability (p. 13); Sec. 2.5 (S9); Table 7 (S9)
The paper states that code, a download/checksum script, the frozen pre-registration, the deviation log and a machine-readable file of all estimates 'accompany this article', but none was supplied. Our raw re-analysis makes up for most of this for the main tables. S9 (multiple imputation) could not be refitted, however, because the imputation model, number of imputations and seed are not reported (check 07). Fix: release the materials with the paper and report the mice specification.
Analytic sample described in Sec. 2.2 is not the one used for Tables 2 and 6
minor · Sec. 2.2; Table 1; Table 2; Table 6; Table 8
Sec. 2.2/Table 1 define a single analytic sample (valid sleep and PHQ-9). Table 2 short-sleep prevalences and all Table 6 trend estimates, however, reproduce only when each outcome is computed on all examined adults with a valid value for that outcome. On the Table 1 sample, crude short sleep would be I 19.14, P 24.11, L 20.60 instead of the printed 19.6, 23.8, 20.7, and trend coefficients shift by up to 0.028 (checks 02, 09). A one-record mismatch in cycle I (Table 1 n 4,864 vs 4,865 under the stated rules) is explained by an undocumented exclusion of the single SLD012 = 14.5 h response, although Sec. 2.2 says 'No other exclusions were applied' (check 03). Neither choice changes conclusions, but the methods should describe what was actually done.
Design description of cycle L incomplete
minor · Sec. 2.2; Sec. 4.3
Sec. 4.3 says the L design 'no longer oversampled by race or income', which is correct. The DEMO_L documentation also states that 'person-level oversampling based on age group was added' (check 11). This is relevant because the H2 difference is concentrated in one age group, and the paper does not mention it.
Reference errors
minor · References [10], [22], [25], [28]; Sec. 4.3
[25] is cited as 'NCHS Data Brief 190' but is Vital Health Stat Series 1 No. 190 (2022). [28] is cited as 'NCHS Data Brief 175' but is Vital Health Stat Series 2 No. 175 (2017). [22] has named authors (Terry et al.) but is cited to NCHS. [10] supports 'the series has risen since 2005' only partially: its abstract reports increases only for severe depression and for moderate depression at 20–39 (check 18). L's examined response rate is printed as 25.6% by NCHS vs 25.7% in the paper, a rounding difference (check 11).
Small table inconsistencies
minor · Table 1 (cycle F, education); Table 8; Sec. 3.1
The Table 1 education column for cycle F sums to 99.7 (18.2 + 23.0 + 30.6 + 27.9). From raw data, 11 participants with missing education are kept in the denominator but not shown as a row (check 17). Table 1 (weighted, analytic sample) and Table 8 (unweighted, eligible sample) report PIR missingness on different bases (10.6/12.7% vs 14.0/13.1%). Both are labelled, but the text in Sec. 3.1 should say which is which.
Pre-registration: protocol v1.0 frozen 9 Nov 2026 13:26:43 UTC before any outcome estimate; H1-H3 and S1-S11 pre-specified; four deviations listed (incl. dry run exposing cycle-level prevalences; PD variance correction with SE 1.063 (P) and 1.566 (L)).
Adherence cannot be verified. The frozen protocol, its SHA-256 hash, the deviation log and the code are said to be 'in the experiment directory' but none was supplied (inputs/ holds only paper.pdf and request.json). The paper gives no public registry identifier or URL. The PDF CreationDate is 2026-11-09 18:23:52 UTC, 4 h 57 min 09 s after the stated freeze at 13:26:43 UTC, so all analysis, deviations and writing fall in that window and an independent timestamp for the freeze is not available. The parts that can be checked reproduce exactly. Deviation 4: in a sleep-only model the standardised PD equals the crude weighted difference (P 3.5061 = 3.5061, L 7.6071 = 7.6071 pp), and its SE is 1.0631 (P) and 1.5662 (L) vs the paper's 1.063 and 1.566 (tolerance 0.005). Corrected adjusted PDs: P 2.94 (0.66, 5.22) vs 2.9 (0.7, 5.2); L 6.95 (3.55, 10.36) vs 7.0 (3.5, 10.4); L-P 4.02 (0.09, 7.95) p = 0.0453 vs 4.0 (0.1, 7.9) p = 0.045. This supersedes the slightly narrower beta-only CI in check 04. The 2.2-2.4-fold inflation of the original error could not be reconstructed, because the faulty formula is not given.
pdfinfo CreationDate vs the stated freeze time; inventory of the submitted materials; text search for a registry identifier/URL; raw-data recomputation of deviation 4 (sleep-only-model PD = crude weighted difference, its SE) and of the corrected adjusted PDs with full linearisation of the marginally standardised PD.
Design facts used to interpret H2: PHQ-9 interviewer-administered (CAPI) in the MEC through P and self-administered by ACASI in L; sleep/household interview could be by telephone in L; examination response 46.9% (2017-March 2020) -> 25.7% (2021-2023); L without oversampling by race or income; L has 15 design df.
P_DPQ: 'asked at the MEC, by trained interviewers, using CAPI'. DPQ_L: 'asked at the MEC using the Audio Computer-Assisted Personal Interview (ACASI) system. In previous NHANES, these questions were administered as part of the face-to-face MEC Interview'. SLQ_L: 'CAPI ... in the participant's home or by telephone'. Examined unweighted response rate: P 46.9% (paper 46.9%). L: NCHS prints 25.6%, the paper 25.7%. Recomputed 8,860/22,660 x 65.6% = 25.65%, so the difference is rounding only (tolerance 0.1 pp). DEMO_L analytic notes confirm 'no person-level oversampling by race and Hispanic origin and income', but they also state 'person-level oversampling based on age group was added', which the paper does not mention. DEMO_L has 15 strata and 30 PSUs, giving design df 15 (paper 15). One wording point: NCHS labels the L screener mode as 'Audio Computer-Assisted Personal Interview (ACASI)'. The paper's 'audio computer-assisted self-interview' describes the same self-administered instrument.
Downloaded the NCHS documentation pages (P_DPQ, DPQ_L, SLQ_L, DEMO_L) and the unweighted response-rate tables for 2017-March 2020 and August 2021-August 2023, extracted the mode-of-administration and design sentences, and parsed the 'Total, All Ages' rows. Strata and PSUs counted in DEMO_L.xpt.
Literature anchors: Zhai et al. RR 1.31/1.42 (seven cohorts); NHIS 2024 30.5%; BRFSS 2020 33.2%; DB527 13.1% vs 8.3% (ages 12+) and 16.6/10.8/8.7% by adult age group; Fan et al. 8.45%/12.20%; DB303 8.1/10.4/5.5%; the estimator reproduced the six NCHS values within 0.1 pp.
All quoted numbers appear in the sources. Zhai: 'Seven prospective studies', RR 1.31 (1.04-1.64) short and 1.42 (1.04-1.92) long. DB559: '30.5% of adults had less than 7 hours'. Pankowska: '33.2% of adults reported short sleep'. Fan: 8.45 % -> 12.20 %, adults 20+. DB527: 13.1% (12+, L), 2017-March 2020 8.3 (0.5), and by age 16.6/10.8/8.7. DB303: 8.1%, women 10.4%, men 5.5%. Raw recomputation (tolerance 0.1 pp): DB303 crude 8.06/10.43/5.55% vs 8.1/10.4/5.5; DB527 L crude 16.56/10.85/8.73% vs 16.6/10.8/8.7. The 'within 0.1 pp' validation claim is therefore reproducible for the five adult values. The ages 12+ figures (13.1%, 8.3%) cannot be recomputed because youth PHQ-9 data are RDC-only, and the paper did not claim to. Note: Fan et al. label 2021-2023 as 'pandemic and post-pandemic'.
Retrieved the PubMed abstracts (NCBI E-utilities) of refs [1], [4], [5], [7], [8], [26] and searched them for the quoted strings. www.cdc.gov returns 403 to curl, so the DB527 and DB303 figure tables come from WebFetch extracts of the NCHS pages saved in work/verify/docs. Recomputed DB303 from raw cycles H+I (pooled, MEC weight/2) and the DB527 adult age groups from cycle L.
Mode-effect evidence: Li et al. 2012 [36] reported PHQ depression prevalence 9.16% by telephone interview vs 6.28% face-to-face, cited in support of 'self-administration and the absence of an interviewer ... can increase the reporting of depressive symptoms'.
The numbers are quoted correctly: 9.16% (SE 0.15) vs 6.28% (SE 0.59), exact match. The source does not support the sentence they are attached to. The instrument is the PHQ-8 diagnostic algorithm (with PHQ-8 >= 10: 9.22% vs 5.15%), not the PHQ-9 >= 10 definition used in the paper. The two 'modes' are two different surveys: BRFSS 2006 by CATI and NHANES 2005-2006 by CAPI. Both modes are interviewer-administered, so the study says nothing about self-administration or ACASI. The difference also confounds mode with survey (sampling frame, response, context). The abstract reports no age-specific mode effect, so it gives no empirical basis either way for the paper's argument that a mode effect would be age-invariant. This bears on the main interpretive caveat for H2 (Sec. 4.3). The caveat itself is reasonable; the cited quantitative evidence for it is mismatched.
Retrieved the PubMed abstract of ref. [36] (PMID 21523617) and tested the quoted numbers, the instrument, the modes compared and whether a self-administered mode was studied.
Cycle L (August 2021-August 2023) is labelled 'after the COVID-19 pandemic' / 'post-pandemic' in the title, abstract, body, conclusion and plain-language summary.
642 of 761 fieldwork days (84.4%) fell before the end of the WHO PHEIC, and 648/761 (85.2%) before the end of the US PHE. The 'post-pandemic' cycle was therefore collected mostly during the declared emergency. A per-person share cannot be computed because public DEMO_L gives only 6-month exam periods. The label occurs on 14 lines. These include the title ('Before and After the COVID-19 Pandemic'), the abstract's opening sentence ('Depression among US adults rose after the COVID-19 pandemic'), the conclusion ('the post-pandemic rise in depression') and the plain-language summary ('After the pandemic this link may have become somewhat stronger'). The paper also disclaims a pandemic effect three times: the abstract (L25), Sec. 4.3 (L669) and Sec. 5 (L720). So the framing is internally inconsistent. 'Post-pandemic rise' and 'after the pandemic' attach a period/causal label that the data and the authors' own caveats do not support; 'during and after' or 'August 2021-August 2023 vs 2017-March 2020' would be accurate. The plain-language phrase 'may have become somewhat stronger' rests on the H3 ratio 1.21 (0.89, 1.63), p = 0.216 (reproduced in check 04), and is more suggestive than the abstract's 'too imprecise to establish or exclude a change'. The plain-language summary's statement 'We wrote down our questions and methods before looking at the answers' sits uneasily with deviation 3 (Sec. 2.6: the dry run exposed approximate cycle-level prevalences before H1/H2 were tested).
Calendar arithmetic on the cycle-L fieldwork window against the end of the WHO PHEIC (2023-05-05) and of the US federal PHE (2023-05-11). Line-by-line search of paper.txt for the labels and for the disclaimers.
Sensitivity analyses (Table 7, Sec. 3.5): S1 18+ crude short -3.1 (-5.1, -1.2) and depression 4.1 (2.2, 6.0); S2 PHQ-8 2.7 (1.1, 4.3); S3 -3.3 (-5.5, -1.1); S4 -1.7 (-3.6, 0.3) p=0.088; S5 short -2.4 (-4.4, -0.4) and mean sleep 0.12 (0.06, 0.18) h; S6 -3.2 / 4.1; S7 4.1 (2.2, 6.0). Suspected transcription: S1 depression row identical to the primary age-adjusted row.
All 10 sensitivity contrasts reproduce within 0.15 pp (0.015 h) and p within 0.005: S1 short -3.15 (-5.05, -1.25) p=0.0018; S1 depression 4.10 (2.24, 5.95) p=0.0001; S2 2.69 (1.09, 4.29) p=0.0016; S3 -3.29 (-5.52, -1.06) p=0.0048; S4 -1.67 (-3.59, 0.26) p=0.0876; S5 short -2.42 (-4.43, -0.41) p=0.0193; S5 mean sleep 0.12 (0.06, 0.18) h; S6 -3.22 (-5.19, -1.25) and 4.10 (2.21, 6.00); S7 4.06 (2.16, 5.95). The suspected transcription error is refuted: the 18+ crude depression difference is genuinely 4.10 pp. It coincides with the primary age-adjusted value after rounding, whereas the 20+ crude difference is 3.75 (1.88, 5.62). So 'left both crude differences almost unchanged' (Sec. 3.5) is fair (3.75 -> 4.10 for depression). The PHQ-8 result (2.7 vs 4.1 pp) and the S4 result (-1.7, CI crossing 0) are reported accurately, and the text interprets both.
Raw recomputation from public P and L files. Each L-P contrast uses per-outcome samples, direct age standardisation (or crude for S1), Taylor linearisation and df 40. S5 uses interview weights on all interviewed adults. S6 uses a weighting-class adjustment of the MEC weight within sex x age-group x race5 cells. S7 prorates the PHQ-9 when >= 7 items are answered.
Power statement: with design effect ~2, MDE ~2 pp for depression and ~3 pp for short sleep (80% power, alpha 0.05); H3 detectable only for a PR change by a factor of ~1.6.
Deff-2 MDE: depression 2.18 pp (paper 'about 2'), short sleep 2.77 pp (paper 'about 3'). Realised MDE from reported CIs: H1 2.77 pp, H2 2.70 pp, H3 ratio-of-PRs factor 1.54 (paper 'about 1.6'). Note: realised design effects were heterogeneous - depression P 2.26, L 3.08; short sleep P 3.39, L 0.64 - so the deff-2 assumption was optimistic for depression in L (realised depression MDE with design SEs 2.59 pp), though the statement as written holds.
Prospective MDE = (z.975+z.80)sqrt(2sum p(1-p)/n) from crude prevalences and per-outcome n of cycles P and L (public files); realised design effects = design variance/SRS variance; realised MDEs from the reported primary-contrast CIs (t, 40 df). Tolerance: rounds to the stated 'about' value (0.5 pp; 0.1 on the PR factor).
Internal bookkeeping: Table 1 category percentages sum to 100; Table 8 percentages = counts / eligible n; Sec. 3.1 text values match Tables 1 and 8; Table 4 unadjusted PRs = ratios of crude prevalences; long sleep associated in 'five of the eight cycles'.
All checks pass except one minor item. Table 8: all 48 count/percent cells are consistent with the eligible n. Sec. 3.1 text: 22.4%->31.3%, 16.9%->8.4%, 7.7%/14.1% and 14.0%/13.1% all match the tables. Table 4: all 16 unadjusted PRs lie within the rounding interval of the crude-prevalence ratio (e.g. L <7 h: 16.9/9.3 = 1.817 vs printed 1.82). Five of eight is confirmed: the adjusted >=9 h PR CI excludes 1 in F, G, I, P and L. The one mismatch is the Table 1 education column for cycle F (2009-2010), which sums to 99.7 (18.2 + 23.0 + 30.6 + 27.9), outside the 0.20 rounding tolerance. In the public file, 11 analytic-sample participants with missing education are left in the denominator; this gives 18.21 / 23.02 / 30.63 / 27.95 (sum 99.81), while excluding them gives 18.24 / 23.07 / 30.69 / 28.00. So the column uses an inconsistent denominator (missing education counted in the denominator but not shown as a weighted row), and 27.95 is rounded down. This is cosmetic and minor. Note: Table 1 income 'Missing' (weighted, analytic sample: P 10.6%, L 12.7%) and Table 8 (unweighted, eligible: 14.0% / 13.1%) use different bases; both are labelled.
Parsed the tables from work/paper.txt and recomputed the arithmetic. Tolerances: Table 1 column sums within 0.05 pp per category; Table 8 count/n within 0.05 pp of the printed percentage; PR within the interval implied by rounding both crude % to 0.1 (+0.005). Raw-data recomputation of the cycle-F education distribution to explain the one mismatch.
Bibliography: all 36 references exist and are cited with the correct publication type and number (e.g. [25] Akinbami et al. listed as 'NCHS Data Brief 190', [28] Parker et al. listed as 'NCHS Data Brief 175').
Existence: 35 of 36 references were located. Crossref resolved 17. PubMed/Crossref located the rest: [4] PMID 42096866, [7] 40839857, [11] 42775219 (doi 10.1155/da/1800832), [13] 36272761, [16] 42965254, [17] Research Square doi 10.21203/rs.3.rs-6475145/v1, [18] 39876061, [19] 42732358, [20] 40596991, [21] 40597928, [22] 38768042, [25] 35593699, [26] 29638213, [28] 30248016, [29] 11676466, [30] 29775435, [36] 21523617. [23] is the 'Brief Overview and Analytic Guidance' document linked on the NHANES 2021-2023 overview page. [33] Holm (1979) has no DOI and is not in PubMed, so it was not verified by lookup. Type and number errors: [25] is cited as 'NCHS Data Brief 190' but is Vital Health Stat 1(190):1-36, 2022 (a series report, not a data brief). [28] is cited as 'NCHS Data Brief 175' but is Vital Health Stat 2(175):1-22, 2017. [22] is cited with NCHS as corporate author, whereas PubMed lists Terry AL, Chiappa MM, McAllister J et al. as authors (minor). Year differences flagged by Crossref for [14] (2022 vs cited 2023) and [15] (2018 vs 2019) are online-first dates against issue year; PubMed gives Sleep 2023;46(1) for [14], so these are not errors. Support: [36] does not support the self-administration claim (check 13). [10] (Yu et al. 2020) reports increasing trends only for severe depression in adults and moderate depression at ages 20-39, not for PHQ-9 >= 10 overall; it gives partial support for 'the series has risen since 2005' ([11] supports it directly). [9], [12], [13] and [19] support the statements they are cited for. For [14], the '≈20 percentage points' shift and the 'increase between 2015-2016 and 2017-2018 under a stricter cut-off' are not in the abstract, so they could not be checked from the abstract alone.
tools/check_refs.py (Crossref/arXiv by title; 17/36 resolved). Every unresolved item was re-queried in PubMed (E-utilities esearch/esummary; cached in verify/docs/refs_pubmed.json) and, where PubMed had no hit, in the Crossref bibliographic search. The indexed source, number and year were compared with the citation. Abstracts of load-bearing citations ([9]-[14], [19], plus [36] in check 13) were fetched and the cited statement compared with the abstract.
Depression (PHQ-9>=10) prevalence per cycle (Table 2: e.g. P 8.4 crude/8.5 adj, L 12.2/12.6 (10.9,14.4)) and H2 contrast L-P = 4.1 pp (2.2, 6.0), p<0.001; subgroup contrasts in Table 3
All 8 cycles: n(depression) identical (4,284 ... 5,210); all 48 crude/age-adjusted point estimates and CI limits identical after rounding to 0.1 pp (max diff 0.00). H2: obtained 4.08 (2.17, 5.99), p=0.0001, df 40 vs claimed 4.1 (2.2, 6.0) p<0.001. Crude L-P 3.75 (1.88, 5.62) vs 3.8 (1.9, 5.6). Male 3.59 (1.46,5.73), female 4.62 (1.91,7.33), 20-39 7.47 (3.41,11.52), 40-59 2.21 (0.23,4.19) p=0.029, 60+ 1.27 (-0.97,3.52) p=0.258, P-I 1.06 (-0.25,2.37) p=0.111 - all match Table 3 to the reported precision. Cycle-L age-specific values 16.56/10.85/8.73 match the NCHS DB527 validation targets 16.6/10.8/8.7. Tolerance 0.1 pp.
Independent re-analysis of the 24 public NHANES XPT files (DEMO/DPQ, cycles D-L) with own Taylor-linearisation code (strata/PSU, design df), Korn-Graubard CIs for crude, t CIs for direct age standardisation (Eq. 1), contrast df = sum of cycle df
Short sleep and mean workday sleep per cycle (Table 2) and H1 L-P = -3.2 pp (-5.2, -1.3), p=0.002 (Holm 0.004); sex/age contrasts, DiD 4.0 (0.9, 7.1), joint age tests p=0.478 / 0.019; S3 -3.3, S4 -1.7 (-3.6, 0.3) p=0.088, S5 -2.4 (-4.4, -0.4)
All 48 Table 2 short-sleep cells and 16 mean-sleep cells reproduced to the reported precision (max diff 0.00 pp / 0.00 h) when short sleep is computed among all MEC-examined adults with valid sleep (not the PHQ-complete analytic sample of Table 1, which gives e.g. I 19.14, P 24.11, L 20.60). The cycle-L crude interval (19.8, 21.6) is reproduced only with the uncapped KG variant; the NCHS-capped variant gives (19.58, 21.83) because the design effect is 0.64. H1 obtained -3.23 (-5.20, -1.26) p=0.0020; crude -3.15 (-5.07,-1.23); male -5.29 (-7.51,-3.07); female -1.27 (-4.04,1.51) p=0.362; 20-39 -4.29 p=0.0012; male 20-39 -8.03; DiD 4.03 (0.91,7.14) p=0.0127; joint age F(2,40) p=0.478 (sleep) and 0.019 (depression); depression DiD 1.03 (-1.99,4.04) p=0.495; mean sleep +0.13 (0.06,0.20), men +0.23, women +0.04 p=0.451; P-I 4.38 (2.34,6.42). Sensitivity: S4 -1.67 (-3.59,0.26) p=0.0876; S3 -3.29 (-5.52,-1.06) p=0.0048; S5 -2.42 (-4.43,-0.41) p=0.0193. All match. Tolerance 0.1 pp / 0.01 h.
Independent re-analysis of public NHANES DEMO/SLQ files (SLD010H series B, SLD012 series A, SLD013 for weekly average) with own Taylor-linearisation code; Korn-Graubard CIs computed both with the NCHS cap (effective n <= n) and without it (R survey::svyciprop beta)
Sample flow: Table 8 eligible n and item-missing counts, Table 1 unweighted n (total 42,222) and weighted composition, education missing, Table 4 adjusted-model n
Minor and explained. 117 of 120 counts match exactly (eligible n 4,773 ... 6,064; PHQ-missing 489 ... 854; education and PIR missing; Table 1 n and Table 4 n in 7 of 8 cycles), and all 48 checked Table 1 weighted percentages (male, 60+, <HS, college, NH White, PIR missing) match within 0.1. Cycle I: under the stated rule (valid sleep, no other exclusions) we get sleep-missing 27 vs 28, Table 1 n 4,865 vs 4,864, model n 4,864 vs 4,863, total 42,223 vs 42,222. Treating the single SLD012 = 14.5 h response as missing reproduces the paper exactly (28 / 4,864 / 4,863), so the paper appears to have applied an undocumented 14-h upper bound (the P and L range is 2-14) that Sec. 2.2 ("No other exclusions were applied") does not mention. No effect on any reported prevalence at 0.1 pp precision.
Counted records in the merged public DEMO/SLQ/DPQ files (MEC-examined, age 20+), weighted percentages with MEC weights
Adjusted PRs for short sleep 1.41 (1.12, 1.78) in P and 1.71 (1.38, 2.11) in L (Eq. 2); H3 ratio 1.21 (0.89, 1.63), p = 0.216; all Table 4 PRs, Table 5 ratios, Table 10 P/L subgroup PRs, Table 7 model-based sensitivity analyses.
All 8 cycles: adjusted short and long PRs and CIs and model n reproduce Table 4 exactly after rounding (max abs diff 0.010, in one unadjusted cell). P 1.414 (1.120, 1.784) vs 1.41 (1.12, 1.78); L 1.707 (1.383, 2.106) vs 1.71 (1.38, 2.11). H3 ratio 1.207 (0.892, 1.635) p = 0.216 vs 1.21 (0.89, 1.63) p = 0.216. Long-sleep ratio 0.998 (0.766, 1.299) p = 0.986 vs 1.00 (0.77, 1.30) 0.986. All five subgroup ratios and their P/L PRs in Table 10 match to 0.01, e.g. 40-59: 2.027 (1.100, 3.735) p = 0.025 vs 2.03 (1.10, 3.73) 0.025. Table 7 S2 1.420 (0.922, 2.188) p = 0.109 vs 1.42 (0.92, 2.18) 0.110; S3 1.356 p = 0.045; S4 1.350 (1.004, 1.815) p = 0.047; S7 1.210 p = 0.207; S8 1.158 p = 0.353; S10 OR ratio 1.295 p = 0.159, all matching. Note 1: the unadjusted PRs and the unadjusted ratio 1.19 (0.88, 1.61) p = 0.249 reproduce only on the Table 1 sample including the education-missing records (n 7821/5157), whereas the Table 4 n column refers to the adjusted model (model n gives L unadjusted 1.811 vs 1.82 and p = 0.257). Note 2: adjusted PDs P 2.94 (0.68, 5.20) vs 2.9 (0.7, 5.2) and L 6.95 (3.73, 10.18) vs 7.0 (3.5, 10.4); difference 4.02 (0.24, 7.79) p = 0.038 vs 4.0 (0.1, 7.9) p = 0.045. Point estimates match; our beta-only delta-method SE is slightly smaller, so the CI difference reflects the variance method, not an error. Tolerance 0.01 for ratios, 0.005 for p.
Independent refit from public NHANES files: survey-weighted Poisson (log link) by IRLS with linearised sandwich variance (strata/PSU), domain estimation, t CIs with design df, ratio of PRs with df 40; logistic + marginal standardisation for PDs (delta method on beta only).
Direct age standardisation (Eq. 1) of Table 9 age-group prevalences reproduces the Table 2 age-adjusted values and the Table 9 sex-specific age-adjusted rows; crude overall values are consistent with Table 1 age shares.
(a) All 16 overall age-adjusted values (short sleep and depression, 8 cycles) recomputed within 0.15 pp. Example: L depression 0.396616.6 + 0.371810.8 + 0.2316*8.7 = 12.6 vs Table 2 12.6. (b) All 32 sex-specific age-adjusted values are within 0.15 pp. (c) Crude values from the Table 1 age shares are within 0.25 pp; the largest gap is short sleep in G, 36.66 vs 36.8, which the different denominators explain. The 0.15 pp tolerance covers input and output rounding. Independently, checks 01/02 reproduce the same values from raw data.
Arithmetic from the printed tables only: 16 overall, 32 sex-specific and 16 crude recombinations.
Table 3 contrast arithmetic and inference: differences equal differences of Table 2/9 entries; CIs and p consistent with per-cycle CIs and design df (contrast df 40); Holm adjusted p for H1-H3 0.004, <0.001, 0.216.
All 22 crude point differences equal differences of the Table 2/9 entries within 0.15 pp (e.g. M40-59 short 27.5 - 31.6 = -4.1 vs -4.0, a rounding difference). The 7 age-adjusted contrasts recombine to the printed CIs within 0.15 pp: H1 -3.30 (-5.28, -1.32) p = 0.0017 vs -3.2 (-5.2, -1.3) p = 0.002; H2 4.10 (2.17, 6.03) vs 4.1 (2.2, 6.0). The female short-sleep p (0.347 vs 0.362) is within 0.02, which is all that rounded inputs allow; the exact value was reproduced from raw data in check 02. Holm with the exact p values from checks 01/02/04 (0.00198, 0.000124, 0.216) gives 0.004, 0.0004, 0.216, matching 0.004, <0.001, 0.216. All 19 raw-data rows match to the printed precision, e.g. dep F20-39 7.86 (2.71, 13.01) vs 7.9 (2.7, 13.0); mean sleep M20-39 0.37 (0.22, 0.52) vs 0.37 (0.22, 0.52); mean sleep 40-59 0.10 (-0.03, 0.22) p = 0.124 vs p = 0.124. Tolerance: 0.15 pp (0.015 h) on limits, one rounding unit on raw rows.
(i) Arithmetic from the printed tables: SEs backed out from the t-based CIs (df L 15, P 25, I 15) and recombined with df 40. (ii) Holm recomputed. (iii) Raw-data recomputation of the 19 sex x age and mean-sleep age rows not covered by checks 01/02.
Ratio-of-PR arithmetic in Table 5 and Table 7 (overall 1.21, long 1.00, unadjusted 1.19, men 1.12, women 1.29, 20-39 0.85, 40-59 2.03, 60+ 1.19; S2 1.42, S3 1.36, S4 1.35, S7 1.21, S8 1.16, S9 1.22, S10 1.29) and the equality F tests follow from the cycle-specific PRs.
All 15 ratios recombined from the printed PRs fall within 0.02 of the printed ratio and limits, and p within 0.02. Examples: H3 1.213 (0.895, 1.643) p = 0.207 vs 1.21 (0.89, 1.63) p = 0.216, a difference from rounded inputs (raw-data refit in check 04 gives exactly 0.216); S9 multiple imputation 1.213 (0.896, 1.642) p = 0.205 vs 1.22 (0.90, 1.64) p = 0.200. S9 itself could not be refitted because the imputation model and seed are not reported. Equality tests: I=P=L F(2,55) = 0.982 p = 0.381 vs 0.95 p = 0.395; D..H F(4,79) = 0.813 p = 0.521 vs 0.80 p = 0.526. Joint short&long test from raw data: F(2,40) = 0.930 p = 0.403 vs 0.93 p = 0.403. Tolerances: 0.02 on ratios/limits/p; 0.1 on F and 0.05 on p for the F tests.
Log-scale SEs backed out from the printed PR CIs (df P 25, L 15), recombined with df 40. Wald equality tests across independent cycles. The joint short&long test was refitted from raw data with the within-cycle covariance.
Benjamini-Hochberg q values within F2 (14 tests), F3 (9 tests) and F4 (8 tests) as reported in Tables 3, 5 and 6.
All 31 q values match within 0.002. F2 examples: short female 0.4224 vs 0.422; short F-M 0.0296 vs 0.030; dep 40-59 0.0514 vs 0.051; dep age joint 0.0380 vs 0.038; short male 0.0001 vs <0.001. F3: female ratio 0.4020 vs 0.403; 40-59 ratio and PD difference 0.2025 vs 0.204; others 0.6694 vs 0.669. F4: P-I depression 0.2216 vs 0.222; D-P linear 0.0080 vs 0.009 (the printed p 0.003 is rounded; 0.009 implies p about 0.0034); quadratic terms 0.2533 vs 0.253; series A linear 0.4171 vs 0.417. With "<0.001" crudely bounded at 0.0005, three "<0.001" q values would come out at 0.002-0.0035, so the printed q values require the exact p. Family sizes (14/9/8) match Sec. 2.5. Tolerance 0.002.
BH step-up recomputed. For F2 the exact p values from the raw re-analysis (checks 01/02) are used; for F3 the printed p. For F4 the exact P-I p (check 02) and the D-L linear p backed out from its CI are used. A variant bounding "<0.001" at 0.0005 is printed for information.
Trend tests (Table 6): linear depression trend D-L 0.526 (0.360, 0.693), df 134, q < 0.001; quadratic 0.107 p = 0.190; D-P linear 0.237 (0.081, 0.394) p = 0.003; short sleep series B linear -0.033 / quadratic -0.067, series A linear 0.027 p = 0.365; df equal sums of cycle design df.
All six Table 6 rows reproduce exactly (max deviation 0.000 at 3 decimals) when the sample is every examined adult with a valid outcome: dep D-L linear 0.526 (0.360, 0.693) p < 1e-6; quadratic 0.107 (-0.054, 0.267) p = 0.1899; D-P linear 0.237 (0.081, 0.394) p = 0.0032; short D-H linear -0.033 (-0.137, 0.070) p = 0.5204; quadratic -0.067 (-0.161, 0.028) p = 0.1632; I-L linear 0.027 (-0.032, 0.087) p = 0.3641. Design df by cycle D15 E16 F16 G17 H15 I15 P25 L15 give the sums 134/119/79/55 exactly. Restricting to the analytic sample (valid sleep and PHQ-9) changes the coefficients by up to 0.028, e.g. short D-H linear -0.007. Like Table 2 (check 02), Table 6 is therefore computed per outcome on all valid records, not on the single analytic sample described in Sec. 2.2/Table 1.
Raw refit: pooled survey-weighted logistic regression on orthogonal-polynomial contrasts of the cycle midpoints (unit-norm columns on the unique midpoints), adjusted for 3 age groups, linearised variance with strata/PSU, df = sum of cycle design df. Also p arithmetic from the printed CIs, and design df counted from the DEMO files.
Exists (doi 10.21203/rs.3.rs-6475145/v1), but it is a non-peer-reviewed preprint not indexed in PubMed. It is cited only as an example of studies treating sleep as a covariate, so it is not load-bearing. Flagged for transparency only.