Summary
The paper asks whether the share of public GitHub pushes made outside 09:00–18:00 local time (O1) or on a local weekend (O2) departed from its 2018–mid-2022 trend after GitHub Copilot and ChatGPT became available. A keyed-hash 2% sample of accounts yields 5,302 developers reached before a stop time, of whom 4,873 received a local time zone per era from their (2026) profile location or commit time-zone offsets. Monthly unweighted means of developer shares (103 months, 98 fitted) are modelled with segmented OLS (Eq. 1: level and slope change at December 2022, pandemic indicator, month-of-year dummies, Newey–West SEs), complemented by a developer fixed-effects panel (M2), an event study, placebo breaks, regional heterogeneity and 26 robustness checks. Decisions follow a pre-specified four-category rule combining Holm-adjusted tests with TOST against ±1 pp.
Both primary estimates are negative and inconclusive (O1 −0.85 pp [−2.27, 0.57]; O2 −1.25 pp [−2.77, 0.26]). The paper argues that pandemic-sized increases (~3 pp) are excluded, that within-developer estimates are near zero, that placebo breaks give similar departures and that robustness checks do not change the conclusion; among regions only Eastern Asia shows a decrease in O1 (−10.20 pp), interpreted as the unwinding of a pandemic-era rise. The study was produced end to end by an automated research system and is presented as pre-registered.
Strengths
Core estimates are exactly reproducible from the published figures
Table 4; Eq. (1) §3.7; Figures 2, 4, 5, 7; Tables 5, 7, 8
Figures 2–7 are vector graphics that encode the aggregated monthly series and every plotted estimate. Refitting Eq. (1) on the 103 monthly means recovered from Figure 2 reproduces Table 4 to the second decimal (O1 −0.850 [−2.266, 0.567], p = 0.236; O2 −1.253 [−2.766, 0.260], p = 0.103), the drawn fit and counterfactual to <1e-4 pp, all 16 aggregate-series robustness estimates (R1, R3, R4, R11, R20, R26, agent-era break), the P1 placebo p-values (12/36, 15/36), the regional Cochran's Q (11.23, Holm p 0.0073, I² 0.82) and the Figure 5 TVD (0.0248 vs 0.025). Checks 01–05, 07, 11.
Explicit decision framework that allows an 'inconclusive' verdict
§3.8; §4.3; Table 4
The SESOI of ±1 pp, TOST, Holm/BH families and a four-category rule are stated before the results (§3.8), and the main text reports both primaries as inconclusive rather than as 'no effect' (§4.3, Table 4). Every category in Table 4 and Table 5 recomputes from the printed intervals (check 02, 13). The anticipated half-width (1.4–2.2 pp) is compared with the observed 90% half-widths (1.2, 1.3 pp; recomputed 1.188 and 1.267).
Transparent deviation log and reporting of infeasible analyses
Table 2 (p. 4); §3.9; §4.6; Table 5
Table 2 logs seven deviations including the substantial D5 (extended local-time resolution) and flags D6/D7 as written after a development run, moving R8 and R21 to exploratory. Pre-specified analyses that failed the group minimum (S2 countries, S3 Italy) are reported as infeasible rather than dropped silently (§4.6, Table 5).
Serious handling of known confounders and archive breaks
§3.6; §3.10; §5.1; Figure 5; Table 7
The design anticipates the pandemic (indicator plus three alternatives R10, R20, R26), composition (M2, R7–R9), platform changes (R11, R12, R25) and the GH Archive capture drop. The 24 May 2025 drop is real and correctly sized in the public archive (144,617 vs 'about 145,500' pushes/hour before; check 10), and the agent-era analysis is correctly kept descriptive because of it.
Candid limitations and mostly careful causal language
§2.5; §6.4; §7; Table 3
§2.5 and §6.4 state that an ITS without a control group cannot attribute a change to the tools; §7 lists push ≠ work, fixed offsets without DST for about half of the developer-months, pandemic modelling and the capture drop. Table 3 discloses how local time was obtained (offset commit 49.7%, city/zone 40.3% of 66,672 developer-months, check 09).
Literature base exists and is largely quoted correctly
§2.1–2.2; §3.8; References
All 50 references exist (43 resolved automatically, 7 located manually). Spot checks of full texts confirm the figures taken from Peng et al. [1] (55.8% faster), Claes et al. [8] (median start 10:03, ~60% in window) and Mao et al. [13] (0.372 vs 0.358 = 1.4 pp, one of the two SESOI anchors). Check 15.
Weaknesses
The 'pre-registered' claim in the title cannot be verified
major · Title; Abstract; §3.1; Table 2 (D5–D7); §3.11; §8; ref. [40]
§3.1 dates the freeze to 9 Nov 2026 19:52:27 UTC and prints SHA-256 hashes, but cites only [40] (Nosek et al. 2018, a general essay on preregistration), not a registry record; the paper contains no URL, DOI, OSF/AsPredicted/Zenodo identifier or trusted timestamp (check 12). The 'accompanying experiment package' (§3.11) is not linked and was not supplied. The timeline is very compressed: freeze → data stop 00:35 UTC (4 h 43 min, itself deviation D5, so the stop time was fixed after the freeze) → PDF creation 02:37 UTC, i.e. all outcomes, 33 robustness rows, event study, placebos, groups and writing within about two hours. D6/D7 were written after an undated 'development run had produced estimates'. §8's 'fixed and time-stamped before these numbers were calculated, so the results could not steer the method' is therefore an unsupported assertion. Fix: deposit the frozen plan with a public, third-party timestamp (OSF, Zenodo, RFC 3161), link it, publish the development-run log with timestamps, and until then qualify 'pre-registered' in the title and abstract.
Abstract understates non-null results the paper's own rule produces
major · Abstract; §5.2; §6.2; Table 6 (p. 10); Table 8 (p. 14)
The abstract says '26 pre-specified robustness checks did not alter this conclusion' and that 'only Eastern Asia changed'. Applying the §3.8 rule to the printed intervals: R18 (10:00–18:00 window) gives O1 [−3.45, −0.90], Holm p = 0.0011 → 'decrease' (check 08); the pre-specified secondary S4 band ≤ UTC−5 gives O2 −2.33 [−3.79, −0.86], p = 0.0022, still 'decrease' after BH over six bands (p = 0.0131) (check 13). Four further O2 intervals exclude zero (R1 lag 12, R6, R11, R17; raw p 0.031–0.046, inconclusive only after Holm), including R11, which §3.10 says must be reported beside the primary. §6.2 concedes that 'conclusions about change depend on where the working day is assumed to start', but the abstract does not. Table 6 lacks a decision column and is labelled 'Exploratory' although §3.9 lists S4 as secondary. Fix: report R18 and the S4 band in the abstract and state that the O2 sign is negative in all 31 rows.
Linear counterfactual is fragile, especially for O2
major · Eq. (1) §3.7; §4.5; Figure 3; Table 4 footnote †; §5.3; Figure 7; Table 8 R10
Δpost extrapolates a 54-month linear trend 44 months ahead from a pre-period dominated by a pandemic shock modelled as a flat indicator. The paper's own event study rejects a linear O2 pre-trend (χ² = 40.8, 8 df, p = 2.3e-6; 2018–2019 bins swing between −2.06 and +2.70 pp; check 06), yet O2 stays confirmatory with only a dagger in Table 4. R10 flips the O1 sign (+1.50). The O2 placebo distribution is mostly positive (23/36 P1, 34/67 P2, up to +5.96 pp) because fake breaks near the pandemic onset dominate, so 'placebo breakpoints produced departures of similar size' mixes magnitudes of opposite sign (check 05). The P2 rank (24th, 17th) compares the primary estimate computed with the launch-window exclusion against 67 placebo estimates computed without it; the like-for-like 2022-12 value ranks 30th and 28th. Fix: report the O2 confirmatory result conditional on the failed pre-trend test, add a model with a flexible pre-trend or a comparison series (e.g. non-developer GitHub activity, pre-AI-era cohorts), and rank like-for-like placebo estimates.
Within-developer 'close to zero' and the compositional interpretation exceed the evidence
major · Abstract; §4.4; §6.1; Table 4 (M2 columns)
The abstract, §4.4 and §6.1 call the M2 estimates (+0.02 [−2.70, 2.73], +0.40 [−1.69, 2.48]) 'close to zero'. Under the paper's own TOST against ±1 pp they are not equivalent (p = 0.240 and 0.286; 90% CIs [−2.26, 2.30], [−1.35, 2.15]), i.e. 'inconclusive' by §3.8 (check 14). 'About twice as wide' holds for O1 (1.91×) but not O2 (1.38×). §4.4 then states that 'taken at face value, the small negative population-level estimates reflect a change in who is active rather than a change in when the same developers push', a compositional conclusion that the difference test (p = 0.58 / 0.21 assuming independence) does not support and that M1 vs M2 weighting differences (months vs developer-months, growing sample) would confound even if precise. Fix: report M2 as inconclusive and drop or clearly label the compositional reading as speculation.
External benchmark and platform-change evidence misdescribed
major · §2.1; §2.2; §3.8; §6.2; refs [10], [22]
§2.2 says McDermott & Hansen [10] converted events to each user's local time; the cited 20% → 24% weekend rise is their Fig. 3, global activity with 'Timestamps ... relative to UTC'. §6.2 nevertheless claims the paper's local-time series 'reproduces' it, a validation claim for the local-time pipeline that compares unlike measures, and §3.8 scales the 1 pp SESOI as 'about a quarter of the pandemic rise' from it. §2.1 says Moldon et al. [22] found streak removal lowered the weekend share 'by less than a tenth of a percentage point ..., identified with a regression discontinuity'; the <0.1 pp figure is a descriptive before/after change (Table II, 0.2188 → 0.2179), while the RD estimates are 2.4–9.9 pp (Table III). This understates how much a single platform change moved timing shares, which is exactly the threat R11/R12/R25 address (check 15). Fix: correct both descriptions, drop the 'reproduces' claim or compare against a UTC version of the paper's own series, and re-justify the SESOI.
Data provenance: no package, and an undocumented archive change after October 2025
major · §3.2; §3.3; §3.11; §5.1; Table 8 (R16, R17); ref. [36]
Neither code, frozen plan nor aggregated series is linked (§3.11), so beyond what the vector figures encode nothing can be rerun; M2, the event-study χ², §5.4 and R12/R25 are not reproducible. In the public archive the October gap begins 8 Oct 2025 17:00 UTC, not 9 Oct (check 10b), and after the gap PushEvent payloads lose 'commits', 'size' and 'distinct_size' (bytes per event ~1,230 → ~635) while pushes per hour rise (check 10c). The paper does not mention this change although it covers 10 of 44 post-launch months; it matters for R17 (commit-weighted shares), for the undefined 'AI-agent markers' (§3.3, R16/R16b), and for the file-size completeness filter (§3.2). Primary push timestamps are probably unaffected, but the paper should say so. Fix: publish the package with a DOI, define the agent markers, document archive version/access dates, and state how R17 and the size filter handle the post-October 2025 format.
Sampling is described as 'random' but conditions on resolution and whole-period activity
minor · Abstract; §3.3; §3.4; §4.1; Table 2 (D5)
The 4,873 developers are the 91.9% of the 5,302-account random prefix whose time zone could be resolved (429 non-random failures), and eligibility requires ≥10 sampled pushes and ≥6 active months over 2018–2026, which conditions on activity after the break. Local time uses a 2026 profile location applied back to 2018 and fixed offsets from up to two commits per era without DST (§3.4, §7). Only 5,302 of 130,751 eligible accounts (4.06%) were reached, which drives the imprecision. Fix: describe the sample as 'developers with a resolved time zone from a random sample', compare resolved vs unresolved accounts on observables, and run a sensitivity analysis restricted to developers active before 2022.
Post hoc explanation of the Eastern Asia result
minor · Abstract; §4.6; §6.3; §8
Eastern Asia O1 −10.20 [−15.25, −5.16] is a pre-registered 'decrease' (verified, check 07/13). §6.3 and §8 explain it as 'a return to normal after the pandemic rather than an effect of the tools'. No pre-specified analysis separates these explanations; the claim should be presented as one hypothesis, not a conclusion, especially in the plain-language summary.
Internal inconsistencies in counts and labels
minor · Table 6; Table 2 (D3); Table 8; Figure 6; §5.2; §4.5
Table 6 band counts sum to 5,359, 486 more than the 4,873 developers the bands are said to 'cover' (check 09). D3 refers to '5 of 1,576' profiles while 1,628 developers had been resolved at the freeze. R13 is 'no offset only' in Table 8, 'location-based only' in Figure 6 and 'commit offsets for every developer rather than locations' in §5.2, i.e. opposite rules for the check cited in §7 as moving O1 by >1 pp (1.15 pp). R16 is 'no agent pushes' (Table 8) vs 'no agent-marked months' (Figure 6). The event-study O1 bins from 2025 are described as 'close to zero', but the Dec 2025–Feb 2026 bin is −1.42 pp (check 06). Figure 6 omits the 'R19 same sample original outcome' row.
Newey–West without small-sample correction
minor · §3.7; Table 4
The reported CIs correspond to Newey–West (lag 6) without the n/(n−k) factor; with p = 16 parameters and n = 98 the corrected intervals are [−2.40, 0.70] (O1) and [−2.91, 0.40] (O2), p 0.278 and 0.136 (check 01). No decision changes, but with 16 parameters on 98 months the choice should be stated and the corrected version shown as sensitivity.
Questions for the authors
Can you provide a public, third-party timestamp (registry entry or trusted timestamp) for the frozen plan and the hashes in §3.1, and a timestamped log of the 'development run' that preceded D6/D7? What outcomes did that run produce?
Was the data-collection stop time (10 Nov 2026 00:35 UTC, D5) chosen before or after any outcome estimate had been computed, and by what rule?
Why is O2 retained as a confirmatory test after the event study rejects a linear pre-trend (χ² = 40.8)? Was a decision for this case pre-specified?
How are the 'AI-agent markers' (§3.3, R16/R16b) defined, and how are they and R17's commit weights computed for Oct 2025–Jul 2026, when PushEvent payloads no longer list commits?
Why do the Table 6 band counts sum to 5,359 rather than 4,873? Are developers counted in several bands?
Which rule does R13 implement: offsets only, locations only, or 'no offset'?
Is the P2 placebo ranking intended to compare the primary estimate (with the launch-window exclusion) against placebo estimates without it? What are the ranks like for like?
How do the 429 unresolved developers and the developers first active after 2022 differ from the analysed sample, and do the primary estimates change if the sample is restricted to developers active before the break?
Verification
Only the PDF was available. Because Figures 2–7 are vector graphics, I recovered the 103 monthly means per outcome and every plotted estimate and refitted the main model independently. The headline M1 estimates, CIs and p-values, Table 4 adjustments and decisions, yearly means, 16 aggregate robustness estimates, the P1 placebo p-values, the regional Q/I² statistics and the capture-drop TVD all reproduce within rounding (checks 01–07, 11). The 24 May 2025 capture drop is confirmed in the public GH Archive. Discrepancies concern framing and provenance rather than the core numbers: two non-null results under the paper's own rule are missing from the abstract (08, 13), the M2 'close to zero' claim fails the paper's own equivalence test (14), Table 6 counts do not add up (09), the October 2025 gap is misdated and a payload change after it is undocumented (10b, 10c), and two references are misdescribed (15). The pre-registration and the experiment package could not be checked (12). Developer-level analyses (M2, event-study χ², §5.4) could not be rerun.
Primary M1 estimates: O1 Δpost −0.85 [−2.27, 0.57] p=0.236; O2 −1.25 [−2.77, 0.26] p=0.103; O1 level −0.58 [−1.62, 0.46], slope −0.15/yr; O2 level −1.32 [−2.39, −0.24] p=0.017, slope +0.04/yrConfirmedAbstract; Table 4 (p. 7); §4.3; Eq. (1) §3.7; Figure 2 (p. 8)
O1: Δpost −0.850 [−2.266, 0.567] p=0.236 (claimed −0.85 [−2.27, 0.57] p=0.236); level −0.580 [−1.616, 0.456], slope −0.150. O2: Δpost −1.253 [−2.766, 0.260] p=0.103 (claimed −1.25 [−2.77, 0.26] p=0.103); level −1.315 [−2.387, −0.244] p=0.017, slope +0.035 (claimed +0.04). Fitted values and counterfactual match the drawn Figure 2 polylines to <1e-4 pp. Tolerance ±0.02 estimates, ±0.05 CI bounds. Note: the reported inference uses NW without the n/(n−k) factor; with it the CIs widen to [−2.40, 0.70] and [−2.91, 0.40] (p 0.278, 0.136), which does not change any decision. The pandemic indicator coefficient is +2.25 pp (O1) and +1.20 pp (O2).
script: 01_m1_refit.pyRecovered the 103 monthly means per panel from the vector markers of Figure 2 (tick calibration residual <1e-7 pp); refitted Eq. (1) by OLS on 98 months (Jul–Nov 2022 excluded, pandemic Mar 2020–Dec 2021, 11 MoY dummies), Newey–West Bartlett lag 6 without small-sample factor, t(82). Compared fitted values and counterfactual with the polylines drawn in Figure 2.
Table 4 inference is internally consistent (90% CI, p, Holm/BH p, TOST p, decisions; observed 90% half-widths 1.2 and 1.3 pp)ConfirmedTable 4 (p. 7); §3.8; §4.3
All 4 rows agree within rounding: e.g. O2 90% CI [−2.517, 0.017] (claimed [−2.52, 0.01]), p 0.105 (0.103), Holm 0.209 (0.207), TOST 0.628 (0.630); O3 BH 0.328 (0.332). 90% half-widths 1.188 (O1) and 1.267 (O2) match the quoted 1.2 and 1.3 pp. All four decisions recompute as inconclusive. Tolerance 0.01 on p, 0.02 on CI bounds.
script: 02_table4_consistency.pyBacked out SE from each 95% CI with t(82); recomputed 90% CI, two-sided p, Holm (O1,O2), BH (O1w,O3), TOST vs ±1 pp; applied the four-category rule
Yearly means O1 43.1/46.7/47.4/46.1/45.8/47.3/47.2 and O2 20.6/23.4/22.4/24.0; both shares rose about 3 points during the pandemic; largest rise in 2020Confirmed§4.2; Abstract; §6.2; Figure 2
All 11 quoted yearly means reproduce within 0.05 pp (e.g. O1 2018 43.12 vs 43.1; O2 2023 22.35 vs 22.4; tolerance 0.1). Largest year-on-year rise is 2020 for both (O1 +3.27, O2 +2.38). Pandemic-era rise: O1 +3.62 (2018→2020), O2 +2.73 (2018→2020) or +2.17 (Mar 2020–Dec 2021 mean vs Jan 2018–Feb 2020 mean), so ‘about 3 points’ is accurate for O1 and generous for O2. Second-largest rise is 2024→2025 (+1.20 and +1.23 pp), coinciding with the agent era and the May 2025 capture drop. Note the M1 pandemic coefficient is only +2.25 (O1) and +1.20 (O2), because the linear trend absorbs part of the rise; both are still above the CI upper bounds 0.57 and 0.26.
script: 03_yearly_means.pyCalendar-year unweighted means of the 103 monthly values recovered from Figure 2
Aggregate-series alternative specifications R1 (lag 3, 12), R3, R4, R11, R20, R26 and the agent-era second breakConfirmedTable 7 (p. 12); Table 8 (p. 14); §4.3; §5.1–5.2
16/16 estimates and CIs reproduced within 0.01 pp (tolerance 0.05), and the 6 Table 7 p-values exactly: e.g. R3 O1 −0.740 [−1.589, 0.110] p=0.087 (claimed −0.74 [−1.59, 0.11] p=0.087); R11 O2 −1.413 [−2.758, −0.068] (claimed −1.41 [−2.76, −0.07]); agent era O2 +1.470 [0.037, 2.902] p=0.045 (claimed +1.47 [0.04, 2.90] p=0.045), n=94. Unreported p-values implied: R1 lag 12 O2 p=0.032, R11 O2 p=0.040. R12 and R25 could not be refitted (break dates and month exclusions not given).
script: 04_aggregate_robustness.pyRefitted each variant on the Figure 2 series with the check-01 code; R26 as pandemic×τ interaction; agent era as extra level+slope at June 2025 with Feb–May 2025 excluded
Placebos: P1 36 fake breaks, empirical p 0.333 (O1), 0.417 (O2); P2 67 candidate breaks, observed estimate ranks 24th (O1) and 17th (O2)Confirmed§5.3; Figure 7 (p. 15); Abstract
P1: refit on data Jan 2018–May 2022 with no exclusion window gives 12/36 = 0.333 (O1) and 15/36 = 0.417 (O2), exactly as claimed; the refit range (O1 −7.07..+1.14, O2 −1.54..+7.79) matches the histogram extent and both histograms sum to 36. P2: the 67 plotted estimates are reproduced to <0.001 pp by M1 with NO launch-window exclusion (the R3 rule); the ranks 24th and 17th are reproduced only when the primary estimate (−0.85, −1.25, computed WITH the exclusion) is ranked among these no-exclusion values; the plotted 2022-12 value itself (−0.740, −0.928) would rank 30th and 28th. The comparison mixes two specifications (not stated in the text) but does not change the qualitative conclusion. Context: the O2 placebo distribution is mostly positive (P1: 23 of 36 positive; P2: 34 of 67 positive, up to +5.96), driven by fake breaks around the pandemic onset, so ‘departures of similar size’ are mostly in the opposite direction.
script: 05_placebo.pyExtracted the P1 histograms and the 67 plotted P2 estimates from vector Figure 7; refitted M1 at every fake T0 on the Figure 2 series under several readings of the placebo rule
Event study: linear pre-trend Wald chi2 = 13.57 (8 df, p = 0.094) for O1 and 40.8 (p < 0.001) for O2; off-hours bins about 1-2 pp below reference in 2023-2024 and close to zero from 2025; weekend bins between -2.0 and +0.4 pp in 2023-2024 and -0.4 to +1.9 pp from 2025.ConfirmedSec. 4.5; Figure 3 (p. 9); Table 4 footnote
p(chi2=13.57, 8) = 0.0937 vs claimed 0.094; p(chi2=40.8, 8) = 2.27e-06 vs claimed < 0.001. O2 bins starting in 2023-2024: min -2.03, max +0.40 (claimed -2.0, +0.4); starting 2025+: min -0.39, max +1.86 (claimed -0.4, +1.9), tolerance 0.05 pp. If bins are grouped by middle month instead, the 2025+ minimum becomes -0.93 (Dec 2024-Feb 2025 bin), so the stated range depends on the grouping. O1 bins starting 2023-2024: -1.93 -1.63 -1.24 -1.67 -1.96 -2.21 -1.95 -1.26 (7 of 8 within 1-2 pp); 2025+: +0.34 -0.12 -0.03 -1.42 +0.48 -0.60 (5 of 6 within 0.6 pp; the Dec 2025-Feb 2026 bin is -1.42, so 'close to zero' is loose). Nine non-pandemic pre-period bins with the reference fixed at 0 is consistent with 8 df. Independent-bins approximation: O1 chi2 = 7.78 (df 8, p = 0.455), O2 chi2 = 30.89 (p = 0.0001), which agrees qualitatively that O2 departs from a line; the exact chi2 values could not be reproduced without the covariance. Note: The event study uses developer fixed effects (dev-level data not released), so the chi2 statistics themselves are not reproducible; only their p-values and the plotted bins were checked. The non-linear O2 pre-period (2018-2019 bins swing between -2.06 and +2.70) undermines the linear counterfactual for a confirmatory outcome.
script: 06_event_study.pyRecomputed chi2 p-values with scipy; extracted all 33 bin estimates and 95% CIs per panel from the vector Figure 3 (/Fm2), calibrated on axis ticks. Bin x positions are the middle month (reference Mar-May 2022 at 2022.28; grey launch-window bins Jun-Aug and Sep-Nov 2022), so bins were grouped by the year in which they start. Approximated the Wald test from the 9 pre-pandemic bins assuming independent bins (approximation only; the bin covariance is not published).
Heterogeneity across the three qualifying regions: O1 Cochran's Q = 11.23 (2 df, Holm p = 0.0073, I^2 = 0.82); O2 Q = 0.96 (p = 0.62); Eastern Asia O1 = -10.20 [-15.25, -5.16], BH p < 0.001, a 'decrease'; other six region/US estimates between -2.63 and +2.13 pp, all inconclusive.ConfirmedSec. 4.6; Table 5 (p. 10); Table 6; Figure 4 (p. 11)
Figure values match every Table 5 point estimate and every Table 6 estimate and CI within 0.005 pp (tolerance 0.15). O1: Q = 11.23 (claimed 11.23), raw p = 0.0036, Holm p = 0.0073 (claimed 0.0073), I^2 = 0.82 (claimed 0.82). O2: Q = 0.96, p = 0.619 (claimed 0.96, 0.62). Eastern Asia O1 -10.20 [-15.25, -5.16], raw p = 1.3e-4; BH p = 3.8e-4 (family: 3 regions, O1), 7.6e-4 (3 regions x 2 outcomes), 1.01e-3 (with US added), so 'BH p < 0.001' holds except under the widest family, where it rounds to 0.001. Decision rule (+-2 pp): Eastern Asia O1 = decrease; the other seven region/US estimates are inconclusive (90% CIs far wider than +-2 pp, e.g. Northern America O1 [-5.07, 8.96]); range of the six non-EA estimates -2.63..+2.13 as claimed. Note: Q treats the three region estimates as independent, which holds since regions are disjoint; the US is a subset of Northern America (275 of 335) and is correctly kept out of Q.
script: 07_heterogeneity.pyExtracted the 10 group estimates and 95% CIs per outcome from vector Figure 4 panel (b) (/Fm3; x calibrated on ticks -15..10, residual 7e-8; the drawn equivalence band is exactly -2.000..+2.000). Backed out SE = half-width / t(0.975, 82); recomputed fixed-effect Q, I^2, chi2(2) p, Holm over the two outcomes, raw and BH-adjusted p for Eastern Asia under three plausible families, and the +-2 pp decision category.
Sec. 5.2 robustness statements (O2 negative in every check, -1.71 R6 to -0.18 R10; four O2 intervals exclude zero, upper bounds -0.04 to -0.14; only R18 10-18 excludes zero for O1; R16 < 0.3 pp; R14/R22/R25 within 0.6 pp; R13 largest local-time change) and the abstract's '26 pre-specified robustness checks did not alter this conclusion'; Table 7 p-values agree with CIs.DiscrepancySec. 5.2; Sec. 6.1; Table 8 (p. 14); Figure 6 (p. 13); Table 7 (p. 12); Abstract
All Sec. 5.2 numerical statements hold: O2 negative in all 31 O2 rows, min -1.71 (R6), max -0.18 (R10); exactly four O2 CIs exclude zero (R1 lag12 hi -0.11, R6 -0.04, R11 -0.07, R17 -0.14); O1 range -2.50 (R19 same-sample) to +1.50 (R10), R10 the only positive O1; only R18 10-18 excludes zero for O1 [-3.45, -0.90]; |R16 - primary| = 0.26/0.10 (< 0.3; R16b is 0.33 for O1); R14a 0.05/0.06, R14b 0.53/0.42, R22 0.03/0.00, R25 0.07/0.47 (all < 0.6); R13 moves O1 by 1.15 pp. No CI asymmetry > 0.015 pp. Table 7 implied p 0.087/0.268/0.473/0.133/0.105/0.044 vs reported 0.087/0.265/0.472/0.134/0.104/0.045 (tolerance 0.005). Figure 6 matches Table 8 within 0.005 pp over 61 estimates; the 'R19 same sample original outcome' row is not plotted. DISCREPANCY with the abstract: applying the paper's own rule, R18 10-18 gives O1 raw p = 0.0011, Holm p = 0.0011 -> 'decrease', so one pre-specified check does alter the O1 conclusion (the paper concedes this in Sec. 6.2: 'conclusions about change depend on where the working day is assumed to start'). The four O2 rows that exclude zero have raw p 0.031-0.046 but Holm p 0.061-0.091, so they remain inconclusive under Holm. Label conflicts: R13 is 'no offset only' (Table 8), 'location-based only' (Fig. 6) and 'commit offsets for every developer rather than locations' (Sec. 5.2), i.e. opposite rules; R16 is 'no agent pushes' (Table 8, Sec. 5.2) vs 'no agent-marked months' (Fig. 6).
script: 08_robustness_text.pyTranscribed all 34 rows of Table 8 (primary + 33 rows covering 26 R-numbers) and Table 7; re-derived every quantitative statement; checked CI symmetry; computed implied p from each CI with t(82) and Holm over (O1, O2) within each row, then applied the pre-registered decision rule (90% CI vs +-1 pp) to each row; checked Table 7 p against CI with t(months - 16) (t(76) for the agent-era model); extracted all 61 plotted estimates and CIs from vector Figure 6 (/Fm5) and compared with Table 8.
Sample accounting: Table 3 sources sum to 66,672 developer-months; 'half rely on commit offsets and 40% on a geocoded city or zone'; 5,302 = 4.1% of 130,751; Table 5 regions sum to the 1,869 developers with a country; S4 bands 'cover all 4,873 developers with a time zone' (Table 6); 103/54/44/98 months; tau_post = 1.79; M2 uses 4,853 developers and 63,114 developer-months.DiscrepancyTable 3 (p. 5); Sec. 3.3-3.4; Sec. 3.6; Sec. 4.1; Sec. 4.4; Table 5 (p. 10); Table 6 (p. 10); Table 2 D3/D5
Consistent: Table 3 sources sum to 66,672 exactly; offset commit 49.7% ('half'; all fixed-offset sources together 55.4%), city or zone 40.3% ('40%'); 5,302/130,751 = 4.06% ('4.1%'); the 11 Table 5 regions sum to exactly 1,869; US 275 + Canada 60 = 335 = Northern America; listed Eastern Asian countries 194 <= 206; months 103/54/44/98 and mean tau 1.7917 (claimed 1.79) correct; M2 shortfall 3,558 developer-months = 712/month if it comes from the 5 excluded months (plausible against the 284-956 range) and 20 developers dropped. DISCREPANCY: Table 6 band counts 1,034 + 681 + 878 + 955 + 982 + 829 = 5,359, which is 486 (10.0%) more than the 4,873 developers the text says the bands 'cover'; the paper does not explain whether developers are counted in several bands (e.g. per era or per January). Minor unexplained count: D3 refers to '5 of 1,576' profiles whereas 1,628 developers had been resolved at the freeze. Note also that 4,873 of 5,302 (91.9%) is the subset with a resolved time zone, so the abstract's 'random sample of 4,873 developers' conditions on a non-random resolution step (429 failures).
script: 09_sample_accounting.pyArithmetic on the transcribed counts (sums, shares, month counts, mean tau, subset checks of countries within regions, M2 shortfall).
GH Archive provenance: 'On 24 May 2025 the number of public pushes captured per hour fell from about 145,500 to about 96,000'.Confirmed§5.1 (p. 9)
12:00 UTC sizes fall from 120–151 MB on weekdays 12–23 May to 64–90 MB from 26 May (weekends 78.7–83.3 → 54.7–60.3 MB). PushEvents/hour average 144,617 before (claimed 'about 145,500', ratio 0.99) and 87,881 after (claimed 'about 96,000', ratio 0.92), both within ±15%. After/before ratio 0.61 vs 0.66 implied by the paper.
script: 10_gharchive_capture.pyPublic data.gharchive.org files: sizes of the 12:00 UTC file every day 10 May–10 Jun 2025 via 1-byte range requests; PushEvents counted in 16 whole hourly files (02/08/14/20 UTC on 20–21 May and 27–28 May 2025). Tolerance ±15% on hourly push counts.
9–14 October 2025 had a 'near-total data loss' and is excluded (§3.2, §3.6).Discrepancy§3.2 (p. 4); §3.6
Hourly files exist but hold only 0.1–0.6 MB from 8 Oct 2025 17:00 UTC to 14 Oct 2025 20:00 UTC (vs ~70–100 MB/hour before), so the gap starts about 7 hours before '9–14 October'. The paper's 20%-of-median size rule would catch the 8 Oct hours, so the practical effect is small, but the stated gap is inexact. After the gap file sizes stay at ~21–38 MB/hour through Jul 2026 (vs 82–94 MB in Jun–Sep 2025), see check 10c.
script: 10b_gharchive_october.pySizes of every hourly GH Archive file on 6–17 Oct 2025 and two days per month Apr 2025–Jul 2026 via 1-byte range requests.
No change in what GH Archive captures after the October 2025 gap is reported (§3.2, §5.1, §7 list only the May 2025 drop and the October gap).Discrepancy§3.2; §5.1; §7; §3.3 (agent markers); R17 in Table 8
PushEvents/hour at 14 UTC rise from 96,940 (Sep 2025) to 116,956 (Nov 2025) and 133,644 (Jun 2026), so the smaller files are not a capture loss but a payload change: PushEvent payloads lose the 'commits', 'size' and 'distinct_size' fields, and mean bytes per PushEvent fall from ~1,230 to ~635. This undocumented change covers 10 of the 44 post-launch months (Oct 2025–Jul 2026). Reasoned, not computed: R17 (commit-weighted shares) and any commit-message-based agent marker cannot use archive commit lists after Oct 2025, and the size-based file filter compares sizes across a format change.
script: 10c_gharchive_after_gap.pyCounted PushEvents and inspected payload fields at 14 UTC on two days each in Sep 2025, Nov 2025 and Jun 2026.
The 24 May 2025 capture drop barely altered the UTC-hour profile of all public pushes (total variation distance 0.025, Figure 5).ConfirmedSec. 5.1 (p. 9); Figure 5 (p. 12)
Both profiles sum to 100.000 %. TVD = 0.0248 vs claimed 0.025 (tolerance 0.003). The plotted profiles are nearly flat (before 3.59-5.32 %, after 3.96-4.69 % per UTC hour, uniform = 4.17 %); the 'after' profile is flatter than 'before'. The TVD is consistent with the figure; whether the figure reflects the archive is assessed separately in check 10.
script: 11_capture_tvd.pyExtracted the 2 x 24 bar heights from vector Figure 5 (/Fm4), calibrated on y ticks 0-5 % (residual 1e-8) and dropping the legend swatches; checked that each profile sums to 100 % and recomputed TVD = 0.5 * sum |p - q|.
The study was pre-registered: plan frozen 9 Nov 2026 19:52:27 UTC before any outcome was computed [40], SHA-256 hashes given; deviations D1-D7 logged; resolution continued to a stop time fixed in advance (10 Nov 2026 00:35 UTC); code, plan and series published in an 'accompanying experiment package'.Not checkableTitle; Abstract; Sec. 3.1; Table 2 (p. 4); Sec. 3.11; Sec. 8; ref. [40]
No URL, DOI, registry identifier or repository link appears anywhere in the paper (0 hits for each pattern); the 'accompanying experiment package' (Sec. 3.11) was not supplied to the reviewer (request has no code_url). Reference [40] resolves on Crossref to Nosek et al., 'The preregistration revolution', PNAS 115:2600-2606 (2018), a general methods article, not a registration record, so the freeze time is supported only by the authors' own statement. The SHA-256 hashes cannot be checked against anything. Timeline from the paper and PDF metadata (CreationDate D:20261110023727, UTC): freeze 19:52:27 -> data-collection stop 00:35:00 (4 h 43 min) -> PDF created 02:37:27 (2 h 02 min after the stop; 6 h 45 min after the freeze), so all outcome computation, 33 robustness rows, event study, placebos, group models and writing happened within about two hours, consistent with an automated pipeline. The stop time 'fixed in advance' is itself deviation D5, i.e. set after the freeze, and D6/D7 were written after 'a development run had produced estimates', a run that is not dated. Sec. 8's 'fixed and time-stamped before these numbers were calculated, so the results could not steer the method' is therefore not verifiable from the materials.
script: 12_prereg_provenance.pyRead PDF metadata with pypdf and computed the timeline; regex search of the full text for any URL, DOI, OSF/AsPredicted/Zenodo/GitHub identifier or registry statement; Crossref lookup of the DOI of [40].
The pre-registered four-category decision rule is applied consistently: Table 4 primaries and secondaries 'inconclusive', Eastern Asia O1 'decrease', other group estimates inconclusive; abstract 'only Eastern Asia changed'.Confirmed§3.8; Table 4 (p. 7); §4.6; Table 5, Table 6 (p. 10); Figure 4; Abstract
Every category the paper reports follows from its rule. All four Table 4 rows are inconclusive (O1 90% [−2.03, 0.33], padj 0.236; O2 [−2.52, 0.01], padj 0.207; O1w padj 0.389; O3 padj 0.332). Eastern Asia O1 −10.20 has 90% CI [−14.42, −5.98] and BH p 0.0004 (3 tests) or 0.0008 (6 tests), so it is a decrease. The other seven group estimates are inconclusive (padj ≥ 0.49 for the US, ≥ 0.63 for the regions). No group 90% CI lies inside ±2 pp, so the abstract's 'showed no departure' for Northern America, Northern and Western Europe and the US means 'not significant', not equivalence. S4: the text is accurate (O2 negative in 5 of 6 bands; O1 range −6.11..+2.42; all six O1 intervals include zero). However, band ≤ UTC−5 O2 −2.33 [−3.79, −0.86] has p = 0.0022, 90% CI [−3.56, −1.10], and is a 'decrease' under the rule, even after BH over the 6 bands (p = 0.0131). Neither Table 6 (no decision column, labelled 'Exploratory' although §3.9 lists S4 as secondary) nor the abstract states it. The abstract's 'only Eastern Asia changed' is literally restricted to regions and is correct for them. Together with R18 (check 08), two non-null results under the paper's own rule are absent from the abstract.
script: 13_decision_rules.pyApplied the §3.8 rule (adjusted p<0.05; 90% CI vs ±1 pp global / ±2 pp groups) with SEs backed out of the 95% CIs under t(82): Table 4 rows with printed padj and 90% CIs; region estimates with BH per outcome (3 tests) and over both outcomes (6 tests); the US as a one-member country family; the six S4 bands uncorrected as pre-specified and with BH(6) as sensitivity; R18 from check 08.
Within-developer (M2) estimates are 'close to zero' (+0.02 [−2.70, 2.73], +0.40 [−1.69, 2.48]); their intervals are 'about twice as wide' as M1's; 'taken at face value' the population-level negatives reflect a change in who is active.DiscrepancyAbstract; §4.4; §6.1; Table 4 (p. 7)
Width ratios: O1 5.43/2.84 = 1.91, O2 4.17/3.03 = 1.38, O1w 1.74, O3 1.30. 'About twice as wide' (tolerance 1.8–2.2) holds for O1 only; for O2 the M2 interval is 38% wider. None of the four M2 estimates is equivalent to zero under the paper's own rule: TOST p vs ±1 pp is 0.240 (O1) and 0.286 (O2). The 90% CIs [−2.26, 2.30] and [−1.35, 2.15] would only be 'equivalent' with bounds of ±2.30 and ±2.15 pp. So 'close to zero' (Abstract; §4.4; §6.1 'for all four outcomes') describes point estimates, not something the data establish; by §3.8 these would be 'inconclusive'. The M1−M2 difference has p = 0.577 (O1) and 0.207 (O2) if the two estimators are independent, and p = 0.195 and < 0.001 if ρ = 1. §4.4's 'difference ... not itself established' is fair, but the following sentence ('reflect a change in who is active rather than a change in when the same developers push') draws a compositional conclusion that the stated uncertainty does not license; §6.1 softens it to 'suggests'. M1 and M2 also differ in weighting: M1 weights months equally, M2 weights developer-months, and the monthly sample grows over time. So an M1−M2 gap would not isolate composition even if it were precise (reasoned, not computed).
script: 14_within_vs_between.pyFrom the Table 4 CIs: M2/M1 width ratios; SE of M1−M2 bounded by ρ = 1 and ρ = 0; z-test of the difference; TOST of each M2 estimate against the paper's own ±1 pp SESOI (normal and t(82) references, because the clustered df are not reported).
Load-bearing literature claims: Peng et al. [1] 55.8% faster; McDermott & Hansen [10] weekend share 'from about 20% to more than 24%' after March 2020 with GitHub events 'converted ... to the local time of each user's profile location'; Mao et al. [13] 1.4 pp non-working-hours difference; Moldon et al. [22] streak removal 'lowered the weekend share of contributions by less than a tenth of a percentage point ..., an effect identified with a regression discontinuity'; Claes et al. [8] median start 10:03, about 60% of commits inside the window; all 50 references exist.DiscrepancySec. 2.1 (p. 2-3), Sec. 2.2, Sec. 3.8 (SESOI justification), Sec. 6.2; References [1]-[50]
Existence: 43/50 matched Crossref/arXiv; the other 7 ([7] HBR Feb 2026, [11] GitHub blog 6 May 2020, [13] arXiv:2603.27130, [30] arXiv:2406.11046, [33] arXiv:2608.00101, [36] gharchive.org, [46] Holm 1979) were located by web search, so no reference is fabricated; [13], [30], [33] are cited without arXiv identifiers. Supported: [1] '55.8% faster' (6 hits); [8] 'median start time ... is at 10:03' and 'commits done in eight-hour work days is 60% on average' (source says both 86 and 87 projects); [13] '0.372 vs. 0.358' = 1.4 pp (paper 1.4 pp, exact). Misstated: [22] Table II all-developer weekend share 0.2188 -> 0.2179 = -0.09 pp is a descriptive before/after figure; the regression-discontinuity estimates (Table III, beta2 = -0.0985, -0.0365, -0.0241, p < 0.001, i.e. -9.9 to -2.4 pp) are 25-100 times larger, so 'less than a tenth of a percentage point ..., identified with a regression discontinuity' conflates the two and understates how much a platform change moved timing shares (relevant to the paper's own platform-change threat). [10] The 'from a baseline of 20 percent to over 24 percent' figure refers to Fig. 3, 'the proportion of global GitHub event activity taking place on weekends', whose caption states 'Timestamps are relative to UTC'; only the city panels are 'adjusted to local time zones'. The paper attributes the figure to a local-time conversion (Sec. 2.2) and claims its local-time series 'reproduces' it (Sec. 6.2), and uses it to scale the 1 pp SESOI (Sec. 3.8, 'about a quarter of the pandemic rise'), so the comparison is between unlike measures. The SESOI's other anchor ([13], 1.4 pp) is correctly quoted.
script: 15_refs_spotcheck.pytools/check_refs.py on all 50 references (log 15_check_refs_tool.log); web search for the 7 unmatched entries; full texts of [1], [8], [10], [13], [22] downloaded (arXiv PDF/HTML, NBER PDF), converted with pdftotext and grepped for the quoted figures (15_refs_spotcheck.py).
References
50 references checked
- [22] Moldon, Strohmaier & Wachs, How Gamification Affects Software Developers (ICSE 2021; arXiv:2006.02371)
Cited for claims it does not support (Sec. 2.1). The paper says the 2016 removal of streak counters 'lowered the weekend share of contributions by less than a tenth of a percentage point ..., an effect identified with a regression discontinuity'. The <0.1 pp figure is the descriptive all-developer before/after change in Table II (0.2188 -> 0.2179). The regression-discontinuity estimates in Table III are beta2 = -0.0985, -0.0365 and -0.0241 (p < 0.001), i.e. 2.4-9.9 pp for developers with >= 30 contributions. Magnitude and method are conflated, and the paper understates how much a platform change moved timing shares (work/verify/15_refs_spotcheck.log).
- [10] McDermott & Hansen, Labor reallocation and remote work during COVID-19 (NBER w29598)
Mischaracterised (Sec. 2.2, Sec. 3.8, Sec. 6.2). The rise 'from a baseline of 20 percent to over 24 percent' refers to Fig. 3, the 'proportion of global GitHub event activity taking place on weekends', whose caption reads 'Timestamps are relative to UTC'. Only the city-level figures are 'adjusted to local time zones'. The paper says [10] 'converted GitHub events to the local time of each user's profile location' to obtain this figure. It then claims its own local-time series 'reproduces' the rise (Sec. 6.2) and uses the rise to scale the 1 pp SESOI ('about a quarter of the pandemic rise', Sec. 3.8). These are unlike measures.
- [40] Nosek et al., The preregistration revolution (PNAS 2018)
Cited as the record for the pre-registration freeze (Sec. 3.1: 'frozen on 9 November 2026 at 19:52:27 UTC ... [40]'). [40] is a general methods essay. It does not document this plan, and no registry entry, DOI or URL for the plan is given, so the timestamp and the SHA-256 hashes cannot be checked against a public record.
- [13] Mao et al., A large-scale comprehensive measurement of AI-generated code (arXiv:2603.27130)
Exists and supports the use: '0.372 vs. 0.358' gives 1.4 pp, as quoted. However, it is cited only as 'arXiv preprint, 2026' with no identifier, and Crossref/arXiv title search did not resolve it. Add arXiv:2603.27130.
- [30] Quispe & Grijalba, Impact of the availability of ChatGPT on software development (arXiv preprint, 2024)
Exists (arXiv:2406.11046) but is cited without an identifier, and tools/check_refs.py did not resolve it. Add the arXiv ID.
- [33] Liu et al., Agentic coding in the wild: Characterizing GitHub Copilot traces at production scale (arXiv preprint, 2026)
Exists (arXiv:2608.00101, July 2026) but is cited without an identifier and was not resolved automatically. Its traces come from US regions only, which is relevant to the Sec. 2.3 point that the source does not state whether times are local.
- [7] Ranganathan & Ye, AI doesn't reduce work - it intensifies it, 2026
No venue given. Located as a Harvard Business Review magazine article (9 Feb 2026) reporting in-progress qualitative fieldwork. It is not peer-reviewed evidence, and the citation should name HBR.
- [11] Forsgren, Octoverse spotlight (GitHub, 2020)
Exists as a GitHub blog post/report (6 May 2020). It is a vendor report, not peer-reviewed, and is cited as a 'Technical report' without a URL.
- [36] Grigorik, GH Archive, 2011
A data source (gharchive.org), cited without a URL or access date. Given the undocumented change in PushEvent payloads after the October 2025 gap (check 10), the version and access date of the archive matter for provenance.
Suggestions
Deposit the frozen plan, deviation log and hashes with a third-party timestamp and cite the registry entry instead of [40]; until then soften 'pre-registered' and the §8 statement that results 'could not steer the method'.
Publish the experiment package (code, aggregated series, estimates) with a DOI and cite it in §3.11.
Rewrite the abstract's robustness sentence to name R18 (O1 decrease with a 10:00–18:00 window) and the S4 band ≤ UTC−5 (O2 decrease), and note that O2 is negative in all 31 robustness rows with four intervals excluding zero.
Describe M2 as inconclusive under ±1 pp, remove 'about twice as wide' for O2, and drop or label as speculation the compositional interpretation in §4.4.
Add a counterfactual that does not rely on a 44-month linear extrapolation: a flexible pre-trend, a comparison series unaffected by AI tools, or a difference-in-differences across developer groups with different AI exposure.
Correct the descriptions of [10] (UTC, global activity) and [22] (descriptive <0.1 pp vs RD 2.4–9.9 pp), and re-justify the 1 pp SESOI accordingly.
Document the GH Archive version and access dates, the October 2025 gap from 8 Oct 17:00 UTC, and the post-gap PushEvent payload change; define the AI-agent markers and explain how R16/R17 work after October 2025.
Fix the label conflicts (R13, R16), reconcile Table 6 counts with the 4,873 developers, add a decision column to Table 6, and make placebo comparisons like for like.
State whether ethics review was sought for geocoding profile locations of identifiable users, and how location data are stored and deleted.
Ethics and integrity
The study geocodes free-text profile locations and commit time-zone offsets of 5,302 identifiable GitHub users; only hashes and aggregates are said to be published, but no ethics review, data-protection basis or retention policy is stated. The paper was generated end to end by an automated system; its 'pre-registered' status rests on unverifiable self-reported timestamps and two citations misstate their sources, which readers should weigh. No text in the PDF attempts to instruct the reviewer.
Scope of this review
Only the PDF was supplied: no LaTeX source, code, data package, developer-level tables or pre-registration document, and the paper gives no link to them. Aggregate-level results were checked by refitting models on series recovered from the vector figures; developer-level analyses (M2, event-study Wald statistics, §5.4 push vs author time agreement, R12/R25 whose break dates are not given) could not be reproduced. GH Archive provenance was checked on samples of hourly files, not the full 103-month archive.