A carefully controlled and internally consistent case study whose reported numbers reproduce almost exactly from the paper's own tables and vector figures, but whose evidence base (two entries, 12 MS2 test micrographs, a six-configuration LoG grid) is narrow, and one Sec. 5.4 value contradicts Fig. 6a.
Soundness
3/4
Presentation
3/4
Contribution
2/4
Confidence
4/5
✓ 15 confirmed✕ 3 discrepancy
Summary
The paper compares a small (922,895-parameter) U-Net that regresses Gaussian particle-centre heatmaps with a Laplacian-of-Gaussian (LoG) blob detector for cryo-EM particle picking, on two CryoPPP subsets: EMPIAR-10017 (β-galactosidase, 84 micrographs, 17 test) and a 60-micrograph subset of EMPIAR-10075 (MS2, 12 test). Both methods share a 32-px border mask, NMS at one particle radius, Hungarian one-to-one matching at one radius, and thresholds frozen before testing (U-Net on validation, LoG configuration and threshold on train+validation).
The U-Net reaches test F1 0.879 / 0.851 versus 0.805 / 0.388 for LoG; paired micrograph-level bootstrap gives ΔF1 +0.074 [0.033, 0.106] and +0.463 [0.304, 0.599]. The authors attribute the gain to precision (ΔP +0.163 / +0.578) rather than recall, show that U-Net precision–recall curves lie above LoG's, that the U-Net wins on 27 of 29 micrographs (exact Wilcoxon p = 0.0056, 4.9×10⁻⁴), and discuss the two exceptions (an apparently mislabelled micrograph and one with non-uniform illumination). A single-seed MSE ablation, training/inference timing, and an explicit limitations section complete the study, which is framed as a controlled case study rather than a ranking against published deep pickers.
Strengths
Single, a-priori evaluation protocol for both methods
Sec. 3.5, 3.6, 4.1, 4.2
Scored region (32-px border), NMS distance, matching tolerance (one radius) and matching rule (Hungarian, micro P/R/F1) are fixed before testing and shared by both methods; thresholds are frozen without test data, and the LoG baseline is tuned on train+validation (more data than the U-Net threshold). This removes the usual confound of comparing pickers reported under different matching rules, which is the paper's stated motivation (Sec. 1).
Micrograph treated as the unit of uncertainty, and the statistics reproduce
Sec. 4.3, Sec. 5.1, Table 3, Fig. 6
Paired stratified micrograph bootstrap, exact two-sided Wilcoxon signed-rank tests and leave-one-micrograph-out ranges are the right tools for 29 correlated test images. Using per-micrograph F1 extracted from Fig. 6, the exact Wilcoxon p-values reproduce to the stated digits (0.00557, 4.88×10⁻⁴, 7.58×10⁻⁶; check 05), and the leave-one-out lower endpoints 0.404 and 0.095 are reproduced by removing the two sparse MS2 micrographs (check 15).
Exceptional internal consistency across tables and figures
Tables 2–4; Figs. 3 and 6
Every number I could recompute agrees: F1 from P and R in all six rows of Table 2 (check 01), all Table 3 and Table 4 differences (check 02), the pooled rows (check 03), mean per-micrograph F1 0.825/0.800 vs 0.786/0.445 from Fig. 6 (check 04), and AP 0.907±0.002 / 0.761 / 0.902±0.007 / 0.253 recomputed from the digitised Fig. 3 curves, whose operating-point markers average exactly to the Table 2 P/R (check 09).
Overlay micrographs are selected by rule (median and lowest seed-mean U-Net F1), including the U-Net's worst case. The selected micrographs are indeed the median and minimum in Fig. 6 (check 17), and the TP/FP/FN markers drawn in the full-image Fig. 5 panels match the caption counts exactly (39/11/10, 30/78/19, 10/1/14, 17/97/7; check 06).
Architecture and data reported precisely enough to reconstruct
Sec. 3.1, 3.2, Table 1
The stated parameter counts 922,895 (width 14) and 1,205,009 (width 16) are reproduced exactly by one architecture reading (check 07), the 2.18 GMAC on a 256² crop follows from it (check 08), and the pixel sizes, diameters and particle counts agree with the EMPIAR API and the CryoPPP README (checks 10, 11). The axis-convention choice among eight flips/transposes is documented rather than hidden (Sec. 3.1).
Weaknesses
The 'tuned' LoG baseline has a very small search space, which bounds how much the headline gaps mean
major · Sec. 3.6; Sec. 5.3 (sparse MS2 micrographs); Table 2 LoG rows; Sec. 6.2
LoG tuning covers only two scale ranges × three low-pass settings (six configurations) plus a threshold grid. The analytic scale optimum σ = r/√2 lies inside the searched range (check 12), so scale is not the problem; but the large MS2 gap (LoG F1 0.388, ΔF1 +0.463) is explained in Sec. 5.3 by LoG placing 240 and 230 picks on aggregate-covered micrographs, exactly the failure that cheap classical remedies address (local background subtraction/flat-fielding, larger low-pass, an upper bound on blob response or local variance, a minimum distance to dark contamination). The paper itself names background subtraction as the remedy for the U-Net's illumination failure (Sec. 6.2). Without such variants, the result is 'a U-Net beats a lightly tuned LoG', and the size of the precision gain — the main claim — may be overstated relative to a reasonable classical pipeline. Fix: add a small set of standard classical pre/post-processing steps to the LoG grid (tuned on train+val as now) and report how ΔP and ΔF1 change, at least on EMPIAR-10075.
Narrow evidence base, especially for EMPIAR-10075, with an unexplained subset selection
major · Sec. 3.1, Table 1, Table 2, Sec. 6.3
The MS2 result rests on 12 test micrographs and 446 particles (4–67 per micrograph); the LoG F1 interval is [0.232, 0.568] and leaving out a single micrograph moves ΔF1 from 0.463 to 0.404 (Sec. 5.1; check 15). The 60 MS2 micrographs are 20% of the 300 labelled in CryoPPP (2,433 of 12,682 particles; check 11), chosen to keep data under 10 GB, but the selection rule is not stated. The pooled row is 96% EMPIAR-10017 particles and adds little independent evidence (check 03). Two entries at one binning cannot support statements about CryoPPP in general. Fix: state how the 60 micrographs were chosen; evaluate the frozen models and thresholds on some or all of the 240 unused labelled MS2 micrographs (no retraining needed, the data limit applies to training storage); ideally add one or two further CryoPPP entries.
Sec. 5.4 U-Net F1 on the illumination micrograph contradicts Fig. 6a
minor · Sec. 5.4 (p. 7); Fig. 6a (p. 8); Sec. 6.2
Sec. 5.4 states that on the non-uniform-illumination micrograph 'the U-Net recall fell to 0.34 (F1 0.46), while LoG reached an F1 of 0.76'. In the vector data of Fig. 6a the corresponding line runs from LoG 0.756 to U-Net (seed mean) 0.242 (check 04); every other Fig. 6 statistic in the text (means, 15/17 and 12/12 counts, overlay F1 values) matches exactly, so the figure is likely right and 0.46 is either a single seed (which would require the other two seeds to average ≈0.13) or a typo. This is the U-Net's only genuine failure case and the leave-one-out maxima (0.089, 0.118) can only be verified once its counts are known (check 15). The main conclusion survives either way, but the text should give the seed-mean value and per-seed range, plus the expert/pick counts as for the other examples.
Questions for the authors
Is the U-Net F1 of 0.46 on the non-uniform-illumination EMPIAR-10017 micrograph (Sec. 5.4) a single seed? Fig. 6a shows a seed-mean of about 0.24; what are the per-seed values and the expert/pick counts on that micrograph?
How were the 60 EMPIAR-10075 micrographs selected from the 300 labelled in CryoPPP, and do the frozen models and thresholds give a similar ΔF1 and ΔPrecision on the 240 labelled micrographs that were not used?
How much of the EMPIAR-10075 precision gap remains if LoG is given local background subtraction or a cap on blob response/local variance to suppress aggregates, tuned on train+validation as now?
Hungarian matching is run on unconstrained Euclidean distance and pairs are then thresholded at one radius (Sec. 4.1). Does a matching restricted to pairs within the tolerance (cost capped at r) change any TP counts?
On which set was the 0.11% nearest-neighbour fraction for EMPIAR-10075 computed (Sec. 3.5)?
Would test F1 change noticeably if the last checkpoint (epoch 60) were used instead of the validation-selected epoch?
Verification
Eighteen checks were run, all with saved scripts and logs. Fifteen are confirmed: F1 and all paired differences in Tables 2–4, the pooled rows, the exact Wilcoxon p-values (reproduced from per-micrograph F1 digitised from Fig. 6), AP and PR-curve claims recomputed from Fig. 3's vector polylines, overlay counts and marker counts in Fig. 5, rule-based example selection, the leave-one-out ranges (as far as reported counts allow), the parameter count and MACs, geometry, EMPIAR/CryoPPP metadata, the LoG scale optimum, and all 32 references (including the full text of [21]). Among these, the ablation check (16) reproduces the numbers but finds the 'within or near the seed spread' wording generous. Three discrepancies were found, none in a headline number: the Sec. 5.4 U-Net F1 0.46 on the illumination micrograph versus 0.242 in Fig. 6a (check 04), an unattainable 0.11% nearest-neighbour fraction for the stated EMPIAR-10075 denominator (check 13), and an inaccurate bias-initialisation rationale (check 14). Because the main results (Tables 2–3, Fig. 3, Fig. 6 aggregate statistics) reproduce exactly, the main conclusion survives these discrepancies. No training or inference could be rerun because no code or data products were released.
Exists (arXiv 2404.10178) but was published in WACV 2025 (doi 10.1109/wacv61041.2025.00381); cite the peer-reviewed version. Citation-format issue only.
Not on Crossref; resolves on arXiv as 2608.28838 (28 Aug 2026) and its full text does support the cited finding (EMPIAR-10081 is the only one of four entries where F1 and resolution rankings agree). Flagged because it is a very recent, non-peer-reviewed preprint that carries the Discussion's argument (Sec. 2.2, Sec. 6.2); give the arXiv identifier.
[27] Loshchilov and Hutter. Decoupled weight decay regularization. arXiv preprint, 2017; [28] Paszke et al. PyTorch. arXiv preprint, 2019.
Both exist (arXiv 1711.05101, 1912.01703) but have peer-reviewed versions (ICLR 2019; NeurIPS 2019). Citation-format issue only; no arXiv identifiers are given for any arXiv reference ([20], [21], [24], [27], [28]).
Suggestions
Evaluate the frozen U-Net models and LoG configuration on the 240 labelled EMPIAR-10075 micrographs not used in the study; this strengthens the weakest part of the evidence at negligible cost.
Extend the LoG search with standard classical steps (flat-field/background subtraction, larger low-pass, an upper cap on blob response or local variance) and report how ΔPrecision and ΔF1 change.
Correct Sec. 5.4 to the seed-mean U-Net F1 on the illumination micrograph (Fig. 6a shows ≈0.24), give per-seed values and TP/FP/FN counts, and make the abstract's 'similar recall' subset-specific.
Release the evaluation code (border mask, NMS, Hungarian matching, bootstrap), the micrograph lists of each split, per-micrograph TP/FP/FN tables, and the trained weights.
Run the MSE ablation with the same three seeds and phrase the result as a seed-based comparison.
State the decoder structure (two 3×3 conv+BN per level after concatenation; 1×1 reduce conv with bias) and revise the bias-initialisation rationale; state the denominator of the nearest-neighbour fractions in Sec. 3.5.
Add one learned picker with a general pretrained model (e.g. crYOLO or Topaz) under the same protocol, or one further CryoPPP entry, to position the result beyond a classical baseline.
Cite peer-reviewed versions of [20], [27], [28] and give arXiv identifiers for arXiv-only references, especially [21].
Ethics and integrity
No concerns. The study uses public EMPIAR/CryoPPP data with no human subjects, and no text in the paper attempts to instruct reviewers. The only affiliation listed is 'Apevon Science'; individual authors, funding and competing interests are not stated and should be added.
Scope of this review
Only the PDF was provided: no LaTeX source, code, trained weights, split lists or per-micrograph tables. Training, inference, LoG tuning and matching could therefore not be rerun, and checks rely on recomputation from reported numbers, on per-micrograph values and curves digitised from the vector graphics of Figs. 3, 5 and 6 (precision ~0.001–0.002), and on public EMPIAR/CryoPPP metadata. Fig. 4 shows only a 512×512 crop, so its marker counts could not be checked. Whether the threshold and checkpoint selections were performed as described, and whether the test micrographs were untouched, cannot be verified without code.
Sec. 1 (last paragraph), Sec. 5.4, Sec. 6.2, Sec. 6.3, Sec. 7
The paper states that precision against incomplete curation is a lower bound, that the U-Net may learn curator habits, that 2D F1 need not track reconstruction quality (supported by [21], whose full text I checked; check 18), that the illumination micrograph is a genuine U-Net failure, and that no learned picker was run. The Conclusion does not overreach beyond these caveats.
No code, model weights or per-micrograph results released
minor · Whole paper; Sec. 3.1 (split seed 20261103), Sec. 4.1
The headline numbers cannot be regenerated: training, threshold selection, LoG configuration and matching could not be rerun. Per-micrograph F1 had to be recovered by digitising vector graphics (~0.001 precision). Given that the paper's value proposition is a reproducible common protocol, releasing the evaluation code (border mask, NMS, Hungarian matching), the split seed's micrograph lists, and per-micrograph TP/FP/FN tables would make it directly reusable by others.
'Similar recall' overstates the EMPIAR-10075 result
minor · Abstract; Table 3; Sec. 5.2; Sec. 7
The abstract and conclusion say the gain arose 'almost entirely from precision at similar recall', but on EMPIAR-10075 ΔRecall is +0.104 [+0.018, +0.176], an interval that excludes zero (Table 3), and recall accounts for a non-trivial share of the +0.463 ΔF1. Sec. 5.2 reports this correctly; the abstract and Sec. 7 should be qualified per subset.
Ablation wording understates the MSE effect on EMPIAR-10017
minor · Sec. 5.5; Table 4
Sec. 5.5 says the MSE F1 changes (−0.008, −0.003, −0.008) are 'within or near the spread across focal seeds (0.002–0.004)'. On EMPIAR-10017 and pooled, −0.008 is 4 seed SDs (check 16); the single MSE run still lies inside a 95% one-seed prediction interval, so the honest statement is 'not distinguishable with one seed'. Conversely, the AP-based ranking claim holds clearly on 10017 (20 SD) but not on 10075 (MSE AP 0.878 inside the one-seed interval [0.867, 0.937]). Running the MSE ablation with the same three seeds would settle both.
Model selection and threshold selection reuse the same small validation set
minor · Sec. 3.4; Sec. 4.2; Fig. 7
Both the checkpoint epoch (34, 34, 51) and the decision threshold are chosen by best mean validation F1 on 8 + 6 micrographs. The test set remains untouched, so this is not leakage, but validation F1 is optimistic and the chosen epoch is noisy with so few images. A sentence on the sensitivity of test F1 to the checkpoint (e.g. last vs best epoch, which Fig. 7 suggests plateaus after ~25 epochs) would help.
Nearest-neighbour percentage for EMPIAR-10075 is arithmetically impossible for the stated denominator
minor · Sec. 3.5 (p. 4)
Sec. 3.5 reports that 0.11% of EMPIAR-10075 particles have a labelled neighbour closer than one radius. With 2,433 labelled particles (Table 1) the attainable values are 0.082% (k = 2) or 0.123% (k = 3); 446 test particles also cannot give 0.11%. It is attainable for the full entry (12,682 particles, k = 14; check 13), suggesting it was computed on all 300 labelled micrographs. The impact is negligible, but the denominator should be stated. (0.28% for EMPIAR-10017 is attainable.)
Architecture description and bias-initialisation rationale are slightly inaccurate or incomplete
minor · Sec. 3.2 (p. 3); Sec. 3.3, Eq. (1)
(i) The parameter count is reproduced only if each decoder level has two 3×3 conv+BN layers after concatenation and the 1×1 reduce convs carry a bias without BN (check 07); Sec. 3.2 does not state this. (ii) The output bias is initialised to a sigmoid of 0.01, described as 'close to the fraction of centre pixels', but the centre-pixel fraction is 6.4×10⁻⁴ (10017) and 4.4×10⁻⁵ (10075), 16× and 227× smaller; 0.01 is closer to the mean target value (0.045, 0.015) and is the standard focal-loss prior (check 14). Neither affects results; both should be corrected for exact reproducibility.
Minor precision of wording: 'every recall', 'median' of 12, citation formats
minor · Sec. 5.2, Fig. 3; Sec. 4.3; References [20], [21], [27], [28]
The U-Net PR curve lies above LoG at every recall except the final vertex on EMPIAR-10017 (recall 0.933–0.939, margin down to −0.05 for one seed; check 09) — negligible but 'every' should be 'every operating point except the last'. For 12 micrographs the 'median-F1' micrograph is the 6th of 12 (check 17); say so. Several references are cited as arXiv preprints although peer-reviewed versions exist ([20] WACV 2025, [27] ICLR 2019, [28] NeurIPS 2019), and no arXiv identifiers are given, including for the 2026 preprint [21] that carries the Sec. 6.2 argument.
Geometry and bookkeeping: 7.08/4.64 A/px after 4x binning; diameters 27.0/58.2 px = 191/270 A; radii 13.5 px = 95.6 A, 29.1 px = 135.1 A; sigma_t = 3.4/7.3 px; splits 59/8/17 and 42/6/12; 15,000 iterations; ~25 min per run
All items consistent. 29.1 px x 4.64 A = 135.02 A vs stated 135.1 A is explained by rounding of the diameter: any unrounded diameter in 58.211-58.250 px rounds to 58.2 px, 270 A and 135.1 A. 70/10/20 of 84 = 58.8/8.4/16.8 -> 59/8/17; 60 x 24.6 s = 24.6 min. Raw pixel sizes 1.77 and 1.16 A assumed here are checked against EMPIAR in check 11. 144 float32 4096^2 micrographs are 9.66 GB, near the stated 9.71 GB total.
Arithmetic with half-unit rounding intervals; feasibility search for the diameter rounding
Data description matches public records: 10017 has 84 labelled Falcon-II micrographs with 49,391 particles, raw pixel sizes give 7.08 / 4.64 A/px after 4x binning, CryoPPP diameters give 27.0 / 58.2 binned px, CryoPPP covers 34 EMPIAR entries, 10075 subset is part of the labelled micrographs
EMPIAR-10017: 84 micrographs, 4096x4096, 1.77 A/px, title 'Beta-galactosidase Falcon-II micrographs ...' -> 7.08 A/px. EMPIAR-10075: 4096x4096, 1.16 A/px -> 4.64 A/px. CryoPPP README: 10017 84 micrographs, diameter 108 px, 49,391 particles (matches); 10075 300 labelled micrographs, 233 px, 12,682 particles. 108/4 = 27.0 and 233/4 = 58.25 binned px (paper 58.2; radius 29.125 px = 135.14 A, explaining the 135.1 A in Sec. 4.1). CryoPPP abstract: 34 EMPIAR datasets. The paper's 60-micrograph MS2 subset is 20% of the 300 labelled micrographs (2,433 of 12,682 particles); the selection rule for the 60 is not described.
Queried the EMPIAR REST API for entries 10017 and 10075, the CryoPPP GitHub README table, and the CryoPPP abstract via Crossref
For a disc of radius r the scale-normalised LoG response peaks at sigma = 2^-1/2 r; the LoG scale range is [a,b] x 2^-1/2 r with (a,b) in {(0.7,1.3),(0.5,1.5)}.
Closed form sigma^2 (LoG*disc)(0) = -(r/sigma)^2 exp(-r^2/(2 sigma^2)), unique stationary point sigma = r/sqrt(2) (exact). Numerically, argmax sigma = 9.518 px vs r/sqrt2 = 9.546 px (r = 13.5) and 20.515 vs 20.577 px (r = 29.1), ratio 0.997 (within the 0.005 r grid step). Selected range (0.7,1.3) maps to sigma in [6.68, 12.41] px and [14.40, 26.75] px, so the analytic optimum lies inside the searched range. The claim is mathematically correct; the search over only two scale ranges and five scales remains a limited baseline-tuning budget (a separate fairness point, not a defect of this statement).
sympy closed form of sigma^2 (LoG * disc)(0) and its stationary point; numerical check summing the analytic LoG kernel over a 4x-supersampled pixel disc for r = 13.5 px (10017) and 29.1 px (10075), 161 sigma values in [0.4,1.2] r.
Only 0.28% (EMPIAR-10017) and 0.11% (EMPIAR-10075) of labelled particles have a labelled neighbour closer than one particle radius.
EMPIAR-10017: 0.28% is attainable for N = 49,391 (k = 136-140) and for the test set N = 10,161 (k = 28). EMPIAR-10075: 0.11% is not attainable for the stated labelled count N = 2,433 (k = 2 gives 0.0822%, k = 3 gives 0.1233%) nor for the test count N = 446 (k = 1 gives 0.2242%), under rounding or truncation; no denominator between 2,300 and 2,599 makes 0.11% attainable. It is attainable for the full CryoPPP EMPIAR-10075 entry (N = 12,682, k = 14, 0.1104%), which suggests the statistic was computed on all 300 labelled micrographs rather than the 60 used, or that the value is misreported. Low impact: either way the fraction is tiny, so NMS at one radius removes few true particles; but the denominator should be stated.
GRIM-style integer feasibility: enumerate integer counts k with round(100k/N, 2) equal to the reported percentage (also truncation) for each plausible denominator N.
Targets use sigma_t = 0.25 r (3.4 and 7.3 px), positives in Eq. (1) are pixels with y >= 0.999, and the output bias is initialised so the initial sigmoid output is 0.01, 'close to the fraction of centre pixels'.
sigma_t = 3.375 and 7.275 px (paper 3.4 and 7.3, rounding OK). The 4-neighbour target is 0.9571 (10017) and 0.9906 (10075), below 0.999, and a second centre would have to be within 0.151/0.325 px to create another positive, so N+ equals the number of centres (simulation: 588 of 588 and 41 of 41). The fraction of centre pixels is 6.38e-4 (10017) and 4.40e-5 (10075), so the 0.01 prior is 16x and 227x that fraction and is not 'close' to it; 0.01 is instead close to the mean target value (0.0457 and 0.0146 by upper bound; 0.0452 and 0.0147 simulated, the latter with 41 rather than 40.55 centres) and is the standard focal-loss prior. Minor: affects only the stated rationale, not results.
Analytic target values at neighbouring pixels; centre-pixel fraction and mean target value per micrograph from 49,391/84 and 2,433/60 centres over the 960x960 scored region; simulation of max-combined Gaussian targets with random centres (min spacing r).
Leaving out any single test micrograph kept dF1 within 0.065-0.089 (EMPIAR-10017), 0.404-0.484 (EMPIAR-10075) and 0.095-0.118 (pooled).
Reconstructed full-set dF1 0.0741 / 0.4641 / 0.1029 (paper 0.074 / 0.463 / 0.102). Removing the reported micrographs gives: 10017 lowest-F1 0.0712 (pooled 0.1021); 10017 median 0.0736 (0.1043); 10075 median 0.4713 (0.1008); 10075 lowest 0.4632 (0.1004); sparse 4-centre 0.4053 (0.0953); sparse 5-centre 0.4088 (0.0957); all inside the stated ranges (tol 0.002), and the sparse micrographs reproduce the stated lower endpoints 0.404 and 0.095 almost exactly. The sparse-micrograph LoG counts imply TP = 2 on both (22/244 = 0.0164, 22/235 = 0.0170, matching Fig. 6b). Caveat: the illumination micrograph's counts are not reported; with the Sec. 5.4 U-Net F1 0.46 the stated maxima (0.089, 0.118) hold only if it has <= 575 expert centres, and with the Fig. 6a value 0.242 only if it has <= 175 (median is 647), so its consistency cannot be fully checked and depends on the 0.46/0.242 conflict (check 04). Note also that removing the lowest-F1 micrograph, where LoG wins, lowers rather than raises dF1, because LoG's 720 FP there weigh more in its micro precision.
Reconstructed aggregate micro TP/picks from Table 2 P, R and Table 1 expert totals; removed each micrograph whose counts are reported (U-Net counts are seed 2, a proxy for the seed mean) and recomputed dF1; for the illumination micrograph (no counts given) scanned expert count 100-1200 and LoG recall 0.70-0.98.
MSE ablation changes test F1 by -0.008/-0.003/-0.008, 'within or near the spread across focal-loss seeds (0.002-0.004)'; AP falls by 0.040 and 0.024, indicating that focal loss mainly improved candidate ranking.
All Table 4 deltas recompute exactly (-0.008, -0.003, -0.008, -0.040, -0.024). In SD units the F1 deltas are 4.0, 0.8 and 4.0 SD, so 'within or near the spread' understates the 10017 and pooled changes (4 SD), although the single MSE F1 values lie inside the one-seed prediction intervals ([0.8691, 0.8889], [0.8311, 0.8709], [0.8681, 0.8879]); with SD 0.0015 the 10017 and pooled MSE values fall just outside. The AP drop on 10017 (20 SD, 0.867 outside [0.8971, 0.9169]) is clearly beyond seed variation, but on 10075 (3.4 SD) the MSE AP 0.878 lies inside the one-seed interval [0.8672, 0.9368], so the AP evidence for the ranking claim rests on one subset. The paper itself notes the ablation uses a single seed.
Recomputed deltas; expressed them in seed-SD units; compared the single MSE run with a 95% t prediction interval for one new focal-loss seed (n = 3, t_0.975,2 = 4.303, half-width 4.968 SD), also with SD reduced by 0.0005 for 3-dp rounding.
Rule-selected overlays are the median and lowest seed-mean U-Net F1 micrographs; the two sparse EMPIAR-10075 micrographs with 4 and 5 centres received 240 and 230 LoG picks while U-Net seeds produced 0-10 FP.
In each case the LoG F1 from the counts matches a Fig. 6 point to 4 dp: 10017 median 0.8362 (U-Net seed mean 0.9162, rank 9/17, the exact median); 10017 lowest 0.0741 (U-Net 0.0353, rank 1/17); 10075 median 0.3822 (U-Net 0.7855, rank 6/12, the lower of the two middle values 0.785/0.808, since 12 is even); 10075 lowest 0.2464 (U-Net 0.5431, rank 1/12). The seed-2 U-Net F1 values (0.9154, 0.0269, 0.7879, 0.5714) differ from the seed means as expected. Sparse micrographs: Fig. 6b LoG 0.0164 and 0.0170 imply exactly TP = 2 (2.001, 1.998) with 240+4 and 230+5 picks+centres. U-Net seed means 0.686 and 0.708 are attainable with 0-10 FP. Minor: for 12 micrographs the 'median-F1' micrograph is the 6th of 12, which should be stated.
Recomputed F1 from the Sec. 5.3/Fig. 4-5 counts, located the matching points among the digitised Fig. 6 seed-mean values (fig6_values.json from check 04), and checked their rank; back-solved LoG TP on the sparse micrographs from Fig. 6b F1 and the stated pick counts.
References exist and support what they are cited for, in particular [6] crYOLO (200-2500 training particles, up to five micrographs/s), [21] (2D F1 ranking matched resolution ranking on only one of four EMPIAR entries), [2] CryoPPP (34 EMPIAR entries, expert curation), [11] review with detection and reconstruction metrics.
All 32 references resolve: 25 directly on Crossref, [17], [19], [20] by Crossref title query (exact titles), [20], [21], [24], [27] on arXiv (2404.10178, 2608.28838, 1904.07850, 1711.05101) and [28] as arXiv 1912.01703. [6] abstract: 'After training the network with 200-2500 particles per dataset ... a speed of up to five micrographs per second', which matches Sec. 2.2 verbatim. [2] abstract: 34 EMPIAR datasets, 2D class and 3D map validation, consistent with Sec. 3.1. [11] abstract: AI pickers compared with ML metrics and 3D reconstruction resolution, consistent. [21] (Itsuji et al., arXiv 2608.28838, 28 Aug 2026) uses EMPIAR-10081/10093/10345/10532, and its full text states that EMPIAR-10081 'is the only entry where the F1 and resolution rankings agree', supporting 'one of four EMPIAR entries'. It is a recent preprint without peer review, yet it carries the Discussion's argument. Citation-format issues only: [20] was published at WACV 2025 (doi 10.1109/wacv61041.2025.00381) but is cited as an arXiv preprint; [27] (ICLR 2019) and [28] (NeurIPS 2019) are also cited as arXiv preprints. (The full-text check of [21] is in work/verify/18b_ref21_fulltext.py.)
tools/check_refs.py (Crossref) for all 32; arXiv API title search for arXiv-only refs (with Semantic Scholar fallback); Europe PMC abstracts for [6], [2], [11]; full text of [21] (arXiv PDF via pdftotext) grepped for the cited finding.
Table 2 F1 values follow from reported precision and recall; U-Net bootstrap CIs 'span 0.12' on both subsets (Sec. 5.1)
All six rows match: e.g. U-Net 10017 claimed 0.879, obtained 0.8792; LoG 10075 claimed 0.388, obtained 0.3877; pooled LoG 0.775 vs 0.7754. U-Net CI widths 0.117 (10017) and 0.122 (10075), consistent with 'span 0.12'. All point estimates lie inside their CIs; the 10017 U-Net CI is strongly asymmetric (0.074 below vs 0.043 above the point estimate).
Recomputed F1 = 2PR/(P+R) for all six rows with the +/-0.0005 rounding interval of P and R (tolerance 0.0005 for LoG, 0.003 for seed-averaged U-Net rows); recomputed CI widths
Table 3 paired differences (dF1 +0.074, +0.463, +0.102 pooled; dP, dR) and Table 4 deltas equal differences of the point estimates; 27 of 29 = 15 + 12
All 9 Table 3 point differences and 5 Table 4 deltas agree within rounding (largest deviation: 10017 dP claimed +0.163 vs 0.895-0.731 = +0.164; pooled dF1 claimed +0.102 vs +0.103). Table 4 focal cells equal Table 2; 15+12 = 27 of 17+12 = 29.
Subtracted rounded Table 2/Table 4 point estimates; tolerance 0.0015
Pooled rows of Table 2 (U-Net 0.892/0.864/0.878; LoG 0.687/0.890/0.775) follow from summing counts of the two subsets with 10,161 and 446 test particles
LoG pooled recomputed P 0.6863 (range 0.6857-0.6869; claimed 0.687), R 0.8904 (claimed 0.890), F1 0.7751 (claimed 0.775). U-Net pooled point recompute 0.8924/0.8641/0.8780 vs claimed 0.892/0.864/0.878. Implied LoG counts: 9,104 TP of 12,454 picks (10017) and 340 TP of 1,308 picks (10075). Pooled values are dominated by EMPIAR-10017 (96% of particles), so the pooled row adds little independent information.
Reconstructed TP = R*N and picks = TP/P per subset, pooled, and sampled P,R within rounding intervals (LoG) or +/-1 seed SD (U-Net), 200,000 draws
Per-micrograph F1 (Fig. 6) supports 15/17 and 12/12 U-Net-better counts, mean per-micrograph F1 U-Net 0.825/0.800 vs LoG 0.786/0.445, and Sec. 5.4: on the non-uniform-illumination micrograph 'the U-Net recall fell to 0.34 (F1 0.46), while LoG reached an F1 of 0.76'
Confirmed: 15 of 17 and 12 of 12 lines favour U-Net; mean per-micrograph F1 U-Net 0.825 / 0.800 and LoG 0.786 / 0.445 (all four match the text exactly); medians 0.916/0.830 (10017) and 0.797/0.490 (10075); Fig. 4/5 caption values match the corresponding lines (e.g. 10017 median micrograph LoG 0.836 vs caption 0.84; 10075 median LoG 0.382 vs 0.38). Discrepancy: the illumination micrograph is drawn at LoG 0.756 -> U-Net (seed mean) 0.242, whereas Sec. 5.4 states U-Net F1 0.46. 0.46 would require the other two seeds to average about 0.13 if it is a single seed; the text does not say it is. The lowest 10075 U-Net seed-mean F1 is 0.543 (Fig. 5d, seed 2: 0.57). Saved values to work/verify/fig6_values.json.
Extracted the 29 per-micrograph line segments and the axis ticks from the vector graphics of page 8 (pdftocairo -svg) and mapped them linearly to F1 (precision of extraction ~0.001)
Exact test on Fig. 6 data: 10017 W- = 20 (negative ranks 3 and 17), p = 0.00557 (claimed 0.0056); 10075 W- = 0, p = 4.88e-4 (claimed 4.9e-4); pooled W- = 30, p = 7.58e-6 (claimed 7.6e-6). W- = 20/0/30 are the only values reproducing the claimed p, so the paper used the exact test; the normal approximation would give 0.0075, 0.0022, 5.0e-5.
Exact signed-rank null distribution (enumeration) and scipy.stats.wilcoxon on the 29 per-micrograph F1 differences extracted from Fig. 6 (check 04)
Overlay counts give the stated F1 (0.92, 0.84, 0.03, 0.07, 0.79, 0.38, 0.57, 0.25), TP+FN is identical across methods, and Fig. 5 shows exactly the stated TP/FP/FN markers
All 8 F1 values match (e.g. 10017 median U-Net 0.9154 -> 0.92; 10017 lowest U-Net 0.0269 -> 0.03; 10075 lowest U-Net 0.5714 -> 0.57). TP+FN agree between methods: 725, 112, 49, 24. Fig. 5 markers: (a) 39/11/10, (b) 30/78/19, (d) 10/1/14, (e) 17/97/7 TP/FP/FN, exactly as captioned; expert outlines in (c) 49 and (f) 24 = TP+FN.
F1 = 2TP/(2TP+FP+FN) from caption counts (tolerance 0.005); counted vector marker glyphs by stroke colour and image placement in the SVG of page 8 (Fig. 5 shows full micrographs; Fig. 4 only the central 512x512 so not countable)
The U-Net with widths 14/28/56/112 and bottleneck 112 has 922,895 trainable parameters; base width 16 would give 1,205,009
Exactly one reading reproduces both numbers: bias-free 3x3 convs each with BN (2 params/channel), 1x1 reduce convs with bias and no BN, two 3x3 conv+BN per decoder level after concatenation, bottleneck 8w, 1x1 head with bias: w=14 -> 922,895, w=16 -> 1,205,009 (exact). The decoder's two 3x3 convs per level are implied by the count but not stated in Sec. 3.2.
Analytic parameter count over 32 readings of the architecture description (conv biases, BN after reduce conv, bottleneck width, number of 3x3 convs per decoder level)
The forward pass requires 2.18 GMAC on a 256 x 256 crop
Obtained 2.1800 GMAC (claimed 2.18) when the 1x1 reduce conv is applied after bilinear upsampling, as Sec. 3.2 describes; reducing before upsampling would give 2.113 G. Tolerance 0.01 G. Base width 16 would need 2.85 G.
Analytic conv MAC count for the architecture variant that reproduces 922,895 parameters (check 07), same padding
Fig. 3: U-Net PR curves above LoG at every recall the U-Net reaches; best precision at recall >= 0.8 0.93-0.94 vs 0.78 (10017) and 0.87-0.89 vs 0.26 (10075); AP 0.907 +/- 0.002 vs 0.761 and 0.902 +/- 0.007 vs 0.253; LoG max recall 0.98/0.99 vs 0.94/0.97 at precision <= 0.72/0.24; frozen operating points equal Table 2
AP from curves: U-Net 10017 seeds 0.909/0.906/0.908, mean 0.907, SD 0.002 (claimed 0.907 +/- 0.002); LoG 0.761 (0.761); U-Net 10075 0.903/0.895/0.909, mean 0.902, SD 0.007 (0.902 +/- 0.007); LoG 0.253 (0.253). Best precision at R >= 0.8: 0.941/0.939/0.930 vs LoG 0.782 (10017); 0.890/0.869/0.885 vs 0.261 (10075). Max recall U-Net 0.939 / 0.973, LoG 0.979 / 0.987; LoG precision beyond the U-Net max recall <= 0.655 / 0.176. Mean of the three U-Net operating-point markers = (R 0.864, P 0.895) and (0.866, 0.838), exactly Table 2; LoG markers (0.896, 0.731), (0.762, 0.260). Dominance holds everywhere on 10075 (min margin +0.023) and on 10017 except at the last vertex of each U-Net curve (recall 0.933-0.939, where the U-Net curve drops to all candidates; margin down to -0.050 for one seed), a negligible qualification of 'every recall'.
Digitised the 8 vector polylines and 8 operating-point markers of Fig. 3 from the SVG of page 6 using the axis ticks; recomputed AP as area under the precision envelope, max recall, best precision at R >= 0.8, pointwise dominance on a 400-point recall grid; digitisation precision ~0.002