A substantial, released and verifiably implemented agent harness, but the benchmark evidence is single-run point estimates with sub-noise margins and no ablations, and the RSI case study tunes data on the very benchmarks it reports, with some Table 2 scores inconsistent with the stated task counts.
Soundness
2/4
Presentation
3/4
Contribution
3/4
Confidence
4/5
✓ 12 confirmed✕ 2 discrepancy◐ 3 inconclusive○ 1 not checkable
Summary
The paper presents CoworkOS, an open-source (MIT) harness for long-horizon LLM agent work. A Main agent keeps strategic authority (what to do, whom to delegate to, what to accept), a Coordinator Runtime manages durable execution state (queues, leases, acknowledgements, Inboxes), and Subagents execute delegated work synchronously or asynchronously, with peer messaging routed through the Coordinator (Sec. 3.1). A shared application layer lets humans and agents edit the same documents, slides, CAD models and Blender scenes under a revision-checked commit rule (Eq. 7) with change summaries of other actors' edits (Eq. 8). Background Jobs with watchers, hierarchical workspaces and a persistent semantic memory are meant to make work resumable after interruption (Sec. 3.3).
Empirically, the paper runs the same backbone (GLM-5.3) inside four harnesses (OpenHands, Codex, DeepSeek Harness, CoworkOS) on 12 benchmarks in four domains and claims CoworkOS is top-two in every domain average and has "the strongest overall performance" (Table 1, Sec. 4.1.2). It adds 24 qualitative application cases (Sec. 4.2, App. A.2) and a case study in which CoworkOS, driven by GLM-5.3, runs five rounds of automated post-training on Qwen3.6-35B-A3B, raising an eight-benchmark average from 33.19 to 42.95 (Sec. 4.3, Table 2).
Strengths
Released implementation whose core mechanisms work as described
Sec. 3.1.2, 3.2.1–3.2.2, 3.3.1; src/applications/resource-store.ts:469, src/applications/context-builder.ts:88, src/runtime/goal-management.ts:241, src/runtime/coordinator-runtime.ts:465-540
The MIT-licensed TypeScript code base (~265 source files, ~400 test files) implements the mechanisms Sec. 3 describes. I built it (Node 22.19.0) and drove the compiled services directly. A stale-base commit is rejected with revision_conflict and the content is left unchanged (Eq. 7). Change summaries exclude the agent's own edits but show them to other agents (Eq. 8). A continuing goal that has neither a watcher nor a review time is refused (Sec. 3.3.1). Peer messages are queued into the recipient's Inbox and refused for completed targets (Sec. 3.1.2). The relevant repo test files pass: 34/34, 23/23, 2/2 and 11/11 (checks 13–16).
Clear treatment of durability and authority, an often-neglected engineering problem
Sec. 3.1.1–3.1.2 (Eqs. 3–6), Sec. 3.3.1
The split between strategic authority (Main) and execution management (Coordinator) is stated precisely. So are the durability primitives: leases, acknowledgements after entry into saved context, source cursors committed together with events, and bounded-wait decision requests with fallbacks. These address real failure modes of long-running agents (lost messages, duplicated work, silent overwrites) that most agent papers leave out.
Broad, same-backbone benchmark comparison with internally consistent tables
Sec. 4.1, Table 1, App. A.1
Twelve benchmarks across Tool Use, Productivity, Search and ML Coding are run with one fixed backbone and decoding configuration for all four harnesses. Per-benchmark protocols are documented, including an explicit GDPVal Elo formula and averaging rule. All 12 domain averages, all bold/underline marks and every rank statement in Sec. 4.1.2 reproduce from the table cells (check 03). The GDPVal Elo values and Productivity averages are mutually consistent, with exactly one admissible tie-adjusted win count per harness (check 04).
Unusually transparent RSI record
Sec. 4.3, Fig. 8, Table 2 and lineage table (App. A.3, p33)
Table 2 reports every benchmark in every round. The lineage table gives trajectory counts, sources, epochs, multipliers and steps, and the Fig. 8 caption states that the per-round changes are concurrent interventions rather than causal effects. The headline 33.19 → 42.95 (+9.76), all eight Δ values and 23 narrative statements reproduce exactly from the table (checks 01, 02), as do all data-composition sums and percentages (check 09).
Weaknesses
Headline ranking claims rest on single-run margins far below sampling noise
major · Table 1 (p15); Sec. 4.1.2; Abstract; Contribution 4 (p4)
No run counts, seeds, confidence intervals or paired tests are reported. Several of CoworkOS's "best" results are tiny on small task sets: ML Coding average 56.26 vs 56.14, MLE-Bench-Lite 76.90 vs 76.70 (22 tasks), MLS-Bench 35.62 vs 35.57 (30 tasks), WideSearch 82.47 vs 82.05. CoworkOS also trails DSH in Tool Use (55.22 vs 55.31). Using the task counts in App. A.1, every CoworkOS-minus-best-competitor gap has ∣z∣<1. A normal bootstrap gives P(CoworkOS first in ML Coding) ≈ 0.46–0.51 and P(top two in all four domains) ≈ 0.58–0.79 (check 06). The ordering itself reproduces (check 05), but "strongest overall" and "best on X" are not supported as stated. Much of the overall lead over DSH comes from a single benchmark: DSH's WideSearch score is 72.48, against about 82 for the other three harnesses. Fix: repeat the runs (≥3 seeds or resampled task orders), report CIs or paired per-task tests, and soften the claims to "comparable" where differences are not significant.
No ablations: gains cannot be attributed to the proposed architecture
major · Sec. 3 vs Sec. 4.1; Sec. 3.3.1
Table 1 compares whole systems that differ in prompts, tools, skills and control flow. No experiment removes or varies the Coordinator, async delegation, peer channel, resource protocol, memory reconstruction or application layer. Many of the benchmarks (BrowseComp, OfficeQA Pro, τ³-Banking) probably do not exercise the cowork or human-in-the-loop features that are the paper's motivation. Interruption recovery is a headline property, yet there is no failure-injection or resume experiment. Fix: ablate the key components on a subset of benchmarks, and add a controlled interruption/resume test and a human-in-the-loop co-editing task.
Questions for the authors
How many runs per benchmark and harness underlie Table 1? Can you report per-benchmark standard deviations or paired per-task significance tests against DSH?
What exactly were the baseline configurations (harness versions, prompts, tool and skill access, timeouts)? Did Codex and DSH have access to the same skills and application/MCP integrations as CoworkOS?
Were all RSI rounds scored on the same task sets? Why are Automation 39.49/44.15, OfficeQA 56.14/62.11 and τ³ 18.57 in rounds R3/R5 not attainable as k/600, k/133 and k/97?
How does the RSI result hold up on benchmarks never used for per-round diagnosis or data targeting, and against a non-adaptive control that uses the same number of GLM-5.3 trajectories?
What batch size and packing were used in each RSI round, and how large was the 'Old' replay pool in R2–R5?
Which components (Coordinator, async delegation, memory reconstruction, application layer) drive the Table 1 differences? Have you run any ablation?
Who or what is the 'fixed baseline' opponent in the GDPVal Elo comparison, and who judged the pairs?
Verification
18 checks were run: 12 confirmed (including the reference check), 2 discrepancies, 3 inconclusive and 1 not checkable. Arithmetic in both main tables is clean. The Table 1 domain averages, marks and rank statements reproduce, as do the GDPVal Elo/win-rate consistency, the RSI headline 33.19 → 42.95 (+9.76), the 23 RSI narrative statements and all data-composition sums. I built the released code and drove four core mechanisms (revision-checked commits, own-edit-excluding change summaries, the goal wait rule and peer messaging), and they behave as described. The discrepancies are granularity (GRIM-style) mismatches. Five RSI cells, all in rounds R3/R5, and several Table 1 cells (all of OfficeQA Pro) are not attainable on the stated task counts, which points to undisclosed changes in denominator or scoring. Uncertainty analysis shows that every CoworkOS winning margin is within one standard error. The RSI gains concentrate on benchmarks that received targeted data, and the RSI training schedule is hard to reconcile. The benchmark numbers themselves could not be regenerated because no evaluation code or per-task results are released.
Used as one of the eight RSI evaluation benchmarks (Appendix A.3, Fig. 8(b), '+9.07' R4->R5 claim) but has no citation; no BFCL entry exists in citation.bib.
GLM-5.3 / Qwen3.6-35B-A3B (backbone and base models)
Models central to the experiments (backbone for Table 1, base model for RSI) are named but not cited; no bib entry or model card reference.
mlsbench2026 (Lyu et al., MLS-Bench, arXiv 2605.08678)
Exists and matches title, but the abstract does not mention a Lite subset or a 30-task split; the subset used in Table 1 is not documented by the cited source (unverified rather than contradicted).
deepseek2026harness (DeepSeek Harness: Everything Is a Plugin)
Cited only as an 'Official source repository' without URL/version; a main baseline in Table 1 whose version/configuration cannot be identified from the citation.
harveylab2026 (Harvey LAB: The Legal Agent Benchmark)
Cited only as an 'Official source repository' without URL/version; the benchmark subset and N behind Table 1/RSI scores cannot be identified from the citation.
young2025harnesses (Effective Harnesses for Long-Running Agents, Anthropic Engineering)
Blog post not resolvable by Crossref/arXiv (not machine-verifiable); also listed in the bibliography but never cited in the text.
Real works (arXiv/ICLR) but present in the bibliography without any in-text citation.
Suggestions
Report ≥3 runs per cell with standard deviations or bootstrap CIs, and use paired per-task tests for the CoworkOS-vs-DSH comparisons; rephrase "strongest overall" accordingly.
Add component ablations (no Coordinator/async delegation, no memory reconstruction, no application layer) on 3–4 representative benchmarks.
Add an interruption/resume experiment (kill the process mid-task, then resume) and a human co-editing task that actually exercises the application layer and Eq. 7 conflicts.
For RSI, add held-out benchmarks never used for diagnosis, plus a non-adaptive control with the same trajectory budget, and run at least two seeds for the final round.
State the exact denominator and scoring rule for every cell of Tables 1 and 2, and explain any excluded tasks.
Document baseline harness versions, prompts, tool/skill access and limits, and report tokens, cost and wall-clock per harness.
Release the evaluation harness, per-task outputs (the referenced benchmark-results.json), RSI trajectories, BM25 filter code and SFT configs.
Show all 12 benchmarks in Fig. 2(c), or flag the omissions in the caption, and identify the GDPVal Elo opponent and judge.
Add a related-work section on agent harnesses and multi-agent runtimes, cite BFCL v4 and the models used, and add a limitations section.
Ethics and integrity
No integrity concerns beyond the reporting issues above. No prompt-injection or reviewer-directed text was found in the LaTeX source. The author-contributions section is incomplete (a TODO comment remains in the source), and the paper is credited to 'CoworkOS Team' without individual authors. The system can control desktop applications autonomously; the paper notes that this is user-pausable (Sec. 3.2.3), but it does not discuss safeguards for unattended long-running Jobs.
Scope of this review
No benchmark evaluation harness, per-task results, baseline configurations, RSI training data or training configs are released, so Tables 1 and 2 were checked only for internal arithmetic and granularity consistency, not regenerated. The benchmarks and the GLM-5.3 backbone were not run (no API access or compute budget for this review), and the repo's end-to-end smoke scripts need model API credentials. Four core runtime mechanisms were verified by building the code and calling the compiled services. The full unit suite was partially run (1227 ok / 42 not ok, with failures in web-UI tests that lack generated assets, plus one hanging live-UI test). The 24 qualitative cases were not re-executed.
Qualitative appendix states the limits of each observation
Sec. 4.2, Fig. 6, App. A.2
For each of the 24 cases, App. A.2 says what the observation does not establish (for example that a packing case is "not used to rank packing algorithms"), and it includes counter-examples where CoworkOS is not better. This candour is rare and helps readers calibrate.
RSI case study tunes training data on its own evaluation benchmarks
major · Sec. 4.3.3–4.3.4 (p19–21); Fig. 8; Table 2 and lineage table (p33)
Six of the eight RSI evaluation benchmarks received benchmark-specific trajectories selected after per-round diagnosis on those benchmarks: e.g., R3 has 437 of 558 trajectories (78.3%) from AutomationBench, and R5 has 203 of 332 (61.1%) from BFCL. Fig. 8 itself annotates "Further tailor data to key benchmarks". The targeted benchmarks account for 8.50 of the 9.76-point gain; the two untargeted ones (Harvey +7.75, GDPVal +2.34) average +5.05 (check 12). There is no held-out benchmark, no control arm (same data budget from a fixed pipeline, or untargeted GLM-5.3 trajectories), and only one training run per round. The improvement also comes from a stronger teacher (GLM-5.3) distilling into a different, smaller model, so "self-improvement" and "cross-task generalization" (Sec. 4.3.3) are overstated. Fix: report held-out benchmarks never used for diagnosis, add a non-adaptive control, and frame the result as controller-driven distillation.
Some RSI scores are not attainable on the stated task counts, all in rounds R3 and R5
Granularity checks show that 13 of the 18 binary-scored Table 2 cells equal k/N for the stated N, but five do not, and all five fall in rounds R3 and R5. Automation 39.49 and 44.15 are not k/600 (nearest 39.50, 44.17). OfficeQA 56.14 and 62.11 are not k/133 (nearest 56.39, 62.41). τ³-Banking 18.57 is not k/97 (nearest 18.56). These values fit smaller denominators (e.g., 13/70, 64/114, 59/95), which suggests some rounds were scored on subsets of tasks (for example after dropping failed runs) or under a different rule. Neither is disclosed (check 08). Each cell is at most 0.30 points from an attainable value, so the +9.76 headline survives in magnitude. However, the rounds may not be scored on the same task set, and this undermines the round-to-round comparisons that the narrative relies on (for example "recovers to 39.22" at R3). Similar granularity mismatches occur in Table 1: no OfficeQA Pro value fits k/133 at two decimals, and OpenHands' τ³ score of 40.20 is not k/97 (check 07). Fix: state the exact denominator and scoring rule for every cell, and release per-task results.
Baseline configuration and cost undocumented
major · Sec. 4.1.1 'Harnesses' (p14); App. A.1; references deepseek2026harness
The paper says only that the three baselines were run "under the above task and model configurations" (Sec. 4.1.1). Harness versions, system prompts, tool and skill sets, timeouts and retry policies are not given, and the reader cannot tell whether baselines had access to the same skills and application integrations. Codex is tuned for OpenAI models, and its very low scores with GLM-5.3 (JobBench 23.42, Harvey 12.70, GDPVal Elo 925) suggest a possible configuration handicap. Tokens, cost and wall-clock time, which matter for long-horizon harnesses, are not reported. DeepSeek Harness is cited only as a repository, with no version.
Main results cannot be regenerated from released artifacts
major · Repo RUC-NLPIR/CoworkOS; figures/benchmark-results-table.tex; Sec. 4.1, 4.3
The repository contains no benchmark harness and no per-task outputs. The benchmark-results.json cited in the LaTeX source of Table 1 is absent. There are no baseline configurations, no RSI training/SFT configs, no BM25 filtering code and no trajectories; 11 of the 13 evaluation benchmarks are never mentioned in the repo (check 17). Tables 1 and 2 can therefore only be checked for internal consistency. Fix: release the evaluation scripts, raw per-task results and the RSI data/configs.
RSI training schedule is internally hard to reconcile
minor · Lineage table and 'Training' paragraph (App. A.3, p33); Sec. 4.3.4
Trajectories × multiplier × epochs / steps gives about 20 samples per step in R0–R1, but 324 in R2 and 2,232 in R3, a 110× range (check 10). This is coherent only if batch size or packing changed by two orders of magnitude, which is not reported. The text says R2 "stops replaying seed and previously trained data", but the table lists an "Old 1x" pool for R2–R5 without giving its size. In addition, the BM25 near-duplicate threshold of 0.8 (App. A.3) is not interpretable without a normalisation, because raw BM25 scores are unbounded, and no train/eval overlap statistics are given.
Fig. 2(c) silently drops two of the twelve benchmarks
minor · Fig. 2(c) and caption (p3); App. A.1 'GDPVal' (p26)
The overview radar shows 10 benchmarks. Its plotted values match Table 1 (check 11), but it omits GDPVal and OfficeQA Pro, and OfficeQA Pro is where CoworkOS ranks 3rd of 4. The caption's "10 benchmarks" is accurate, but the omission is not explained, while the abstract and text speak of 12. The opponent of the GDPVal pairwise comparison (the "fixed baseline of 1000") is never identified.
Qualitative comparison is unblinded and unprotocolled
minor · Sec. 4.2, Fig. 6, App. A.2
The 24 cases are judged by an unnamed reviewer, presumably an author, without blinding, a rubric or inter-rater agreement. Case selection, the number of attempts per harness and the reason for omitting OpenHands are not stated. The bold main-text sentences in Sec. 4.2 claim more than the cautious appendix supports.
Thin related work, unused formalism, missing citations and limitations
minor · Sec. 1–2; Sec. 5; citation.bib
There is no related-work section. Closely related harness work (SWE-agent, Anthropic's long-running-agent harness note) is in the bibliography but never cited in the text. BFCL v4, an RSI evaluation benchmark, has no citation, and the GLM-5.3 and Qwen3.6 models are not referenced. The task formalism of Sec. 2 (Eqs. 1–2, including the budget B) is never used in the evaluation, and budgets and costs are not measured. There is no limitations section.
Lineage table epochs, sampling multipliers and steps (R0 66, R1 69, R2 10, R3 24, R4 22, R5 20) describe a coherent training schedule
Implied samples per step: R0 20.4, R1 23.0, R2 324.0, R3 2232.0, R4 1976.7, R5 1593.6 (110x range; 20-558 even if epochs are not multiplied in). This is only coherent if the global batch size or packing changed by about two orders of magnitude between R1 and R3, which is not reported; per-round training compute therefore cannot be compared. In addition, the text says R2 'stops replaying seed and previously trained data', while the table lists 'Old 1x' for R2-R5 without stating the size of the 'Old' pool. Batch size and replay-pool size would be needed to settle this.
Implied sample presentations per optimisation step = trajectories x multiplier x epochs / steps, per round
Fig 2(c) presents CoworkOS 'across 10 benchmarks in four domains' (Table 1 has 12); plotted values match Table 1
Panel (c) shows 10 axes; all 10 CoworkOS values printed there match Table 1 exactly (64.34, 49.48, 51.83, 59.88, 22.74, 67.14, 82.47, 81.80, 76.90, 35.62). The omitted benchmarks are GDPVal (CoworkOS ranks 1st, but Elo is on a different scale) and OfficeQA Pro (CoworkOS ranks 3rd of 4). The caption is accurate; the omission is not flagged in the caption, and the abstract/text refer to 12 benchmarks.
Extracted words with bounding boxes from the figure source PDF (figures/cowork_v4.pdf), restricted to panel (c), and matched labels and numbers against Table 1
RSI gains 'span function calling, complex tool execution, document question answering, and professional tasks' and provide 'positive evidence of cross-task generalization'
Six of eight evaluation benchmarks received benchmark-specific training trajectories (AB, OQ, tau3 in R3-R5; SB in R3-R4; BFCL 203 and JobBench 2 in R5). Mean R5-R0 gain: targeted +11.33 (+14.03 excluding JobBench), untargeted (Harvey, GDPVal) +5.05; targeted benchmarks account for 8.50 of the 9.76-point average gain. Fig 8 itself annotates 'Further tailor data to key benchmarks' and 'BFCL plateau prompts call-focused data'. Harvey (+7.75) and GDPVal (+2.34) did improve without targeted data, which is some evidence of transfer, but with no held-out benchmark, no control arm and single runs, 'cross-task generalization' cannot be separated from benchmark-directed data tailoring and teacher distillation.
Classified the eight RSI benchmarks as targeted/untargeted using the lineage table, compared R5-R0 gains; extracted Fig 8 annotation text
Resource protocol (Eq. 7): a commit whose base revision differs from the current revision is rejected as a conflict instead of overwriting; accepted commits increment the revision and record the actor
Stale commit rejected with code revision_conflict {expectedRevision:0, currentRevision:1}; content unchanged ('USER line'); fresh commit accepted -> revision 2 with actor agent:main; history [[1,user],[2,agent]]. Repo test file: 34/34 pass, including 'concurrent writes on one baseline: exactly one wins and the other gets 409'. Guard at resource-store.ts:469.
Built the released repo with Node 22.19.0 (npm ci --ignore-scripts; npm run build:cli) and drove the compiled ApplicationService: user edit at base 0, agent edit with stale base 0, agent edit at base 1; ran test/application-resource-store.test.js
A continuing goal must have an active observation source or an explicit review time; the runtime rejects a wait that provides neither
'wait' and 'continue' with neither watcher nor nextReviewAt are rejected ('持续目标需要活动监控或明确的下次复盘时间...'); 'wait' with nextReviewAt +2 min accepted (revision 2). Repo tests 23/23 pass.
Drove compiled GoalManagement.progress on a fresh background goal without watchers; ran test/long-term-goals.test.js and test/goal-tools.test.js
Peer (Subagent-to-Subagent) messages go through the Coordinator into the recipient's Inbox with a message id and queued status; messages to terminated targets are rejected or marked undelivered
Message to live target returned {messageId: 'peer-message_...', status: 'queued'}; receiver Inbox holds one item kind=peer_message, state=ready, sender=sender. Message to completed target rejected ('peer call done-call is already completed'). Repo test 2/2 pass, including 'a Subagent peer message reaches another Subagent's next real model turn' (scripted model provider).
Drove compiled CoordinatorRuntime.sendPeerMessage against a scheduler queue with a leased sender, a queued target and a completed target; inspected the receiver Inbox; ran test/subagent-message-loop.test.js
Application context (Eq. 8) includes intervening changes by other actors since the agent's last acknowledged revision and excludes the agent's own edits
After edits rev1 user, rev2 agent:main, rev3 user: 'main' sees [rev1 user, rev3 user] (own edit excluded), 'reviewer' sees all three. Repo test file 11/11 pass. Filter at context-builder.ts:88 is by (actor type, actor id).
Drove compiled ApplicationService.activeContext with interleaved user/agent edits; queried as the editing agent ('main') and as a different agent ('reviewer'); ran test/application-context.test.js
Table 1 and Table 2 can be regenerated from the released materials (code link given as the paper's artifact)
No benchmark harness, per-task outputs, result file (benchmark-results.json is cited in the LaTeX source but absent), baseline configurations (OpenHands / DeepSeek Harness never mentioned), RSI training/SFT configs, BM25 filtering code, seed or generated trajectories are released. 11 of the 13 evaluation benchmarks have zero mentions in the repo; Qwen3.6 appears only in the README sentence repeating the 33.19 -> 42.95 headline. The headline numbers can only be checked for internal consistency (checks 01-12), not regenerated. Evaluation scripts, per-task results and training records would be needed. Main entry points (smoke:multi-subagent, smoke:coding) require COWORK_API_BASE/COWORK_API_KEY and could not be run. Full unit suite (node scripts/test.mjs --skip-build, log /tmp/fullsuite.log, stopped by the 1500 s timeout, exit=124): 1227 ok / 42 not ok out of 1269 reported tests; test/live-ui.test.js hung with no output for more than 15 min. The failures are web-UI tests missing generated assets (web/assets/math/generated/katex.js, cad/generated/model.js) because the web build was skipped; they are unrelated to the paper's claims.
Scanned all 1,769 non-.git files of the released repository for benchmark names, baseline harness names, result files, SFT/training configs and BM25 filtering code
Reference list: cited works exist and support the citations (benchmark suite, baselines, related work)
All 16 arXiv-id references resolve, with title similarity of at least 0.94. 12 of 14 venue papers were found on arXiv by title search with similarity of at least 0.99. The AgentBench and tau-bench title searches returned nothing, probably because of the query string; both are known ICLR papers. Abstract spot-checks support the cited usage: OfficeQA Pro has '133 questions' and Treasury; tau-Knowledge mentions banking; Workspace-Bench mentions 'Lite'. The MLS-Bench abstract does not mention a Lite subset. Gaps: BFCL v4, one of the 8 RSI benchmarks, has no citation; the GLM-5.3 and Qwen3.6 models are not cited; 5 bib entries are never cited in the text; DeepSeek Harness and Harvey LAB are cited only as repositories (not machine-checkable).
Summarised reference lookups (Crossref/arXiv title and id resolution) and spot-checked abstracts of load-bearing references against their use in the paper
RSI headline: 8-benchmark average rises 33.19 (R0) -> 42.95 (R5), +9.76; all round averages and Delta R5-R0 values follow from Table 2 cells
Averages obtained 33.1875/36.9400/35.9012/39.2150/40.7125/42.9488 vs claimed 33.19/36.94/35.90/39.22/40.71/42.95 (all within 0.005; R3 39.215 sits exactly on the rounding boundary). All 8 deltas match exactly (+33.15, +7.35, +6.47, +8.25, +7.75, +14.91, +2.34, -2.13); headline +9.76 reproduced.
Recomputed six row means and eight R5-R0 differences from the Table 2 cells
RSI narrative statements: R5 > R0 on 7/8; R5 at six-round max on 6/8 incl. ties; average rises R3->R5; Automation +33.15 and SkillsBench +14.91 largest; BFCL +9.07 R4->R5 and below baseline R1-R4; Fig 8 deltas; Table 2 bold/underline marks
All 23 checks pass: only JobBench ends below R0 (22.66 -> 20.53); R5 is the maximum on Automation, BFCL, OfficeQA, tau3 (tie), Harvey, SkillsBench; BFCL R4->R5 = +9.07; R1 +3.75, R2 -1.04, R3 AB +8.66, R4 OfficeQA +3.26; all bold/underline marks consistent with the 'distinct scores' rule.
Programmatically re-derived 23 statements and marks from Table 2 values
Table 1 domain averages (Tool Use, Search, ML Coding) are equal-weight means; bold/underline marks correct; Sec 4.1.2 rank statements (top-2 in every domain average; best on SkillsBench, JobBench, WideSearch, MLE-Bench-Lite, MLS-Bench; joins DSH in top two on tau3/AutomationBench; top two on Harvey, WorkspaceBench-Lite, GDPVal, BrowseComp)
All 12 recomputed averages match within 0.005 (e.g., CoworkOS Tool Use 55.2167 vs 55.22, ML Coding 56.2600 vs 56.26; DSH Tool Use 55.3067 vs 55.31). All 16 bold/underline marks consistent. All rank statements hold. CoworkOS is 3rd of 4 on OfficeQA Pro (65.90 vs 66.30 best), which the text phrases as 'close to that of OpenHands and Codex' - accurate but not a top-two result.
Recomputed 12 domain means (rounding after aggregation) and per-row rankings for all 16 rows
GDPVal Elo (1122, 925, 1157, 1164) and Productivity averages (55.27, 40.11, 56.13, 57.54) are mutually consistent under E=1000+400log10[(W+T/2)/(L+T/2)], W+L+T=220, win rate 100(W+T/2)/220 in the average
Exactly one consistent value per harness: OpenHands W+T/2=147.0 (win rate 66.82%), Codex 86.5 (39.32%), DSH 156.5 (71.14%), CoworkOS 158.5 (72.05%). Elo and Productivity averages are internally consistent. Note CoworkOS vs DSH differ by 2 tie-adjusted wins out of 220, and the identity of the 'fixed baseline' opponent is not described.
Enumerated all half-integer W+T/2 in [0,220]; intersected the sets compatible with the rounded Elo and with the rounded Productivity average
CoworkOS has 'the strongest overall performance among the four evaluated harnesses' and ranks top-two in every domain average
As point estimates CoworkOS leads on every aggregate: mean of domain averages 62.79 vs DSH 61.05 (+1.74), OpenHands 58.37, Codex 52.61; mean of 12 benchmarks 60.85 vs DSH 59.34 (+1.50); row wins 6/12 vs DSH 3; mean rank 1.58 vs DSH 2.33. The overall margin over DSH is largely driven by DSH's WideSearch score (72.48 vs ~82 for the others). Statistical support is assessed in check 06.
Computed mean of domain averages, mean over 12 benchmarks (GDPVal as win rate from check 04), row wins and mean rank from Table 1
Table 1 margins (e.g., ML Coding 56.26 vs 56.14, MLE-Bench-Lite 76.90 vs 76.70, MLS-Bench 35.62 vs 35.57, WideSearch 82.47 vs 82.05, Productivity 57.54 vs 56.13) reflect real performance differences (no variance reported)
No reported number is contradicted, but every CoworkOS-minus-best-competitor gap has |z| < 1: MLE-Bench-Lite +0.20 (z=0.02-0.03, n=22), MLS-Bench +0.05 (z~0.00, n=30), WideSearch +0.42 (z=0.11-0.22), GDPVal +0.91 pp win rate (z=0.21), SkillsBench +1.04 (z=0.14-0.27), JobBench +3.73 (z=0.43-0.86). Bootstrap: P(CoworkOS 1st) in ML Coding 0.46-0.51, Productivity 0.60-0.71; P(top-2 in all four domains) 0.58-0.79; P(1st overall) 0.73-0.87. The 'best on X' and 'highest average' statements are point-estimate rankings that single runs cannot resolve; repeated runs or paired tests would be needed.
Binomial SE for pass-rate benchmarks, Bernoulli-bound (and half-bound) SE for bounded continuous scores using A.1 task counts; unpaired gap z-scores; fixed-seed (12345) normal bootstrap of domain rankings, 20,000 draws; N for AutomationBench/WorkspaceBench (100) and BrowseComp (1000) assumed
Table 1 pass-rate scores are consistent with the stated task counts (tau3-Banking 97 tasks, OfficeQA Pro 133 questions; AutomationBench and BrowseComp N unstated)
tau3: 46.39 (45/97), 55.67 (54/97), 49.48 (48/97) attainable, but OpenHands 40.20 is not (39/97 = 40.21; only matches if rounded to 1 dp). OfficeQA Pro: none of 66.20/66.30/62.30/65.90 equals k/133 at 2 dp; at 1-dp precision only 66.2 (88/133) fits, 66.3, 62.3 and 65.9 do not. Either OfficeQA Pro is not binary-scored on 133 questions (the paper says only 'native EvalScope/OfficeQA rules'), or the denominator differs. AutomationBench values fit only N=600 or 1200 (public set = 600 tasks, consistent); BrowseComp values fit only N=1000 in [20,1300], which is not the 1,266-question full set and is not stated in A.1. Minor reporting issue; it does not change rankings.
GRIM-style search for integer k with round(100k/N, 2) equal to the reported value; 1-dp variant; search over N for AutomationBench and BrowseComp
Table 2 (RSI) pass-rate cells are consistent with the benchmark task counts (tau3 97, OfficeQA Pro 133, AutomationBench 600)
13 of 18 binary-score cells are attainable (e.g., Automation R0 11.00 = 66/600, OfficeQA R0 55.64 = 74/133, tau3 R4/R5 22.68 = 22/97), but 5 cells are not, and all five lie in rounds R3 and R5: Automation R3 39.49 (nearest 237/600 = 39.50) and R5 44.15 (nearest 265/600 = 44.17); OfficeQA R3 56.14 (nearest 75/133 = 56.39) and R5 62.11 (nearest 83/133 = 62.41); tau3 R3 18.57 (nearest 18/97 = 18.56). Values fit smaller denominators (tau3 13/70, OfficeQA 64/114 and 59/95, Automation /585-590 and /564-598), suggesting that some rounds were scored on a subset of tasks (e.g., excluding invalid runs) or used a different scoring rule, which is not disclosed. The effect on the averages is small (each cell differs by at most 0.30 pts from the nearest attainable value) but it bears on comparability across rounds.
GRIM-style search for integer k per cell; search for alternative denominators N-40..N and multi-run means
All counts sum exactly; 437/558 = 78.32% (claimed 78.3%); 203/332 = 61.14% (61.1%); 332-203 = 129; 244/1589 = 15.36% (15.4%). Note that all 1,345 seed trajectories come from search-style sources (BrowseComp, DSQA, WideSearch sets), none of which are among the eight RSI evaluation benchmarks.