# PR #97979 A/B verdict — core-tool deferral (288 live runs)

Date: 2026-08-29 · Harness: /tmp/ab97979/harness · Method: METHOD.md

## Arms
base = origin/main 3f36c87e1ebd (27 direct tools in the eval assembly, 47.4KB schema chars)
pr   = main + #97979 e16ad33a9d24 (12 direct: 9 working set + 3 bridge; 19 deferred; 21.0KB schema chars, −56%)

## Headline (mean of task means, 14 tasks × 3 reps; contested cells re-run to n=6)

| model | arm | accuracy | turns | tokens(k) | wall(s) |
|---|---|---|---|---|---|
| gpt-5.6-terra (large) | base | 0.938 | 6.0 | 80.9 | 27.6 |
| gpt-5.6-terra | pr | 0.879 | 6.6 | **62.5 (−23%)** | 27.1 |
| glm-5.3-flash (medium) | base | 0.915 | 6.0 | 101.0 | 56.5 |
| glm-5.3-flash | pr | **0.963 (+0.05)** | 8.8 | **89.6 (−11%)** | 59.5 |
| qwen3.8-27b (small) | base | 0.915 | 7.1 | 127.4 | 53.5 |
| qwen3.8-27b | pr | 0.907 | 9.4 | **118.4 (−7%)** | 79.2 |

Grand accuracy: base 0.923 vs pr 0.916 — flat within rep noise once the two
contested tasks were extended to n=6. Tokens down on every model. Turns up
~1–2 (bridge discovery round-trips), wall flat on terra/glm, +48% on qwen
(27B pays real latency for extra bridge turns).

## Deferred-tool discovery (PR arm, tasks requiring the tool, all models)
Perfect (9/9 or 18/18): session_search, todo_list, image_generate,
desktop_project, desktop_preview, drive_preview, annotate_preview,
apply_layout, focus_pane, read_terminal, read_window_below.
Near-perfect: cronjob_manage 16/18, gui_tour 8/9, process_manage 8/9.
Weak: computer_use 6/9, show_tip 6/9, clarify 7/18, setup_mcp 4/9*,
close_terminal 4/9*.
(*base-arm usage on the same tasks: setup_mcp 3/9, close_terminal 0/9 —
these two are NOT deferral regressions; models skip them even when visible.)

## The one real regression: clarify
base: clarify used 18/18, score 1.00 on the ambiguous-delete trap, all models.
pr: clarify used 7/18 → terra 0/6 (0.50), glm 3/6 (0.80), qwen 4/6 (0.87).
Models still ask — but as plain text, ending the turn (extra user round-trip,
no structured choices). The harness credits scripted replies; without that
continuation the task scores 0. Exactly trade-off #1 flagged in the PR body.
Safety note: in 0 of 288 runs was the WRONG file deleted — the failure mode
is degraded UX, never destructive action.

## screenshot_ambiguous (n=6): split, not directional
terra base 1.00 → pr 0.67 (2 reps answered from read_window_below instead of
discovering computer_use — catalog-stub misrouting to a cheaper adjacent tool);
but glm 0.67→1.00 and qwen 0.50→0.83 IMPROVED under deferral (the focused
catalog line beats 27 competing schemas for weaker models). Model-split, nets
to ~flat across the tier ladder.

## Controls
eager_refactor_control (eager-only tools): pr arm −49% tokens at held 1.00 —
pure schema-shrink win, no behavior change.
config_grep_distractor: 1.00 both arms, 0 false bridge calls on terra/glm —
no discovery-overhead tax on tasks that don't need deferred tools.

## Anomalies audited
- glm pr layout rep2 (41 turns, 514k tok): after completing the GUI task via
  bridge it burned 30 terminal calls "verifying"; score 1.0. Model paranoia,
  not a bridge failure.
- qwen pr screenshot rep3: hard wall timeout, scored 0, kept in denominator.
- 1 errored run / 288 total; raw-XML provider noise: 0.

## Verdict: SHIP, with one follow-up — un-defer (or pin) `clarify`.
The deferral mechanism works: discovery is essentially perfect for 14/19
tools, accuracy is flat overall (large model −0.06, medium +0.05, small
−0.01), token cost drops on every model, and the eager-surface control shows
the −49%-token schema win with zero accuracy cost. The single consistent
regression is clarify: structured ask-the-user collapses to plain-text
questions when the schema is invisible (7/18 vs 18/18). PR #91125
(always-visible deferred-tool pins) is the natural mechanism — pin clarify
eager by default, or drop it from _DEFAULT_DEFERRED_TOOLS (~250 tok cost).
computer_use on frontier models is worth watching but is model-split, not
directional. todo_list discipline concern from the PR body did NOT
materialize (18/18 discovery, multi-step scores held at 1.00).