ab-v4 raw output capture on dashboard-search-filters-composite shows
minimax-m2.7 emitting <think>...</think> + 4 op_tool tags that fit
inside the 4096 default — its measured completion-token average for
this run was 697, well under the cap. So thinking-budget truncation
is NOT the actual root cause of minimax's lower multi-tool hit rate
(25% vs gpt+deepseek 50%); the real issues are instruction-following
(mixed Strategy A + B despite the explicit forbidance, invented
"canvas" parent_id placeholder).
Still doubling the cap defensively: composite multi-tool outputs can
chain 12-13 op_tool tags + thinking, and "fit easy" today doesn't
mean "fits headroom-free on a longer brief tomorrow." The bump is
free on the happy path (provider stops generating when done, doesn't
bill unused headroom) and only ever helps when the model would
otherwise hit a real ceiling.
Real follow-up for minimax: instruction compliance — the no-mix rule
needs to land harder than a single trailing sentence. Probably wants
the rule moved to top-of-prompt + a few-shot bad-example contrast.
Out of scope here; tracked under Phase 2 prompt design.
ab-v4's dashboard-search-filters-composite garbaged on gpt-5.4 with
"codex timed out after 300000ms" — Codex's own agent framing
(~20k tokens) plus our 18.9k composite system prompt plus thinking
budget plus a 13-tool composite output is enough to blow the old
5-minute cap. The same model produced 13 chained op_tool tags on
team-people-page-composite within the window, so we know it's
generation latency under heavy briefs, not a hung CLI.
Doubles the default to 600000ms and adds AB_CORPUS_CODEX_TIMEOUT_MS
to override either way (lower it to surface slowness as a hard fail
when iterating, raise it for one-off long-form runs). Pairs with
the existing AB_CORPUS_CALL_TIMEOUT_MS for openai-compat clients.
Doesn't fix the deepseek empty-content failure on the same prompt
(already covered by retries=2 + exp backoff) or the minimax mix
+ invented "canvas" parent_id (model-side instruction skip; out of
scope for this commit).
ab-v3 left 36 ark empty + 15 timeout + 4 429 + 6 deepseek empty + 4
minimax timeout AFTER the existing retries=1 fired 88 times. Linear
backoff 250ms*(attempt+1) was too tight when stepping up to
retries=2 (500ms then 750ms isn't a typical Ark recovery window).
Switches to exponential 250ms*4^attempt — 250 / 1000 / 4000ms
spacing — and bumps ark + deepseek to retries=2. minimax opts in to
retries=1 (its 4 errors were wall-clock timeouts, not the model's
<think> truncation that retry can't fix anyway).
10 existing retry tests still pass; the retries=2 case now sleeps
1.25s instead of 0.75s, still well under vitest's default timeout.
ab-v3 succeeds ab-v1 (frozen 2026-04-28). Carries forward all 40
v1 obvious yaml files unchanged so the v1↔v3 overlap stays
comparable, then layers in two new dimensions.
**1. Token cost.** All clients (openai-compat, ark, bailian,
deepseek, minimax, codex-cli, stub-model) now return a
`ChatCallResult { content, usage }` instead of bare string.
Provider usage stats (`prompt_tokens` / `completion_tokens`) plumb
through realModelCall → run.ts → scoreRun → ScoreRow.{prompt,completion}Tokens.
aggregate adds avgPromptTokens{Baseline,Treatment} +
avgCompletionTokens{Baseline,Treatment} per ModelSummary.
write-report emits a new "Token cost" table with Δ columns so
narrow-tools-saves-tokens (the ab-v2 hypothesis) is measurable.
avgUsage skips rows with 0/0 usage so codex-cli (CLI doesn't
surface tokens) and harness errors don't deflate the average to
near-zero — they show '—' instead.
**2. Composite difficulty.** New 'composite' value alongside
obvious / optional. Composite prompts express multi-tool intents
where no single expected_tool_if_any applies. classifyRouting
routes composite-treatment runs into multi-tool / fallback /
garbage (3-bucket sum to 1, distinct from obvious's 4-bucket
right/wrong/fallback/garbage). aggregate adds m6_multi_tool +
m6_fallback + m6_garbage; write-report emits a "Composite routing"
table that gracefully degrades to a placeholder when no composite
yaml exists yet.
Harness side: scripts/ab-corpus/run.ts accepts --corpus ab-v3
(enum + parseArgs guard); dry-run on the v1-mirror corpus produces
a 160-row report including populated token table.
Tests: 4 new aggregate cases (composite, token avg with skip-zero,
NaN-when-no-data) + 4 new score-run cases (composite routing
multi-tool/fallback/garbage/baseline-n/a) + 2 new score-run cases
(usage plumbing) + 2 new openai-compat cases (usage parsing,
missing-usage fallback). Existing 5 retry tests updated for new
return shape. 3727 → 3740 vitest tests, all green; tsc + format
clean.
Token-cost docs and composite docs go straight into types.ts /
score-run.ts / aggregate.ts JSDoc — keeps the contract close to
the code that owns it.
ab-v2 (2026-04-28) saw kimi-k2.6 garbage rate hit 17.5% — every
failure was Ark returning empty `choices[0].message.content` or
hitting the 120s wall clock, not a model-quality issue (the model
itself routed to the right element tool 80% of the time when it
did respond). Same pattern at 7.5% on deepseek-v4-pro.
Adds optional `retries` to `callOpenAICompat` with a transient-error
allowlist: empty content, abort/timeout, HTTP 5xx, HTTP 429. Linear
250ms × (attempt+1) backoff. HTTP 4xx other than 429 stays fatal so
auth/bad-request failures don't burn retry budget.
Wires `retries: 1` through clients/ark.ts and clients/deepseek.ts.
MiniMax + Bailian + Codex stay untouched — their ab-v2 failures
were model-quality (DSL escape errors, output truncation), where
retry wastes a call without changing the outcome.
Adds an 8-case fixture in scripts/ab-corpus/clients/__tests__/ that
mocks fetch to verify: first-try success, empty-then-success,
5xx-then-success, 429-then-success, 401 fatal, retries-default-zero,
retries exhausted, and retries=2 (3 attempts total). Extends the
apps/web vitest include glob to pick up scripts/**/__tests__/.
DeepSeek wasn't in the harness; A/B v2 needed it for the 5-model run.
api.deepseek.com is OpenAI-compatible, so the client mirrors the
minimax pattern (callOpenAICompat with DEEPSEEK_API_KEY env, override-
able DEEPSEEK_BASE_URL).
Router: /^deepseek/i routes to the new client. Aliases `deepseek` /
`deepseek-pro` resolve to `deepseek-v4-pro` (current flagship per
docs); `deepseek-v4-flash` and the deprecated `deepseek-chat` /
`deepseek-reasoner` ids pass through verbatim until the 2026-07-24
sunset upstream.
Verified live with mobile-bio-textarea smoke (B emits batch_design,
T emits add_textarea_v0). Full ab-v2 results across all 5 models in
docs/notes/2026-04-28-ab-v2-results.md.
origin's v0.8.0 had cherry-picks of the v0.7.5 deepseek/image-search
fixes (a727632a, a5952bc8) overlapping local 2073cf5b / 04f4fbc1, plus
new commits (model-selector ark-coding deepseek-v4-pro/flash IDs that
ARK rejects, fetch error.cause unwrap, CI agent-native build, op
export docs cleanup, main merge). Resolved the ark-coding list in
favor of HEAD's deepseek-v3.2 entry (only model ARK Coding Plan
actually supports — see openpencil-docs note).
`/models` now returns only deepseek-v4-pro and deepseek-v4-flash;
deepseek-chat / deepseek-reasoner sunset 2026-07-24 and the
deepseek-v3.2 hard-coded in the ark-coding fallback list never
existed. Both v4 models default to thinking enabled and the API
toggles via `{"thinking":{"type":"disabled"}}` — keep
`thinkingMode: 'disabled'` so the app's fast/non-thinking default
stays intact (server reasoning paths honor it; the Zig openai-compat
path doesn't emit the toggle yet, so calls through that path still
get provider-default thinking until it's wired). v4-pro promoted to
full tier; legacy aliases pinned to an exact RegExp so future
deepseek-* variants don't inherit a forced disabled mode.
Bandaid for the unwired toggle: v4-pro gets `timeoutMultiplier: 2`
because its default-on reasoning blows past the orchestrator's
planning timeout on long system prompts (observed in dev: planning
phase falls back, sub-agent then succeeds — UX degraded but
functional). Drop the multiplier once the Zig path actually sends
`thinking:{type:disabled}`.
Don't add a BUILTIN_MODEL_LISTS.deepseek entry — DeepSeek exposes
/v1/models, so let `fetchProviderModels` pull the live catalog
through `/api/ai/provider-models` instead of pinning a snapshot
(the ark-coding `deepseek-v3.2` ghost above shows what those
snapshots drift into).
The first 12-prompt × 2-model × 2-variant sweep (48 API calls)
ran 29 minutes before I killed it. A Kimi call on the 10th
prompt hung indefinitely — no client-side timeout — and the
harness writes scores.jsonl + report.md ONLY at the end, so
partial progress was unrecoverable. Lost 9/12 completed prompts
because the aggregate step never ran.
Two fixes:
1. openai-compat.ts: AbortController with default 120s timeout
(overridable via AB_CORPUS_CALL_TIMEOUT_MS env). When a call
exceeds the budget, the harness catches the abort, records it
as __HARNESS_ERROR__ (routing=garbage, M1=false), and moves on.
Verified by dialing the timeout to 60s — GLM-5.1's first call
took >60s, got aborted cleanly, run continued to completion
instead of hanging.
2. run.ts: append each ScoreRow to scores.jsonl immediately after
scoring. Truncate at start (so re-runs overwrite). Lost-work
window now bounded to "the currently-executing API call," not
"everything since the run started." report.md and report.json
still write once at the end (aggregate needs the full set) but
scores.jsonl alone is enough for any partial-run analysis.
Post-hardening validation (live 方舟 CP runs):
- mobile-upload-dropzone → add_upload_dropzone_v0 ✓ right-tool
- dashboard-dark-modal → add_modal_shell_v1 (theme=dark) ✓
Second one is the first end-to-end proof that the v1 theme-aware
tool family routes correctly with a real LLM — GLM-5.1 inferred
\`theme: "dark"\` from the natural-language prompt.
Volcengine 方舟 (Ark) added GLM-5.1 and Kimi-K2.6 to its coding
plan on 2026-04-22 — single ARK_CODING_KEY covers both. Harness
now prefers this route over the previous paths:
- glm-5.1 was routed to clients/glm.ts (GLM official CP via
open.bigmodel.cn with GLM_OFFICIAL_CODING_KEY). Now routed to
new clients/ark.ts. The old glm.ts file is kept on disk for
historical comparison but not wired into the default router —
callers who want to A/B the old GLM-official path vs. new Ark
path can import callGlm directly.
- kimi-k2.6 is new — added as a dedicated router branch above
the kimi-k2.5 (bailian) branch so the version-specific match
lands on Ark.
Old kimi-k2.5 continues to route through clients/bailian.ts
(DashScope aggregator) for continuity with earlier A/B runs.
Key management (unchanged from the harness convention):
- ARK_CODING_KEY — Volcengine 方舟 CP UUID format key. Export
in shell before running --live; never committed.
- Existing MINIMAX_API_KEY / GLM_OFFICIAL_CODING_KEY /
DASHSCOPE_BAILIAN_CODING_KEY all still honored for their
respective routes.
Throw message updated so missing-key errors surface the correct
env var for each route.
Harness at scripts/ab-corpus/ wires the pen-ai-skills corpus evaluator to
real model endpoints and pen-mcp handlers:
run.ts — CLI entry (--dry-run / --live / --models A,B,C / --only ID)
apply.ts — ApplyFn impl dispatching tool_call → element handler
and batch_design DSL → handleBatchDesign, against a
fresh tmp .op per run (isolated, auto-cleanup)
build-prompt.ts — B variant strips elements.md + appends batch_design
<op_tool> format instruction; T keeps elements + adds
element-tool PRIMARY / batch_design FALLBACK
instruction. Uniform <op_tool> wrapper in both arms
isolates "tool set width" as the only A/B variable.
stub-model.ts — fixture-based offline model for --dry-run
real-model.ts — router by model id (minimax* / gpt-*/o* / glm-5.1 /
glm-* / kimi-*)
clients/
openai-compat.ts — generic chat/completions POST
minimax.ts — api.minimax.io/v1, MINIMAX_API_KEY
codex-cli.ts — spawns `codex exec` (GPT-5.4 via Codex Pro sub)
bailian.ts — coding.dashscope.aliyuncs.com/v1 CP,
DASHSCOPE_BAILIAN_CODING_KEY (hosts glm-4.7, kimi-k2.5)
glm.ts — open.bigmodel.cn/api/coding/paas/v4 official CP,
GLM_OFFICIAL_CODING_KEY
write-report.ts — Report → report.md + report.json in out dir;
4-way routing breakdown table per model
Kept entirely outside packages/ — scripts are a local dev tool, not part
of the published SDK. API keys never hit disk or git.
v1 run results logged separately in openpencil-docs
superpowers/notes/2026-04-20-ab-v1-results.md (5 models × 24 prompts).
