ab-v3 left 36 ark empty + 15 timeout + 4 429 + 6 deepseek empty + 4
minimax timeout AFTER the existing retries=1 fired 88 times. Linear
backoff 250ms*(attempt+1) was too tight when stepping up to
retries=2 (500ms then 750ms isn't a typical Ark recovery window).
Switches to exponential 250ms*4^attempt — 250 / 1000 / 4000ms
spacing — and bumps ark + deepseek to retries=2. minimax opts in to
retries=1 (its 4 errors were wall-clock timeouts, not the model's
<think> truncation that retry can't fix anyway).
10 existing retry tests still pass; the retries=2 case now sleeps
1.25s instead of 0.75s, still well under vitest's default timeout.
ab-v3 succeeds ab-v1 (frozen 2026-04-28). Carries forward all 40
v1 obvious yaml files unchanged so the v1↔v3 overlap stays
comparable, then layers in two new dimensions.
**1. Token cost.** All clients (openai-compat, ark, bailian,
deepseek, minimax, codex-cli, stub-model) now return a
`ChatCallResult { content, usage }` instead of bare string.
Provider usage stats (`prompt_tokens` / `completion_tokens`) plumb
through realModelCall → run.ts → scoreRun → ScoreRow.{prompt,completion}Tokens.
aggregate adds avgPromptTokens{Baseline,Treatment} +
avgCompletionTokens{Baseline,Treatment} per ModelSummary.
write-report emits a new "Token cost" table with Δ columns so
narrow-tools-saves-tokens (the ab-v2 hypothesis) is measurable.
avgUsage skips rows with 0/0 usage so codex-cli (CLI doesn't
surface tokens) and harness errors don't deflate the average to
near-zero — they show '—' instead.
**2. Composite difficulty.** New 'composite' value alongside
obvious / optional. Composite prompts express multi-tool intents
where no single expected_tool_if_any applies. classifyRouting
routes composite-treatment runs into multi-tool / fallback /
garbage (3-bucket sum to 1, distinct from obvious's 4-bucket
right/wrong/fallback/garbage). aggregate adds m6_multi_tool +
m6_fallback + m6_garbage; write-report emits a "Composite routing"
table that gracefully degrades to a placeholder when no composite
yaml exists yet.
Harness side: scripts/ab-corpus/run.ts accepts --corpus ab-v3
(enum + parseArgs guard); dry-run on the v1-mirror corpus produces
a 160-row report including populated token table.
Tests: 4 new aggregate cases (composite, token avg with skip-zero,
NaN-when-no-data) + 4 new score-run cases (composite routing
multi-tool/fallback/garbage/baseline-n/a) + 2 new score-run cases
(usage plumbing) + 2 new openai-compat cases (usage parsing,
missing-usage fallback). Existing 5 retry tests updated for new
return shape. 3727 → 3740 vitest tests, all green; tsc + format
clean.
Token-cost docs and composite docs go straight into types.ts /
score-run.ts / aggregate.ts JSDoc — keeps the contract close to
the code that owns it.
ab-v2 (2026-04-28) saw kimi-k2.6 garbage rate hit 17.5% — every
failure was Ark returning empty `choices[0].message.content` or
hitting the 120s wall clock, not a model-quality issue (the model
itself routed to the right element tool 80% of the time when it
did respond). Same pattern at 7.5% on deepseek-v4-pro.
Adds optional `retries` to `callOpenAICompat` with a transient-error
allowlist: empty content, abort/timeout, HTTP 5xx, HTTP 429. Linear
250ms × (attempt+1) backoff. HTTP 4xx other than 429 stays fatal so
auth/bad-request failures don't burn retry budget.
Wires `retries: 1` through clients/ark.ts and clients/deepseek.ts.
MiniMax + Bailian + Codex stay untouched — their ab-v2 failures
were model-quality (DSL escape errors, output truncation), where
retry wastes a call without changing the outcome.
Adds an 8-case fixture in scripts/ab-corpus/clients/__tests__/ that
mocks fetch to verify: first-try success, empty-then-success,
5xx-then-success, 429-then-success, 401 fatal, retries-default-zero,
retries exhausted, and retries=2 (3 attempts total). Extends the
apps/web vitest include glob to pick up scripts/**/__tests__/.
`/models` now returns only deepseek-v4-pro and deepseek-v4-flash;
deepseek-chat / deepseek-reasoner sunset 2026-07-24 and the
deepseek-v3.2 hard-coded in the ark-coding fallback list never
existed. Both v4 models default to thinking enabled and the API
toggles via `{"thinking":{"type":"disabled"}}` — keep
`thinkingMode: 'disabled'` so the app's fast/non-thinking default
stays intact (server reasoning paths honor it; the Zig openai-compat
path doesn't emit the toggle yet, so calls through that path still
get provider-default thinking until it's wired). v4-pro promoted to
full tier; legacy aliases pinned to an exact RegExp so future
deepseek-* variants don't inherit a forced disabled mode.
Bandaid for the unwired toggle: v4-pro gets `timeoutMultiplier: 2`
because its default-on reasoning blows past the orchestrator's
planning timeout on long system prompts (observed in dev: planning
phase falls back, sub-agent then succeeds — UX degraded but
functional). Drop the multiplier once the Zig path actually sends
`thinking:{type:disabled}`.
Don't add a BUILTIN_MODEL_LISTS.deepseek entry — DeepSeek exposes
/v1/models, so let `fetchProviderModels` pull the live catalog
through `/api/ai/provider-models` instead of pinning a snapshot
(the ark-coding `deepseek-v3.2` ghost above shows what those
snapshots drift into).
Volcengine 方舟 (Ark) added GLM-5.1 and Kimi-K2.6 to its coding
plan on 2026-04-22 — single ARK_CODING_KEY covers both. Harness
now prefers this route over the previous paths:
- glm-5.1 was routed to clients/glm.ts (GLM official CP via
open.bigmodel.cn with GLM_OFFICIAL_CODING_KEY). Now routed to
new clients/ark.ts. The old glm.ts file is kept on disk for
historical comparison but not wired into the default router —
callers who want to A/B the old GLM-official path vs. new Ark
path can import callGlm directly.
- kimi-k2.6 is new — added as a dedicated router branch above
the kimi-k2.5 (bailian) branch so the version-specific match
lands on Ark.
Old kimi-k2.5 continues to route through clients/bailian.ts
(DashScope aggregator) for continuity with earlier A/B runs.
Key management (unchanged from the harness convention):
- ARK_CODING_KEY — Volcengine 方舟 CP UUID format key. Export
in shell before running --live; never committed.
- Existing MINIMAX_API_KEY / GLM_OFFICIAL_CODING_KEY /
DASHSCOPE_BAILIAN_CODING_KEY all still honored for their
respective routes.
Throw message updated so missing-key errors surface the correct
env var for each route.
