ab-v3 succeeds ab-v1 (frozen 2026-04-28). Carries forward all 40
v1 obvious yaml files unchanged so the v1↔v3 overlap stays
comparable, then layers in two new dimensions.
**1. Token cost.** All clients (openai-compat, ark, bailian,
deepseek, minimax, codex-cli, stub-model) now return a
`ChatCallResult { content, usage }` instead of bare string.
Provider usage stats (`prompt_tokens` / `completion_tokens`) plumb
through realModelCall → run.ts → scoreRun → ScoreRow.{prompt,completion}Tokens.
aggregate adds avgPromptTokens{Baseline,Treatment} +
avgCompletionTokens{Baseline,Treatment} per ModelSummary.
write-report emits a new "Token cost" table with Δ columns so
narrow-tools-saves-tokens (the ab-v2 hypothesis) is measurable.
avgUsage skips rows with 0/0 usage so codex-cli (CLI doesn't
surface tokens) and harness errors don't deflate the average to
near-zero — they show '—' instead.
**2. Composite difficulty.** New 'composite' value alongside
obvious / optional. Composite prompts express multi-tool intents
where no single expected_tool_if_any applies. classifyRouting
routes composite-treatment runs into multi-tool / fallback /
garbage (3-bucket sum to 1, distinct from obvious's 4-bucket
right/wrong/fallback/garbage). aggregate adds m6_multi_tool +
m6_fallback + m6_garbage; write-report emits a "Composite routing"
table that gracefully degrades to a placeholder when no composite
yaml exists yet.
Harness side: scripts/ab-corpus/run.ts accepts --corpus ab-v3
(enum + parseArgs guard); dry-run on the v1-mirror corpus produces
a 160-row report including populated token table.
Tests: 4 new aggregate cases (composite, token avg with skip-zero,
NaN-when-no-data) + 4 new score-run cases (composite routing
multi-tool/fallback/garbage/baseline-n/a) + 2 new score-run cases
(usage plumbing) + 2 new openai-compat cases (usage parsing,
missing-usage fallback). Existing 5 retry tests updated for new
return shape. 3727 → 3740 vitest tests, all green; tsc + format
clean.
Token-cost docs and composite docs go straight into types.ts /
score-run.ts / aggregate.ts JSDoc — keeps the contract close to
the code that owns it.
origin's v0.8.0 had cherry-picks of the v0.7.5 deepseek/image-search
fixes (a727632a, a5952bc8) overlapping local 2073cf5b / 04f4fbc1, plus
new commits (model-selector ark-coding deepseek-v4-pro/flash IDs that
ARK rejects, fetch error.cause unwrap, CI agent-native build, op
export docs cleanup, main merge). Resolved the ark-coding list in
favor of HEAD's deepseek-v3.2 entry (only model ARK Coding Plan
actually supports — see openpencil-docs note).
`/models` now returns only deepseek-v4-pro and deepseek-v4-flash;
deepseek-chat / deepseek-reasoner sunset 2026-07-24 and the
deepseek-v3.2 hard-coded in the ark-coding fallback list never
existed. Both v4 models default to thinking enabled and the API
toggles via `{"thinking":{"type":"disabled"}}` — keep
`thinkingMode: 'disabled'` so the app's fast/non-thinking default
stays intact (server reasoning paths honor it; the Zig openai-compat
path doesn't emit the toggle yet, so calls through that path still
get provider-default thinking until it's wired). v4-pro promoted to
full tier; legacy aliases pinned to an exact RegExp so future
deepseek-* variants don't inherit a forced disabled mode.
Bandaid for the unwired toggle: v4-pro gets `timeoutMultiplier: 2`
because its default-on reasoning blows past the orchestrator's
planning timeout on long system prompts (observed in dev: planning
phase falls back, sub-agent then succeeds — UX degraded but
functional). Drop the multiplier once the Zig path actually sends
`thinking:{type:disabled}`.
Don't add a BUILTIN_MODEL_LISTS.deepseek entry — DeepSeek exposes
/v1/models, so let `fetchProviderModels` pull the live catalog
through `/api/ai/provider-models` instead of pinning a snapshot
(the ark-coding `deepseek-v3.2` ghost above shows what those
snapshots drift into).
The first 12-prompt × 2-model × 2-variant sweep (48 API calls)
ran 29 minutes before I killed it. A Kimi call on the 10th
prompt hung indefinitely — no client-side timeout — and the
harness writes scores.jsonl + report.md ONLY at the end, so
partial progress was unrecoverable. Lost 9/12 completed prompts
because the aggregate step never ran.
Two fixes:
1. openai-compat.ts: AbortController with default 120s timeout
(overridable via AB_CORPUS_CALL_TIMEOUT_MS env). When a call
exceeds the budget, the harness catches the abort, records it
as __HARNESS_ERROR__ (routing=garbage, M1=false), and moves on.
Verified by dialing the timeout to 60s — GLM-5.1's first call
took >60s, got aborted cleanly, run continued to completion
instead of hanging.
2. run.ts: append each ScoreRow to scores.jsonl immediately after
scoring. Truncate at start (so re-runs overwrite). Lost-work
window now bounded to "the currently-executing API call," not
"everything since the run started." report.md and report.json
still write once at the end (aggregate needs the full set) but
scores.jsonl alone is enough for any partial-run analysis.
Post-hardening validation (live 方舟 CP runs):
- mobile-upload-dropzone → add_upload_dropzone_v0 ✓ right-tool
- dashboard-dark-modal → add_modal_shell_v1 (theme=dark) ✓
Second one is the first end-to-end proof that the v1 theme-aware
tool family routes correctly with a real LLM — GLM-5.1 inferred
\`theme: "dark"\` from the natural-language prompt.
Two independent changes rolled together since they both serve the
same goal — "can real LLMs actually route to the tools we shipped
today?":
1. scripts/ab-corpus/run.ts gains a `--corpus` flag (ab-v0 |
ab-v1, default ab-v0 for back-compat). The harness was
hardcoded to ab-v0 — adding ab-v1 prompts was worthless
without a way to run them. Validated 17 prompts × 2 models ×
2 variants during a live 方舟-CP run.
2. 5 new ab-v1 prompts cover the 2026-04-24 tool batch:
- mobile-upload-dropzone.yaml → add_upload_dropzone_v0
- mobile-otp-verification.yaml → add_otp_input_v0
- mobile-file-attachment.yaml → add_attachment_row_v0
- mobile-chat-message.yaml → add_chat_bubble_v0
- dashboard-dark-modal.yaml → add_modal_shell_v1
corpus-loader.test.ts bumps its count assertion 12 → 17 and
extends the tool-coverage set. Also loosens the regex to accept
`_v\d+$` (was `_v0$`) so add_modal_shell_v1 passes. No other
test file needed changes — the existing registry-parity and
mock-llm tests already use `_v\d+$` or the registry directly.
.gitignore gains:
- .playwright-mcp/ (MCP Playwright session artifacts)
- editor-*.png (local verification screenshots)
- scripts/ab-corpus/runs/ (live-run outputs / reports)
None of those belong in version control — they're artifacts
from local verification runs.
User hit at dev-server startup:
Module "node:fs" has been externalized for browser compatibility.
Cannot access "node:fs.readdirSync" in client code.
Chain: apps/web's design-parser.ts imports `parseModelOutput` from
`@zseven-w/pen-ai-skills`; main barrel re-exports everything from
`./corpus`; `./corpus/index.ts` re-exports `loadCorpus` which imports
`node:fs`. Vite pulls the whole graph into the client bundle → crash
on the first browser-side module evaluation.
Fix: remove `loadCorpus` from `./corpus/index.ts`. The barrel now
only exposes pure-string helpers (parser, scorer, aggregator, types)
— all browser-safe. `loadCorpus` stays in `corpus-loader.ts` but
Node-only consumers (`scripts/ab-corpus/run.ts`) import it directly
via a relative path. Package.json only declares the main entry in
`exports`, so sub-path imports via the package name fail at runtime
(pkg runs under Bun for the harness) — relative file path avoids
that gate.
Verification:
- `bun scripts/ab-corpus/run.ts --only X --dry-run` still runs end
to end
- tsc --noEmit exit 0
- Full test suite 1866/1866
Browser-side verification (user): restart Vite dev server — the
design-parser import no longer pulls node:fs through the barrel.
Harness at scripts/ab-corpus/ wires the pen-ai-skills corpus evaluator to
real model endpoints and pen-mcp handlers:
run.ts — CLI entry (--dry-run / --live / --models A,B,C / --only ID)
apply.ts — ApplyFn impl dispatching tool_call → element handler
and batch_design DSL → handleBatchDesign, against a
fresh tmp .op per run (isolated, auto-cleanup)
build-prompt.ts — B variant strips elements.md + appends batch_design
<op_tool> format instruction; T keeps elements + adds
element-tool PRIMARY / batch_design FALLBACK
instruction. Uniform <op_tool> wrapper in both arms
isolates "tool set width" as the only A/B variable.
stub-model.ts — fixture-based offline model for --dry-run
real-model.ts — router by model id (minimax* / gpt-*/o* / glm-5.1 /
glm-* / kimi-*)
clients/
openai-compat.ts — generic chat/completions POST
minimax.ts — api.minimax.io/v1, MINIMAX_API_KEY
codex-cli.ts — spawns `codex exec` (GPT-5.4 via Codex Pro sub)
bailian.ts — coding.dashscope.aliyuncs.com/v1 CP,
DASHSCOPE_BAILIAN_CODING_KEY (hosts glm-4.7, kimi-k2.5)
glm.ts — open.bigmodel.cn/api/coding/paas/v4 official CP,
GLM_OFFICIAL_CODING_KEY
write-report.ts — Report → report.md + report.json in out dir;
4-way routing breakdown table per model
Kept entirely outside packages/ — scripts are a local dev tool, not part
of the published SDK. API keys never hit disk or git.
v1 run results logged separately in openpencil-docs
superpowers/notes/2026-04-20-ab-v1-results.md (5 models × 24 prompts).
