origin's v0.8.0 had cherry-picks of the v0.7.5 deepseek/image-search
fixes (a727632a, a5952bc8) overlapping local 2073cf5b / 04f4fbc1, plus
new commits (model-selector ark-coding deepseek-v4-pro/flash IDs that
ARK rejects, fetch error.cause unwrap, CI agent-native build, op
export docs cleanup, main merge). Resolved the ark-coding list in
favor of HEAD's deepseek-v3.2 entry (only model ARK Coding Plan
actually supports — see openpencil-docs note).
`/models` now returns only deepseek-v4-pro and deepseek-v4-flash;
deepseek-chat / deepseek-reasoner sunset 2026-07-24 and the
deepseek-v3.2 hard-coded in the ark-coding fallback list never
existed. Both v4 models default to thinking enabled and the API
toggles via `{"thinking":{"type":"disabled"}}` — keep
`thinkingMode: 'disabled'` so the app's fast/non-thinking default
stays intact (server reasoning paths honor it; the Zig openai-compat
path doesn't emit the toggle yet, so calls through that path still
get provider-default thinking until it's wired). v4-pro promoted to
full tier; legacy aliases pinned to an exact RegExp so future
deepseek-* variants don't inherit a forced disabled mode.
Bandaid for the unwired toggle: v4-pro gets `timeoutMultiplier: 2`
because its default-on reasoning blows past the orchestrator's
planning timeout on long system prompts (observed in dev: planning
phase falls back, sub-agent then succeeds — UX degraded but
functional). Drop the multiplier once the Zig path actually sends
`thinking:{type:disabled}`.
Don't add a BUILTIN_MODEL_LISTS.deepseek entry — DeepSeek exposes
/v1/models, so let `fetchProviderModels` pull the live catalog
through `/api/ai/provider-models` instead of pinning a snapshot
(the ark-coding `deepseek-v3.2` ghost above shows what those
snapshots drift into).
The first 12-prompt × 2-model × 2-variant sweep (48 API calls)
ran 29 minutes before I killed it. A Kimi call on the 10th
prompt hung indefinitely — no client-side timeout — and the
harness writes scores.jsonl + report.md ONLY at the end, so
partial progress was unrecoverable. Lost 9/12 completed prompts
because the aggregate step never ran.
Two fixes:
1. openai-compat.ts: AbortController with default 120s timeout
(overridable via AB_CORPUS_CALL_TIMEOUT_MS env). When a call
exceeds the budget, the harness catches the abort, records it
as __HARNESS_ERROR__ (routing=garbage, M1=false), and moves on.
Verified by dialing the timeout to 60s — GLM-5.1's first call
took >60s, got aborted cleanly, run continued to completion
instead of hanging.
2. run.ts: append each ScoreRow to scores.jsonl immediately after
scoring. Truncate at start (so re-runs overwrite). Lost-work
window now bounded to "the currently-executing API call," not
"everything since the run started." report.md and report.json
still write once at the end (aggregate needs the full set) but
scores.jsonl alone is enough for any partial-run analysis.
Post-hardening validation (live 方舟 CP runs):
- mobile-upload-dropzone → add_upload_dropzone_v0 ✓ right-tool
- dashboard-dark-modal → add_modal_shell_v1 (theme=dark) ✓
Second one is the first end-to-end proof that the v1 theme-aware
tool family routes correctly with a real LLM — GLM-5.1 inferred
\`theme: "dark"\` from the natural-language prompt.
Two independent changes rolled together since they both serve the
same goal — "can real LLMs actually route to the tools we shipped
today?":
1. scripts/ab-corpus/run.ts gains a `--corpus` flag (ab-v0 |
ab-v1, default ab-v0 for back-compat). The harness was
hardcoded to ab-v0 — adding ab-v1 prompts was worthless
without a way to run them. Validated 17 prompts × 2 models ×
2 variants during a live 方舟-CP run.
2. 5 new ab-v1 prompts cover the 2026-04-24 tool batch:
- mobile-upload-dropzone.yaml → add_upload_dropzone_v0
- mobile-otp-verification.yaml → add_otp_input_v0
- mobile-file-attachment.yaml → add_attachment_row_v0
- mobile-chat-message.yaml → add_chat_bubble_v0
- dashboard-dark-modal.yaml → add_modal_shell_v1
corpus-loader.test.ts bumps its count assertion 12 → 17 and
extends the tool-coverage set. Also loosens the regex to accept
`_v\d+$` (was `_v0$`) so add_modal_shell_v1 passes. No other
test file needed changes — the existing registry-parity and
mock-llm tests already use `_v\d+$` or the registry directly.
.gitignore gains:
- .playwright-mcp/ (MCP Playwright session artifacts)
- editor-*.png (local verification screenshots)
- scripts/ab-corpus/runs/ (live-run outputs / reports)
None of those belong in version control — they're artifacts
from local verification runs.
The Ark router regex was `/^kimi-k2\.6/i` — required the `k`
prefix. But mapKimiArkId's alias list accepted both `kimi-k2.6`
AND `kimi-2.6` (no-prefix form). Result: `kimi-2.6` failed the
Ark regex, fell through to the generic `/^kimi/i` branch, got
routed to Bailian — which doesn't host K2.6. Bailian would
return HTTP 400 "model not supported" with no hint that the id
belonged on Ark.
Fix: Ark router regex now `/^kimi-k?2\.6(-ark)?$/i` — optional
`k` prefix + optional `-ark` suffix, anchored at end to prevent
accidentally matching a hypothetical later version. mapKimiArkId
normalizes all four accepted aliases (kimi-k2.6, kimi-2.6,
kimi-k2.6-ark, kimi-2.6-ark) to the canonical on-Ark id.
Same latent bug fixed on the glm-5.1 route: regex `/^glm-5\.1/i`
would prefix-match a hypothetical `glm-5.10` and wrongly route it
to Ark. Tightened to `/^glm-5\.1(-coding|-ark)?$/i` with the same
anchored-end + suffix-allowlist pattern.
Caught by Codex stop-hook review during 2026-04-22 session.
Volcengine 方舟 (Ark) added GLM-5.1 and Kimi-K2.6 to its coding
plan on 2026-04-22 — single ARK_CODING_KEY covers both. Harness
now prefers this route over the previous paths:
- glm-5.1 was routed to clients/glm.ts (GLM official CP via
open.bigmodel.cn with GLM_OFFICIAL_CODING_KEY). Now routed to
new clients/ark.ts. The old glm.ts file is kept on disk for
historical comparison but not wired into the default router —
callers who want to A/B the old GLM-official path vs. new Ark
path can import callGlm directly.
- kimi-k2.6 is new — added as a dedicated router branch above
the kimi-k2.5 (bailian) branch so the version-specific match
lands on Ark.
Old kimi-k2.5 continues to route through clients/bailian.ts
(DashScope aggregator) for continuity with earlier A/B runs.
Key management (unchanged from the harness convention):
- ARK_CODING_KEY — Volcengine 方舟 CP UUID format key. Export
in shell before running --live; never committed.
- Existing MINIMAX_API_KEY / GLM_OFFICIAL_CODING_KEY /
DASHSCOPE_BAILIAN_CODING_KEY all still honored for their
respective routes.
Throw message updated so missing-key errors surface the correct
env var for each route.
User hit at dev-server startup:
Module "node:fs" has been externalized for browser compatibility.
Cannot access "node:fs.readdirSync" in client code.
Chain: apps/web's design-parser.ts imports `parseModelOutput` from
`@zseven-w/pen-ai-skills`; main barrel re-exports everything from
`./corpus`; `./corpus/index.ts` re-exports `loadCorpus` which imports
`node:fs`. Vite pulls the whole graph into the client bundle → crash
on the first browser-side module evaluation.
Fix: remove `loadCorpus` from `./corpus/index.ts`. The barrel now
only exposes pure-string helpers (parser, scorer, aggregator, types)
— all browser-safe. `loadCorpus` stays in `corpus-loader.ts` but
Node-only consumers (`scripts/ab-corpus/run.ts`) import it directly
via a relative path. Package.json only declares the main entry in
`exports`, so sub-path imports via the package name fail at runtime
(pkg runs under Bun for the harness) — relative file path avoids
that gate.
Verification:
- `bun scripts/ab-corpus/run.ts --only X --dry-run` still runs end
to end
- tsc --noEmit exit 0
- Full test suite 1866/1866
Browser-side verification (user): restart Vite dev server — the
design-parser import no longer pulls node:fs through the barrel.
Harness at scripts/ab-corpus/ wires the pen-ai-skills corpus evaluator to
real model endpoints and pen-mcp handlers:
run.ts — CLI entry (--dry-run / --live / --models A,B,C / --only ID)
apply.ts — ApplyFn impl dispatching tool_call → element handler
and batch_design DSL → handleBatchDesign, against a
fresh tmp .op per run (isolated, auto-cleanup)
build-prompt.ts — B variant strips elements.md + appends batch_design
<op_tool> format instruction; T keeps elements + adds
element-tool PRIMARY / batch_design FALLBACK
instruction. Uniform <op_tool> wrapper in both arms
isolates "tool set width" as the only A/B variable.
stub-model.ts — fixture-based offline model for --dry-run
real-model.ts — router by model id (minimax* / gpt-*/o* / glm-5.1 /
glm-* / kimi-*)
clients/
openai-compat.ts — generic chat/completions POST
minimax.ts — api.minimax.io/v1, MINIMAX_API_KEY
codex-cli.ts — spawns `codex exec` (GPT-5.4 via Codex Pro sub)
bailian.ts — coding.dashscope.aliyuncs.com/v1 CP,
DASHSCOPE_BAILIAN_CODING_KEY (hosts glm-4.7, kimi-k2.5)
glm.ts — open.bigmodel.cn/api/coding/paas/v4 official CP,
GLM_OFFICIAL_CODING_KEY
write-report.ts — Report → report.md + report.json in out dir;
4-way routing breakdown table per model
Kept entirely outside packages/ — scripts are a local dev tool, not part
of the published SDK. API keys never hit disk or git.
v1 run results logged separately in openpencil-docs
superpowers/notes/2026-04-20-ab-v1-results.md (5 models × 24 prompts).
