Task 2.3 — representative v1 tool walkthrough for Plan 14 byte-parity contract.
Light mode is byte-equal to add_heading_v0 (V0_LATIN_PRESETS table reused);
dark/system modes use resolveTheme() for fill color and typography token refs.
Adds theme enum [light, dark, system] to add_heading_v1 MCP schema, elements.md
decision tree, shim-server-parity CASES, and SERVER_BUILDERS. All 3937 tests pass.
P1.5 consumer compatibility gate before P2 element-tool v1 builders.
resolveNodeForCanvas was already resolving gap/padding/opacity/color refs,
but missed text typography fields (fontSize, lineHeight, letterSpacing) and
cornerRadius. Since both skia-engine and pen-renderer consume resolver output
before layout runs, unresolved string refs would pass arithmetic as NaN.
layout/engine.ts also patched to guard typeof === 'number' at the 4 text
font-size/line-height access points — defensive fallback to 16/1.x even when
called on pre-resolution nodes (e.g. MCP server, normalizer pipeline).
Adds 21-test p1-5-consumer-compat.test.ts covering all 6 consumer paths.
All 1183 pen-core tests pass. format:check and tsc --noEmit clean.
Introduce DEFAULT_PALETTE_FALLBACK (56-entry map built from semantic-palette
at module init) and wire it into resolveVariableRef so that v1 'system' mode
on an un-seeded doc resolves to canonical hex/numeric defaults rather than
undefined — fulfilling the equivalence guarantee from spec §5.3.
Update modal-shell-v1 test to reflect the new contract: un-seeded doc +
Mode:Dark now yields #1E293B (color-surface dark) instead of undefined.
The 4 extra single-value tokens (color-accent-dark, color-info-surface,
color-warning-text-strong, color-danger-text-strong) introduced in P1.1.6
violated spec §3.1 / §7.4 — those hex were INTENDED to merge into existing
tokens with ≤ 5% accepted color drift, not become new tokens.
Replaced with MERGE_MAP in measure-v0-hex-coverage.ts that tracks the 4
near-shade redirections (#1D4ED8→color-accent, #EFF6FF→color-info-bg,
#B45309→color-warning-text, #B91C1C→color-danger-text). Cover rate
calculation now reports direct + merge breakdown.
Final palette token count: 56 (28 color + 18 type + 2 letterSpacing +
5 spacing + 3 radius). Cover rate: 28 direct + 4 merge = 32/32 = 100.0%.
spacing-1..5 = 4/8/12/16/24 px — a 4-point base scale covering xs through xl.
All tokens use type="number" with plain scalar values. Palette total grows to 57.
type-display-letter-spacing (-0.5 px) for large display text, and
type-uppercase-label-letter-spacing (1.5 px) for uppercase / overline labels.
Palette total grows to 52.
size + weight + line-height for 6 roles: display / h1 / h2 / h3 / body / caption.
All 18 tokens use type="number" with a plain scalar value (no theme axis).
getSemanticPaletteHex() omits numeric tokens from its return map.
Palette grows from 32 color → 50 total variables.
8 light/dark alert pairs (info/success/warning/danger bg+text) and 6
single-value chart series colors added to PALETTE. PaletteEntry union
type introduced to support LightDarkEntry | SingleColorEntry | SingleNumberEntry.
getSemanticPalette(), getSemanticPaletteHex(), applySemanticPalette()
updated to handle all three entry shapes. Palette grows from 14 → 28 tokens.
Two diet/teaching changes to elements.md, both motivated by today's
ab-v4 smoke results:
(a) parent_id teaching — minimax-m2.7 invented "members-section" /
"canvas" as parent_id values on its multi-tool composite output,
then every one of its 14 tags failed apply with "parent_id X not
found in document". Same pattern as yesterday. Adds an explicit
rule near the top of elements.md (right under the multi-tool
banner) — `parent_id` is REAL or OMITTED, never invented. The
`<page>` / `<panel>` / `<sidebar>` placeholders in the cookbook
recipes are documentation conventions; in actual output, OMIT
the field. Names the failure mode by reproducing the error
message format so models learn to avoid it.
(b) Phase 3 token diet — strip ". Different from <tool> (...)"
disambiguation suffixes from the PREFER list (22 entries had
them, ~80-200 chars each = ~2.5kb / ~625 tokens saved). The
primary keyword + tool-name + capability description survives
intact; the cross-references pointing at sibling tools get
dropped. Risk: slight increase in wrong-tool routing on
ambiguous prompts. Worth it for the size reduction; the
decision tree alone still shows the tool family in context.
Verified by mechanical diff — perl in-place edit, then visual
review confirms no other content was touched. 3785 vitest pass,
format clean, tsc silent.
Net effect on T-prompt size (chars / 4 estimate):
T + composite + mobile: was 17,930 → now 17,475 (-455 chars)
T + composite + dashboard: was 18,437 → now 17,983 (-454 chars)
T + obvious + dashboard: ~14,009 → ~13,556 (-453 chars)
Per-arm savings are smaller than the raw 2.5kb trim because (a)
adds ~700 chars of parent_id teaching. Net win: ~450 chars / ~110
tokens per call. Modest but compounds across 520+ ab-v4 runs.
Sweep follow-up to 113bd55a — same defensive pattern (reject
unknown enum strings at the entry boundary) applied to every other
builder that indexed a Record<EnumLiteral, T> with a value sourced
from raw JSON args.
Builders + enums covered:
- buildTag — TagTone (default | accent | success | warning | error)
- buildCallout — CalloutTone (info | success | warning | danger | note)
- buildActivityLog — tone (info | success | warning | danger | neutral)
- buildInviteRow — InviteStatus (pending | expired | accepted)
- buildMemberRow — trailing.tone for status_dot (online | busy | away
| offline). role_badge / menu variants skip the check (no tone field)
Same failure mode each one fixed: when a model invents an
out-of-enum string (gpt-5.4 did this with `level: "caption"` in
ab-v4), the lookup `TONES[bad]` / `STATUS_TONE[bad]` returned
undefined, the next property access crashed mid-batch with a
cryptic `undefined is not an object`, and the surrounding dispatch
loop dropped every remaining tag (until df33e937 + 07639f6d landed
the per-shape continuation + partial-doc scoring earlier today).
With validation in place, a bad enum becomes a clean per-shape
error message + the rest of the batch still applies.
13 new edge-case tests cover throw on bad input + valid path on
every enum value + omitted-default for each builder. 3785 vitest
pass, format clean, tsc silent.
Builders not touched: heading.ts (already done in 113bd55a).
Builders that don't fit this pattern (no enum→Record lookup of a
user-controlled string): everything else surveyed via grep on
`Record<.*Tone|Status|Level|Mode|Kind`.
Codex stop-time review caught the previous fix (df33e937) handing
the scorer a partial PenDocument that the scorer immediately
ignored. score-run.ts:87 short-circuited on `!applied.ok || !applied.doc`,
so even though apply.ts now surfaces 12 of 13 successfully-applied
tags as a populated `applied.doc`, the row still scored as a total
failure (M1=false, M3=false, m3_failure_reason="apply failed before
shape checks") — exactly the noise df33e937 was meant to eliminate.
Loosens the short-circuit to `!applied.doc` only. When apply.ok=false
but apply.doc is populated, the scorer now:
- runs the issue detector against the partial doc (issues surface)
- keeps M1 strict (apply.ok=false → M1=false regardless of detector)
- decouples M3 from M1: M3 = shape.ok against the partial doc
- sets m3_failure_reason to the shape miss when shape fails;
otherwise to "partial apply (M3 met by what landed): <error>" so
the row reads "tag 12 of 13 broke, but role coverage still met"
instead of silently swallowing the partial signal
- surfaces applied.error in row.applyError so per-shape failure
messages flow into reports
Two new tests cover the new path:
1. partial apply + shape match → M1=false, M3=true, reason mentions
"partial apply"
2. partial apply + shape miss → M1=false, M3=false, shape-miss
reason wins (structural verdict trumps the partial-apply notice)
Plumbing chain across today's session is now consistent:
- apply.ts continues past per-shape failures (df33e937)
- score-run.ts scores the partial doc that lands (this commit)
- scoring no longer over-attributes to "apply failed" when the model
actually produced most of the brief
3774 vitest pass (+2), format clean, tsc silent.
Codex stop-time review caught the previous commit (113bd55a) message
overstating apply.ts's behavior — I claimed "scripts/ab-corpus/apply
both catch per-shape and keep running the remaining tags" but the
loop at line 55 had no inner try/catch. A single throw from any
handleElementToolCall (e.g. the heading invalid-level reject 113bd55a
just added) would bubble up through the outer try at line 40 and
return early, dropping every remaining tag in a composite batch on
the floor — gpt-5.4's 13-tag team-people-page response would lose
tags 12-13 instead of just tag 12.
Wraps each handleElementToolCall in its own try/catch + accumulates
failures into a per-shape list. ELEMENT_TOOL_NAMES miss is also a
push-and-continue (was a return). When `failures.length > 0` we
return ok:false with a message listing every failed tag, AND the
partial PenDocument that DID land — so M3 (role coverage) can still
score the 11 tags that worked. M1 stays strict (any failure → false).
Mirrors apps/web/src/services/ai/element-tools-dispatcher::
dispatchElementToolCalls's "collect-errors-keep-going" semantics —
production already worked this way; ab-corpus now does too.
3772 vitest pass, format clean, tsc silent. Existing dry-run + live
sweeps exercise the path; a focused apply.ts unit test would need
pen-mcp setup that the harness's existing build-prompt test sidesteps,
so leaving that as a followup.
ab-v4 partial sweep (2026-05-01) caught gpt-5.4 emitting
`add_heading_v0({"content":"Pending invitations","level":"caption"})`
as the 12th tag of a 13-tag composite multi-tool response. Even
though the MCP tool def has `enum: ['display','h1','h2','h3']`, the
ab-corpus harness and the in-process production dispatcher both call
buildHeading() with raw JSON args (no jsonschema gate), so the model's
invented "caption" reached the preset lookup. LATIN_PRESETS["caption"]
is undefined, and the next line `fontSize: preset.fontSize` crashed
the WHOLE batch with `undefined is not an object (evaluating
'preset.fontSize')` — the 11 valid tags ahead of it never landed.
Adds an entry-point validation in buildHeading: if `level` is set and
not in the {display, h1, h2, h3} set, throw with a clear message.
The dispatch loops in apps/web/element-tools-dispatcher and
scripts/ab-corpus/apply both catch per-shape and keep running the
remaining tags, so a single bad level on tag 12 no longer kills tags
1-11 + 13.
3 new edge-case tests in element-builders-edge-cases.test.ts
cover the throw + the four valid levels + the omitted-default case.
3772 vitest pass (+3), format clean, tsc silent.
Other element builders likely have the same pattern (preset lookup
on a string enum without runtime validation) — separate sweep, not
shotgunning here.
Codex stop-time review caught the next inconsistency: f5d9a29c
switched the orchestrator to tryParseAllElementToolOutputs +
dispatchElementToolCalls, but the plural parser cheerfully returned
both element-tool AND batch_design shapes side by side. A
non-compliant model (saw this on minimax-m2.7 in the ab-v4
search-filters composite — 3 element tools + 1 batch_design
scaffolding tag) would slip the forbidden mixed strategy through
the dispatcher, applying the element calls AND the batch_design
DSL together — exact thing the prompt forbids and exact thing the
ab-corpus output-parser silently rejects on the harness side.
Aligns the production parser with corpus output-parser.ts: when
ANY element-tool tag is present in the response, batch_design tags
are DROPPED. Pure Strategy B (no element-tool tags, only batch_design
fallback) keeps working — the drop only fires on mixed output.
Two new regression tests:
- mixed input → only element-tool shapes returned
- pure batch_design input → batch-design-dsl shape returned
3769 vitest pass (+2), format clean, tsc silent. Together with
f5d9a29c and 1a14a6c2, production now has prompt + parser + dispatch
all consistent with the Strategy A/B contract — no path can smuggle
mixed output past any of them.
The Strategy A prompt I shipped in 1a14a6c2 invites the model to
chain N op_tool tags ("settings panel with 4 toggle rows is 5 tool
calls"), but the orchestrator sub-agent was still calling the SINGULAR
tryParseElementToolOutput → dispatchElementToolCall path, which
silently kept only the first tag. A composite-T response with 5 tags
would render only the section header and drop the 4 setting rows on
the floor — exact thing the prompt promises won't happen. Codex
stop-time review caught it.
design-parser and element-tools-dispatcher already had the plural
counterparts (tryParseAllElementToolOutputs, dispatchElementToolCalls)
plumbed end-to-end with one history batch wrapping the whole loop.
Switches the orchestrator to use those.
Failure handling: BatchDispatchResult exposes per-shape DispatchResult
in `results`. When status != 'applied' we concatenate the failed
shapes' messages tagged by toolName so the UI's diagnostic preview
shows which tag(s) broke instead of a generic "dispatch failed".
Partial successes still surface their inserted nodes through
onApplyPartial — the user sees what landed, plus an error summary
naming the broken pieces.
3767 vitest pass, format clean, tsc silent. End-to-end: web app
chat / orchestrator now actually realizes the multi-tool gain my
1a14a6c2 prompt change advertised.
The orchestrator sub-agent's ELEMENT_TOOL_OUTPUT_FORMAT was still
running the pre-Codex-fix wording from before today's ab-corpus pass:
- "Respond with one <op_tool> tag, nothing else"
- "Do not combine multiple tags"
Same self-defeating prompt that gave ab-v3 0/25 composite multi-tool
runs. Production code path stayed broken while the harness kept
getting fixed. Caught when investigating ab-v4's gpt-5.4
search-filters garbage — orchestrator-sub-agent's leading comment
explicitly says it's kept verbatim against the ab-corpus version.
Aligns with the latest scripts/ab-corpus/build-prompt.ts version:
- "Respond with one or more <op_tool> tags" (multi-tool allowed)
- STRATEGY A — element tools, one tag per component, with a 3-tag
worked example
- EMBEDDED COVERAGE — production-specific block listing the subset
of add_*_v0 tools the embedded orchestrator can actually execute,
inserted between Strategy A and Strategy B (the ab-corpus harness
has full coverage so it doesn't need this block)
- STRATEGY B — single batch_design covering the whole response when
any component falls outside EMBEDDED COVERAGE
- Explicit "Do not mix Strategy A and Strategy B" guard, naming the
parser's silent-drop behavior
design-parser.ts::tryParseElementToolOutput already collects every
`<op_tool>` tag into tool_calls (line 61: `parsed.kind === 'tool_calls'
&& parsed.calls.length > 0`), so the multi-tool path works end-to-end
on the production parser side too — no parser change needed.
3767 vitest pass, format clean, tsc silent. Real-user impact: web app
chat / orchestrator runs against minimax / glm / kimi / deepseek now
get the same multi-tool teaching that took composite routing from
0% to 42% in ab-v4.
Codex stop-time review caught that the Phase 2 domain filter
corrupted output when two adjacent kept blocks abutted: the regex's
trailing `\s*` consumed every character of whitespace BEFORE the next
match's leading `\s*`, so block N's body ended on its closing ``` and
block N+1's body started on its `### heading` with no separator at
all. Output: ` ```### Audit / activity feed` on a single line — fence
left unclosed, heading swallowed.
Replace the greedy `\s*` on both sides of each marker with a literal
`\n`, so the regex only consumes the single newline immediately
adjoining the comment. Surrounding blank lines stay in the
surrounding text where they belong, separating adjacent kept blocks
naturally.
Adds a regression guard test that searches the output for ` ```###`
(closing fence directly followed by a level-3 heading on the same
line). Codex effectively asked for it.
3767 vitest pass, format clean. Side effect: dropped blocks now
leave their surrounding blank lines intact, so dropped-block savings
shrink by ~3-5 chars per block; total Phase 2 savings stay within
the 500-char floor the existing test asserts.
ab-v3 / ab-v4 showed Phase 1A (cookbook strip on obvious difficulty)
shaved ~4.4k tokens off T-obvious. Phase 2 adds a per-category gate
that strips cookbook recipes whose domain doesn't match the prompt's
category — mobile briefs don't see dashboard recipes, dashboard
briefs don't see mobile / landing recipes, etc.
Mechanism: HTML comment block markers in elements.md
(`<!-- @domain:dashboard --> ... <!-- /@domain -->`) plus a
stripNonMatchingDomains() pass in buildSystemPrompt that drops blocks
whose tag list doesn't include the active category. Untagged content
is "general" and stays in every variant — the safe default.
Tagged 7 single-domain cookbook recipes:
- dashboard: Team / members list, Audit / activity feed, Faceted
search filter sidebar, Dashboard KPI strip
- landing: Pricing section
- mobile: Onboarding "How it works", Support chat thread
Cross-domain recipes (Login, Signup, Settings page, OTP, Empty
inbox) stay untagged so they load for every category. Decision tree
+ PREFER list also untagged today; the per-tool annotations there
would be a much larger judgment pass for marginal additional savings.
Token measurements (chars / 4 estimate):
full mobile dashboard landing
- T + composite 19.0k 17.9k 18.4k 17.7k
(-1.1k) (-0.6k) (-1.3k)
- T + obvious 14.6k 13.5k 14.0k 13.3k
(-1.1k) (-0.6k) (-1.3k)
Modest absolute savings — Phase 2 only filters cookbook RECIPES (in
elements.md), and most cookbook content is in elements-cookbook.md
which Phase 1A already strips on obvious. To hit the 6-8k T target
we still need decision-tree compression or PREFER-list trim, but
both are lossier than this gate. Phase 3 candidates noted in the
ab-v4 results doc.
real-model.ts plumbs call.prompt.category through to buildSystemPrompt.
3766 vitest pass (+6 category filter tests including a 500-char
floor regression guard that the filter actually shaves bytes).
ab-v4 raw output capture on dashboard-search-filters-composite shows
minimax-m2.7 emitting <think>...</think> + 4 op_tool tags that fit
inside the 4096 default — its measured completion-token average for
this run was 697, well under the cap. So thinking-budget truncation
is NOT the actual root cause of minimax's lower multi-tool hit rate
(25% vs gpt+deepseek 50%); the real issues are instruction-following
(mixed Strategy A + B despite the explicit forbidance, invented
"canvas" parent_id placeholder).
Still doubling the cap defensively: composite multi-tool outputs can
chain 12-13 op_tool tags + thinking, and "fit easy" today doesn't
mean "fits headroom-free on a longer brief tomorrow." The bump is
free on the happy path (provider stops generating when done, doesn't
bill unused headroom) and only ever helps when the model would
otherwise hit a real ceiling.
Real follow-up for minimax: instruction compliance — the no-mix rule
needs to land harder than a single trailing sentence. Probably wants
the rule moved to top-of-prompt + a few-shot bad-example contrast.
Out of scope here; tracked under Phase 2 prompt design.
ab-v4's dashboard-search-filters-composite garbaged on gpt-5.4 with
"codex timed out after 300000ms" — Codex's own agent framing
(~20k tokens) plus our 18.9k composite system prompt plus thinking
budget plus a 13-tool composite output is enough to blow the old
5-minute cap. The same model produced 13 chained op_tool tags on
team-people-page-composite within the window, so we know it's
generation latency under heavy briefs, not a hung CLI.
Doubles the default to 600000ms and adds AB_CORPUS_CODEX_TIMEOUT_MS
to override either way (lower it to surface slowness as a hard fail
when iterating, raise it for one-off long-form runs). Pairs with
the existing AB_CORPUS_CALL_TIMEOUT_MS for openai-compat clients.
Doesn't fix the deepseek empty-content failure on the same prompt
(already covered by retries=2 + exp backoff) or the minimax mix
+ invented "canvas" parent_id (model-side instruction skip; out of
scope for this commit).
Codex stop-time review (3rd round) caught residual "T prompt
includes mixed-strategy instructions": even after the prompt
forbade mixing, elements.md still taught the mixed pattern in
several places that are also part of the T system prompt.
Cleaned out every spot that paired batch_design with add_*_v0:
- Login screen / Pricing section / Dashboard KPI strip recipes:
dropped the leading `batch_design: foo = I("page", {...})` line
and renamed `<foo>` / `<row>` placeholders to `<page>` so each
recipe is now Strategy A (element tools only). Lost: explicit
page-level layout/padding/horizontal row — acceptable, the
recipes still teach the tool selection + chain pattern.
- Intro paragraph: dropped "override via a follow-up batch_design
U-op if needed" (taught a per-component fallback that the parser
drops).
- Banner: rewrote the fall-back clause from "when no element tool
fits a specific component shape" to "when at least one component
truly needs a custom shape no element tool covers — and then use
a SINGLE batch_design for the WHOLE response, never mixed."
- "STILL use batch_design when" list: collapsed 3 mixed-strategy
bullets ("larger composite via batch_design then element tool",
"post-hoc styling via batch_design U-ops") into 3 clean
Strategy-B-only bullets, all explicitly emit a SINGLE batch_design.
- Removed the "## Composition pattern" section entirely — its 3-step
plan was a textbook mixed pattern (batch_design root → element
tool inserts → batch_design U-op styling) that depended on real
MCP multi-round semantics the corpus harness can't provide.
- Composition rules of thumb: replaced "Don't mix N-tool and
batch_design DSL ops in a single call" + "Style overrides come
AFTER structure" with a single "Don't mix in the same output"
rule that names the corpus parser behavior explicitly.
3760 vitest pass, format clean, tsc silent. T-obvious 14.5k tokens
(down ~100 from the cleanup); composite still 18.9k.
The earlier T instruction split tool selection into PRIMARY / COMPOSITE
(element tools) and FALLBACK (batch_design). FALLBACK was scoped
"when no element tool fits a given component shape" — implying you
could mix per-component. But output-parser.ts silently drops every
batch_design tag whenever any element call is also present (filter
ELEMENT_TOOL_NAME_RE then return). A mixed response
<op_tool>{"name":"add_section_header_v0",...}</op_tool>
<op_tool>{"name":"batch_design", ...scaffolding...}</op_tool>
would lose the batch_design half and only run the element call —
the brief is half-applied without anyone noticing. Codex stop-time
review caught the mismatch.
Reframes T as a binary choice:
- STRATEGY A: every component fits an add_*_v0 tool — emit one tag
per component
- STRATEGY B: at least one component needs batch_design — emit a
SINGLE batch_design covering the whole brief
Plus an explicit "Do not mix Strategy A and Strategy B" guard line.
Tests assert both strategy markers are present in every T variant
and the dropped per-component-fallback phrase is gone.
The elements.md cookbook still teaches batch_design + element tool
composition for real MCP multi-round usage; the corpus T arm is
single-shot so the trailing T_TOOL_CALL_INSTRUCTIONS overrides via
"last instruction wins".
Codex stop-time review caught a contradiction: elements.md taught
"emit one <op_tool> per component" while T_TOOL_CALL_INSTRUCTIONS
explicitly forbade it ("Respond with one tag, nothing else" + "Do
not combine multiple tags"). Live models were probably reading the
later, more authoritative trailing instructions and dropping back to
batch_design — which matches ab-v3's 0/25 composite-T multi-tool
runs even though the cookbook had recipes.
Rewrites the T output-format block to:
- declare "one or more <op_tool> tags" up front
- add an explicit COMPOSITE clause showing chained tags for
multi-component briefs (settings panel / team list / audit feed /
onboarding) with a 3-tag worked example
- keep the FALLBACK clause for batch_design when no element tool fits
- keep the "no prose between tags" rule
Tests now assert the multi-tool marker is present in every T variant
AND the forbidding phrases ("one tag, nothing else", "Do not combine
multiple tags") are GONE — the regression guard Codex effectively
asked for. Also pulls in an oxfmt auto-format on the measurement
helper that landed unformatted in a3c5bf2d.
One-shot diagnostic that prints char + token-estimate sizes for every
(variant, difficulty) combination of buildSystemPrompt. Used to verify
the elements-cookbook diet is shaving the bytes the test floor
predicts (>10kb) before running an actual sweep. Confirms today's
numbers: B 15kb, T-obvious 57kb, T-composite 75kb — diet saves ~17kb
(~4.4k tokens) on every obvious prompt. Phase 2 (per-domain split)
still needed to hit the 6-8k target.
ab-v3 left T prompts at ~22k tokens (vs ~6-7k for B). The 18kb
elements-cookbook teaches per-tool arg shapes, which is what
composite multi-tool chains genuinely need; single-tool obvious
prompts can route correctly from just the decision tree + PREFER
list alone.
buildSystemPrompt now takes opts.difficulty. T+obvious strips the
cookbook (saves ~18kb on the 47/52 ab-v3 obvious prompts);
T+composite, T+optional, and undefined keep both halves. B variant
unchanged — still strips both, so the A/B comparison stays clean.
Verified by 8 new build-prompt.test.ts cases including a 10kb floor
on the obvious-vs-composite delta. ab-v4 will measure whether the
diet hurts arg compliance on weak models; per-domain split is the
Phase 2 fallback if obvious-T garbage rate creeps up.
ab-v3 left 36 ark empty + 15 timeout + 4 429 + 6 deepseek empty + 4
minimax timeout AFTER the existing retries=1 fired 88 times. Linear
backoff 250ms*(attempt+1) was too tight when stepping up to
retries=2 (500ms then 750ms isn't a typical Ark recovery window).
Switches to exponential 250ms*4^attempt — 250 / 1000 / 4000ms
spacing — and bumps ark + deepseek to retries=2. minimax opts in to
retries=1 (its 4 errors were wall-clock timeouts, not the model's
<think> truncation that retry can't fix anyway).
10 existing retry tests still pass; the retries=2 case now sleeps
1.25s instead of 0.75s, still well under vitest's default timeout.
Codex stop-time review flagged the new ab-v3 composite cookbook
recipes calling add_section_header_v0 with a `subtitle` arg the tool
doesn't accept (silently dropped at runtime today, but teaches live
models to emit invalid shapes). The same bug was in the dry-run stub
fixtures.
Split each header into add_section_header_v0(title) +
add_body_text_v0(content) — semantically what the ab-v3 briefs ask
for, and reinforces the multi-tool chaining the cookbook now teaches.
Also fills in the missing required `number` arg on the onboarding
recipe's final completed step card (schema requires it even when
completed=true renders a check instead of the number).
ab-v3 live sweep showed 0/25 composite-T runs walked the multi-tool
path — every model fell back to batch_design or garbage when given a
brief like "5 member rows + 1 invite row" or "6 audit log entries".
Root cause: elements.md's decision tree opens with "pick first match"
(single-tool framing) and the cookbook's chained examples sit ~400
lines deep, so models stop at the lookup and never realize they can
emit N <op_tool> blocks.
This adds:
- top-of-file "MULTI-TOOL OUTPUT IS THE NORM" banner with a concrete
5-call settings example, before the decision tree
- decision tree heading rewritten to "(per component — pick first
match)" so per-component framing is in scope from the start
- 4 new cookbook recipes mirroring the ab-v3 composite prompts:
team / members list (rows + invite), audit / activity feed,
faceted search filter sidebar, onboarding step cards
Markdown-only edit to a single skill file; vite-plugin-skills
re-compiles the registry. Banner verified present in the generated
registry, all 3750 vitest tests pass. Verifies in next ab-v4 sweep.
Codex stop-hook caught: ab-v3 introduced composite-difficulty prompts
that *expect* multi-tool emit (e.g. 5× member_row + 1× invite_row
for a team page), but `ParsedOutput.tool_call` was a single
{name, arguments} so the parser silently dropped every call after
the first. apply.ts only invoked one tool, M3 min_roles couldn't
pass on legitimately-routed multi-tool runs, and byTool stats
under-counted. The composite routing 'multi-tool' bucket was
correctly assigned in classifyRouting, but downstream the pipeline
behaved as if the model emitted a single call.
This commit replaces `kind: 'tool_call'` with
`kind: 'tool_calls'` (NON-EMPTY list) across every consumer:
- types.ts: ParsedOutput tagged union; new ParsedOpToolCall.
ScoreRow.toolName → toolNames: string[].
- output-parser.ts: collects ALL element-tool tags in emit order;
unknown-tool path also surfaces as single-element tool_calls so
routing keeps the same wrong-tool semantics.
- score-run.ts: classifyRouting uses Array.includes for obvious
prompts (right-tool when ANY emitted call matches expected_tool —
over-production isn't a routing miss). Composite stays multi-tool
on any non-empty list.
- aggregate.ts byTool: tallies EVERY name in toolNames, so a
composite row that emits 6× add_activity_log_v0 + 1×
add_section_header_v0 contributes 6+1 = 7 invocations across two
tools (with row-level m1_legal applied to both buckets — apply is
all-or-nothing).
- apply.ts: loops over parsed.calls and invokes
handleElementToolCall in emit order. Any single call failing
aborts the row (M1=false); we don't partial-apply.
- mock-llm.ts mockLlmParsed: collects all `<op_tool>` tags into the
list (composite-prompt mocks can carry multi-call raw strings).
- apps/web design-parser.tryParseElementToolOutput: maps tool_calls
→ its single-shape DesignOutputShape contract using the FIRST
call (the multi-tag path `tryParseAllElementToolOutputs` was
already correct).
Tests: 3746 → 3750 vitest. New cases:
- output-parser: surfaces ALL element-tool tags in emit order with
intermixed batch_design scaffolds dropped (3 element calls from
5 tags).
- score-run: right-tool when expected appears alongside extras;
composite multi-call captures every name in toolNames.
- aggregate: 6× activity_log + 1× section_header → byTool reports
6 and 1 invocations respectively.
dry-run on ab-v3 produces a 208-row report; tsc + format clean.
Adds 7 new obvious prompts covering the v0.8.0 element tools that
weren't in ab-v1 (tools 91-97):
setting_row / member_row / filter_group / invite_row /
activity_log / event_card / step_card
One yaml per tool, same single-component "Design ONLY..." pattern
as ab-v1, with must_contain_roles mirroring the role names emitted
by the corresponding builder in pen-core/src/element-builders/.
Adds 5 composite prompts that exercise the new M6 routing
breakdown:
- dashboard-settings-page-composite (4× setting_row)
- dashboard-team-people-page-composite (5× member_row + 1×
invite_row)
- dashboard-search-filters-composite (2× filter_group + result
list)
- dashboard-audit-feed-composite (6× activity_log)
- mobile-onboarding-flow-composite (4× step_card)
Each composite prompt omits expected_tool_if_any (multi-tool
intent) and uses min_roles to enforce the multi-element shape.
corpus-loader: composite added to VALID_DIFFICULTIES; composite
prompts MUST NOT specify expected_tool_if_any (validation error
points the user back to difficulty=obvious if a single tool fits).
6 new corpus-loader tests cover the v3 yaml inventory + composite
validation rules. 3740 → 3746 vitest tests, all green.
ab-v3 corpus now has 52 prompts: 40 inherited from v1 (unchanged
for v1↔v3 comparability) + 7 obvious + 5 composite.
ab-v3 succeeds ab-v1 (frozen 2026-04-28). Carries forward all 40
v1 obvious yaml files unchanged so the v1↔v3 overlap stays
comparable, then layers in two new dimensions.
**1. Token cost.** All clients (openai-compat, ark, bailian,
deepseek, minimax, codex-cli, stub-model) now return a
`ChatCallResult { content, usage }` instead of bare string.
Provider usage stats (`prompt_tokens` / `completion_tokens`) plumb
through realModelCall → run.ts → scoreRun → ScoreRow.{prompt,completion}Tokens.
aggregate adds avgPromptTokens{Baseline,Treatment} +
avgCompletionTokens{Baseline,Treatment} per ModelSummary.
write-report emits a new "Token cost" table with Δ columns so
narrow-tools-saves-tokens (the ab-v2 hypothesis) is measurable.
avgUsage skips rows with 0/0 usage so codex-cli (CLI doesn't
surface tokens) and harness errors don't deflate the average to
near-zero — they show '—' instead.
**2. Composite difficulty.** New 'composite' value alongside
obvious / optional. Composite prompts express multi-tool intents
where no single expected_tool_if_any applies. classifyRouting
routes composite-treatment runs into multi-tool / fallback /
garbage (3-bucket sum to 1, distinct from obvious's 4-bucket
right/wrong/fallback/garbage). aggregate adds m6_multi_tool +
m6_fallback + m6_garbage; write-report emits a "Composite routing"
table that gracefully degrades to a placeholder when no composite
yaml exists yet.
Harness side: scripts/ab-corpus/run.ts accepts --corpus ab-v3
(enum + parseArgs guard); dry-run on the v1-mirror corpus produces
a 160-row report including populated token table.
Tests: 4 new aggregate cases (composite, token avg with skip-zero,
NaN-when-no-data) + 4 new score-run cases (composite routing
multi-tool/fallback/garbage/baseline-n/a) + 2 new score-run cases
(usage plumbing) + 2 new openai-compat cases (usage parsing,
missing-usage fallback). Existing 5 retry tests updated for new
return shape. 3727 → 3740 vitest tests, all green; tsc + format
clean.
Token-cost docs and composite docs go straight into types.ts /
score-run.ts / aggregate.ts JSDoc — keeps the contract close to
the code that owns it.
ab-v2 (2026-04-28) saw kimi-k2.6 garbage rate hit 17.5% — every
failure was Ark returning empty `choices[0].message.content` or
hitting the 120s wall clock, not a model-quality issue (the model
itself routed to the right element tool 80% of the time when it
did respond). Same pattern at 7.5% on deepseek-v4-pro.
Adds optional `retries` to `callOpenAICompat` with a transient-error
allowlist: empty content, abort/timeout, HTTP 5xx, HTTP 429. Linear
250ms × (attempt+1) backoff. HTTP 4xx other than 429 stays fatal so
auth/bad-request failures don't burn retry budget.
Wires `retries: 1` through clients/ark.ts and clients/deepseek.ts.
MiniMax + Bailian + Codex stay untouched — their ab-v2 failures
were model-quality (DSL escape errors, output truncation), where
retry wastes a call without changing the outcome.
Adds an 8-case fixture in scripts/ab-corpus/clients/__tests__/ that
mocks fetch to verify: first-try success, empty-then-success,
5xx-then-success, 429-then-success, 401 fatal, retries-default-zero,
retries exhausted, and retries=2 (3 attempts total). Extends the
apps/web vitest include glob to pick up scripts/**/__tests__/.
DeepSeek wasn't in the harness; A/B v2 needed it for the 5-model run.
api.deepseek.com is OpenAI-compatible, so the client mirrors the
minimax pattern (callOpenAICompat with DEEPSEEK_API_KEY env, override-
able DEEPSEEK_BASE_URL).
Router: /^deepseek/i routes to the new client. Aliases `deepseek` /
`deepseek-pro` resolve to `deepseek-v4-pro` (current flagship per
docs); `deepseek-v4-flash` and the deprecated `deepseek-chat` /
`deepseek-reasoner` ids pass through verbatim until the 2026-07-24
sunset upstream.
Verified live with mobile-bio-textarea smoke (B emits batch_design,
T emits add_textarea_v0). Full ab-v2 results across all 5 models in
docs/notes/2026-04-28-ab-v2-results.md.
shim-server-parity.test.ts grew to 838 lines after the recent batch of
element tools added 7 fixture entries. Move the CASES table and the
build* imports to shim-server-parity-cases.ts (a sibling .ts, not
.test.ts so vitest doesn't pick it up as a separate suite); keep the
mocks, helpers, and describe blocks in the test file (now 139 lines).
Both files are well under the repo's 800-line ceiling. vitest's
vi.mock hoisting still works because the mock is registered at the
test file's module-resolution time, before the cases file's imports
resolve — confirmed by the 104 parity assertions still passing.
The previous schema description and JSDoc invited "Step 1" as a valid
value for `number`, but that prose has 6 chars and overflows the 36px
circle marker. Two changes:
- Builder: add clipContent: true on the marker frame so any caller
who ignores the docs at least gets a clipped (not bleeding) render.
- Schema + JSDoc: drop the misleading "Step 1" example, document the
1–3 character contract, and steer prose toward `title` instead.
Closes the obvious gaps remaining in the family:
- add_filter_group_v0 — sidebar facet (heading + checkbox-style options
with optional counts). Distinct from nav_chip_row (horizontal scrolling
chips), tag (single applied chip), segmented_control (mutex tabs).
- add_invite_row_v0 — pending invite row (avatar + email/role + status
pill + trailing action). Distinct from member_row (a JOINED member,
no status pill or action) and list_row (no avatar / status / action).
- add_activity_log_v0 — single-line audit feed entry (optional tinted
icon dot + actor in bold + action + right-aligned timestamp). Uses
StyledTextSegment[] content for the bold/regular split. Distinct from
timeline (multi-event vertical with connectors) and notification_row
(title + body, no actor focus).
- add_event_card_v0 — single calendar event tile (date column with
month band + day number, then title + time + location). Distinct from
calendar_grid (the full month grid) and card_row (no date column).
- add_step_card_v0 — onboarding step card (numbered circle / check +
title + description). Distinct from stepper (horizontal progress nav
with connectors) and faq_item (collapsible Q&A header).
9 touchpoints per tool: pen-core builder + index + barrel + types,
pen-mcp handler + dispatcher + ext-4 schema, apps/web shim +
SERVER_BUILDERS, parity test (+5 cases), elements.md decision tree
items 86-89 + 6 PREFER mappings with cross-links to existing tools,
elements-cookbook.md arg-shape examples (8 entries across 5 tools).
