Two diet/teaching changes to elements.md, both motivated by today's
ab-v4 smoke results:
(a) parent_id teaching — minimax-m2.7 invented "members-section" /
"canvas" as parent_id values on its multi-tool composite output,
then every one of its 14 tags failed apply with "parent_id X not
found in document". Same pattern as yesterday. Adds an explicit
rule near the top of elements.md (right under the multi-tool
banner) — `parent_id` is REAL or OMITTED, never invented. The
`<page>` / `<panel>` / `<sidebar>` placeholders in the cookbook
recipes are documentation conventions; in actual output, OMIT
the field. Names the failure mode by reproducing the error
message format so models learn to avoid it.
(b) Phase 3 token diet — strip ". Different from <tool> (...)"
disambiguation suffixes from the PREFER list (22 entries had
them, ~80-200 chars each = ~2.5kb / ~625 tokens saved). The
primary keyword + tool-name + capability description survives
intact; the cross-references pointing at sibling tools get
dropped. Risk: slight increase in wrong-tool routing on
ambiguous prompts. Worth it for the size reduction; the
decision tree alone still shows the tool family in context.
Verified by mechanical diff — perl in-place edit, then visual
review confirms no other content was touched. 3785 vitest pass,
format clean, tsc silent.
Net effect on T-prompt size (chars / 4 estimate):
T + composite + mobile: was 17,930 → now 17,475 (-455 chars)
T + composite + dashboard: was 18,437 → now 17,983 (-454 chars)
T + obvious + dashboard: ~14,009 → ~13,556 (-453 chars)
Per-arm savings are smaller than the raw 2.5kb trim because (a)
adds ~700 chars of parent_id teaching. Net win: ~450 chars / ~110
tokens per call. Modest but compounds across 520+ ab-v4 runs.
Sweep follow-up to 113bd55a — same defensive pattern (reject
unknown enum strings at the entry boundary) applied to every other
builder that indexed a Record<EnumLiteral, T> with a value sourced
from raw JSON args.
Builders + enums covered:
- buildTag — TagTone (default | accent | success | warning | error)
- buildCallout — CalloutTone (info | success | warning | danger | note)
- buildActivityLog — tone (info | success | warning | danger | neutral)
- buildInviteRow — InviteStatus (pending | expired | accepted)
- buildMemberRow — trailing.tone for status_dot (online | busy | away
| offline). role_badge / menu variants skip the check (no tone field)
Same failure mode each one fixed: when a model invents an
out-of-enum string (gpt-5.4 did this with `level: "caption"` in
ab-v4), the lookup `TONES[bad]` / `STATUS_TONE[bad]` returned
undefined, the next property access crashed mid-batch with a
cryptic `undefined is not an object`, and the surrounding dispatch
loop dropped every remaining tag (until df33e937 + 07639f6d landed
the per-shape continuation + partial-doc scoring earlier today).
With validation in place, a bad enum becomes a clean per-shape
error message + the rest of the batch still applies.
13 new edge-case tests cover throw on bad input + valid path on
every enum value + omitted-default for each builder. 3785 vitest
pass, format clean, tsc silent.
Builders not touched: heading.ts (already done in 113bd55a).
Builders that don't fit this pattern (no enum→Record lookup of a
user-controlled string): everything else surveyed via grep on
`Record<.*Tone|Status|Level|Mode|Kind`.
Codex stop-time review caught the previous fix (df33e937) handing
the scorer a partial PenDocument that the scorer immediately
ignored. score-run.ts:87 short-circuited on `!applied.ok || !applied.doc`,
so even though apply.ts now surfaces 12 of 13 successfully-applied
tags as a populated `applied.doc`, the row still scored as a total
failure (M1=false, M3=false, m3_failure_reason="apply failed before
shape checks") — exactly the noise df33e937 was meant to eliminate.
Loosens the short-circuit to `!applied.doc` only. When apply.ok=false
but apply.doc is populated, the scorer now:
- runs the issue detector against the partial doc (issues surface)
- keeps M1 strict (apply.ok=false → M1=false regardless of detector)
- decouples M3 from M1: M3 = shape.ok against the partial doc
- sets m3_failure_reason to the shape miss when shape fails;
otherwise to "partial apply (M3 met by what landed): <error>" so
the row reads "tag 12 of 13 broke, but role coverage still met"
instead of silently swallowing the partial signal
- surfaces applied.error in row.applyError so per-shape failure
messages flow into reports
Two new tests cover the new path:
1. partial apply + shape match → M1=false, M3=true, reason mentions
"partial apply"
2. partial apply + shape miss → M1=false, M3=false, shape-miss
reason wins (structural verdict trumps the partial-apply notice)
Plumbing chain across today's session is now consistent:
- apply.ts continues past per-shape failures (df33e937)
- score-run.ts scores the partial doc that lands (this commit)
- scoring no longer over-attributes to "apply failed" when the model
actually produced most of the brief
3774 vitest pass (+2), format clean, tsc silent.
Codex stop-time review caught the previous commit (113bd55a) message
overstating apply.ts's behavior — I claimed "scripts/ab-corpus/apply
both catch per-shape and keep running the remaining tags" but the
loop at line 55 had no inner try/catch. A single throw from any
handleElementToolCall (e.g. the heading invalid-level reject 113bd55a
just added) would bubble up through the outer try at line 40 and
return early, dropping every remaining tag in a composite batch on
the floor — gpt-5.4's 13-tag team-people-page response would lose
tags 12-13 instead of just tag 12.
Wraps each handleElementToolCall in its own try/catch + accumulates
failures into a per-shape list. ELEMENT_TOOL_NAMES miss is also a
push-and-continue (was a return). When `failures.length > 0` we
return ok:false with a message listing every failed tag, AND the
partial PenDocument that DID land — so M3 (role coverage) can still
score the 11 tags that worked. M1 stays strict (any failure → false).
Mirrors apps/web/src/services/ai/element-tools-dispatcher::
dispatchElementToolCalls's "collect-errors-keep-going" semantics —
production already worked this way; ab-corpus now does too.
3772 vitest pass, format clean, tsc silent. Existing dry-run + live
sweeps exercise the path; a focused apply.ts unit test would need
pen-mcp setup that the harness's existing build-prompt test sidesteps,
so leaving that as a followup.
ab-v4 partial sweep (2026-05-01) caught gpt-5.4 emitting
`add_heading_v0({"content":"Pending invitations","level":"caption"})`
as the 12th tag of a 13-tag composite multi-tool response. Even
though the MCP tool def has `enum: ['display','h1','h2','h3']`, the
ab-corpus harness and the in-process production dispatcher both call
buildHeading() with raw JSON args (no jsonschema gate), so the model's
invented "caption" reached the preset lookup. LATIN_PRESETS["caption"]
is undefined, and the next line `fontSize: preset.fontSize` crashed
the WHOLE batch with `undefined is not an object (evaluating
'preset.fontSize')` — the 11 valid tags ahead of it never landed.
Adds an entry-point validation in buildHeading: if `level` is set and
not in the {display, h1, h2, h3} set, throw with a clear message.
The dispatch loops in apps/web/element-tools-dispatcher and
scripts/ab-corpus/apply both catch per-shape and keep running the
remaining tags, so a single bad level on tag 12 no longer kills tags
1-11 + 13.
3 new edge-case tests in element-builders-edge-cases.test.ts
cover the throw + the four valid levels + the omitted-default case.
3772 vitest pass (+3), format clean, tsc silent.
Other element builders likely have the same pattern (preset lookup
on a string enum without runtime validation) — separate sweep, not
shotgunning here.
Codex stop-time review caught the next inconsistency: f5d9a29c
switched the orchestrator to tryParseAllElementToolOutputs +
dispatchElementToolCalls, but the plural parser cheerfully returned
both element-tool AND batch_design shapes side by side. A
non-compliant model (saw this on minimax-m2.7 in the ab-v4
search-filters composite — 3 element tools + 1 batch_design
scaffolding tag) would slip the forbidden mixed strategy through
the dispatcher, applying the element calls AND the batch_design
DSL together — exact thing the prompt forbids and exact thing the
ab-corpus output-parser silently rejects on the harness side.
Aligns the production parser with corpus output-parser.ts: when
ANY element-tool tag is present in the response, batch_design tags
are DROPPED. Pure Strategy B (no element-tool tags, only batch_design
fallback) keeps working — the drop only fires on mixed output.
Two new regression tests:
- mixed input → only element-tool shapes returned
- pure batch_design input → batch-design-dsl shape returned
3769 vitest pass (+2), format clean, tsc silent. Together with
f5d9a29c and 1a14a6c2, production now has prompt + parser + dispatch
all consistent with the Strategy A/B contract — no path can smuggle
mixed output past any of them.
The Strategy A prompt I shipped in 1a14a6c2 invites the model to
chain N op_tool tags ("settings panel with 4 toggle rows is 5 tool
calls"), but the orchestrator sub-agent was still calling the SINGULAR
tryParseElementToolOutput → dispatchElementToolCall path, which
silently kept only the first tag. A composite-T response with 5 tags
would render only the section header and drop the 4 setting rows on
the floor — exact thing the prompt promises won't happen. Codex
stop-time review caught it.
design-parser and element-tools-dispatcher already had the plural
counterparts (tryParseAllElementToolOutputs, dispatchElementToolCalls)
plumbed end-to-end with one history batch wrapping the whole loop.
Switches the orchestrator to use those.
Failure handling: BatchDispatchResult exposes per-shape DispatchResult
in `results`. When status != 'applied' we concatenate the failed
shapes' messages tagged by toolName so the UI's diagnostic preview
shows which tag(s) broke instead of a generic "dispatch failed".
Partial successes still surface their inserted nodes through
onApplyPartial — the user sees what landed, plus an error summary
naming the broken pieces.
3767 vitest pass, format clean, tsc silent. End-to-end: web app
chat / orchestrator now actually realizes the multi-tool gain my
1a14a6c2 prompt change advertised.
The orchestrator sub-agent's ELEMENT_TOOL_OUTPUT_FORMAT was still
running the pre-Codex-fix wording from before today's ab-corpus pass:
- "Respond with one <op_tool> tag, nothing else"
- "Do not combine multiple tags"
Same self-defeating prompt that gave ab-v3 0/25 composite multi-tool
runs. Production code path stayed broken while the harness kept
getting fixed. Caught when investigating ab-v4's gpt-5.4
search-filters garbage — orchestrator-sub-agent's leading comment
explicitly says it's kept verbatim against the ab-corpus version.
Aligns with the latest scripts/ab-corpus/build-prompt.ts version:
- "Respond with one or more <op_tool> tags" (multi-tool allowed)
- STRATEGY A — element tools, one tag per component, with a 3-tag
worked example
- EMBEDDED COVERAGE — production-specific block listing the subset
of add_*_v0 tools the embedded orchestrator can actually execute,
inserted between Strategy A and Strategy B (the ab-corpus harness
has full coverage so it doesn't need this block)
- STRATEGY B — single batch_design covering the whole response when
any component falls outside EMBEDDED COVERAGE
- Explicit "Do not mix Strategy A and Strategy B" guard, naming the
parser's silent-drop behavior
design-parser.ts::tryParseElementToolOutput already collects every
`<op_tool>` tag into tool_calls (line 61: `parsed.kind === 'tool_calls'
&& parsed.calls.length > 0`), so the multi-tool path works end-to-end
on the production parser side too — no parser change needed.
3767 vitest pass, format clean, tsc silent. Real-user impact: web app
chat / orchestrator runs against minimax / glm / kimi / deepseek now
get the same multi-tool teaching that took composite routing from
0% to 42% in ab-v4.
Codex stop-time review caught that the Phase 2 domain filter
corrupted output when two adjacent kept blocks abutted: the regex's
trailing `\s*` consumed every character of whitespace BEFORE the next
match's leading `\s*`, so block N's body ended on its closing ``` and
block N+1's body started on its `### heading` with no separator at
all. Output: ` ```### Audit / activity feed` on a single line — fence
left unclosed, heading swallowed.
Replace the greedy `\s*` on both sides of each marker with a literal
`\n`, so the regex only consumes the single newline immediately
adjoining the comment. Surrounding blank lines stay in the
surrounding text where they belong, separating adjacent kept blocks
naturally.
Adds a regression guard test that searches the output for ` ```###`
(closing fence directly followed by a level-3 heading on the same
line). Codex effectively asked for it.
3767 vitest pass, format clean. Side effect: dropped blocks now
leave their surrounding blank lines intact, so dropped-block savings
shrink by ~3-5 chars per block; total Phase 2 savings stay within
the 500-char floor the existing test asserts.
ab-v3 / ab-v4 showed Phase 1A (cookbook strip on obvious difficulty)
shaved ~4.4k tokens off T-obvious. Phase 2 adds a per-category gate
that strips cookbook recipes whose domain doesn't match the prompt's
category — mobile briefs don't see dashboard recipes, dashboard
briefs don't see mobile / landing recipes, etc.
Mechanism: HTML comment block markers in elements.md
(`<!-- @domain:dashboard --> ... <!-- /@domain -->`) plus a
stripNonMatchingDomains() pass in buildSystemPrompt that drops blocks
whose tag list doesn't include the active category. Untagged content
is "general" and stays in every variant — the safe default.
Tagged 7 single-domain cookbook recipes:
- dashboard: Team / members list, Audit / activity feed, Faceted
search filter sidebar, Dashboard KPI strip
- landing: Pricing section
- mobile: Onboarding "How it works", Support chat thread
Cross-domain recipes (Login, Signup, Settings page, OTP, Empty
inbox) stay untagged so they load for every category. Decision tree
+ PREFER list also untagged today; the per-tool annotations there
would be a much larger judgment pass for marginal additional savings.
Token measurements (chars / 4 estimate):
full mobile dashboard landing
- T + composite 19.0k 17.9k 18.4k 17.7k
(-1.1k) (-0.6k) (-1.3k)
- T + obvious 14.6k 13.5k 14.0k 13.3k
(-1.1k) (-0.6k) (-1.3k)
Modest absolute savings — Phase 2 only filters cookbook RECIPES (in
elements.md), and most cookbook content is in elements-cookbook.md
which Phase 1A already strips on obvious. To hit the 6-8k T target
we still need decision-tree compression or PREFER-list trim, but
both are lossier than this gate. Phase 3 candidates noted in the
ab-v4 results doc.
real-model.ts plumbs call.prompt.category through to buildSystemPrompt.
3766 vitest pass (+6 category filter tests including a 500-char
floor regression guard that the filter actually shaves bytes).
ab-v4 raw output capture on dashboard-search-filters-composite shows
minimax-m2.7 emitting <think>...</think> + 4 op_tool tags that fit
inside the 4096 default — its measured completion-token average for
this run was 697, well under the cap. So thinking-budget truncation
is NOT the actual root cause of minimax's lower multi-tool hit rate
(25% vs gpt+deepseek 50%); the real issues are instruction-following
(mixed Strategy A + B despite the explicit forbidance, invented
"canvas" parent_id placeholder).
Still doubling the cap defensively: composite multi-tool outputs can
chain 12-13 op_tool tags + thinking, and "fit easy" today doesn't
mean "fits headroom-free on a longer brief tomorrow." The bump is
free on the happy path (provider stops generating when done, doesn't
bill unused headroom) and only ever helps when the model would
otherwise hit a real ceiling.
Real follow-up for minimax: instruction compliance — the no-mix rule
needs to land harder than a single trailing sentence. Probably wants
the rule moved to top-of-prompt + a few-shot bad-example contrast.
Out of scope here; tracked under Phase 2 prompt design.
ab-v4's dashboard-search-filters-composite garbaged on gpt-5.4 with
"codex timed out after 300000ms" — Codex's own agent framing
(~20k tokens) plus our 18.9k composite system prompt plus thinking
budget plus a 13-tool composite output is enough to blow the old
5-minute cap. The same model produced 13 chained op_tool tags on
team-people-page-composite within the window, so we know it's
generation latency under heavy briefs, not a hung CLI.
Doubles the default to 600000ms and adds AB_CORPUS_CODEX_TIMEOUT_MS
to override either way (lower it to surface slowness as a hard fail
when iterating, raise it for one-off long-form runs). Pairs with
the existing AB_CORPUS_CALL_TIMEOUT_MS for openai-compat clients.
Doesn't fix the deepseek empty-content failure on the same prompt
(already covered by retries=2 + exp backoff) or the minimax mix
+ invented "canvas" parent_id (model-side instruction skip; out of
scope for this commit).
Codex stop-time review (3rd round) caught residual "T prompt
includes mixed-strategy instructions": even after the prompt
forbade mixing, elements.md still taught the mixed pattern in
several places that are also part of the T system prompt.
Cleaned out every spot that paired batch_design with add_*_v0:
- Login screen / Pricing section / Dashboard KPI strip recipes:
dropped the leading `batch_design: foo = I("page", {...})` line
and renamed `<foo>` / `<row>` placeholders to `<page>` so each
recipe is now Strategy A (element tools only). Lost: explicit
page-level layout/padding/horizontal row — acceptable, the
recipes still teach the tool selection + chain pattern.
- Intro paragraph: dropped "override via a follow-up batch_design
U-op if needed" (taught a per-component fallback that the parser
drops).
- Banner: rewrote the fall-back clause from "when no element tool
fits a specific component shape" to "when at least one component
truly needs a custom shape no element tool covers — and then use
a SINGLE batch_design for the WHOLE response, never mixed."
- "STILL use batch_design when" list: collapsed 3 mixed-strategy
bullets ("larger composite via batch_design then element tool",
"post-hoc styling via batch_design U-ops") into 3 clean
Strategy-B-only bullets, all explicitly emit a SINGLE batch_design.
- Removed the "## Composition pattern" section entirely — its 3-step
plan was a textbook mixed pattern (batch_design root → element
tool inserts → batch_design U-op styling) that depended on real
MCP multi-round semantics the corpus harness can't provide.
- Composition rules of thumb: replaced "Don't mix N-tool and
batch_design DSL ops in a single call" + "Style overrides come
AFTER structure" with a single "Don't mix in the same output"
rule that names the corpus parser behavior explicitly.
3760 vitest pass, format clean, tsc silent. T-obvious 14.5k tokens
(down ~100 from the cleanup); composite still 18.9k.
The earlier T instruction split tool selection into PRIMARY / COMPOSITE
(element tools) and FALLBACK (batch_design). FALLBACK was scoped
"when no element tool fits a given component shape" — implying you
could mix per-component. But output-parser.ts silently drops every
batch_design tag whenever any element call is also present (filter
ELEMENT_TOOL_NAME_RE then return). A mixed response
<op_tool>{"name":"add_section_header_v0",...}</op_tool>
<op_tool>{"name":"batch_design", ...scaffolding...}</op_tool>
would lose the batch_design half and only run the element call —
the brief is half-applied without anyone noticing. Codex stop-time
review caught the mismatch.
Reframes T as a binary choice:
- STRATEGY A: every component fits an add_*_v0 tool — emit one tag
per component
- STRATEGY B: at least one component needs batch_design — emit a
SINGLE batch_design covering the whole brief
Plus an explicit "Do not mix Strategy A and Strategy B" guard line.
Tests assert both strategy markers are present in every T variant
and the dropped per-component-fallback phrase is gone.
The elements.md cookbook still teaches batch_design + element tool
composition for real MCP multi-round usage; the corpus T arm is
single-shot so the trailing T_TOOL_CALL_INSTRUCTIONS overrides via
"last instruction wins".
Codex stop-time review caught a contradiction: elements.md taught
"emit one <op_tool> per component" while T_TOOL_CALL_INSTRUCTIONS
explicitly forbade it ("Respond with one tag, nothing else" + "Do
not combine multiple tags"). Live models were probably reading the
later, more authoritative trailing instructions and dropping back to
batch_design — which matches ab-v3's 0/25 composite-T multi-tool
runs even though the cookbook had recipes.
Rewrites the T output-format block to:
- declare "one or more <op_tool> tags" up front
- add an explicit COMPOSITE clause showing chained tags for
multi-component briefs (settings panel / team list / audit feed /
onboarding) with a 3-tag worked example
- keep the FALLBACK clause for batch_design when no element tool fits
- keep the "no prose between tags" rule
Tests now assert the multi-tool marker is present in every T variant
AND the forbidding phrases ("one tag, nothing else", "Do not combine
multiple tags") are GONE — the regression guard Codex effectively
asked for. Also pulls in an oxfmt auto-format on the measurement
helper that landed unformatted in a3c5bf2d.
One-shot diagnostic that prints char + token-estimate sizes for every
(variant, difficulty) combination of buildSystemPrompt. Used to verify
the elements-cookbook diet is shaving the bytes the test floor
predicts (>10kb) before running an actual sweep. Confirms today's
numbers: B 15kb, T-obvious 57kb, T-composite 75kb — diet saves ~17kb
(~4.4k tokens) on every obvious prompt. Phase 2 (per-domain split)
still needed to hit the 6-8k target.
ab-v3 left T prompts at ~22k tokens (vs ~6-7k for B). The 18kb
elements-cookbook teaches per-tool arg shapes, which is what
composite multi-tool chains genuinely need; single-tool obvious
prompts can route correctly from just the decision tree + PREFER
list alone.
buildSystemPrompt now takes opts.difficulty. T+obvious strips the
cookbook (saves ~18kb on the 47/52 ab-v3 obvious prompts);
T+composite, T+optional, and undefined keep both halves. B variant
unchanged — still strips both, so the A/B comparison stays clean.
Verified by 8 new build-prompt.test.ts cases including a 10kb floor
on the obvious-vs-composite delta. ab-v4 will measure whether the
diet hurts arg compliance on weak models; per-domain split is the
Phase 2 fallback if obvious-T garbage rate creeps up.
ab-v3 left 36 ark empty + 15 timeout + 4 429 + 6 deepseek empty + 4
minimax timeout AFTER the existing retries=1 fired 88 times. Linear
backoff 250ms*(attempt+1) was too tight when stepping up to
retries=2 (500ms then 750ms isn't a typical Ark recovery window).
Switches to exponential 250ms*4^attempt — 250 / 1000 / 4000ms
spacing — and bumps ark + deepseek to retries=2. minimax opts in to
retries=1 (its 4 errors were wall-clock timeouts, not the model's
<think> truncation that retry can't fix anyway).
10 existing retry tests still pass; the retries=2 case now sleeps
1.25s instead of 0.75s, still well under vitest's default timeout.
Codex stop-time review flagged the new ab-v3 composite cookbook
recipes calling add_section_header_v0 with a `subtitle` arg the tool
doesn't accept (silently dropped at runtime today, but teaches live
models to emit invalid shapes). The same bug was in the dry-run stub
fixtures.
Split each header into add_section_header_v0(title) +
add_body_text_v0(content) — semantically what the ab-v3 briefs ask
for, and reinforces the multi-tool chaining the cookbook now teaches.
Also fills in the missing required `number` arg on the onboarding
recipe's final completed step card (schema requires it even when
completed=true renders a check instead of the number).
ab-v3 live sweep showed 0/25 composite-T runs walked the multi-tool
path — every model fell back to batch_design or garbage when given a
brief like "5 member rows + 1 invite row" or "6 audit log entries".
Root cause: elements.md's decision tree opens with "pick first match"
(single-tool framing) and the cookbook's chained examples sit ~400
lines deep, so models stop at the lookup and never realize they can
emit N <op_tool> blocks.
This adds:
- top-of-file "MULTI-TOOL OUTPUT IS THE NORM" banner with a concrete
5-call settings example, before the decision tree
- decision tree heading rewritten to "(per component — pick first
match)" so per-component framing is in scope from the start
- 4 new cookbook recipes mirroring the ab-v3 composite prompts:
team / members list (rows + invite), audit / activity feed,
faceted search filter sidebar, onboarding step cards
Markdown-only edit to a single skill file; vite-plugin-skills
re-compiles the registry. Banner verified present in the generated
registry, all 3750 vitest tests pass. Verifies in next ab-v4 sweep.
Codex stop-hook caught: ab-v3 introduced composite-difficulty prompts
that *expect* multi-tool emit (e.g. 5× member_row + 1× invite_row
for a team page), but `ParsedOutput.tool_call` was a single
{name, arguments} so the parser silently dropped every call after
the first. apply.ts only invoked one tool, M3 min_roles couldn't
pass on legitimately-routed multi-tool runs, and byTool stats
under-counted. The composite routing 'multi-tool' bucket was
correctly assigned in classifyRouting, but downstream the pipeline
behaved as if the model emitted a single call.
This commit replaces `kind: 'tool_call'` with
`kind: 'tool_calls'` (NON-EMPTY list) across every consumer:
- types.ts: ParsedOutput tagged union; new ParsedOpToolCall.
ScoreRow.toolName → toolNames: string[].
- output-parser.ts: collects ALL element-tool tags in emit order;
unknown-tool path also surfaces as single-element tool_calls so
routing keeps the same wrong-tool semantics.
- score-run.ts: classifyRouting uses Array.includes for obvious
prompts (right-tool when ANY emitted call matches expected_tool —
over-production isn't a routing miss). Composite stays multi-tool
on any non-empty list.
- aggregate.ts byTool: tallies EVERY name in toolNames, so a
composite row that emits 6× add_activity_log_v0 + 1×
add_section_header_v0 contributes 6+1 = 7 invocations across two
tools (with row-level m1_legal applied to both buckets — apply is
all-or-nothing).
- apply.ts: loops over parsed.calls and invokes
handleElementToolCall in emit order. Any single call failing
aborts the row (M1=false); we don't partial-apply.
- mock-llm.ts mockLlmParsed: collects all `<op_tool>` tags into the
list (composite-prompt mocks can carry multi-call raw strings).
- apps/web design-parser.tryParseElementToolOutput: maps tool_calls
→ its single-shape DesignOutputShape contract using the FIRST
call (the multi-tag path `tryParseAllElementToolOutputs` was
already correct).
Tests: 3746 → 3750 vitest. New cases:
- output-parser: surfaces ALL element-tool tags in emit order with
intermixed batch_design scaffolds dropped (3 element calls from
5 tags).
- score-run: right-tool when expected appears alongside extras;
composite multi-call captures every name in toolNames.
- aggregate: 6× activity_log + 1× section_header → byTool reports
6 and 1 invocations respectively.
dry-run on ab-v3 produces a 208-row report; tsc + format clean.
Adds 7 new obvious prompts covering the v0.8.0 element tools that
weren't in ab-v1 (tools 91-97):
setting_row / member_row / filter_group / invite_row /
activity_log / event_card / step_card
One yaml per tool, same single-component "Design ONLY..." pattern
as ab-v1, with must_contain_roles mirroring the role names emitted
by the corresponding builder in pen-core/src/element-builders/.
Adds 5 composite prompts that exercise the new M6 routing
breakdown:
- dashboard-settings-page-composite (4× setting_row)
- dashboard-team-people-page-composite (5× member_row + 1×
invite_row)
- dashboard-search-filters-composite (2× filter_group + result
list)
- dashboard-audit-feed-composite (6× activity_log)
- mobile-onboarding-flow-composite (4× step_card)
Each composite prompt omits expected_tool_if_any (multi-tool
intent) and uses min_roles to enforce the multi-element shape.
corpus-loader: composite added to VALID_DIFFICULTIES; composite
prompts MUST NOT specify expected_tool_if_any (validation error
points the user back to difficulty=obvious if a single tool fits).
6 new corpus-loader tests cover the v3 yaml inventory + composite
validation rules. 3740 → 3746 vitest tests, all green.
ab-v3 corpus now has 52 prompts: 40 inherited from v1 (unchanged
for v1↔v3 comparability) + 7 obvious + 5 composite.
ab-v3 succeeds ab-v1 (frozen 2026-04-28). Carries forward all 40
v1 obvious yaml files unchanged so the v1↔v3 overlap stays
comparable, then layers in two new dimensions.
**1. Token cost.** All clients (openai-compat, ark, bailian,
deepseek, minimax, codex-cli, stub-model) now return a
`ChatCallResult { content, usage }` instead of bare string.
Provider usage stats (`prompt_tokens` / `completion_tokens`) plumb
through realModelCall → run.ts → scoreRun → ScoreRow.{prompt,completion}Tokens.
aggregate adds avgPromptTokens{Baseline,Treatment} +
avgCompletionTokens{Baseline,Treatment} per ModelSummary.
write-report emits a new "Token cost" table with Δ columns so
narrow-tools-saves-tokens (the ab-v2 hypothesis) is measurable.
avgUsage skips rows with 0/0 usage so codex-cli (CLI doesn't
surface tokens) and harness errors don't deflate the average to
near-zero — they show '—' instead.
**2. Composite difficulty.** New 'composite' value alongside
obvious / optional. Composite prompts express multi-tool intents
where no single expected_tool_if_any applies. classifyRouting
routes composite-treatment runs into multi-tool / fallback /
garbage (3-bucket sum to 1, distinct from obvious's 4-bucket
right/wrong/fallback/garbage). aggregate adds m6_multi_tool +
m6_fallback + m6_garbage; write-report emits a "Composite routing"
table that gracefully degrades to a placeholder when no composite
yaml exists yet.
Harness side: scripts/ab-corpus/run.ts accepts --corpus ab-v3
(enum + parseArgs guard); dry-run on the v1-mirror corpus produces
a 160-row report including populated token table.
Tests: 4 new aggregate cases (composite, token avg with skip-zero,
NaN-when-no-data) + 4 new score-run cases (composite routing
multi-tool/fallback/garbage/baseline-n/a) + 2 new score-run cases
(usage plumbing) + 2 new openai-compat cases (usage parsing,
missing-usage fallback). Existing 5 retry tests updated for new
return shape. 3727 → 3740 vitest tests, all green; tsc + format
clean.
Token-cost docs and composite docs go straight into types.ts /
score-run.ts / aggregate.ts JSDoc — keeps the contract close to
the code that owns it.
ab-v2 (2026-04-28) saw kimi-k2.6 garbage rate hit 17.5% — every
failure was Ark returning empty `choices[0].message.content` or
hitting the 120s wall clock, not a model-quality issue (the model
itself routed to the right element tool 80% of the time when it
did respond). Same pattern at 7.5% on deepseek-v4-pro.
Adds optional `retries` to `callOpenAICompat` with a transient-error
allowlist: empty content, abort/timeout, HTTP 5xx, HTTP 429. Linear
250ms × (attempt+1) backoff. HTTP 4xx other than 429 stays fatal so
auth/bad-request failures don't burn retry budget.
Wires `retries: 1` through clients/ark.ts and clients/deepseek.ts.
MiniMax + Bailian + Codex stay untouched — their ab-v2 failures
were model-quality (DSL escape errors, output truncation), where
retry wastes a call without changing the outcome.
Adds an 8-case fixture in scripts/ab-corpus/clients/__tests__/ that
mocks fetch to verify: first-try success, empty-then-success,
5xx-then-success, 429-then-success, 401 fatal, retries-default-zero,
retries exhausted, and retries=2 (3 attempts total). Extends the
apps/web vitest include glob to pick up scripts/**/__tests__/.
DeepSeek wasn't in the harness; A/B v2 needed it for the 5-model run.
api.deepseek.com is OpenAI-compatible, so the client mirrors the
minimax pattern (callOpenAICompat with DEEPSEEK_API_KEY env, override-
able DEEPSEEK_BASE_URL).
Router: /^deepseek/i routes to the new client. Aliases `deepseek` /
`deepseek-pro` resolve to `deepseek-v4-pro` (current flagship per
docs); `deepseek-v4-flash` and the deprecated `deepseek-chat` /
`deepseek-reasoner` ids pass through verbatim until the 2026-07-24
sunset upstream.
Verified live with mobile-bio-textarea smoke (B emits batch_design,
T emits add_textarea_v0). Full ab-v2 results across all 5 models in
docs/notes/2026-04-28-ab-v2-results.md.
shim-server-parity.test.ts grew to 838 lines after the recent batch of
element tools added 7 fixture entries. Move the CASES table and the
build* imports to shim-server-parity-cases.ts (a sibling .ts, not
.test.ts so vitest doesn't pick it up as a separate suite); keep the
mocks, helpers, and describe blocks in the test file (now 139 lines).
Both files are well under the repo's 800-line ceiling. vitest's
vi.mock hoisting still works because the mock is registered at the
test file's module-resolution time, before the cases file's imports
resolve — confirmed by the 104 parity assertions still passing.
The previous schema description and JSDoc invited "Step 1" as a valid
value for `number`, but that prose has 6 chars and overflows the 36px
circle marker. Two changes:
- Builder: add clipContent: true on the marker frame so any caller
who ignores the docs at least gets a clipped (not bleeding) render.
- Schema + JSDoc: drop the misleading "Step 1" example, document the
1–3 character contract, and steer prose toward `title` instead.
Closes the obvious gaps remaining in the family:
- add_filter_group_v0 — sidebar facet (heading + checkbox-style options
with optional counts). Distinct from nav_chip_row (horizontal scrolling
chips), tag (single applied chip), segmented_control (mutex tabs).
- add_invite_row_v0 — pending invite row (avatar + email/role + status
pill + trailing action). Distinct from member_row (a JOINED member,
no status pill or action) and list_row (no avatar / status / action).
- add_activity_log_v0 — single-line audit feed entry (optional tinted
icon dot + actor in bold + action + right-aligned timestamp). Uses
StyledTextSegment[] content for the bold/regular split. Distinct from
timeline (multi-event vertical with connectors) and notification_row
(title + body, no actor focus).
- add_event_card_v0 — single calendar event tile (date column with
month band + day number, then title + time + location). Distinct from
calendar_grid (the full month grid) and card_row (no date column).
- add_step_card_v0 — onboarding step card (numbered circle / check +
title + description). Distinct from stepper (horizontal progress nav
with connectors) and faq_item (collapsible Q&A header).
9 touchpoints per tool: pen-core builder + index + barrel + types,
pen-mcp handler + dispatcher + ext-4 schema, apps/web shim +
SERVER_BUILDERS, parity test (+5 cases), elements.md decision tree
items 86-89 + 6 PREFER mappings with cross-links to existing tools,
elements-cookbook.md arg-shape examples (8 entries across 5 tools).
Two bugs in elements.md after add_setting_row_v0 landed:
1. PREFER mapping still routed "settings row" to add_list_row_v0 — direct
contradiction with the new add_setting_row_v0 entry below it.
2. The "Settings page" recipe called add_list_row_v0 with a trailing_kind:
"switch" arg, but list-row has no such param; it would silently render
without a switch (or fail validation in stricter clients).
Reroute settings-row prose mapping to the new tool and rewrite the recipe
to use add_setting_row_v0 with proper trailing variants.
Leading icon + (title over optional subtitle) + trailing control with
4 variants: chevron / value text / switch / badge. Distinct from
add_list_row_v0 (trailing is always icon, no switch/value/badge) and
add_form_field_v0 (label-above-input for forms).
Wires all 9 touchpoints: pen-core builder + index + barrel re-export,
pen-mcp handler + dispatcher case + ext-4 schema, apps/web shim +
Nitro SERVER_BUILDERS, elements.md decision tree #83 + PREFER mapping,
elements-cookbook arg-shape examples, plus shim-server parity case.
The tool-level description still listed 11 specific sections (schema /
layout / roles / text / style / icons / examples / guidelines /
planning / elements / design-md) even though the actual catalog has
28. Replace with a pointer at inputSchema.section.enum, which is
already derived from listPromptSections() — the description now
can't go stale as sections are added or removed.
D0 parity snapshot refreshed for the new description text.
The published section enum had drifted to 14 entries while
SECTION_MAP grew to 28 (copywriting / overflow / cjk / variables +
8 codegen-* + elements-cookbook). External MCP clients calling with
the missing names hit schema-validation rejection even though the
implementation could serve them. Codex caught the immediate
elements-cookbook gap; widening the fix because the same pattern was
already silently broken for half the catalog.
- enum now derives from listPromptSections() at module load, no
hand-maintained list to drift
- description points at listPromptSections() rather than enumerating
individual sections (which was its own drift vector)
- design-prompt-elements adds a sync drift-guard: enum set must
exactly equal listPromptSections() set
- D0 parity snapshot refreshed for the new enum values
3712 tests green.
Trimming the Minimal usage block out of elements.md (65c31832) lost
arg-shape templates that the A/B harness depends on. Text-only LLMs
in the treatment arm see only the markdown skill content — no MCP
tools/list, no published inputSchema — so without the per-tool
example payloads they have to guess argument names and break M1.
Restore the full block as a sibling skill `elements-cookbook` (same
hasMcpTools flag, slightly later priority so it loads alongside
elements). Wire it through buildFullPrompt + the
get_design_prompt(section='elements-cookbook') section map. Update
the A/B harness to also strip the cookbook body when building the
baseline prompt — leaving it in B would leak tool names + arg shapes
back into the no-tools variant and re-bias the comparison.
Both files now under the 800-line per-file ceiling.
elements.md grew to 859 lines after the 81-90 batch shipped — past the
repo's per-file ceiling. The 366-line "Minimal usage" section was the
biggest contributor and the most redundant: MCP `tools/list` already
publishes the full inputSchema for every element tool (arg names,
types, descriptions, requireds), so the LLM has authoritative arg
shape from the wire. The decision tree + PREFER mappings already
teach WHEN to pick each tool. Drop the inline usage examples; keep
the composition pattern + cookbook recipes (which schemas can't
convey) plus invariants and failure-mode guidance.
493 lines remaining; design-prompt-elements + every drift guard still
green.
Adds the desktop-leaning batch needed to round the family to 90:
- add_user_card_v0 — compact avatar+name+role row
- add_drawer_shell_v0 — full-height side panel header
- add_combobox_v0 — open-state autocomplete with dropdown
- add_toolbar_v0 — desktop icon button row + dividers
- add_callout_v0 — inline doc tip block, 5 tones
- add_share_row_v0 — circular social-share buttons
- add_inline_action_v0 — message + Undo-style action
- add_legend_item_v0 — chart legend marker+label+value
- add_inbox_message_v0 — email/inbox row with unread dot
- add_profile_header_v0 — large profile hero block
All ten go through the standard 9-touchpoint wiring and land in a
new ext-4 schema shard so existing shards stay under 800 lines.
Drift guards (contract / registry parity / shim-server parity) cover
each new name; per-tool handler tests are deferred — every tool's
structure is exercised through the parity build call already.
Long cell content rode `width=auto` and the cell itself had
`height=fit_content` with no clip, so a string longer than its
allotted column would push past the cell edge into the adjacent
column at render time. Switch each cell to `width/height=
fill_container + clipContent=true`, give the text child
`width=fill_container + textGrowth=fixed-width` so the layout
engine wraps inside the cell, and clip the row itself so a runaway
cell can't push siblings off-row either. Lock the contract in the
handler test.
