elements.md still carried two stale claims from the P6 spec era when I
incorrectly assumed the apps/web sub-agent always emitted JSONL:
1. Frontmatter comment (line 17): "embedded orchestrator in apps/web
emits single-shot JSON and cannot call MCP tools — this skill would
be 1500 tokens of dead weight there, so it stays excluded."
Wrong now. With VITE_ENABLE_ELEMENT_TOOLS=1 the embedded orchestrator
sets `hasMcpTools` and the sub-agent DOES emit `<op_tool>` blocks
parsed by tryParseAllElementToolOutputs and dispatched via
element-tools-dispatcher.ts. The skill loads in BOTH paths.
2. Theme handling section opener: "This section applies to the MCP
tool-call path only … the web-app sub-agent JSONL path forbids tool
calls — there, write $color-* / $type-* refs directly in JSONL".
Wrong now. With element tools enabled both paths use `<op_tool>` and
the same `theme: 'system'` advice applies uniformly. The pointer to
"DESIGN SYSTEM TOKENS in jsonl-format.md" is dead — that skill was
just dropped from the element-tool path.
Removed the stale comment, rewrote the comment positively to describe
the dual-path loading. Removed the misleading sub-section header so the
"Default to theme: 'system'" rule applies cleanly to every caller.
Two ranking bugs were silently sending mobile food/wellness/fintech briefs
to a desktop landing-page palette:
1. Substring tag inference. /red|red/ matched 'Featured', /health/ matched
'Healthy' (a category in the food brief), so a food prompt picked up a
spurious 'wellness' tag and a desktop wellness guide jumped above the
mobile food guide via tag-overlap math. Added \b word boundaries to
every English keyword in inferTagsFromPrompt; CJK rules unchanged
because \b doesn't apply.
2. Industry vs style tag weighting + platform mismatch penalty. Each
matched tag was worth +10 regardless of meaning, and a platform
mismatch was a tiny -3 vs +0. So a desktop ecommerce-modern guide
beating mobile warm-food on the same brief was just `clean+modern+
rounded` overlapping more than `warm-tones+friendly+rounded` while
the platform penalty was negligible.
Now: industry tags (warm-tones / wellness / fintech / developer /
monospace) score 30, generic style tags 10, platform mismatch -30.
Empirically pushes warm-food-mobile-light to the top of the food
brief shortlist (verified with the actual expanded prompt that the
user's MiniMax-M2.7 run logged).
Same fix applies to every brief that was getting "wrong palette" results
because the planner snippets only contain the top-4 ranked guides — if
the right answer falls past 4, the planner literally never sees it and
the model invents its own (default-blue) palette.
Also: jsonl-format-simplified.md (basic-tier sub-agent prompt) now mirrors
jsonl-format.md's design-system-tokens teaching — basic-tier models like
MiniMax-M2.7 currently emit 0% typography refs because the simplified
prompt doesn't mention $type-* refs at all. The expanded simplified
prompt is 3981 chars, well under the bumped budget=1700 (=6800 char cap).
CRITICAL contract moved to top-of-file as the same defense-in-depth
pattern applied earlier to jsonl-format.md.
Single ClipInfo can't faithfully encode `(rrect ∩ rrect)` whenever one rect
cuts inside the other's corner. The previous fix collapsed nested clips
into one ClipInfo and dropped one side's rounded corner — which meant a
rounded modal containing rounded cards would silently lose either the
modal's rounding or the card's rounding at paint time.
Fix: replace the single `RenderNode.clipRect: ClipInfo | undefined` with
`clipStack: ClipInfo[]`. Flatten time accumulates a stack from outer-most
ancestor down to the immediate clip-introducing parent. Paint time pushes
each entry as its own canvas.save+clipRect/clipRRect — Skia's clip stack
intersects them naturally, so each level's rounded corner is enforced
independently.
Touched:
- types.ts: export ClipInfo, replace clipRect with clipStack
- document-flattener.ts: thread `clipStack: ClipInfo[]` through recursion;
push to a copy when isRootFrame || explicitClip
- node-renderer.ts paint: loop over clipStack, push N save+clip ops, pop
the same N at the end
- renderer.ts (root frame label loop) + skia-engine.ts (root frame label
loop) + focus-fit.ts (auto-fit excludes clipped descendants) +
global-export.ts (page bounds): all check clipStack.length instead of
truthy single field
- skia-interaction.ts: drag/resize/rotate snapshots store and restore
clipStack arrays (deep-cloned per entry)
- Tests updated + 1 new test: rounded modal containing rounded card
preserves both rrects on the inner content's clip stack
Previously `flattenToRenderNodes` overrode the inherited `clipCtx` whenever
a frame had `clipContent: true` (e.g. a card masking its rounded image),
which let the card's children paint past any outer clip — including the
root frame's artboard clip and any horizontal scrolling row's clip.
Repro: a horizontal `clipContent: true` row containing 3 rounded cards
whose total width exceeds the row's visible width. The 3rd card's children
(thumbnail, name text, etc) painted all the way out to the card's own
right edge — past the row, past the root frame, onto the canvas
background.
Fix: introduce `intersectClip(inner, outer)` and use it whenever a nested
clipContent is enabled. `inner` is the new clip we want to introduce (the
current frame's own bounds + cornerRadius); `outer` is the inherited clip
from the ancestor chain. We intersect the rectangles and drop the rounded
corner only if the inner was actually cut on either axis (a single
ClipInfo can't faithfully encode a rrect ∩ rect when the rect cuts inside
a corner).
`outer.rx` is intentionally not propagated — the ambient canvas clip
stack at paint time already enforces the outer rounded shape, so each new
clip just needs to refine the rectangular extent.
Test added: overflowing horizontal scroll row with 3 rounded cards.
Pre-fix: 3rd card's inner text gets clipRect={x:324, w:150}, escaping
the row clip. Post-fix: clipRect={x:324, w:76, rx:0}, properly clipped.
Two-part fix for the web-app chat path (both built-in and CLI mode go through
sub-agent JSONL output, not MCP tool calls — `jsonl-format.md` line 50 forbids
tool calls). Without this, every fill in generated designs was a hex literal
even after the 5/3 design-system-aware work — the 188 v1 element tools and
DEFAULT_PALETTE_FALLBACK were dead weight here.
A. STYLE GUIDE injection now uses ref + (hex) double form so the model sees
`$color-accent` paired with the resolved hex it represents:
Before: - Background: #FFF8F0 Surface: #FFFFFF
After: - Background: `$color-bg-deep` (resolves to #FFF8F0)
- Surface: `$color-surface` (#FFFFFF)
Applied to both `buildSubAgentStyleGuideInstruction` (selectedStyleGuideContent
path) and the inline `plan.styleGuide` injection in orchestrator-sub-agent.ts.
B. `seedDocVariablesFromStyleGuide` runs once before sub-agent execution: when
`doc.variables` is empty AND a style guide is selected, it maps the palette
to v1 token names (`color-bg-deep` / `color-accent` / etc) and seeds them
into `doc.variables`. This makes refs emitted by the model resolve to the
user's chosen palette at render time instead of falling back to the default
#2563EB blue.
A and B are coupled — A alone would make designs render in the wrong color
(every design becomes blue regardless of style guide); B alone leaves the model
mimicking the hex from the prompt. Both must ship together.
Also:
- jsonl-format.md: DESIGN SYSTEM TOKENS section + example fills converted to
refs + CRITICAL contract moved to top (so future budget overruns can't
truncate it). Budget bumped 1500 → 1700 for safety margin.
- elements.md: Theme handling section now flags MCP path vs JSONL path so
models on either path know which guidance applies.
GAP-1: add 'fontWeight' to the text-node key list in resolveNodeForCanvas so
$type-*-weight refs resolve to a number before reaching the renderer.
GAP-2: replace the early-exit `if (!variables || Object.keys(variables).length === 0)
return node` with `if (!variables) variables = {}` so DEFAULT_PALETTE_FALLBACK
fires even when the document has an empty variables map (un-seeded v1 docs).
Adds 2 new tests to fallback-equivalence.test.ts covering both gaps.
Switch all 89 element tool recommendations in elements.md from v0 to v1
with theme: 'system' as the default. Add "Theme handling" section explaining
when to use system/light/dark/v0. v0 entries retained as "Byte-frozen escape
hatch" sub-entries for rare byte-parity requirements.
Adds theme-aware v1 builders for icon_button, image_placeholder,
inbox_message, inline_action, input_with_action, invite_row, kbd,
legend_item, link, and list_row. Group A (zero-color: icon_button,
link, list_row) — no hardcoded colors in v0, all three modes identical.
Group B (kbd) — key bg → surface2, stroke → border in dark/system.
Group C (remaining 6) — surface/text/border/accent/alertColors tokens
applied in dark/system modes, full byte-parity with v0 in light mode.
Extends ext-6 shard (357→647 lines, within 800-line ceiling) housing
all 20 batch-4 + batch-5 tool schema definitions. All 9 touchpoints
wired per playbook: builder, index.ts, pen-core barrel, handler,
dispatcher, ext-6 shard, client shim, server builder, elements.md entries.
Verified: format:check clean, tsc --noEmit clean, 4086/4086 tests pass.
Adds theme-aware v1 builders for cookie_banner, data_table_row,
date_picker, drawer_shell, empty_state, event_card, fab, faq_item,
filter_group, and form_field. Group A (zero-color: empty_state,
form_field) — no hardcoded colors in v0, all three modes identical.
Group B (fab) — accent bg is brand-invariant, maps to accent token
in dark/system; icon stays white in all modes. Group C (remaining 7)
— surface/text/border/accent tokens applied in dark/system modes, full
byte-parity with v0 in light mode.
Creates ext-6 shard (ext-5 was at 798-line ceiling) housing all 10
new tool schema definitions (357 lines). All 9 touchpoints wired per
playbook: builder, index.ts, pen-core barrel, handler, dispatcher,
ext-6 shard, client shim, server builder, elements.md entries.
Verified: format:check clean, tsc --noEmit clean, 4076/4076 tests pass.
Adds theme-aware v1 builders for chart_bars, chart_line, chart_pie,
chat_bubble, checkbox, chip_input, code_block, color_swatch, combobox,
and comment. Group A (chart tools) maps bar/line color to chart-1 token
and pie default palette to chart-1..6 tokens in dark/system modes.
Group B (color_swatch) is theme-invariant — swatch color is caller-
supplied and passes through unchanged. Group C (chat_bubble, checkbox,
chip_input, code_block, combobox, comment) resolves surface/text/border
via semantic palette tokens. All light modes are byte-parity with v0.
Adds theme-aware v1 builders for alert, bottom_nav, breadcrumb,
activity_ring, carousel_dots, action_menu, attachment_row,
calendar_grid, avatar_group, and callout. Group A (zero-color:
alert/bottom_nav/breadcrumb/activity_ring) produce identical output
across all three theme modes. Group B (carousel_dots) maps active=
text-primary, inactive=border in dark/system modes. Group C (action_menu/
attachment_row/calendar_grid/avatar_group) resolve surface/text/border via
semantic palette. Group D (callout) maps tone-keyed bg/fg to alert palette
tokens in dark/system modes. All light modes are byte-parity with v0.
avatar-v1, badge-v1, divider-v1, body_text-v1, icon_label-v1 — each with
full 9-touchpoint coverage (pen-core builder + index + pen-mcp handler +
schema shard + dispatcher + apps/web shim + SERVER_BUILDERS + parity test
+ elements.md). Light mode is byte-equal to v0; dark/system modes produce
identical output since all 5 tools emit zero hardcoded color fills — theme
param accepted for API consistency across all v1 tools. New shard
element-tool-defs-ext-5.ts created (ext-4 was at 739 lines). All 2026
pen-core + pen-mcp tests pass; format:check + tsc clean.
card_row-v1, setting_row-v1, member_row-v1, activity_log-v1 — each with
full 9-touchpoint coverage (pen-core builder + index + pen-mcp handler +
schema shard + dispatcher + apps/web shim + SERVER_BUILDERS + parity test
+ elements.md). Light mode is byte-equal to v0; dark/system use resolveTheme()
for all color fills. activity_log-v1 maps tone×theme to alertColors tokens
(info/success/warning/danger) with neutral falling back to surface/textMuted.
All 3998 tests pass. Completes P2 representative phase.
Task 2.3 — representative v1 tool walkthrough for Plan 14 byte-parity contract.
Light mode is byte-equal to add_heading_v0 (V0_LATIN_PRESETS table reused);
dark/system modes use resolveTheme() for fill color and typography token refs.
Adds theme enum [light, dark, system] to add_heading_v1 MCP schema, elements.md
decision tree, shim-server-parity CASES, and SERVER_BUILDERS. All 3937 tests pass.
P1.5 consumer compatibility gate before P2 element-tool v1 builders.
resolveNodeForCanvas was already resolving gap/padding/opacity/color refs,
but missed text typography fields (fontSize, lineHeight, letterSpacing) and
cornerRadius. Since both skia-engine and pen-renderer consume resolver output
before layout runs, unresolved string refs would pass arithmetic as NaN.
layout/engine.ts also patched to guard typeof === 'number' at the 4 text
font-size/line-height access points — defensive fallback to 16/1.x even when
called on pre-resolution nodes (e.g. MCP server, normalizer pipeline).
Adds 21-test p1-5-consumer-compat.test.ts covering all 6 consumer paths.
All 1183 pen-core tests pass. format:check and tsc --noEmit clean.
Introduce DEFAULT_PALETTE_FALLBACK (56-entry map built from semantic-palette
at module init) and wire it into resolveVariableRef so that v1 'system' mode
on an un-seeded doc resolves to canonical hex/numeric defaults rather than
undefined — fulfilling the equivalence guarantee from spec §5.3.
Update modal-shell-v1 test to reflect the new contract: un-seeded doc +
Mode:Dark now yields #1E293B (color-surface dark) instead of undefined.
The 4 extra single-value tokens (color-accent-dark, color-info-surface,
color-warning-text-strong, color-danger-text-strong) introduced in P1.1.6
violated spec §3.1 / §7.4 — those hex were INTENDED to merge into existing
tokens with ≤ 5% accepted color drift, not become new tokens.
Replaced with MERGE_MAP in measure-v0-hex-coverage.ts that tracks the 4
near-shade redirections (#1D4ED8→color-accent, #EFF6FF→color-info-bg,
#B45309→color-warning-text, #B91C1C→color-danger-text). Cover rate
calculation now reports direct + merge breakdown.
Final palette token count: 56 (28 color + 18 type + 2 letterSpacing +
5 spacing + 3 radius). Cover rate: 28 direct + 4 merge = 32/32 = 100.0%.
spacing-1..5 = 4/8/12/16/24 px — a 4-point base scale covering xs through xl.
All tokens use type="number" with plain scalar values. Palette total grows to 57.
type-display-letter-spacing (-0.5 px) for large display text, and
type-uppercase-label-letter-spacing (1.5 px) for uppercase / overline labels.
Palette total grows to 52.
size + weight + line-height for 6 roles: display / h1 / h2 / h3 / body / caption.
All 18 tokens use type="number" with a plain scalar value (no theme axis).
getSemanticPaletteHex() omits numeric tokens from its return map.
Palette grows from 32 color → 50 total variables.
8 light/dark alert pairs (info/success/warning/danger bg+text) and 6
single-value chart series colors added to PALETTE. PaletteEntry union
type introduced to support LightDarkEntry | SingleColorEntry | SingleNumberEntry.
getSemanticPalette(), getSemanticPaletteHex(), applySemanticPalette()
updated to handle all three entry shapes. Palette grows from 14 → 28 tokens.
Two diet/teaching changes to elements.md, both motivated by today's
ab-v4 smoke results:
(a) parent_id teaching — minimax-m2.7 invented "members-section" /
"canvas" as parent_id values on its multi-tool composite output,
then every one of its 14 tags failed apply with "parent_id X not
found in document". Same pattern as yesterday. Adds an explicit
rule near the top of elements.md (right under the multi-tool
banner) — `parent_id` is REAL or OMITTED, never invented. The
`<page>` / `<panel>` / `<sidebar>` placeholders in the cookbook
recipes are documentation conventions; in actual output, OMIT
the field. Names the failure mode by reproducing the error
message format so models learn to avoid it.
(b) Phase 3 token diet — strip ". Different from <tool> (...)"
disambiguation suffixes from the PREFER list (22 entries had
them, ~80-200 chars each = ~2.5kb / ~625 tokens saved). The
primary keyword + tool-name + capability description survives
intact; the cross-references pointing at sibling tools get
dropped. Risk: slight increase in wrong-tool routing on
ambiguous prompts. Worth it for the size reduction; the
decision tree alone still shows the tool family in context.
Verified by mechanical diff — perl in-place edit, then visual
review confirms no other content was touched. 3785 vitest pass,
format clean, tsc silent.
Net effect on T-prompt size (chars / 4 estimate):
T + composite + mobile: was 17,930 → now 17,475 (-455 chars)
T + composite + dashboard: was 18,437 → now 17,983 (-454 chars)
T + obvious + dashboard: ~14,009 → ~13,556 (-453 chars)
Per-arm savings are smaller than the raw 2.5kb trim because (a)
adds ~700 chars of parent_id teaching. Net win: ~450 chars / ~110
tokens per call. Modest but compounds across 520+ ab-v4 runs.
Sweep follow-up to 113bd55a — same defensive pattern (reject
unknown enum strings at the entry boundary) applied to every other
builder that indexed a Record<EnumLiteral, T> with a value sourced
from raw JSON args.
Builders + enums covered:
- buildTag — TagTone (default | accent | success | warning | error)
- buildCallout — CalloutTone (info | success | warning | danger | note)
- buildActivityLog — tone (info | success | warning | danger | neutral)
- buildInviteRow — InviteStatus (pending | expired | accepted)
- buildMemberRow — trailing.tone for status_dot (online | busy | away
| offline). role_badge / menu variants skip the check (no tone field)
Same failure mode each one fixed: when a model invents an
out-of-enum string (gpt-5.4 did this with `level: "caption"` in
ab-v4), the lookup `TONES[bad]` / `STATUS_TONE[bad]` returned
undefined, the next property access crashed mid-batch with a
cryptic `undefined is not an object`, and the surrounding dispatch
loop dropped every remaining tag (until df33e937 + 07639f6d landed
the per-shape continuation + partial-doc scoring earlier today).
With validation in place, a bad enum becomes a clean per-shape
error message + the rest of the batch still applies.
13 new edge-case tests cover throw on bad input + valid path on
every enum value + omitted-default for each builder. 3785 vitest
pass, format clean, tsc silent.
Builders not touched: heading.ts (already done in 113bd55a).
Builders that don't fit this pattern (no enum→Record lookup of a
user-controlled string): everything else surveyed via grep on
`Record<.*Tone|Status|Level|Mode|Kind`.
Codex stop-time review caught the previous fix (df33e937) handing
the scorer a partial PenDocument that the scorer immediately
ignored. score-run.ts:87 short-circuited on `!applied.ok || !applied.doc`,
so even though apply.ts now surfaces 12 of 13 successfully-applied
tags as a populated `applied.doc`, the row still scored as a total
failure (M1=false, M3=false, m3_failure_reason="apply failed before
shape checks") — exactly the noise df33e937 was meant to eliminate.
Loosens the short-circuit to `!applied.doc` only. When apply.ok=false
but apply.doc is populated, the scorer now:
- runs the issue detector against the partial doc (issues surface)
- keeps M1 strict (apply.ok=false → M1=false regardless of detector)
- decouples M3 from M1: M3 = shape.ok against the partial doc
- sets m3_failure_reason to the shape miss when shape fails;
otherwise to "partial apply (M3 met by what landed): <error>" so
the row reads "tag 12 of 13 broke, but role coverage still met"
instead of silently swallowing the partial signal
- surfaces applied.error in row.applyError so per-shape failure
messages flow into reports
Two new tests cover the new path:
1. partial apply + shape match → M1=false, M3=true, reason mentions
"partial apply"
2. partial apply + shape miss → M1=false, M3=false, shape-miss
reason wins (structural verdict trumps the partial-apply notice)
Plumbing chain across today's session is now consistent:
- apply.ts continues past per-shape failures (df33e937)
- score-run.ts scores the partial doc that lands (this commit)
- scoring no longer over-attributes to "apply failed" when the model
actually produced most of the brief
3774 vitest pass (+2), format clean, tsc silent.
ab-v4 partial sweep (2026-05-01) caught gpt-5.4 emitting
`add_heading_v0({"content":"Pending invitations","level":"caption"})`
as the 12th tag of a 13-tag composite multi-tool response. Even
though the MCP tool def has `enum: ['display','h1','h2','h3']`, the
ab-corpus harness and the in-process production dispatcher both call
buildHeading() with raw JSON args (no jsonschema gate), so the model's
invented "caption" reached the preset lookup. LATIN_PRESETS["caption"]
is undefined, and the next line `fontSize: preset.fontSize` crashed
the WHOLE batch with `undefined is not an object (evaluating
'preset.fontSize')` — the 11 valid tags ahead of it never landed.
Adds an entry-point validation in buildHeading: if `level` is set and
not in the {display, h1, h2, h3} set, throw with a clear message.
The dispatch loops in apps/web/element-tools-dispatcher and
scripts/ab-corpus/apply both catch per-shape and keep running the
remaining tags, so a single bad level on tag 12 no longer kills tags
1-11 + 13.
3 new edge-case tests in element-builders-edge-cases.test.ts
cover the throw + the four valid levels + the omitted-default case.
3772 vitest pass (+3), format clean, tsc silent.
Other element builders likely have the same pattern (preset lookup
on a string enum without runtime validation) — separate sweep, not
shotgunning here.
ab-v3 / ab-v4 showed Phase 1A (cookbook strip on obvious difficulty)
shaved ~4.4k tokens off T-obvious. Phase 2 adds a per-category gate
that strips cookbook recipes whose domain doesn't match the prompt's
category — mobile briefs don't see dashboard recipes, dashboard
briefs don't see mobile / landing recipes, etc.
Mechanism: HTML comment block markers in elements.md
(`<!-- @domain:dashboard --> ... <!-- /@domain -->`) plus a
stripNonMatchingDomains() pass in buildSystemPrompt that drops blocks
whose tag list doesn't include the active category. Untagged content
is "general" and stays in every variant — the safe default.
Tagged 7 single-domain cookbook recipes:
- dashboard: Team / members list, Audit / activity feed, Faceted
search filter sidebar, Dashboard KPI strip
- landing: Pricing section
- mobile: Onboarding "How it works", Support chat thread
Cross-domain recipes (Login, Signup, Settings page, OTP, Empty
inbox) stay untagged so they load for every category. Decision tree
+ PREFER list also untagged today; the per-tool annotations there
would be a much larger judgment pass for marginal additional savings.
Token measurements (chars / 4 estimate):
full mobile dashboard landing
- T + composite 19.0k 17.9k 18.4k 17.7k
(-1.1k) (-0.6k) (-1.3k)
- T + obvious 14.6k 13.5k 14.0k 13.3k
(-1.1k) (-0.6k) (-1.3k)
Modest absolute savings — Phase 2 only filters cookbook RECIPES (in
elements.md), and most cookbook content is in elements-cookbook.md
which Phase 1A already strips on obvious. To hit the 6-8k T target
we still need decision-tree compression or PREFER-list trim, but
both are lossier than this gate. Phase 3 candidates noted in the
ab-v4 results doc.
real-model.ts plumbs call.prompt.category through to buildSystemPrompt.
3766 vitest pass (+6 category filter tests including a 500-char
floor regression guard that the filter actually shaves bytes).
Codex stop-time review (3rd round) caught residual "T prompt
includes mixed-strategy instructions": even after the prompt
forbade mixing, elements.md still taught the mixed pattern in
several places that are also part of the T system prompt.
Cleaned out every spot that paired batch_design with add_*_v0:
- Login screen / Pricing section / Dashboard KPI strip recipes:
dropped the leading `batch_design: foo = I("page", {...})` line
and renamed `<foo>` / `<row>` placeholders to `<page>` so each
recipe is now Strategy A (element tools only). Lost: explicit
page-level layout/padding/horizontal row — acceptable, the
recipes still teach the tool selection + chain pattern.
- Intro paragraph: dropped "override via a follow-up batch_design
U-op if needed" (taught a per-component fallback that the parser
drops).
- Banner: rewrote the fall-back clause from "when no element tool
fits a specific component shape" to "when at least one component
truly needs a custom shape no element tool covers — and then use
a SINGLE batch_design for the WHOLE response, never mixed."
- "STILL use batch_design when" list: collapsed 3 mixed-strategy
bullets ("larger composite via batch_design then element tool",
"post-hoc styling via batch_design U-ops") into 3 clean
Strategy-B-only bullets, all explicitly emit a SINGLE batch_design.
- Removed the "## Composition pattern" section entirely — its 3-step
plan was a textbook mixed pattern (batch_design root → element
tool inserts → batch_design U-op styling) that depended on real
MCP multi-round semantics the corpus harness can't provide.
- Composition rules of thumb: replaced "Don't mix N-tool and
batch_design DSL ops in a single call" + "Style overrides come
AFTER structure" with a single "Don't mix in the same output"
rule that names the corpus parser behavior explicitly.
3760 vitest pass, format clean, tsc silent. T-obvious 14.5k tokens
(down ~100 from the cleanup); composite still 18.9k.
Codex stop-time review flagged the new ab-v3 composite cookbook
recipes calling add_section_header_v0 with a `subtitle` arg the tool
doesn't accept (silently dropped at runtime today, but teaches live
models to emit invalid shapes). The same bug was in the dry-run stub
fixtures.
Split each header into add_section_header_v0(title) +
add_body_text_v0(content) — semantically what the ab-v3 briefs ask
for, and reinforces the multi-tool chaining the cookbook now teaches.
Also fills in the missing required `number` arg on the onboarding
recipe's final completed step card (schema requires it even when
completed=true renders a check instead of the number).
ab-v3 live sweep showed 0/25 composite-T runs walked the multi-tool
path — every model fell back to batch_design or garbage when given a
brief like "5 member rows + 1 invite row" or "6 audit log entries".
Root cause: elements.md's decision tree opens with "pick first match"
(single-tool framing) and the cookbook's chained examples sit ~400
lines deep, so models stop at the lookup and never realize they can
emit N <op_tool> blocks.
This adds:
- top-of-file "MULTI-TOOL OUTPUT IS THE NORM" banner with a concrete
5-call settings example, before the decision tree
- decision tree heading rewritten to "(per component — pick first
match)" so per-component framing is in scope from the start
- 4 new cookbook recipes mirroring the ab-v3 composite prompts:
team / members list (rows + invite), audit / activity feed,
faceted search filter sidebar, onboarding step cards
Markdown-only edit to a single skill file; vite-plugin-skills
re-compiles the registry. Banner verified present in the generated
registry, all 3750 vitest tests pass. Verifies in next ab-v4 sweep.
Codex stop-hook caught: ab-v3 introduced composite-difficulty prompts
that *expect* multi-tool emit (e.g. 5× member_row + 1× invite_row
for a team page), but `ParsedOutput.tool_call` was a single
{name, arguments} so the parser silently dropped every call after
the first. apply.ts only invoked one tool, M3 min_roles couldn't
pass on legitimately-routed multi-tool runs, and byTool stats
under-counted. The composite routing 'multi-tool' bucket was
correctly assigned in classifyRouting, but downstream the pipeline
behaved as if the model emitted a single call.
This commit replaces `kind: 'tool_call'` with
`kind: 'tool_calls'` (NON-EMPTY list) across every consumer:
- types.ts: ParsedOutput tagged union; new ParsedOpToolCall.
ScoreRow.toolName → toolNames: string[].
- output-parser.ts: collects ALL element-tool tags in emit order;
unknown-tool path also surfaces as single-element tool_calls so
routing keeps the same wrong-tool semantics.
- score-run.ts: classifyRouting uses Array.includes for obvious
prompts (right-tool when ANY emitted call matches expected_tool —
over-production isn't a routing miss). Composite stays multi-tool
on any non-empty list.
- aggregate.ts byTool: tallies EVERY name in toolNames, so a
composite row that emits 6× add_activity_log_v0 + 1×
add_section_header_v0 contributes 6+1 = 7 invocations across two
tools (with row-level m1_legal applied to both buckets — apply is
all-or-nothing).
- apply.ts: loops over parsed.calls and invokes
handleElementToolCall in emit order. Any single call failing
aborts the row (M1=false); we don't partial-apply.
- mock-llm.ts mockLlmParsed: collects all `<op_tool>` tags into the
list (composite-prompt mocks can carry multi-call raw strings).
- apps/web design-parser.tryParseElementToolOutput: maps tool_calls
→ its single-shape DesignOutputShape contract using the FIRST
call (the multi-tag path `tryParseAllElementToolOutputs` was
already correct).
Tests: 3746 → 3750 vitest. New cases:
- output-parser: surfaces ALL element-tool tags in emit order with
intermixed batch_design scaffolds dropped (3 element calls from
5 tags).
- score-run: right-tool when expected appears alongside extras;
composite multi-call captures every name in toolNames.
- aggregate: 6× activity_log + 1× section_header → byTool reports
6 and 1 invocations respectively.
dry-run on ab-v3 produces a 208-row report; tsc + format clean.
Adds 7 new obvious prompts covering the v0.8.0 element tools that
weren't in ab-v1 (tools 91-97):
setting_row / member_row / filter_group / invite_row /
activity_log / event_card / step_card
One yaml per tool, same single-component "Design ONLY..." pattern
as ab-v1, with must_contain_roles mirroring the role names emitted
by the corresponding builder in pen-core/src/element-builders/.
Adds 5 composite prompts that exercise the new M6 routing
breakdown:
- dashboard-settings-page-composite (4× setting_row)
- dashboard-team-people-page-composite (5× member_row + 1×
invite_row)
- dashboard-search-filters-composite (2× filter_group + result
list)
- dashboard-audit-feed-composite (6× activity_log)
- mobile-onboarding-flow-composite (4× step_card)
Each composite prompt omits expected_tool_if_any (multi-tool
intent) and uses min_roles to enforce the multi-element shape.
corpus-loader: composite added to VALID_DIFFICULTIES; composite
prompts MUST NOT specify expected_tool_if_any (validation error
points the user back to difficulty=obvious if a single tool fits).
6 new corpus-loader tests cover the v3 yaml inventory + composite
validation rules. 3740 → 3746 vitest tests, all green.
ab-v3 corpus now has 52 prompts: 40 inherited from v1 (unchanged
for v1↔v3 comparability) + 7 obvious + 5 composite.
ab-v3 succeeds ab-v1 (frozen 2026-04-28). Carries forward all 40
v1 obvious yaml files unchanged so the v1↔v3 overlap stays
comparable, then layers in two new dimensions.
**1. Token cost.** All clients (openai-compat, ark, bailian,
deepseek, minimax, codex-cli, stub-model) now return a
`ChatCallResult { content, usage }` instead of bare string.
Provider usage stats (`prompt_tokens` / `completion_tokens`) plumb
through realModelCall → run.ts → scoreRun → ScoreRow.{prompt,completion}Tokens.
aggregate adds avgPromptTokens{Baseline,Treatment} +
avgCompletionTokens{Baseline,Treatment} per ModelSummary.
write-report emits a new "Token cost" table with Δ columns so
narrow-tools-saves-tokens (the ab-v2 hypothesis) is measurable.
avgUsage skips rows with 0/0 usage so codex-cli (CLI doesn't
surface tokens) and harness errors don't deflate the average to
near-zero — they show '—' instead.
**2. Composite difficulty.** New 'composite' value alongside
obvious / optional. Composite prompts express multi-tool intents
where no single expected_tool_if_any applies. classifyRouting
routes composite-treatment runs into multi-tool / fallback /
garbage (3-bucket sum to 1, distinct from obvious's 4-bucket
right/wrong/fallback/garbage). aggregate adds m6_multi_tool +
m6_fallback + m6_garbage; write-report emits a "Composite routing"
table that gracefully degrades to a placeholder when no composite
yaml exists yet.
Harness side: scripts/ab-corpus/run.ts accepts --corpus ab-v3
(enum + parseArgs guard); dry-run on the v1-mirror corpus produces
a 160-row report including populated token table.
Tests: 4 new aggregate cases (composite, token avg with skip-zero,
NaN-when-no-data) + 4 new score-run cases (composite routing
multi-tool/fallback/garbage/baseline-n/a) + 2 new score-run cases
(usage plumbing) + 2 new openai-compat cases (usage parsing,
missing-usage fallback). Existing 5 retry tests updated for new
return shape. 3727 → 3740 vitest tests, all green; tsc + format
clean.
Token-cost docs and composite docs go straight into types.ts /
score-run.ts / aggregate.ts JSDoc — keeps the contract close to
the code that owns it.
The previous schema description and JSDoc invited "Step 1" as a valid
value for `number`, but that prose has 6 chars and overflows the 36px
circle marker. Two changes:
- Builder: add clipContent: true on the marker frame so any caller
who ignores the docs at least gets a clipped (not bleeding) render.
- Schema + JSDoc: drop the misleading "Step 1" example, document the
1–3 character contract, and steer prose toward `title` instead.
Closes the obvious gaps remaining in the family:
- add_filter_group_v0 — sidebar facet (heading + checkbox-style options
with optional counts). Distinct from nav_chip_row (horizontal scrolling
chips), tag (single applied chip), segmented_control (mutex tabs).
- add_invite_row_v0 — pending invite row (avatar + email/role + status
pill + trailing action). Distinct from member_row (a JOINED member,
no status pill or action) and list_row (no avatar / status / action).
- add_activity_log_v0 — single-line audit feed entry (optional tinted
icon dot + actor in bold + action + right-aligned timestamp). Uses
StyledTextSegment[] content for the bold/regular split. Distinct from
timeline (multi-event vertical with connectors) and notification_row
(title + body, no actor focus).
- add_event_card_v0 — single calendar event tile (date column with
month band + day number, then title + time + location). Distinct from
calendar_grid (the full month grid) and card_row (no date column).
- add_step_card_v0 — onboarding step card (numbered circle / check +
title + description). Distinct from stepper (horizontal progress nav
with connectors) and faq_item (collapsible Q&A header).
9 touchpoints per tool: pen-core builder + index + barrel + types,
pen-mcp handler + dispatcher + ext-4 schema, apps/web shim +
SERVER_BUILDERS, parity test (+5 cases), elements.md decision tree
items 86-89 + 6 PREFER mappings with cross-links to existing tools,
elements-cookbook.md arg-shape examples (8 entries across 5 tools).
