fix(ab-corpus): forbid mixing batch_design with element tools in T

The earlier T instruction split tool selection into PRIMARY / COMPOSITE
(element tools) and FALLBACK (batch_design). FALLBACK was scoped
"when no element tool fits a given component shape" — implying you
could mix per-component. But output-parser.ts silently drops every
batch_design tag whenever any element call is also present (filter
ELEMENT_TOOL_NAME_RE then return). A mixed response

  <op_tool>{"name":"add_section_header_v0",...}</op_tool>
  <op_tool>{"name":"batch_design", ...scaffolding...}</op_tool>

would lose the batch_design half and only run the element call —
the brief is half-applied without anyone noticing. Codex stop-time
review caught the mismatch.

Reframes T as a binary choice:
- STRATEGY A: every component fits an add_*_v0 tool — emit one tag
  per component
- STRATEGY B: at least one component needs batch_design — emit a
  SINGLE batch_design covering the whole brief

Plus an explicit "Do not mix Strategy A and Strategy B" guard line.
Tests assert both strategy markers are present in every T variant
and the dropped per-component-fallback phrase is gone.

The elements.md cookbook still teaches batch_design + element tool
composition for real MCP multi-round usage; the corpus T arm is
single-shot so the trailing T_TOOL_CALL_INSTRUCTIONS overrides via
"last instruction wins".
This commit is contained in:
Fini 2026-04-29 09:49:45 +08:00
parent b7e6097da5
commit f6746eb91c
2 changed files with 30 additions and 15 deletions

View file

@ -9,16 +9,21 @@ import { buildSystemPrompt } from '../build-prompt';
// the diet silently no-ops.
const T_MULTI_TOOL_MARKER = 'Respond with one or more `<op_tool>` tags';
const T_COMPOSITE_MARKER = 'COMPOSITE (multi-component briefs';
const T_STRATEGY_A_MARKER = 'STRATEGY A — element tools';
const T_STRATEGY_B_MARKER = 'STRATEGY B — batch_design fallback';
const T_NO_MIX_MARKER = 'Do not mix Strategy A and Strategy B';
const B_BATCH_MARKER =
'<op_tool>{"name": "batch_design", "arguments": {"operations": "<DSL_STRING>"}}</op_tool>';
// Earlier versions of the T instructions explicitly forbade multi-tool
// output ("Respond with one tag" / "Do not combine multiple tags");
// that wording undercut the multi-tool teaching baked into elements.md.
// Codex stop-time review caught the contradiction — guard against
// regression by asserting the forbidding text is GONE.
// A subsequent revision flipped to allow chaining but quietly invited
// mixed batch_design + element tags per-component, which the corpus
// parser silently drops. Codex stop-time review caught both. Guard
// against re-introducing either failure mode.
const FORBIDDEN_SINGLE_TAG_PHRASE = 'one `<op_tool>` tag, nothing else';
const FORBIDDEN_NO_COMBINE_PHRASE = 'Do not combine multiple tags';
const FORBIDDEN_PER_COMPONENT_FALLBACK_PHRASE = 'when no element tool fits a given component shape';
describe('buildSystemPrompt', () => {
it('T + composite returns the largest prompt (cookbook present)', () => {
@ -64,11 +69,12 @@ describe('buildSystemPrompt', () => {
expect(distinct.size).toBe(1);
});
it('every T variant carries the multi-tool output-format marker', () => {
it('every T variant carries the multi-tool + Strategy A/B output-format markers', () => {
for (const difficulty of ['obvious', 'optional', 'composite'] as const) {
const built = buildSystemPrompt('T', { difficulty });
expect(built.system).toContain(T_MULTI_TOOL_MARKER);
expect(built.system).toContain(T_COMPOSITE_MARKER);
expect(built.system).toContain(T_STRATEGY_A_MARKER);
expect(built.system).toContain(T_STRATEGY_B_MARKER);
}
});
@ -80,9 +86,21 @@ describe('buildSystemPrompt', () => {
}
});
it('B variant carries the batch_design marker but not the multi-tool split', () => {
it('T variant explicitly forbids mixing Strategy A and Strategy B (parser drops mixed)', () => {
for (const difficulty of ['obvious', 'optional', 'composite'] as const) {
const built = buildSystemPrompt('T', { difficulty });
expect(built.system).toContain(T_NO_MIX_MARKER);
// Earlier wording invited per-component fallback ("when no element
// tool fits a given component shape") which the parser silently
// drops when element calls are also present. Guard the rephrase.
expect(built.system).not.toContain(FORBIDDEN_PER_COMPONENT_FALLBACK_PHRASE);
}
});
it('B variant carries the batch_design marker but not the Strategy A/B split', () => {
const built = buildSystemPrompt('B');
expect(built.system).toContain(B_BATCH_MARKER);
expect(built.system).not.toContain(T_COMPOSITE_MARKER);
expect(built.system).not.toContain(T_STRATEGY_A_MARKER);
expect(built.system).not.toContain(T_STRATEGY_B_MARKER);
});
});

View file

@ -45,21 +45,18 @@ const T_TOOL_CALL_INSTRUCTIONS = [
'',
'OUTPUT FORMAT — EMIT AS TOOL CALL(S):',
'',
'Respond with one or more `<op_tool>` tags, nothing else. The harness reads every `<op_tool>` tag in your output, so chain as many as the brief implies. Choose tools based on intent:',
'Respond with one or more `<op_tool>` tags, nothing else. The harness reads every `<op_tool>` tag in your output, so chain as many as the brief implies. Pick ONE strategy for the whole response — do NOT mix element tools with batch_design in the same output:',
'',
'PRIMARY (single component): emit one tag for the add_*_v0 element tool that fits.',
' <op_tool>{"name": "add_X_v0", "arguments": {...}}</op_tool>',
'',
'COMPOSITE (multi-component briefs — settings panel with N rows, team list with N members, audit feed with N entries, onboarding screen with N steps): emit ONE `<op_tool>` per component under the same parent. Tags appear in render order (top-to-bottom for vertical, left-to-right for horizontal).',
'STRATEGY A — element tools (preferred): when every component in the brief fits an `add_*_v0` element tool, emit one tag per component, in render order (top-to-bottom for vertical layouts, left-to-right for horizontal). Single-component briefs produce exactly one tag; multi-component briefs (settings panel with N rows, team list with N members, audit feed with N entries, onboarding screen with N step cards) produce N+1 or more.',
' <op_tool>{"name": "add_section_header_v0", "arguments": {"title": "Notifications"}}</op_tool>',
' <op_tool>{"name": "add_setting_row_v0", "arguments": {...}}</op_tool>',
' <op_tool>{"name": "add_setting_row_v0", "arguments": {...}}</op_tool>',
'',
'FALLBACK: when no element tool fits a given component shape, emit a single batch_design call.',
'STRATEGY B — batch_design fallback: when the brief includes any component shape that NO element tool covers (heterogeneous custom layout, post-hoc styling, bespoke scaffolding), emit a SINGLE batch_design call covering the WHOLE response. Multiple `<op_tool>` tags carrying batch_design or any mix of batch_design + element tools is not supported — the harness drops the batch_design half and only the element calls run.',
' <op_tool>{"name": "batch_design", "arguments": {"operations": "<DSL_STRING>"}}</op_tool>',
'The `operations` value is a single string containing the batch_design DSL.',
'The `operations` value is a single string containing the batch_design DSL covering every component in the brief.',
'',
'Do not add prose before, after, or between tags.',
'Do not add prose before, after, or between tags. Do not mix Strategy A and Strategy B in the same output — choose element tools for every component or batch_design for every component.',
'',
].join('\n');