openpencil/scripts/ab-corpus
Fini f385f433e1 feat(ab-corpus): gate elements-cookbook on T arm by difficulty
ab-v3 left T prompts at ~22k tokens (vs ~6-7k for B). The 18kb
elements-cookbook teaches per-tool arg shapes, which is what
composite multi-tool chains genuinely need; single-tool obvious
prompts can route correctly from just the decision tree + PREFER
list alone.

buildSystemPrompt now takes opts.difficulty. T+obvious strips the
cookbook (saves ~18kb on the 47/52 ab-v3 obvious prompts);
T+composite, T+optional, and undefined keep both halves. B variant
unchanged — still strips both, so the A/B comparison stays clean.

Verified by 8 new build-prompt.test.ts cases including a 10kb floor
on the obvious-vs-composite delta. ab-v4 will measure whether the
diet hurts arg compliance on weak models; per-domain split is the
Phase 2 fallback if obvious-T garbage rate creeps up.
2026-04-29 09:49:42 +08:00
..
__tests__ feat(ab-corpus): gate elements-cookbook on T arm by difficulty 2026-04-29 09:49:42 +08:00
clients feat(ab-corpus): exponential backoff + bump retries=2 on ark/deepseek 2026-04-29 09:49:41 +08:00
apply.ts fix(ab-corpus): plumb multi-tool output end-to-end for composite 2026-04-29 08:35:00 +08:00
build-prompt.ts feat(ab-corpus): gate elements-cookbook on T arm by difficulty 2026-04-29 09:49:42 +08:00
real-model.ts feat(ab-corpus): gate elements-cookbook on T arm by difficulty 2026-04-29 09:49:42 +08:00
run.ts feat(ab-corpus): bootstrap ab-v3 with token cost + composite difficulty 2026-04-29 07:45:00 +08:00
stub-model.ts fix(ai-skills): drop invalid section_header subtitle from cookbook + stubs 2026-04-29 09:49:40 +08:00
write-report.ts feat(ab-corpus): bootstrap ab-v3 with token cost + composite difficulty 2026-04-29 07:45:00 +08:00