openpencil

History

Fini 2cd197a7ec test(ai): ab-v9 manifest matrix harness scripts/ab-v9/run_matrix.py runs the ab-v3 corpus (52 prompts) through the full Rust orchestrator per provider (op-smoke headless, OPENPENCIL_MANIFEST=1) and scores M3 expected-shape (required roles in the saved .op tree) + M5 element selection, appending rows per cell for crash-safe resume. Keys come from env only (MM_KEY/ARK_KEY/DS_KEY). op-smoke grows OPENPENCIL_SMOKE_KEEP_THINKING=1 to keep MiniMax reasoning ON: ab-v9 showed M3-nothink emits lazy minimal manifests (17%, ~10s answers) while M3-with-thinking lands 60% with composite tied-best at ~110s — the MiniMax production routing target.	2026-06-10 21:04:07 +08:00
..
run_matrix.py	test(ai): ab-v9 manifest matrix harness	2026-06-10 21:04:07 +08:00

Fini 2cd197a7ec test(ai): ab-v9 manifest matrix harness

scripts/ab-v9/run_matrix.py runs the ab-v3 corpus (52 prompts) through
the full Rust orchestrator per provider (op-smoke headless,
OPENPENCIL_MANIFEST=1) and scores M3 expected-shape (required roles in
the saved .op tree) + M5 element selection, appending rows per cell for
crash-safe resume. Keys come from env only (MM_KEY/ARK_KEY/DS_KEY).

op-smoke grows OPENPENCIL_SMOKE_KEEP_THINKING=1 to keep MiniMax
reasoning ON: ab-v9 showed M3-nothink emits lazy minimal manifests
(17%, ~10s answers) while M3-with-thinking lands 60% with composite
tied-best at ~110s — the MiniMax production routing target.

2026-06-10 21:04:07 +08:00

run_matrix.py

test(ai): ab-v9 manifest matrix harness

2026-06-10 21:04:07 +08:00