openpencil/docs
Rob Coenen 1ebaac7be2 docs(ai): re-measure at a realistic token budget, document reasoning starvation
The probe behind this page used max_tokens 2000, which starves reasoning
models: they spend the whole budget thinking and emit no tool calls, with
finish_reason length and no error. That looks identical to a model that
cannot call tools, and two models were wrongly marked "avoid" because of it.

Re-ran every Scaleway model at 16384. All five are clean -- qwen3.6-35b-a3b
goes from 1/3 to 3/3, mistral-small-3.2-24b from 2/3 to 3/3. Tool calling no
longer separates them, so the recommendation now follows price, vision and
context: mistral-small-3.2-24b is cheapest with vision, gemma-4-26b-a4b-it
for double the context.

Document the failure mode, the reasoning_effort lever (only "none" had any
effect on Scaleway; "low" did nothing), and ask contributors to record the
token budget alongside results. Nebius and TensorX rows are marked as
measured at 2000 so they are not read as comparable.

Refs #454

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-03 19:08:31 +03:00
..
byok-providers.md docs(ai): re-measure at a realistic token budget, document reasoning starvation 2026-08-03 19:08:31 +03:00