95566e4ed2
ab-v3 succeeds ab-v1 (frozen 2026-04-28). Carries forward all 40
v1 obvious yaml files unchanged so the v1↔v3 overlap stays
comparable, then layers in two new dimensions.
**1. Token cost.** All clients (openai-compat, ark, bailian,
deepseek, minimax, codex-cli, stub-model) now return a
`ChatCallResult { content, usage }` instead of bare string.
Provider usage stats (`prompt_tokens` / `completion_tokens`) plumb
through realModelCall → run.ts → scoreRun → ScoreRow.{prompt,completion}Tokens.
aggregate adds avgPromptTokens{Baseline,Treatment} +
avgCompletionTokens{Baseline,Treatment} per ModelSummary.
write-report emits a new "Token cost" table with Δ columns so
narrow-tools-saves-tokens (the ab-v2 hypothesis) is measurable.
avgUsage skips rows with 0/0 usage so codex-cli (CLI doesn't
surface tokens) and harness errors don't deflate the average to
near-zero — they show '—' instead.
**2. Composite difficulty.** New 'composite' value alongside
obvious / optional. Composite prompts express multi-tool intents
where no single expected_tool_if_any applies. classifyRouting
routes composite-treatment runs into multi-tool / fallback /
garbage (3-bucket sum to 1, distinct from obvious's 4-bucket
right/wrong/fallback/garbage). aggregate adds m6_multi_tool +
m6_fallback + m6_garbage; write-report emits a "Composite routing"
table that gracefully degrades to a placeholder when no composite
yaml exists yet.
Harness side: scripts/ab-corpus/run.ts accepts --corpus ab-v3
(enum + parseArgs guard); dry-run on the v1-mirror corpus produces
a 160-row report including populated token table.
Tests: 4 new aggregate cases (composite, token avg with skip-zero,
NaN-when-no-data) + 4 new score-run cases (composite routing
multi-tool/fallback/garbage/baseline-n/a) + 2 new score-run cases
(usage plumbing) + 2 new openai-compat cases (usage parsing,
missing-usage fallback). Existing 5 retry tests updated for new
return shape. 3727 → 3740 vitest tests, all green; tsc + format
clean.
Token-cost docs and composite docs go straight into types.ts /
score-run.ts / aggregate.ts JSDoc — keeps the contract close to
the code that owns it.
48 lines
1.6 KiB
TypeScript
48 lines
1.6 KiB
TypeScript
/**
|
|
* 百炼 (Alibaba DashScope Bailian) Coding Plan wrapper. Exposes
|
|
* third-party models — including GLM-5 and Kimi K2.5 — through a
|
|
* single DashScope endpoint, so the same API key (`DASHSCOPE_BAILIAN_CODING_KEY`)
|
|
* covers multiple target models. Cribbed from builtin-provider-presets.ts
|
|
* `bailian-coding` preset.
|
|
*
|
|
* Default baseURL: https://coding.dashscope.aliyuncs.com/v1 (CN).
|
|
* Override via `BAILIAN_BASE_URL` if you need the -intl variant.
|
|
*/
|
|
|
|
import type { ChatCallResult } from './openai-compat';
|
|
import { callOpenAICompat } from './openai-compat';
|
|
|
|
// User supplied the 百炼 CP (Coding Plan) key, which targets the
|
|
// coding-plan endpoint (not the regular compatible-mode endpoint).
|
|
// See builtin-provider-presets.ts `bailian-coding` preset for the
|
|
// canonical URL — this is separate from plain `bailian` tier.
|
|
const BAILIAN_BASE_URL = process.env.BAILIAN_BASE_URL ?? 'https://coding.dashscope.aliyuncs.com/v1';
|
|
const BAILIAN_API_KEY_ENV = 'DASHSCOPE_BAILIAN_CODING_KEY';
|
|
|
|
export interface CallBailianArgs {
|
|
model: string;
|
|
system: string;
|
|
user: string;
|
|
temperature?: number;
|
|
maxTokens?: number;
|
|
}
|
|
|
|
export async function callBailian(args: CallBailianArgs): Promise<ChatCallResult> {
|
|
const apiKey = process.env[BAILIAN_API_KEY_ENV];
|
|
if (!apiKey) {
|
|
throw new Error(
|
|
`${BAILIAN_API_KEY_ENV} not set — export it (the 百炼 CP sk-sp-... key) before running with --live`,
|
|
);
|
|
}
|
|
return callOpenAICompat({
|
|
baseURL: BAILIAN_BASE_URL,
|
|
apiKey,
|
|
model: args.model,
|
|
system: args.system,
|
|
user: args.user,
|
|
temperature: args.temperature,
|
|
maxTokens: args.maxTokens,
|
|
label: 'bailian',
|
|
});
|
|
}
|