sorokin.oleg.edu/learnharness
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 6 Wiki Activity

feat: model + thinking control — the stats bar's picker, on pi-ai's catalog

Browse Source 
The model-control wave (user-decided design, 2026-09-12): ONE controller
for the conversation agents (prober/teacher are functionally one agent);
background sub-agents stay config-pinned. pi-ai owns the registry — the
shipped 366-model OpenRouter catalog (cost, context, modality,
thinkingLevelMap), getSupportedThinkingLevels/clampThinkingLevel for the
per-model thinking truth, and streamSimple's automatic clamp. No second
registry to drift out of sync.

Seams:
- config.ts: ReasoningEffort retired for pi's ThinkingLevel
  (off|minimal|low|medium|high|xhigh|max). GLM roles go honest at "low" —
  pi's catalog says GLM-5.3-flash accepts exactly low|high|max, and the old
  "minimal" was silently clamped to low by streamSimple since the swap;
  nothing changes behaviorally. visualMaker → qwen/qwen3.8-flash (fixes
  the dead qwen/qwen3.8-max slug — the catalog only ships -0902; also the
  user's cost pick, 0.15/0.47 per-1M).
- src/agent/models.ts: piEngineFor(role, settings?) — provider+id resolved
  from the catalog, thinking level from settings ?? config, CLAMPED to the
  model's supported list (a settings change between models can never send
  an invalid level). PiEngine gains `level` (observability/tests).
- ChatRecord.settings {model?: {provider, id}, thinkingLevel?} — written
  by PUT /api/chats/[id]/settings (replace-whole, no updatedAt churn),
  read by chat-handler at turn composition: agents are stateless per turn,
  so the change is next-turn-effective by construction. The turns log
  still records the ACTUAL billed model (the honest witness).
- GET /api/models: the configured-provider catalog as client-safe DTOs
  (core/contracts/models.ts) + config defaults; a second provider joins
  the picker by registering it in piModels() — nothing else changes.
- PUT settings validation gate (src/server/model-catalog.ts): unknown
  provider/model or unsupported level → 400 carrying the supported list,
  never a silent clamp.
- scripts/sync-pricing.ts: regenerated PRICES/CONTEXT_LIMITS from pi-ai's
  shipped openrouter.json (offline; same source as the engine + picker).
  All 366 models priced; glm's stale 0.15/0.5 → pi's 0.075/0.25, limit
  1310720 → 1048576 — cost is inferred at read time, so history re-prices.
- UI (components/model-picker.tsx): stats-bar right side becomes model +
  thinking buttons; menus open ABOVE; fuzzy search over "provider id
  name" (core/fuzzy.ts — pi's /model pattern); model rows show name/id/
  price/context; the thinking menu lists exactly the chosen model's
  supported levels (GLM: low|high|max — no dead options, no "off").
  Model switch drops an incompatible pinned level (falls back to config,
  which clamps honestly) rather than carrying a 400.

Tests (350/350, +15): catalog projection + defaults, the settings gate,
pi thinking truth pinned against upgrades (glm low|high|max; minimal→low;
off→low — GLM cannot disable thinking), the engine factory's
override/fallback/clamp/throw matrix, fuzzy ranking, store round-trip,
and the re-priced folds (usage, stats bar, activity dock).

Live smoke (disposable driver, deleted): catalog route 366 + defaults;
glm+minimal → 400 + supported; turn 1 billed qwen/qwen3.8-flash under
the override; mid-session switch flipped turn 2 to glm; clear-overrides
restored the config pin. ALL CHECKS PASS on real OpenRouter.
This commit is contained in:
amosblomqvist
2026-09-12 06:02:40 +03:00
parent 740293c361
commit de0c286311
26 changed files with 1649 additions and 131 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/sync-pricing.ts
+50 -64
View File
@@ -1,85 +1,72 @@
/**
* Regenerate src/core/prices.ts tables from OpenRouter's published pricing.
* Regenerate src/core/prices.ts tables from pi-ai's shipped OpenRouter
* catalog (providers/data/openrouter.json).
*
* Usage: npx tsx scripts/sync-pricing.ts
*
* Pulls GET https://openrouter.ai/api/v1/models, filters to the models in
* config.ts MODELS (+ known spelling variants), and rewrites the PRICES +
* CONTEXT_LIMITS tables in src/core/prices.ts — the tables are generated;
* hand edits are overwritten. Per-token prices convert to per-1M-token USD;
* fields OpenRouter doesn't publish (null) are omitted. A model missing from
* OpenRouter's list gets no entry — cost renders as absent ("—"), never zero.
* The catalog is pi-ai's curated, versioned snapshot of OpenRouter's model
* list — the SAME source piEngineFor resolves engines from and
* /api/models serves the picker from — so the money channel's lookup table
* can never disagree with the engine. Fully offline (the JSON ships in
* node_modules); refreshes happen by bumping @earendil-works/pi-ai and
* re-running this script. The tables are generated; hand edits are
* overwritten. A model missing from the catalog gets no entry — cost
* renders as absent ("—"), never zero.
*/
import { MODELS } from "../config.ts";
import * as fs from "node:fs";
import * as path from "node:path";
interface OpenRouterModel {
id: string;
context_length?: number;
pricing?: Record<string, string | null>;
}
/** Spelling variants worth keeping priced alongside a config id (the config's
* id must always resolve even if OpenRouter renames the slug). */
const VARIANTS: Record<string, string[]> = {
"qwen/qwen3.8-max": ["qwen/qwen3.8-max-0902"],
"qwen/qwen3.8-max-0902": ["qwen/qwen3.8-max"],
};
const API = "https://openrouter.ai/api/v1/models";
const CATALOG = path.join(
import.meta.dirname,
"..",
"node_modules",
"@earendil-works",
"pi-ai",
"dist",
"providers",
"data",
"openrouter.json",
);
const OUT = path.join(import.meta.dirname, "..", "src", "core", "prices.ts");
const perM = (v: string | null | undefined): number | undefined => {
if (v === null || v === undefined || v === "") return undefined;
const n = Number(v) * 1e6;
return Number.isFinite(n) ? Number(n.toFixed(4)) : undefined;
};
function priceLine(id: string, model: OpenRouterModel): string {
const p = model.pricing ?? {};
const fields = [
`input: ${perM(p.prompt)}`,
`output: ${perM(p.completion)}`,
`cacheRead: ${perM(p.input_cache_read)}`,
...(perM(p.input_cache_write) !== undefined ? [`cacheWrite: ${perM(p.input_cache_write)}`] : []),
];
return `\t${JSON.stringify(id)}: { ${fields.join(", ")} },`;
/** One raw catalog entry (fields we consume; the file carries more). */
interface CatalogModel {
id: string;
name: string;
cost: { input: number; output: number; cacheRead: number; cacheWrite: number };
contextWindow: number;
}
async function main(): Promise<void> {
const res = await fetch(API);
if (!res.ok) throw new Error(`OpenRouter models request failed: ${res.status} ${res.statusText}`);
const { data } = (await res.json()) as { data: OpenRouterModel[] };
const byId = new Map(data.map((m) => [m.id, m]));
function main(): void {
// Keyed by api ("openai-completions" | "anthropic-messages"), each a
// Record<id, entry> — every OpenRouter id is unique across the two maps.
const catalog = JSON.parse(fs.readFileSync(CATALOG, "utf-8")) as Record<
string,
Record<string, CatalogModel>
>;
const byId = new Map<string, CatalogModel>();
for (const api of Object.values(catalog)) for (const [id, m] of Object.entries(api)) byId.set(id, m);
const wanted = Object.values(MODELS).map((m) => m.model);
const all = [...new Set([...wanted, ...wanted.flatMap((id) => VARIANTS[id] ?? [])])];
const priced = new Map<string, { price: string; limit: string }>();
for (const id of all) {
const model = byId.get(id);
if (!model) {
console.warn(`!! ${id}: not on OpenRouter — entry omitted (cost renders as "—")`);
continue;
}
const limit = model.context_length;
priced.set(id, {
price: priceLine(id, model),
limit: `\t${JSON.stringify(id)}: ${limit ?? "undefined"},`,
});
}
// Sort by id for a stable, diff-friendly generated table.
const priced: { price: string; limit: string }[] = [...byId.entries()]
.sort(([a], [b]) => a.localeCompare(b))
.map(([id, m]) => ({
price: `\t${JSON.stringify(id)}: { input: ${m.cost.input}, output: ${m.cost.output}, cacheRead: ${m.cost.cacheRead}, cacheWrite: ${m.cost.cacheWrite} },`,
limit: `\t${JSON.stringify(id)}: ${m.contextWindow},`,
}));
const tables = `/** Token prices, USD per 1M tokens. A missing model makes cost render
* as absent ("—"), never zero. Refresh: npx tsx scripts/sync-pricing.ts */
* as absent ("—"), never zero. Refresh: bump @earendil-works/pi-ai, then
* npx tsx scripts/sync-pricing.ts (reads pi-ai's shipped catalog — the
* same source the engine and the picker use). */
export const PRICES: Record<string, ModelPrice> = {
${[...priced.values()].map((p) => p.price).join("\n")}
${priced.map((p) => p.price).join("\n")}
};
/** Context-window limits (tokens) per model — generated alongside PRICES. */
export const CONTEXT_LIMITS: Record<string, number> = {
${[...priced.values()].map((p) => p.limit).join("\n")}
${priced.map((p) => p.limit).join("\n")}
};
`;
@@ -89,8 +76,7 @@ ${[...priced.values()].map((p) => p.limit).join("\n")}
if (start === -1 || end === -1) throw new Error("prices.ts markers not found — file structure changed");
fs.writeFileSync(OUT, src.slice(0, start) + tables + "\n" + src.slice(end), "utf-8");
console.log(`prices.ts regenerated: ${all.length} models checked, ${priced.size} priced.`);
for (const id of all) if (!priced.has(id)) console.warn(` unpriced: ${id}`);
console.log(`prices.ts regenerated from pi-ai's catalog: ${priced.length} models priced.`);
}
void main();
main();