sorokin.oleg.edu/learnharness
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 7 Wiki Activity

feat(models): clamp-on-write for pinned thinking levels + DEFAULT_THINKING_ORDER in the core barrel

Browse Source 
Fix 1 — clamp, never reject (owner ruling, interlude 2026-09-22): a
model swap with a pinned thinking level must JUST WORK. The owner hit
the write-side 400 live (Claude Fable 5.1/xhigh → GLM 5.3 flash:
"z-ai/glm-5.3-flash does not support thinking level xhigh"). The
supported-level verdicts in validateChatSettings and
validateModelsSettings are DEAD — the WRITE clamps the pin to the new
model's nearest supported level (pi's clampThinkingLevel, the same
translation piEngineForSpec applies at runtime) and stores the clamped
result; the server is the truth — the record shows the clamped level
after write.

New pure helpers in model-catalog.ts, applied by ALL THREE write
routes (section, defaults, per-chat — one verdict seam, so every
surface that used the old 400 gets the clamp uniformly):
- clampModelsLevels: every slot's level clamps against the SLOT'S OWN
  model (the parser already requires a model on each slot write).
- clampChatLevel: the body's own model, else the record's resolved
  strong slot (the chain's fallback); no model anywhere → rides
  unchecked (runtime clamps at engine build; the turn gate refuses
  the model-less chat anyway).

Fix 2 — DEFAULT_THINKING_ORDER moves from config.ts to core/settings
(core barrel, AUTH_KEYS precedent — one source of truth, no drift).
The UI imports it to auto-populate a thinking level on FRESH model
picks; the ordering has no other client role. Legacy unset records
keep the runtime defaultThinkingLevel fallback at engine build
untouched. Import sites updated: agent/models.ts, tests.

Verified live on 3004: the owner's exact case stores max (not 400);
a bare chat-level write clamps to low against the strong slot's glm;
unknown provider/model still 400s with the provider list.

Tests: the level-verdict suites became clamp suites (the owner's case
pinned verbatim: xhigh→max, minimal→low, off→low, supported
pass-through, level-less slots ride); the machine-aware
expectBareLevelVerdict chain is retired — clampChatLevel takes the
strong slot as an explicit param, so the tests are pure again.

Gates: typecheck, lint, 839/839, build.
This commit is contained in:
amosblomqvist
2026-09-22 21:44:43 +03:00
parent 5ab52f07f6
commit 9387418864
8 changed files with 233 additions and 138 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/model-catalog.test.ts
+79 -48
View File
@@ -1,9 +1,22 @@
import { describe, it } from "node:test";
import assert from "node:assert/strict";
import { modelCatalog, validateChatSettings, validateModelsSettings, catalogDefaultsFor, catalogProviders, catalogSlots, toModelDto, rolePinsFor, catalogRolePins } from "../src/server/model-catalog.ts";
import {
catalogDefaultsFor,
catalogProviders,
catalogRolePins,
catalogSlots,
clampChatLevel,
clampModelsLevels,
modelCatalog,
rolePinsFor,
toModelDto,
validateChatSettings,
validateModelsSettings,
} from "../src/server/model-catalog.ts";
import type { EffectiveSlot } from "../src/core/settings-resolution.ts";
import type { SettingsValidation } from "../src/server/model-catalog.ts";
import { clampThinkingLevel, getSupportedThinkingLevels } from "@earendil-works/pi-ai";
import { DEFAULT_THINKING_ORDER } from "../config.ts";
import { DEFAULT_THINKING_ORDER } from "../src/core/settings.ts";
import { piEngineForSpec, piModels } from "../src/agent/models.ts";
import { BACKGROUND_MODEL_ROLES } from "../src/core/settings.ts";
import { resolveSettings } from "../src/core/settings-resolution.ts";
@@ -29,26 +42,11 @@ function glmModel() {
return glm!;
}
/** The record's resolved tier as seen by the validation fallbacks —
* machine-aware truth (data/settings.json belongs to settings-store.test.ts
* under the one-fs-owner discipline, so READ it, never pin it): on a
* scratch worktree the record is empty, on the owner's machine slots are
* configured; the expected verdicts below follow whichever holds. */
/** The machine-aware expected verdict for a bare-level validation: mirrors
* the fallback chain exactly — no resolved tier → rides unchecked; a tier
* whose model left the catalog → rejected; else the supported-level check. */
function expectBareLevelVerdict(verdict: SettingsValidation, tier: "strong" | "cheap" | "vision") {
const slot = effectiveSettings().models[tier];
if (!slot) {
assert.deepEqual(verdict, { ok: true }, "no model to check against → accepted (clamps at run)");
return;
}
const model = piModels().getModel(slot.provider, slot.id);
if (!model) {
assert.equal(verdict.ok, false, "a vanished model is rejected honestly");
return;
}
assert.equal(verdict.ok, getSupportedThinkingLevels(model).includes("minimal"));
/** Another reasoning specimen with a different supported set. */
function qwenModel() {
const qwen = piModels().getModel("openrouter", "qwen/qwen3.8-max-0902");
assert.ok(qwen, "qwen3.8-max in catalog");
return qwen!;
}
describe("the model catalog projection", () => {
@@ -217,21 +215,46 @@ describe("the models-section validation gate", () => {
);
});
it("thinkingLevel resolves against the slot's model; level-only slots check nothing", () => {
// level checked against the named model
assert.equal(
validateModelsSettings({ strong: { model: { provider: "openrouter", id: "z-ai/glm-5.3-flash" }, thinkingLevel: "max" } }).ok,
true,
it("thinking levels never reject a slot write (owner, interlude 2026-09-22)", () => {
// clamp, never reject: the model swap with a pinned level JUST
// WORKS — validation accepts ANY level; the WRITE clamps to the
// new model's nearest supported and stores the clamped result.
assert.deepEqual(
validateModelsSettings({ strong: { model: { provider: "openrouter", id: "z-ai/glm-5.3-flash" }, thinkingLevel: "minimal" } }),
{ ok: true },
);
assert.equal(
validateModelsSettings({ strong: { model: { provider: "openrouter", id: "z-ai/glm-5.3-flash" }, thinkingLevel: "minimal" } }).ok,
false,
assert.deepEqual(
validateModelsSettings({ strong: { model: { provider: "openrouter", id: "z-ai/glm-5.3-flash" }, thinkingLevel: "xhigh" } }),
{ ok: true },
);
// (a level-only slot never passes the WRITE parser — owner, 2026-09-22;
// the validation gate falls back to the tier's RESOLVED model —
// machine-aware: the record's strong tier decides the verdict)
const levelOnly = validateModelsSettings({ strong: { thinkingLevel: "minimal" } });
expectBareLevelVerdict(levelOnly, "strong");
});
it("clampModelsLevels: the slot's own model translates the pin (the owner's exact case)", () => {
// Claude Fable 5.1 xhigh → GLM 5.3 flash: the pin clamps to glm's
// nearest supported (low|high|max) and the CLAMPED value is stored.
const glm = { model: { provider: "openrouter", id: "z-ai/glm-5.3-flash" }, thinkingLevel: "xhigh" as const };
assert.deepEqual(clampModelsLevels({ strong: glm }), {
strong: { model: { provider: "openrouter", id: "z-ai/glm-5.3-flash" }, thinkingLevel: "max" },
});
assert.deepEqual(clampModelsLevels({ strong: { ...glm, thinkingLevel: "minimal" as const } }).strong, {
model: { provider: "openrouter", id: "z-ai/glm-5.3-flash" },
thinkingLevel: "low",
});
// glm cannot disable thinking: off clamps UP to low
assert.equal(
clampModelsLevels({ strong: { ...glm, thinkingLevel: "off" as const } }).strong?.thinkingLevel,
"low",
);
// a supported level passes through untouched
assert.deepEqual(clampModelsLevels({ strong: { ...glm, thinkingLevel: "high" as const } }).strong, {
model: { provider: "openrouter", id: "z-ai/glm-5.3-flash" },
thinkingLevel: "high",
});
// a slot with no level rides as-is (client populates on fresh
// picks per DEFAULT_THINKING_ORDER — the server never invents one)
assert.deepEqual(clampModelsLevels({ strong: { model: glm.model } }), {
strong: { model: { provider: "openrouter", id: "z-ai/glm-5.3-flash" } },
});
});
it("accepts empty settings (the clear-slots write)", () => {
@@ -240,21 +263,29 @@ describe("the models-section validation gate", () => {
});
describe("the settings validation gate", () => {
it("accepts a supported level and rejects an unsupported one with the list", () => {
const ok = validateChatSettings({ model: { provider: "openrouter", id: "z-ai/glm-5.3-flash" }, thinkingLevel: "max" });
assert.deepEqual(ok, { ok: true });
const bad = validateChatSettings({ model: { provider: "openrouter", id: "z-ai/glm-5.3-flash" }, thinkingLevel: "minimal" });
assert.equal(bad.ok, false);
if (!bad.ok) {
assert.match(bad.error, /does not support thinking level/);
assert.deepEqual(bad.supported, ["low", "high", "max"]);
}
it("thinking levels never reject a chat write (owner, interlude 2026-09-22)", () => {
assert.deepEqual(
validateChatSettings({ model: { provider: "openrouter", id: "z-ai/glm-5.3-flash" }, thinkingLevel: "minimal" }),
{ ok: true },
);
assert.deepEqual(validateChatSettings({ thinkingLevel: "xhigh" }), { ok: true });
});
it("a bare thinking level checks against the chat model, else the strong slot, else rides unchecked", () => {
// machine-aware: with a record-carried strong slot the level checks
// against it; with none it rides unchecked (clamps at run).
expectBareLevelVerdict(validateChatSettings({ thinkingLevel: "minimal" }), "strong");
it("clampChatLevel: the body's model, else the strong slot, translates the pin", () => {
const glm = { provider: "openrouter", id: "z-ai/glm-5.3-flash" };
const strong: EffectiveSlot = { provider: "openrouter", id: "z-ai/glm-5.3-flash" };
// the body's own model wins
assert.deepEqual(
clampChatLevel({ model: { provider: "openrouter", id: "qwen/qwen3.8-max-0902" }, thinkingLevel: "xhigh" as const }, strong),
{ model: { provider: "openrouter", id: "qwen/qwen3.8-max-0902" }, thinkingLevel: clampThinkingLevel(qwenModel(), "xhigh") },
);
// no body model → the record's strong slot is the fallback
assert.equal(clampChatLevel({ thinkingLevel: "xhigh" as const }, strong).thinkingLevel, "max");
// no model anywhere → rides unchecked (runtime clamps; the turn
// gate refuses the model-less chat anyway)
assert.deepEqual(clampChatLevel({ thinkingLevel: "minimal" as const }, null), { thinkingLevel: "minimal" });
// no pinned level → untouched
assert.deepEqual(clampChatLevel({ model: { ...glm } }, strong), { model: { ...glm } });
});
it("rejects unknown providers and models", () => {