feat(models): BYOK all the way down — factory pins deleted, per-need refusal gates

Owner decision 2026-09-22: 'it doesn't make sense to have any default
model configured anywhere; the application is bring your own keys.'

config.ts: MODELS deleted; ModelRole becomes an explicit 9-role union;
DEFAULT_THINKING_ORDER (medium, low, high, minimal, xhigh, max) bakes
the unset-level default — first entry the model supports; explicitly
set levels keep pi's native nearest-clamp.

core/settings-resolution: the factory layer dissolves — the chain is
role override → slot → NOTHING. resolveSettings(record) takes no
factory. Empty chains resolve to null/undefined; a thinkingLevel-only
slot is inert (stored, never run). resolveConversationSpec: chat.model
override over the strong slot, null when both empty. resolveRoleSpec
spans all nine roles for the job/extraction engines.

Refusal envelope (core/settings.ts): modelsNotConfigured(missing) —
{error, code: 'models_not_configured', missing: [slot tiers]} +
ModelsNotConfiguredError with .envelope. Per-need gates, each honest:
- chat-handler: universal turn gate (new + continuing turns) returns
  503 before beginTurn — no liveness, no write-ahead persist, no graph
  trigger before the gate fires.
- runner.runJob: unresolvable engine lands the job as error with the
  envelope wording; no phantom loop, no result steer.
- extraction: refuseIfStagesUnconfigured throws before any stage
  (turn-service info-skips so coverage retries next turn; the import
  route surfaces per-file code+missing); labeler-less runs skip
  cluster naming with a console line and still merge.

Write side: slot entries must carry a model — level-only slot writes
are 400s (absent slot key = clear). piEngineForSpec resolves unset
thinking levels via DEFAULT_THINKING_ORDER; piEngineFor now throws
honestly; thinkingLevelFor/modelIdFor deleted; teacher-agent, pi-run,
visual-maker, pdf-reader all read engine.level.

model-catalog: defaults/slots/rolePins null when the chain is empty
(contracts nullable) — no phantom ids in payloads. Stores drop
MODEL_PINS entirely.

Tests: pin-leaners converted to explicit injection (per-chat model
pins on scripted chats — exactly what the setup flow writes; faux
engines for agent-construction tests). New coverage: the 503 gate
envelope, the runner refusal, the extraction refusal + labeler skip.

Gates: typecheck, lint, 815/815, build.
This commit is contained in:
amosblomqvist
2026-09-22 16:59:16 +03:00
parent 8675b19bef
commit bfa906e2f8
26 changed files with 889 additions and 562 deletions
+53 -31
View File
@@ -1,7 +1,7 @@
import type { UIMessage } from "ai";
import type { Usage } from "@earendil-works/pi-ai";
import { createProberAgent, createTeacherAgent, type AgentBuildOptions, type TeachingAgent } from "../agent/teacher-agent.ts";
import { piEngineForSpec } from "../agent/models.ts";
import { piEngineForSpec, type PiEngine } from "../agent/models.ts";
import { createJobRunner } from "../agent/subagents/runner.ts";
import { createExtractionPipeline } from "../agent/subagents/extraction.ts";
import { runGraphContextJob, type GraphContextDeps } from "../agent/subagents/graph-context.ts";
@@ -10,9 +10,15 @@ import type { ModelRole } from "../../config.ts";
import { claimSteers, deliverJobResult, setContextRespawn, setJobRespawn, setTurnRunner } from "./jobs.ts";
import { recordToPiMessages } from "./pi/record-adapter.ts";
import { createPiTurnResponse } from "./pi/ui-stream.ts";
import { effectiveSettings, MODEL_PINS, stores } from "./stores.ts";
import { effectiveSettings, stores } from "./stores.ts";
import { searchAvailable, searchProvider } from "./search.ts";
import { extractionEnabled, graphContextEnabled, resolveRoleSpec } from "../core/settings-resolution.ts";
import {
extractionEnabled,
graphContextEnabled,
resolveConversationSpec,
resolveRoleSpec,
} from "../core/settings-resolution.ts";
import { modelsNotConfigured } from "../core/settings.ts";
import { beginTurn, endTurn } from "./turn-liveness.ts";
import { persistTurn, triggerExtraction, triggerGraphContext, withMessageIds } from "./turn-service.ts";
import type { CallUsage } from "../core/contracts.ts";
@@ -40,12 +46,12 @@ const jobRunner = createJobRunner({
chatStore: stores.chatStore,
jobStore: stores.jobStore,
deliver: deliverJobResult,
// Model slots wave (2026-09-20): job engines resolve the full chain —
// role override → slot → factory pin — read FRESH per job (a settings
// PUT between dispatch and spawn lands). Closes the old gap where
// per-role overrides reached rolePins but never the engines.
// BYOK (owner, 2026-09-22): job engines resolve the chain — role
// override → slot → NOTHING — read FRESH per job (a settings PUT
// between dispatch and spawn lands); an empty chain flows through as
// undefined and the runner refuses the spawn honestly.
engineFor: (role) =>
piEngineForSpec(resolveRoleSpec(role, stores.settingsStore.readSettings(), MODEL_PINS)),
resolveEngine(role),
// Search port wave (2026-09-20): the researcher's web_search runs on the
// injected port — resolved FRESH per job (key rotations land immediately).
// null = honestly unavailable; the dispatch-level drop (below) uses the
@@ -60,13 +66,12 @@ export const extraction = createExtractionPipeline({
// The REAL embedder (local transformer model): post-merge embedding
// batch — semantic clusters are computed and persisted here.
embedder: embedNewNodes,
// Model slots wave (2026-09-20): the extraction stages resolve the FULL
// chain — role override → slot → factory pin — read FRESH per stage call
// (the flagged gap: ExtractionModels deps were engine VALUES, keeping
// extractor/auditor/labeler config-pin-bound while the job runner went
// slot-native). Direct test values still win inside the pipeline.
// BYOK (owner, 2026-09-22): the extraction stages resolve the chain —
// role override → slot → NOTHING — read FRESH per stage call; an empty
// chain flows through as undefined and the pipeline refuses the run
// with the typed ModelsNotConfiguredError. Direct test values still win.
engineFor: (role) =>
piEngineForSpec(resolveRoleSpec(role, stores.settingsStore.readSettings(), MODEL_PINS)),
resolveEngine(role),
});
/** Graph context injection (vision-context-injection.md): the runner's
@@ -97,6 +102,15 @@ setTurnRunner(async (chatId, messages) => {
await res.arrayBuffer().catch(() => undefined);
return res.status;
});
/** The settings-resolved engine for one role — BYOK (owner, 2026-09-22):
* the chain (role override → slot → NOTHING) may end empty, and undefined
* flows through to the per-need refusal gates (runner/extraction), read
* FRESH so a settings PUT between dispatch and spawn lands. */
function resolveEngine(role: ModelRole): PiEngine | undefined {
const spec = resolveRoleSpec(role, stores.settingsStore.readSettings());
return spec ? piEngineForSpec(spec) : undefined;
}
setJobRespawn((jobId) => void jobRunner.runJob(jobId));
setContextRespawn((jobId) => void runGraphContextJob(graphContext, jobId));
@@ -135,6 +149,25 @@ export async function handleChatRequest(
return Response.json({ error: "messages must be an array" }, { status: 400 });
}
// THE UNIVERSAL TURN GATE (BYOK, owner, 2026-09-22): a turn needs the
// CONVERSATION chain — the chat's own model override over the strong
// slot — and there is no factory floor; when both are empty the turn
// refuses honestly with the machine-readable 503 envelope (new sessions
// and continuing ones alike — wake-pipeline turns ride this same path).
// Placement is deliberate: BEFORE beginTurn (no liveness effects), the
// write-ahead persist (nothing durable changes), and every downstream
// resolution (jobs/extraction gate per-need on their own tiers).
const effective = effectiveSettings();
const conversation = resolveConversationSpec(
stores.chatStore.getChat(chatId)?.settings,
effective.models.strong,
);
if (!conversation) {
const envelope = modelsNotConfigured(["strong"]);
console.info(`[chat ${chatId}] turn refused — ${JSON.stringify(envelope)}`);
return Response.json(envelope, { status: 503 });
}
// One turn per chat, ever. A second request while a turn runs means a
// stale/remounted client raced a live one — its message list is outdated,
// and letting it through would double-run the teacher and clobber the
@@ -158,10 +191,8 @@ export async function handleChatRequest(
) as UIMessage[];
// THE SETTINGS VIEW (settings-storage wave, 2026-09-19): resolved fresh
// per turn — graph gates, tool toggles, and (via the store) the record's
// chat defaults all read from ONE small file. No caching: a stale toggle
// after a settings PUT would be worse than the read.
const effective = effectiveSettings();
// per turn (the gate above owns the read) — graph gates, tool toggles,
// and the record's chat defaults all read from ONE small file.
// Write-ahead persistence: the request body IS the client's canonical
// state, so it goes on disk BEFORE the stream starts. The learner's answer
@@ -220,19 +251,10 @@ export async function handleChatRequest(
// steers, usage, extraction) — only the agent differs.
const teaching = outgoing.some((m) => m.parts.some((p) => p.type === "tool-probe_done"));
const role: ModelRole = teaching ? "teacher" : "prober";
// The chat's model-control settings (prober/teacher only — one controller
// for the conversation agents; piEngineFor falls back to config on absent
// fields and clamps the thinking level to the model's supported list).
const settings = stores.chatStore.getChat(chatId)?.settings;
// Model slots wave (2026-09-20): the conversation chain is the chat's
// own override over the STRONG slot (prober + teacher's tier) over the
// factory pin — each field independent, same discipline as the roles.
const strong = effective.models.strong;
const engine = piEngineForSpec({
provider: settings?.model?.provider ?? strong.provider,
id: settings?.model?.id ?? strong.id,
thinkingLevel: settings?.thinkingLevel ?? strong.thinkingLevel,
});
// BYOK (owner, 2026-09-22): the conversation chain was resolved by the
// turn gate above (chat.model override → strong slot → refused) — the
// engine build is pure from here; thinking levels clamp natively.
const engine = piEngineForSpec(conversation);
// Provider-qualified stamp (model slots wave): matches the price table's
// provider:id keys.
const modelId = `${engine.model.provider}:${engine.model.id}`;
+66 -42
View File
@@ -39,63 +39,77 @@ import { BACKGROUND_MODEL_ROLES, type BackgroundModelRole, type ModelSlotName, t
/** The picker's client-safe model entry (contract: core/contracts/models.ts). */
export type ModelDto = ModelCatalogEntry;
/** The conversation agents' resolved default pin (pure form — the record's
* chat section in, CatalogDefaults out; `catalogDefaults()` below is the
* fs wrapper). Absent fields (or no record — BYOK first boot) fall back
* to the config pin, exactly as before. */
export function catalogDefaultsFor(chat: ChatSettings | undefined, strong: EffectiveSlot): CatalogDefaults {
/** The conversation agents' resolved default (pure form — the record's chat
* section + the strong slot in, CatalogDefaults out; `catalogDefaults()`
* below is the fs wrapper). BYOK (owner, 2026-09-22): there is no factory
* floor — when neither the chat section nor the strong slot names a
* model, the default is NULL and the turn gate refuses with the 503
* models_not_configured envelope. An unset thinkingLevel resolves from
* DEFAULT_THINKING_ORDER at engine build (reported as absent here). */
export function catalogDefaultsFor(
chat: ChatSettings | undefined,
strong: EffectiveSlot | null,
): CatalogDefaults | null {
const model = chat?.model ?? (strong ? { provider: strong.provider, id: strong.id } : undefined);
if (!model) return null;
const level = chat?.thinkingLevel ?? strong?.thinkingLevel;
return {
model: chat?.model ?? { provider: strong.provider, id: strong.id },
thinkingLevel: chat?.thinkingLevel ?? strong.thinkingLevel,
model,
...(level !== undefined ? { thinkingLevel: level } : {}),
};
}
/** The conversation agents' config pin — what a settings-less chat runs.
* (settings-storage wave, 2026-09-19): the user's picked default rides in
* the settings record's `chat` section; the picker's defaults row shows
* it once chosen. */
export function catalogDefaults(): CatalogDefaults {
/** The conversation agents' resolved default — what a settings-less chat
* runs. (settings-storage wave, 2026-09-19): the user's picked default
* rides in the settings record's `chat` section; the picker's defaults
* row shows it once chosen. */
export function catalogDefaults(): CatalogDefaults | null {
return catalogDefaultsFor(settingsStore.readSettings()?.chat, effectiveSettings().models.strong);
}
/** Project the resolved per-role specs to the client DTO (pure — the
* settings UI's "Default: <actual pinned model>" caption reads this).
* thinkingLevel is always concrete in the resolved view; the contract
* keeps it optional so the UI tolerates a future role without one. */
export function rolePinsFor(models: Record<BackgroundModelRole, EffectiveModelSpec>): RolePins {
* BYOK (owner, 2026-09-22): an empty chain projects to NULL — the caption
* honestly shows "not configured", never a phantom default. */
export function rolePinsFor(
models: Record<BackgroundModelRole, EffectiveModelSpec | undefined>,
): RolePins {
const pins = {} as RolePins;
for (const role of BACKGROUND_MODEL_ROLES) {
const spec = models[role];
pins[role] = {
model: { provider: spec.provider, id: spec.id },
thinkingLevel: spec.thinkingLevel,
};
pins[role] = spec
? {
model: { provider: spec.provider, id: spec.id },
...(spec.thinkingLevel !== undefined ? { thinkingLevel: spec.thinkingLevel } : {}),
}
: null;
}
return pins;
}
/** The six background roles' RESOLVED pins (settings interlude,
* 2026-09-19): the user's agent-models override falling back to the
* config.ts MODELS pin — the same resolution the agent factories use
* (src/server/stores.ts effectiveSettings), so the caption can never
* drift from what actually runs. */
/** The six background roles' RESOLVED pins: the user's agent-models
* override → tier slot → nothing — the same resolution the agent wiring
* uses (src/server/stores.ts effectiveSettings), so the caption can
* never drift from what actually runs. */
export function catalogRolePins(): RolePins {
return rolePinsFor(effectiveSettings().agent.models);
}
/** The three slot tiers' RESOLVED pins (model slots wave, 2026-09-20) —
* effectiveSettings().models verbatim: slot over factory rep pin, the
* same resolution the engine wiring uses, so the Models settings page's
* captions cannot drift from what actually runs. */
/** The three slot tiers' RESOLVED pins — effectiveSettings().models
* verbatim: the same resolution the engine wiring uses, so the Models
* settings page's captions cannot drift from what actually runs. NULL
* per tier that holds no model. */
export function catalogSlots(): SlotPins {
const slots = effectiveSettings().models;
const pins = {} as SlotPins;
for (const name of ["strong", "cheap", "vision"] as const) {
const slot: EffectiveSlot = slots[name];
pins[name] = {
model: { provider: slot.provider, id: slot.id },
thinkingLevel: slot.thinkingLevel,
};
const slot: EffectiveSlot | null = slots[name];
pins[name] = slot
? {
model: { provider: slot.provider, id: slot.id },
...(slot.thinkingLevel !== undefined ? { thinkingLevel: slot.thinkingLevel } : {}),
}
: null;
}
return pins;
}
@@ -183,11 +197,14 @@ export function validateChatSettings(settings: ChatSettings): SettingsValidation
}
}
if (settings.thinkingLevel !== undefined) {
// The conversation chain's fallback is the strong slot (model slots
// wave, 2026-09-20) — the model a settings-less chat would run.
// The conversation chain's fallback is the strong slot — the model
// a settings-less chat would run. BYOK (owner, 2026-09-22): with no
// model to check against (no chat.model, no strong slot), the level
// rides unchecked — it clamps against whatever the chain later holds.
const strong = effectiveSettings().models.strong;
const provider = settings.model?.provider ?? strong.provider;
const id = settings.model?.id ?? strong.id;
const provider = settings.model?.provider ?? strong?.provider;
const id = settings.model?.id ?? strong?.id;
if (!provider || !id) return { ok: true };
const model = piModels().getModel(provider, id);
if (!model) return { ok: false, error: `No model "${id}" in the ${provider} catalog` };
const supported = getSupportedThinkingLevels(model);
@@ -202,9 +219,10 @@ export function validateChatSettings(settings: ChatSettings): SettingsValidation
return { ok: true };
}
/** Validate a models-section write (PUT /api/settings/models) against the
* catalog — the same gate the chat section gets. Model swaps must name a
* real provider+model; a thinkingLevel must be supported by the model the
* slot would resolve to (the slot's own model, else the factory rep pin).
* catalog — the same gate the chat section gets. A slot must NAME a real
* provider+model (owner, 2026-09-22: no factory floor exists, so a level
* alone can never run — the parser rejects it, this gate re-checks the
* model); a thinkingLevel must be supported by the slot's own model.
* keyPresent is NOT gated here (a stored pin for a provider whose key
* lapsed stays valid — it fails honestly at stream time, and the picker
* stops offering it). */
@@ -230,8 +248,14 @@ function validateSlotSpec(name: ModelSlotName, slot: { model?: { provider: strin
}
}
if (slot.thinkingLevel !== undefined) {
const provider = slot.model?.provider ?? effectiveSettings().models[name].provider;
const id = slot.model?.id ?? effectiveSettings().models[name].id;
// The write-side parser requires slot.model (owner, 2026-09-22 — a
// level-only slot can never run), so this fallback is tolerance for
// future callers; with no model to check against, accept (the
// engine build clamps whatever the chain later holds).
const resolved = effectiveSettings().models[name];
const provider = slot.model?.provider ?? resolved?.provider;
const id = slot.model?.id ?? resolved?.id;
if (!provider || !id) return { ok: true };
const model = piModels().getModel(provider, id);
if (!model) return { ok: false, error: `No model "${id}" in the ${provider} catalog` };
const supported = getSupportedThinkingLevels(model);
+7 -4
View File
@@ -309,11 +309,14 @@ export function parseModelsSettingsBody(body: unknown): SettingsBodyResult<Model
}
const parsed = parseRoleOverride(raw[key], key);
if (!parsed.ok) return { ok: false, error: parsed.error };
if (Object.keys(parsed.override).length) {
settings[key as (typeof MODEL_SLOTS)[number]] = parsed.override;
} else {
return { ok: false, error: `slot "${key}" must set model or thinkingLevel (absent slot = unset)` };
// BYOK (owner, 2026-09-22): a slot IS a model choice — no factory
// floor exists, so a thinkingLevel alone can never run; every slot
// write must carry the model (the level is an optional rider). An
// absent slot key clears the tier (replace-whole section).
if (!parsed.override.model) {
return { ok: false, error: `slot "${key}" must set a model (thinkingLevel alone cannot run — absent slot = unset)` };
}
settings[key as (typeof MODEL_SLOTS)[number]] = parsed.override;
}
return { ok: true, settings };
}
+5 -24
View File
@@ -6,12 +6,11 @@
* file (and the adapter implementations in src/store/).
*/
import { MODELS } from "../../config.ts";
import { chatStore } from "../store/chats.ts";
import { graphStore } from "../store/graph.ts";
import { jobStore } from "../store/jobs.ts";
import { settingsStore } from "../store/settings.ts";
import { resolveSettings, type EffectiveSettings, type ModelRoleSpecs } from "../core/settings-resolution.ts";
import { resolveSettings, type EffectiveSettings } from "../core/settings-resolution.ts";
/** The wired adapters. Satisfies core/graph.ts's GraphDeps structurally. */
export const stores = {
chatStore,
@@ -22,32 +21,14 @@ export const stores = {
export type Stores = typeof stores;
/** config.ts's MODELS as the compiled factory layer the resolution reads —
* ALL NINE roles since the model-slots wave (2026-09-20): the strong
* slot's factory fallback IS the prober pin. core stays pure (it takes
* this as a parameter, never imports config at runtime). Per-role
* OVERRIDES stay background-only (the conversation pair is
* chat-settings-controlled) — that split lives in core's
* BACKGROUND_MODEL_ROLES, unchanged. */
const MODEL_PINS: ModelRoleSpecs = {
prober: MODELS.prober,
teacher: MODELS.teacher,
researcher: MODELS.researcher,
demoBuilder: MODELS.demoBuilder,
visualMaker: MODELS.visualMaker,
pdfReader: MODELS.pdfReader,
extractor: MODELS.extractor,
auditor: MODELS.auditor,
labeler: MODELS.labeler,
};
export { MODEL_PINS };
/**
* The resolved view of the settings record (core/settings-resolution.ts):
* every knob concrete, record overrides over config.ts factory pins. Read
* every knob concrete over its factory default — EXCEPT MODELS (BYOK all
* the way down, owner 2026-09-22): no factory pins exist, so unset model
* slots resolve to null and per-need refusal gates fire downstream. Read
* fresh on every call — the record is a small file and turns are rare, so
* no caching layer (stale toggles after a settings PUT would be worse).
*/
export function effectiveSettings(): EffectiveSettings {
return resolveSettings(settingsStore.readSettings(), MODEL_PINS);
return resolveSettings(settingsStore.readSettings());
}
+8
View File
@@ -22,6 +22,7 @@ import type { ChatStore, GraphStore } from "../core/ports.ts";
import { EXTRACTION } from "../../config.ts";
import type { ExtractionPipeline } from "../agent/subagents/extraction.ts";
import { startGraphContextInjection, type GraphContextDeps } from "../agent/subagents/graph-context.ts";
import { ModelsNotConfiguredError } from "../core/settings.ts";
/**
* The canonical state write: the client owns the message list between turns
@@ -98,6 +99,13 @@ export function triggerExtraction(
const covered = coverageOf(record.messages as UIMessage[], deps.graphStore.readCoverage(), chatId);
if (covered.uncoveredTokens < EXTRACTION.chunkTokens) return;
void extraction.runExtraction(chatId).catch((err) => {
// BYOK (owner, 2026-09-22): an unconfigured write channel SKIPS
// (the coverage ledger keeps the messages uncovered — the next
// trigger retries once slots are set); only real failures log.
if (err instanceof ModelsNotConfiguredError) {
console.info(`[chat ${chatId}] extraction skipped — ${err.envelope.error} (missing: ${err.envelope.missing.join(", ")})`);
return;
}
console.error(`[chat ${chatId}] extraction pipeline failed:`, err);
});
} catch (err) {