sorokin.oleg.edu/learnharness
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 6 Wiki Activity

Wave 7 chunk B: remove the embeddings machinery — the ladder collapses

Browse Source 
The owner's wave-7 ruling: embeddings die entirely (git history is the
stash); clustering is birth-session derived (chunk A, 950ddfd), labels
are session names, and the prober's graph tools replace injection.

- settings: GraphSettings collapses to { enabled } — embedding,
  embeddingModel, clustering, labeling all die. Migration story: stale
  record keys are UNREAD and dropped by the sanitizer's reconstruction;
  the next graph-section PUT (replace-whole) persists the collapsed
  shape. The settings-panel GraphSection renders one master row.
- deleted: src/server/embeddings.ts (adapters, resolveEmbeddingPortFor,
  embedNewNodes, embeddingPort, embeddingModelCatalog), the
  GET /api/models/embeddings route, scripts/embed-graph.ts, tests for
  all three, and the /api/models/embeddings picker surfaces in the UI.
- prices: EMBEDDING_PRICES overlay + callCost fallback die. cost-summary:
  the embedding stage fold dies — back to three stages (extraction/
  audit/naming) per the owner ruling; historical stages.embedding spend
  in old run records is simply not displayed.
- ports: EmbeddingPort/EmbeddingResult/EmbeddingTask die. The embeddings
  LEDGER surface (readEmbeddings/writeEmbeddings + EmbeddingsLedger +
  EMBEDDINGS_FILE + graph-deletion's vector pruning) STAYS — inert data,
  kept only so graph-context.ts compiles until Teacher's pass removes
  it; the seam dies in the follow-up interlude.
- chat-handler (approved cross-lane unwiring, wave-6 precedent): the
  extraction pipeline's embedder becomes a documented no-op (nothing
  embedded, nothing spent, ledger untouched) until Teacher's pass
  removes the phase; the graph-context injection dispatch gate, the
  graphContext deps wiring, the setContextRespawn seam, and
  turn-service's triggerGraphContext are gone — injection is dead
  code pending Teacher's file deletions.
- scripts: backfill-graph/import-notes pass the same no-op embedder;
  replay-layout derives clusters from birthSessionPartition (the same
  partition the real settle uses) instead of the embeddings ledger.
- settings-body: parseGraphSettingsBody reads the master switch only —
  old-shape keys in a body are ignored, not rejected; bad shapes 400.
- next.config: the @huggingface tracing-exclusion guards die with the
  last reference.

Cross-lane flags: extraction.ts/graph-context.ts keep stale embed
comments + the embed phase until Teacher's full pass (their parallel
interlude owns those files — zero overlap kept by design);
semantic-clusters.ts + its test stay one more round (extraction.test.ts,
Teacher's file, still imports it — they delete it with the namer cases);
ChatSettings.graphContextTokens + the graph-context-control UI stay
until Teacher+UI.

Gates: tsc clean, eslint clean, 885/885 tests, build compiles.
Live-verified on the port-3004 dev tree: PUT /api/settings/graph with
dead keys collapses on write, bad shapes 400, GET reads {enabled: true},
/api/models/embeddings 404s.
This commit is contained in:
amosblomqvist
2026-09-23 15:02:55 +03:00
parent 0fcffdffaf
commit c86430e8bf
26 changed files with 120 additions and 1823 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/cost-summary.test.ts
-33
View File
@@ -96,39 +96,6 @@ describe("graphCostSummary (the cost pill's fold)", () => {
assert.equal(s.unpricedRuns, 1); // run-level: only the both-stages run
});
test("the embedding stage (wave 6): folds when present, absent when the run embedded nothing", () => {
const EMBED_MODEL = "openai:text-embedding-3-small";
const embStep = callCost(step(5000, 0), EMBED_MODEL)!;
assert.ok(embStep > 0, "the overlay prices the known embedding models");
const withEmbed = extractionRun(100, PRICED, PRICED);
withEmbed.stages.embedding = { model: EMBED_MODEL, steps: [step(5000, 0)] };
const s = graphCostSummary({
chats: [{ title: "a", extractions: [withEmbed] }, { title: "b", extractions: [extractionRun(200, PRICED, PRICED)] }],
sources: [],
naming: [],
});
assert.equal(s.stages.embedding.runs, 1, "only the run that embedded");
assert.equal(s.stages.embedding.dollars, embStep);
assert.equal(s.stages.embedding.tokens.inputTokens, 5000);
assert.equal(s.stages.extraction.runs, 2, "the two-stage era still folds");
assert.equal(s.stages.extraction.runs, 2);
// Run-level: the embedding dollars ride the run's total.
const extStep = callCost(step(1000, 200), PRICED)!;
const audStep = callCost(step(2000, 100), PRICED)!;
assert.equal(s.runs.find((r) => r.startedAt === 100)!.dollars, extStep + audStep + embStep);
assert.equal(s.runs.find((r) => r.startedAt === 200)!.dollars, extStep + audStep);
assert.equal(s.totalDollars, (extStep + audStep) * 2 + embStep);
});
test("an UNPRICED embedding model stays honest: counted, never a silent zero; a priced sibling saves the run total", () => {
const unpricedEmbed = extractionRun(100, PRICED, "unknown-model");
unpricedEmbed.stages.embedding = { model: "some/new-embed", steps: [step(900, 0)] };
const s = graphCostSummary({ chats: [{ title: "x", extractions: [unpricedEmbed] }], sources: [], naming: [] });
assert.equal(s.stages.embedding.unpricedRuns, 1);
assert.equal(s.stages.embedding.dollars, 0);
assert.notEqual(s.runs[0]!.dollars, null, "the priced extractor saves the run total");
});
test("recent runs cap: newest survive", () => {
const extractions = [1, 2, 3, 4].map((n) => extractionRun(n * 100, PRICED, PRICED));
const s = graphCostSummary({
tests/embedding-settings.test.tsx
-91
View File
@@ -1,91 +0,0 @@
import React from "react";
(globalThis as Record<string, unknown>).React = React;
import { renderToStaticMarkup } from "react-dom/server";
import { describe, it } from "node:test";
import assert from "node:assert/strict";
import {
embeddingPriceLabel,
GraphSection,
needsEmbeddingModel,
} from "../components/settings/appearance-section.tsx";
import { EMBEDDING_PRICES, type GraphSettings } from "../src/core/index.ts";
/**
* Wave 6 chunk B — the embeddings settings seams: the price line for
* catalog rows (EMBEDDING_PRICES lookup, honestly "—" when unlisted),
* the model-dependency predicate that drives the ladder gray-out, and
* GraphSection's SSR shape with and without a configured model.
*
* The fetches (catalog, node count) live in effects — dead under
* renderToStaticMarkup, so the render pins the STORED-state markup only;
* the interactive round-trips are live-verified on the dev server.
*/
describe("embeddingPriceLabel", () => {
it("prices a listed model: USD per 1M input tokens", () => {
// Straight from the hand-maintained table (openai small = $0.02).
assert.equal(embeddingPriceLabel("openai:text-embedding-3-small"), "$0.02/1M in");
});
it("prices every entry in EMBEDDING_PRICES without zero or 'NaN' leaking", () => {
for (const id of Object.keys(EMBEDDING_PRICES)) {
const label = embeddingPriceLabel(id);
assert.match(label, /^\$\d+(\.\d+)?\/1M in$/);
assert.notEqual(label, "$0/1M in");
}
});
it("trims a trailing zero the model-menu way ($0.2, not $0.20)", () => {
assert.equal(embeddingPriceLabel("google:gemini-embedding-2"), "$0.2/1M in");
});
it("stays honestly '—' for unlisted models (never zero)", () => {
assert.equal(embeddingPriceLabel("openrouter:some/model-not-in-table"), "—");
});
});
describe("needsEmbeddingModel", () => {
it("spares only the master switch", () => {
assert.equal(needsEmbeddingModel("enabled"), false);
assert.equal(needsEmbeddingModel("embedding"), true);
assert.equal(needsEmbeddingModel("clustering"), true);
assert.equal(needsEmbeddingModel("labeling"), true);
});
});
describe("GraphSection (SSR)", () => {
const render = (graph: GraphSettings | undefined) =>
renderToStaticMarkup(<GraphSection graph={graph} onSectionChange={() => {}} />);
it("grays out + hints the model-dependent rungs when no model is set", () => {
const html = render({ enabled: true } as GraphSettings);
// Master switch: never model-dimmed.
const enabledRow = html.split('data-testid="graph-row-enabled"')[1]?.split("</div>")[0] ?? "";
assert.ok(!enabledRow.includes("opacity-40"), "master switch must not dim on unset model");
// The three dependent rungs dim and each carries the hint.
for (const key of ["embedding", "clustering", "labeling"] as const) {
const row = html.split(`data-testid="graph-row-${key}"`)[1]?.split('data-testid="graph-row-')[0] ?? "";
assert.ok(row.includes("opacity-40"), `${key} row dims without a model`);
assert.ok(row.includes("Needs an embedding model"), `${key} row carries the hint`);
}
assert.ok(html.includes('data-testid="graph-row-embedding-model"'));
});
it("keeps the ladder touchable once a model is set", () => {
const html = render({
enabled: true,
embeddingModel: "openai:text-embedding-3-small",
} as GraphSettings);
for (const key of ["embedding", "clustering", "labeling"] as const) {
const row = html.split(`data-testid="graph-row-${key}"`)[1]?.split('data-testid="graph-row-')[0] ?? "";
assert.ok(!row.includes("opacity-40"), `${key} row stays lit with a model`);
assert.ok(!row.includes("Needs an embedding model"), "no hint with a model");
}
});
it("renders the picker row (catalog fetch is an effect — inert in SSR)", () => {
const html = render(undefined);
assert.ok(html.includes("Loading…"), "pre-fetch trigger state renders");
});
});
tests/embeddings.test.ts
-474
View File
@@ -1,474 +0,0 @@
/**
* BYOK provider embeddings (wave 6, 2026-09-23): the adapters' wire
* protocols (injected fetch — never the network), the slug/keys
* resolution rules (null = honestly off; a set model with a missing key
* is an honest error), the post-merge batch's staleness/re-embed/skip
* semantics (scripted port — never the machine's settings/auth files),
* and the federated catalog's federation rules.
*/
import assert from "node:assert/strict";
import { describe, it } from "node:test";
import { callCost, EMBEDDING_PRICES } from "../src/core/prices.ts";
import {
embedNewNodes,
embeddingModelCatalog,
googleEmbeddingProvider,
openaiEmbeddingProvider,
openrouterEmbeddingProvider,
parseEmbeddingModelSlug,
resolveEmbeddingPortFor,
type EmbeddingResolution,
} from "../src/server/embeddings.ts";
import type { EmbeddingPort, EmbeddingTask } from "../src/core/ports.ts";
import type { EmbeddingsLedger, GraphNode } from "../src/core/contracts/graph.ts";
// ─── scripted fetch + in-memory graph (no fs, no network) ────────────────────
type RecordedCall = { url: string; init: RequestInit };
function jsonResponse(body: unknown, status = 200): Response {
return new Response(JSON.stringify(body), {
status,
headers: { "content-type": "application/json" },
});
}
function recordingFetch(
route: (url: string, body: unknown) => Response | Promise<Response>,
): { calls: RecordedCall[]; fetch: (input: string, init: RequestInit) => Promise<Response> } {
const calls: RecordedCall[] = [];
const fetchFn = async (input: string, init: RequestInit): Promise<Response> => {
calls.push({ url: input, init });
const body = init.body ? (JSON.parse(String(init.body)) as unknown) : null;
return route(input, body);
};
return { calls, fetch: fetchFn };
}
function node(id: string, statement = `statement of ${id}`): GraphNode {
return { id, name: id, statement, aliases: [], provenance: [], createdAt: "2026-09-23T00:00:00.000Z" };
}
function fakeGraphStore(start?: Partial<EmbeddingsLedger>): {
ledger: EmbeddingsLedger;
writes: EmbeddingsLedger[];
store: {
readNodes(): GraphNode[];
readEmbeddings(): EmbeddingsLedger;
writeEmbeddings(ledger: EmbeddingsLedger): void;
};
nodes: GraphNode[];
} {
const nodes = [node("n1", "alpha fact"), node("n2", "beta fact")];
const ledger: EmbeddingsLedger = {
model: start?.model ?? "test-embed",
dim: start?.dim ?? 2,
vectors: { ...start?.vectors },
clusters: [...start?.clusters ?? []],
updatedAt: start?.updatedAt ?? "2026-09-23T00:00:00.000Z",
};
const writes: EmbeddingsLedger[] = [];
return {
ledger,
writes,
nodes,
store: {
readNodes: () => nodes,
readEmbeddings: () => ledger,
writeEmbeddings: (next) => {
writes.push(next);
Object.assign(ledger, next);
},
},
};
}
/** A scripted port: unit vectors by text hash, records (texts, task) calls. */
function scriptedPort(model = "openai:test-embed", dim = 2, tokensPerCall = 120): {
calls: Array<{ texts: string[]; task: EmbeddingTask }>;
port: EmbeddingPort;
} {
const calls: Array<{ texts: string[]; task: EmbeddingTask }> = [];
return {
calls,
port: {
model,
async embed(texts, task) {
calls.push({ texts: [...texts], task });
return {
vectors: texts.map((_, i) => Array.from({ length: dim }, (_, j) => (j === i ? 1 : 0))),
model,
dim,
promptTokens: tokensPerCall,
};
},
},
};
}
const okPort = (port: EmbeddingPort): EmbeddingResolution => ({ ok: true, port });
// ─── usage-on-error + pay-once (owner ruling 2026-09-23, interlude) ──────────
describe("embedNewNodes partial-failure accounting (Teacher review follow-up)", () => {
it("a mid-run batch failure: landed vectors persist, usage survives, failed carries the error, next run resumes", async () => {
// 40 nodes → batch 1 (32) lands, batch 2 (8) throws.
const many: GraphNode[] = Array.from({ length: 40 }, (_, i) => node(`n${i}`, `statement ${i}`));
const ledger: EmbeddingsLedger = { model: "m", dim: 2, vectors: {}, clusters: [], updatedAt: "" };
let writes = 0;
let call = 0;
const failing: EmbeddingPort = {
model: "m",
async embed(texts) {
call++;
if (call === 2) throw new Error("provider exploded mid-run");
return {
vectors: texts.map((_, i) => (i === 0 ? [1, 0] : [0, 1])),
model: "m",
dim: 2,
promptTokens: 100,
};
},
};
const store = {
readNodes: () => many,
readEmbeddings: () => ledger,
writeEmbeddings: (next: EmbeddingsLedger) => {
writes++;
Object.assign(ledger, next);
},
};
const result = await embedNewNodes(store, () => okPort(failing));
// Usage-on-error: batch 1 billed — the money channel keeps it.
assert.equal(result.failed, "provider exploded mid-run");
assert.equal(result.embedded, 32, "only the landed batch counts as embedded");
assert.equal(result.usage?.steps.length, 1);
assert.equal(result.usage?.steps[0]?.inputTokens, 100);
// Pay-once: the landed vectors are persisted — a retry embeds only the rest.
assert.equal(writes, 1);
assert.equal(Object.keys(ledger.vectors).length, 32);
assert.equal(ledger.model, "m");
const resume = await embedNewNodes(store, () => okPort(scriptedPort("m", 2).port));
assert.equal(resume.embedded, 8, "the retry resumes — it re-embeds only the 8 remaining");
assert.equal(resume.failed, undefined);
assert.equal(Object.keys(ledger.vectors).length, 40);
assert.equal(resume.usage?.steps[0]?.inputTokens, 120, "scripted port tokens — the retry billed once");
});
it("a FIRST-batch failure: nothing written, nothing billed, the old ledger stays intact", async () => {
const g = fakeGraphStore({ vectors: { n1: [1, 0] } });
const alwaysFails: EmbeddingPort = {
model: "m",
embed: async () => {
throw new Error("total outage");
},
};
const result = await embedNewNodes(g.store, () => okPort(alwaysFails));
assert.equal(result.failed, "total outage");
assert.equal(result.embedded, 0);
assert.equal(result.usage, undefined, "no call landed — nothing to bill");
assert.equal(g.writes.length, 0, "the ledger is untouched (its own model/dim still true)");
assert.ok(g.ledger.vectors.n1, "the old vector survives a zero-landed failure");
});
it("success still carries no `failed` key and counts every todo node", async () => {
const g = fakeGraphStore();
const scripted = scriptedPort(g.ledger.model, 2);
const result = await embedNewNodes(g.store, () => okPort(scripted.port));
assert.equal(result.failed, undefined);
assert.equal(result.embedded, 2);
assert.equal(result.skipped, undefined);
});
});
// ─── the money channel's embedding overlay ─────────────────────────────────
describe("the embedding price overlay (hand-maintained, outside the sync block)", () => {
it("prices the known embedding models at their verified input rates", () => {
// USD per 1M input tokens, verified against provider pages 2026-09-23.
assert.equal(callCost({ inputTokens: 1_000_000, outputTokens: 0 }, "openai:text-embedding-3-small"), 0.02);
assert.equal(callCost({ inputTokens: 1_000_000, outputTokens: 0 }, "openai:text-embedding-3-large"), 0.13);
assert.equal(callCost({ inputTokens: 1_000_000, outputTokens: 0 }, "google:gemini-embedding-001"), 0.15);
assert.equal(callCost({ inputTokens: 1_000_000, outputTokens: 0 }, "google:gemini-embedding-2"), 0.2);
assert.equal(callCost({ inputTokens: 1_000_000, outputTokens: 0 }, "openrouter:openai/text-embedding-3-small"), 0.02);
});
it("unknown embedding models stay unpriced (—, never zero), tokens still captured", () => {
assert.equal(callCost({ inputTokens: 100, outputTokens: 0 }, "openrouter:mystery/embed"), null);
assert.equal(!("openrouter:mystery/embed" in EMBEDDING_PRICES), true);
});
});
// ─── the slug grammar ─────────────────────────────────────────────────────────
describe("parseEmbeddingModelSlug", () => {
it("parses the three shipped providers", () => {
assert.deepEqual(parseEmbeddingModelSlug("openai:text-embedding-3-small"), { provider: "openai", id: "text-embedding-3-small" });
assert.deepEqual(parseEmbeddingModelSlug("openrouter:openai/text-embedding-3-large"), { provider: "openrouter", id: "openai/text-embedding-3-large" });
assert.deepEqual(parseEmbeddingModelSlug("google:gemini-embedding-001"), { provider: "google", id: "gemini-embedding-001" });
// OpenRouter ids may carry a :free suffix (live catalog fact) — the
// provider is the part before the FIRST colon.
assert.deepEqual(parseEmbeddingModelSlug("openrouter:liquid/lfm-2.5-embedding-350m:free"), { provider: "openrouter", id: "liquid/lfm-2.5-embedding-350m:free" });
});
it("rejects malformed slugs (no guessing)", () => {
assert.equal(parseEmbeddingModelSlug("text-embedding-3-small"), null, "no provider prefix");
assert.equal(parseEmbeddingModelSlug("anthropic:embed"), null, "unshipped provider");
assert.equal(parseEmbeddingModelSlug("openai:"), null, "empty id");
assert.equal(parseEmbeddingModelSlug(":id"), null, "empty provider");
});
});
// ─── resolution: BYOK honesty rules ───────────────────────────────────────────
describe("resolveEmbeddingPortFor", () => {
it("null/absent setting = honestly off", () => {
for (const slug of [null, undefined, ""]) {
const res = resolveEmbeddingPortFor(slug, () => "key");
assert.equal(res.ok, false);
assert.match((res as { reason: string }).reason, /no embedding model configured/);
}
});
it("a set model with no key is an honest error naming the key", () => {
const res = resolveEmbeddingPortFor("openai:text-embedding-3-small", () => undefined);
assert.equal(res.ok, false);
assert.match((res as { reason: string }).reason, /OPENAI_API_KEY/);
});
it("an invalid slug never resolves", () => {
const res = resolveEmbeddingPortFor("bedrock:whatever", () => "key");
assert.equal(res.ok, false);
assert.match((res as { reason: string }).reason, /invalid embedding model slug/);
});
it("a keyed model resolves to its provider adapter", () => {
const res = resolveEmbeddingPortFor("openai:text-embedding-3-small", () => "sk-test");
assert.equal(res.ok, true);
assert.equal((res as { port: EmbeddingPort }).port.model, "openai:text-embedding-3-small");
});
});
// ─── the adapters' wire protocols (injected fetch) ────────────────────────────
describe("openrouterEmbeddingProvider", () => {
it("POSTs /api/v1/embeddings with Bearer auth and the task input_type", async () => {
const { calls, fetch } = recordingFetch(() =>
jsonResponse({ data: [{ embedding: [1, 0], index: 0 }], usage: { prompt_tokens: 42 } }),
);
const port = openrouterEmbeddingProvider("openrouter:openai/text-embedding-3-small", "k", fetch);
const result = await port.embed(["hello"], "document");
assert.equal(calls.length, 1);
assert.equal(calls[0]!.url, "https://openrouter.ai/api/v1/embeddings");
assert.equal((calls[0]!.init.headers as Record<string, string>).authorization, "Bearer k");
assert.deepEqual(JSON.parse(String(calls[0]!.init.body)), {
model: "openai/text-embedding-3-small",
input: ["hello"],
input_type: "search_document",
});
assert.deepEqual(result.vectors, [[1, 0]]);
assert.equal(result.promptTokens, 42);
await port.embed(["q"], "query");
assert.equal((JSON.parse(String(calls[1]!.init.body)) as { input_type: string }).input_type, "search_query");
});
});
describe("openaiEmbeddingProvider", () => {
it("POSTs /v1/embeddings, Bearer auth, no task hint; index ordering honored", async () => {
const { calls, fetch } = recordingFetch(() =>
jsonResponse({
data: [{ embedding: [0, 1], index: 1 }, { embedding: [1, 0], index: 0 }],
usage: { prompt_tokens: 7, total_tokens: 7 },
}),
);
const port = openaiEmbeddingProvider("openai:text-embedding-3-small", "sk", fetch);
const result = await port.embed(["a", "b"], "document");
assert.equal(calls[0]!.url, "https://api.openai.com/v1/embeddings");
assert.equal((calls[0]!.init.headers as Record<string, string>).authorization, "Bearer sk");
const body = JSON.parse(String(calls[0]!.init.body)) as Record<string, unknown>;
assert.deepEqual(body.input, ["a", "b"]);
assert.equal("input_type" in body, false, "OpenAI takes no task hint");
// Rows arrive out of order — the adapter must restore input order.
assert.deepEqual(result.vectors, [[1, 0], [0, 1]]);
assert.equal(result.promptTokens, 7);
});
});
describe("googleEmbeddingProvider", () => {
it("POSTs :batchEmbedContents with x-goog-api-key and per-text taskType", async () => {
const { calls, fetch } = recordingFetch((_url, body) => {
const n = ((body as { requests?: unknown[] } | null)?.requests ?? []).length;
return jsonResponse({ embeddings: Array.from({ length: n }, (_, i) => ({ values: i === 0 ? [1, 0] : [0, 1] })) });
});
const port = googleEmbeddingProvider("google:gemini-embedding-001", "gk", fetch);
const result = await port.embed(["a", "b"], "document");
assert.equal(calls[0]!.url, "https://generativelanguage.googleapis.com/v1beta/models/gemini-embedding-001:batchEmbedContents");
assert.equal((calls[0]!.init.headers as Record<string, string>)["x-goog-api-key"], "gk");
assert.deepEqual(JSON.parse(String(calls[0]!.init.body)), {
requests: [
{ model: "models/gemini-embedding-001", content: { parts: [{ text: "a" }] }, taskType: "RETRIEVAL_DOCUMENT" },
{ model: "models/gemini-embedding-001", content: { parts: [{ text: "b" }] }, taskType: "RETRIEVAL_DOCUMENT" },
],
});
assert.deepEqual(result.vectors, [[1, 0], [0, 1]]);
assert.equal(result.promptTokens, undefined, "batch responses carry no token count we can trust");
await port.embed(["q"], "query");
const second = JSON.parse(String(calls[1]!.init.body)) as { requests: Array<{ taskType: string }> };
assert.equal(second.requests[0]!.taskType, "RETRIEVAL_QUERY");
});
it("a short/missing vector is an honest error, never a silent misalignment", async () => {
const { fetch } = recordingFetch(() => jsonResponse({ embeddings: [{ values: [1, 0] }] }));
const port = googleEmbeddingProvider("google:gemini-embedding-001", "gk", fetch);
await assert.rejects(port.embed(["a", "b"], "document"), /1\/2 vectors/);
});
});
describe("adapter error mapping (all providers)", () => {
it("HTTP failures become clear Errors carrying the status", async () => {
for (const factory of [openrouterEmbeddingProvider, openaiEmbeddingProvider, googleEmbeddingProvider]) {
const { fetch } = recordingFetch(() => jsonResponse({ error: { message: "bad key" } }, 401));
const port = factory("openai:text-embedding-3-small", "k", fetch);
await assert.rejects(port.embed(["x"], "document"), /API 401/);
}
});
});
// ─── the post-merge batch (scripted port — the machine's files stay unread) ───
describe("embedNewNodes (the extraction tail)", () => {
it("embeds missing nodes, stamps model/dim, records one usage step per API call", async () => {
const g = fakeGraphStore();
const scripted = scriptedPort("openai:text-embedding-3-small", 2, 120);
const result = await embedNewNodes(g.store, () => okPort(scripted.port));
assert.deepEqual(result, {
embedded: 2,
total: 2,
usage: {
model: "openai:text-embedding-3-small",
steps: [
{ inputTokens: 120, outputTokens: 0, cacheReadTokens: undefined, cacheWriteTokens: undefined, reasoningTokens: undefined },
],
},
});
assert.equal(g.writes.length, 1, "the ledger is written once per batch");
assert.equal(g.ledger.model, "openai:text-embedding-3-small");
assert.equal(g.ledger.dim, 2);
assert.ok(g.ledger.vectors.n1 && g.ledger.vectors.n2);
assert.deepEqual(scripted.calls[0]!.texts, ["alpha fact", "beta fact"]);
assert.equal(scripted.calls[0]!.task, "document");
assert.equal(result.usage?.steps.length, 1);
});
it("a MODEL CHANGE voids the ledger and re-embeds everything", async () => {
// The ledger holds a stale-model vector for a node that still exists.
const g = fakeGraphStore({ model: "openai:old-embed", vectors: { n1: [9, 9] } });
const scripted = scriptedPort("openai:new-embed", 2);
const result = await embedNewNodes(g.store, () => okPort(scripted.port));
assert.equal(result.embedded, 2, "the stale vector dies — BOTH nodes re-embed");
assert.deepEqual(scripted.calls[0]!.texts, ["alpha fact", "beta fact"]);
assert.equal(g.ledger.model, "openai:new-embed");
assert.ok(!g.ledger.vectors.n1!.every((x, i) => x === [9, 9][i]), "the old vector was replaced");
});
it("null model = info skip: nothing written, nothing embedded, the reason rides the result", async () => {
const g = fakeGraphStore({ vectors: { n1: [1, 0] } });
const result = await embedNewNodes(g.store, () => ({
ok: false,
reason: "no embedding model configured (graph.embeddingModel is null)",
}));
assert.deepEqual(result, { embedded: 0, total: 1, skipped: "no embedding model configured (graph.embeddingModel is null)" });
assert.equal(g.writes.length, 0, "a skipped batch never touches the ledger");
});
it("a set model with a missing key is the same skip path (its reason says so)", async () => {
const g = fakeGraphStore();
const result = await embedNewNodes(g.store, () => ({
ok: false,
reason: 'embedding model is set to "openai:x" but its OPENAI_API_KEY key is missing',
}));
assert.match(result.skipped ?? "", /OPENAI_API_KEY/);
});
it("prunes ghost vectors for nodes that no longer exist", async () => {
const g = fakeGraphStore({ vectors: { n1: [1, 0], ghost: [0, 1] } });
const scripted = scriptedPort(g.ledger.model, 2);
const result = await embedNewNodes(g.store, () => okPort(scripted.port));
assert.equal(result.embedded, 1, "n1 already has a vector; only n2 embeds");
assert.ok(!("ghost" in g.ledger.vectors), "the ghost vector is pruned");
});
it("batches at 32 (a 40-node backfill = two API calls)", async () => {
const many: GraphNode[] = Array.from({ length: 40 }, (_, i) => node(`n${i}`, `statement ${i}`));
const ledger: EmbeddingsLedger = { model: "m", dim: 2, vectors: {}, clusters: [], updatedAt: "" };
const writes: EmbeddingsLedger[] = [];
const scripted = scriptedPort("m", 2);
const result = await embedNewNodes(
{
readNodes: () => many,
readEmbeddings: () => ledger,
writeEmbeddings: (next) => writes.push(next),
},
() => okPort(scripted.port),
);
assert.equal(result.embedded, 40);
assert.equal(scripted.calls.length, 2);
assert.equal(scripted.calls[0]!.texts.length, 32);
assert.equal(scripted.calls[1]!.texts.length, 8);
assert.equal(writes.length, 1);
});
});
// ─── the federated catalog (scripted keys + fetch) ────────────────────────────
describe("embeddingModelCatalog", () => {
it("no keys = empty catalog", async () => {
const models = await embeddingModelCatalog(
recordingFetch(() => jsonResponse({})).fetch,
() => undefined,
);
assert.deepEqual(models, []);
});
it("federates every keyed provider with its filter rules", async () => {
const { fetch, calls } = recordingFetch((url) => {
if (url.includes("openrouter.ai/api/v1/embeddings/models")) {
return jsonResponse({ data: [{ id: "openai/text-embedding-3-small", name: "Text Embedding 3 Small" }, { id: "other/embed" }] });
}
if (url.includes("api.openai.com")) {
return jsonResponse({ data: [{ id: "text-embedding-3-large" }, { id: "text-embedding-3-small" }, { id: "gpt-6-sol" }] });
}
if (url.includes("generativelanguage")) {
return jsonResponse({
models: [
{ name: "models/gemini-embedding-001", displayName: "Gemini Embedding", supportedGenerationMethods: ["embedContent", "generateContent"] },
{ name: "models/gemini-6-pro", supportedGenerationMethods: ["generateContent"] },
],
});
}
return jsonResponse({});
});
const models = await embeddingModelCatalog(fetch, (id) => (id === "OPENROUTER_API_KEY" || id === "OPENAI_API_KEY" || id === "GEMINI_API_KEY" ? "k" : undefined));
assert.equal(calls[0]!.url, "https://openrouter.ai/api/v1/embeddings/models?limit=500", "the real catalog path (the inverse 404s — verified live)");
assert.deepEqual(models, [
{ id: "openrouter:openai/text-embedding-3-small", name: "Text Embedding 3 Small" },
{ id: "openrouter:other/embed", name: "other/embed" },
{ id: "openai:text-embedding-3-large", name: "text-embedding-3-large" },
{ id: "openai:text-embedding-3-small", name: "text-embedding-3-small" },
{ id: "google:gemini-embedding-001", name: "Gemini Embedding" },
]);
});
it("a failing provider never poisons the rest", async () => {
const { fetch } = recordingFetch((url) => {
if (url.includes("openrouter.ai")) return jsonResponse({}, 500); // their side broke
if (url.includes("api.openai.com")) return jsonResponse({ data: [{ id: "text-embedding-3-small" }] });
return jsonResponse({});
});
const models = await embeddingModelCatalog(fetch, (id) => (id === "OPENROUTER_API_KEY" || id === "OPENAI_API_KEY" ? "k" : undefined));
assert.deepEqual(models, [{ id: "openai:text-embedding-3-small", name: "text-embedding-3-small" }]);
});
});
tests/render-settings.test.tsx
+16 -29
View File
@@ -134,37 +134,24 @@ describe("AgentSection (thin SSR)", () => {
});
});
describe("GraphSection — the dependency ladder", () => {
// Wave 6: without an embedding model the three dependent rungs are
// inert (dimmed, disabled, hinted) — pinned in embedding-settings.
// These ladder pins isolate the CHAIN logic with a model configured.
const MODEL: GraphSettings = { enabled: true, embeddingModel: "openai:text-embedding-3-small" };
it("defaults with a model set: all four rungs on, none disabled", () => {
const html = renderToStaticMarkup(<GraphSection graph={MODEL} onSectionChange={() => {}} />);
for (const key of ["enabled", "embedding", "clustering", "labeling"]) {
assert.ok(html.includes(`data-testid="graph-row-${key}"`), key);
assert.ok(!tagOf(html, `toggle-graph-${key}`).includes("disabled"), `${key} touchable`);
}
describe("GraphSection — the collapsed master switch (wave 7)", () => {
// Wave 7 (owner ruling 2026-09-23): the section is the master switch
// ONLY — the embedding/clustering/labeling rungs and the BYOK model
// picker are gone (clustering is the free birth-session partition,
// labels ARE session names). One row, one toggle.
it("renders the single master-switch row", () => {
const html = renderToStaticMarkup(<GraphSection graph={undefined} onSectionChange={() => {}} />);
assert.ok(html.includes('data-testid="graph-row-enabled"'));
assert.ok(!html.includes("graph-row-embedding"), "the dead rungs render nothing");
assert.ok(!html.includes("Embedding model"), "the picker is gone");
assert.ok(!html.includes("graph-row-clustering"));
assert.ok(!html.includes("graph-row-labeling"));
assert.ok(tagOf(html, "toggle-graph-enabled").includes('aria-checked="true"'), "absent record = factory ON");
});
it("master off grays EVERY dependent (stored values persist, resolve inert)", () => {
const html = renderToStaticMarkup(
<GraphSection graph={{ ...MODEL, enabled: false, embedding: true, clustering: true, labeling: true }} onSectionChange={() => {}} />,
);
for (const key of ["embedding", "clustering", "labeling"]) {
assert.ok(tagOf(html, `toggle-graph-${key}`).includes("disabled"), `${key} disabled`);
}
assert.ok(!tagOf(html, "toggle-graph-enabled").includes("disabled"), "master itself stays touchable");
});
it("a mid-ladder break grays only what's below it", () => {
const html = renderToStaticMarkup(
<GraphSection graph={{ ...MODEL, embedding: true, clustering: false }} onSectionChange={() => {}} />,
);
assert.ok(!tagOf(html, "toggle-graph-embedding").includes("disabled"), "embedding touchable");
assert.ok(!tagOf(html, "toggle-graph-clustering").includes("disabled"), "clustering touchable");
assert.ok(tagOf(html, "toggle-graph-labeling").includes("disabled"), "labeling grayed");
it("a stored off record shows the switch off", () => {
const html = renderToStaticMarkup(<GraphSection graph={{ enabled: false }} onSectionChange={() => {}} />);
assert.ok(tagOf(html, "toggle-graph-enabled").includes('aria-checked="false"'));
});
});
tests/settings-body.test.ts
+9 -31
View File
@@ -150,47 +150,25 @@ describe("parseAppearanceSettingsBody", () => {
});
describe("parseGraphSettingsBody", () => {
it("parses the four-rung ladder replace-whole", () => {
const r = parseGraphSettingsBody({ enabled: false, embedding: false });
it("parses the collapsed section replace-whole (wave 7: the master switch ONLY)", () => {
const r = parseGraphSettingsBody({ enabled: false });
assert.ok(r.ok);
assert.deepEqual(r.settings, { enabled: false, embedding: false });
assert.deepEqual(r.settings, { enabled: false });
assert.equal(parseGraphSettingsBody({}).ok, true);
});
it("rejects non-boolean rungs and unknown bodies", () => {
it("rejects non-boolean switches and unknown bodies", () => {
assert.equal(parseGraphSettingsBody({ enabled: "off" }).ok, false);
assert.equal(parseGraphSettingsBody("nope").ok, false);
});
it("the ladder DEPENDENCY (graph → embedding → clustering → labeling) is a UI concern, not a parse concern", () => {
// Nothing stops a stored {enabled: false, labeling: true} — the
// values persist but resolve inert (grays out in the panel).
it("the dead wave-6 rungs in a body are ignored — the section collapses on write", () => {
// Old clients may still send {embedding: true, embeddingModel: …};
// the parser reads ONLY the master switch — the dead keys are not
// an error, they are just not part of the section anymore.
const r = parseGraphSettingsBody({ enabled: false, clustering: true, labeling: true });
assert.ok(r.ok);
});
it("embeddingModel (wave 6): null or a shipped-provider slug; anything else is a 400", () => {
const nulled = parseGraphSettingsBody({ embeddingModel: null });
assert.ok(nulled.ok);
assert.deepEqual(nulled.settings, { embeddingModel: null });
for (const slug of [
"openai:text-embedding-3-small",
"openrouter:openai/text-embedding-3-large",
"google:gemini-embedding-001",
// OpenRouter ids may carry a :free-style suffix — the provider is
// the part before the FIRST colon (live catalog fact, 2026-09-23).
"openrouter:liquid/lfm-2.5-embedding-350m:free",
]) {
const r = parseGraphSettingsBody({ embeddingModel: slug });
assert.ok(r.ok, slug);
assert.deepEqual(r.settings, { embeddingModel: slug });
}
// Bad shapes never become a live model.
assert.equal(parseGraphSettingsBody({ embeddingModel: "text-embedding-3-small" }).ok, false, "no provider prefix");
assert.equal(parseGraphSettingsBody({ embeddingModel: "bedrock:whatever" }).ok, false, "unshipped provider");
assert.equal(parseGraphSettingsBody({ embeddingModel: "openai:" }).ok, false, "empty id");
assert.equal(parseGraphSettingsBody({ embeddingModel: 42 }).ok, false, "non-string");
assert.equal(parseGraphSettingsBody({ embeddingModel: "openai:has spaces" }).ok, false, "whitespace id");
assert.deepEqual(r.settings, { enabled: false });
});
});
tests/settings-resolution.test.ts
+8 -19
View File
@@ -11,7 +11,6 @@ import assert from "node:assert/strict";
import { describe, it } from "node:test";
import {
extractionEnabled,
graphContextEnabled,
graphToolsEnabled,
resolveConversationSpec,
resolveModelSpec,
@@ -43,7 +42,7 @@ describe("resolveSettings", () => {
assert.equal(eff.appearance.proseFontSize, 16);
assert.equal(eff.appearance.sessionNameSerif, true);
assert.equal(eff.appearance.mdRendering.headingWeight, 900);
assert.deepEqual(eff.graph, { enabled: true, embedding: true, clustering: true, labeling: true, embeddingModel: null }); // BYOK wave 6: no factory embedding model
assert.deepEqual(eff.graph, { enabled: true }); // wave 7 collapse: the master switch ONLY
});
it("explicit record values win over every factory default", () => {
@@ -76,9 +75,8 @@ describe("resolveSettings", () => {
assert.equal(eff.appearance.mode, "light");
assert.equal(eff.appearance.proseFontSize, 20);
assert.equal(eff.appearance.mdRendering.headingWeight, 600);
// graph: enabled=false explicit; absent rungs stay factory-on (stored
// but inert — the ladder grays them out in the UI).
assert.deepEqual(eff.graph, { enabled: false, embedding: true, clustering: true, labeling: true, embeddingModel: null }); // wave 6: unset embeddingModel resolves to the null floor
// wave 7 collapse: the section IS the master switch.
assert.deepEqual(eff.graph, { enabled: false });
});
it("partial role override: thinkingLevel alone borrows the SLOT's model (or nothing, without one)", () => {
@@ -224,29 +222,20 @@ describe("resolveConversationSpec (the turn gate's chain)", () => {
});
describe("the graph channel gates (the wave's seam honoring)", () => {
it("WRITE channel (extraction): dies with the master switch only", () => {
assert.equal(extractionEnabled({ enabled: true, embedding: true, clustering: true, labeling: true, embeddingModel: null }), true);
assert.equal(extractionEnabled({ enabled: true, embedding: false, clustering: false, labeling: false, embeddingModel: null }), true);
assert.equal(extractionEnabled({ enabled: false, embedding: true, clustering: true, labeling: true, embeddingModel: null }), false);
it("WRITE channel (extraction): dies with the master switch — the ONLY graph setting since wave 7", () => {
assert.equal(extractionEnabled({ enabled: true }), true);
assert.equal(extractionEnabled({ enabled: false }), false);
});
it("READ channel (graph-context): needs master AND the embedding rung", () => {
assert.equal(graphContextEnabled({ enabled: true, embedding: true, clustering: false, labeling: false, embeddingModel: null }), true);
assert.equal(graphContextEnabled({ enabled: true, embedding: false, clustering: true, labeling: true, embeddingModel: null }), false);
assert.equal(graphContextEnabled({ enabled: false, embedding: true, clustering: true, labeling: true, embeddingModel: null }), false);
});
it("both gates ride resolveSettings' factory defaults (all on → both open)", () => {
it("the write gate rides resolveSettings' factory defaults (all on → open)", () => {
const eff = resolveSettings(null);
assert.equal(extractionEnabled(eff.graph), true);
assert.equal(graphContextEnabled(eff.graph), true);
const off = resolveSettings({ version: 1, graph: { enabled: false } });
assert.equal(extractionEnabled(off.graph), false);
assert.equal(graphContextEnabled(off.graph), false);
});
it("PROBER GRAPH TOOLS gate (wave 7, owner ruling 2026-09-23): the agent toggle AND the graph master switch compose", () => {
const graph = { enabled: true, embedding: true, clustering: true, labeling: true, embeddingModel: null };
const graph = { enabled: true };
assert.equal(graphToolsEnabled({ graph, agent: { graphTools: {} } }), true, "absent subsection = ON");
assert.equal(graphToolsEnabled({ graph, agent: { graphTools: { enabled: true } } }), true);
assert.equal(graphToolsEnabled({ graph, agent: { graphTools: { enabled: false } } }), false);
tests/settings.test.ts
+2 -2
View File
@@ -21,7 +21,7 @@ describe("FACTORY_SETTINGS (the compiled factory layer)", () => {
it("pins the owner-decided defaults", () => {
assert.equal(CURRENT_SETTINGS_VERSION, 1);
assert.deepEqual(FACTORY_SETTINGS.general, { autoUpdate: true });
assert.deepEqual(FACTORY_SETTINGS.graph, { enabled: true, embedding: true, clustering: true, labeling: true, embeddingModel: null }); // BYOK wave 6: no factory embedding model
assert.deepEqual(FACTORY_SETTINGS.graph, { enabled: true }); // wave 7 collapse: the master switch ONLY
const a = FACTORY_SETTINGS.appearance;
assert.equal(a.mode, "dark");
assert.equal(a.background, null); // the mode's own default, not a factory hex
@@ -75,7 +75,7 @@ describe("sanitizeUserSettings", () => {
sessionNameSerif: false,
mdRendering: { headingWeight: 700, boldColor: "#ff8800", italicColor: null },
},
graph: { enabled: false, embedding: true, clustering: false, labeling: true },
graph: { enabled: false },
};
assert.deepEqual(sanitizeUserSettings(record), record);
});