Wave 7 close-out: the embedding ledger seam dies; notes birth via mapping lines

Part 1 — the ledger + remnants interlude (master-dispatched, owner ruling
2026-09-23: nothing configures the graph beyond chat models):

- The embedding ledger surface is gone: readEmbeddings/writeEmbeddings
  ports, the EmbeddingsLedger contract, EMBEDDINGS_FILE (config, store
  impls, graphRevision source list), and graph-deletion's vector-pruning
  step. On-disk data files stay untouched (inert history).
- The extraction embedder seam dies wholesale: ExtractionPipelineDeps
  drops the embedder dep (chat-handler + backfill/import-notes stubs go
  with it), the 'embedding' phase leaves ExtractionPhase, the usage
  capture/persist helpers die, and stages.embedding leaves
  ExtractionRunUsage (historical records keep their shape — reads are
  lenient). Extraction is two stages again, like the local-embedder era.
- Consumer-less remnants: latestQuizOutcomes + QUIZ_EVENT_KINDS,
  clusterContentId, matchClusterLabelings + Community + ClusterLabeling,
  the readClusterLabels/appendClusterLabel port seams + store impls +
  LEDGER_FILES.clusterLabels. NamingRunUsage/readNamingRuns STAY
  (standing ruling — the cost pill renders historical naming spend);
  clusterHue + MIN_CLUSTER_SIZE stay (live).
- The 'labeler' role dies: BACKGROUND_MODEL_ROLES, ROLE_SLOT, ModelRole,
  the catalog pin projections, and the UI roster strings. RolePins keys
  shrink with the union.

The notes birth fallback (owner-approved 2026-09-23, surfaced by the
interlude's live smoke: production notes imports create provenance-EMPTY
nodes (importNotes passes messages: null — the file's anchor is the
mapping line), so the provenance-only birth rule left every imported
file unclustered and invisible to the prober's session list, against the
locked file-birth ruling. A provenance-empty node now births under its
FIRST mapping line's chatId (ledger order); only notes-born nodes ever
hit the fallback (chat nodes always carry provenance; a re-encountered
node's link-append re-opens the provenance rule). Deletion-safe: mapping
lines of deleted chats are stripped alongside their provenance.

Live-verified on port 3004: a fresh notes import clusters under its
src_ id with the file path as the label, end to end with no embedder.

845/845 tests, typecheck/lint/build green.
EOF
)
This commit is contained in:
amosblomqvist
2026-09-23 15:37:34 +03:00
parent 1c3a5e121c
commit d1b46132b9
30 changed files with 181 additions and 595 deletions
+6 -63
View File
@@ -1,71 +1,14 @@
/**
* CLUSTER LABEL MATCHING (vision §1). DETECTION is birth-session
* CLUSTER PRESENTATION RULES (vision §1). DETECTION is birth-session
* clustering since wave 7 (core/session-clusters.ts; the semantic
* embedder and its Louvain predecessor are both dead). What REMAINS here
* is the
* cluster contract + core-survival label matching, which is
* detection-agnostic: it consumes the same { members } shape semantic
* clustering emits.
* embedder, its Louvain predecessor, and the LLM label matcher are all
* dead — labels are session names now). What remains here is the
* one presentation rule the snapshot and the renderer share.
*
* Labelings key to sorted MEMBER SETS, never to detection indices —
* a labeling record outlives any reshuffle of the partition.
*
* Browser-safe pure logic (no node builtins, no fs): the server runs
* matching for the snapshot payload, tests run it directly.
* Browser-safe pure logic (no node builtins, no fs).
*/
import type { ClusterLabeling } from "./contracts.ts";
/** A detected community. Members are SORTED node ids — the member set IS
* the identity a labeling record can outlive a reshuffle with. */
export interface Community {
members: string[];
}
/** Communities below this size are not rendered, not named — presentation
* rule (user-decided 2026-09-13): a region of the world model needs at
* least three propositions to be worth a hull and a label. */
export const MIN_CLUSTER_SIZE = 3;
/**
* Claim labeling records for detected communities — CORE-SURVIVAL (DECIDED
* 2026-09-13): a community keeps a record's label while AT LEAST HALF of
* the record's core survives in it (overlap * 2 >= core.length; exactly
* half survives). Both sides are single-successor: one community holds at
* most one label, one record is claimed by at most one community — on a
* split, only the branch carrying the majority of the core keeps the name;
* on a merge, the best-overlapping record wins.
*
* Greedy assignment by overlap size (largest first; ties broken by newer
* record). Returns a claim (or null) per community, in input order —
* unclaimed communities are the namer's input. Pure: same inputs, same
* claims.
*/
export function matchClusterLabelings(
communities: readonly Community[],
labelings: readonly ClusterLabeling[],
): (ClusterLabeling | null)[] {
interface Pair {
community: number;
labeling: ClusterLabeling;
overlap: number;
}
const pairs: Pair[] = [];
communities.forEach((community, ci) => {
const members = new Set(community.members);
for (const labeling of labelings) {
const overlap = labeling.core.reduce((n, id) => n + (members.has(id) ? 1 : 0), 0);
if (overlap * 2 >= labeling.core.length && overlap > 0) {
pairs.push({ community: ci, labeling, overlap });
}
}
});
pairs.sort((a, b) => b.overlap - a.overlap || (a.labeling.createdAt < b.labeling.createdAt ? 1 : -1));
const claims: (ClusterLabeling | null)[] = communities.map(() => null);
const takenRecords = new Set<string>();
for (const pair of pairs) {
if (claims[pair.community] || takenRecords.has(pair.labeling.id)) continue;
claims[pair.community] = pair.labeling;
takenRecords.add(pair.labeling.id);
}
return claims;
}
export const MIN_CLUSTER_SIZE = 3;
-4
View File
@@ -89,10 +89,6 @@ export interface ExtractionRunUsage {
stages: {
extractor: ExtractionStageUsage;
auditor: ExtractionStageUsage;
/** Wave 6 (2026-09-23): the provider-embedding tail — present only
* when an embedding model is configured and the run embedded
* (local embeddings were free, so this post-dates them). */
embedding?: ExtractionStageUsage;
};
}
+2 -38
View File
@@ -93,7 +93,8 @@ export interface NamingRunUsage {
/** ISO — wall-clock time of the run. */
at: string;
durationMs: number;
/** The labeler role's model id. */
/** The naming run's labeler role model id (the role died in wave 7;
* the field stays as the historical record's shape). */
model: string;
/** Every billed call (usually ONE — the namer is single-submit). */
steps: CallUsage[];
@@ -323,43 +324,6 @@ export interface AuditResult {
}
/** The GET /api/graph payload — everything the home view's PURE renderer needs. */
/** cluster-labels.jsonl line — one LLM-named community (vision §1,
* DECIDED 2026-09-13). Append-only history: records are immutable, never
* edited or deleted; dead labelings remain as history. IDENTITY is the
* sorted member set ("core"), hashed into the id, so re-naming the same
* community dedupes and a labeling survives graph edits as long as at
* least half its core survives in some single community (core-survival;
* under half → the community is renamed, splits can keep only one
* successor). */
/** The embedding ledger shape (config.EMBEDDINGS_FILE): vectors for
* EVERY embedded node, all produced by one model at one dimensionality. */
export interface EmbeddingsLedger {
/** The embedding model id (e.g. the HF repo) — mismatch = stale
* vectors, the writer re-embeds everything. */
model: string;
/** Vector dimensionality (MRL-truncated on write). */
dim: number;
/** nodeId → L2-normalized vector (cosine similarity = dot product). */
vectors: Record<string, number[]>;
/** The topic partition over those vectors, computed by the SAME
* embedder run (the user-decided trigger: recluster at the end of
* every embedding batch, never per read — the agglomerative pass is
* ~1.5s at current scale). Consumers (snapshot, settle, namer) read
* this instead of recomputing. Empty when nothing is embedded. */
clusters: Array<{ members: string[] }>;
/** ISO timestamp of the last write. */
updatedAt: string;
}
export interface ClusterLabeling {
/** "clu_" + content hash of the sorted core. */
id: string;
/** The short domain label (1–4 words), e.g. "Differential forms". */
label: string;
/** The member node ids the label was minted for, sorted — the survival core. */
core: string[];
createdAt: string;
}
/** One rendered cluster in the snapshot payload (vision §1). Members are
* node ids; the renderer draws the hull from member positions at draw
-24
View File
@@ -249,29 +249,5 @@ export function deleteChatTraces(graphStore: GraphStore, chatId: string, mode: C
graphStore.writePositions(prunedPositions);
}
// 6. Embeddings: prune dead vectors and dead cluster members — a dead
// member in a hull list would break rendering (its position is
// gone). Clusters emptied by pruning drop out; sub-minimum ones
// vanish at the read-side filter. The model/dim headers stay.
const embeddings = graphStore.readEmbeddings();
const prunedVectors = Object.fromEntries(
Object.entries(embeddings.vectors).filter(([id]) => live.has(id) || !knownNodeIds.has(id)),
);
const prunedClusters = embeddings.clusters
.map((cluster) => ({ members: cluster.members.filter((id) => live.has(id) || !knownNodeIds.has(id)) }))
.filter((cluster) => cluster.members.length > 0);
if (
Object.keys(prunedVectors).length !== Object.keys(embeddings.vectors).length ||
prunedClusters.length !== embeddings.clusters.length ||
prunedClusters.some((c, i) => c.members.length !== embeddings.clusters[i]!.members.length)
) {
graphStore.writeEmbeddings({
...embeddings,
vectors: prunedVectors,
clusters: prunedClusters,
updatedAt: new Date().toISOString(),
});
}
return stats;
}
+8 -45
View File
@@ -38,7 +38,7 @@
* adapters in.
*/
import { createHash, randomBytes } from "node:crypto";
import { randomBytes } from "node:crypto";
import { isToolUIPart, type UIMessage } from "ai";
import type { CoverageEntry } from "./contracts/graph.ts";
import type {
@@ -54,7 +54,7 @@ import type {
UnderstandingLevel,
} from "./contracts.ts";
import { MIN_CLUSTER_SIZE } from "./clusters.ts";
import { birthSessionPartition } from "./session-clusters.ts";
import { birthMappingIndex, birthSessionOf, birthSessionPartition } from "./session-clusters.ts";
import type { QuestionAnswerData } from "./question-core.ts";
import type { ChatStore, GraphStore } from "./ports.ts";
@@ -333,16 +333,6 @@ function quizEvent(
// ─── derivation (reads only — everything below is recomputable) ─────────────
/** The content identity of a community: "clu_" + sha256 of the sorted
* member ids (12 hex). Deterministic — the same member set always mints
* the same id — so re-naming races and idempotent re-runs dedupe at the
* store seam (appendClusterLabel). Lives here (server-only core), not in
* clusters.ts, because clusters.ts is barrel-exported and must stay
* browser-safe (no node:crypto). */
export function clusterContentId(members: readonly string[]): string {
return "clu_" + createHash("sha256").update([...members].sort().join("\n")).digest("hex").slice(0, 12);
}
export interface DerivedGraph {
nodes: GraphNode[];
edges: GraphEdge[];
@@ -394,34 +384,6 @@ export function getGraph(graphStore: GraphStore): DerivedGraph {
return { nodes: [...byId.values()], edges: merged, events: graphStore.readEvents() };
}
const QUIZ_EVENT_KINDS: ReadonlySet<string> = new Set(["correct", "miss", "dont-know"]);
/**
* The latest quiz outcome per node — a raw EVENT FACT, not a node state
* (the transitional node-state classes were removed 2026-09-08: nothing
* colors nodes by quiz results anymore). Its last consumer, the
* graph-as-context selection, died in wave 7 (the injection is gone —
* the prober reads its map through tools now); the event fact stays as
* raw ledger truth. Nodes with no quiz events have no entry.
*/
export function latestQuizOutcomes(nodes: GraphNode[], events: LedgerEvent[]): Map<string, LedgerEventKind> {
const latest = new Map<string, { at: string; kind: LedgerEventKind; index: number }>();
events.forEach((event, index) => {
if (event.nodeId === null) return;
if (!QUIZ_EVENT_KINDS.has(event.kind)) return;
const current = latest.get(event.nodeId);
if (!current || event.at > current.at || (event.at === current.at && index > current.index)) {
latest.set(event.nodeId, { at: event.at, kind: event.kind, index });
}
});
const out = new Map<string, LedgerEventKind>();
for (const node of nodes) {
const last = latest.get(node.id);
if (last) out.set(node.id, last.kind);
}
return out;
}
// ─── the /api/graph payload ──────────────────────────────────────────────────
/** Everything the home graph view needs, in one fetch (shape: contracts). */
@@ -505,7 +467,7 @@ export function graphSnapshot(deps: GraphDeps): GraphSnapshot {
// presentation rule (a region needs >= 3 propositions for a hull). A
// birth session whose record is gone (deleted chat) keeps its members
// but renders unlabeled — honest, never guessed.
const communities = birthSessionPartition(nodes);
const communities = birthSessionPartition(nodes, deps.graphStore.readMappings());
const clusters: Cluster[] = communities
.filter((community) => community.members.length >= MIN_CLUSTER_SIZE)
.map((community) => ({
@@ -562,7 +524,9 @@ export interface GraphSessionEntry {
* first (undatable records sink, in stable id order). */
export function graphSessionIndex(deps: GraphDeps): GraphSessionEntry[] {
const counts = new Map<string, number>();
for (const cluster of birthSessionPartition(getGraph(deps.graphStore).nodes)) {
const born = getGraph(deps.graphStore);
const mappingFallback = birthMappingIndex(deps.graphStore.readMappings());
for (const cluster of birthSessionPartition(born.nodes, deps.graphStore.readMappings())) {
counts.set(cluster.id, cluster.members.length);
}
const chats = new Map(deps.chatStore.listChats().map((chat) => [chat.id, chat]));
@@ -619,9 +583,8 @@ export interface GraphSessionProposition {
* ledger order (the same tie rule the snapshot's nodeObservations
* uses). Members sorted by node id — deterministic. */
export function graphSessionPropositions(deps: GraphDeps, sessionId: string): GraphSessionProposition[] {
const born = getGraph(deps.graphStore).nodes.filter(
(node) => node.provenance[0] !== undefined && node.provenance[0]!.chatId === sessionId,
);
const mappingFallback = birthMappingIndex(deps.graphStore.readMappings());
const born = getGraph(deps.graphStore).nodes.filter((node) => birthSessionOf(node, mappingFallback) === sessionId);
const latest = new Map<string, { at: string; note: string; index: number }>();
deps.graphStore.readEvents().forEach((event, index) => {
if (event.kind !== "observation" || event.nodeId === null || event.chatId !== sessionId) return;
+3 -3
View File
@@ -1,7 +1,6 @@
import type { GraphEdge, GraphNode } from "./contracts.ts";
import type { GraphEdge, GraphNode, NodeMapping, Position } from "./contracts.ts";
import { incrementalLayout } from "./layout.ts";
import { birthSessionPartition } from "./session-clusters.ts";
import type { Position } from "./contracts.ts";
/**
* The merge-time settle: recompute the resting layout from the CURRENT
@@ -20,6 +19,7 @@ import type { Position } from "./contracts.ts";
export function settleGraphLayout(graphStore: {
readNodes(): GraphNode[];
readEdges(): GraphEdge[];
readMappings(): NodeMapping[];
readPositions(): Record<string, Position>;
writePositions(positions: Record<string, Position>): void;
}): void {
@@ -39,7 +39,7 @@ export function settleGraphLayout(graphStore: {
// core/session-clusters.ts, wave 7) — one cluster concept everywhere.
// The unique list is the deduped set the partition needs (lines may
// repeat an id — first defines).
const communities = birthSessionPartition(unique);
const communities = birthSessionPartition(unique, graphStore.readMappings());
const next = incrementalLayout(unique, edges, previous, { clusters: communities });
graphStore.writePositions(Object.fromEntries(next));
}
-17
View File
@@ -27,9 +27,7 @@ import type {
SessionCheckpoint,
TurnUsage,
NotesSource,
ClusterLabeling,
NamingRunUsage,
EmbeddingsLedger,
} from "./contracts.ts";
import type { UserSettings } from "./settings.ts";
@@ -102,11 +100,6 @@ export interface GraphStore {
appendCoverage(entry: CoverageEntry): void;
readCoverage(): CoverageEntry[];
/** Append one cluster labeling — idempotent by id (the id hashes the
* sorted core, so identical member sets dedupe). */
appendClusterLabel(labeling: ClusterLabeling): void;
readClusterLabels(): ClusterLabeling[];
/** Append one cluster-naming run's usage (graph-global money channel) —
* idempotent per startedAt. */
appendNamingUsage(entry: NamingRunUsage): void;
@@ -128,16 +121,6 @@ export interface GraphStore {
/** Replace the whole ledger (the settle is a full-graph computation). */
writePositions(positions: Record<string, Position>): void;
/** The embeddings ledger (config.EMBEDDINGS_FILE) — INERT DATA since
* wave 7 (owner ruling 2026-09-23: the provider embedder is dead;
* clustering is birth-session, labels are session names; the
* graph-context injection is gone with it). The whole seam dies in
* the follow-up interlude; graph-deletion's vector pruning is
* harmless housekeeping meanwhile. */
readEmbeddings(): EmbeddingsLedger;
/** Replace the whole ledger (graph-deletion's vector pruning only). */
writeEmbeddings(ledger: EmbeddingsLedger): void;
/** THE DESTRUCTIVE SEAM (session-deletion wave 6, owner 2026-09-22 —
* the first non-append operation on these ledgers): atomically
* replace the full contents of any subset of the six rewritable
+36 -9
View File
@@ -21,14 +21,24 @@
*
* Notes imports are birth-sessions too (owner ruling 2026-09-23): their
* nodes carry a "src_…" source id instead of a chat id; the caller labels
* them with the file's path (kind: file in the session views).
* them with the file's path (kind: file in the session views). The
* mapping-line fallback (owner-approved 2026-09-23, surfaced by the
* interlude's live smoke): production notes imports create provenance-
* EMPTY nodes (the notes front door passes messages: null — the file's
* anchor is the mapping line), so a provenance-empty node's birth session
* is its FIRST mapping line's chatId in ledger order. Only notes-born
* nodes ever hit the fallback — chat extraction always writes provenance
* points, and a re-encountered node's link-append adds one, which
* re-opens the provenance rule above. Deletion-safe: a deleted chat's
* mapping lines are stripped alongside its provenance; notes mappings
* survive because files are never deleted.
*
* Browser-safe pure logic (no node builtins, no fs): same contract style
* as the rest of the core graph derivation. Not re-exported through the
* core barrel — the shell reads clusters from the snapshot; only the
* server-side derivations (snapshot, settle) and tests import this.
*/
import type { GraphNode } from "./contracts.ts";
import type { GraphNode, NodeMapping } from "./contracts.ts";
/** One birth-session cluster: the session id + the SORTED member node
* ids born in it. The shape is deliberately the { members } contract
@@ -41,11 +51,23 @@ export interface BirthSessionCluster {
members: string[];
}
/** A node's birth session: its first surviving provenance point's chatId,
* or null when the node carries no provenance at all (unclustered). */
export function birthSessionOf(node: GraphNode): string | null {
/** A node's birth session: its first surviving provenance point's chatId;
* for a provenance-empty node, its first mapping line's chatId (the notes
* fallback — see the module doc). null when neither exists. */
export function birthSessionOf(node: GraphNode, mappingFallback?: ReadonlyMap<string, string>): string | null {
const first = node.provenance[0];
return first === undefined ? null : first.chatId;
if (first !== undefined) return first.chatId;
return mappingFallback?.get(node.id) ?? null;
}
/** The notes fallback index: nodeId → the chatId of the node's FIRST
* mapping line in ledger order. */
export function birthMappingIndex(mappings: readonly NodeMapping[]): Map<string, string> {
const index = new Map<string, string>();
for (const mapping of mappings) {
if (!index.has(mapping.graphNodeId)) index.set(mapping.graphNodeId, mapping.chatId);
}
return index;
}
/**
@@ -53,15 +75,20 @@ export function birthSessionOf(node: GraphNode): string | null {
* (first line defines — the getGraph rule), groups by birth session, and
* returns the groups size-desc with members sorted (deterministic output:
* ties break by session id ascending). Nodes with no surviving provenance
* are left unclustered — they belong to no session the graph can name.
* fall back to their first mapping line (notes-born); a node with neither
* is left unclustered — it belongs to no session the graph can name.
*/
export function birthSessionPartition(nodes: readonly GraphNode[]): BirthSessionCluster[] {
export function birthSessionPartition(
nodes: readonly GraphNode[],
mappings: readonly NodeMapping[] = [],
): BirthSessionCluster[] {
const mappingFallback = birthMappingIndex(mappings);
const members = new Map<string, string[]>();
const seen = new Set<string>();
for (const node of nodes) {
if (seen.has(node.id)) continue;
seen.add(node.id);
const birth = birthSessionOf(node);
const birth = birthSessionOf(node, mappingFallback);
if (birth === null) continue;
const bucket = members.get(birth);
if (bucket) bucket.push(node.id);
+1 -3
View File
@@ -45,7 +45,6 @@ export const BACKGROUND_MODEL_ROLES = [
"visualMaker",
"extractor",
"auditor",
"labeler",
] as const;
export type BackgroundModelRole = (typeof BACKGROUND_MODEL_ROLES)[number];
@@ -64,7 +63,7 @@ export interface RoleModelOverride {
/** The capability tiers a user can pin models into (owner-settled mapping):
* strong = prober + teacher (also the per-session default-teacher
* starting point); cheap = researcher, demo-builder, extractor, auditor,
* labeler; vision = visual-maker + pdf-reader. Resolution per role:
; vision = visual-maker + pdf-reader. Resolution per role:
* ROLE OVERRIDE → slot → factory pin. */
export const MODEL_SLOTS = ["strong", "cheap", "vision"] as const;
export type ModelSlotName = (typeof MODEL_SLOTS)[number];
@@ -174,7 +173,6 @@ export const ROLE_SLOT: Readonly<Record<ModelRole, ModelSlotName>> = {
demoBuilder: "cheap",
extractor: "cheap",
auditor: "cheap",
labeler: "cheap",
visualMaker: "vision",
pdfReader: "vision",
};