Birth-session clustering + the prober graph-tools data seam (wave 7 chunk A)

Embeddings are dying (chunk B removes the machinery); the clusters they
used to carry become the session that created them — derived on read.

- New src/core/session-clusters.ts: birthSessionPartition groups merged
  nodes by their FIRST SURVIVING provenance point's chatId (owner-locked
  2026-09-23). Deletion is self-healing by construction: stripped-birth
  survivors re-attribute to their earliest remaining contributor, and
  empty-provenance nodes stay unclustered. Notes imports are birth-sessions
  too (owner ruling) — their label is the file path.
- graphSnapshot: clusters = the birth partition filtered >= MIN_CLUSTER_SIZE
  (the hull presentation rule), id = session id, label = session name (chat
  title / file path; a deleted record renders unlabeled). The persisted
  cluster-labelings join dies; no LLM naming pass needed anymore.
- settleGraphLayout derives the same partition from readNodes — drops its
  readEmbeddings dep. One cluster concept everywhere.
- Prober data seam (core/graph.ts, pure ports-driven like the snapshot):
  graphSessionIndex (every birth session: id, name, chat|file kind,
  createdAt/updatedAt, birth-proposition count, recency-ordered) and
  graphSessionPropositions (a session's birth propositions with the
  extractor's latest understanding level and THAT event's own timestamp —
  owner ruling: quiz outcomes are ledger evidence, they never move the
  evaluation date). Teacher's tool handlers consume these in-process.
- New agent.graphTools subsection { enabled?: boolean } (owner ruling:
  the toggle gates prober tool REGISTRATION — agent capability, agent
  section, its own subsection, NOT the tool-toggles map; absent = ON).
  Lenient sanitizer, strict settings-body parser, passthrough resolution,
  and graphToolsEnabled(effective) = graph.enabled && toggle for Teacher's
  registry wiring.

Gates: tsc clean, eslint clean, 923/923 tests, next build compiles.
Live-verified on the worktree dev data: 10 nodes → 2 clusters labeled
with their chat titles; session index + propositions carry levels and
their own timestamps.
This commit is contained in:
amosblomqvist
2026-09-23 14:51:37 +03:00
parent fb66606555
commit 0fcffdffaf
12 changed files with 551 additions and 61 deletions
+129 -17
View File
@@ -51,8 +51,10 @@ import type {
GraphSnapshot,
LedgerEvent,
LedgerEventKind,
UnderstandingLevel,
} from "./contracts.ts";
import { matchClusterLabelings, MIN_CLUSTER_SIZE } from "./clusters.ts";
import { MIN_CLUSTER_SIZE } from "./clusters.ts";
import { birthSessionPartition } from "./session-clusters.ts";
import type { QuestionAnswerData } from "./question-core.ts";
import type { ChatStore, GraphStore } from "./ports.ts";
@@ -496,23 +498,20 @@ export function graphSnapshot(deps: GraphDeps): GraphSnapshot {
(nodeObservations[nodeId] ??= {})[chatId] = note;
}
// Semantic topic communities (DECIDED 2026-09-13, replacing structural
// Louvain): average-linkage clustering over embeddings — related
// topics, not reasoning chains. Edge-orphans join topics by meaning;
// stragglers stay loose (filtered at the presentation threshold
// below). The partition is computed ONCE per embedding batch by the
// embedder (the agglomerative pass is ~1.5s at current scale) and
// PERSISTED in the embedding ledger — reads just load it. Nodes whose
// embedding hasn't landed yet simply sit unclustered until the next
// read after the embedder runs.
const communities = deps.graphStore.readEmbeddings().clusters;
const claims = matchClusterLabelings(communities, deps.graphStore.readClusterLabels());
// BIRTH-SESSION CLUSTERS (wave 7, owner-locked 2026-09-23): embeddings
// are dead — a cluster is the SESSION THAT CREATED ITS PROPOSITIONS,
// derived on read from provenance (core/session-clusters.ts). The label
// is the session's NAME (chat title / file path): no LLM naming pass, no
// persisted partition to invalidate. MIN_CLUSTER_SIZE stays the hull
// presentation rule (a region needs >= 3 propositions for a hull). A
// birth session whose record is gone (deleted chat) keeps its members
// but renders unlabeled — honest, never guessed.
const communities = birthSessionPartition(nodes);
const clusters: Cluster[] = communities
.map((community, index) => ({ community, claim: claims[index] }))
.filter(({ community }) => community.members.length >= MIN_CLUSTER_SIZE)
.map(({ community, claim }) => ({
id: claim?.id ?? clusterContentId(community.members),
label: claim?.label ?? null,
.filter((community) => community.members.length >= MIN_CLUSTER_SIZE)
.map((community) => ({
id: community.id,
label: sessions[community.id] ?? fileSources[community.id] ?? null,
members: community.members,
}));
@@ -533,5 +532,118 @@ export function graphSnapshot(deps: GraphDeps): GraphSnapshot {
};
}
// ─── Prober graph tools — the data seam (wave 7, owner-locked 2026-09-23) ───
/** One session in the prober's graph view: every provenance anchor the
* graph can name — a real chat (deep-linkable) or an imported notes file
* (owner ruling 2026-09-23: files are birth-sessions too, kind "file").
* The prober's session-list tool renders these; the per-session view is
* graphSessionPropositions below. Pure derivation over the same ports
* the snapshot reads — Teacher's tool handlers call these in-process. */
export interface GraphSessionEntry {
/** chatId or notes source id ("src_…") — the birth-session key. */
id: string;
/** The chat's title, or the file's path — the cluster label's source. */
name: string;
/** A real chat (its record still exists) or an imported notes file. */
kind: "chat" | "file";
/** When the session began: the chat's createdAt, or the file's
* importedAt. null when no record can date it. */
createdAt: string | null;
/** Most recent activity: the chat's updatedAt, or the file's
* importedAt. null when no record can date it. */
updatedAt: string | null;
/** How many propositions were BORN in this session (birth-session
* clustering's count — shared/re-encountered nodes count for their
* birth session only). */
propositions: number;
}
/** The prober's session list: every birth session, most-recently-active
* first (undatable records sink, in stable id order). */
export function graphSessionIndex(deps: GraphDeps): GraphSessionEntry[] {
const counts = new Map<string, number>();
for (const cluster of birthSessionPartition(getGraph(deps.graphStore).nodes)) {
counts.set(cluster.id, cluster.members.length);
}
const chats = new Map(deps.chatStore.listChats().map((chat) => [chat.id, chat]));
const sources = new Map(deps.graphStore.readSources().map((s) => [s.id, s]));
const entries: GraphSessionEntry[] = [];
for (const [id, propositions] of counts) {
const chat = chats.get(id); if (chat) {
entries.push({
id,
name: chat.title,
kind: "chat",
createdAt: chat.createdAt,
updatedAt: chat.updatedAt,
propositions,
});
continue;
}
const source = sources.get(id);
entries.push({
id,
name: source?.filePath ?? id,
kind: "file",
createdAt: source?.importedAt ?? null,
updatedAt: source?.importedAt ?? null,
propositions,
});
}
return entries.sort(
(a, b) =>
(b.updatedAt ?? "").localeCompare(a.updatedAt ?? "") ||
(b.createdAt ?? "").localeCompare(a.createdAt ?? "") ||
(a.id < b.id ? -1 : 1),
);
}
/** One birth proposition of one session, for the prober's per-session
* view. `understanding` is the extractor's latest observation for
* (node, session) — the named-word scale (solid/shaky/accepted/resisted/
* not yet engaged) — and `evaluatedAt` is THAT event's own timestamp
* (owner ruling 2026-09-23: quiz outcomes are evidence in the ledger,
* but they never move the evaluation date — the level's event does). */
export interface GraphSessionProposition {
nodeId: string;
/** The compact handle — the same name the canvas shows. */
name: string;
understanding: UnderstandingLevel | null;
/** The `at` of the event that minted `understanding`; null when the
* session never evaluated this proposition. */
evaluatedAt: string | null;
}
/** The propositions BORN in one session, with the session's latest
* understanding of each. Latest observation wins by `at`, then by
* ledger order (the same tie rule the snapshot's nodeObservations
* uses). Members sorted by node id — deterministic. */
export function graphSessionPropositions(deps: GraphDeps, sessionId: string): GraphSessionProposition[] {
const born = getGraph(deps.graphStore).nodes.filter(
(node) => node.provenance[0] !== undefined && node.provenance[0]!.chatId === sessionId,
);
const latest = new Map<string, { at: string; note: string; index: number }>();
deps.graphStore.readEvents().forEach((event, index) => {
if (event.kind !== "observation" || event.nodeId === null || event.chatId !== sessionId) return;
if (!event.note) return;
const current = latest.get(event.nodeId);
if (!current || event.at > current.at || (event.at === current.at && index > current.index)) {
latest.set(event.nodeId, { at: event.at, note: event.note, index });
}
});
return born
.map((node) => {
const observation = latest.get(node.id);
return {
nodeId: node.id,
name: node.name,
understanding: (observation?.note as UnderstandingLevel | undefined) ?? null,
evaluatedAt: observation?.at ?? null,
};
})
.sort((a, b) => (a.nodeId < b.nodeId ? -1 : 1));
}
// Re-export: the contract owns the shape; legacy importers still reach it.
export type { EmissionStats };
+12 -11
View File
@@ -1,5 +1,6 @@
import type { EmbeddingsLedger, GraphEdge, GraphNode } from "./contracts.ts";
import type { GraphEdge, GraphNode } from "./contracts.ts";
import { incrementalLayout } from "./layout.ts";
import { birthSessionPartition } from "./session-clusters.ts";
import type { Position } from "./contracts.ts";
/**
@@ -11,17 +12,16 @@ import type { Position } from "./contracts.ts";
* finish before the next /api/graph fetch reflects the change.
*
* SPLIT FROM layout.ts deliberately (2026-09-13): layout.ts is in the
* browser barrel, and the settle needs the Louvain partition for the
* cluster centroid pull — the heavy deps must not ship in the client
* bundle. Server code imports this module DIRECTLY; it is never
* re-exported through src/core/index.ts.
* browser barrel, and the settle needs the cluster partition for the
* cluster centroid pull — this module must not ship in the client bundle.
* Server code imports it DIRECTLY; it is never re-exported through
* src/core/index.ts.
*/
export function settleGraphLayout(graphStore: {
readNodes(): GraphNode[];
readEdges(): GraphEdge[];
readPositions(): Record<string, Position>;
writePositions(positions: Record<string, Position>): void;
readEmbeddings(): EmbeddingsLedger;
}): void {
const nodes = graphStore.readNodes();
if (nodes.length === 0) return;
@@ -34,11 +34,12 @@ export function settleGraphLayout(graphStore: {
});
const edges = graphStore.readEdges();
const previous = new Map(Object.entries(graphStore.readPositions()));
// The cluster partition for the centroid pull: the SAME persisted
// semantic topics the snapshot renders (computed by the embedder
// once per batch, stored in the embedding ledger) — one cluster
// concept everywhere.
const communities = graphStore.readEmbeddings().clusters;
// The cluster partition for the centroid pull: the SAME birth-session
// partition the snapshot renders (derived from provenance on read —
// core/session-clusters.ts, wave 7) — one cluster concept everywhere.
// The unique list is the deduped set the partition needs (lines may
// repeat an id — first defines).
const communities = birthSessionPartition(unique);
const next = incrementalLayout(unique, edges, previous, { clusters: communities });
graphStore.writePositions(Object.fromEntries(next));
}
+73
View File
@@ -0,0 +1,73 @@
/**
* BIRTH-SESSION CLUSTERING (wave 7, owner-locked 2026-09-23): embeddings
* are dead — a proposition's cluster is the SESSION THAT CREATED IT,
* derived deterministically from provenance. Cluster labels are session
* names (the chat title or the imported file's path): no model call, no
* key, no configuration. What the agglomerative pass grouped by MEANING
* proximity, this groups by WHERE KNOWLEDGE WAS BORN — the owner's call
* that the session that minted a set of propositions is the honest
* "region of the world model" they belong to.
*
* Birth session (owner ruling 2026-09-23): a node's FIRST surviving
* provenance point's chatId — the create line's first point at read time
* (getGraph merges lines by id, first line defines, provenance unions in
* line order — so provenance[0] is the birth point of the surviving
* lines). This makes deletion self-healing with ZERO special handling:
* delete-nodes strips a session's create lines and its provenance points;
* a SURVIVING shared node re-attributes to its earliest remaining
* contributor (provenance[0] walks forward), and a node left with NO
* provenance at all is simply unclustered. The partition is derived on
* every read — there is no persisted partition to invalidate.
*
* Notes imports are birth-sessions too (owner ruling 2026-09-23): their
* nodes carry a "src_…" source id instead of a chat id; the caller labels
* them with the file's path (kind: file in the session views).
*
* Browser-safe pure logic (no node builtins, no fs): same contract style
* as the rest of the core graph derivation. Not re-exported through the
* core barrel — the shell reads clusters from the snapshot; only the
* server-side derivations (snapshot, settle) and tests import this.
*/
import type { GraphNode } from "./contracts.ts";
/** One birth-session cluster: the session id + the SORTED member node
* ids born in it. The shape is deliberately the { members } contract
* the layout consumer takes; `id` rides along for consumers that need
* the session identity (the snapshot's cluster id + label join). */
export interface BirthSessionCluster {
/** The birth session's id — a chatId or a notes source id ("src_…"). */
id: string;
/** Node ids born in this session, sorted. */
members: string[];
}
/** A node's birth session: its first surviving provenance point's chatId,
* or null when the node carries no provenance at all (unclustered). */
export function birthSessionOf(node: GraphNode): string | null {
const first = node.provenance[0];
return first === undefined ? null : first.chatId;
}
/**
* The birth-session partition over the given node LINES: dedupes by id
* (first line defines — the getGraph rule), groups by birth session, and
* returns the groups size-desc with members sorted (deterministic output:
* ties break by session id ascending). Nodes with no surviving provenance
* are left unclustered — they belong to no session the graph can name.
*/
export function birthSessionPartition(nodes: readonly GraphNode[]): BirthSessionCluster[] {
const members = new Map<string, string[]>();
const seen = new Set<string>();
for (const node of nodes) {
if (seen.has(node.id)) continue;
seen.add(node.id);
const birth = birthSessionOf(node);
if (birth === null) continue;
const bucket = members.get(birth);
if (bucket) bucket.push(node.id);
else members.set(birth, [node.id]);
}
return [...members.entries()]
.map(([id, list]) => ({ id, members: [...list].sort() }))
.sort((a, b) => b.members.length - a.members.length || (a.id < b.id ? -1 : 1));
}
+18
View File
@@ -81,6 +81,9 @@ export interface EffectiveSettings {
* passthrough delta — engine resolution (pin → auto → unavailable)
* lives in server/search.ts, where keys are visible. */
search: { engine?: SearchEngineId };
/** Prober graph tools (wave 7): passthrough delta as stored —
* absent/undefined enabled = ON; consumers use graphToolsEnabled. */
graphTools: { enabled?: boolean };
};
/** The resolved strong/cheap/vision slots: each tier's concrete model or
* NULL when the tier holds no model (no factory floor — BYOK all the
@@ -178,6 +181,7 @@ export function resolveSettings(record: UserSettings | null): EffectiveSettings
tools: agent.tools ?? {},
prompts: agent.prompts ?? {},
search: agent.search ?? {},
graphTools: agent.graphTools ?? {},
},
models: {
strong: resolveSlot(modelsSection.strong),
@@ -228,3 +232,17 @@ export function extractionEnabled(graph: Required<GraphSettings>): boolean {
export function graphContextEnabled(graph: Required<GraphSettings>): boolean {
return graph.enabled && graph.embedding;
}
/**
* The PROBER GRAPH TOOLS gate (wave 7, owner ruling 2026-09-23): the
* toggle is an agent-capability switch (it gates tool registration), but
* the tools only mean anything when the graph itself runs — so the
* effective gate composes the graph master switch AND the toggle.
* Teacher's tool factories consume this at registration time.
*/
export function graphToolsEnabled(effective: {
graph: Required<GraphSettings>;
agent: { graphTools: { enabled?: boolean } };
}): boolean {
return effective.graph.enabled && (effective.agent.graphTools.enabled ?? true);
}
+16
View File
@@ -200,6 +200,14 @@ export interface AgentSettings {
* engine must still have its key, else search is honestly unavailable.
* Absent + no key = unavailable (no silent fallback, no default). */
search?: { engine?: SearchEngineId };
/** Prober graph tools (wave 7, owner ruling 2026-09-23): read-only
* knowledge-graph views for the prober agent — a session list +
* per-session propositions with understanding levels. Its SUBJECT
* is the graph but its MECHANISM is agent capability (tool
* registration), so it lives here in its OWN subsection, not in the
* tool-toggles map. Absent = enabled (the toggle discipline); the
* effective gate composes graph.enabled AND this switch. */
graphTools?: { enabled?: boolean };
/** System-prompt override SWITCHES keyed by segment path (absent key =
* off). Switch-on requires the override file to exist (enforced by the
* route); the switch is explicit so users keep a file on disk but run
@@ -419,6 +427,14 @@ function sanitizeAgent(raw: Record<string, unknown>): AgentSettings {
const engine = (raw.search as Record<string, unknown>).engine;
if (engine === "tavily" || engine === "brave") out.search = { engine };
}
// Prober graph tools (wave 7): its own agent-section subsection —
// lenient read side (a bad shape drops the key and the default ON
// resumes; the write side's parser is strict).
if (raw.graphTools !== null && typeof raw.graphTools === "object" && !Array.isArray(raw.graphTools)) {
const enabled = (raw.graphTools as Record<string, unknown>).enabled;
if (enabled === undefined) out.graphTools = {};
else if (typeof enabled === "boolean") out.graphTools = { enabled };
}
return out;
}
+12
View File
@@ -195,6 +195,18 @@ export function parseAgentSettingsBody(body: unknown): SettingsBodyResult<AgentS
settings.search = { engine: search.engine };
}
}
// Prober graph tools (wave 7, owner ruling 2026-09-23): an agent-section
// subsection, not a tool-toggles key — the mechanism is agent capability
// (tool registration), the subject is the graph. Strict write side:
// object shape only; `enabled` boolean when present. Absent = ON.
if (raw.graphTools !== undefined) {
const graphTools = asRecord(raw.graphTools);
if (!graphTools) return { ok: false, error: "graphTools must be an object" };
if (graphTools.enabled !== undefined && typeof graphTools.enabled !== "boolean") {
return { ok: false, error: "graphTools.enabled must be a boolean" };
}
settings.graphTools = graphTools.enabled === undefined ? {} : { enabled: graphTools.enabled };
}
return { ok: true, settings };
}