extraction cleanup: events ledger dies, markdown source view, auditor-as-linker (2026-09-27 rulings)

The graph becomes raw reasoning chains only. Quiz outcomes and
understanding observations are no longer formalized; assessment
moves to the future practice module.

Events ledger removed end-to-end: LedgerEvent/LedgerEventKind/
UnderstandingLevel/ExtractionAttribution/AuditResult.dropped and
snapshot nodeObservations deleted from contracts; quizEvent builder
and the events step of mergeFindings gone; appendEvent/readEvents
dropped from the port and store; LEDGER_FILES loses events.jsonl
with a one-time purge of a stale file next to the nodes ledger;
graph-deletion and the delete dialog lose event purge + quiz-answer
impact counts; GraphSessionProposition becomes a pure birth list.

Whole-session markdown source view: new core/session-markdown.ts is
the shared formatting engine (id-anchored speaker blocks, quiz/
recall/question blocks, tool one-liners) — the extractor's prompt
view and the future user-facing export render from one engine.
Coverage fingerprints deliberately stay on the old line format
(content-hash semantics; re-formatting never re-opens covered
history). The whole session goes to the extractor — no phase cut.

Auditor refocused as the identity linker: never sees the source;
explores the graph via new auditor-specific tools in
agent/tools/graph.ts (find — glob over name/statement/aliases;
list_sessions; list_propositions with full statement text; no
timestamps) instead of an injected full-graph dump; equivalence
bias — paraphrases merge, only genuinely distinct propositions stay
apart; no drop channel, the extractor is fully trusted; may draw
bridging edges (typed follows/supports) between existing nodes and
the batch, resolved by the merger with chat-scoped message-less
provenance. Agentic loop, maxTurns 16.

UI: the node detail card gains an edges section above the sessions
list (follows-from / supported-by / what-follows / what-it-supports,
rendered only when present, click-to-select neighbors); the
understanding-note chips are gone.

Prober list_propositions reworded to what a session covered —
names only, no levels. docs/architecture.md + README updated to
the as-built pipeline. Gates: typecheck, 1059/1059, lint, build.
This commit is contained in:
amosblomqvist
2026-09-28 16:54:44 +03:00
parent cd09c12b56
commit 1b50013551
22 changed files with 909 additions and 1064 deletions
+1 -1
View File
@@ -154,7 +154,7 @@ data/ chats/ · graph/ (six ledgers + positions/source
**The async sub-agent protocol (ack-and-steer).** `research` / `demo` / `visual` are server-executed tools: the call resolves immediately mid-turn with a fire-and-forget ack and the teacher keeps teaching; a background runner works in parallel (the client polls `/api/activity` and `/api/jobs`). On completion the runner builds a **steer message** — a user-role `UIMessage` whose text part is the model-facing result and whose `data-subagent-artifact` part carries the full result for the client — and the **wake pipeline** delivers it: a per-chat serialized queue waits for the turn to end and no question to be pending, then fires a server-side teacher turn that weaves the result into the lesson. Delivery is exactly-once (deterministic steer ids, delivered-flag checks at every claim point); stalled jobs are recovered on read; aborted jobs deliver an abort steer. The client adopts the wake turn through the same `turnInFlight` machinery as returning mid-stream (busy shimmer, composer gate) and renders steers as artifacts — never a bubble. Sub-agents **verify, never one-shot**: the demo-builder headlessly executes its HTML in system Chrome before delivery; the visual-maker renders each SVG and looks at it (vision-capable model — the `vision` slot in settings). **The async sub-agent protocol (ack-and-steer).** `research` / `demo` / `visual` are server-executed tools: the call resolves immediately mid-turn with a fire-and-forget ack and the teacher keeps teaching; a background runner works in parallel (the client polls `/api/activity` and `/api/jobs`). On completion the runner builds a **steer message** — a user-role `UIMessage` whose text part is the model-facing result and whose `data-subagent-artifact` part carries the full result for the client — and the **wake pipeline** delivers it: a per-chat serialized queue waits for the turn to end and no question to be pending, then fires a server-side teacher turn that weaves the result into the lesson. Delivery is exactly-once (deterministic steer ids, delivered-flag checks at every claim point); stalled jobs are recovered on read; aborted jobs deliver an abort steer. The client adopts the wake turn through the same `turnInFlight` machinery as returning mid-stream (busy shimmer, composer gate) and renders steers as artifacts — never a bubble. Sub-agents **verify, never one-shot**: the demo-builder headlessly executes its HTML in system Chrome before delivery; the visual-maker renders each SVG and looks at it (vision-capable model — the `vision` slot in settings).
**The knowledge graph (two channels, one writer).** The write channel: the extraction pipeline — extractor (LLM, tree-blind: sees only the transcript view — propositions as true statements, `follows`/`supports` edges, retroactive quiz attribution, one-word understanding observations) → auditor (LLM: sees findings + the current graph; drops anything without verbatim evidence, stamps each survivor with an identity verdict — link to an existing node or create) → merger (`mergeFindings`, deterministic: applies the verdicts, link-appends phrasings/provenance, rebuilds quiz events from the transcript's tool parts, dedupes, checkpoints coverage) → layout settle. Triggered when a chat's **uncovered tokens** (message-view fingerprints, `coverage.jsonl`) cross `graph.extractionChunkTokens` (a settings knob, default 4096, floor 500 — read fresh per turn); failure checkpoints nothing, the next trigger re-runs. The same pipeline serves the **notes front door** (`POST /api/graph/import`: any `*.md`/`*.txt` becomes a source; re-imports are idempotent by path hash). The read channel: `GET /api/graph` derives the snapshot from the ledgers — a pure read; the home view is a hand-rolled Canvas 2D renderer (one rAF loop, dirty-flag idle, d3-zoom camera, zoom-gated labels, cluster hulls) over a d3-force layout, with server-persisted render settings and drag-committed positions. Clusters are **birth-session groupings**: a node joins the cluster of the session that birthed it (a chat, or an imported notes file as a file-kind session), derived at read — no embedding pass, no naming pass. The teacher reaches the graph on its own terms through the gated prober tools (`list_sessions` / `list_propositions`, the `agent.graphTools` setting) — there is no automatic context injection. **The knowledge graph (two channels, one writer).** The write channel: the extraction pipeline — extractor (LLM: sees only the session, rendered as readable markdown by the shared `core/session-markdown.ts` engine — propositions as true statements, `follows`/`supports` edges with verbatim evidence; fully trusted, graph-blind) → auditor (LLM, an agentic loop that never sees the source: explores the existing graph through `find` / `list_sessions` / `list_propositions` tools, stamps each proposition with an identity verdict — link to an existing node (equivalence bias: paraphrases merge) or create — and may draw bridging edges between existing nodes and the new batch) → merger (`mergeFindings`, deterministic: applies the verdicts, link-appends phrasings/provenance, resolves and dedupes edges, checkpoints coverage) → layout settle. No quiz outcomes or understanding assessments live in the graph (2026-09-27 cleanup); assessment becomes practice-module territory. Triggered when a chat's **uncovered tokens** (message-view fingerprints, `coverage.jsonl`) cross `graph.extractionChunkTokens` (a settings knob, default 4096, floor 500 — read fresh per turn); failure checkpoints nothing, the next trigger re-runs. The same pipeline serves the **notes front door** (`POST /api/graph/import`: any `*.md`/`*.txt` becomes a source; re-imports are idempotent by path hash). The read channel: `GET /api/graph` derives the snapshot from the ledgers — a pure read; the home view is a hand-rolled Canvas 2D renderer (one rAF loop, dirty-flag idle, d3-zoom camera, zoom-gated labels, cluster hulls) over a d3-force layout, with server-persisted render settings and drag-committed positions. Clusters are **birth-session groupings**: a node joins the cluster of the session that birthed it (a chat, or an imported notes file as a file-kind session), derived at read — no embedding pass, no naming pass. The teacher reaches the graph on its own terms through the gated prober tools (`list_sessions` / `list_propositions`, the `agent.graphTools` setting) — there is no automatic context injection.
**Dictation.** Optional speech-to-text: the server mints short-lived Deepgram tokens; audio streams browser→Deepgram over WebSocket, never through the server. **Dictation.** Optional speech-to-text: the server mints short-lived Deepgram tokens; audio streams browser→Deepgram over WebSocket, never through the server.
+44 -16
View File
@@ -1563,13 +1563,54 @@ export function GraphView() {
> >
<div className="text-sm font-medium leading-snug">{selected.node.name}</div> <div className="text-sm font-medium leading-snug">{selected.node.name}</div>
<div className="mt-1 text-xs leading-relaxed text-muted-foreground">{selected.node.statement}</div> <div className="mt-1 text-xs leading-relaxed text-muted-foreground">{selected.node.statement}</div>
{(() => {
// EDGES (owner 2026-09-27): edge classification was entirely
// hidden in the UI — the card now shows every edge this node
// participates in, grouped by direction + type, above the
// sessions list. Only present groups render.
const id = selected.node.id;
const nodeById = new Map((snapshot?.nodes ?? []).map((n) => [n.id, n]));
const label = (nid: string) => nodeById.get(nid)?.name ?? nid;
const groups: { title: string; names: string[] }[] = [
{
title: "Follows from",
names: (snapshot?.edges ?? []).filter((e) => e.from === id && e.type === "follows").map((e) => label(e.to)),
},
{
title: "Supported by",
names: (snapshot?.edges ?? []).filter((e) => e.from === id && e.type === "supports").map((e) => label(e.to)),
},
{
title: "What follows from this",
names: (snapshot?.edges ?? []).filter((e) => e.to === id && e.type === "follows").map((e) => label(e.from)),
},
{
title: "What this supports",
names: (snapshot?.edges ?? []).filter((e) => e.to === id && e.type === "supports").map((e) => label(e.from)),
},
].filter((g) => g.names.length > 0);
if (groups.length === 0) return null;
return (
<div className="mt-3 flex flex-col gap-1">
{groups.map((g) => (
<div key={g.title} className="flex flex-col gap-0.5">
<div className="text-xs text-muted-foreground">{g.title}</div>
<div className="flex flex-wrap gap-1">
{g.names.map((name, i) => (
<span key={`${name}-${i}`} className="rounded-full border px-2 py-0.5 text-[11px] text-muted-foreground">
{name}
</span>
))}
</div>
</div>
))}
</div>
);
})()}
{selected.sourceIds.length > 0 ? ( {selected.sourceIds.length > 0 ? (
<div className="mt-3 flex flex-col gap-1"> <div className="mt-3 flex flex-col gap-1">
<div className="text-xs text-muted-foreground">Sessions</div> <div className="text-xs text-muted-foreground">Sessions</div>
{selected.sourceIds.map((sourceId) => { {selected.sourceIds.map((sourceId) => {
// That source's understanding observation — the
// extractor's named-word note (confidence history).
const note = snapshot?.nodeObservations?.[selected.node.id]?.[sourceId];
// An imported-notes file: a STATIC entry — the path, // An imported-notes file: a STATIC entry — the path,
// plain text, never a link (no session to open). // plain text, never a link (no session to open).
if (snapshot?.fileSources?.[sourceId] !== undefined) { if (snapshot?.fileSources?.[sourceId] !== undefined) {
@@ -1579,11 +1620,6 @@ export function GraphView() {
<span className="min-w-0 flex-1 truncate px-1 py-1 text-sm text-muted-foreground"> <span className="min-w-0 flex-1 truncate px-1 py-1 text-sm text-muted-foreground">
{snapshot.fileSources[sourceId]} {snapshot.fileSources[sourceId]}
</span> </span>
{note && (
<span className="max-w-28 shrink-0 truncate whitespace-nowrap rounded-full border px-2 py-0.5 text-[10px] text-muted-foreground">
{note}
</span>
)}
</div> </div>
); );
} }
@@ -1596,14 +1632,6 @@ export function GraphView() {
> >
{snapshot?.sessions[sourceId] ?? sourceId} {snapshot?.sessions[sourceId] ?? sourceId}
</Link> </Link>
{/* The note is a fixed-scale word (solid/shaky/...) — the
session name is the main thing; the chip is bounded so
it can never push or overflow the row. */}
{note && (
<span className="max-w-28 shrink-0 truncate whitespace-nowrap rounded-full border px-2 py-0.5 text-[10px] text-muted-foreground">
{note}
</span>
)}
</div> </div>
); );
})} })}
-1
View File
@@ -58,7 +58,6 @@ export type ModelRole =
export const LEDGER_FILES = { export const LEDGER_FILES = {
nodes: path.join(PATHS.graph, "nodes.jsonl"), nodes: path.join(PATHS.graph, "nodes.jsonl"),
edges: path.join(PATHS.graph, "edges.jsonl"), edges: path.join(PATHS.graph, "edges.jsonl"),
events: path.join(PATHS.graph, "events.jsonl"),
mappings: path.join(PATHS.graph, "mappings.jsonl"), mappings: path.join(PATHS.graph, "mappings.jsonl"),
sessions: path.join(PATHS.graph, "sessions.jsonl"), sessions: path.join(PATHS.graph, "sessions.jsonl"),
/** Per-message extraction coverage (contracts/graph.ts CoverageEntry; /** Per-message extraction coverage (contracts/graph.ts CoverageEntry;
+10 -10
View File
@@ -331,24 +331,24 @@ flowchart TB
TRIG["trigger seam (turn-service): coverageOf(record, coverageLedger)<br/>— uncovered tokens ≥ graph.extractionChunkTokens (settings)?"] -->|"yes, single-flight per source"| RUN["extraction.runExtraction(chatId)"] TRIG["trigger seam (turn-service): coverageOf(record, coverageLedger)<br/>— uncovered tokens ≥ graph.extractionChunkTokens (settings)?"] -->|"yes, single-flight per source"| RUN["extraction.runExtraction(chatId)"]
IMPORT["POST /api/graph/import (notes front door)<br/>deterministic src id = sha256(path) — idempotent re-imports"] -->|"importNotes: shared pipeline, second extractor prompt"| RUN IMPORT["POST /api/graph/import (notes front door)<br/>deterministic src id = sha256(path) — idempotent re-imports"] -->|"importNotes: shared pipeline, second extractor prompt"| RUN
RUN --> BATCH["batchRuns — greedy token-budgeted batches of oldest uncovered messages"] RUN --> BATCH["batchRuns — greedy token-budgeted batches of oldest uncovered messages"]
BATCH --> EXT["EXTRACTOR (LLM) — sees ONLY the transcript view<br/>(tree-blind; the same messageViewLines the fingerprint hashes)"] BATCH --> EXT["EXTRACTOR (LLM) — sees ONLY the session markdown view<br/>(tree-blind, graph-blind; shared engine: core/session-markdown.ts)"]
EXT --> AUD["AUDITOR (LLM) — sees findings + current graph;<br/>verifies evidence verbatim, judges identity (link-or-create), drops + logs unbacked"] EXT --> AUD["AUDITOR (LLM, agentic loop) — sees findings ONLY, never the source;<br/>explores the graph via find / list_sessions / list_propositions tools,<br/>judges identity (link-or-create, equivalence bias) + may draw bridging edges"]
AUD --> MERGE["MERGER (deterministic, core/graph.ts) — the ONLY ledger writer:<br/>identity via validated linkTo · dedupe edges · rebuild quiz events from tool parts · coverage checkpoint"] AUD --> MERGE["MERGER (deterministic, core/graph.ts) — the ONLY ledger writer:<br/>identity via validated linkTo · bridging-edge resolution · dedupe edges · coverage checkpoint"]
MERGE --> SETTLE["settleGraphLayout — resting positions ledger"] MERGE --> SETTLE["settleGraphLayout — resting positions ledger"]
MERGE -->|"failure at ANY stage"| NOTHING["checkpoints nothing — the next trigger re-runs"] MERGE -->|"failure at ANY stage"| NOTHING["checkpoints nothing — the next trigger re-runs"]
``` ```
**Tree-blind and post-hoc** (vision §1): the extractor reads ONLY the transcript — never the session's plan tree — and proposes free-form propositions (true statements; a misconception is stated as its true correction), `follows`/`supports` edges, per-proposition understanding observations (a closed one-word scale), and message-granular provenance. Every node is an objective, world-facing fact; the learner's relationship to it lives entirely in the event ledger. **Raw reasoning chains only** (2026-09-27 cleanup): the extractor reads ONLY the session — the whole thing, rendered as readable markdown by the shared engine (`core/session-markdown.ts`; the same renderer will serve future user-facing session export) — and proposes free-form propositions (true statements; a misconception is stated as its true correction) and `follows`/`supports` edges with verbatim evidence. Every node is an objective, world-facing fact. No quiz outcomes, no understanding assessments — the events ledger and `UnderstandingLevel` are deleted; the learner's relationship to a proposition becomes practice-module territory (assessment records, later).
**Division of trust** (the pipeline's core rule): the LLM stages supply only judgment — *what was learned*, *what follows/supports what*, *which proposition each quiz tested*, and the auditor's *identity* verdict (whether a proposition IS an existing node). Everything the model could silently mis-copy — event kinds (`correct`/`miss`/`dont-know`), timestamps, idempotency refs, notes, provenance points — is rebuilt deterministically by the merger from the transcript's tool parts. The extractor files verbatim evidence quotes; the auditor drops anything unbacked and logs every drop. Conservative throughout: duplicates over wrong merges, drop over invent. **Division of trust** (the pipeline's core rule): the extractor is fully trusted and is the only stage that ever sees the source. The auditor never sees the transcript — it receives the findings JSON alone and explores the existing graph through read-only tools (`find`, `list_sessions`, `list_propositions` — statement text included, no timestamps) to reach identity verdicts: `linkTo` an existing node or let the merger create one. Equivalence bias (owner ruling 2026-09-27): paraphrases, rewordings, and compressions of the same claim MERGE; only genuinely distinct propositions stay apart. The auditor may also draw bridging edges — typed `follows`/`supports` connections between existing graph nodes and the batch's propositions (the auditor is the asserter: chat-scoped provenance, no message anchor). The deterministic merger validates and appends: linkTo resolution, dedupe by `(from, to, type)`, coverage checkpoints. There is no drop channel — nothing rejects the extractor's output.
**Coverage, not quiz-counting** (supersedes the v1 every-N-quiz-answers trigger): each message's *view* (the same lines the extractor sees) is hashed into a per-chat fingerprint ledger (`coverage.jsonl`); "covered" means exactly "what the model saw." Uncovered = fingerprint diff, never dual-tracked; runs are greedy token-budgeted batches (`graph.extractionChunkTokens` — a settings knob, default 4096, floor 500, read fresh each turn) of the oldest uncovered messages — quiz-less sessions extract too. A coarse `sessions.jsonl` checkpoint still marks each merged run. The trigger seam (`turn-service.ts`) is trigger-agnostic from the pipeline's side: when a better mechanism lands (session end, idle, explicit flush), only the seam changes. Single-flight per source; the run registry powers the coverage UI and true abort. **Coverage, not quiz-counting** (supersedes the v1 every-N-quiz-answers trigger): each message's *view* (the compact line format `core/coverage.ts` has always hashed) is hashed into a per-chat fingerprint ledger (`coverage.jsonl`); "covered" means exactly "a run committed on this content." The prompt view is the richer markdown document — a deliberate 2026-09-27 split: re-formatting presentation never re-opens covered history. Uncovered = fingerprint diff, never dual-tracked; runs are greedy token-budgeted batches (`graph.extractionChunkTokens` — a settings knob, default 4096, floor 500, read fresh each turn) of the oldest uncovered messages — quiz-less sessions extract too. A coarse `sessions.jsonl` checkpoint still marks each merged run. The trigger seam (`turn-service.ts`) is trigger-agnostic from the pipeline's side: when a better mechanism lands (session end, idle, explicit flush), only the seam changes. Single-flight per source; the run registry powers the coverage UI and true abort.
**Two front doors, one pipeline**: chat transcripts (quiz-triggered) and raw personal notes (`POST /api/graph/import` — each file becomes a `NotesSource` playing the `chatId` role, deterministic id from the path hash). **Two front doors, one pipeline**: chat transcripts (quiz-triggered) and raw personal notes (`POST /api/graph/import` — each file becomes a `NotesSource` playing the `chatId` role, deterministic id from the path hash).
**The read channel is pure derivation.** `GET /api/graph` reads the ledgers (`graphSnapshot`: merged nodes/edges, folded observations, birth-session clusters, positions) — the home view never triggers a write. Node states are always recomputed from the ledger, never stored. **The read channel is pure derivation.** `GET /api/graph` reads the ledgers (`graphSnapshot`: merged nodes/edges, birth-session clusters, positions) — the home view never triggers a write. Node detail cards show a node's edges (follows-from / supported-by / what-follows / what-it-supports) and its birth sessions.
**The teacher probes, nothing is injected** (supersedes the semantic-injection experiment, killed 2026-09-23): the teacher reaches the graph on its own terms through two read-only prober tools — `list_sessions` (birth sessions, the learner's files) and `list_propositions` (a session's extracted statements) — gated by the `agent.graphTools` setting (composing with `graph.enabled`). No topic embedding, no top-K selection, no session-start steer; the agent decides what to look at, and the graph never writes into a lesson unbidden. **The teacher probes, nothing is injected** (supersedes the semantic-injection experiment, killed 2026-09-23): the teacher reaches the graph on its own terms through two read-only prober tools — `list_sessions` (birth sessions, the learner's files) and `list_propositions` (the names of what a session covered) — gated by the `agent.graphTools` setting (composing with `graph.enabled`). No topic embedding, no top-K selection, no session-start steer; the agent decides what to look at, and the graph never writes into a lesson unbidden.
**Rendering**: a hand-rolled Canvas 2D renderer (`components/graph-canvas.ts` — one rAF loop, dirty-flag idle, d3-zoom camera, zoom-gated label fade, cluster hulls computed at draw time) over a d3-force layout in core (`layout.ts`, warm-started `incrementalLayout` + `settleGraphLayout`). Render settings and drag-committed positions persist server-side; the camera persists per-device in localStorage. **Rendering**: a hand-rolled Canvas 2D renderer (`components/graph-canvas.ts` — one rAF loop, dirty-flag idle, d3-zoom camera, zoom-gated label fade, cluster hulls computed at draw time) over a d3-force layout in core (`layout.ts`, warm-started `incrementalLayout` + `settleGraphLayout`). Render settings and drag-committed positions persist server-side; the camera persists per-device in localStorage.
@@ -406,9 +406,9 @@ The Electron main is a launcher + security guard, never a host for app logic: si
| **One composition root per concern** — `stores.ts` (storage), `chat-handler.ts` (a turn) | | **One composition root per concern** — `stores.ts` (storage), `chat-handler.ts` (a turn) |
| **One writer per ledger** — the merger is the sole graph writer; chat records written only by the turn pipeline; job records only by the runner; positions PUT is the only sanctioned graph-adjacent exception | | **One writer per ledger** — the merger is the sole graph writer; chat records written only by the turn pipeline; job records only by the runner; positions PUT is the only sanctioned graph-adjacent exception |
| **The store never decides** — adapters persist and shape-guard; merge/dedup/derivation semantics are reader-side, in core | | **The store never decides** — adapters persist and shape-guard; merge/dedup/derivation semantics are reader-side, in core |
| **Derived, never dual-tracked** — append-only evidence; every state recomputed at read (snapshot, plan statuses, phases, understanding, costs, coverage) | | **Derived, never dual-tracked** — append-only evidence; every state recomputed at read (snapshot, plan statuses, phases, costs, coverage) |
| **Idempotency via deterministic keys** — steer ids, ledger refs, coverage fingerprints, mapping `(chatId, sourceId)`, cluster content hashes; re-running anything converges | | **Idempotency via deterministic keys** — steer ids, ledger refs, coverage fingerprints, mapping `(chatId, sourceId)`, cluster content hashes; re-running anything converges |
| **LLMs judge; code mechanizes** — grading, merging, event kinds, fingerprints, layout are deterministic; a failed LLM stage checkpoints nothing | | **LLMs judge; code mechanizes** — grading, merging, fingerprints, layout are deterministic; a failed LLM stage checkpoints nothing |
| **The persisted record is the only source of truth** — agents are stateless per-turn rebuilds; no unanswered tool call reaches a provider | | **The persisted record is the only source of truth** — agents are stateless per-turn rebuilds; no unanswered tool call reaches a provider |
| **Failure containment** — side channels (context, extraction, usage, naming) are fire-and-forget behind try/catch seams; a side-channel failure never touches the response or the persist; the turn registry clears on every path | | **Failure containment** — side channels (context, extraction, usage, naming) are fire-and-forget behind try/catch seams; a side-channel failure never touches the response or the persist; the turn registry clears on every path |
| **Absence is not zero** — unreported tokens, absent prices, absent levels render as absent, never fabricated | | **Absence is not zero** — unreported tokens, absent prices, absent levels render as absent, never fabricated |
+2 -2
View File
@@ -27,7 +27,7 @@ async function main() {
const stats = await pipeline.runExtraction(chat.id); const stats = await pipeline.runExtraction(chat.id);
console.log( console.log(
`[backfill] ${chat.id} (${chat.title.slice(0, 40)}): ` + `[backfill] ${chat.id} (${chat.title.slice(0, 40)}): ` +
`${stats.nodes} nodes, ${stats.edges} edges, ${stats.events} events in ${((Date.now() - t0) / 1000).toFixed(1)}s`, `${stats.nodes} nodes, ${stats.edges} edges in ${((Date.now() - t0) / 1000).toFixed(1)}s`,
); );
} catch (err) { } catch (err) {
console.error(`[backfill] ${chat.id} FAILED:`, err); console.error(`[backfill] ${chat.id} FAILED:`, err);
@@ -35,7 +35,7 @@ async function main() {
} }
const graph = getGraph(graphStore); const graph = getGraph(graphStore);
console.log( console.log(
`[backfill] done: ${graph.nodes.length} nodes, ${graph.edges.length} edges, ${graph.events.length} events`, `[backfill] done: ${graph.nodes.length} nodes, ${graph.edges.length} edges`,
); );
} }
+4 -5
View File
@@ -33,7 +33,7 @@ async function main() {
getChunkTokens: () => effectiveSettings().graph.extractionChunkTokens, getChunkTokens: () => effectiveSettings().graph.extractionChunkTokens,
}); });
const queue = [...files]; const queue = [...files];
const totals = { nodes: 0, edges: 0, events: 0, failed: 0 }; const totals = { nodes: 0, edges: 0, failed: 0 };
const workers = Array.from({ length: Math.min(limit, files.length) }, async () => { const workers = Array.from({ length: Math.min(limit, files.length) }, async () => {
for (;;) { for (;;) {
const file = queue.shift(); const file = queue.shift();
@@ -44,9 +44,8 @@ async function main() {
const stats = await pipeline.importNotes({ filePath, text: fs.readFileSync(filePath, "utf-8") }); const stats = await pipeline.importNotes({ filePath, text: fs.readFileSync(filePath, "utf-8") });
totals.nodes += stats.nodes; totals.nodes += stats.nodes;
totals.edges += stats.edges; totals.edges += stats.edges;
totals.events += stats.events;
console.log( console.log(
`[import-notes] ${file}: ${stats.nodes} nodes, ${stats.edges} edges, ${stats.events} events in ${((Date.now() - t0) / 1000).toFixed(1)}s`, `[import-notes] ${file}: ${stats.nodes} nodes, ${stats.edges} edges in ${((Date.now() - t0) / 1000).toFixed(1)}s`,
); );
} catch (err) { } catch (err) {
totals.failed++; totals.failed++;
@@ -58,8 +57,8 @@ async function main() {
const graph = getGraph(stores.graphStore); const graph = getGraph(stores.graphStore);
console.log( console.log(
`[import-notes] done: +${totals.nodes} nodes, +${totals.edges} edges, +${totals.events} events, ${totals.failed} failed — ` + `[import-notes] done: +${totals.nodes} nodes, +${totals.edges} edges, ${totals.failed} failed — ` +
`graph now ${graph.nodes.length} nodes, ${graph.edges.length} edges, ${graph.events.length} events`, `graph now ${graph.nodes.length} nodes, ${graph.edges.length} edges`,
); );
} }
+151 -296
View File
@@ -20,33 +20,43 @@ import type {
ChatRecord, ChatRecord,
ExtractionRunUsage, ExtractionRunUsage,
ExtractionStageUsage, ExtractionStageUsage,
GraphNode,
SessionFindings, SessionFindings,
} from "../../core/contracts.ts"; } from "../../core/contracts.ts";
import { getGraph, mergeFindings, type EmissionStats, type GraphDeps } from "../../core/graph.ts"; import { getGraph, mergeFindings, type EmissionStats, type GraphDeps } from "../../core/graph.ts";
import type { GraphStore } from "../../core/ports.ts"; import type { GraphStore } from "../../core/ports.ts";
import { settleGraphLayout } from "../../core/layout-settle.ts"; import { settleGraphLayout } from "../../core/layout-settle.ts";
import { batchRuns, batchTag, coverageOf, messageViewLines } from "../../core/coverage.ts"; import { batchRuns, coverageOf, messageViewLines } from "../../core/coverage.ts";
import { sessionMarkdown } from "../../core/session-markdown.ts";
import { auditorGraphTools } from "../tools/graph.ts";
/** /**
* The extraction pipeline (vision §1 construction, 2026-09-02 redesign) — the * The extraction pipeline (vision §1 construction, 2026-09-02 redesign;
* sole graph writer. Three stages, one orchestrator: * relinked 2026-09-27) — the sole graph writer. Three stages, one
* orchestrator:
* *
* extractor (LLM) transcript → SessionFindings (free-form propositions, * extractor (LLM) the ONE stage that sees the raw source (a session
* follows/supports edges, quiz attribution, understanding * rendered as markdown by the shared session-markdown
* notes — each with verbatim evidence). TREE-BLIND: it * engine, or a notes file verbatim) → SessionFindings:
* sees only the transcript, never the plan tree. * propositions + follows/supports edges, each with
* auditor (LLM) findings + THE CURRENT GRAPH → AuditResult: the * verbatim evidence. TREE-BLIND, graph-blind, FULLY
* verified subset with an identity verdict per * TRUSTED — nothing downstream re-checks it.
* proposition (linkTo an existing node, or create), plus * auditor (LLM) never sees the source. Its job is IDENTITY: for each
* the drops for the log. * proposition, linkTo an existing graph node or create —
* decided by EXPLORING the graph through tools (find /
* list_sessions / list_propositions), not by an injected
* graph dump (context bloat, owner 2026-09-27). It may
* also draw BRIDGING edges from existing graph nodes to
* the batch's propositions.
* merger (core) mergeFindings() → deterministic ledger appends * merger (core) mergeFindings() → deterministic ledger appends
* *
* The LLM stages supply only the judgment calls; the deterministic rebuild * The events ledger is GONE (2026-09-27): quiz outcomes and understanding
* in the merger owns everything that must never be mis-copied (event kinds, * observations are no longer formalized in the graph — the graph is purely
* timestamps, refs, provenance points). Both agents are single-tool-call * extracted reasoning chains (nodes + edges). Understanding-level
* workers: their whole deliverable is one TypeBox-validated submit call — a * assessment is practice-module territory.
* ToolLoopAgent purely so the model can retry a schema-invalid call. *
* The extractor is a single-tool-call worker (one TypeBox-validated submit
* call — a pi agent purely so the model can retry a schema-invalid call).
* The auditor is a small AGENTIC loop: exploration tools plus one submit.
* *
* Created via createExtractionPipeline(deps) — the composition root * Created via createExtractionPipeline(deps) — the composition root
* (chat-handler) injects the storage ports + models; this module never * (chat-handler) injects the storage ports + models; this module never
@@ -60,11 +70,9 @@ import { batchRuns, batchTag, coverageOf, messageViewLines } from "../../core/co
* session, no message ids). A notes-mode extractor prompt replaces the * session, no message ids). A notes-mode extractor prompt replaces the
* session one; the auditor and merger are shared (identity + deterministic * session one; the auditor and merger are shared (identity + deterministic
* appends don't care where text came from). Provenance is the file itself: * appends don't care where text came from). Provenance is the file itself:
* source ids ("src_" + path hash) play the chatId role in mappings, events * source ids ("src_" + path hash) play the chatId role in mappings and
* and checkpoints, and the source record (graphStore sources seam) carries * checkpoints, and the source record (graphStore sources seam) carries
* the file path + the run's usage. No quiz events are ever built from * the file path + the run's usage.
* notes — quiz outcomes inside the text surface only through the
* extractor's understanding observations (user-decided 2026-09-13).
* *
* Observability (§5 baseline): every stage logs one line per step, its * Observability (§5 baseline): every stage logs one line per step, its
* start, and its duration; the pipeline logs a failure line before * start, and its duration; the pipeline logs a failure line before
@@ -72,32 +80,23 @@ import { batchRuns, batchTag, coverageOf, messageViewLines } from "../../core/co
* stats (or failed) for every run. * stats (or failed) for every run.
*/ */
// ─── the transcript view (what the extractor + auditor see) ───────────────── // ─── the source view (what the extractor sees) ──────────────────────────────
/** /**
* A quiz tool part's outcome, one line: the attribution evidence. * A session rendered as readable markdown — the EXTRACTOR's view (owner
* (question text + what the learner did — kind/correct are rebuilt by the * ruling 2026-09-27): the whole session goes in, formatted by the ONE
* merger; the extractor only needs enough to attribute.) * shared session-markdown engine (core/session-markdown.ts — the same
*/ * engine the user-facing session export will use), with the message-id
/** * anchors preserved (the extractor reports them back per
* The compact transcript both LLM stages see: user/teacher text VERBATIM, * proposition/edge — they become provenance). TREE-BLIND by design:
* each tagged with its message id (provenance anchors — the extractor * no plan tree, no mark_done.
* reports them back per proposition/edge), and quiz outcome lines. *
* TREE-BLIND by design (2026-09-02): no plan tree, no mark_done — * Presentation-only: coverage fingerprints hash core/coverage.ts's line
* extraction is free-form, decoupled from the session's plan. Job tool * format, which is SEPARATE and unchanged — re-formatting the view never
* parts (research/demo/visual) are noise for extraction — their outcomes * re-opens covered history (the deliberate 2026-09-27 split).
* are summarized as one line each.
*/ */
export function transcriptView(record: ChatRecord): string { export function transcriptView(record: ChatRecord): string {
// The shared seam (core/coverage.ts messageViewLines): the LLM's prompt return sessionMarkdown(record.messages, { title: record.title });
// view and the coverage fingerprint use ONE line builder, so "covered"
// means — exactly — what the extractor saw. Wave D: the record is native
// history — the view reads the native messages directly.
const lines = [`SESSION: ${record.title}`];
record.messages.forEach((message, index) => {
lines.push(...messageViewLines(message, (message as { id?: string }).id || `m${index}`));
});
return lines.join("\n");
} }
// ─── the extractor ────────────────────────────────────────────────────────── // ─── the extractor ──────────────────────────────────────────────────────────
@@ -111,24 +110,6 @@ const messagesSchema = Type.Array(Type.String(), {
description: "The message ids (from the [id] tags in the transcript) where this came up — provenance.", description: "The message ids (from the [id] tags in the transcript) where this came up — provenance.",
}); });
const understandingSchema = Type.Optional(
Type.Union(
[
Type.Literal("solid"),
Type.Literal("shaky"),
Type.Literal("accepted"),
Type.Literal("resisted"),
Type.Literal("not yet engaged"),
],
{
description:
"ONE word — the learner's grasp of this proposition in THIS session, from the fixed scale " +
'"solid" | "shaky" | "accepted" | "resisted" | "not yet engaged". Nothing else: no numbers, ' +
"no justification, no explanation.",
},
),
);
const propositionProps = { const propositionProps = {
id: Type.String({ description: 'Local id: "p1", "p2", … — referenced by edges and attribution.' }), id: Type.String({ description: 'Local id: "p1", "p2", … — referenced by edges and attribution.' }),
name: Type.String({ name: Type.String({
@@ -142,14 +123,13 @@ const propositionProps = {
"The proposition as one reasoning step — a TRUE, self-contained, world-facing statement. " + "The proposition as one reasoning step — a TRUE, self-contained, world-facing statement. " +
'Never reference the learner, the teacher, or the session: not "the learner understands X" but ' + 'Never reference the learner, the teacher, or the session: not "the learner understands X" but ' +
'"X". A misconception is stated as the TRUE correction ("Nietzsche did not endorse ' + '"X". A misconception is stated as the TRUE correction ("Nietzsche did not endorse ' +
'might-makes-right social Darwinism") — the wrongness lives in the quiz outcomes, never here.', 'might-makes-right social Darwinism") — wrongness is never recorded in the graph.',
}), }),
aliases: Type.Array(Type.String(), { aliases: Type.Array(Type.String(), {
description: "Verbatim phrasings this session used for the same proposition, if any.", description: "Verbatim phrasings this session used for the same proposition, if any.",
}), }),
messages: messagesSchema, messages: messagesSchema,
evidence: evidenceSchema, evidence: evidenceSchema,
understanding: understandingSchema,
}; };
const edgeSchema = Type.Object({ const edgeSchema = Type.Object({
@@ -177,20 +157,6 @@ const findingsProps = {
"Relations between propositions: follows (derivable) / supports (grounds without entailing). Pass [] when there are none — never omit the field.", "Relations between propositions: follows (derivable) / supports (grounds without entailing). Pass [] when there are none — never omit the field.",
}), }),
), ),
attribution: Type.Optional(
Type.Object(
{
quiz: Type.Record(Type.String(), Type.Union([Type.String(), Type.Null()]), {
description:
"toolCallId of EVERY quiz line → the proposition id it was really testing. Attribute RETROACTIVELY — the whole transcript is visible; map each quiz to the proposition it tests (often articulated elsewhere in the session). null when genuinely unattributable.",
}),
},
{
description:
"Quiz attribution — your only event judgment. Pass { quiz: {} } when the session has no quiz lines — never omit the field.",
},
),
),
}; };
const findingsSchema = Type.Object(findingsProps); const findingsSchema = Type.Object(findingsProps);
@@ -203,7 +169,7 @@ const auditProps = {
linkTo: Type.Optional( linkTo: Type.Optional(
Type.String({ Type.String({
description: description:
"The id of the EXISTING graph node this proposition IS — your identity verdict. Absent = create a new node. When unsure, omit: duplicates over wrong merges.", "The id of the EXISTING graph node this proposition IS — your identity verdict. Absent = create a new node. EQUIVALENCE BIAS (2026-09-27): paraphrases, rewordings, and compressions of the same claim MERGE; only genuinely distinct propositions stay apart.",
}), }),
), ),
}), }),
@@ -211,34 +177,15 @@ const auditProps = {
description: "Every distinct proposition the session touched — true statements, misconception-free.", description: "Every distinct proposition the session touched — true statements, misconception-free.",
}, },
), ),
dropped: Type.Optional(
Type.Array(
Type.Object({
ref: Type.String({
description: 'Proposition id, "from→to" for an edge, or attribution toolCallId.',
}),
reason: Type.String(),
}),
{
description:
"Everything you dropped, with the reason — the audit trail. Pass [] when you dropped nothing — never omit the field.",
},
),
),
}; };
const auditSchema = Type.Object(auditProps); const auditSchema = Type.Object(auditProps);
// The notes front door's shapes: everything session findings have, minus // The notes front door's shapes are IDENTICAL to the session shapes now
// quiz attribution — raw notes have no quiz tool parts, so there is // (the only difference was quiz attribution, which no longer exists):
// nothing to attribute (and no event to build: outcomes inside the text // one schema, one findings type, both front doors.
// surface only through understanding observations, user-decided). type NotesFindings = SessionFindings;
const { attribution: _omitted, ...notesFindingsProps } = findingsProps; type NotesAuditResult = AuditResult;
const notesFindingsSchema = Type.Object(notesFindingsProps);
const { attribution: _omitted2, ...notesAuditProps } = auditProps;
const notesAuditSchema = Type.Object(notesAuditProps);
type NotesFindings = Omit<SessionFindings, "attribution">;
type NotesAuditResult = Omit<AuditResult, "attribution">;
// ── the weak-model ??-fills (the old zod defaults, applied at submit) ── // ── the weak-model ??-fills (the old zod defaults, applied at submit) ──
@@ -246,39 +193,26 @@ function fillFindings(raw: Static<typeof findingsSchema>): SessionFindings {
return { return {
propositions: raw.propositions, propositions: raw.propositions,
edges: raw.edges ?? [], edges: raw.edges ?? [],
attribution: raw.attribution ?? { quiz: {} },
}; };
} }
function fillNotesFindings(raw: Static<typeof notesFindingsSchema>): NotesFindings {
return { propositions: raw.propositions, edges: raw.edges ?? [] };
}
function fillAudit(raw: Static<typeof auditSchema>): AuditResult { function fillAudit(raw: Static<typeof auditSchema>): AuditResult {
return { return {
propositions: raw.propositions, propositions: raw.propositions,
edges: raw.edges ?? [], edges: raw.edges ?? [],
attribution: raw.attribution ?? { quiz: {} },
dropped: raw.dropped ?? [],
}; };
} }
function fillNotesAudit(raw: Static<typeof notesAuditSchema>): NotesAuditResult {
return { propositions: raw.propositions, edges: raw.edges ?? [], dropped: raw.dropped ?? [] };
}
const NOTES_EXTRACTOR_INSTRUCTIONS = `You are the extraction specialist in a learning application. You read ONE source — const NOTES_EXTRACTOR_INSTRUCTIONS = `You are the extraction specialist in a learning application. You read ONE source —
the user's own NOTES, in an ARBITRARY format (personal notes, an exported conversation, a document — anything) — and the user's own NOTES, in an ARBITRARY format (personal notes, an exported conversation, a document — anything) — and
report what those notes show the author engaging with, as structured findings for their global knowledge graph. report what those notes show the author engaging with, as structured findings for their global knowledge graph.
There is no session and no chat structure here: the source is raw text, the author's own material. Read it as such. There is no session and no chat structure here: the source is raw text, the author's own material. Read it as such.
You are the ONLY stage that sees this source — you are fully trusted; report carefully.
Report, with verbatim evidence quotes for everything: The graph is a set of TRUE propositions about the world, full stop. A node is never anything about the author or the
source — no "the author understands X", no "mentioned in these notes": the node is just X. A misconception is NOT a
NODE THEORY — the graph is a set of TRUE propositions about the world, full stop. A node is never anything about the node of its own wrong claim — state the TRUE correction as the proposition.
author or the source. The author's relationship to a proposition is expressed ENTIRELY by your understanding note; it
never leaks into the node itself. So no "the author understands X", no "mentioned in these notes": the node is just X.
A misconception is NOT a node of its own wrong claim — state the TRUE correction as the proposition.
1. PROPOSITIONS — every distinct reasoning-step-sized proposition the notes actually engage with: 1. PROPOSITIONS — every distinct reasoning-step-sized proposition the notes actually engage with:
- Each has a "name" — a VERY short label, a few words, a compact handle like a glossary headword ("Will to power", - Each has a "name" — a VERY short label, a few words, a compact handle like a glossary headword ("Will to power",
@@ -288,20 +222,14 @@ A misconception is NOT a node of its own wrong claim — state the TRUE correcti
one reasoning step each — never a topic blob, never a sub-step. one reasoning step each — never a topic blob, never a sub-step.
- "messages": ALWAYS an empty list — provenance is the file itself. - "messages": ALWAYS an empty list — provenance is the file itself.
- "aliases": verbatim phrasings the notes used for the same proposition. - "aliases": verbatim phrasings the notes used for the same proposition.
- "understanding": ONE word from the fixed scale — "solid", "shaky", "accepted", "resisted", or "not yet engaged" —
judging the AUTHOR's grasp of the proposition as the notes show it: their own commentary, self-assessments,
recorded quiz outcomes, the way they write about it. Strictly one word, no numbers, no justification (a
wrong-shaped note is rejected by the schema). When the notes give no signal, omit the field.
2. EDGES between propositions: 2. EDGES between propositions:
- "follows": to is DERIVABLE from from (understanding from makes understanding to achievable). - "follows": to is DERIVABLE from from (understanding from makes understanding to achievable).
- "supports": from GROUNDS to without entailing it (helpful context, not a prerequisite). - "supports": from GROUNDS to without entailing it (helpful context, not a prerequisite).
- Only relations the notes actually articulate — with evidence. - Only relations the notes actually articulate — with evidence.
There is no quiz attribution for notes — the source has no quiz events, and the schema has no attribution field. Evidence discipline: every quote must appear VERBATIM in the source — this is your self-grounding, and the proof the
statements come from the material rather than invention.
Evidence discipline: every quote must appear VERBATIM in the source. The auditor rejects anything unbacked. When in
doubt, report; the auditor decides.
Include EVERY top-level field of the call even when empty — "edges": [] when there are no relations. NEVER omit Include EVERY top-level field of the call even when empty — "edges": [] when there are no relations. NEVER omit
a field because it is empty. a field because it is empty.
@@ -309,19 +237,18 @@ a field because it is empty.
Your entire deliverable is ONE submit_findings call.`; Your entire deliverable is ONE submit_findings call.`;
const EXTRACTOR_INSTRUCTIONS = `You are the extraction specialist in a learning application. You read ONE session const EXTRACTOR_INSTRUCTIONS = `You are the extraction specialist in a learning application. You read ONE session
transcript and report what was learned — as structured findings for the learner's global knowledge graph. rendering and report what that session teaches — as structured findings for the learner's global knowledge graph.
The transcript view shows: the conversation verbatim (LEARNER / TEACHER lines, each tagged with its message id) The session is rendered as readable markdown: Teacher and Learner passages in order (each header carries its message
and QUIZ outcome lines. id — the [id] anchor), Quiz / Recall-check / Question blocks for the embedded checks, and one-line notes where other
tools ran. Read ALL of it as the session's teaching content.
Report, with verbatim evidence quotes for everything: You are the ONLY stage that sees this source, and you are FULLY TRUSTED: nothing downstream re-checks your work.
Extract carefully — the graph will hold exactly what you report.
NODE THEORY — the graph is a set of TRUE propositions about the world, full stop. A node is never anything The graph is a set of TRUE propositions about the world, full stop. A node is never anything about the learner or the
about the learner or the session. The learner's relationship to a proposition — whether they know it, missed session — no "the learner understands X", no "discussed in this session": the node is just X. A misconception is NOT
it, resisted it — is expressed ENTIRELY by the quiz outcomes and your understanding note; it never leaks into a node of its own wrong claim — state the TRUE correction as the proposition.
the node itself. So no "the learner understands X", no "discussed in this session": the node is just X. A
misconception is NOT a node of its own wrong claim — state the TRUE correction as the proposition, and let the
quiz outcomes carry that the learner believed the wrong version.
1. PROPOSITIONS — every distinct reasoning-step-sized proposition the session touched: 1. PROPOSITIONS — every distinct reasoning-step-sized proposition the session touched:
- Each has a "name" — a VERY short label, a few words, a compact handle like a glossary headword - Each has a "name" — a VERY short label, a few words, a compact handle like a glossary headword
@@ -332,26 +259,19 @@ quiz outcomes carry that the learner believed the wrong version.
in the graph only as this one statement). in the graph only as this one statement).
- Include corrections of misconceptions the teacher articulated, and true facts explained in prose. - Include corrections of misconceptions the teacher articulated, and true facts explained in prose.
- One reasoning step per proposition: never a topic blob, never a sub-step. - One reasoning step per proposition: never a topic blob, never a sub-step.
- "messages": the [id] tags of every message where the proposition came up — provenance, report it. - "messages": the id from the header anchor of every passage where the proposition came up — provenance, report it.
- "aliases": verbatim phrasings this session used for the same proposition. - "aliases": verbatim phrasings this session used for the same proposition.
- "understanding": ONE word from the fixed scale — "solid", "shaky", "accepted", "resisted", or
"not yet engaged" — judging the learner's grasp of the proposition in THIS session. Strictly one
word, no numbers, no justification, no explanation (a wrong-shaped note is rejected by the schema).
2. EDGES between propositions: 2. EDGES between propositions:
- "follows": to is DERIVABLE from from (understanding from makes understanding to achievable). - "follows": to is DERIVABLE from from (understanding from makes understanding to achievable).
- "supports": from GROUNDS to without entailing it (helpful context, not a prerequisite). - "supports": from GROUNDS to without entailing it (helpful context, not a prerequisite).
- Only relations the transcript actually articulates — with evidence. - Only relations the session actually articulates — with evidence.
3. ATTRIBUTION — every QUIZ line gets a mapping: toolCallId → the proposition id it was really testing. Evidence discipline: every quote must appear VERBATIM in the source markdown (a Quiz block's question counts as the
Attribute RETROACTIVELY — a quiz often tests a proposition the session only articulates later; the source). This is your self-grounding: the proof that statements come from the material rather than invention.
question text is your guide. If you truly cannot tell, use null.
Evidence discipline: every quote must appear VERBATIM in the transcript view (a QUIZ line counts as the Include EVERY top-level field of the call even when empty — "edges": [] when there are no relations. NEVER omit a
transcript). The auditor rejects anything unbacked. When in doubt, report; the auditor decides. field because it is empty.
Include EVERY top-level field of the call even when empty — "edges": [], "attribution": { quiz: {} } when there
are no quizzes. NEVER omit a field because it is empty.
Your entire deliverable is ONE submit_findings call.`; Your entire deliverable is ONE submit_findings call.`;
@@ -375,10 +295,17 @@ async function runSubmitAgent<T>(opts: {
signal: AbortSignal | undefined; signal: AbortSignal | undefined;
onStep: (usage: Usage, model: string, step: number, finishReason: string, tools: string) => void; onStep: (usage: Usage, model: string, step: number, finishReason: string, tools: string) => void;
neverSubmitted: string; neverSubmitted: string;
/** Extra tools beside the submit call (the auditor's graph-exploration
* set, 2026-09-27): a small agentic loop instead of a bare retry. */
extraTools?: Record<string, AgentTool<any>>;
/** Turn budget for the loop (default 4 = retry-only; the auditor runs
* an exploration loop instead — 16). */
maxTurns?: number;
}): Promise<T> { }): Promise<T> {
let submitted: T | undefined; let submitted: T | undefined;
const result = await runSubagent( const result = await runSubagent(
[ [
...(opts.extraTools ? Object.values(opts.extraTools) : []),
{ {
name: opts.toolName, name: opts.toolName,
label: opts.toolName, label: opts.toolName,
@@ -395,7 +322,7 @@ async function runSubmitAgent<T>(opts: {
engine: opts.engine, engine: opts.engine,
systemPrompt: opts.systemPrompt, systemPrompt: opts.systemPrompt,
prompt: opts.prompt, prompt: opts.prompt,
maxTurns: 4, maxTurns: opts.maxTurns ?? 4,
abortSignal: opts.signal, abortSignal: opts.signal,
onAssistantMessage: (message, step) => onAssistantMessage: (message, step) =>
opts.onStep( opts.onStep(
@@ -450,8 +377,8 @@ function runNotesExtractor(
role: "extractor", role: "extractor",
toolName: "submit_findings", toolName: "submit_findings",
toolDescription: "Submit the notes findings.", toolDescription: "Submit the notes findings.",
parameters: notesFindingsSchema, parameters: findingsSchema,
fill: fillNotesFindings, fill: fillFindings,
systemPrompt: NOTES_EXTRACTOR_INSTRUCTIONS, systemPrompt: NOTES_EXTRACTOR_INSTRUCTIONS,
prompt, prompt,
engine, engine,
@@ -464,68 +391,52 @@ function runNotesExtractor(
// ─── the auditor ─────────────────────────────────────────────────────────── // ─── the auditor ───────────────────────────────────────────────────────────
const AUDITOR_INSTRUCTIONS = `You are the audit stage of a learning application's extraction pipeline. You receive const AUDITOR_INSTRUCTIONS = `You are the linking stage of a learning application's extraction pipeline. An extractor
ONE source — a session transcript, or the author's personal notes in an ARBITRARY format (the notes front door) — a has read a source (a teaching session, or the author's personal notes) and reported findings: propositions and
set of proposed findings (propositions, edges, and quiz attribution where the source has quiz lines), and the follows/supports edges. You NEVER see the source itself — the extractor is fully trusted, and you do not re-check
CURRENT knowledge graph. Your jobs: the faithfulness net, and identity. its work. Your job is IDENTITY: connecting those findings to the CURRENT knowledge graph, preparing the merge.
FAITHFULNESS — verify every candidate against the source text: You explore the graph through your tools: find (glob over names + statements + aliases), list_sessions, and
- PROPOSITIONS: every evidence quote must appear (near-)verbatim in the transcript view, and the statement list_propositions (full statement text, per session). The graph may be large — search, don't assume.
must be a faithful reading of it. The node theory applies: a statement must be a TRUE, self-contained,
world-facing fact — reword learner- or session-relative statements ("the learner understands X" → "X"), state
misconceptions as the true correction, or drop them when no objective fact is backed. Drop invented or
bloated statements. Conservative: when two proposed propositions are really one, keep the better-stated one
and drop the other.
- EDGES: the evidence must actually articulate the relation, in the direction claimed. Drop
direction-swapped or speculative edges.
- ATTRIBUTION: the QUIZ line's question must genuinely test the attributed proposition. Drop wrong
attributions to null, or re-point them to the proposition the question actually tests. (Session
sources only — notes findings carry no attribution; there is nothing to verify.)
- UNDERSTANDING levels: verify against the quiz outcomes and the session's evidence — or, for notes,
the author's own words; each note must be
exactly one word from the fixed scale ("solid", "shaky", "accepted", "resisted", "not yet engaged") —
reword or drop anything else. No justifications.
IDENTITY — link-or-create, per proposition: IDENTITY — link-or-create, per proposition:
- You see the current graph (every node: id, name, statement, aliases). For each verified proposition, - For each proposition, search the graph (its key words, its synonyms, its phrasings). Does an EXISTING node already
decide: does an EXISTING node already state this same proposition? state this same proposition?
- If yes: set "linkTo" to that node's id. The session's phrasings will join its aliases and its provenance; - If yes: set "linkTo" to that node's id. The session's phrasings join its aliases and its provenance; the node
the node keeps its existing name (labels stay stable across sessions — ignore your name for linked nodes). keeps its existing name (labels stay stable — ignore your name for linked nodes).
- If no: omit "linkTo" and a new node is created, with the extractor's name as its label. Reword only a - If no: omit "linkTo" and a new node is created with the extractor's name as its label.
clearly unglanceable name (an unwieldy sentence of a name); keep it short. - EQUIVALENCE BIAS (owner ruling): paraphrases, rewordings, and compressions of the same claim MERGE. When two
- When unsure, omit — duplicates are recoverable later; wrong merges are not. statements are the same claim at different precision or from different angles, LINK them. Only genuinely distinct
propositions — different claims about the world — stay apart. You are NOT guarding against wrong merges; you are
hunting for right ones.
Also drop propositions not referenced by anything else ONLY if they have no other backing — an unreferenced but BRIDGING EDGES — the one place you create edges:
well-evidenced proposition is still knowledge. - You may add edges whose endpoints are an EXISTING graph node id and one of the batch's propositions (use the
existing node's real id as "from" or "to"). This is how the new material gets welded into the old graph: when the
existing graph's content makes a proposition follow, or grounds it, draw the edge.
- Keep edges typed like the extractor's: "follows" = derivable; "supports" = grounds without entailing.
- NEVER drop or rewrite the extractor's own propositions or edges — pass them through unchanged (adding linkTo is
the only edit). Never invent propositions.
Never invent new candidates. Your entire deliverable is ONE submit_review call containing the verified findings Reproduce every proposition and edge you received, with their evidence and messages verbatim. Your entire deliverable
(the same shapes you received, with linkTo added) plus the dropped list. is ONE submit_review call.
Include EVERY top-level field of the call even when empty — "edges": [], "dropped": []. NEVER omit a field Include EVERY top-level field of the call even when empty — "edges": []. NEVER omit a field because it is empty.`;
because it is empty.`;
/** The current graph, compact — the auditor's identity context. */ /** The auditor — one shape for both front doors now (the notes schema was
function graphView(nodes: GraphNode[]): string { * identical once attribution died). An agentic loop: graph-exploration
if (nodes.length === 0) return "(the graph is currently empty — every proposition is a new node)"; * tools plus the one submit call. */
const lines = nodes.map(
(n) =>
`- ${n.id} :: ${n.name}\n statement: ${n.statement}` +
(n.aliases.length > 0 ? `\n aliases: ${n.aliases.join(" | ")}` : ""),
);
return lines.join("\n");
}
/** The auditor over session findings (the chat front door). */
function runAuditor( function runAuditor(
engine: PiEngine | undefined, engine: PiEngine | undefined,
prompt: string, prompt: string,
signal: AbortSignal | undefined, signal: AbortSignal | undefined,
onStep: (usage: Usage, model: string, step: number, finishReason: string, tools: string) => void, onStep: (usage: Usage, model: string, step: number, finishReason: string, tools: string) => void,
tools: Record<string, AgentTool<any>>,
): Promise<AuditResult> { ): Promise<AuditResult> {
return runSubmitAgent({ return runSubmitAgent({
role: "auditor", role: "auditor",
toolName: "submit_review", toolName: "submit_review",
toolDescription: "Submit the verified findings subset (with identity verdicts) + the dropped list.", toolDescription: "Submit the findings with identity verdicts (linkTo) and any bridging edges — for the merge.",
parameters: auditSchema, parameters: auditSchema,
fill: fillAudit, fill: fillAudit,
systemPrompt: AUDITOR_INSTRUCTIONS, systemPrompt: AUDITOR_INSTRUCTIONS,
@@ -534,29 +445,8 @@ function runAuditor(
signal, signal,
onStep, onStep,
neverSubmitted: "the auditor never submitted a review", neverSubmitted: "the auditor never submitted a review",
}); extraTools: tools,
} maxTurns: 16,
/** The auditor over notes findings (the format-agnostic front door): same
* faithfulness + identity job, notes-mode schema (no attribution). */
function runNotesAuditor(
engine: PiEngine | undefined,
prompt: string,
signal: AbortSignal | undefined,
onStep: (usage: Usage, model: string, step: number, finishReason: string, tools: string) => void,
): Promise<NotesAuditResult> {
return runSubmitAgent({
role: "auditor",
toolName: "submit_review",
toolDescription: "Submit the verified findings subset (with identity verdicts) + the dropped list.",
parameters: notesAuditSchema,
fill: fillNotesAudit,
systemPrompt: AUDITOR_INSTRUCTIONS,
prompt,
engine,
signal,
onStep,
neverSubmitted: "the auditor never submitted a review",
}); });
} }
@@ -683,7 +573,7 @@ export interface ExtractionPipeline {
/** The format-agnostic front door (2026-09-13): run the same pipeline /** The format-agnostic front door (2026-09-13): run the same pipeline
* over ONE raw notes file. The source id is deterministic per file * over ONE raw notes file. The source id is deterministic per file
* path, so re-importing the same file re-runs extraction against the * path, so re-importing the same file re-runs extraction against the
* same source identity (mappings + events dedupe by it). */ * same source identity (mappings dedupe by it). */
importNotes(input: NotesImportInput): Promise<EmissionStats>; importNotes(input: NotesImportInput): Promise<EmissionStats>;
} }
@@ -770,12 +660,12 @@ export function createExtractionPipeline(deps: ExtractionPipelineDeps): Extracti
async function extract(chatId: string, view: string): Promise<SessionFindings> { async function extract(chatId: string, view: string): Promise<SessionFindings> {
const signal = runningExtractions.get(chatId)?.controller.signal; const signal = runningExtractions.get(chatId)?.controller.signal;
signal?.throwIfAborted(); // an abort during a sync gap ends the run here signal?.throwIfAborted(); // an abort during a sync gap ends the run here
console.info(`[extraction ${chatId}] extractor starting (${view.split("\n").length} transcript lines)`); console.info(`[extraction ${chatId}] extractor starting (${view.split("\n").length} markdown lines)`);
const t0 = performance.now(); const t0 = performance.now();
const findings = await runExtractor( const findings = await runExtractor(
stageEngine("extractor", deps.extractor), stageEngine("extractor", deps.extractor),
`Transcript:\n\n${view}`, view,
signal, signal,
(usage, model, step, finishReason, tools) => { (usage, model, step, finishReason, tools) => {
recordStep(chatId, "extractor", callUsage(usage), model); recordStep(chatId, "extractor", callUsage(usage), model);
@@ -786,36 +676,36 @@ export function createExtractionPipeline(deps: ExtractionPipelineDeps): Extracti
return findings; return findings;
} }
async function audit(chatId: string, findings: SessionFindings, view: string): Promise<AuditResult> { async function audit(chatId: string, findings: SessionFindings): Promise<AuditResult> {
const signal = runningExtractions.get(chatId)?.controller.signal; const signal = runningExtractions.get(chatId)?.controller.signal;
signal?.throwIfAborted(); // an abort during the extract→audit gap ends here signal?.throwIfAborted(); // an abort during the extract→audit gap ends here
const graph = getGraph(graphStore); const graph = getGraph(graphStore);
console.info( console.info(
`[extraction ${chatId}] auditor starting (${findings.propositions.length} propositions, ${findings.edges.length} edges, ` + `[extraction ${chatId}] auditor starting (${findings.propositions.length} propositions, ${findings.edges.length} edges, ` +
`${Object.keys(findings.attribution.quiz).length} attributions, ${graph.nodes.length} graph nodes)`, `${graph.nodes.length} graph nodes)`,
); );
const t0 = performance.now(); const t0 = performance.now();
// The auditor NEVER sees the source (owner 2026-09-27): findings
// JSON + identity instructions, and the graph reached through
// exploration tools instead of an injected dump.
const auditResult = await runAuditor( const auditResult = await runAuditor(
stageEngine("auditor", deps.auditor), stageEngine("auditor", deps.auditor),
`Transcript:\n\n${view}\n\nProposed findings:\n\n${JSON.stringify(findings, null, 2)}\n\n` + `Proposed findings from the extractor (the source itself is not available to you — the extractor is fully trusted):\n\n` +
`Current knowledge graph (${graph.nodes.length} nodes):\n\n${graphView(graph.nodes)}`, `${JSON.stringify(findings, null, 2)}\n\n` +
`The current knowledge graph has ${graph.nodes.length} node(s). Explore it with your tools and link what links.`,
signal, signal,
(usage, model, step, finishReason, tools) => { (usage, model, step, finishReason, tools) => {
recordStep(chatId, "auditor", callUsage(usage), model); recordStep(chatId, "auditor", callUsage(usage), model);
console.info(`[extraction] auditor step ${step}: ${finishReason}, tools: ${tools}`); console.info(`[extraction] auditor step ${step}: ${finishReason}, tools: ${tools}`);
}, },
auditorGraphTools({ chatStore, graphStore }),
); );
if (auditResult.dropped.length > 0) { const bridged = auditResult.edges.filter((e) => !e.from.startsWith("p") || !e.to.startsWith("p")).length;
console.info(
`[extraction ${chatId}] audit dropped ${auditResult.dropped.length}: ` +
auditResult.dropped.map((d) => `${d.ref} (${d.reason})`).join("; "),
);
}
console.info( console.info(
`[extraction ${chatId}] audit kept ${auditResult.propositions.length}/${findings.propositions.length} propositions, ` + `[extraction ${chatId}] audit: ${auditResult.propositions.length} propositions through, ` +
`${auditResult.edges.length}/${findings.edges.length} edges, ` + `${auditResult.propositions.filter((p) => p.linkTo !== undefined).length} linked to existing nodes, ` +
`${auditResult.propositions.filter((p) => p.linkTo !== undefined).length} linked to existing nodes`, `${auditResult.edges.length} edges (${bridged} bridging)`,
); );
console.info(`[extraction ${chatId}] auditor done in ${Math.round(performance.now() - t0)}ms`); console.info(`[extraction ${chatId}] auditor done in ${Math.round(performance.now() - t0)}ms`);
return auditResult; return auditResult;
@@ -848,7 +738,7 @@ export function createExtractionPipeline(deps: ExtractionPipelineDeps): Extracti
} }
async function runExtraction(chatId: string): Promise<EmissionStats> { async function runExtraction(chatId: string): Promise<EmissionStats> {
if (inFlight.has(chatId)) return { nodes: 0, edges: 0, events: 0 }; if (inFlight.has(chatId)) return { nodes: 0, edges: 0 };
inFlight.add(chatId); inFlight.add(chatId);
const t0 = performance.now(); const t0 = performance.now();
// COVERAGE (vision-extraction-coverage.md, 2026-09-15): extract ONLY // COVERAGE (vision-extraction-coverage.md, 2026-09-15): extract ONLY
@@ -859,7 +749,7 @@ export function createExtractionPipeline(deps: ExtractionPipelineDeps): Extracti
// error below loses at most the in-flight batch — every earlier // error below loses at most the in-flight batch — every earlier
// batch stays covered, the rest stays visibly uncovered. // batch stays covered, the rest stays visibly uncovered.
let batchesDone = 0; let batchesDone = 0;
const total: EmissionStats = { nodes: 0, edges: 0, events: 0 }; const total: EmissionStats = { nodes: 0, edges: 0 };
try { try {
const record = chatStore.getChat(chatId); const record = chatStore.getChat(chatId);
if (!record) throw new Error(`Chat not found: ${chatId}`); if (!record) throw new Error(`Chat not found: ${chatId}`);
@@ -871,7 +761,7 @@ export function createExtractionPipeline(deps: ExtractionPipelineDeps): Extracti
const covered = coverageOf(record.messages, graphStore.readCoverage(), chatId); const covered = coverageOf(record.messages, graphStore.readCoverage(), chatId);
if (covered.uncovered.length === 0) { if (covered.uncovered.length === 0) {
console.info(`[extraction ${chatId}] nothing uncovered — no-op`); console.info(`[extraction ${chatId}] nothing uncovered — no-op`);
return { nodes: 0, edges: 0, events: 0 }; return { nodes: 0, edges: 0 };
} }
const chunkTokens = deps.getChunkTokens?.() ?? FACTORY_SETTINGS.graph.extractionChunkTokens; const chunkTokens = deps.getChunkTokens?.() ?? FACTORY_SETTINGS.graph.extractionChunkTokens;
// The batcher's one rule (owner, Wave D): a tool-call message and // The batcher's one rule (owner, Wave D): a tool-call message and
@@ -883,17 +773,19 @@ export function createExtractionPipeline(deps: ExtractionPipelineDeps): Extracti
`${covered.uncoveredCount} messages, ${batches.length} batch(es) of ≤${chunkTokens} tok`, `${covered.uncoveredCount} messages, ${batches.length} batch(es) of ≤${chunkTokens} tok`,
); );
for (const batch of batches) { for (const batch of batches) {
// The batch's prompt view: SESSION header + the batch's lines, // The batch's source view: the ONE shared markdown engine
// same provenance [id] tags the fingerprints used (coverage // (owner 2026-09-27) over the batch's messages — the same
// and prompt share ONE line builder — core/coverage.ts). // rendering the user-facing session export will use, id
const view = [ // anchors preserved. The fingerprint's line format is
`SESSION: ${record.title}`, // separate and unchanged (no mass re-extraction).
...batch.flatMap((slot) => messageViewLines(slot.message, slot.message.id)), const view = sessionMarkdown(
].join("\n"); batch.map((slot) => slot.message),
{ title: record.title },
);
setPhase(chatId, "extracting", "chat"); setPhase(chatId, "extracting", "chat");
const findings = await extract(chatId, view); const findings = await extract(chatId, view);
setPhase(chatId, "auditing", "chat"); setPhase(chatId, "auditing", "chat");
const auditResult = await audit(chatId, findings, view); const auditResult = await audit(chatId, findings);
setPhase(chatId, "merging", "chat"); setPhase(chatId, "merging", "chat");
// SESSION-DELETION GUARD (wave 6): a chat deleted mid-run must never // SESSION-DELETION GUARD (wave 6): a chat deleted mid-run must never
// merge. The deletion route aborts this run first and runs its purge // merge. The deletion route aborts this run first and runs its purge
@@ -911,20 +803,17 @@ export function createExtractionPipeline(deps: ExtractionPipelineDeps): Extracti
// one and it re-opens: honest by construction. // one and it re-opens: honest by construction.
const foldedAt = new Date().toISOString(); const foldedAt = new Date().toISOString();
const stats = mergeFindings(chatId, auditResult, { chatStore, graphStore }, { const stats = mergeFindings(chatId, auditResult, { chatStore, graphStore }, {
messages: batch.map((slot) => slot.message),
foldedAt, foldedAt,
coverage: batch coverage: batch
.filter((slot) => messageViewLines(slot.message, slot.message.id).length > 0) .filter((slot) => messageViewLines(slot.message, slot.message.id).length > 0)
.map((slot) => ({ chatId, fp: slot.fp, tokens: slot.tokens, foldedAt })), .map((slot) => ({ chatId, fp: slot.fp, tokens: slot.tokens, foldedAt })),
batchTag: batchTag(batch),
}); });
total.nodes += stats.nodes; total.nodes += stats.nodes;
total.edges += stats.edges; total.edges += stats.edges;
total.events += stats.events;
batchesDone++; batchesDone++;
console.info( console.info(
`[extraction ${chatId}] batch ${batchesDone}/${batches.length} merged: ` + `[extraction ${chatId}] batch ${batchesDone}/${batches.length} merged: ` +
`${stats.nodes} nodes, ${stats.edges} edges, ${stats.events} events`, `${stats.nodes} nodes, ${stats.edges} edges`,
); );
} }
const stats = total; const stats = total;
@@ -933,7 +822,7 @@ export function createExtractionPipeline(deps: ExtractionPipelineDeps): Extracti
settleGraphLayout(graphStore); settleGraphLayout(graphStore);
persistRunUsage(chatId, Math.round(performance.now() - t0), stats); persistRunUsage(chatId, Math.round(performance.now() - t0), stats);
console.info( console.info(
`[extraction ${chatId}] merged: ${stats.nodes} nodes, ${stats.edges} edges, ${stats.events} events ` + `[extraction ${chatId}] merged: ${stats.nodes} nodes, ${stats.edges} edges ` +
`(${Math.round(performance.now() - t0)}ms total)`, `(${Math.round(performance.now() - t0)}ms total)`,
); );
return stats; return stats;
@@ -989,42 +878,12 @@ export function createExtractionPipeline(deps: ExtractionPipelineDeps): Extracti
return findings; return findings;
} }
async function notesAudit(input: { id: string; filePath: string; text: string }, findings: NotesFindings): Promise<NotesAuditResult> { // The notes auditor is the SAME identity-linker as the chat one (one
const signal = runningExtractions.get(input.id)?.controller.signal; // schema now that attribution is gone) — audit() above, verbatim.
signal?.throwIfAborted(); // an abort during the extract→audit gap ends here
const graph = getGraph(graphStore);
console.info(
`[extraction ${input.id}] notes auditor starting (${findings.propositions.length} propositions, ` +
`${findings.edges.length} edges, ${graph.nodes.length} graph nodes)`,
);
const t0 = performance.now();
const auditResult = await runNotesAuditor(
stageEngine("auditor", deps.auditor),
`Source:\n\n${notesView(input)}\n\nProposed findings:\n\n${JSON.stringify(findings, null, 2)}\n\n` +
`Current knowledge graph (${graph.nodes.length} nodes):\n\n${graphView(graph.nodes)}`,
signal,
(usage, model, step, finishReason, tools) => {
recordStep(input.id, "auditor", callUsage(usage), model);
console.info(`[extraction] auditor step ${step}: ${finishReason}, tools: ${tools}`);
},
);
if (auditResult.dropped.length > 0) {
console.info(
`[extraction ${input.id}] audit dropped ${auditResult.dropped.length}: ` +
auditResult.dropped.map((d) => `${d.ref} (${d.reason})`).join("; "),
);
}
console.info(
`[extraction ${input.id}] audit kept ${auditResult.propositions.length}/${findings.propositions.length} propositions, ` +
`${auditResult.propositions.filter((p) => p.linkTo !== undefined).length} linked to existing nodes`,
);
console.info(`[extraction ${input.id}] notes auditor done in ${Math.round(performance.now() - t0)}ms`);
return auditResult;
}
async function runNotesExtraction(input: { id: string; filePath: string; text: string; importedAt: string }): Promise<EmissionStats> { async function runNotesExtraction(input: { id: string; filePath: string; text: string; importedAt: string }): Promise<EmissionStats> {
const { id } = input; const { id } = input;
if (inFlight.has(id)) return { nodes: 0, edges: 0, events: 0 }; if (inFlight.has(id)) return { nodes: 0, edges: 0 };
inFlight.add(id); inFlight.add(id);
const t0 = performance.now(); const t0 = performance.now();
try { try {
@@ -1036,18 +895,14 @@ export function createExtractionPipeline(deps: ExtractionPipelineDeps): Extracti
setPhase(id, "extracting", "notes", input.filePath); setPhase(id, "extracting", "notes", input.filePath);
const findings = await notesExtract(input); const findings = await notesExtract(input);
setPhase(id, "auditing", "notes", input.filePath); setPhase(id, "auditing", "notes", input.filePath);
const notesResult = await notesAudit(input, findings); const auditResult = await audit(id, findings);
setPhase(id, "merging", "notes", input.filePath); setPhase(id, "merging", "notes", input.filePath);
// Normalize to the merger's findings shape: notes carry no quiz const stats = mergeFindings(id, auditResult, { chatStore, graphStore }, { foldedAt: input.importedAt });
// attribution, and the merger scans no quiz parts for notes
// (messages: null) — nothing to attribute, nothing to build.
const auditResult: AuditResult = { ...notesResult, attribution: { quiz: {} } };
const stats = mergeFindings(id, auditResult, { chatStore, graphStore }, { messages: null, foldedAt: input.importedAt });
// Same post-merge settle as the chat path. // Same post-merge settle as the chat path.
settleGraphLayout(graphStore); settleGraphLayout(graphStore);
persistRunUsage(id, Math.round(performance.now() - t0), stats); persistRunUsage(id, Math.round(performance.now() - t0), stats);
console.info( console.info(
`[extraction ${id}] merged: ${stats.nodes} nodes, ${stats.edges} edges, ${stats.events} events ` + `[extraction ${id}] merged: ${stats.nodes} nodes, ${stats.edges} edges ` +
`(${Math.round(performance.now() - t0)}ms total)`, `(${Math.round(performance.now() - t0)}ms total)`,
); );
return stats; return stats;
@@ -1055,7 +910,7 @@ export function createExtractionPipeline(deps: ExtractionPipelineDeps): Extracti
persistRunUsage(id, Math.round(performance.now() - t0), undefined); persistRunUsage(id, Math.round(performance.now() - t0), undefined);
if (err instanceof Error && err.name === "AbortError") { if (err instanceof Error && err.name === "AbortError") {
console.info(`[extraction ${id}] aborted (${Math.round(performance.now() - t0)}ms) — nothing merged, nothing checkpointed`); console.info(`[extraction ${id}] aborted (${Math.round(performance.now() - t0)}ms) — nothing merged, nothing checkpointed`);
return { nodes: 0, edges: 0, events: 0 }; return { nodes: 0, edges: 0 };
} }
console.error(`[extraction ${id}] notes pipeline failed (${Math.round(performance.now() - t0)}ms):`, err); console.error(`[extraction ${id}] notes pipeline failed (${Math.round(performance.now() - t0)}ms):`, err);
throw err; throw err;
+95 -6
View File
@@ -1,7 +1,7 @@
import { Type } from "typebox"; import { Type } from "typebox";
import type { AgentTool } from "@earendil-works/pi-agent-core"; import type { AgentTool } from "@earendil-works/pi-agent-core";
import { serverExecute } from "../tool-execute.ts"; import { serverExecute } from "../tool-execute.ts";
import { graphSessionIndex, graphSessionPropositions } from "../../core/graph.ts"; import { getGraph, graphSessionIndex, graphSessionPropositions } from "../../core/graph.ts";
import type { ChatStore, GraphStore } from "../../core/ports.ts"; import type { ChatStore, GraphStore } from "../../core/ports.ts";
/** /**
@@ -63,18 +63,107 @@ export function proberGraphTools(ctx: GraphToolContext): Record<string, AgentToo
name: "list_propositions", name: "list_propositions",
label: "list_propositions", label: "list_propositions",
description: description:
"List the propositions BORN in one knowledge-graph session — what was first learned there — each with his latest understanding level on the scale solid / shaky / accepted / resisted / not yet engaged, and the date that level was set. " + "List the propositions BORN in one knowledge-graph session — what was covered there. " +
"Names, levels, and dating only — no statement text, no session contents. " + "Names only — no statement text, no session contents (owner ruling 2026-09-23: the map's ORGANIZATION is your business, its prose is not).",
"The level is that session's own latest evaluation; other sessions may have moved it since, so cross-session levels can differ.",
parameters: listPropositionsInput, parameters: listPropositionsInput,
execute: serverExecute("list_propositions", listPropositionsInput, async ({ session }): Promise<string> => { execute: serverExecute("list_propositions", listPropositionsInput, async ({ session }): Promise<string> => {
const propositions = graphSessionPropositions(deps, session); const propositions = graphSessionPropositions(deps, session);
if (propositions.length === 0) { if (propositions.length === 0) {
return `No propositions born in session "${session}". Call list_sessions for the current ids.`; return `No propositions born in session "${session}". Call list_sessions for the current ids.`;
} }
const lines = propositions.map( const lines = propositions.map((p) => `- ${p.name}`);
(p) => `- ${p.name} — ${p.understanding ?? "never evaluated"}${p.evaluatedAt ? ` (as of ${p.evaluatedAt})` : ""}`, return `${propositions.length} proposition(s) covered in this session:\n${lines.join("\n")}`;
}),
},
};
}
// ─── the auditor's graph tools (2026-09-27 relink) ───────────────────────────
/**
* THE AUDITOR'S TOOLSET — the extraction auditor explores the graph to
* place new propositions instead of receiving it injected (owner ruling
* 2026-09-27: a full-graph dump bloats context as the graph grows; the
* auditor looks up identity candidates itself). Auditor-specific
* versions, deliberately different from the prober's:
*
* find glob over node names + statements + aliases — the
* identity workhorse; returns id + name + statement
* (statement text is ESSENTIAL for identity work,
* unlike the prober's names-only rule).
* list_sessions the birth-session index, no timestamps.
* list_propositions full proposition text (same reasoning as find).
*
* No timestamps anywhere: identity work needs content, not chronology.
* Read-only, pure core seams, same ports the snapshot reads.
*/
/** Glob match — `*` spans anything, case-insensitive. `*needle*` is the
* expected shape; a bare term gets implicit surrounding stars. */
function globMatch(pattern: string, text: string): boolean {
const raw = pattern.includes("*") ? pattern : `*${pattern}*`;
const re = new RegExp(
`^${raw.replace(/[.*+?^${}()|[\]\\]/g, (ch) => (ch === "*" ? "[\\s\\S]*" : `\\${ch}`))}$`,
"i",
);
return re.test(text);
}
export function auditorGraphTools(ctx: GraphToolContext): Record<string, AgentTool<any>> {
const deps = { chatStore: ctx.chatStore, graphStore: ctx.graphStore };
const findInput = Type.Object({
query: Type.String({
minLength: 1,
description: "Glob pattern over node names, statements, and aliases — e.g. *will to power*. A bare term gets implicit surrounding *.",
}),
});
return {
find: {
name: "find",
label: "find",
description:
"Search the knowledge graph for existing propositions — glob over names, statements, and aliases. " +
"Returns each match's id, name, and full statement. This is how you decide whether a new proposition IS an existing one: search its key words, its synonyms, its phrasings.",
parameters: findInput,
execute: serverExecute("find", findInput, async ({ query }): Promise<string> => {
const nodes = getGraph(deps.graphStore).nodes;
const hits = nodes.filter(
(n) => globMatch(query, n.name) || globMatch(query, n.statement) || n.aliases.some((a) => globMatch(query, a)),
); );
if (hits.length === 0) return `No existing proposition matches "${query}".`;
const lines = hits.map((n) => `- ${n.id} :: ${n.name}\n statement: ${n.statement}`);
return `${hits.length} match(es):\n${lines.join("\n")}`;
}),
},
list_sessions: {
name: "list_sessions",
label: "list_sessions",
description:
"List the knowledge graph's birth sessions — every chat or imported-notes file that ever contributed a proposition. " +
"Each line: name, kind, proposition count, id. Use the id with list_propositions to see what a session covered.",
parameters: Type.Object({}),
execute: serverExecute("list_sessions", Type.Object({}), async (): Promise<string> => {
const sessions = graphSessionIndex(deps);
if (sessions.length === 0) return "The knowledge graph is empty — no session has contributed propositions yet.";
const lines = sessions.map((s, i) => `${i + 1}. ${s.name} (${s.kind}, ${s.propositions} proposition${s.propositions === 1 ? "" : "s"}) — id: ${s.id}`);
return `${sessions.length} session(s):\n${lines.join("\n")}`;
}),
},
list_propositions: {
name: "list_propositions",
label: "list_propositions",
description:
"List the propositions BORN in one session, with FULL statement text — the content-level view for identity work " +
"(deciding whether a new proposition is really one of these).",
parameters: listPropositionsInput,
execute: serverExecute("list_propositions", listPropositionsInput, async ({ session }): Promise<string> => {
const propositions = graphSessionPropositions(deps, session);
if (propositions.length === 0) return `No propositions born in session "${session}". Call list_sessions for the current ids.`;
const nodes = new Map(getGraph(deps.graphStore).nodes.map((n) => [n.id, n]));
const lines = propositions.map((p) => {
const node = nodes.get(p.nodeId);
return `- ${p.nodeId} :: ${p.name}\n statement: ${node?.statement ?? "(unknown)"}`;
});
return `${propositions.length} proposition(s) born in this session:\n${lines.join("\n")}`; return `${propositions.length} proposition(s) born in this session:\n${lines.join("\n")}`;
}), }),
}, },
+1 -1
View File
@@ -9,7 +9,7 @@ import type { ThinkingLevel } from "@earendil-works/pi-agent-core";
import type { LhMessage } from "../native-messages.ts"; import type { LhMessage } from "../native-messages.ts";
import type { Phase3 } from "../phases.ts"; import type { Phase3 } from "../phases.ts";
import type { PlanNodeStatus, PlanTree } from "./plan.ts"; import type { PlanNodeStatus, PlanTree } from "./plan.ts";
import type { EmissionStats, UnderstandingLevel } from "./graph.ts"; import type { EmissionStats } from "./graph.ts";
// ─── Chat records (the session store's on-disk format) ─────────────────────── // ─── Chat records (the session store's on-disk format) ───────────────────────
+43 -98
View File
@@ -5,10 +5,11 @@
* 2026-09-02 redesign): every node is a TRUE proposition; edges are follows * 2026-09-02 redesign): every node is a TRUE proposition; edges are follows
* (derivable) and supports (grounds without entailing); provenance is * (derivable) and supports (grounds without entailing); provenance is
* message-granular; extraction is free-form and post-hoc, fully decoupled * message-granular; extraction is free-form and post-hoc, fully decoupled
* from the session's plan tree; the event ledger records quiz outcomes plus * from the session's plan tree. The graph is an extraction of the reasoning
* the extractor's per-proposition understanding observations. Anything not * chains from all teaching sessions (and imported notes) — NOTHING else:
* yet decided (the state model, strength derivation, understanding * no quiz outcomes, no understanding assessments (2026-09-27 cleanup —
* treatment, selection strategies) is deliberately NOT here — transitional * understanding becomes practice-module territory). Anything not
* yet decided (the state model, strength derivation, selection strategies) is deliberately NOT here — transitional
* stubs carry the renderer until those land. * stubs carry the renderer until those land.
* *
* Part of the domain-split contracts (src/core/contracts/): each domain file * Part of the domain-split contracts (src/core/contracts/): each domain file
@@ -31,13 +32,13 @@ export interface EmissionStats {
nodes: number; nodes: number;
/** New edge assertions appended. */ /** New edge assertions appended. */
edges: number; edges: number;
/** Ledger events rebuilt (quiz outcomes + observations). */
events: number;
} }
/** One provenance point — message-granular (DECIDED 2026-09-02): nodes and /** One provenance point — message-granular (DECIDED 2026-09-02): nodes and
* edges record every (chat, message) where they came up, deep-linkable to * edges record every (chat, message) where they came up, deep-linkable to
* the exact transcript point. */ * the exact transcript point. Auditor-drawn bridging edges carry the chat
* but NO message anchor (nothing in the transcript asserts them — the
* auditor is the asserter); messageId is "" there. */
export interface ProvenancePoint { export interface ProvenancePoint {
/** The chat this point anchors to — or a notes source id ("src_…"): /** The chat this point anchors to — or a notes source id ("src_…"):
* imports carry no message anchors (provenance is the file itself, * imports carry no message anchors (provenance is the file itself,
@@ -50,11 +51,10 @@ export interface ProvenancePoint {
* (DECIDED 2026-09-13): personal notes in ARBITRARY formats run through the * (DECIDED 2026-09-13): personal notes in ARBITRARY formats run through the
* same extractor → auditor → merger pipeline, with the file itself as the * same extractor → auditor → merger pipeline, with the file itself as the
* provenance anchor. There is no chat: the detail card shows the file path * provenance anchor. There is no chat: the detail card shows the file path
* as a STATIC entry (not a link), and understanding observations still * as a STATIC entry (not a link). The run's usage log lives here,
* apply (judged from the author's own voice in the notes). The run's usage * mirroring ChatRecord.extractions. Ids are "src_" + hash of the file
* log lives here, mirroring ChatRecord.extractions. Ids are "src_" + hash * path — deterministic, so re-importing the same file re-runs extraction
* of the file path — deterministic, so re-importing the same file re-runs * against the same source id (mappings dedupe by it). */
* extraction against the same source id (mappings/events dedupe by it). */
export interface NotesSource { export interface NotesSource {
id: string; id: string;
filePath: string; filePath: string;
@@ -149,39 +149,9 @@ export interface CoverageEntry {
foldedAt: string; foldedAt: string;
} }
/**
* Event kinds (DECIDED 2026-09-02): quiz outcomes only — correct / miss /
* dont-know — plus `observation`, the extractor's per-proposition
* understanding note joining the ledger as a session-scoped observation
* (named-word scale; treatment pending the state-model ideation). No
* `taught` events at all; no misconception kind — a misconception is a
* TRUE correction in the graph, and the wrongness lives here, in the
* ledger, until the state model exists.
*/
export type LedgerEventKind = "correct" | "miss" | "dont-know" | "observation";
export interface LedgerEvent {
/** null = not yet attributed. */
nodeId: string | null;
kind: LedgerEventKind;
/** The chat that produced this evidence — or a notes source id
* ("src_…") for imported-notes observations. */
chatId: string;
at: string;
/** Idempotency key: the quiz part's toolCallId, or `obs:<sourceId>` for
* understanding observations — re-emission of the same chat must never
* double-record an event. */
ref?: string;
/** The learner's quiz note (diagnostic gold), or the understanding
* LEVEL word for observations — a session-relative judgment. */
note?: string;
/** The quiz question text, when nodeId is null (later attribution input). */
question?: string;
}
/** /**
* Store format (src/store/graph.ts): JSONL ledgers at config.LEDGER_FILES * Store format (src/store/graph.ts): JSONL ledgers at config.LEDGER_FILES
* (nodes/edges/events/mappings/sessions), all append-only. * (nodes/edges/mappings/sessions), all append-only.
* *
* nodes.jsonl one line per node — the FIRST line for an id defines * nodes.jsonl one line per node — the FIRST line for an id defines
* name/statement/createdAt; later lines for the * name/statement/createdAt; later lines for the
@@ -198,27 +168,23 @@ export interface LedgerEvent {
* (never extract the same turns twice). * (never extract the same turns twice).
* *
* The EXTRACTION PIPELINE is the sole graph writer (extractor → auditor → * The EXTRACTION PIPELINE is the sole graph writer (extractor → auditor →
* merger), tree-blind and post-hoc: the extractor reads only the transcript * merger), tree-blind and post-hoc (2026-09-27 cleanup — clear separation
* and proposes propositions/edges/quiz-attribution with verbatim evidence; * of work): the extractor is the ONLY stage that sees the raw source
* the auditor sees the extractor's output PLUS the current graph, owns * (whole session as readable markdown, or raw notes) and extracts the
* faithfulness (evidence check) and identity (link-or-create: point each * reasoning chains — propositions (lonely relevant ones included) plus
* proposition at the existing node it is, or let it be created — recording * follows/supports edges, with verbatim evidence; the auditor NEVER sees
* the session's phrasings as aliases); the merger (core/graph.ts, * the source — its job is preparing the merge: identity verdicts
* deterministic) applies the verdicts and appends. * (link-or-create, merging equivalent forms of the same proposition) via
* graph-exploration tools (find / list_sessions / list_propositions), and
* it may draw BRIDGING edges from existing graph nodes to the new
* propositions; the merger (core/graph.ts, deterministic) applies the
* verdicts and appends. Quiz events and understanding observations are
* GONE — the pipeline is the only session→graph bridge, and it carries
* propositions + edges, nothing else.
*/ */
// ─── Extraction findings (extractor output → audit → merger input) ─────────── // ─── Extraction findings (extractor output → audit → merger input) ───────────
/**
* The extractor's standardized per-session understanding scale (DECIDED
* 2026-09-08): a CLOSED named-word list — no numbers, no prose, no
* justifications (a justification, if the agent wants one, is dropped;
* it is not stored). The level is session- and extraction-run-relative
* judgment about the learner, shown next to its session — never a node
* property.
*/
export type UnderstandingLevel = "solid" | "shaky" | "accepted" | "resisted" | "not yet engaged";
/** One proposed proposition — a node candidate (or a re-encounter of one). */ /** One proposed proposition — a node candidate (or a re-encounter of one). */
export interface ExtractedProposition { export interface ExtractedProposition {
/** Extractor-local id ("p1", …) referenced by edges + attribution. */ /** Extractor-local id ("p1", …) referenced by edges + attribution. */
@@ -231,7 +197,8 @@ export interface ExtractedProposition {
/** The minimal proposition — a true, self-contained, world-facing /** The minimal proposition — a true, self-contained, world-facing
* statement, at full precision. No learner / teacher / session * statement, at full precision. No learner / teacher / session
* references: misconceptions are stated as the true correction, and * references: misconceptions are stated as the true correction, and
* wrongness lives in the event ledger, never in the node. */ * wrongness is never recorded in the graph (2026-09-27: the event
* ledger is gone). */
statement: string; statement: string;
/** Verbatim phrasings the session used. */ /** Verbatim phrasings the session used. */
aliases: string[]; aliases: string[];
@@ -239,17 +206,17 @@ export interface ExtractedProposition {
messages: string[]; messages: string[];
/** Verbatim transcript quotes backing the proposition. */ /** Verbatim transcript quotes backing the proposition. */
evidence: string[]; evidence: string[];
/** The extractor's understanding level for this proposition IN THIS
* session — one word from the UnderstandingLevel scale. Joins the
* ledger as an observation event (note = the level word); treatment
* pending the state-model ideation. */
understanding?: UnderstandingLevel;
} }
/** One proposed relation between two propositions. */ /** One proposed relation. Endpoints are extractor-local proposition ids
* ("p1", …); the AUDITOR may also emit bridging edges whose from/to is an
* EXISTING graph node id (connecting the new content to what the graph
* already holds) — the merger resolves both. */
export interface ExtractedEdge { export interface ExtractedEdge {
/** Proposition id that is followed / supported. */ /** Proposition id (or, for auditor-drawn bridges, a graph node id)
* that is followed / supported. */
from: string; from: string;
/** Proposition id (or graph node id) that follows / is supported. */
to: string; to: string;
type: EdgeType; type: EdgeType;
/** The message ids where this relation was asserted — provenance. */ /** The message ids where this relation was asserted — provenance. */
@@ -258,26 +225,10 @@ export interface ExtractedEdge {
evidence: string[]; evidence: string[];
} }
/** /** The extractor's output shape: the raw reasoning chains. */
* The extractor's quiz attribution — its ONLY say about quiz events. Event
* kinds, timestamps, idempotency refs, notes and question texts are all
* rebuilt deterministically by the merger from the transcript's quiz tool
* parts; the LLM supplies exactly the judgment call: WHICH proposition each
* quiz tested. Tree-blind, so EVERY quiz attributes here (there is no tree
* mapping to lean on) — retroactively against the whole transcript.
*/
export interface ExtractionAttribution {
/** quiz toolCallId → proposition id; null / missing = unattributed (the
* event lands with nodeId null + the question text, a later-
* attribution input). */
quiz: Record<string, string | null>;
}
/** The extractor's output shape. */
export interface SessionFindings { export interface SessionFindings {
propositions: ExtractedProposition[]; propositions: ExtractedProposition[];
edges: ExtractedEdge[]; edges: ExtractedEdge[];
attribution: ExtractionAttribution;
} }
/** The auditor's verdict on one proposition: the findings, plus `linkTo` — /** The auditor's verdict on one proposition: the findings, plus `linkTo` —
@@ -286,20 +237,19 @@ export interface AuditedProposition extends ExtractedProposition {
/** The existing GraphNode id this proposition IS — the auditor's /** The existing GraphNode id this proposition IS — the auditor's
* identity verdict. On link, the session's phrasings (statement, * identity verdict. On link, the session's phrasings (statement,
* aliases) join the node's aliases and the provenance unions. * aliases) join the node's aliases and the provenance unions.
* Conservative: when unsure, leave it out — duplicates over wrong * EQUIVALENCE BIAS (2026-09-27): paraphrases, rewordings, and
* merges. */ * compressions of the same claim MERGE — do not create a duplicate
* because the wording differs. Only genuinely distinct propositions
* stay apart. */
linkTo?: string; linkTo?: string;
} }
/** The auditor's output: the verified findings (with identity verdicts), /** The auditor's output: the findings with identity verdicts, plus any
* plus the drops (the audit trail of what was rejected and why). */ * bridging edges drawn to existing graph nodes. No drop channel — the
* auditor never rejects; the extractor is fully trusted (2026-09-27). */
export interface AuditResult { export interface AuditResult {
propositions: AuditedProposition[]; propositions: AuditedProposition[];
edges: ExtractedEdge[]; edges: ExtractedEdge[];
attribution: ExtractionAttribution;
/** Dropped candidates: proposition id, edge "from→to", or attribution
* toolCallId — with the reason. */
dropped: { ref: string; reason: string }[];
} }
/** The GET /api/graph payload — everything the home view's PURE renderer needs. */ /** The GET /api/graph payload — everything the home view's PURE renderer needs. */
@@ -328,11 +278,6 @@ export interface GraphSnapshot {
fileSources: Record<string, string>; fileSources: Record<string, string>;
/** nodeId → chatIds that touched the node (click-through provenance). */ /** nodeId → chatIds that touched the node (click-through provenance). */
nodeSessions: Record<string, string[]>; nodeSessions: Record<string, string[]>;
/** nodeId → chatId → that session's understanding observation (the
* extractor's per-session named-word note; session-relative data,
* shown next to the session list — the confidence history). Multiple
* notes for one (node, chat) collapse to the latest. */
nodeObservations: Record<string, Record<string, string>>;
/** Structural communities (Louvain over follows+supports), size ≥ 3, /** Structural communities (Louvain over follows+supports), size ≥ 3,
* joined with persisted LLM labelings. Purely structural — never * joined with persisted LLM labelings. Purely structural — never
* depends on layout. */ * depends on layout. */
+8 -35
View File
@@ -13,17 +13,16 @@
* *
* preserve-graph the chat's extraction bookkeeping dies — its coverage * preserve-graph the chat's extraction bookkeeping dies — its coverage
* rows and its legacy checkpoint lines — and NOTHING else: * rows and its legacy checkpoint lines — and NOTHING else:
* nodes, edges, events, mappings stay byte-identical. * nodes, edges, mappings stay byte-identical.
* The graph keeps the session's contributions; the * The graph keeps the session's contributions; the
* snapshot simply no longer resolves the dead chatId to * snapshot simply no longer resolves the dead chatId to
* a title (nodeSessions/nodeObservations filter unknown * a title (nodeSessions filters unknown ids at read
* ids at read time — no UI ghosting). * time — no UI ghosting).
* *
* delete-nodes "this session never happened": every trace of the * delete-nodes "this session never happened": every trace of the
* chat leaves the ledgers — its provenance points on * chat leaves the ledgers — its provenance points on
* every node and edge line, its link lines (their * every node and edge line, its link lines (their
* phrasings die with them), its quiz events and * phrasings die with them), its mapping lines (the audit trail), its
* observations, its mapping lines (the audit trail), its
* coverage/checkpoint rows — and orphans die: a node * coverage/checkpoint rows — and orphans die: a node
* whose references all came from this chat disappears * whose references all came from this chat disappears
* entirely, pruned from the positions and embedding * entirely, pruned from the positions and embedding
@@ -58,7 +57,6 @@ import type {
CoverageEntry, CoverageEntry,
GraphEdge, GraphEdge,
GraphNode, GraphNode,
LedgerEvent,
NodeMapping, NodeMapping,
SessionCheckpoint, SessionCheckpoint,
} from "./contracts.ts"; } from "./contracts.ts";
@@ -76,8 +74,6 @@ export interface ChatDeletionStats {
nodesRemoved: number; nodesRemoved: number;
/** Edge assertion LINES removed (empty-provenance or dead endpoint). */ /** Edge assertion LINES removed (empty-provenance or dead endpoint). */
edgesRemoved: number; edgesRemoved: number;
/** Events removed (quiz outcomes + observations). */
eventsRemoved: number;
/** Mapping lines removed (the chat's audit trail). */ /** Mapping lines removed (the chat's audit trail). */
mappingsRemoved: number; mappingsRemoved: number;
/** Coverage rows removed. */ /** Coverage rows removed. */
@@ -89,7 +85,6 @@ export interface ChatDeletionStats {
const EMPTY_STATS: ChatDeletionStats = { const EMPTY_STATS: ChatDeletionStats = {
nodesRemoved: 0, nodesRemoved: 0,
edgesRemoved: 0, edgesRemoved: 0,
eventsRemoved: 0,
mappingsRemoved: 0, mappingsRemoved: 0,
coverageRemoved: 0, coverageRemoved: 0,
checkpointsRemoved: 0, checkpointsRemoved: 0,
@@ -126,7 +121,6 @@ export function deleteChatTraces(graphStore: GraphStore, chatId: string, mode: C
// ── delete-nodes: the full purge ────────────────────────────────────── // ── delete-nodes: the full purge ──────────────────────────────────────
const nodeLines = graphStore.readNodes(); const nodeLines = graphStore.readNodes();
const edgeLines = graphStore.readEdges(); const edgeLines = graphStore.readEdges();
const events = graphStore.readEvents();
const mappings = graphStore.readMappings(); const mappings = graphStore.readMappings();
// Does the chat touch the graph at all? (A never-extracted chat still // Does the chat touch the graph at all? (A never-extracted chat still
@@ -136,7 +130,6 @@ export function deleteChatTraces(graphStore: GraphStore, chatId: string, mode: C
const touched = const touched =
nodeLines.some((line) => line.provenance.some((pt) => pt.chatId === chatId)) || nodeLines.some((line) => line.provenance.some((pt) => pt.chatId === chatId)) ||
edgeLines.some((line) => line.provenance.some((pt) => pt.chatId === chatId)) || edgeLines.some((line) => line.provenance.some((pt) => pt.chatId === chatId)) ||
events.some((event) => event.chatId === chatId) ||
mappings.some((m) => m.chatId === chatId); mappings.some((m) => m.chatId === chatId);
if (!touched) { if (!touched) {
if (stats.coverageRemoved > 0 || stats.checkpointsRemoved > 0) { if (stats.coverageRemoved > 0 || stats.checkpointsRemoved > 0) {
@@ -149,7 +142,7 @@ export function deleteChatTraces(graphStore: GraphStore, chatId: string, mode: C
// remains — another chat/src's provenance point on any of its lines, // remains — another chat/src's provenance point on any of its lines,
// a surviving mapping line (notes-front-door nodes carry EMPTY // a surviving mapping line (notes-front-door nodes carry EMPTY
// provenance: the file IS the anchor, recorded through the mapping // provenance: the file IS the anchor, recorded through the mapping
// line), or a surviving event naming it. (Edge endpoints are deliberately // line). (Edge endpoints are deliberately
// NOT survival references: an edge's asserting session always left // NOT survival references: an edge's asserting session always left
// provenance on its endpoints, so endpoint-only liveness cannot // provenance on its endpoints, so endpoint-only liveness cannot
// occur in a consistent ledger — and counting it would keep nodes // occur in a consistent ledger — and counting it would keep nodes
@@ -163,9 +156,6 @@ export function deleteChatTraces(graphStore: GraphStore, chatId: string, mode: C
for (const mapping of mappings) { for (const mapping of mappings) {
if (mapping.chatId !== chatId) live.add(mapping.graphNodeId); if (mapping.chatId !== chatId) live.add(mapping.graphNodeId);
} }
for (const event of events) {
if (event.chatId !== chatId && event.nodeId !== null) live.add(event.nodeId);
}
// 2. Node lines. Per-line (each line is ONE extraction run's append, so // 2. Node lines. Per-line (each line is ONE extraction run's append, so
// per-session alias provenance exists at line granularity): // per-session alias provenance exists at line granularity):
@@ -219,19 +209,16 @@ export function deleteChatTraces(graphStore: GraphStore, chatId: string, mode: C
outEdges.push(points.length === line.provenance.length ? line : { ...line, provenance: points }); outEdges.push(points.length === line.provenance.length ? line : { ...line, provenance: points });
} }
// 4. Events, mappings: the chat's rows die outright. // 4. Mappings: the chat's rows die outright.
const outEvents = events.filter((event) => event.chatId !== chatId);
const outMappings = mappings.filter((m) => m.chatId !== chatId); const outMappings = mappings.filter((m) => m.chatId !== chatId);
stats.nodesRemoved = [...new Set(nodeLines.map((line) => line.id))].filter((id) => !live.has(id)).length; stats.nodesRemoved = [...new Set(nodeLines.map((line) => line.id))].filter((id) => !live.has(id)).length;
stats.edgesRemoved = edgeLines.length - outEdges.length; stats.edgesRemoved = edgeLines.length - outEdges.length;
stats.eventsRemoved = events.length - outEvents.length;
stats.mappingsRemoved = mappings.length - outMappings.length; stats.mappingsRemoved = mappings.length - outMappings.length;
graphStore.rewriteLedgers({ graphStore.rewriteLedgers({
nodes: outNodes, nodes: outNodes,
edges: outEdges, edges: outEdges,
events: outEvents,
mappings: outMappings, mappings: outMappings,
coverage, coverage,
checkpoints, checkpoints,
@@ -264,8 +251,6 @@ export function deleteChatTraces(graphStore: GraphStore, chatId: string, mode: C
* *
* - every line of the node (create + link lines) * - every line of the node (create + link lines)
* - every EDGE touching it (from or to), whatever session asserted it * - every EDGE touching it (from or to), whatever session asserted it
* - every EVENT naming it (quiz outcomes + observations), from every
* session — the evidence history dies with the thing it was about
* - every mapping line that anchored it (the audit trail; for a * - every mapping line that anchored it (the audit trail; for a
* notes-born node the file IS the anchor — its mapping line is the * notes-born node the file IS the anchor — its mapping line is the
* only one) * only one)
@@ -287,20 +272,16 @@ export function deleteChatTraces(graphStore: GraphStore, chatId: string, mode: C
export interface NodeDeletionStats { export interface NodeDeletionStats {
/** Edge assertion LINES removed (every edge touching the node). */ /** Edge assertion LINES removed (every edge touching the node). */
edgesRemoved: number; edgesRemoved: number;
/** Events removed (quiz outcomes + observations naming the node). */
eventsRemoved: number;
/** Mapping lines removed (the node's audit trail). */ /** Mapping lines removed (the node's audit trail). */
mappingsRemoved: number; mappingsRemoved: number;
} }
/** The confirm dialog's pre-flight facts (the GET on the same node id): /** The confirm dialog's pre-flight facts (the GET on the same node id):
* what full death would take with it. sessions is the deduped set of * what full death would take with it. sessions is the deduped set of
* chats/notes sources that reference the node — provenance points, * chats/notes sources that reference the node — provenance points and
* mapping lines, and events all count. */ * mapping lines count. */
export interface NodeDeletionImpact { export interface NodeDeletionImpact {
edges: number; edges: number;
quizEvents: number;
observations: number;
sessions: string[]; sessions: string[];
} }
@@ -313,7 +294,6 @@ function nodeLinesOf(graphStore: GraphStore, nodeId: string): GraphNode[] {
export function nodeDeletionImpact(graphStore: GraphStore, nodeId: string): NodeDeletionImpact | null { export function nodeDeletionImpact(graphStore: GraphStore, nodeId: string): NodeDeletionImpact | null {
if (nodeLinesOf(graphStore, nodeId).length === 0) return null; if (nodeLinesOf(graphStore, nodeId).length === 0) return null;
const edges = graphStore.readEdges().filter((line) => line.from === nodeId || line.to === nodeId); const edges = graphStore.readEdges().filter((line) => line.from === nodeId || line.to === nodeId);
const events = graphStore.readEvents().filter((event) => event.nodeId === nodeId);
const sessions = new Set<string>(); const sessions = new Set<string>();
for (const line of nodeLinesOf(graphStore, nodeId)) { for (const line of nodeLinesOf(graphStore, nodeId)) {
for (const pt of line.provenance) sessions.add(pt.chatId); for (const pt of line.provenance) sessions.add(pt.chatId);
@@ -321,11 +301,8 @@ export function nodeDeletionImpact(graphStore: GraphStore, nodeId: string): Node
for (const mapping of graphStore.readMappings()) { for (const mapping of graphStore.readMappings()) {
if (mapping.graphNodeId === nodeId) sessions.add(mapping.chatId); if (mapping.graphNodeId === nodeId) sessions.add(mapping.chatId);
} }
for (const event of events) sessions.add(event.chatId);
return { return {
edges: edges.length, edges: edges.length,
quizEvents: events.filter((event) => event.kind !== "observation").length,
observations: events.filter((event) => event.kind === "observation").length,
sessions: [...sessions].sort(), sessions: [...sessions].sort(),
}; };
} }
@@ -341,11 +318,9 @@ export function deleteNodeTraces(graphStore: GraphStore, nodeId: string): NodeDe
if (nodeLines.length === 0) return null; if (nodeLines.length === 0) return null;
const edgeLines = graphStore.readEdges(); const edgeLines = graphStore.readEdges();
const events = graphStore.readEvents();
const mappings = graphStore.readMappings(); const mappings = graphStore.readMappings();
const outEdges = edgeLines.filter((line) => line.from !== nodeId && line.to !== nodeId); const outEdges = edgeLines.filter((line) => line.from !== nodeId && line.to !== nodeId);
const outEvents = events.filter((event) => event.nodeId !== nodeId);
const outMappings = mappings.filter((m) => m.graphNodeId !== nodeId); const outMappings = mappings.filter((m) => m.graphNodeId !== nodeId);
// Only the touched ledgers: the node's own lines always die (its file // Only the touched ledgers: the node's own lines always die (its file
@@ -354,7 +329,6 @@ export function deleteNodeTraces(graphStore: GraphStore, nodeId: string): NodeDe
graphStore.rewriteLedgers({ graphStore.rewriteLedgers({
nodes: outNodes, nodes: outNodes,
...(outEdges.length !== edgeLines.length ? { edges: outEdges } : {}), ...(outEdges.length !== edgeLines.length ? { edges: outEdges } : {}),
...(outEvents.length !== events.length ? { events: outEvents } : {}),
...(outMappings.length !== mappings.length ? { mappings: outMappings } : {}), ...(outMappings.length !== mappings.length ? { mappings: outMappings } : {}),
}); });
@@ -367,7 +341,6 @@ export function deleteNodeTraces(graphStore: GraphStore, nodeId: string): NodeDe
return { return {
edgesRemoved: edgeLines.length - outEdges.length, edgesRemoved: edgeLines.length - outEdges.length,
eventsRemoved: events.length - outEvents.length,
mappingsRemoved: mappings.length - outMappings.length, mappingsRemoved: mappings.length - outMappings.length,
}; };
} }
+73 -245
View File
@@ -2,45 +2,45 @@
* The knowledge graph: merger (the deterministic final stage of the * The knowledge graph: merger (the deterministic final stage of the
* extraction pipeline — vision §1 construction) + derivation (reads). * extraction pipeline — vision §1 construction) + derivation (reads).
* *
* The extraction pipeline is the SOLE graph writer, tree-blind and post-hoc: * The extraction pipeline is the SOLE graph writer, tree-blind and post-hoc
* (2026-09-27 cleanup — clear separation of work):
* *
* extractor (LLM) → SessionFindings: propositions, edges, quiz * extractor (LLM) → SessionFindings: the raw reasoning chains —
* attribution, per-proposition understanding notes — * propositions (lonely relevant ones included) and
* each with verbatim evidence, from the transcript ALONE * follows/supports edges, each with verbatim evidence,
* auditor (LLM) → AuditResult: the verified subset, with the IDENTITY * from the source ALONE (whole session as markdown,
* VERDICT per proposition (linkTo an existing node, or * or raw notes). Fully trusted: no stage re-checks it.
* create), and the drops for the log * auditor (LLM) → AuditResult: identity verdicts (linkTo an existing
* node, or create) via graph-exploration tools, plus
* any BRIDGING edges it draws between existing graph
* nodes and the new propositions. Never sees the
* source; never drops.
* merger (HERE) → mergeFindings(): deterministic ledger appends * merger (HERE) → mergeFindings(): deterministic ledger appends
* *
* The merger is deliberately the ONLY stage that touches the ledgers, and it * The merger is deliberately the ONLY stage that touches the ledgers, and it
* is pure code: the LLM supplies exactly the judgment calls (what the * is pure code: the LLM supplies exactly the judgment calls (what the
* propositions are, what follows/supports what, which proposition a quiz * propositions are, what follows/supports what, whether a proposition IS an
* tested, whether a proposition IS an existing node), while event * existing node), while timestamps/provenance — anything the model could
* kinds/timestamps/refs/provenance — anything the model could silently * silently mis-copy — are rebuilt deterministically. Idempotent +
* mis-copy — are rebuilt deterministically from the transcript's tool parts * incremental: edges dedupe by (from,to,type)-already-asserted-by-this-chat,
* and the message ids in the findings. Idempotent + incremental: refs are * mapping lines dedupe by (chat, sourceId), and coverage entries /
* chat-scoped idempotency keys, edges dedupe by (from,to,type)-already- * sessions.jsonl checkpoints which messages and chats have been extracted
* asserted-by-this-chat, mapping lines dedupe by (chat, sourceId), and * (never the same turns twice). No events: the events ledger is GONE
* sessions.jsonl checkpoints which chats have been extracted (never the * (2026-09-27) — the graph carries propositions + edges, nothing else.
* same turns twice).
* *
* Identity across re-encounters is the AUDITOR's judgment (link-or-create), * Identity across re-encounters is the AUDITOR's judgment (link-or-create
* not string matching: the conservative v1 exact-statement merge was * with an equivalence bias — paraphrases merge), not string matching.
* replaced by the 2026-09-02 redesign. A re-extracted session's * A re-extracted session's propositions link to the existing nodes; the
* propositions link to the existing nodes; the link appends a node line * link appends a node line whose phrasings join `aliases` and whose
* whose phrasings join `aliases` and whose provenance unions (getGraph * provenance unions (getGraph merges lines by id).
* merges lines by id).
* *
* Derivation below (getGraph / deriveNodeStates / graphSnapshot) is reads * Derivation below (getGraph / graphSnapshot) is reads only — everything
* only — everything is recomputed from the ledgers. Every function takes * is recomputed from the ledgers. Every function takes the storage ports
* the storage ports (core/ports.ts) as explicit parameters — pure core, no * (core/ports.ts) as explicit parameters — pure core, no adapter imports;
* adapter imports; the composition root (src/server/stores.ts) wires the fs * the composition root (src/server/stores.ts) wires the fs adapters in.
* adapters in.
*/ */
import { randomBytes } from "node:crypto"; import { randomBytes } from "node:crypto";
import type { LhMessage } from "./native-messages.ts";
import type { ToolResultMessage } from "@earendil-works/pi-ai";
import type { CoverageEntry } from "./contracts/graph.ts"; import type { CoverageEntry } from "./contracts/graph.ts";
import type { import type {
AuditResult, AuditResult,
@@ -50,13 +50,9 @@ import type {
GraphNode, GraphNode,
ProvenancePoint, ProvenancePoint,
GraphSnapshot, GraphSnapshot,
LedgerEvent,
LedgerEventKind,
UnderstandingLevel,
} from "./contracts.ts"; } from "./contracts.ts";
import { MIN_CLUSTER_SIZE } from "./clusters.ts"; import { MIN_CLUSTER_SIZE } from "./clusters.ts";
import { birthMappingIndex, birthSessionOf, birthSessionPartition } from "./session-clusters.ts"; import { birthMappingIndex, birthSessionOf, birthSessionPartition } from "./session-clusters.ts";
import type { QuestionAnswerData } from "./question-core.ts";
import type { ChatStore, GraphStore } from "./ports.ts"; import type { ChatStore, GraphStore } from "./ports.ts";
/** The storage ports every graph operation needs, passed explicitly. */ /** The storage ports every graph operation needs, passed explicitly. */
@@ -71,71 +67,22 @@ function newGraphNodeId(): string {
} }
/** What one merge run merges FROM (the caller supplies it — the merger never /** What one merge run merges FROM (the caller supplies it — the merger never
* reads the chat store): a session's transcript + its coverage checkpoint, * reads the chat store): the coverage checkpoint for a session batch, or —
* or — for the format-agnostic notes front door (2026-09-13) — `messages: * for the format-agnostic notes front door (2026-09-13) — no coverage at
* null`: raw notes carry no tool parts, so no quiz events are rebuilt (quiz * all: the whole-file checkpoint stamps the import.
* outcomes inside notes surface only through the extractor's understanding
* observations), and the whole-file checkpoint stamps the import.
* *
* Coverage (vision-extraction-coverage.md, settled 2026-09-15): a chat merge * Coverage (vision-extraction-coverage.md, settled 2026-09-15): a chat merge
* passes the BATCH it extracted — the caller computed the fingerprints * passes the BATCH it extracted — the caller computed the fingerprints
* before the run, from the same snapshot the merger sees — and step 4 * before the run, from the same snapshot the merger sees — and the final
* appends one CoverageEntry per message on success ("covered" means * step appends one CoverageEntry per message on success ("covered" means
* exactly this merge happened). `batchTag` prefixes the observation refs so * exactly this merge happened). The notes path passes neither and keeps
* re-runs of the SAME batch dedupe while a re-opened (mutated) message's * the whole-file checkpoint. */
* fresh understanding observations land as new events; the notes path
* passes neither and keeps the whole-file checkpoint. */
export interface MergeSource { export interface MergeSource {
messages: LhMessage[] | null;
foldedAt: string; foldedAt: string;
/** The batch being merged (chat path): one entry per message in /** The batch being merged (chat path): one entry per message in the
* `messages`, same order — undefined (notes path / coverage-less * extracted batch — undefined (notes path / coverage-less callers) →
* callers) → the legacy whole-chat checkpoint is written instead. */ * the legacy whole-chat checkpoint is written instead. */
coverage?: CoverageEntry[]; coverage?: CoverageEntry[];
/** The batch's deterministic tag (core/coverage.ts batchTag) — obs refs. */
batchTag?: string;
}
// ─── transcript scanning (shared by the merger + trigger) ─────────────────────
/** A flattened, order-preserving view of every tool call (+ its result
* state) in a native transcript. Wave D: the pairing is native — calls
* come from assistant content blocks; a result map built from the
* toolResult messages carries the outcome (a dangling call is
* input-available, an error result is output-error). */
interface ToolPartView {
name: string;
toolCallId: string;
state: string;
input: unknown;
output: unknown;
}
function toolParts(messages: LhMessage[]): ToolPartView[] {
const results = new Map<string, ToolResultMessage>();
for (const m of messages) {
if ((m as { role?: string }).role === "toolResult") {
results.set((m as ToolResultMessage).toolCallId, m as ToolResultMessage);
}
}
const out: ToolPartView[] = [];
for (const m of messages) {
if ((m as { role?: string }).role !== "assistant") continue;
const content = (m as { content?: unknown }).content;
if (!Array.isArray(content)) continue;
for (const block of content as Array<{ type?: string; id?: string; name?: string; arguments?: unknown }>) {
if (block.type !== "toolCall" || typeof block.id !== "string") continue;
const result = results.get(block.id);
out.push({
name: block.name ?? "",
toolCallId: block.id,
state: result ? (result.isError ? "output-error" : "output-available") : "input-available",
input: block.arguments,
output: result && !result.isError ? result.details : undefined,
});
}
}
return out;
} }
/** (chatId, messageIds) → provenance points; order-preserving, deduped. */ /** (chatId, messageIds) → provenance points; order-preserving, deduped. */
@@ -167,21 +114,14 @@ const samePoint = (a: ProvenancePoint, b: ProvenancePoint): boolean =>
* Mapping lines record each verdict: one per (chat, sourceId). Idempotent: * Mapping lines record each verdict: one per (chat, sourceId). Idempotent:
* a proposition id already mapped for this chat reuses its resolution. * a proposition id already mapped for this chat reuses its resolution.
* *
* Events are rebuilt deterministically from the transcript: quiz outcomes
* (correct / miss / dont-know; kinds + timestamps + refs from the tool
* parts, nodeId from the extractor's attribution through this run's
* mapping) plus one observation event per proposition carrying an
* understanding note (ref `obs:<sourceId>`). No taught events, ever.
*
* `chatId` is the run's source key: a real chat, or a notes source id * `chatId` is the run's source key: a real chat, or a notes source id
* ("src_…") for the format-agnostic import path — same ledgers, same * ("src_…") for the format-agnostic import path — same ledgers, same
* idempotency, one graph. The `source` parameter carries the transcript * idempotency, one graph. The `source` parameter carries the checkpoint
* being merged (null messages = raw notes: no quiz events) and the * stamp; the merger itself is chat-store-free.
* checkpoint stamp; the merger itself is chat-store-free.
*/ */
export function mergeFindings(chatId: string, audit: AuditResult, deps: GraphDeps, source: MergeSource): EmissionStats { export function mergeFindings(chatId: string, audit: AuditResult, deps: GraphDeps, source: MergeSource): EmissionStats {
const { graphStore: ledgers } = deps; const { graphStore: ledgers } = deps;
const stats: EmissionStats = { nodes: 0, edges: 0, events: 0 }; const stats: EmissionStats = { nodes: 0, edges: 0 };
// 1. Resolution: this run's propositions → graph node ids. Existing // 1. Resolution: this run's propositions → graph node ids. Existing
// mapping lines for THIS chat are loaded first — the audit trail doubles // mapping lines for THIS chat are loaded first — the audit trail doubles
@@ -238,63 +178,41 @@ export function mergeFindings(chatId: string, audit: AuditResult, deps: GraphDep
} }
} }
// 2. Edges. Resolve endpoints through the mapping; dedupe by // 2. Edges. Resolve endpoints through the mapping — with BRIDGING
// (from,to,type) already asserted by THIS chat (an existing assertion // (2026-09-27): an endpoint the extractor never proposed but that
// by another chat stays untouched — provenance unions at read time). // names an EXISTING graph node is the auditor drawing a connection
// between the graph it explored and this batch's propositions. Such
// edges carry chat-scoped provenance with NO message anchor (nothing
// in the transcript asserts them — the auditor is the asserter);
// transcript-articulated edges keep message-granular provenance.
// Dedupe by (from,to,type) already asserted by THIS chat (an existing
// assertion by another chat stays untouched — provenance unions at
// read time).
const assertedByChat = new Set<string>(); const assertedByChat = new Set<string>();
for (const line of ledgers.readEdges()) { for (const line of ledgers.readEdges()) {
const key = `${line.from}\u0000${line.to}\u0000${line.type}`; const key = `${line.from}\u0000${line.to}\u0000${line.type}`;
if (line.provenance.some((pt) => pt.chatId === chatId)) assertedByChat.add(key); if (line.provenance.some((pt) => pt.chatId === chatId)) assertedByChat.add(key);
} }
for (const edge of audit.edges) { for (const edge of audit.edges) {
const from = mapping.get(edge.from); const resolve = (endpoint: string): string | undefined =>
const to = mapping.get(edge.to); mapping.get(endpoint) ?? (existingIds.has(endpoint) ? endpoint : undefined);
if (!from || !to || from === to) continue; // unmapped endpoint — skip const from = resolve(edge.from);
const to = resolve(edge.to);
if (!from || !to || from === to) continue; // unresolvable endpoint — skip
const bridged = !mapping.has(edge.from) || !mapping.has(edge.to);
const key = `${from}\u0000${to}\u0000${edge.type}`; const key = `${from}\u0000${to}\u0000${edge.type}`;
if (assertedByChat.has(key)) continue; if (assertedByChat.has(key)) continue;
ledgers.appendEdge({ from, to, type: edge.type, provenance: pointsOf(chatId, edge.messages) }); ledgers.appendEdge({
from,
to,
type: edge.type,
provenance: bridged ? [{ chatId, messageId: "" }] : pointsOf(chatId, edge.messages),
});
assertedByChat.add(key); assertedByChat.add(key);
stats.edges++; stats.edges++;
} }
// 3. Events, rebuilt deterministically in transcript order. Refs are // 3. Checkpoint — THE COVERAGE DISCIPLINE (vision-extraction-coverage.md):
// chat-scoped idempotency keys (quiz toolCallId, obs:<sourceId>) —
// a ref already in the ledger means this evidence was extracted before.
const seenRefs = new Set<string>();
for (const event of ledgers.readEvents()) {
if (event.chatId === chatId && event.ref) seenRefs.add(event.ref);
}
for (const part of source.messages ? toolParts(source.messages) : []) {
if (part.name !== "quiz" || part.state !== "output-available") continue;
const event = quizEvent(part, chatId, audit, mapping, now);
if (event?.ref !== undefined && !seenRefs.has(event.ref)) {
ledgers.appendEvent(event);
seenRefs.add(event.ref);
stats.events++;
}
}
// Understanding observations: one per proposition carrying a note,
// resolved to the node the verdict mapped it to. Refs carry the batch
// tag when there is one: run-local source ids ("p1"…) shift position
// between runs, so a bare obs:p1 could never both dedupe a re-run of
// the same batch AND let a re-opened message's fresh level through.
for (const p of audit.propositions) {
if (p.understanding === undefined || p.understanding.length === 0) continue;
const ref = source.batchTag !== undefined ? `obs:${source.batchTag}:${p.id}` : `obs:${p.id}`;
if (seenRefs.has(ref)) continue;
ledgers.appendEvent({
nodeId: mapping.get(p.id) ?? null,
kind: "observation",
chatId,
at: now,
ref,
note: p.understanding,
});
seenRefs.add(ref);
stats.events++;
}
// 4. Checkpoint — THE COVERAGE DISCIPLINE (vision-extraction-coverage.md):
// only now, after every append succeeded, is the batch covered. The // only now, after every append succeeded, is the batch covered. The
// chat path appends one CoverageEntry per message (per-batch atomic — // chat path appends one CoverageEntry per message (per-batch atomic —
// an abort or failure mid-run leaves earlier batches covered, this // an abort or failure mid-run leaves earlier batches covered, this
@@ -308,50 +226,11 @@ export function mergeFindings(chatId: string, audit: AuditResult, deps: GraphDep
return stats; return stats;
} }
/** Build the ledger event for one answered quiz tool part (or null to skip).
* Deterministic in everything except the attribution-supplied nodeId. */
function quizEvent(
part: ToolPartView,
chatId: string,
audit: AuditResult,
mapping: Map<string, string>,
now: string,
): LedgerEvent | null {
const out = part.output;
if (typeof out !== "object" || out === null) return null;
const answer = out as Partial<QuestionAnswerData>;
if (answer.kind !== "quiz") return null; // ask questions are not graded
if (answer.dismissed === true) return null; // no answer — no evidence
let kind: LedgerEventKind;
if (answer.dontKnow === true) kind = "dont-know";
else if (answer.correct === true) kind = "correct";
else kind = "miss";
// Tree-blind: EVERY quiz attributes through the extractor's judgment —
// the vision's retroactive attribution, with no tree mapping to lean on.
const attributedId = audit.attribution.quiz[part.toolCallId];
const nodeId = attributedId ? (mapping.get(attributedId) ?? null) : null;
const event: LedgerEvent = {
nodeId,
kind,
chatId,
at: typeof answer.answeredAt === "string" ? answer.answeredAt : now,
ref: part.toolCallId,
};
if (typeof answer.note === "string" && answer.note.length > 0) event.note = answer.note;
// The question text is the attribution input for unattributed quizzes.
if (nodeId === null && typeof answer.questionText === "string") event.question = answer.questionText;
return event;
}
// ─── derivation (reads only — everything below is recomputable) ───────────── // ─── derivation (reads only — everything below is recomputable) ─────────────
export interface DerivedGraph { export interface DerivedGraph {
nodes: GraphNode[]; nodes: GraphNode[];
edges: GraphEdge[]; edges: GraphEdge[];
events: LedgerEvent[];
} }
/** /**
@@ -396,14 +275,14 @@ export function getGraph(graphStore: GraphStore): DerivedGraph {
merged.push(edge); merged.push(edge);
} }
} }
return { nodes: [...byId.values()], edges: merged, events: graphStore.readEvents() }; return { nodes: [...byId.values()], edges: merged };
} }
// ─── the /api/graph payload ────────────────────────────────────────────────── // ─── the /api/graph payload ──────────────────────────────────────────────────
/** Everything the home graph view needs, in one fetch (shape: contracts). */ /** Everything the home graph view needs, in one fetch (shape: contracts). */
export function graphSnapshot(deps: GraphDeps): GraphSnapshot { export function graphSnapshot(deps: GraphDeps): GraphSnapshot {
const { nodes, edges, events } = getGraph(deps.graphStore); const { nodes, edges } = getGraph(deps.graphStore);
const nodeChats = new Map<string, Set<string>>(); const nodeChats = new Map<string, Set<string>>();
const touch = (nodeId: string, chatId: string): void => { const touch = (nodeId: string, chatId: string): void => {
@@ -424,9 +303,6 @@ export function graphSnapshot(deps: GraphDeps): GraphSnapshot {
touch(edge.to, pt.chatId); touch(edge.to, pt.chatId);
} }
} }
for (const event of events) {
if (event.nodeId !== null) touch(event.nodeId, event.chatId);
}
// Labels for every source the graph can point back at: real chats // Labels for every source the graph can point back at: real chats
// (title, deep-linkable) and imported notes files (path, static — the // (title, deep-linkable) and imported notes files (path, static — the
@@ -453,27 +329,6 @@ export function graphSnapshot(deps: GraphDeps): GraphSnapshot {
if (ids.length > 0) nodeSessions[nodeId] = ids; if (ids.length > 0) nodeSessions[nodeId] = ids;
} }
// Understanding observations per (node, chat) — the extractor's
// session-relative note (latest wins within a chat). Read-time join for
// the detail card: the confidence history next to the session list.
const nodeObservations: Record<string, Record<string, string>> = {};
const latestObs = new Map<string, { at: string; note: string; index: number }>();
events.forEach((event, index) => {
if (event.kind !== "observation" || event.nodeId === null) return;
if (!event.note) return;
const key = `${event.nodeId}\u0000${event.chatId}`;
const current = latestObs.get(key);
if (!current || event.at > current.at || (event.at === current.at && index > current.index)) {
latestObs.set(key, { at: event.at, note: event.note, index });
}
});
for (const [key, { note }] of latestObs) {
const sep = key.indexOf("\u0000");
const nodeId = key.slice(0, sep);
const chatId = key.slice(sep + 1);
(nodeObservations[nodeId] ??= {})[chatId] = note;
}
// BIRTH-SESSION CLUSTERS (wave 7, owner-locked 2026-09-23): embeddings // BIRTH-SESSION CLUSTERS (wave 7, owner-locked 2026-09-23): embeddings
// are dead — a cluster is the SESSION THAT CREATED ITS PROPOSITIONS, // are dead — a cluster is the SESSION THAT CREATED ITS PROPOSITIONS,
// derived on read from provenance (core/session-clusters.ts). The label // derived on read from provenance (core/session-clusters.ts). The label
@@ -502,7 +357,6 @@ export function graphSnapshot(deps: GraphDeps): GraphSnapshot {
sessions, sessions,
fileSources, fileSources,
nodeSessions, nodeSessions,
nodeObservations,
clusters, clusters,
...(Object.keys(positions).length > 0 ? { positions } : {}), ...(Object.keys(positions).length > 0 ? { positions } : {}),
}; };
@@ -577,48 +431,22 @@ export function graphSessionIndex(deps: GraphDeps): GraphSessionEntry[] {
); );
} }
/** One birth proposition of one session, for the prober's per-session /** One birth proposition of one session — WHAT WAS COVERED there (2026-09-27
* view. `understanding` is the extractor's latest observation for * cleanup: no understanding levels; assessment is practice-module
* (node, session) — the named-word scale (solid/shaky/accepted/resisted/ * territory and nothing writes it until that module exists). */
* not yet engaged) — and `evaluatedAt` is THAT event's own timestamp
* (owner ruling 2026-09-23: quiz outcomes are evidence in the ledger,
* but they never move the evaluation date — the level's event does). */
export interface GraphSessionProposition { export interface GraphSessionProposition {
nodeId: string; nodeId: string;
/** The compact handle — the same name the canvas shows. */ /** The compact handle — the same name the canvas shows. */
name: string; name: string;
understanding: UnderstandingLevel | null;
/** The `at` of the event that minted `understanding`; null when the
* session never evaluated this proposition. */
evaluatedAt: string | null;
} }
/** The propositions BORN in one session, with the session's latest /** The propositions BORN in one session — what that session first
* understanding of each. Latest observation wins by `at`, then by * articulated. Members sorted by node id — deterministic. */
* ledger order (the same tie rule the snapshot's nodeObservations
* uses). Members sorted by node id — deterministic. */
export function graphSessionPropositions(deps: GraphDeps, sessionId: string): GraphSessionProposition[] { export function graphSessionPropositions(deps: GraphDeps, sessionId: string): GraphSessionProposition[] {
const mappingFallback = birthMappingIndex(deps.graphStore.readMappings()); const mappingFallback = birthMappingIndex(deps.graphStore.readMappings());
const born = getGraph(deps.graphStore).nodes.filter((node) => birthSessionOf(node, mappingFallback) === sessionId); const born = getGraph(deps.graphStore).nodes.filter((node) => birthSessionOf(node, mappingFallback) === sessionId);
const latest = new Map<string, { at: string; note: string; index: number }>();
deps.graphStore.readEvents().forEach((event, index) => {
if (event.kind !== "observation" || event.nodeId === null || event.chatId !== sessionId) return;
if (!event.note) return;
const current = latest.get(event.nodeId);
if (!current || event.at > current.at || (event.at === current.at && index > current.index)) {
latest.set(event.nodeId, { at: event.at, note: event.note, index });
}
});
return born return born
.map((node) => { .map((node) => ({ nodeId: node.id, name: node.name }))
const observation = latest.get(node.id);
return {
nodeId: node.id,
name: node.name,
understanding: (observation?.note as UnderstandingLevel | undefined) ?? null,
evaluatedAt: observation?.at ?? null,
};
})
.sort((a, b) => (a.nodeId < b.nodeId ? -1 : 1)); .sort((a, b) => (a.nodeId < b.nodeId ? -1 : 1));
} }
-4
View File
@@ -18,7 +18,6 @@ import type {
GraphNode, GraphNode,
JobKind, JobKind,
JobRecord, JobRecord,
LedgerEvent,
ExtractionRunUsage, ExtractionRunUsage,
NodeMapping, NodeMapping,
PlanTree, PlanTree,
@@ -84,8 +83,6 @@ export interface GraphStore {
readEdges(): GraphEdge[]; readEdges(): GraphEdge[];
/** Append one quiz outcome / understanding observation. */ /** Append one quiz outcome / understanding observation. */
appendEvent(event: LedgerEvent): void;
readEvents(): LedgerEvent[];
/** Append one identity-map line (chat's tree node → graph node). */ /** Append one identity-map line (chat's tree node → graph node). */
appendMapping(mapping: NodeMapping): void; appendMapping(mapping: NodeMapping): void;
@@ -130,7 +127,6 @@ export interface GraphStore {
rewriteLedgers(next: { rewriteLedgers(next: {
nodes?: GraphNode[]; nodes?: GraphNode[];
edges?: GraphEdge[]; edges?: GraphEdge[];
events?: LedgerEvent[];
mappings?: NodeMapping[]; mappings?: NodeMapping[];
coverage?: CoverageEntry[]; coverage?: CoverageEntry[];
checkpoints?: SessionCheckpoint[]; checkpoints?: SessionCheckpoint[];
+126
View File
@@ -0,0 +1,126 @@
/**
* Session → markdown — the ONE formatting engine (owner ruling 2026-09-27).
*
* The extractor's view of a teaching session and the future user-facing
* session export share this engine: a session is rendered as clean,
* readable markdown — speaker headers, quiz/check-in blocks, tool notes —
* with the message-id anchors preserved (extraction provenance depends on
* the extractor reporting the `[id]` tags it saw; `anchors: false` mints
* the same document without them, for the user export).
*
* Pure core: native messages in (LhMessage), string out. Content parity
* with the coverage line builder (core/coverage.ts messageViewLines) is
* deliberate — every message that produces view lines produces a markdown
* block, in the same order — but the formats are SEPARATE by design
* (2026-09-27): coverage fingerprints hash the line format and stay stable
* (no mass re-extraction when presentation changes); markdown is
* presentation. A failed tool call renders nothing (retry context, not
* learning evidence); custom markers are invisible, same as the view.
*/
import type { LhMessage } from "./native-messages.ts";
import type { ToolResultMessage } from "@earendil-works/pi-ai";
/** The user-facing text blocks of a native message (string or text parts). */
function textOf(content: unknown): string[] {
if (typeof content === "string") return content.trim() ? [content] : [];
if (!Array.isArray(content)) return [];
return content
.filter((c): c is { type: "text"; text: string } => (c as { type?: string }).type === "text")
.map((c) => c.text)
.filter((t) => t.trim().length > 0);
}
/** One tool result's learner-visible outcome, rendered as body lines. */
function toolResultBody(result: ToolResultMessage): { heading: string; body: string[] } {
const out = result.details as
| Partial<{
questionText: string;
correct: boolean;
dontKnow: boolean;
note: string;
text: string;
answers: string[];
other: string;
dismissed: boolean;
}>
| undefined;
if (result.toolName === "quiz" || result.toolName === "explain") {
const heading = result.toolName === "quiz" ? "Quiz" : "Recall check";
const question = typeof out?.questionText === "string" && out.questionText ? out.questionText : "(no question text)";
const body = [`**Q:** ${question}`];
if (out?.dismissed === true) body.push("The learner skipped this check.");
else if (out?.dontKnow === true) body.push("The learner did not know.");
else if (out?.correct === true) body.push("The learner answered correctly.");
else if (out?.correct === false) body.push("The learner answered incorrectly.");
else if (typeof out?.text === "string" && out.text.trim()) body.push(`The learner answered in his own words: ${out.text.trim()}`);
if (typeof out?.note === "string" && out.note) body.push(`*(his thinking: ${out.note})*`);
return { heading, body };
}
if (result.toolName === "ask_user_question") {
const question = typeof out?.questionText === "string" && out.questionText ? out.questionText : "(no question text)";
const body = [`**Q:** ${question}`];
const chosen = [
...(Array.isArray(out?.answers) ? out!.answers! : []),
...(typeof out?.other === "string" && out.other.trim() ? [out.other.trim()] : []),
];
if (typeof out?.text === "string" && out.text.trim()) body.push(`The learner answered: ${out.text.trim()}`);
else if (chosen.length > 0) body.push(`The learner chose: ${chosen.join(", ")}`);
else if (out?.dismissed === true) body.push("The learner skipped this question.");
if (typeof out?.note === "string" && out.note) body.push(`*(his thinking: ${out.note})*`);
return { heading: "Question", body };
}
// Parity with the line builder: other clean tool results are one line.
return { heading: "Tool", body: [`*${result.toolName} result delivered*`] };
}
/**
* ONE message as a markdown block — `null` when the message renders
* nothing (a failed tool call, a bare marker, an empty text part set).
* `mid` is the provenance anchor shown when `anchors` is on.
*/
export function messageMarkdown(message: LhMessage, mid: string, anchors = true): string | null {
const role = (message as { role?: string }).role;
const anchor = anchors ? ` · ${mid}` : "";
const parts: string[] = [];
if (role === "user") {
for (const text of textOf((message as { content: unknown }).content)) {
parts.push(`### Learner${anchor}\n\n${text.trim()}`);
}
} else if (role === "assistant") {
for (const text of textOf((message as { content: unknown }).content)) {
parts.push(`### Teacher${anchor}\n\n${text.trim()}`);
}
} else if (role === "toolResult") {
const result = message as ToolResultMessage;
if (result.isError) return null; // retry context, not learning evidence
const { heading, body } = toolResultBody(result);
parts.push(`### ${heading}${anchor}\n\n${body.join("\n\n")}`);
}
// custom markers are invisible, same as the extraction view.
return parts.length > 0 ? parts.join("\n\n") : null;
}
/**
* A whole session (or one extraction BATCH — same engine, subset of
* messages) as one markdown document. Blocks render in message order;
* messages that render nothing are skipped silently. The optional title
* mints the document header (`# <title>`).
*/
export function sessionMarkdown(
messages: ReadonlyArray<LhMessage>,
opts: { title?: string; anchors?: boolean } = {},
): string {
const blocks: string[] = [];
if (opts.title !== undefined && opts.title.length > 0) blocks.push(`# ${opts.title}`);
for (const message of messages) {
const block = messageMarkdown(message, (message as { id?: string }).id ?? "", opts.anchors ?? true);
if (block !== null) blocks.push(block);
}
return blocks.join("\n\n");
}
-1
View File
@@ -57,7 +57,6 @@ export function graphRevision(): number {
for (const file of [ for (const file of [
LEDGER_FILES.nodes, LEDGER_FILES.nodes,
LEDGER_FILES.edges, LEDGER_FILES.edges,
LEDGER_FILES.events,
LEDGER_FILES.mappings, LEDGER_FILES.mappings,
LEDGER_FILES.sessions, LEDGER_FILES.sessions,
LEDGER_FILES.coverage, LEDGER_FILES.coverage,
+18 -22
View File
@@ -1,5 +1,5 @@
/** /**
* GraphStore — the fs JSONL adapter: five append-only ledgers at * GraphStore — the fs JSONL adapter: four append-only ledgers at
* config.LEDGER_FILES (contracts.ts "v1 emission"). * config.LEDGER_FILES (contracts.ts "v1 emission").
* nodes.jsonl GraphNode — one line per node write: the FIRST * nodes.jsonl GraphNode — one line per node write: the FIRST
* line for an id defines it; a later * line for an id defines it; a later
@@ -10,7 +10,6 @@
* edges.jsonl GraphEdge — one line per edge ASSERTION (a second * edges.jsonl GraphEdge — one line per edge ASSERTION (a second
* session asserting the same (from,to,type) * session asserting the same (from,to,type)
* appends another line; reads merge) * appends another line; reads merge)
* events.jsonl LedgerEvent — quiz outcomes + understanding observations
* mappings.jsonl NodeMapping — the session→graph identity map: * mappings.jsonl NodeMapping — the session→graph identity map:
* (chatId, sourceId) → graphNodeId * (chatId, sourceId) → graphNodeId
* sessions.jsonl SessionCheckpoint— emission checkpoints (foldedAt) * sessions.jsonl SessionCheckpoint— emission checkpoints (foldedAt)
@@ -34,8 +33,6 @@ import type {
GraphEdge, GraphEdge,
GraphNode, GraphNode,
ExtractionRunUsage, ExtractionRunUsage,
LedgerEvent,
LedgerEventKind,
NotesSource, NotesSource,
NodeMapping, NodeMapping,
Position, Position,
@@ -110,15 +107,6 @@ const isGraphEdge = (v: unknown): v is GraphEdge =>
Array.isArray(v.provenance) && Array.isArray(v.provenance) &&
v.provenance.every(isProvenancePoint); v.provenance.every(isProvenancePoint);
const EVENT_KINDS: readonly LedgerEventKind[] = ["correct", "miss", "dont-know", "observation"];
const isLedgerEvent = (v: unknown): v is LedgerEvent =>
isRecord(v) &&
(v.nodeId === null || isString(v.nodeId)) &&
(EVENT_KINDS as readonly string[]).includes(v.kind as string) &&
isString(v.chatId) &&
isString(v.at);
const isNodeMapping = (v: unknown): v is NodeMapping => const isNodeMapping = (v: unknown): v is NodeMapping =>
isRecord(v) && isString(v.chatId) && isString(v.sourceId) && isString(v.graphNodeId); isRecord(v) && isString(v.chatId) && isString(v.sourceId) && isString(v.graphNodeId);
@@ -149,13 +137,6 @@ export const graphStore: GraphStore = {
return readValid(LEDGER_FILES.edges, isGraphEdge); return readValid(LEDGER_FILES.edges, isGraphEdge);
}, },
appendEvent(event: LedgerEvent): void {
appendLine(LEDGER_FILES.events, event);
},
readEvents(): LedgerEvent[] {
return readValid(LEDGER_FILES.events, isLedgerEvent);
},
appendMapping(mapping: NodeMapping): void { appendMapping(mapping: NodeMapping): void {
appendLine(LEDGER_FILES.mappings, mapping); appendLine(LEDGER_FILES.mappings, mapping);
}, },
@@ -217,7 +198,6 @@ export const graphStore: GraphStore = {
rewriteLedgers(next: { rewriteLedgers(next: {
nodes?: GraphNode[]; nodes?: GraphNode[];
edges?: GraphEdge[]; edges?: GraphEdge[];
events?: LedgerEvent[];
mappings?: NodeMapping[]; mappings?: NodeMapping[];
coverage?: CoverageEntry[]; coverage?: CoverageEntry[];
checkpoints?: SessionCheckpoint[]; checkpoints?: SessionCheckpoint[];
@@ -225,7 +205,6 @@ export const graphStore: GraphStore = {
const files = { const files = {
nodes: LEDGER_FILES.nodes, nodes: LEDGER_FILES.nodes,
edges: LEDGER_FILES.edges, edges: LEDGER_FILES.edges,
events: LEDGER_FILES.events,
mappings: LEDGER_FILES.mappings, mappings: LEDGER_FILES.mappings,
coverage: LEDGER_FILES.coverage, coverage: LEDGER_FILES.coverage,
checkpoints: LEDGER_FILES.sessions, checkpoints: LEDGER_FILES.sessions,
@@ -326,5 +305,22 @@ function writeSourceMap(map: Record<string, NotesSource>): void {
fs.writeFileSync(SOURCES_FILE, JSON.stringify(map, null, "\t"), "utf-8"); fs.writeFileSync(SOURCES_FILE, JSON.stringify(map, null, "\t"), "utf-8");
} }
// ─── one-way migration: the events ledger is gone (2026-09-27) ────────────────
// Quiz outcomes and understanding observations are no longer formalized in
// the graph — the pipeline writes nodes + edges only. A stale events.jsonl
// left by an earlier build is deleted once, here, at adapter load: dead
// ledgers are exactly the confusion the cleanup removes (owner ruling).
{
const legacyEvents = path.join(path.dirname(LEDGER_FILES.nodes), "events.jsonl");
try {
if (fs.existsSync(legacyEvents)) {
fs.rmSync(legacyEvents);
console.info(`[graph-store] purged legacy events ledger: ${legacyEvents}`);
}
} catch (err) {
console.warn(`[graph-store] could not purge legacy events ledger:`, err);
}
}
const isNotesSource = (v: unknown): v is NotesSource => const isNotesSource = (v: unknown): v is NotesSource =>
isRecord(v) && isString(v.id) && isString(v.filePath) && isString(v.importedAt); isRecord(v) && isString(v.id) && isString(v.filePath) && isString(v.importedAt);
+66 -114
View File
@@ -1,18 +1,20 @@
/** /**
* The extraction pipeline tests (vision §1 construction, 2026-09-02 redesign): * The extraction pipeline tests (vision §1 construction, 2026-09-02 redesign;
* extractor → auditor → merger, the sole graph writer — tree-blind, * relinked 2026-09-27): extractor → auditor → merger, the sole graph writer —
* free-form, with the auditor owning faithfulness and identity. * the extractor alone sees the source (whole-session markdown), the auditor
* never does (identity linking through graph tools), the merger is
* deterministic.
* *
* transcriptView — the compact view both LLM stages see (pure): * transcriptView — the markdown rendering the extractor sees (pure):
* message-id-tagged verbatim text + quiz lines, NO tree * id-anchored Teacher/Learner blocks + check blocks
* pipeline — scripted faux engines drive both agents: findings out, * pipeline — scripted faux engines drive both agents: findings out,
* audit verdict (with linkTo identity) in, deterministic * audit verdict (with linkTo identity) in, deterministic
* merge applied; failure paths checkpoint nothing; * merge applied; failure paths checkpoint nothing;
* single-flight per chat * single-flight per chat
* *
* The merger itself (identity verdicts, event rebuild, idempotency) is * The merger itself (identity verdicts, idempotency) is tested in
* tested in tests/graph.test.ts directly against mergeFindings — here it is * tests/graph.test.ts directly against mergeFindings — here it is the
* the integration: what one full pipeline run does to the ledgers. * integration: what one full pipeline run does to the ledgers.
* *
* Ledger: in-memory (the GraphStore port) — this suite never touches * Ledger: in-memory (the GraphStore port) — this suite never touches
* data/graph. Chats: real ChatRecords in the real chat store (created and * data/graph. Chats: real ChatRecords in the real chat store (created and
@@ -33,7 +35,6 @@ import type {
ExtractionRunUsage, ExtractionRunUsage,
GraphEdge, GraphEdge,
GraphNode, GraphNode,
LedgerEvent,
NodeMapping, NodeMapping,
NotesSource, NotesSource,
SessionCheckpoint, SessionCheckpoint,
@@ -42,7 +43,7 @@ import type {
import type { GraphStore } from "../src/core/ports.ts"; import type { GraphStore } from "../src/core/ports.ts";
import { abortExtraction, createExtractionPipeline, extractionStatuses, transcriptView } from "../src/agent/subagents/extraction.ts"; import { abortExtraction, createExtractionPipeline, extractionStatuses, transcriptView } from "../src/agent/subagents/extraction.ts";
import { ModelsNotConfiguredError, ProviderKeyMissingError, providerKeyMissing, type ModelSlotName } from "../src/core/settings.ts"; import { ModelsNotConfiguredError, ProviderKeyMissingError, providerKeyMissing, type ModelSlotName } from "../src/core/settings.ts";
import { batchTag, coverageOf, coverageSlots, messageViewLines } from "../src/core/coverage.ts"; import { coverageOf, messageViewLines } from "../src/core/coverage.ts";
import { chatStore } from "../src/store/chats.ts"; import { chatStore } from "../src/store/chats.ts";
import { PATHS } from "../config.ts"; import { PATHS } from "../config.ts";
@@ -64,7 +65,6 @@ function fauxEngine(steps: FauxResponseStep[]): PiEngine {
function memoryLedgers() { function memoryLedgers() {
const nodes: GraphNode[] = []; const nodes: GraphNode[] = [];
const edges: GraphEdge[] = []; const edges: GraphEdge[] = [];
const events: LedgerEvent[] = [];
const mappings: NodeMapping[] = []; const mappings: NodeMapping[] = [];
const checkpoints: SessionCheckpoint[] = []; const checkpoints: SessionCheckpoint[] = [];
const coverage: CoverageEntry[] = []; const coverage: CoverageEntry[] = [];
@@ -75,8 +75,6 @@ function memoryLedgers() {
readNodes: () => nodes.map((n) => ({ ...n })), readNodes: () => nodes.map((n) => ({ ...n })),
appendEdge: (e: GraphEdge) => void edges.push(e), appendEdge: (e: GraphEdge) => void edges.push(e),
readEdges: () => edges.map((e) => ({ ...e })), readEdges: () => edges.map((e) => ({ ...e })),
appendEvent: (e: LedgerEvent) => void events.push(e),
readEvents: () => events.map((e) => ({ ...e })),
appendMapping: (m: NodeMapping) => void mappings.push(m), appendMapping: (m: NodeMapping) => void mappings.push(m),
readMappings: () => mappings.map((m) => ({ ...m })), readMappings: () => mappings.map((m) => ({ ...m })),
appendCheckpoint: (c: SessionCheckpoint) => void checkpoints.push(c), appendCheckpoint: (c: SessionCheckpoint) => void checkpoints.push(c),
@@ -110,7 +108,6 @@ function memoryLedgers() {
rewriteLedgers: (next) => { rewriteLedgers: (next) => {
if (next.nodes) nodes.splice(0, nodes.length, ...next.nodes.map((n) => ({ ...n, provenance: [...n.provenance], aliases: [...n.aliases] }))); if (next.nodes) nodes.splice(0, nodes.length, ...next.nodes.map((n) => ({ ...n, provenance: [...n.provenance], aliases: [...n.aliases] })));
if (next.edges) edges.splice(0, edges.length, ...next.edges.map((e) => ({ ...e, provenance: [...e.provenance] }))); if (next.edges) edges.splice(0, edges.length, ...next.edges.map((e) => ({ ...e, provenance: [...e.provenance] })));
if (next.events) events.splice(0, events.length, ...next.events.map((e) => ({ ...e })));
if (next.mappings) mappings.splice(0, mappings.length, ...next.mappings.map((m) => ({ ...m }))); if (next.mappings) mappings.splice(0, mappings.length, ...next.mappings.map((m) => ({ ...m })));
if (next.coverage) coverage.splice(0, coverage.length, ...next.coverage.map((c) => ({ ...c }))); if (next.coverage) coverage.splice(0, coverage.length, ...next.coverage.map((c) => ({ ...c })));
if (next.checkpoints) checkpoints.splice(0, checkpoints.length, ...next.checkpoints.map((c) => ({ ...c }))); if (next.checkpoints) checkpoints.splice(0, checkpoints.length, ...next.checkpoints.map((c) => ({ ...c })));
@@ -224,7 +221,7 @@ after(() => {
}); });
/** What the extractor would produce for the fixture session — tree-blind, /** What the extractor would produce for the fixture session — tree-blind,
* free-form, with message-granular provenance + an understanding note. */ * free-form, with message-granular provenance. */
const FINDINGS: SessionFindings = { const FINDINGS: SessionFindings = {
propositions: [ propositions: [
{ {
@@ -233,8 +230,7 @@ const FINDINGS: SessionFindings = {
statement: "The derivative measures how fast a function changes", statement: "The derivative measures how fast a function changes",
aliases: [], aliases: [],
messages: ["a2"], messages: ["a2"],
evidence: ["TEACHER [a2]: The derivative measures how fast a function changes."], evidence: ["The derivative measures how fast a function changes."],
understanding: "solid",
}, },
{ {
id: "p2", id: "p2",
@@ -242,7 +238,7 @@ const FINDINGS: SessionFindings = {
statement: "The chain rule differentiates composed functions by multiplying local rates.", statement: "The chain rule differentiates composed functions by multiplying local rates.",
aliases: [], aliases: [],
messages: ["a2"], messages: ["a2"],
evidence: ["LEARNER [u1]: teach me the chain rule"], evidence: ["teach me the chain rule"],
}, },
], ],
edges: [ edges: [
@@ -251,10 +247,9 @@ const FINDINGS: SessionFindings = {
to: "p2", to: "p2",
type: "follows", type: "follows",
messages: ["a2"], messages: ["a2"],
evidence: ["LEARNER [u1]: teach me the chain rule"], evidence: ["teach me the chain rule"],
}, },
], ],
attribution: { quiz: { "probe-1": "p1", "check-1": "p2" } },
}; };
/** An engine whose ONE response is the `name` tool call with the given /** An engine whose ONE response is the `name` tool call with the given
@@ -301,27 +296,30 @@ function silentModel(): PiEngine {
// ─── transcriptView (pure) ─────────────────────────────────────────────────── // ─── transcriptView (pure) ───────────────────────────────────────────────────
describe("transcriptView (pure)", () => { describe("transcriptView (pure — the shared markdown engine, owner 2026-09-27)", () => {
it("shows message-id-tagged verbatim conversation + quiz lines — tree-blind, plan ops folded", () => { it("renders id-anchored speaker blocks + check blocks — tree-blind, plan ops folded", () => {
const { record } = fixtureChat(); const { record } = fixtureChat();
const view = transcriptView(record); const view = transcriptView(record);
assert.match(view, /SESSION: Extraction fixture/); // The document header is the session title.
assert.match(view, /^# Extraction fixture$/m);
// Conversation verbatim, each line tagged with its message id — // Conversation verbatim, each block headed by its speaker + the
// the provenance anchors the extractor reports back. // message-id anchor — the provenance the extractor reports back.
assert.match(view, /LEARNER \[u1\] teach me the chain rule/); assert.match(view, /### Learner · u1\n\nteach me the chain rule/);
assert.match(view, /TEACHER \[a2\] The derivative measures how fast a function changes\./); assert.match(view, /### Teacher · a2\n\nThe derivative measures how fast a function changes\./);
// Structured events, attribution-ready. // Checks render as readable blocks (question + outcome).
assert.match(view, /QUIZ \[probe-1\] What is a derivative, roughly\? — the learner answered incorrectly/); assert.match(view, /### Quiz · r_probe-1/);
assert.match(view, /QUIZ \[check-1\] Check — the learner answered correctly/); assert.match(view, /\*\*Q:\*\* What is a derivative, roughly\?/);
assert.match(view, /The learner answered incorrectly\./);
assert.match(view, /The learner answered correctly\./);
// TREE-BLIND: no plan tree, no tree edges, no mark_done lines — plan // TREE-BLIND: no plan tree, no tree edges, no mark_done lines — plan
// ops fold to the generic one-liner like every other non-quiz tool. // ops fold to the generic one-liner like every other non-quiz tool.
assert.doesNotMatch(view, /FINAL PLAN TREE/); assert.doesNotMatch(view, /FINAL PLAN TREE/);
assert.doesNotMatch(view, /TREE EDGES/); assert.doesNotMatch(view, /TREE EDGES/);
assert.match(view, /\(plan_init tool result delivered\)/); assert.match(view, /\*plan_init result delivered\*/);
assert.doesNotMatch(view, /output-available/); assert.doesNotMatch(view, /output-available/);
}); });
@@ -341,9 +339,9 @@ describe("transcriptView (pure)", () => {
]), ]),
].flat()); ].flat());
const view = transcriptView(chatStore.getChat(chat.id)!); const view = transcriptView(chatStore.getChat(chat.id)!);
assert.match(view, /\(research tool result delivered\)/); assert.match(view, /\*research result delivered\*/);
assert.doesNotMatch(view, /long brief/); assert.doesNotMatch(view, /long brief/);
assert.doesNotMatch(view, /TEACHER: {2}/); assert.doesNotMatch(view, /### Teacher/);
}); });
}); });
@@ -355,7 +353,7 @@ describe("the extraction pipeline (extractor → auditor → merger)", () => {
const ledgers = memoryLedgers(); const ledgers = memoryLedgers();
// The auditor passes p1 through and LINKS it to a pre-existing node // The auditor passes p1 through and LINKS it to a pre-existing node
// (identity verdict), drops p2 (unbacked) + the edge with it. // (identity verdict); p2 stays a verdict-free create.
const existingId = "n_existing"; const existingId = "n_existing";
ledgers.appendNode({ ledgers.appendNode({
id: existingId, id: existingId,
@@ -370,8 +368,6 @@ describe("the extraction pipeline (extractor → auditor → merger)", () => {
{ ...FINDINGS.propositions[0]!, linkTo: existingId }, { ...FINDINGS.propositions[0]!, linkTo: existingId },
], ],
edges: [], edges: [],
attribution: { quiz: { "probe-1": "p1" } },
dropped: [{ ref: "p2", reason: "evidence not found" }, { ref: "p1→p2", reason: "endpoint dropped" }],
}; };
const pipeline = createExtractionPipeline({ const pipeline = createExtractionPipeline({
@@ -382,38 +378,18 @@ describe("the extraction pipeline (extractor → auditor → merger)", () => {
}); });
const stats = await pipeline.runExtraction(chatId); const stats = await pipeline.runExtraction(chatId);
// The linked proposition created NO new node; the understanding note // The linked proposition created NO new node.
// became an observation; both quizzes became events. assert.deepEqual(stats, { nodes: 0, edges: 0 });
assert.deepEqual(stats, { nodes: 0, edges: 0, events: 3 });
// Link-append: the phrasing joined the existing node's aliases and // Link-append: the phrasing joined the existing node's aliases and
// the provenance unioned; no second node line defines a new node. // the provenance unioned; no second node line defines a new node.
assert.equal(ledgers.readNodes().length, 2); // the pre-seeded line + the link line assert.equal(ledgers.readNodes().length, 2); // the pre-seeded line + the link line
assert.deepEqual(ledgers.readMappings(), [{ chatId, sourceId: "p1", graphNodeId: existingId }]); assert.deepEqual(ledgers.readMappings(), [{ chatId, sourceId: "p1", graphNodeId: existingId }]);
// Events rebuilt deterministically: quizzes in transcript order, then
// the observation. check-1's attribution pointed at the DROPPED p2 →
// unresolvable → null (the question text is kept for later attribution).
// The observation ref is BATCH-TAGGED (obs:<tag>:p1) so a re-run of
// the same batch dedupes while a re-opened message's update lands as
// a new event (vision-extraction-coverage.md).
const events = ledgers.readEvents();
const record = chatStore.getChat(chatId)!;
const expectedTag = batchTag(coverageSlots(record.messages));
assert.deepEqual(
events.map((e) => [e.kind, e.ref, e.nodeId]),
[
["miss", "probe-1", existingId],
["correct", "check-1", null],
["observation", `obs:${expectedTag}:p1`, existingId],
],
);
assert.equal(events[0]!.nodeId, existingId, "probe attributed via the verdict's mapping");
assert.equal(events[2]!.note, "solid", "the understanding note joins the ledger");
// COVERAGE checkpointed per message (not a whole-chat checkpoint): // COVERAGE checkpointed per message (not a whole-chat checkpoint):
// one entry per transcript message, fingerprinted content each — // one entry per transcript message, fingerprinted content each —
// the same turns are never extracted twice. // the same turns are never extracted twice.
const record = chatStore.getChat(chatId)!;
const covered = ledgers.readCoverage(); const covered = ledgers.readCoverage();
// Wave D: the coverage unit is the VISIBLE native message — bonded // Wave D: the coverage unit is the VISIBLE native message — bonded
// riders with no view lines (a textless call message) are context, // riders with no view lines (a textless call message) are context,
@@ -498,12 +474,10 @@ describe("the extraction pipeline (extractor → auditor → merger)", () => {
it("the LLM cluster-namer is DEAD (wave 7): a merge mints no naming runs — labels are session names now", async () => { it("the LLM cluster-namer is DEAD (wave 7): a merge mints no naming runs — labels are session names now", async () => {
const { chatId } = fixtureChat(); const { chatId } = fixtureChat();
const ledgers = memoryLedgers(); const ledgers = memoryLedgers();
// Identity-pass verdict: p1 becomes a new node, p2 dropped (no links). // Identity-pass verdict: p1 becomes a new node, p2 unlinked.
const verdict: AuditResult = { const verdict: AuditResult = {
propositions: [{ ...FINDINGS.propositions[0]! }], propositions: [{ ...FINDINGS.propositions[0]! }],
edges: [], edges: [],
attribution: { quiz: { "probe-1": "p1" } },
dropped: [{ ref: "p2", reason: "evidence not found" }],
}; };
const pipeline = createExtractionPipeline({ const pipeline = createExtractionPipeline({
chatStore, chatStore,
@@ -512,7 +486,7 @@ describe("the extraction pipeline (extractor → auditor → merger)", () => {
auditor: submitModel("submit_review", verdict), auditor: submitModel("submit_review", verdict),
}); });
const stats = await pipeline.runExtraction(chatId); const stats = await pipeline.runExtraction(chatId);
assert.deepEqual(stats, { nodes: 1, edges: 0, events: 3 }); assert.deepEqual(stats, { nodes: 1, edges: 0 });
// Wave 7 (950ddfd): birth-session clusters carry session-name labels, // Wave 7 (950ddfd): birth-session clusters carry session-name labels,
// so the post-merge naming garnish is gone — no model spend, no // so the post-merge naming garnish is gone — no model spend, no
// naming-run records, no label writes. // naming-run records, no label writes.
@@ -525,9 +499,7 @@ describe("the extraction pipeline (extractor → auditor → merger)", () => {
const ledgers = memoryLedgers(); const ledgers = memoryLedgers();
const verdict: AuditResult = { const verdict: AuditResult = {
propositions: FINDINGS.propositions, propositions: FINDINGS.propositions,
edges: [], // the edge is dropped with it edges: [],
attribution: FINDINGS.attribution,
dropped: [{ ref: "p1→p2", reason: "endpoint dropped" }],
}; };
const consulted: string[] = []; const consulted: string[] = [];
const pipeline = createExtractionPipeline({ const pipeline = createExtractionPipeline({
@@ -558,8 +530,6 @@ describe("the extraction pipeline (extractor → auditor → merger)", () => {
const verdict: AuditResult = { const verdict: AuditResult = {
propositions: FINDINGS.propositions, propositions: FINDINGS.propositions,
edges: [], edges: [],
attribution: FINDINGS.attribution,
dropped: [{ ref: "p1→p2", reason: "endpoint dropped" }],
}; };
const pipeline = createExtractionPipeline({ const pipeline = createExtractionPipeline({
chatStore, chatStore,
@@ -593,7 +563,7 @@ describe("the extraction pipeline (extractor → auditor → merger)", () => {
chatStore, chatStore,
graphStore: ledgers, graphStore: ledgers,
extractor: silentModel(), extractor: silentModel(),
auditor: submitModel("submit_review", { ...FINDINGS, dropped: [] }), auditor: submitModel("submit_review", FINDINGS),
}); });
await assert.rejects(failingExtractor.runExtraction(chatId), /the extractor never submitted findings/); await assert.rejects(failingExtractor.runExtraction(chatId), /the extractor never submitted findings/);
assert.equal(ledgers.readCheckpoints().length, 0); assert.equal(ledgers.readCheckpoints().length, 0);
@@ -616,8 +586,6 @@ describe("the extraction pipeline (extractor → auditor → merger)", () => {
const verdict: AuditResult = { const verdict: AuditResult = {
propositions: [{ ...FINDINGS.propositions[0]!, linkTo: existingId }], propositions: [{ ...FINDINGS.propositions[0]!, linkTo: existingId }],
edges: [], edges: [],
attribution: { quiz: {} },
dropped: [{ ref: "p2", reason: "evidence not found" }],
}; };
const pipeline = createExtractionPipeline({ const pipeline = createExtractionPipeline({
chatStore, chatStore,
@@ -648,8 +616,6 @@ describe("the extraction pipeline (extractor → auditor → merger)", () => {
const verdict: AuditResult = { const verdict: AuditResult = {
propositions: [{ ...FINDINGS.propositions[0]!, linkTo: existingId }], propositions: [{ ...FINDINGS.propositions[0]!, linkTo: existingId }],
edges: [], edges: [],
attribution: { quiz: {} },
dropped: [{ ref: "p2", reason: "evidence not found" }],
}; };
const pipeline = createExtractionPipeline({ const pipeline = createExtractionPipeline({
chatStore, chatStore,
@@ -675,7 +641,6 @@ describe("the extraction pipeline (extractor → auditor → merger)", () => {
assert.equal(record.extractions?.length, 1); assert.equal(record.extractions?.length, 1);
const run = record.extractions![0]!; const run = record.extractions![0]!;
assert.equal(run.stats?.nodes, 0); // linked, not created assert.equal(run.stats?.nodes, 0); // linked, not created
assert.equal(run.stats?.events, 3);
// the scripted engine's model, provider-qualified (model slots wave) // the scripted engine's model, provider-qualified (model slots wave)
assert.equal(run.stages.extractor.model, "faux:faux-1"); assert.equal(run.stages.extractor.model, "faux:faux-1");
assert.equal(run.stages.extractor.steps.length, 1); // the submit call — the run STOPS after a clean submit assert.equal(run.stages.extractor.steps.length, 1); // the submit call — the run STOPS after a clean submit
@@ -696,7 +661,7 @@ describe("the extraction pipeline (extractor → auditor → merger)", () => {
chatStore, chatStore,
graphStore: ledgers, graphStore: ledgers,
extractor: hangingModel(), // hangs until aborted — no step finishes extractor: hangingModel(), // hangs until aborted — no step finishes
auditor: submitModel("submit_review", { propositions: [], edges: [], attribution: { quiz: {} }, dropped: [] }), auditor: submitModel("submit_review", { propositions: [], edges: [] }),
}); });
const runPromise = pipeline.runExtraction(chatId); const runPromise = pipeline.runExtraction(chatId);
@@ -706,7 +671,7 @@ describe("the extraction pipeline (extractor → auditor → merger)", () => {
const stats = await runPromise; const stats = await runPromise;
// A deliberate stop, not a failure: zero emissions, no throw. // A deliberate stop, not a failure: zero emissions, no throw.
assert.deepEqual(stats, { nodes: 0, edges: 0, events: 0 }); assert.deepEqual(stats, { nodes: 0, edges: 0 });
assert.equal(extractionStatuses().length, 0, "the registry clears"); assert.equal(extractionStatuses().length, 0, "the registry clears");
assert.equal(ledgers.readNodes().length, 0, "nothing merged"); assert.equal(ledgers.readNodes().length, 0, "nothing merged");
assert.equal(ledgers.readCheckpoints().length, 0, "nothing checkpointed — the next trigger re-runs"); assert.equal(ledgers.readCheckpoints().length, 0, "nothing checkpointed — the next trigger re-runs");
@@ -751,7 +716,7 @@ describe("the extraction pipeline (extractor → auditor → merger)", () => {
const first = pipeline.runExtraction(chatId); const first = pipeline.runExtraction(chatId);
const second = await pipeline.runExtraction(chatId); // in-flight → zeros const second = await pipeline.runExtraction(chatId); // in-flight → zeros
assert.deepEqual(second, { nodes: 0, edges: 0, events: 0 }); assert.deepEqual(second, { nodes: 0, edges: 0 });
release(); release();
await assert.rejects(first, /the extractor never submitted findings/); await assert.rejects(first, /the extractor never submitted findings/);
@@ -800,7 +765,7 @@ describe("the extraction pipeline (extractor → auditor → merger)", () => {
); );
const g = globalThis as { __lhExtractionRuns?: { running: Map<string, unknown> } }; const g = globalThis as { __lhExtractionRuns?: { running: Map<string, unknown> } };
assert.ok(g.__lhExtractionRuns?.running.has(chatId), "the registry lives on globalThis — no module-instance split can hide runs"); assert.ok(g.__lhExtractionRuns?.running.has(chatId), "the registry lives on globalThis — no module-instance split can hide runs");
assert.deepEqual(await observer.runExtraction(chatId), { nodes: 0, edges: 0, events: 0 }, "single-flight spans pipeline instances"); assert.deepEqual(await observer.runExtraction(chatId), { nodes: 0, edges: 0 }, "single-flight spans pipeline instances");
release(); release();
await assert.rejects(first, /the extractor never submitted findings/); await assert.rejects(first, /the extractor never submitted findings/);
@@ -829,8 +794,8 @@ const NOTES_TEXT = `> [!quote] YOU
> [!abstract] PI > [!abstract] PI
> Vim always starts in Normal mode — keys are commands, not text.`; > Vim always starts in Normal mode — keys are commands, not text.`;
/** What the notes-mode extractor submits: same shapes minus attribution, /** What the notes-mode extractor submits: same shapes as session findings,
* and messages always empty (the file is the provenance). */ * messages always empty (the file is the provenance). */
const NOTES_FINDINGS = { const NOTES_FINDINGS = {
propositions: [ propositions: [
{ {
@@ -840,7 +805,6 @@ const NOTES_FINDINGS = {
aliases: [], aliases: [],
messages: [], messages: [],
evidence: ["Vim always starts in Normal mode — keys are commands, not text."], evidence: ["Vim always starts in Normal mode — keys are commands, not text."],
understanding: "solid",
}, },
], ],
edges: [], edges: [],
@@ -849,11 +813,10 @@ const NOTES_FINDINGS = {
const NOTES_VERDICT = { const NOTES_VERDICT = {
propositions: [NOTES_FINDINGS.propositions[0]], propositions: [NOTES_FINDINGS.propositions[0]],
edges: [], edges: [],
dropped: [],
} as const; } as const;
describe("the notes front door (importNotes — format-agnostic)", () => { describe("the notes front door (importNotes — format-agnostic)", () => {
it("imports raw notes end to end: source record, file-scoped provenance, no quiz events", async () => { it("imports raw notes end to end: source record, file-scoped provenance", async () => {
const ledgers = memoryLedgers(); const ledgers = memoryLedgers();
const seen: { kind?: string; label?: string }[] = []; const seen: { kind?: string; label?: string }[] = [];
const pipeline = createExtractionPipeline({ const pipeline = createExtractionPipeline({
@@ -868,7 +831,7 @@ describe("the notes front door (importNotes — format-agnostic)", () => {
}); });
const stats = await pipeline.importNotes({ filePath: "/vault/learn vim.md", text: NOTES_TEXT }); const stats = await pipeline.importNotes({ filePath: "/vault/learn vim.md", text: NOTES_TEXT });
assert.deepEqual(stats, { nodes: 1, edges: 0, events: 1 }); assert.deepEqual(stats, { nodes: 1, edges: 0 });
// The run surfaced as a NOTES run labeled with the file path. // The run surfaced as a NOTES run labeled with the file path.
assert.deepEqual(seen, [{ kind: "notes", label: "/vault/learn vim.md" }]); assert.deepEqual(seen, [{ kind: "notes", label: "/vault/learn vim.md" }]);
assert.equal(extractionStatuses().length, 0, "the registry clears when the run ends"); assert.equal(extractionStatuses().length, 0, "the registry clears when the run ends");
@@ -888,14 +851,6 @@ describe("the notes front door (importNotes — format-agnostic)", () => {
assert.equal(node.statement, "Vim starts in Normal mode, where keys are commands."); assert.equal(node.statement, "Vim starts in Normal mode, where keys are commands.");
assert.deepEqual(node.provenance, []); assert.deepEqual(node.provenance, []);
assert.deepEqual(ledgers.readMappings(), [{ chatId: source.id, sourceId: "p1", graphNodeId: node.id }]); assert.deepEqual(ledgers.readMappings(), [{ chatId: source.id, sourceId: "p1", graphNodeId: node.id }]);
// One observation event, zero quiz events — outcomes inside notes
// surface only through understanding observations (user-decided).
const events = ledgers.readEvents();
assert.deepEqual(
events.map((e) => [e.kind, e.chatId, e.ref, e.note]),
[["observation", source.id, "obs:p1", "solid"]],
);
// The checkpoint stamps the import. // The checkpoint stamps the import.
assert.equal(ledgers.readCheckpoints()[0]!.chatId, source.id); assert.equal(ledgers.readCheckpoints()[0]!.chatId, source.id);
}); });
@@ -910,10 +865,10 @@ describe("the notes front door (importNotes — format-agnostic)", () => {
auditor: submitModel("submit_review", NOTES_VERDICT), auditor: submitModel("submit_review", NOTES_VERDICT),
}); });
const first = await mk().importNotes({ filePath: "/vault/learn vim.md", text: NOTES_TEXT }); const first = await mk().importNotes({ filePath: "/vault/learn vim.md", text: NOTES_TEXT });
assert.deepEqual(first, { nodes: 1, edges: 0, events: 1 }); assert.deepEqual(first, { nodes: 1, edges: 0 });
const second = await mk().importNotes({ filePath: "/vault/learn vim.md", text: NOTES_TEXT }); const second = await mk().importNotes({ filePath: "/vault/learn vim.md", text: NOTES_TEXT });
assert.deepEqual(second, { nodes: 0, edges: 0, events: 0 }, "the same source id dedupes everything"); assert.deepEqual(second, { nodes: 0, edges: 0 }, "the same source id dedupes everything");
assert.equal(ledgers.readSources().length, 1, "one source identity — the path hashes to the same id"); assert.equal(ledgers.readSources().length, 1, "one source identity — the path hashes to the same id");
assert.equal(ledgers.readSources()[0]!.extractions?.length, 2, "both runs billed their usage"); assert.equal(ledgers.readSources()[0]!.extractions?.length, 2, "both runs billed their usage");
@@ -932,7 +887,7 @@ describe("the notes front door (importNotes — format-agnostic)", () => {
const srcId = extractionStatuses().find((r) => r.chatId.startsWith("src_"))!.chatId; const srcId = extractionStatuses().find((r) => r.chatId.startsWith("src_"))!.chatId;
assert.equal(abortExtraction(srcId), true); assert.equal(abortExtraction(srcId), true);
assert.deepEqual(await runPromise, { nodes: 0, edges: 0, events: 0 }); assert.deepEqual(await runPromise, { nodes: 0, edges: 0 });
assert.equal(ledgers.readNodes().length, 0, "nothing merged"); assert.equal(ledgers.readNodes().length, 0, "nothing merged");
assert.equal(ledgers.readCheckpoints().length, 0, "nothing checkpointed"); assert.equal(ledgers.readCheckpoints().length, 0, "nothing checkpointed");
assert.equal(ledgers.readSources()[0]!.extractions?.length, 1, "partial usage still persisted"); assert.equal(ledgers.readSources()[0]!.extractions?.length, 1, "partial usage still persisted");
@@ -943,8 +898,8 @@ describe("the notes front door (importNotes — format-agnostic)", () => {
// ── the chunked pipeline (extraction coverage — vision-extraction-coverage.md) ── // ── the chunked pipeline (extraction coverage — vision-extraction-coverage.md) ──
/** Pull the pipeline's prompt view out of a model call: the pipeline sends /** Pull the pipeline's prompt view out of a model call: the pipeline sends
* `Transcript:\n\n<view>` as the user turn, so the LAST user message's * the session's markdown document itself as the user turn, so the LAST
* text IS what the extractor saw. */ * user message's text IS what the extractor saw. */
function capturedView(context: unknown): string { function capturedView(context: unknown): string {
const messages = (context as { messages?: { role: string; content: string }[] }).messages ?? []; const messages = (context as { messages?: { role: string; content: string }[] }).messages ?? [];
const last = [...messages].reverse().find((m) => m.role === "user"); const last = [...messages].reverse().find((m) => m.role === "user");
@@ -972,8 +927,6 @@ function passThroughAuditStep(): FauxResponseStep {
{ {
propositions: [{ ...FINDINGS.propositions[0]! }], propositions: [{ ...FINDINGS.propositions[0]! }],
edges: [], edges: [],
attribution: { quiz: {} },
dropped: [{ ref: "p2", reason: "evidence not found" }],
} as unknown as ToolCall["arguments"], } as unknown as ToolCall["arguments"],
{ id: "review-call" }, { id: "review-call" },
), ),
@@ -1014,15 +967,16 @@ describe("the chunked pipeline (extraction coverage)", () => {
// Two batches: the first held messages 1-3, the second 4-5 — a // Two batches: the first held messages 1-3, the second 4-5 — a
// message is never split, batches are consecutive oldest-first. // message is never split, batches are consecutive oldest-first.
assert.equal(views.length, 2, "one extractor call per batch"); assert.equal(views.length, 2, "one extractor call per batch");
assert.ok(views[0]!.includes("LEARNER [u1]") && views[0]!.includes("LEARNER [u3]")); assert.ok(views[0]!.includes("### Learner · u1") && views[0]!.includes("### Learner · u3"));
assert.ok(!views[0]!.includes("LEARNER [u4]"), "the batch stops at the token budget"); assert.ok(!views[0]!.includes("### Learner · u4"), "the batch stops at the token budget");
assert.ok(views[1]!.includes("LEARNER [u4]") && views[1]!.includes("LEARNER [u5]")); assert.ok(views[1]!.includes("### Learner · u4") && views[1]!.includes("### Learner · u5"));
assert.ok(!views[1]!.includes("LEARNER [u3]")); assert.ok(!views[1]!.includes("### Learner · u3"));
// The document header is the session title.
assert.ok(views[0]!.startsWith("# Chunky coverage fixture"));
// p1 was a fresh node in batch 1; batch 2's p1 mapped to the same // p1 was a fresh node in batch 1; batch 2's p1 mapped to the same
// node (chat-scoped mapping) — but its understanding note is a NEW // node (chat-scoped mapping).
// batch-tagged observation. assert.deepEqual(stats, { nodes: 1, edges: 0 });
assert.deepEqual(stats, { nodes: 1, edges: 0, events: 2 });
// Per-batch coverage: every message checkpointed, nothing legacy. // Per-batch coverage: every message checkpointed, nothing legacy.
const coverage = ledgers.readCoverage(); const coverage = ledgers.readCoverage();
@@ -1034,7 +988,7 @@ describe("the chunked pipeline (extraction coverage)", () => {
// new usage entry (no money spent on covered ground). // new usage entry (no money spent on covered ground).
const before = views.length; const before = views.length;
const again = await pipeline.runExtraction(chatId); const again = await pipeline.runExtraction(chatId);
assert.deepEqual(again, { nodes: 0, edges: 0, events: 0 }); assert.deepEqual(again, { nodes: 0, edges: 0 });
assert.equal(views.length, before); assert.equal(views.length, before);
assert.equal(chatStore.getChat(chatId)!.extractions?.length, 1, "the no-op run persists no usage"); assert.equal(chatStore.getChat(chatId)!.extractions?.length, 1, "the no-op run persists no usage");
}); });
@@ -1079,7 +1033,7 @@ describe("the chunked pipeline (extraction coverage)", () => {
const stats = await runPromise; const stats = await runPromise;
// Batch 1's emissions are kept and its messages stay covered. // Batch 1's emissions are kept and its messages stay covered.
assert.deepEqual(stats, { nodes: 1, edges: 0, events: 1 }); assert.deepEqual(stats, { nodes: 1, edges: 0 });
assert.equal(ledgers.readCoverage().length, 3, "batch 1 checkpointed its three messages"); assert.equal(ledgers.readCoverage().length, 3, "batch 1 checkpointed its three messages");
const cov = coverageOf(record.messages, ledgers.readCoverage(), chatId); const cov = coverageOf(record.messages, ledgers.readCoverage(), chatId);
assert.equal(cov.uncoveredCount, 2, "the in-flight batch's messages stay uncovered"); assert.equal(cov.uncoveredCount, 2, "the in-flight batch's messages stay uncovered");
@@ -1095,8 +1049,8 @@ describe("the chunked pipeline (extraction coverage)", () => {
}); });
await finisher.runExtraction(chatId); await finisher.runExtraction(chatId);
assert.equal(reRunViews.length, 1); assert.equal(reRunViews.length, 1);
assert.ok(reRunViews[0]!.includes("LEARNER [u4]") && reRunViews[0]!.includes("LEARNER [u5]")); assert.ok(reRunViews[0]!.includes("### Learner · u4") && reRunViews[0]!.includes("### Learner · u5"));
assert.ok(!reRunViews[0]!.includes("LEARNER [u1]"), "the covered remainder never re-enters the prompt"); assert.ok(!reRunViews[0]!.includes("### Learner · u1"), "the covered remainder never re-enters the prompt");
assert.equal(ledgers.readCoverage().length, 5, "the re-run closed the gap"); assert.equal(ledgers.readCoverage().length, 5, "the re-run closed the gap");
}); });
@@ -1152,7 +1106,7 @@ describe("the chunked pipeline (extraction coverage)", () => {
}); });
await reOpener.runExtraction(chatId); await reOpener.runExtraction(chatId);
assert.equal(reRunViews.length, 1, "one batch: the single re-opened message"); assert.equal(reRunViews.length, 1, "one batch: the single re-opened message");
assert.ok(reRunViews[0]!.includes("LEARNER [s1]") && !reRunViews[0]!.includes("LEARNER [s2]")); assert.ok(reRunViews[0]!.includes("### Learner · s1") && !reRunViews[0]!.includes("### Learner · s2"));
assert.equal(coverageOf(messages(), ledgers.readCoverage(), chatId).uncoveredCount, 0, "re-covered"); assert.equal(coverageOf(messages(), ledgers.readCoverage(), chatId).uncoveredCount, 0, "re-covered");
}); });
}); });
@@ -1170,7 +1124,7 @@ describe("the session-deletion guard (chat deleted mid-run)", () => {
chatStore, chatStore,
graphStore: ledgers, graphStore: ledgers,
extractor: submitModel("submit_findings", FINDINGS), extractor: submitModel("submit_findings", FINDINGS),
auditor: submitModel("submit_review", { propositions: FINDINGS.propositions, edges: [], attribution: { quiz: {} }, dropped: [] }, () => chatStore.deleteChat(chatId)), auditor: submitModel("submit_review", { propositions: FINDINGS.propositions, edges: [] }, () => chatStore.deleteChat(chatId)),
}); });
const stats = await pipeline.runExtraction(chatId); const stats = await pipeline.runExtraction(chatId);
@@ -1179,13 +1133,11 @@ describe("the session-deletion guard (chat deleted mid-run)", () => {
assert.ok(stats); assert.ok(stats);
assert.equal(stats.nodes, 0); assert.equal(stats.nodes, 0);
assert.equal(stats.edges, 0); assert.equal(stats.edges, 0);
assert.equal(stats.events, 0);
// Nothing merged, nothing checkpointed — the deleted chat leaves no // Nothing merged, nothing checkpointed — the deleted chat leaves no
// trace for a later snapshot to resurrect. // trace for a later snapshot to resurrect.
assert.equal(ledgers.readNodes().length, 0); assert.equal(ledgers.readNodes().length, 0);
assert.equal(ledgers.readEdges().length, 0); assert.equal(ledgers.readEdges().length, 0);
assert.equal(ledgers.readEvents().length, 0);
assert.equal(ledgers.readMappings().length, 0); assert.equal(ledgers.readMappings().length, 0);
assert.equal(ledgers.readCoverage().length, 0); assert.equal(ledgers.readCoverage().length, 0);
assert.equal(ledgers.readCheckpoints().length, 0); assert.equal(ledgers.readCheckpoints().length, 0);
+76 -41
View File
@@ -1,19 +1,21 @@
// THE PROBER'S GRAPH TOOLS (wave 7) — pinned behaviors: // THE PROBER'S GRAPH TOOLS (wave 7; understanding levels removed
// 2026-09-27) — pinned behaviors:
// 1. REGISTRATION: the factory registers list_sessions/list_propositions // 1. REGISTRATION: the factory registers list_sessions/list_propositions
// on the prober ONLY when the composition root passes a graphStore // on the prober ONLY when the composition root passes a graphStore
// (the graphToolsEnabled gate lives server-side; absent store = closed // (the graphToolsEnabled gate lives server-side; absent store = closed
// gate — the factory must not reach for settings itself). // gate — the factory must not reach for settings itself).
// 2. OUTPUT: names + levels + dating only — the owner ruling bars // 2. OUTPUT: names only — the owner ruling bars statement text and session
// statement text and session contents from the prober's view of the map. // contents from the prober's view of the map.
// 3. HONESTY: empty graph, unknown session id — the tools answer in // 3. HONESTY: empty graph, unknown session id — the tools answer in
// their own voice, never throw, never guess. // their own voice, never throw, never guess.
import assert from "node:assert/strict"; import assert from "node:assert/strict";
import { describe, it } from "node:test"; import { describe, it } from "node:test";
import { createProberTools } from "../src/agent/tools.ts"; import { createProberTools } from "../src/agent/tools.ts";
import { auditorGraphTools } from "../src/agent/tools/graph.ts";
import type { ChatStore, GraphStore, JobStore } from "../src/core/ports.ts"; import type { ChatStore, GraphStore, JobStore } from "../src/core/ports.ts";
import type { ChatSummary } from "../src/core/contracts/chat.ts"; import type { ChatSummary } from "../src/core/contracts/chat.ts";
import type { GraphEdge, GraphNode, LedgerEvent } from "../src/core/contracts/graph.ts"; import type { GraphEdge, GraphNode } from "../src/core/contracts/graph.ts";
// ─── fixtures ──────────────────────────────────────────────────────────────── // ─── fixtures ────────────────────────────────────────────────────────────────
@@ -31,27 +33,18 @@ const node = (
provenance, provenance,
}); });
const observation = (
nodeId: string,
chatId: string,
note: string,
at: string,
): LedgerEvent =>
({ kind: "observation", nodeId, chatId, note, at }) as unknown as LedgerEvent;
function fakeStores( function fakeStores(
nodes: GraphNode[], nodes: GraphNode[],
over: { over: {
events?: LedgerEvent[];
chats?: Array<Pick<ChatSummary, "id" | "title" | "createdAt" | "updatedAt">>; chats?: Array<Pick<ChatSummary, "id" | "title" | "createdAt" | "updatedAt">>;
mappings?: Array<{ chatId: string; sourceId: string; graphNodeId: string }>;
} = {}, } = {},
): { chatStore: ChatStore; graphStore: GraphStore } { ): { chatStore: ChatStore; graphStore: GraphStore } {
const edges: GraphEdge[] = []; const edges: GraphEdge[] = [];
const graphStore: GraphStore = { const graphStore: GraphStore = {
readNodes: () => nodes, readNodes: () => nodes,
readEdges: () => edges, readEdges: () => edges,
readEvents: () => over.events ?? [], readMappings: () => over.mappings ?? [],
readMappings: () => [],
readCheckpoints: () => [], readCheckpoints: () => [],
readCoverage: () => [], readCoverage: () => [],
readSources: () => [], readSources: () => [],
@@ -134,7 +127,6 @@ describe("list_sessions", () => {
updatedAt: "2026-09-21T10:00:00.000Z", updatedAt: "2026-09-21T10:00:00.000Z",
}, },
], ],
events: [],
}, },
); );
const tools = createProberTools({ ...baseCtx, chatStore, graphStore }); const tools = createProberTools({ ...baseCtx, chatStore, graphStore });
@@ -157,38 +149,28 @@ describe("list_sessions", () => {
}); });
describe("list_propositions", () => { describe("list_propositions", () => {
it("names + latest understanding + that evaluation's date — the latest event wins", async () => { it("names only — what the session covered (2026-09-27: no levels, no dating)", async () => {
const { chatStore, graphStore } = fakeStores( const { chatStore, graphStore } = fakeStores([
[node("n1", "Water cycle", [pt("c1")]), node("n2", "Evaporation", [pt("c1")])], node("n1", "Water cycle", [pt("c1")]),
{ node("n2", "Evaporation", [pt("c1")]),
events: [ ]);
// The birth session's OWN evaluations only — a later
// session's observation must not leak in here.
observation("n1", "c1", "shaky", "2026-09-20T12:00:00.000Z"),
observation("n1", "c1", "solid", "2026-09-22T12:00:00.000Z"),
observation("n1", "c2", "resisted", "2026-09-23T12:00:00.000Z"),
observation("n2", "c1", "not yet engaged", "2026-09-21T12:00:00.000Z"),
],
},
);
const tools = createProberTools({ ...baseCtx, chatStore, graphStore }); const tools = createProberTools({ ...baseCtx, chatStore, graphStore });
const out = (await tools.list_propositions!.execute!("tc", { session: "c1" })).details as string; const out = (await tools.list_propositions!.execute!("tc", { session: "c1" })).details as string;
assert.match(out, /2 proposition\(s\) born in this session:/); assert.match(out, /2 proposition\(s\) covered in this session:/);
assert.match(out, /- Water cycle — solid \(as of 2026-09-22T12:00:00.000Z\)/); assert.match(out, /- Water cycle/);
assert.match(out, /- Evaporation — not yet engaged \(as of 2026-09-21T12:00:00.000Z\)/); assert.match(out, /- Evaporation/);
// Never the other session's word, never the statement text. // The owner ruling: no levels, no dating, never the statement text.
assert.ok(!out.includes("resisted")); assert.ok(!out.includes("as of"));
assert.ok(!out.includes("solid"));
assert.ok(!out.includes("statement of")); assert.ok(!out.includes("statement of"));
}); });
it("a proposition with no observation reads 'never evaluated', undated", async () => { it("no evaluation machinery at all — the covered list is the whole output", async () => {
const { chatStore, graphStore } = fakeStores([node("n1", "Water cycle", [pt("c1")])], { const { chatStore, graphStore } = fakeStores([node("n1", "Water cycle", [pt("c1")])]);
events: [],
});
const tools = createProberTools({ ...baseCtx, chatStore, graphStore }); const tools = createProberTools({ ...baseCtx, chatStore, graphStore });
const out = (await tools.list_propositions!.execute!("tc", { session: "c1" })).details as string; const out = (await tools.list_propositions!.execute!("tc", { session: "c1" })).details as string;
assert.match(out, /- Water cycle — never evaluated/); assert.match(out, /- Water cycle/);
assert.ok(!out.includes("(as of")); assert.ok(!out.includes("never evaluated"));
}); });
it("an unknown session id — honest redirect, not a throw", async () => { it("an unknown session id — honest redirect, not a throw", async () => {
@@ -198,3 +180,56 @@ describe("list_propositions", () => {
assert.equal(out, 'No propositions born in session "no-such". Call list_sessions for the current ids.'); assert.equal(out, 'No propositions born in session "no-such". Call list_sessions for the current ids.');
}); });
}); });
// ─── the auditor's toolset (2026-09-27 relink) ───────────────────────────────
describe("auditor graph tools", () => {
const nodes = [
{ ...node("n1", "Will to power", [pt("c1")]), statement: "The will to power is Nietzsche's central drive.", aliases: ["power drive"] },
{ ...node("n2", "Eternal recurrence", [pt("c1")]), statement: "Eternal recurrence is the thought experiment of living again.", aliases: [] },
];
const { chatStore, graphStore } = fakeStores(nodes);
it("find: glob over names + statements + aliases, returning id + name + statement", async () => {
const tools = auditorGraphTools({ chatStore, graphStore });
const out = (await tools.find!.execute!("tc", { query: "*will to power*" })).details as string;
assert.match(out, /1 match\(es\):/);
assert.match(out, /- n1 :: Will to power/);
assert.match(out, /statement: The will to power is Nietzsche's central drive\./);
// Statements and aliases hit too — and a bare term gets implicit stars.
const byStatement = (await tools.find!.execute!("tc", { query: "living again" })).details as string;
assert.match(byStatement, /n2 :: Eternal recurrence/);
const byAlias = (await tools.find!.execute!("tc", { query: "power drive" })).details as string;
assert.match(byAlias, /n1/);
const miss = (await tools.find!.execute!("tc", { query: "*nothing matches this*" })).details as string;
assert.equal(miss, 'No existing proposition matches "*nothing matches this*".');
});
it("list_sessions: names, kinds, counts, ids — NO timestamps (identity work needs content, not chronology)", async () => {
const { chatStore: cs, graphStore: gs } = fakeStores(nodes, {
chats: [{ id: "c1", title: "Nietzsche seminar", createdAt: "2026-09-20T10:00:00.000Z", updatedAt: "2026-09-21T10:00:00.000Z" }],
});
const tools = auditorGraphTools({ chatStore: cs, graphStore: gs });
const out = (await tools.list_sessions!.execute!("tc", {})).details as string;
assert.match(out, /1 session\(s\):/);
assert.match(out, /Nietzsche seminar \(chat, 2 propositions\) — id: c1/);
assert.ok(!out.includes("2026-09-20"), "no dates in the auditor's view");
});
it("list_propositions: FULL statement text (identity work), no dates", async () => {
const { chatStore: cs, graphStore: gs } = fakeStores(nodes, {
mappings: [
{ chatId: "c1", sourceId: "p1", graphNodeId: "n1" },
{ chatId: "c1", sourceId: "p2", graphNodeId: "n2" },
],
});
const tools = auditorGraphTools({ chatStore: cs, graphStore: gs });
const out = (await tools.list_propositions!.execute!("tc", { session: "c1" })).details as string;
assert.match(out, /2 proposition\(s\) born in this session:/);
assert.match(out, /- n1 :: Will to power/);
assert.match(out, /statement: The will to power is Nietzsche's central drive\./);
assert.ok(!out.includes("2026-09"), "no dates in the auditor's view");
const miss = (await tools.list_propositions!.execute!("tc", { session: "no-such" })).details as string;
assert.equal(miss, 'No propositions born in session "no-such". Call list_sessions for the current ids.');
});
});
+43 -132
View File
@@ -1,9 +1,11 @@
/** /**
* Graph merger + trigger + derivation tests (vision §1, 2026-09-02 redesign). * Graph merger + trigger + derivation tests (vision §1, 2026-09-02 redesign;
* events cleanup 2026-09-27).
* The merger (core/graph.ts mergeFindings) is the pipeline's deterministic * The merger (core/graph.ts mergeFindings) is the pipeline's deterministic
* apply stage: AUDITED VERDICTS in, ledger appends out — link-or-create * apply stage: AUDITED VERDICTS in, ledger appends out — link-or-create
* identity (the auditor's linkTo, never string matching), deterministic quiz * identity (the auditor's linkTo, never string matching) and the
* events, understanding observations, and the idempotency guarantees. * idempotency guarantees. No events ledger: quiz outcomes and
* understanding observations are no longer formalized in the graph.
* *
* Ledger isolation: PATHS is fixed at import time, so the five data/graph * Ledger isolation: PATHS is fixed at import time, so the five data/graph
* ledgers are backed up and cleared before the suite and restored after * ledgers are backed up and cleared before the suite and restored after
@@ -30,15 +32,12 @@ import { LEDGER_FILES, PATHS, SOURCES_FILE } from "../config.ts";
/** The wired fs adapters, exactly what src/server/stores.ts passes in production. */ /** The wired fs adapters, exactly what src/server/stores.ts passes in production. */
const graphDeps = { chatStore, graphStore } as const satisfies GraphDeps; const graphDeps = { chatStore, graphStore } as const satisfies GraphDeps;
/** The session merge source for a chat — what the pipeline passes: the live /** The session merge source for a chat — what the pipeline passes: the
* transcript + its updatedAt checkpoint stamp. (Notes imports pass * fold stamp only (the merger reads no transcript; quiz-event rebuilding
* messages: null — see the notes-merger test below.) */ * died with the events ledger, 2026-09-27). */
function sessionSource(chatId: string): { messages: LhMessage[]; foldedAt: string } { function sessionSource(chatId: string): { foldedAt: string } {
// Wave B: the record is native history — the merger's transcript view
// reads through the one-way projection (same shape the extraction
// pipeline passes: its batch slots are projected messages).
const record = chatStore.getChat(chatId)!; const record = chatStore.getChat(chatId)!;
return { messages: record.messages, foldedAt: record.updatedAt }; return { foldedAt: record.updatedAt };
} }
// ─── ledger isolation (backup-and-clear of data/graph/*.jsonl) ────────────── // ─── ledger isolation (backup-and-clear of data/graph/*.jsonl) ──────────────
@@ -215,9 +214,8 @@ function fixtureMessages(): LhMessage[] {
].flat(); ].flat();
} }
/** The auditor's verdict for the fixture chat: three verified propositions /** The auditor's verdict for the fixture chat: three propositions
* (no linkTo — the graph starts empty), one follows edge, all quizzes * (no linkTo — the graph starts empty), one follows edge. */
* attributed, one understanding note. */
function fixtureAudit(): AuditResult { function fixtureAudit(): AuditResult {
return { return {
propositions: [ propositions: [
@@ -228,7 +226,6 @@ function fixtureAudit(): AuditResult {
aliases: ["alpha feel"], aliases: ["alpha feel"],
messages: ["a1", "a3"], messages: ["a1", "a3"],
evidence: ["fixture evidence"], evidence: ["fixture evidence"],
understanding: "solid",
}, },
{ {
id: "p2", id: "p2",
@@ -256,8 +253,6 @@ function fixtureAudit(): AuditResult {
evidence: ["fixture evidence"], evidence: ["fixture evidence"],
}, },
], ],
attribution: { quiz: { "call-1": "p1", "call-2": null, "call-3": "p1", "call-4": "p2", "call-5": "p3" } },
dropped: [],
}; };
} }
@@ -283,14 +278,13 @@ function scopedGraph(chatIds: string[]) {
return { return {
nodes: graph.nodes.filter((n) => nodeIds.has(n.id)), nodes: graph.nodes.filter((n) => nodeIds.has(n.id)),
edges: graph.edges.filter((e) => e.provenance.some((pt) => chatIds.includes(pt.chatId))), edges: graph.edges.filter((e) => e.provenance.some((pt) => chatIds.includes(pt.chatId))),
events: graph.events.filter((e) => chatIds.includes(e.chatId)),
}; };
} }
describe("the merger (mergeFindings)", () => { describe("the merger (mergeFindings)", () => {
it("creates nodes from the verdicts, rebuilds quiz events deterministically, records observations", () => { it("creates nodes from the verdicts with message-granular provenance", () => {
const stats = mergeFindings(chat1.id, fixtureAudit(), graphDeps, sessionSource(chat1.id)); const stats = mergeFindings(chat1.id, fixtureAudit(), graphDeps, sessionSource(chat1.id));
assert.deepEqual(stats, { nodes: 3, edges: 1, events: 6 }); assert.deepEqual(stats, { nodes: 3, edges: 1 });
const graph = scopedGraph([chat1.id]); const graph = scopedGraph([chat1.id]);
assert.equal(graph.nodes.length, 3); assert.equal(graph.nodes.length, 3);
@@ -321,49 +315,14 @@ describe("the merger (mergeFindings)", () => {
graphStore.readMappings().map((m) => m.sourceId), graphStore.readMappings().map((m) => m.sourceId),
["p1", "p2", "p3"], ["p1", "p2", "p3"],
); );
// Events — kinds, refs, notes, and the attribution paths.
assert.equal(graph.events.length, 6);
const byRef = new Map(graph.events.map((e) => [e.ref, e]));
const miss = byRef.get("call-1")!;
assert.equal(miss.kind, "miss");
assert.equal(miss.nodeId, idByStatement.get("Alpha intuition"), "attributed quiz");
const dontKnow = byRef.get("call-2")!;
assert.equal(dontKnow.kind, "dont-know");
assert.equal(dontKnow.nodeId, null, "unattributable → null");
assert.equal(dontKnow.question, "Q2 text");
const correct = byRef.get("call-3")!;
assert.equal(correct.kind, "correct");
assert.equal(correct.nodeId, idByStatement.get("Alpha intuition"), "attribution wins over the ignored input.nodeId");
assert.equal(correct.note, "recalled the diagram");
const miss2 = byRef.get("call-4")!;
assert.equal(miss2.kind, "miss");
assert.equal(miss2.nodeId, idByStatement.get("Beta rule"));
const dontKnow2 = byRef.get("call-5")!;
assert.equal(dontKnow2.kind, "dont-know");
assert.equal(dontKnow2.nodeId, idByStatement.get("Gamma application"));
const observation = byRef.get("obs:p1")!;
assert.equal(observation.kind, "observation");
assert.equal(observation.nodeId, idByStatement.get("Alpha intuition"));
assert.equal(observation.note, "solid");
// The dismissed quiz and the ask question became nothing.
assert.equal([...byRef.keys()].filter((r) => r === "call-6" || r === "call-7").length, 0);
}); });
it("is idempotent — re-merging the same verdicts appends nothing", () => { it("is idempotent — re-merging the same verdicts appends nothing", () => {
const stats = mergeFindings(chat1.id, fixtureAudit(), graphDeps, sessionSource(chat1.id)); const stats = mergeFindings(chat1.id, fixtureAudit(), graphDeps, sessionSource(chat1.id));
assert.deepEqual(stats, { nodes: 0, edges: 0, events: 0 }); assert.deepEqual(stats, { nodes: 0, edges: 0 });
const graph = scopedGraph([chat1.id]); const graph = scopedGraph([chat1.id]);
assert.equal(graph.nodes.length, 3); assert.equal(graph.nodes.length, 3);
assert.equal(graph.edges.length, 1); assert.equal(graph.edges.length, 1);
assert.equal(graph.events.length, 6);
// exactly one checkpoint line per merge, though — checkpoints are a log // exactly one checkpoint line per merge, though — checkpoints are a log
assert.equal(graphStore.readCheckpoints().filter((c) => c.chatId === chat1.id).length, 2); assert.equal(graphStore.readCheckpoints().filter((c) => c.chatId === chat1.id).length, 2);
}); });
@@ -398,13 +357,11 @@ describe("the merger (mergeFindings)", () => {
}, },
], ],
edges: [], edges: [],
attribution: { quiz: { "call-9": "q1" } },
dropped: [],
}, },
graphDeps, graphDeps,
sessionSource(chat2.id), sessionSource(chat2.id),
); );
assert.deepEqual(stats, { nodes: 0, edges: 0, events: 1 }); assert.deepEqual(stats, { nodes: 0, edges: 0 });
const after = scopedGraph([chat1.id, chat2.id]); const after = scopedGraph([chat1.id, chat2.id]);
assert.equal(after.nodes.length, 3, "no duplicate node"); assert.equal(after.nodes.length, 3, "no duplicate node");
@@ -436,23 +393,18 @@ describe("the merger (mergeFindings)", () => {
aliases: [], aliases: [],
messages: [], messages: [],
evidence: ["fixture evidence"], evidence: ["fixture evidence"],
understanding: "solid",
}, },
], ],
edges: [], edges: [],
attribution: { quiz: {} },
dropped: [],
}, },
graphDeps, graphDeps,
{ messages: null, foldedAt: "2026-09-13T00:00:00.000Z" }, { foldedAt: "2026-09-13T00:00:00.000Z" },
); );
const snap = graphSnapshot(graphDeps); const snap = graphSnapshot(graphDeps);
const node = snap.nodes.find((n) => n.statement === "Vim registers map single keys to editing commands.")!; const node = snap.nodes.find((n) => n.statement === "Vim registers map single keys to editing commands.")!;
assert.equal(snap.fileSources["src_snapfixture"], "/vault/learn vim.md"); assert.equal(snap.fileSources["src_snapfixture"], "/vault/learn vim.md");
assert.deepEqual(snap.nodeSessions[node.id], ["src_snapfixture"], "the file entry joins the node's sessions list"); assert.deepEqual(snap.nodeSessions[node.id], ["src_snapfixture"], "the file entry joins the node's sessions list");
assert.equal(snap.sessions["src_snapfixture"], undefined, "a notes source is never a chat session"); assert.equal(snap.sessions["src_snapfixture"], undefined, "a notes source is never a chat session");
// The understanding observation joins under the source id like any session's.
assert.equal(snap.nodeObservations[node.id]!["src_snapfixture"], "solid");
}); });
it("derives BIRTH-SESSION clusters in graphSnapshot, labeled with the session's name", () => { it("derives BIRTH-SESSION clusters in graphSnapshot, labeled with the session's name", () => {
@@ -478,11 +430,9 @@ describe("the merger (mergeFindings)", () => {
{ from: "p2", to: "p3", type: "follows", messages: [], evidence: ["e"] }, { from: "p2", to: "p3", type: "follows", messages: [], evidence: ["e"] },
{ from: "p1", to: "p3", type: "follows", messages: [], evidence: ["e"] }, { from: "p1", to: "p3", type: "follows", messages: [], evidence: ["e"] },
], ],
attribution: { quiz: {} },
dropped: [],
}, },
graphDeps, graphDeps,
{ messages: null, foldedAt: "2026-09-13T00:00:00.000Z" }, { foldedAt: "2026-09-13T00:00:00.000Z" },
); );
assert.equal(merge.nodes, 3); assert.equal(merge.nodes, 3);
const snap = graphSnapshot(graphDeps); const snap = graphSnapshot(graphDeps);
@@ -496,12 +446,14 @@ describe("the merger (mergeFindings)", () => {
assert.equal(mine!.label, "/vault/cluster notes.md", "a notes file is a birth-session — the label is its path"); assert.equal(mine!.label, "/vault/cluster notes.md", "a notes file is a birth-session — the label is its path");
}); });
it("derives sessions and per-session understanding observations in graphSnapshot", () => { it("derives the sessions index in graphSnapshot", () => {
const snap = graphSnapshot(graphDeps); const snap = graphSnapshot(graphDeps);
const idByStatement = new Map(snap.nodes.map((n) => [n.statement, n.id])); const idByStatement = new Map(snap.nodes.map((n) => [n.statement, n.id]));
// No node-state classes anymore (2026-09-08): the snapshot carries // No node-state classes (2026-09-08) and no understanding
// only the objective graph + sessions; quiz outcomes stay ledger data. // observations (2026-09-27 cleanup): the snapshot carries only the
// objective graph + sessions.
assert.equal("nodeStates" in snap, false); assert.equal("nodeStates" in snap, false);
assert.equal("nodeObservations" in snap, false);
assert.equal(snap.sessions[chat1.id], "Graph fixture"); assert.equal(snap.sessions[chat1.id], "Graph fixture");
assert.equal(snap.sessions[chat2Id], "Graph fixture 2"); assert.equal(snap.sessions[chat2Id], "Graph fixture 2");
@@ -511,12 +463,6 @@ describe("the merger (mergeFindings)", () => {
); );
// Gamma was only ever touched by chat1. // Gamma was only ever touched by chat1.
assert.deepEqual(snap.nodeSessions[idByStatement.get("Gamma application")!], [chat1.id]); assert.deepEqual(snap.nodeSessions[idByStatement.get("Gamma application")!], [chat1.id]);
// Understanding observations: (node → chat → the extractor's note).
// Alpha carries chat1's "solid" level; chat2's q1 has no level.
const alphaObs = snap.nodeObservations[idByStatement.get("Alpha intuition")!] ?? {};
assert.equal(alphaObs[chat1.id], "solid");
assert.equal(alphaObs[chat2Id], undefined);
}); });
it("never string-matches without a verdict — an identical statement without linkTo is a NEW node", () => { it("never string-matches without a verdict — an identical statement without linkTo is a NEW node", () => {
@@ -537,8 +483,6 @@ describe("the merger (mergeFindings)", () => {
}, },
], ],
edges: [], edges: [],
attribution: { quiz: {} },
dropped: [],
}, },
graphDeps, graphDeps,
sessionSource(chat3.id), sessionSource(chat3.id),
@@ -547,10 +491,9 @@ describe("the merger (mergeFindings)", () => {
assert.equal(getGraph(graphStore).nodes.filter((n) => n.statement === "Alpha intuition").length, 2); assert.equal(getGraph(graphStore).nodes.filter((n) => n.statement === "Alpha intuition").length, 2);
}); });
it("merges the notes front door: source id as chatId, no quiz events, checkpoint at import time", () => { it("merges the notes front door: source id as chatId, checkpoint at import time", () => {
// Raw personal notes carry no transcript — the merger gets no // Raw personal notes carry no transcript — no message anchors; the
// message parts, so NO quiz events are rebuilt; the understanding // file is the provenance (and no events exist to build, 2026-09-27).
// observation still lands, keyed to the source id.
const stats = mergeFindings( const stats = mergeFindings(
"src_notesfixture", "src_notesfixture",
{ {
@@ -562,17 +505,14 @@ describe("the merger (mergeFindings)", () => {
aliases: [], aliases: [],
messages: [], // notes carry no message anchors — the file is the provenance messages: [], // notes carry no message anchors — the file is the provenance
evidence: ["fixture evidence"], evidence: ["fixture evidence"],
understanding: "solid",
}, },
], ],
edges: [], edges: [],
attribution: { quiz: {} }, // normalized by the notes pipeline — nothing to attribute
dropped: [],
}, },
graphDeps, graphDeps,
{ messages: null, foldedAt: "2026-09-13T00:00:00.000Z" }, { foldedAt: "2026-09-13T00:00:00.000Z" },
); );
assert.deepEqual(stats, { nodes: 1, edges: 0, events: 1 }); assert.deepEqual(stats, { nodes: 1, edges: 0 });
const node = getGraph(graphStore).nodes.find((n) => n.statement.startsWith("Vim starts"))!; const node = getGraph(graphStore).nodes.find((n) => n.statement.startsWith("Vim starts"))!;
assert.deepEqual(node.provenance, [], "no message anchors — the mapping line carries the source"); assert.deepEqual(node.provenance, [], "no message anchors — the mapping line carries the source");
@@ -581,12 +521,6 @@ describe("the merger (mergeFindings)", () => {
sourceId: "p1", sourceId: "p1",
graphNodeId: node.id, graphNodeId: node.id,
}); });
// The observation event is file-scoped; no quiz events exist.
const events = graphStore.readEvents().filter((e) => e.chatId === "src_notesfixture");
assert.deepEqual(
events.map((e) => [e.kind, e.ref, e.note]),
[["observation", "obs:p1", "solid"]],
);
// The checkpoint stamps the import time. // The checkpoint stamps the import time.
assert.equal(graphStore.readCheckpoints().find((c) => c.chatId === "src_notesfixture")?.foldedAt, "2026-09-13T00:00:00.000Z"); assert.equal(graphStore.readCheckpoints().find((c) => c.chatId === "src_notesfixture")?.foldedAt, "2026-09-13T00:00:00.000Z");
}); });
@@ -673,12 +607,10 @@ describe("session deletion (deleteChatTraces + deleteChatSession, wave 6)", () =
].flat()); ].flat());
const auditA: AuditResult = { const auditA: AuditResult = {
propositions: [ propositions: [
{ id: "p1", name: "Shared core", statement: "Shared core statement", aliases: ["shared feel"], messages: ["du1"], evidence: ["e"], understanding: "solid" }, { id: "p1", name: "Shared core", statement: "Shared core statement", aliases: ["shared feel"], messages: ["du1"], evidence: ["e"] },
{ id: "p2", name: "Solo fact", statement: "Solo fact statement", aliases: ["solo feel"], messages: ["du1"], evidence: ["e"] }, { id: "p2", name: "Solo fact", statement: "Solo fact statement", aliases: ["solo feel"], messages: ["du1"], evidence: ["e"] },
], ],
edges: [{ from: "p1", to: "p2", type: "follows", messages: ["du1"], evidence: ["e"] }], edges: [{ from: "p1", to: "p2", type: "follows", messages: ["du1"], evidence: ["e"] }],
attribution: { quiz: { "d-call-1": "p1" } },
dropped: [],
}; };
mergeFindings(dA.id, auditA, graphDeps, sessionSource(dA.id)); mergeFindings(dA.id, auditA, graphDeps, sessionSource(dA.id));
// Resolve through THIS session's mappings — the shared ledger holds // Resolve through THIS session's mappings — the shared ledger holds
@@ -701,12 +633,10 @@ describe("session deletion (deleteChatTraces + deleteChatSession, wave 6)", () =
].flat()); ].flat());
const auditB: AuditResult = { const auditB: AuditResult = {
propositions: [ propositions: [
{ id: "q1", name: "Shared again", statement: "Shared core, rephrased", aliases: ["B phrasing"], messages: ["eu1"], evidence: ["e"], linkTo: xId, understanding: "shaky" }, { id: "q1", name: "Shared again", statement: "Shared core, rephrased", aliases: ["B phrasing"], messages: ["eu1"], evidence: ["e"], linkTo: xId },
{ id: "q2", name: "B extra", statement: "B extra statement", aliases: ["b extra feel"], messages: ["eu1"], evidence: ["e"] }, { id: "q2", name: "B extra", statement: "B extra statement", aliases: ["b extra feel"], messages: ["eu1"], evidence: ["e"] },
], ],
edges: [{ from: "q1", to: "q2", type: "follows", messages: ["eu1"], evidence: ["e"] }], edges: [{ from: "q1", to: "q2", type: "follows", messages: ["eu1"], evidence: ["e"] }],
attribution: { quiz: { "e-call-1": "q1" } },
dropped: [],
}; };
mergeFindings(dB.id, auditB, graphDeps, sessionSource(dB.id)); mergeFindings(dB.id, auditB, graphDeps, sessionSource(dB.id));
const zId = graphStore.readMappings().find((m) => m.chatId === dB.id && m.sourceId === "q2")!.graphNodeId; const zId = graphStore.readMappings().find((m) => m.chatId === dB.id && m.sourceId === "q2")!.graphNodeId;
@@ -730,7 +660,6 @@ describe("session deletion (deleteChatTraces + deleteChatSession, wave 6)", () =
assert.deepEqual(stats, { assert.deepEqual(stats, {
nodesRemoved: 1, nodesRemoved: 1,
edgesRemoved: 1, edgesRemoved: 1,
eventsRemoved: 2,
mappingsRemoved: 2, mappingsRemoved: 2,
coverageRemoved: 2, coverageRemoved: 2,
checkpointsRemoved: 1, checkpointsRemoved: 1,
@@ -753,11 +682,6 @@ describe("session deletion (deleteChatTraces + deleteChatSession, wave 6)", () =
// Edges: dA's assertion X→Y died; dB's X→Z survived. // Edges: dA's assertion X→Y died; dB's X→Z survived.
const scoped = scopedGraph([dA.id, dB.id]); const scoped = scopedGraph([dA.id, dB.id]);
assert.deepEqual(scoped.edges, [{ from: xId, to: zId, type: "follows", provenance: [{ chatId: dB.id, messageId: "eu1" }] }]); assert.deepEqual(scoped.edges, [{ from: xId, to: zId, type: "follows", provenance: [{ chatId: dB.id, messageId: "eu1" }] }]);
// Events: only dB's quiz + observation remain.
assert.deepEqual(scoped.events.map((e) => [e.kind, e.chatId === dB.id]), [
["miss", true],
["observation", true],
]);
// Mappings: dA's audit trail died; dB's stays. // Mappings: dA's audit trail died; dB's stays.
const mappings = graphStore.readMappings().filter((m) => [dA.id, dB.id].includes(m.chatId)); const mappings = graphStore.readMappings().filter((m) => [dA.id, dB.id].includes(m.chatId));
assert.deepEqual(mappings, [{ chatId: dB.id, sourceId: "q1", graphNodeId: xId }, { chatId: dB.id, sourceId: "q2", graphNodeId: zId }]); assert.deepEqual(mappings, [{ chatId: dB.id, sourceId: "q1", graphNodeId: xId }, { chatId: dB.id, sourceId: "q2", graphNodeId: zId }]);
@@ -768,15 +692,16 @@ describe("session deletion (deleteChatTraces + deleteChatSession, wave 6)", () =
// The snapshot resolves X to dB only — the deleted chat left no ghost. // The snapshot resolves X to dB only — the deleted chat left no ghost.
const snapshot = graphSnapshot(graphDeps); const snapshot = graphSnapshot(graphDeps);
assert.deepEqual(snapshot.nodeSessions[xId], [dB.id]); assert.deepEqual(snapshot.nodeSessions[xId], [dB.id]);
assert.deepEqual(snapshot.nodeObservations[xId], { [dB.id]: "shaky" });
assert.ok(snapshot.sessions[dB.id] && !snapshot.sessions[dA.id]); assert.ok(snapshot.sessions[dB.id] && !snapshot.sessions[dA.id]);
// Atomic rewrites left no temp files, and appends still work. // Atomic rewrites left no temp files, and appends still work.
assert.equal(fs.readdirSync(PATHS.graph).filter((f) => f.includes(".tmp-")).length, 0); assert.equal(fs.readdirSync(PATHS.graph).filter((f) => f.includes(".tmp-")).length, 0);
const eventsBefore = graphStore.readEvents().length; const nodesBefore = graphStore.readNodes().length;
graphStore.appendEvent({ nodeId: null, kind: "correct", chatId: "chat_del_append_probe", at: "2026-09-23T00:04:00.000Z" }); // Foreign provenance: the probe is another chat's data, so later
assert.equal(graphStore.readEvents().length, eventsBefore + 1); // deletions must leave it untouched (the foreign-data doctrine).
graphStore.appendNode({ id: "n_del_append_probe", name: "Probe", statement: "Probe statement", aliases: [], createdAt: "2026-09-23T00:04:00.000Z", provenance: [{ chatId: "chat_foreign_probe", messageId: "" }] });
assert.equal(graphStore.readNodes().length, nodesBefore + 1);
// Idempotent: a second run finds nothing. // Idempotent: a second run finds nothing.
assert.deepEqual(deleteChatTraces(graphStore, dA.id, "delete-nodes"), { ...stats, nodesRemoved: 0, edgesRemoved: 0, eventsRemoved: 0, mappingsRemoved: 0, coverageRemoved: 0, checkpointsRemoved: 0 } satisfies ChatDeletionStats); assert.deepEqual(deleteChatTraces(graphStore, dA.id, "delete-nodes"), { ...stats, nodesRemoved: 0, edgesRemoved: 0, mappingsRemoved: 0, coverageRemoved: 0, checkpointsRemoved: 0 } satisfies ChatDeletionStats);
}); });
it("preserve-graph: only the extraction bookkeeping dies — the graph ledgers are byte-identical", () => { it("preserve-graph: only the extraction bookkeeping dies — the graph ledgers are byte-identical", () => {
@@ -784,7 +709,6 @@ describe("session deletion (deleteChatTraces + deleteChatSession, wave 6)", () =
const before = { const before = {
nodes: graphStore.readNodes(), nodes: graphStore.readNodes(),
edges: graphStore.readEdges(), edges: graphStore.readEdges(),
events: graphStore.readEvents(),
mappings: graphStore.readMappings(), mappings: graphStore.readMappings(),
positions: graphStore.readPositions(), positions: graphStore.readPositions(),
}; };
@@ -792,14 +716,12 @@ describe("session deletion (deleteChatTraces + deleteChatSession, wave 6)", () =
assert.deepEqual(stats, { assert.deepEqual(stats, {
nodesRemoved: 0, nodesRemoved: 0,
edgesRemoved: 0, edgesRemoved: 0,
eventsRemoved: 0,
mappingsRemoved: 0, mappingsRemoved: 0,
coverageRemoved: 2, coverageRemoved: 2,
checkpointsRemoved: 1, checkpointsRemoved: 1,
} satisfies ChatDeletionStats); } satisfies ChatDeletionStats);
assert.deepEqual(graphStore.readNodes(), before.nodes); assert.deepEqual(graphStore.readNodes(), before.nodes);
assert.deepEqual(graphStore.readEdges(), before.edges); assert.deepEqual(graphStore.readEdges(), before.edges);
assert.deepEqual(graphStore.readEvents(), before.events);
assert.deepEqual(graphStore.readMappings(), before.mappings); assert.deepEqual(graphStore.readMappings(), before.mappings);
assert.deepEqual(graphStore.readPositions(), before.positions); assert.deepEqual(graphStore.readPositions(), before.positions);
assert.equal(graphStore.readCoverage().some((row) => row.chatId === dA.id), false); assert.equal(graphStore.readCoverage().some((row) => row.chatId === dA.id), false);
@@ -853,7 +775,7 @@ describe("single-node deletion (deleteNodeTraces + deleteGraphNode, node interlu
// Clean slate per test: each test in this describe builds its own // Clean slate per test: each test in this describe builds its own
// scenario from empty ledgers (the suite's isolation covers the // scenario from empty ledgers (the suite's isolation covers the
// whole file; this covers describe-internal stacking). // whole file; this covers describe-internal stacking).
graphStore.rewriteLedgers({ nodes: [], edges: [], events: [], mappings: [] }); graphStore.rewriteLedgers({ nodes: [], edges: [], mappings: [] });
graphStore.writePositions({}); graphStore.writePositions({});
graphStore.appendNode({ graphStore.appendNode({
id: "n_dead", id: "n_dead",
@@ -884,10 +806,6 @@ describe("single-node deletion (deleteNodeTraces + deleteGraphNode, node interlu
graphStore.appendEdge({ from: "n_dead", to: "n_keep", type: "follows", provenance: [{ chatId: "chat_node_a", messageId: "m1" }] }); graphStore.appendEdge({ from: "n_dead", to: "n_keep", type: "follows", provenance: [{ chatId: "chat_node_a", messageId: "m1" }] });
graphStore.appendEdge({ from: "n_keep", to: "n_dead", type: "supports", provenance: [{ chatId: "chat_node_b", messageId: "m3" }] }); graphStore.appendEdge({ from: "n_keep", to: "n_dead", type: "supports", provenance: [{ chatId: "chat_node_b", messageId: "m3" }] });
graphStore.appendEdge({ from: "n_keep", to: "n_other", type: "follows", provenance: [{ chatId: "chat_node_b", messageId: "m3" }] }); graphStore.appendEdge({ from: "n_keep", to: "n_other", type: "follows", provenance: [{ chatId: "chat_node_b", messageId: "m3" }] });
// Events: two sessions' evidence on X, one on the neighbor.
graphStore.appendEvent({ nodeId: "n_dead", kind: "correct", chatId: "chat_node_a", at: "2026-09-23T00:01:00.000Z", ref: "q1" });
graphStore.appendEvent({ nodeId: "n_dead", kind: "observation", chatId: "chat_node_b", at: "2026-09-23T00:02:00.000Z", note: "shaky", ref: "obs:p1" });
graphStore.appendEvent({ nodeId: "n_keep", kind: "miss", chatId: "chat_node_a", at: "2026-09-23T00:03:00.000Z", ref: "q2" });
// Mappings: the notes anchor + the chat's audit trail. // Mappings: the notes anchor + the chat's audit trail.
graphStore.appendMapping({ chatId: "src_nodefile", sourceId: "p1", graphNodeId: "n_dead" }); graphStore.appendMapping({ chatId: "src_nodefile", sourceId: "p1", graphNodeId: "n_dead" });
graphStore.appendMapping({ chatId: "chat_node_a", sourceId: "p1", graphNodeId: "n_dead" }); graphStore.appendMapping({ chatId: "chat_node_a", sourceId: "p1", graphNodeId: "n_dead" });
@@ -900,11 +818,9 @@ describe("single-node deletion (deleteNodeTraces + deleteGraphNode, node interlu
buildNodeScenario(); buildNodeScenario();
assert.deepEqual(nodeDeletionImpact(graphStore, "n_dead"), { assert.deepEqual(nodeDeletionImpact(graphStore, "n_dead"), {
edges: 2, edges: 2,
quizEvents: 1, // The file anchor + the re-encounter — every referencing
observations: 1, // session, deduped and sorted.
// The file anchor, the re-encounter, and the second session's sessions: ["chat_node_a", "src_nodefile"],
// observation — every referencing session, deduped and sorted.
sessions: ["chat_node_a", "chat_node_b", "src_nodefile"],
}); });
// Unknown node: null (the route's 404). // Unknown node: null (the route's 404).
assert.equal(nodeDeletionImpact(graphStore, "n_missing"), null); assert.equal(nodeDeletionImpact(graphStore, "n_missing"), null);
@@ -914,7 +830,7 @@ describe("single-node deletion (deleteNodeTraces + deleteGraphNode, node interlu
buildNodeScenario(); buildNodeScenario();
const keepLineBefore = JSON.stringify(graphStore.readNodes().find((n) => n.id === "n_keep")); const keepLineBefore = JSON.stringify(graphStore.readNodes().find((n) => n.id === "n_keep"));
const stats = deleteNodeTraces(graphStore, "n_dead"); const stats = deleteNodeTraces(graphStore, "n_dead");
assert.deepEqual(stats, { edgesRemoved: 2, eventsRemoved: 2, mappingsRemoved: 2 } satisfies NodeDeletionStats); assert.deepEqual(stats, { edgesRemoved: 2, mappingsRemoved: 2 } satisfies NodeDeletionStats);
// The node's EVERY line is gone; the neighbor is byte-identical // The node's EVERY line is gone; the neighbor is byte-identical
// (full death is node-scoped — other nodes' sessions keep their // (full death is node-scoped — other nodes' sessions keep their
@@ -930,11 +846,6 @@ describe("single-node deletion (deleteNodeTraces + deleteGraphNode, node interlu
graphStore.readEdges().map((e) => [e.from, e.to]), graphStore.readEdges().map((e) => [e.from, e.to]),
[["n_keep", "n_other"]], [["n_keep", "n_other"]],
); );
// The evidence died with the thing it was about; the neighbor's stays.
assert.deepEqual(
graphStore.readEvents().map((e) => [e.nodeId, e.kind]),
[["n_keep", "miss"]],
);
// The audit trail: X's mappings gone, the neighbor's stay. // The audit trail: X's mappings gone, the neighbor's stay.
assert.deepEqual(graphStore.readMappings().map((m) => m.graphNodeId), ["n_keep"]); assert.deepEqual(graphStore.readMappings().map((m) => m.graphNodeId), ["n_keep"]);
// The resting position died with the node. // The resting position died with the node.
@@ -957,7 +868,7 @@ describe("single-node deletion (deleteNodeTraces + deleteGraphNode, node interlu
provenance: [{ chatId: "chat_node_solo", messageId: "m1" }], provenance: [{ chatId: "chat_node_solo", messageId: "m1" }],
}); });
const stats = deleteNodeTraces(graphStore, "n_solo"); const stats = deleteNodeTraces(graphStore, "n_solo");
assert.deepEqual(stats, { edgesRemoved: 0, eventsRemoved: 0, mappingsRemoved: 0 } satisfies NodeDeletionStats); assert.deepEqual(stats, { edgesRemoved: 0, mappingsRemoved: 0 } satisfies NodeDeletionStats);
assert.equal(graphStore.readNodes().some((n) => n.id === "n_solo"), false); assert.equal(graphStore.readNodes().some((n) => n.id === "n_solo"), false);
}); });
@@ -1007,7 +918,7 @@ describe("single-node deletion (deleteNodeTraces + deleteGraphNode, node interlu
const revisionBefore = graphRevision(); const revisionBefore = graphRevision();
const outcome = deleteGraphNode("n_orch"); const outcome = deleteGraphNode("n_orch");
assert.ok(outcome.ok); assert.ok(outcome.ok);
assert.deepEqual(outcome.stats, { edgesRemoved: 0, eventsRemoved: 0, mappingsRemoved: 0 }); assert.deepEqual(outcome.stats, { edgesRemoved: 0, mappingsRemoved: 0 });
assert.ok(outcome.graphRevision > 0); assert.ok(outcome.graphRevision > 0);
assert.ok(outcome.graphRevision >= revisionBefore, "the version stamp moved"); assert.ok(outcome.graphRevision >= revisionBefore, "the version stamp moved");
assert.equal(graphStore.readNodes().some((n) => n.id === "n_orch"), false); assert.equal(graphStore.readNodes().some((n) => n.id === "n_orch"), false);
+8 -32
View File
@@ -11,7 +11,7 @@
import assert from "node:assert/strict"; import assert from "node:assert/strict";
import { describe, it } from "node:test"; import { describe, it } from "node:test";
import type { GraphEdge, GraphNode, LedgerEvent, NotesSource, ChatSummary } from "../src/core/contracts.ts"; import type { GraphEdge, GraphNode, NotesSource, ChatSummary } from "../src/core/contracts.ts";
import type { ChatStore, GraphStore } from "../src/core/ports.ts"; import type { ChatStore, GraphStore } from "../src/core/ports.ts";
import { import {
graphSessionIndex, graphSessionIndex,
@@ -36,7 +36,6 @@ const node = (id: string, name: string, provenance: Array<{ chatId: string; mess
function fakeStores( function fakeStores(
nodes: GraphNode[], nodes: GraphNode[],
over: { over: {
events?: LedgerEvent[];
chats?: Array<Pick<ChatSummary, "id" | "title" | "createdAt" | "updatedAt">>; chats?: Array<Pick<ChatSummary, "id" | "title" | "createdAt" | "updatedAt">>;
sources?: Array<Pick<NotesSource, "id" | "filePath" | "importedAt">>; sources?: Array<Pick<NotesSource, "id" | "filePath" | "importedAt">>;
mappings?: Array<{ chatId: string; sourceId: string; graphNodeId: string }>; mappings?: Array<{ chatId: string; sourceId: string; graphNodeId: string }>;
@@ -46,7 +45,6 @@ function fakeStores(
const graphStore: GraphStore = { const graphStore: GraphStore = {
readNodes: () => nodes, readNodes: () => nodes,
readEdges: () => edges, readEdges: () => edges,
readEvents: () => over.events ?? [],
readMappings: () => over.mappings ?? [], readMappings: () => over.mappings ?? [],
readCheckpoints: () => [], readCheckpoints: () => [],
readCoverage: () => [], readCoverage: () => [],
@@ -160,12 +158,8 @@ describe("the notes mapping-line fallback", () => {
], "the file births under its src id; the childless chat has no entry (birth sessions only)"); ], "the file births under its src id; the childless chat has no entry (birth sessions only)");
}); });
it("graphSessionPropositions lists the file's births under the src id", () => { it("graphSessionPropositions lists the file's births under the src id (a pure birth list, 2026-09-27)", () => {
const events: LedgerEvent[] = [
{ nodeId: "a", kind: "observation", chatId: "src_abc", at: "2026-09-22T00:00:00.000Z", note: "solid", ref: "obs:p1" },
];
const deps = fakeStores([node("a", "Alpha", []), node("b", "Beta", [])], { const deps = fakeStores([node("a", "Alpha", []), node("b", "Beta", [])], {
events,
mappings: [ mappings: [
{ chatId: "src_abc", sourceId: "p1", graphNodeId: "a" }, { chatId: "src_abc", sourceId: "p1", graphNodeId: "a" },
{ chatId: "src_abc", sourceId: "p2", graphNodeId: "b" }, { chatId: "src_abc", sourceId: "p2", graphNodeId: "b" },
@@ -173,8 +167,8 @@ describe("the notes mapping-line fallback", () => {
sources: [{ id: "src_abc", filePath: "/vault/learn vim.md", importedAt: "2026-09-22T00:00:00.000Z" }], sources: [{ id: "src_abc", filePath: "/vault/learn vim.md", importedAt: "2026-09-22T00:00:00.000Z" }],
}); });
assert.deepEqual(graphSessionPropositions(deps, "src_abc"), [ assert.deepEqual(graphSessionPropositions(deps, "src_abc"), [
{ nodeId: "a", name: "Alpha", understanding: "solid", evaluatedAt: "2026-09-22T00:00:00.000Z" }, { nodeId: "a", name: "Alpha" },
{ nodeId: "b", name: "Beta", understanding: null, evaluatedAt: null }, { nodeId: "b", name: "Beta" },
]); ]);
}); });
}); });
@@ -272,34 +266,16 @@ describe("graphSessionIndex (the prober's session list)", () => {
}); });
describe("graphSessionPropositions (the per-session view)", () => { describe("graphSessionPropositions (the per-session view)", () => {
it("birth propositions with the latest extractor evaluation AND that event's own timestamp", () => { it("birth propositions — a pure list, no evaluations (2026-09-27 cleanup)", () => {
const nodes = [ const nodes = [
node("a", "Alpha", [pt("chat1")]), node("a", "Alpha", [pt("chat1")]),
node("b", "Beta", [pt("chat1")]), node("b", "Beta", [pt("chat1")]),
node("c", "Gamma", [pt("chat2")]), // born elsewhere — never listed under chat1 node("c", "Gamma", [pt("chat2")]), // born elsewhere — never listed under chat1
]; ];
const events: LedgerEvent[] = [ const deps = fakeStores(nodes);
{ nodeId: "a", kind: "observation", chatId: "chat1", at: "2026-09-20T00:00:00.000Z", note: "shaky", ref: "obs:p1" },
// A LATER evaluation supersedes the earlier one for (a, chat1).
{ nodeId: "a", kind: "observation", chatId: "chat1", at: "2026-09-21T00:00:00.000Z", note: "solid", ref: "obs:p2" },
// Quiz evidence (owner ruling 2026-09-23): recorded in the ledger
// but it NEVER moves the evaluation date.
{ nodeId: "a", kind: "correct", chatId: "chat1", at: "2026-09-22T00:00:00.000Z", note: "got it" },
// Another session's evaluation of the same node is not ours.
{ nodeId: "a", kind: "observation", chatId: "chat2", at: "2026-09-23T00:00:00.000Z", note: "resisted" },
// Same-at events: ledger order breaks the tie (the snapshot's rule).
{ nodeId: "b", kind: "observation", chatId: "chat1", at: "2026-09-20T00:00:00.000Z", note: "accepted", ref: "obs:p3" },
{ nodeId: "b", kind: "observation", chatId: "chat1", at: "2026-09-20T00:00:00.000Z", note: "not yet engaged", ref: "obs:p4" },
];
const deps = fakeStores(nodes, { events });
assert.deepEqual(graphSessionPropositions(deps, "chat1"), [ assert.deepEqual(graphSessionPropositions(deps, "chat1"), [
{ nodeId: "a", name: "Alpha", understanding: "solid", evaluatedAt: "2026-09-21T00:00:00.000Z" }, { nodeId: "a", name: "Alpha" },
{ nodeId: "b", name: "Beta", understanding: "not yet engaged", evaluatedAt: "2026-09-20T00:00:00.000Z" }, { nodeId: "b", name: "Beta" },
]);
// A node the session never evaluated shows nulls, never a guess.
const fresh = fakeStores([node("x", "X", [pt("chat1")])], { events });
assert.deepEqual(graphSessionPropositions(fresh, "chat1"), [
{ nodeId: "x", name: "X", understanding: null, evaluatedAt: null },
]); ]);
}); });
}); });
+138
View File
@@ -0,0 +1,138 @@
/**
* The shared session-markdown engine (core/session-markdown.ts, owner ruling
* 2026-09-27): ONE renderer for the extractor's source view and the future
* user-facing session export — id-anchored speaker blocks, readable
* quiz/recall/question blocks, one-line notes for other tools. Pinned here:
* block shapes, anchor handling (on/off), title header, empty-message
* skipping, and content parity with the coverage line builder.
*/
import assert from "node:assert/strict";
import { describe, it } from "node:test";
import type { LhMessage } from "../src/core/native-messages.ts";
import { messageMarkdown, sessionMarkdown } from "../src/core/session-markdown.ts";
import { messageViewLines } from "../src/core/coverage.ts";
import { coverageSlots } from "../src/core/coverage.ts";
function user(id: string, text: string): LhMessage {
return { id, role: "user", content: text, timestamp: Date.now() } as never;
}
function teacher(id: string, text: string): LhMessage {
return {
id,
role: "assistant",
content: [{ type: "text", text }],
api: "openai-completions",
provider: "openrouter",
model: "test",
usage: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
stopReason: "stop",
timestamp: Date.now(),
} as never;
}
function result(
id: string,
toolCallId: string,
toolName: string,
details: unknown,
isError = false,
): LhMessage {
return {
id,
role: "toolResult",
toolCallId,
toolName,
content: [{ type: "text", text: "ok" }],
details,
isError,
timestamp: Date.now(),
} as never;
}
describe("messageMarkdown", () => {
it("renders speaker blocks with the id anchor", () => {
const md = messageMarkdown(user("u1", "teach me the chain rule"), "u1");
assert.equal(md, "### Learner · u1\n\nteach me the chain rule");
const md2 = messageMarkdown(teacher("a1", "The derivative measures change."), "a1");
assert.equal(md2, "### Teacher · a1\n\nThe derivative measures change.");
});
it("anchors off — the same document without the ids (the user export)", () => {
const md = messageMarkdown(user("u1", "hello"), "u1", false);
assert.equal(md, "### Learner\n\nhello");
});
it("quiz blocks: question + outcome + the learner's note", () => {
const md = messageMarkdown(
result("r1", "q1", "quiz", { questionText: "What is a derivative?", correct: false, dontKnow: false, note: "mixed up signs" }),
"r1",
)!;
assert.match(md, /^### Quiz · r1$/m);
assert.match(md, /\*\*Q:\*\* What is a derivative\?/);
assert.match(md, /The learner answered incorrectly\./);
assert.match(md, /\*\(his thinking: mixed up signs\)\*/);
});
it("explain results: the learner's own words are shown", () => {
const md = messageMarkdown(
result("r1", "e1", "explain", { questionText: "Why does Mercury stay near the Sun?", text: "it never strays far" }),
"r1",
)!;
assert.match(md, /^### Recall check · r1$/m);
assert.match(md, /The learner answered in his own words: it never strays far/);
});
it("ask blocks: question + chosen options + free text", () => {
const chosen = messageMarkdown(
result("r1", "a1", "ask_user_question", { questionText: "Which way?", answers: ["second"], other: "also the third" }),
"r1",
)!;
assert.match(chosen, /^### Question · r1$/m);
assert.match(chosen, /The learner chose: second, also the third/);
const free = messageMarkdown(
result("r2", "a2", "ask_user_question", { questionText: "Why?", text: "because" }),
"r2",
)!;
assert.match(free, /The learner answered: because/);
});
it("other clean tool results: one line; failed calls and markers: nothing", () => {
assert.equal(messageMarkdown(result("r1", "j1", "research", { brief: "long" }), "r1"), "### Tool · r1\n\n*research result delivered*");
assert.equal(messageMarkdown(result("r2", "j2", "research", {}, true), "r2"), null, "a failed call is retry context");
assert.equal(messageMarkdown(user("u3", " "), "u3"), null, "empty text renders nothing");
});
});
describe("sessionMarkdown", () => {
it("assembles blocks in order under the title header, skipping nothings", () => {
const md = sessionMarkdown([user("u1", "go"), teacher("a1", "First."), user("u2", " ")], { title: "My session" });
assert.equal(md, "# My session\n\n### Learner · u1\n\ngo\n\n### Teacher · a1\n\nFirst.");
});
it("no title → no header; anchors default on", () => {
const md = sessionMarkdown([user("u1", "go")]);
assert.equal(md, "### Learner · u1\n\ngo");
});
it("content parity with the coverage line builder (2026-09-27 split)", () => {
// Every message that produces view lines produces a markdown block,
// and every message that renders nothing renders nothing in either —
// same messages, two formats, one seam decision. The FINGERPRINT
// stays on the line format (stable — no mass re-extraction when
// presentation changes); markdown is presentation only.
const messages: LhMessage[] = [
user("u1", "teach me"),
teacher("a1", "The chain rule multiplies local rates."),
result("r1", "q1", "quiz", { questionText: "Q?", correct: true, dontKnow: false }),
result("r2", "j1", "research", { brief: "b" }),
result("r3", "j2", "research", {}, true),
];
for (const message of messages) {
const hasLines = messageViewLines(message, message.id).length > 0;
const hasMarkdown = messageMarkdown(message, message.id) !== null;
assert.equal(hasMarkdown, hasLines, `parity for ${message.id}`);
}
});
});
+2 -2
View File
@@ -243,7 +243,7 @@ test("appendExtractionUsage: appends, idempotent per startedAt, not in list payl
startedAt, startedAt,
at: "2026-09-09T00:00:00Z", at: "2026-09-09T00:00:00Z",
durationMs: 42_000, durationMs: 42_000,
stats: { nodes: 1, edges: 2, events: 3 }, stats: { nodes: 1, edges: 2 },
stages: { stages: {
extractor: { model: "openrouter:z-ai/glm-5.3-flash", steps: [call({ inputTokens: 100, outputTokens: 10 })] }, extractor: { model: "openrouter:z-ai/glm-5.3-flash", steps: [call({ inputTokens: 100, outputTokens: 10 })] },
auditor: { model: "openrouter:z-ai/glm-5.3-flash", steps: [call({ inputTokens: 50, outputTokens: 5 })] }, auditor: { model: "openrouter:z-ai/glm-5.3-flash", steps: [call({ inputTokens: 50, outputTokens: 5 })] },
@@ -280,7 +280,7 @@ test("appendSourceUsage: appends to the source record, idempotent per startedAt,
startedAt, startedAt,
at: "2026-09-13T00:00:00Z", at: "2026-09-13T00:00:00Z",
durationMs: 42_000, durationMs: 42_000,
stats: { nodes: 1, edges: 2, events: 3 }, stats: { nodes: 1, edges: 2 },
stages: { stages: {
extractor: { model: "openrouter:z-ai/glm-5.3-flash", steps: [call({ inputTokens: 100, outputTokens: 10 })] }, extractor: { model: "openrouter:z-ai/glm-5.3-flash", steps: [call({ inputTokens: 100, outputTokens: 10 })] },
auditor: { model: "openrouter:z-ai/glm-5.3-flash", steps: [call({ inputTokens: 50, outputTokens: 5 })] }, auditor: { model: "openrouter:z-ai/glm-5.3-flash", steps: [call({ inputTokens: 50, outputTokens: 5 })] },