Extraction pipeline: extractor → auditor → merger — the sole graph writer (stage 5)

The knowledge graph's write channel is now the pivot's target architecture (vision §1 construction): - src/agent/subagents/extraction.ts — the pipeline factory (createExtractionPipeline({ chatStore, graphStore, extractor?, auditor? }); models env LH_EXTRACT_MODEL ?? LH_MODEL): * transcriptView(): the compact view both LLM stages see — final plan tree with node ids, conversation verbatim, QUIZ lines with outcomes, mark_done lines; job tool parts are one-liners, plan ops fold into the tree line. * Extractor: transcript → SessionFindings (propositions incl. misconceptions and prose-only supports edges, attribution for UNTAGGED probes, all with verbatim evidence) via one zod-validated submit_findings call. * Auditor: transcript + findings → the verified subset + drops (the faithfulness net) via submit_review. - src/core/graph.ts — emitChat/foldStaleChats/replayFinalTree retire; mergeFindings() replaces them as the pipeline's deterministic apply stage: exact/alias identity resolution (one mapping line per (chat, graph node)), edge dedupe, and event rebuild FROM THE TRANSCRIPT's tool parts — kinds/timestamps/refs/notes deterministic, the LLM's only event say being WHICH proposition an untagged probe tested (retroactive — the vision's open question #3, resolved structurally: tagged events resolve deterministically; attribution rescues untagged and tagged-but-unmapped ones; unattributable land null + question text). pendingQuizAnswers() is the trigger helper. - Trigger (turn-service.ts triggerExtraction, replacing the old foldTurnIntoGraph body): once a chat accumulates EXTRACTION.everyNQuizAnswers (LH_EXTRACT_N, default 3) new answered quizzes, fire the pipeline async — single-flight per chat, failure checkpoints nothing and re-runs on the next trigger. Deliberately a v1 placeholder isolated in ONE seam, trigger-agnostic pipeline. - GET /api/graph is now a pure read (foldStaleChats gone). - NodeMapping.treeNodeId → sourceId (the mapping key is now the writer's local proposition id — tree node when tree-derived); store strict-read schema updated. Local ledger data regenerates through the pipeline. - Tests: graph.test.ts rewritten around mergeFindings (identity, attribution paths incl. ghost-rescue, idempotency, cross-session provenance, trigger counting), graph-context.test.ts merge section converted to findings-driven, NEW extraction.test.ts — transcriptView units + the full pipeline with mock models (audit drops, failure checkpoints nothing, single-flight). 195/195, tsc/eslint/build clean. Probe attribution (vision open question): probes attach retroactively by the extractor against the whole transcript; a pre-tree probe maps to a proposition the tree only articulates later. Fuzzy/embedding merge adjudication stays deferred (conservative exact/alias only).
2026-08-31 22:36:41 +03:00
parent 59e1fee850
commit af90fadcda
12 changed files with 1168 additions and 320 deletions
@@ -0,0 +1,309 @@
+import { ToolLoopAgent, stepCountIs, tool, type LanguageModel, type UIMessage } from "ai";
+import { openrouter } from "@openrouter/ai-sdk-provider";
+import { z } from "zod";
+import type { AuditResult, ChatRecord, SessionFindings } from "../../core/contracts.ts";
+import { mergeFindings, type EmissionStats, type GraphDeps } from "../../core/graph.ts";
+
+/**
+ * The extraction pipeline (vision §1 construction) — the sole graph writer.
+ * Three stages, one orchestrator:
+ *
+ *   extractor (LLM)  transcript → SessionFindings (propositions, edges,
+ *                    event attribution — each with verbatim evidence)
+ *   auditor (LLM)    findings → AuditResult (the verified subset + drops)
+ *   merger (core)    mergeFindings() → deterministic ledger appends
+ *
+ * The LLM stages supply only the judgment calls; the deterministic rebuild
+ * in the merger owns everything that must never be mis-copied (event kinds,
+ * timestamps, refs). Both agents are single-tool-call workers: their whole
+ * deliverable is one zod-validated submit call — a ToolLoopAgent purely so
+ * the model can retry a schema-invalid call.
+ *
+ * Created via createExtractionPipeline(deps) — the composition root
+ * (chat-handler) injects the storage ports + models; this module never
+ * imports an adapter. runExtraction is single-flight per chat: a second
+ * trigger while one run is in flight is a no-op (the next trigger picks up
+ * whatever the in-flight run missed). On failure nothing is checkpointed —
+ * the next trigger re-runs the whole thing.
+ */
+
+const DEFAULT_EXTRACTION_MODEL =
+	process.env.LH_EXTRACT_MODEL ?? process.env.LH_MODEL ?? "anthropic/claude-sonnet-4.5";
+
+// ─── the transcript view (what the LLM sees) ────────────────────────────────
+
+/**
+ * A quiz tool part's outcome, one line: the attribution evidence.
+ * (question text + what the learner did — kind/correct are rebuilt by the
+ * merger; the extractor only needs enough to attribute.)
+ */
+function quizLine(part: { toolCallId: string; input: unknown; output: unknown }): string {
+	const input = part.input as { question?: unknown } | undefined;
+	const out = part.output as Partial<{ questionText: string; correct: boolean; dontKnow: boolean; note: string }> | undefined;
+	const question = typeof input?.question === "string" ? input.question : out?.questionText ?? "(no question)";
+	const outcome = out?.dontKnow === true ? "did not know" : out?.correct === true ? "answered correctly" : "answered incorrectly";
+	const note = typeof out?.note === "string" && out.note ? ` (note: ${out.note})` : "";
+	return `QUIZ [${part.toolCallId}] ${question} — the learner ${outcome}${note}`;
+}
+
+/**
+ * The compact transcript the extractor + auditor both see: user/teacher
+ * text verbatim (evidence quotes must match it), the final plan tree, and
+ * the structured tool events (quiz outcomes, mark_done). Job tool parts
+ * (research/demo/visual) are noise for extraction — their outcomes are
+ * summarized as one line each.
+ */
+export function transcriptView(record: ChatRecord): string {
+	const lines: string[] = [];
+	lines.push(`SESSION: ${record.title}`);
+
+	if (record.tree && record.tree.nodes.length > 0) {
+		const nodes = record.tree.nodes.map((n) => `${n.id} = ${n.title}`).join("; ");
+		lines.push(`FINAL PLAN TREE (node ids): ${nodes}`);
+		if (record.tree.edges.length > 0) {
+			const edges = record.tree.edges.map((e) => `${e.from} requires ${e.to}`).join("; ");
+			lines.push(`TREE EDGES: ${edges}`);
+		}
+	}
+
+	for (const message of record.messages as UIMessage[]) {
+		const who = message.role === "user" ? "LEARNER" : "TEACHER";
+		for (const part of message.parts) {
+			if (part.type === "text" && part.text.trim()) {
+				lines.push(`${who}: ${part.text.trim()}`);
+			} else if (part.type.startsWith("tool-")) {
+				const name = part.type.slice("tool-".length);
+				const p = part as unknown as { toolCallId: string; state: string; input: unknown; output: unknown };
+				if (name === "quiz" && p.state === "output-available") {
+					lines.push(quizLine(p));
+				} else if (name === "plan_mark_done" && p.state === "output-available") {
+					const input = p.input as { id?: unknown } | undefined;
+					lines.push(`TEACHER marked tree node done [${p.toolCallId}]: ${typeof input?.id === "string" ? input.id : "?"}`);
+				} else if (name.startsWith("plan_")) {
+					// plan ops are folded into the FINAL PLAN TREE line above
+				} else if (p.state === "output-available") {
+					lines.push(`(${name} tool result delivered)`);
+				}
+			}
+		}
+	}
+	return lines.join("\n");
+}
+
+// ─── the extractor ──────────────────────────────────────────────────────────
+
+const evidence = z
+	.array(z.string())
+	.min(1)
+	.describe("Verbatim quotes from the transcript that back this. The auditor will check every quote.");
+
+const findingsSchema = z.object({
+	propositions: z
+		.array(
+			z.object({
+				id: z.string().describe("Local id: \"p1\", \"p2\", … — referenced by edges and attribution."),
+				statement: z.string().describe("The proposition as one reasoning step — a node statement."),
+				aliases: z.array(z.string()).describe("Verbatim alternate phrasings from this session, if any."),
+				kind: z.enum(["truth", "misconception"]).describe("misconception = a wrong idea the learner holds or was warned about."),
+				treeNodeId: z.string().optional().describe("The plan-tree node id this came from, when it did."),
+				evidence,
+			}),
+		)
+		.describe("Every distinct proposition the session touched — from the plan tree AND from prose."),
+	edges: z
+		.array(
+			z.object({
+				from: z.string().describe("Proposition id that is the prerequisite / support."),
+				to: z.string().describe("Proposition id that depends on / is supported by `from`."),
+				type: z.enum(["requires", "supports"]),
+				evidence,
+			}),
+		)
+		.describe("Relations between propositions. `requires` = prerequisite; `supports` = helpful but not prerequisite. Include relations the plan tree does NOT show when the transcript articulates them."),
+	attribution: z
+		.object({
+			quiz: z
+				.record(z.string(), z.string().nullable())
+				.describe("toolCallId of every UNTAGGED quiz line → the proposition id it was really testing. Probe questions asked BEFORE the tree existed are attributed RETROACTIVELY — the whole transcript is visible; map them to the proposition they were probing (often articulated only later). null when genuinely unattributable. Tagged quizzes (with a tree nodeId in the transcript view) resolve automatically — do not include them."),
+		})
+		.describe("Probe attribution — your only event judgment."),
+});
+
+const EXTRACTOR_INSTRUCTIONS = `You are the extraction specialist in a learning application. You read ONE session
+transcript and report what was learned — as structured findings for the learner's global knowledge graph.
+
+The transcript view shows: the session's final plan tree (dependency map, with node ids), the conversation
+verbatim (LEARNER / TEACHER), and structured events (QUIZ lines with the learner's outcome, mark_done lines).
+
+Report, with verbatim evidence quotes for everything:
+
+1. PROPOSITIONS — every distinct reasoning-step-sized proposition the session touched:
+   - Plan-tree nodes become propositions (pass their treeNodeId) — statement from the node title, enriched to a
+     complete proposition if the prose elaborates it.
+   - Propositions that appeared only in prose (teacher explanations, learner questions) get fresh ids.
+   - Include MISCONCEPTIONS — wrong ideas the learner showed or the teacher warned against — with kind
+     "misconception".
+   - One reasoning step per proposition: never a topic blob, never a sub-step.
+   - aliases: verbatim alternate phrasings this session used for the same thing.
+
+2. EDGES between propositions:
+   - The tree's dependency edges are the backbone: from = prerequisite, to = dependent, type "requires".
+   - Also include relations articulated only in prose. Use "supports" when the transcript presents something as
+     helpful but NOT a prerequisite.
+
+3. ATTRIBUTION — UNTAGGED quizzes only:
+   - QUIZ lines whose question carries a tree nodeId resolve automatically; do not attribute those.
+   - The others are PROBE-phase questions (asked before the tree was initialized) that test prerequisite
+     understanding. Attribute them RETROACTIVELY to the proposition they were really probing — even if that
+     proposition is only articulated later in the session. The QUIZ question text is your guide. If you truly
+     cannot tell, use null.
+
+Evidence discipline: every quote must appear VERBATIM in the transcript view (a QUIZ line counts as the
+transcript). The auditor rejects anything unbacked. When in doubt, report; the auditor decides.
+
+Your entire deliverable is ONE submit_findings call.`;
+
+function createExtractor(model: LanguageModel) {
+	return new ToolLoopAgent({
+		model,
+		instructions: EXTRACTOR_INSTRUCTIONS,
+		tools: {
+			submit_findings: tool({
+				description: "Submit the session findings.",
+				inputSchema: findingsSchema,
+				execute: async (findings) => findings,
+			}),
+		},
+		stopWhen: stepCountIs(4),
+	});
+}
+
+// ─── the auditor ───────────────────────────────────────────────────────────
+
+const auditSchema = findingsSchema.extend({
+	dropped: z
+		.array(
+			z.object({
+				ref: z.string().describe("Proposition id, \"from→to\" for an edge, or attribution toolCallId."),
+				reason: z.string(),
+			}),
+		)
+		.describe("Everything you dropped, with the reason — the audit trail."),
+});
+
+const AUDITOR_INSTRUCTIONS = `You are the audit stage of a learning application's extraction pipeline. You receive
+ONE session transcript and a set of proposed findings (propositions, edges, event attribution). Your job is the
+faithfulness net: verify every candidate against the transcript, keep only what is backed, and return the
+verified subset.
+
+Verify:
+- PROPOSITIONS: every evidence quote must appear (near-)verbatim in the transcript view, and the statement must
+  be a faithful reading of it. Drop invented or bloated statements. Conservative: when two proposed propositions
+  are really one, keep the better-stated one and drop the other.
+- EDGES: the evidence must actually articulate the relation, in the direction claimed (from = prerequisite).
+  Drop direction-swapped or speculative edges.
+- ATTRIBUTION: only UNTAGGED quizzes carry attribution — the QUIZ line's question must genuinely test the
+  attributed proposition. Drop wrong attributions to null, or re-point them to the proposition the question
+  actually tests.
+
+Also drop propositions not referenced by anything else ONLY if they have no other backing — an unreferenced but
+well-evidenced proposition is still knowledge.
+
+Never invent new candidates. Your entire deliverable is ONE submit_review call containing the verified findings
+(the same shapes you received) plus the dropped list.`;
+
+function createAuditor(model: LanguageModel) {
+	return new ToolLoopAgent({
+		model,
+		instructions: AUDITOR_INSTRUCTIONS,
+		tools: {
+			submit_review: tool({
+				description: "Submit the verified findings subset + the dropped list.",
+				inputSchema: auditSchema,
+				execute: async (review) => review,
+			}),
+		},
+		stopWhen: stepCountIs(4),
+	});
+}
+
+// ─── the pipeline ───────────────────────────────────────────────────────────
+
+export interface ExtractionModels {
+	/** The extractor's model (tests inject a MockLanguageModelV4). */
+	extractor?: LanguageModel;
+	/** The auditor's model. */
+	auditor?: LanguageModel;
+}
+
+export interface ExtractionPipelineDeps extends GraphDeps, ExtractionModels {}
+
+export interface ExtractionPipeline {
+	/** Extract + audit + merge ONE chat. Single-flight per chat; resolves to
+	 *  the merger's append stats (or throws — no checkpoint on failure). */
+	runExtraction(chatId: string): Promise<EmissionStats>;
+}
+
+export function createExtractionPipeline(deps: ExtractionPipelineDeps): ExtractionPipeline {
+	const { chatStore, graphStore } = deps;
+	const extractorAgent = createExtractor(deps.extractor ?? openrouter(DEFAULT_EXTRACTION_MODEL));
+	const auditorAgent = createAuditor(deps.auditor ?? openrouter(DEFAULT_EXTRACTION_MODEL));
+	const inFlight = new Set<string>();
+
+	async function extract(chatId: string): Promise<SessionFindings> {
+		const record = chatStore.getChat(chatId);
+		if (!record) throw new Error(`Chat not found: ${chatId}`);
+		const view = transcriptView(record);
+
+		const extraction = await extractorAgent.generate({ prompt: `Transcript:\n\n${view}` });
+		for (const part of extraction.content) {
+			if (part.type === "tool-result" && part.toolName === "submit_findings") {
+				return part.output as SessionFindings;
+			}
+		}
+		throw new Error("the extractor never submitted findings");
+	}
+
+	async function audit(chatId: string, findings: SessionFindings): Promise<SessionFindings> {
+		const record = chatStore.getChat(chatId);
+		if (!record) throw new Error(`Chat not found: ${chatId}`);
+		const view = transcriptView(record);
+
+		const review = await auditorAgent.generate({
+			prompt: `Transcript:\n\n${view}\n\nProposed findings:\n\n${JSON.stringify(findings, null, 2)}`,
+		});
+		let auditResult: AuditResult | null = null;
+		for (const part of review.content) {
+			if (part.type === "tool-result" && part.toolName === "submit_review") {
+				auditResult = part.output as AuditResult;
+			}
+		}
+		if (!auditResult) throw new Error("the auditor never submitted a review");
+		if (auditResult.dropped.length > 0) {
+			console.info(
+				`[extraction ${chatId}] audit dropped ${auditResult.dropped.length}: ` +
+					auditResult.dropped.map((d) => `${d.ref} (${d.reason})`).join("; "),
+			);
+		}
+		const { dropped: _dropped, ...verified } = auditResult;
+		return verified;
+	}
+
+	async function runExtraction(chatId: string): Promise<EmissionStats> {
+		if (inFlight.has(chatId)) return { nodes: 0, edges: 0, events: 0 };
+		inFlight.add(chatId);
+		try {
+			const findings = await extract(chatId);
+			const verified = await audit(chatId, findings);
+			// The merger re-reads the chat record itself — events are rebuilt
+			// from the transcript, never from stale copies.
+			const stats = mergeFindings(chatId, verified, { chatStore, graphStore });
+			console.info(`[extraction ${chatId}] merged: ${stats.nodes} nodes, ${stats.edges} edges, ${stats.events} events`);
+			return stats;
+		} finally {
+			inFlight.delete(chatId);
+		}
+	}
+
+	return { runExtraction };
+}
@@ -83,12 +83,14 @@ export interface GraphEdge {
 	provenance: string[];
 }

-/** mappings.jsonl line — the naive-union identity map: a chat's tree node →
- * the graph node it folded into (the merge audit trail; v1 never merges two
- * sessions' tree nodes — W8 exact-title match aside). */
+/** mappings.jsonl line — the session→graph identity map: the extraction
+ *  pipeline's proposition (identified by its source id) → the graph node it
+ *  merged into. The merge audit trail: one line per (chat, graph node). */
 export interface NodeMapping {
 	chatId: string;
-	treeNodeId: string;
+	/** The writer's local id for the proposition — a plan-tree node id when
+	 *  tree-derived, else the extractor's proposition id ("p1", …). */
+	sourceId: string;
 	graphNodeId: string;
 }

@@ -125,18 +127,85 @@ export interface LedgerEvent {
 }

 /**
- * Store format (W4 implements src/store/graph.ts): JSONL ledgers at
- * config.LEDGER_FILES (nodes/edges/events/mappings/sessions). The derived
- * graph is rebuilt by reading the ledgers — store events, derive state.
- * sessions.jsonl tracks which chats have been folded into the graph
- * (emission checkpoint — never fold the same turn twice).
+ * Store format (src/store/graph.ts): JSONL ledgers at config.LEDGER_FILES
+ * (nodes/edges/events/mappings/sessions). The derived graph is rebuilt by
+ * reading the ledgers — store events, derive state. sessions.jsonl is the
+ * extraction checkpoint: which chats the extraction pipeline has processed,
+ * and through which updatedAt (never extract the same turns twice).
 *
- * v1 emission (naive union, deterministic, no LLM): a chat's plan tree folds
- * in — node title → statement (aliases empty), tree edges → requires edges,
- * mark_done → "taught" events; quiz parts with input.nodeId → probe / check
- * events (+ note); quiz parts without nodeId → events with nodeId null.
+ * The EXTRACTION PIPELINE is the sole graph writer (extractor → auditor →
+ * merger): the extractor proposes propositions/edges/attribution with
+ * verbatim evidence, the auditor drops anything unbacked, and the merger
+ * (core/graph.ts, deterministic) resolves identity against the existing
+ * ledgers and appends. See SessionFindings below.
 */

+// ─── Extraction findings (extractor output → audit → merger input) ───────────
+
+/** One proposed proposition — a node candidate. */
+export interface ExtractedProposition {
+	/** Extractor-local id ("p1", …) referenced by edges + attribution. */
+	id: string;
+	/** Canonical one-reasoning-step statement. */
+	statement: string;
+	/** Verbatim alternate phrasings from the session (merge aliases). */
+	aliases: string[];
+	kind: "truth" | "misconception";
+	/** The plan-tree node this came from, when it did (mapping provenance). */
+	treeNodeId?: string;
+	/** Verbatim transcript quotes backing the proposition. */
+	evidence: string[];
+}
+
+/** One proposed relation between two propositions. */
+export interface ExtractedEdge {
+	/** Proposition id that is the prerequisite / support. */
+	from: string;
+	/** Proposition id that depends on / is supported by `from`. */
+	to: string;
+	type: EdgeType;
+	/** Verbatim transcript quotes backing the relation. */
+	evidence: string[];
+}
+
+/** The extractor's event attribution — its ONLY say about events. The kinds,
+ * timestamps, idempotency refs, notes and question texts are all rebuilt
+ * deterministically by the merger from the transcript's quiz / mark_done
+ * tool parts; the LLM supplies exactly the judgment call: WHICH proposition
+ * each one is about (vision open question: probes attribute retroactively,
+ * against the whole transcript — a pre-tree probe can map to a proposition
+ * the tree only articulates later). */
+/** The extractor's event attribution — its ONLY say about events. Event
+ *  kinds, timestamps, idempotency refs, notes and question texts are all
+ *  rebuilt deterministically by the merger from the transcript's tool
+ *  parts, and TAGGED events (quiz nodeId, mark_done tree node) resolve
+ *  deterministically through the proposition mapping. The LLM supplies
+ *  exactly the one judgment call that needs it — the vision's open
+ *  probe-attribution question: WHICH proposition did an UNTAGGED probe
+ *  test? Answered retroactively against the whole transcript — a pre-tree
+ *  probe can map to a proposition the tree only articulates later. */
+export interface ExtractionAttribution {
+	/** UNTAGGED quiz toolCallId → proposition id; missing / null =
+	 *  unattributed (the event lands with nodeId null + the question text,
+	 *  exactly like unattributable probes always did). */
+	quiz: Record<string, string | null>;
+}
+
+/** The extractor's — and, filtered, the auditor's — output shape. */
+export interface SessionFindings {
+	propositions: ExtractedProposition[];
+	edges: ExtractedEdge[];
+	attribution: ExtractionAttribution;
+}
+
+/** The auditor's verdict: the verified subset, plus the drops (for the log —
+ *  the audit trail of what was rejected and why). */
+export interface AuditResult extends SessionFindings {
+	/** Dropped candidates: proposition id, edge "from→to", or attribution
+	 *  toolCallId — with the reason. */
+	dropped: { ref: string; reason: string }[];
+}
+
 /** Per-node state class for the home view (vision §1 "Home page") — derived,
 * never stored: taught if ANY taught event; else the LATEST quiz event decides
 * (known / missed / dont-know); a node with no events is a candidate. */
@@ -1,24 +1,29 @@
 /**
- * Knowledge-graph emission + derivation (W4; vision §1 "One substrate",
- * contracts.ts "v1 emission").
+ * The knowledge graph: merger (the deterministic final stage of the
+ * extraction pipeline — vision §1 construction) + derivation (reads).
 *
- * emitChat(chatId) folds ONE chat into the global graph — naive union,
- * deterministic, no LLM:
- *   plan-tree nodes   → GraphNodes (statement = title, aliases empty;
- *                       W8: a title exactly matching an existing statement
- *                       or alias — case-insensitive, trimmed — maps onto the
- *                       existing node instead of appending a duplicate)
- *   plan-tree edges   → `requires` GraphEdges (from = prerequisite)
- *   answered quizzes  → ledger events (probe-* before the first plan_init
- *                       output, check / dont-know after; ref = toolCallId)
- *   plan_mark_done    → "taught" events (ref = taught:<treeNodeId>)
+ * The extraction pipeline is the SOLE graph writer:
 *
- * Idempotent + incremental: mappings dedupe nodes, (from,to,type) dedupes
- * edge assertions, chat-scoped refs dedupe events, and sessions.jsonl
- * checkpoints which chats have been folded. Re-emitting a chat appends
- * nothing. Every function takes the storage ports (core/ports.ts) as
- * explicit parameters — pure core, no adapter imports; the composition
- * root (src/server/stores.ts) wires the fs adapters in.
+ *   extractor (LLM)  → SessionFindings: propositions, edges, and event
+ *                      attribution, each with verbatim evidence
+ *   auditor (LLM)    → the verified subset (unbacked candidates dropped)
+ *   merger (HERE)    → mergeFindings(): deterministic ledger appends
+ *
+ * The merger is deliberately the ONLY stage that touches the ledgers, and it
+ * is pure code: the LLM supplies exactly the judgment calls (what was
+ * taught, what depends on what, which proposition a quiz tested), while
+ * event kinds/timestamps/refs — anything the model could silently
+ * mis-copy — are rebuilt deterministically from the transcript's tool
+ * parts. Idempotent + incremental: statement/alias matching dedupes nodes,
+ * (from,to,type) dedupes edge assertions, chat-scoped refs dedupe events,
+ * one mapping line per (chat, graph node), and sessions.jsonl checkpoints
+ * which chats have been extracted (never the same turns twice).
+ *
+ * Derivation below (getGraph / deriveNodeStates / graphSnapshot) is reads
+ * only — everything is recomputed from the ledgers. Every function takes
+ * the storage ports (core/ports.ts) as explicit parameters — pure core, no
+ * adapter imports; the composition root (src/server/stores.ts) wires the fs
+ * adapters in.
 */

 import { randomBytes } from "node:crypto";
@@ -30,8 +35,8 @@ import type {
 	LedgerEvent,
 	LedgerEventKind,
 	NodeStateClass,
-	PlanTree,
 	PlanToolResult,
+	SessionFindings,
 } from "./contracts.ts";
 import type { QuestionAnswerData } from "./question-core.ts";
 import type { ChatStore, GraphStore } from "./ports.ts";
@@ -47,7 +52,7 @@ function newGraphNodeId(): string {
 	return `n_${randomBytes(6).toString("hex")}`;
 }

-// ─── transcript scanning ─────────────────────────────────────────────────────
+// ─── transcript scanning (shared by the merger + trigger) ─────────────────────

 /** A flattened, order-preserving view of every tool part in a transcript. */
 interface ToolPartView {
@@ -84,23 +89,7 @@ function planResult(part: ToolPartView): PlanToolResult | null {
 	return out as PlanToolResult;
 }

-/**
- * The FINAL live tree, replayed from plan tool outputs in message order.
- * Fallback for records whose stored `tree` is missing (it is the same state
- * saveTree persists, so the two normally agree); also proves the probe/check
- * boundary scan below sees the same transcript the teacher wrote.
- */
-function replayFinalTree(messages: UIMessage[]): PlanTree {
-	let tree: PlanTree = { nodes: [], edges: [], currentNodeId: null, doneNodeIds: [], version: 0 };
-	for (const part of toolParts(messages)) {
-		if (!part.name.startsWith("plan_")) continue;
-		const result = planResult(part);
-		if (result && result.ok && result.tree) tree = result.tree;
-	}
-	return tree;
-}
-
-// ─── emission ────────────────────────────────────────────────────────────────
+// ─── the merger (extraction pipeline stage 3 — deterministic) ────────────────

 export interface EmissionStats {
 	/** New graph nodes appended (and mappings with them). */
@@ -112,79 +101,86 @@ export interface EmissionStats {
 }

 /**
- * Fold one chat into the graph ledgers. Idempotent — safe to call after every
- * turn (chat-handler onFinish) and again on catch-up (GET /api/graph). Sync
- * and fast: a handful of small-file reads plus one append per new record.
+ * Merge one chat's audited findings into the graph ledgers — the pipeline's
+ * apply stage. Deterministic + idempotent: re-merging the same findings (or
+ * a later re-extraction of the same chat) appends nothing but the new.
 *
- * The storage ports are explicit (tests pass in-memory ledgers; production
- * passes the fs adapters from src/server/stores.ts).
+ * Identity resolution (v1: conservative): a proposition whose statement or
+ * alias EXACTLY matches (case-insensitive, trimmed) an existing node's
+ * statement or alias merges onto it; everything else is a new node. One
+ * mapping line per (chat, graph node) — the merge audit trail. Fuzzy /
+ * embedding adjudication stays deferred (vision §1 consolidation).
+ *
+ * Events are rebuilt deterministically from the transcript's quiz and
+ * plan_mark_done parts (kinds, timestamps, notes, refs). TAGGED events
+ * (quiz nodeId, mark_done tree node) resolve through the proposition
+ * mapping — deterministic; the findings' attribution is consulted ONLY for
+ * untagged probe quizzes: the retroactive probe-attribution resolution (a
+ * pre-tree probe maps to a proposition the tree only articulates later).
+ * Unattributed quizzes land with nodeId null + the question text, exactly
+ * like the unattributable ones always have.
 */
-export function emitChat(chatId: string, deps: GraphDeps): EmissionStats {
-	const { graphStore: ledgers } = deps;
-	const record = deps.chatStore.getChat(chatId);
+export function mergeFindings(chatId: string, findings: SessionFindings, deps: GraphDeps): EmissionStats {
+	const { graphStore: ledgers, chatStore } = deps;
+	const record = chatStore.getChat(chatId);
 	if (!record) return { nodes: 0, edges: 0, events: 0 };
 	const stats: EmissionStats = { nodes: 0, edges: 0, events: 0 };

-	// 1. This chat's existing mappings — the identity map is per-chat (v1 is
-	// naive union: two sessions' tree nodes never merge into one graph node).
+	// 1. Identity resolution. mapping: proposition id (and tree node id)
+	//    → graph node id — from this chat's existing mapping lines, then
+	//    extended as findings resolve. chatNodeIds: graph nodes this chat has
+	//    already mapped onto — the (chat, graph node) mapping dedupe.
 	const mapping = new Map<string, string>();
+	const chatNodeIds = new Set<string>();
 	for (const m of ledgers.readMappings()) {
-		if (m.chatId === chatId) mapping.set(m.treeNodeId, m.graphNodeId);
+		if (m.chatId !== chatId || typeof m.sourceId !== "string") continue;
+		mapping.set(m.sourceId, m.graphNodeId);
+		chatNodeIds.add(m.graphNodeId);
 	}

-	// 2. The tree state to fold. record.tree is the truth (it is the exact
-	// state every plan tool persisted); the transcript replay covers records
-	// that predate tree persistence. Nodes REMOVED from the tree are simply
-	// absent here — if they were emitted by an earlier fold, their mappings
-	// and ledger lines stay (the ledgers are append-only evidence).
-	const tree =
-		record.tree && record.tree.nodes.length > 0 ? record.tree : replayFinalTree(record.messages);
-
-	// 3. Tree nodes without a mapping → GraphNode + mapping. W8 merge
-	// improvement: a title that EXACTLY matches (case-insensitive, trimmed) an
-	// existing node's statement or alias maps onto that node instead of
-	// appending a duplicate — the mapping line is still appended so provenance
-	// stays honest. Fuzzy/embedding matching stays deferred (vision §1
-	// consolidation pipeline).
+	// 2. Propositions → graph nodes. Exact statement/alias match against the
+	//    whole ledger first (cross-session reuse), then against propositions
+	//    created earlier in THIS merge (intra-findings dedupe).
 	const norm = (s: string): string => s.trim().toLowerCase();
 	const existingByTitle = new Map<string, string>();
 	for (const node of ledgers.readNodes()) {
-		const statementKey = norm(node.statement);
-		if (statementKey && !existingByTitle.has(statementKey)) existingByTitle.set(statementKey, node.id);
-		for (const alias of node.aliases) {
-			const aliasKey = norm(alias);
-			if (aliasKey && !existingByTitle.has(aliasKey)) existingByTitle.set(aliasKey, node.id);
+		for (const title of [node.statement, ...node.aliases]) {
+			const key = norm(title);
+			if (key && !existingByTitle.has(key)) existingByTitle.set(key, node.id);
 		}
 	}
 	const now = new Date().toISOString();
-	for (const node of tree.nodes) {
-		if (mapping.has(node.id)) continue;
-		const titleKey = norm(node.title);
-		const existingId = titleKey ? existingByTitle.get(titleKey) : undefined;
-		if (existingId !== undefined) {
-			// Exact match: reuse the node, record the mapping, append nothing.
-			ledgers.appendMapping({ chatId, treeNodeId: node.id, graphNodeId: existingId });
-			mapping.set(node.id, existingId);
-			continue;
+	for (const p of findings.propositions) {
+		if (mapping.has(p.id)) continue;
+		const keys = [p.statement, ...p.aliases].map(norm).filter((k) => k.length > 0);
+		const matchId = keys.map((k) => existingByTitle.get(k)).find((id) => id !== undefined);
+		let graphNodeId: string;
+		if (matchId !== undefined) {
+			graphNodeId = matchId; // reuse — conservative merge, duplicates over wrong merges
+		} else {
+			const graphNode: GraphNode = {
+				id: newGraphNodeId(),
+				statement: p.statement,
+				aliases: [...p.aliases],
+				kind: p.kind,
+				createdAt: now,
+			};
+			ledgers.appendNode(graphNode);
+			stats.nodes++;
+			graphNodeId = graphNode.id;
 		}
-		const graphNode: GraphNode = {
-			id: newGraphNodeId(),
-			// statement is the title; the summary stays out (contracts).
-			statement: node.title,
-			aliases: [],
-			kind: "truth",
-			createdAt: now,
-		};
-		ledgers.appendNode(graphNode);
-		ledgers.appendMapping({ chatId, treeNodeId: node.id, graphNodeId: graphNode.id });
-		existingByTitle.set(titleKey, graphNode.id);
-		mapping.set(node.id, graphNode.id);
-		stats.nodes++;
+		for (const key of keys) if (!existingByTitle.has(key)) existingByTitle.set(key, graphNodeId);
+		if (!chatNodeIds.has(graphNodeId)) {
+			ledgers.appendMapping({ chatId, sourceId: p.treeNodeId ?? p.id, graphNodeId });
+			chatNodeIds.add(graphNodeId);
+		}
+		mapping.set(p.id, graphNodeId);
+		if (p.treeNodeId !== undefined) mapping.set(p.treeNodeId, graphNodeId);
 	}

-	// 4. Tree edges → `requires` edges. Dedupe by (from,to,type) across the
-	// whole ledger: an existing assertion just gains this chatId as
-	// provenance (appended as a new line; reads merge).
+	// 3. Edges. Dedupe by (from,to,type) across the whole ledger: an existing
+	//    assertion just gains this chatId as provenance (appended as a new
+	//    line; reads merge).
 	const asserted = new Map<string, Set<string>>();
 	for (const line of ledgers.readEdges()) {
 		const key = `${line.from}\u0000${line.to}\u0000${line.type}`;
@@ -195,23 +191,25 @@ export function emitChat(chatId: string, deps: GraphDeps): EmissionStats {
 		}
 		for (const c of line.provenance) provenance.add(c);
 	}
-	for (const edge of tree.edges) {
+	for (const edge of findings.edges) {
 		const from = mapping.get(edge.from);
 		const to = mapping.get(edge.to);
 		if (!from || !to || from === to) continue; // unmapped endpoint — skip
-		const key = `${from}\u0000${to}\u0000requires`;
+		const key = `${from}\u0000${to}\u0000${edge.type}`;
 		const provenance = asserted.get(key);
 		if (provenance?.has(chatId)) continue; // already asserted by this chat
-		ledgers.appendEdge({ from, to, type: "requires", provenance: [chatId] });
+		ledgers.appendEdge({ from, to, type: edge.type, provenance: [chatId] });
 		if (provenance) provenance.add(chatId);
 		else asserted.set(key, new Set([chatId]));
 		stats.edges++;
 	}

-	// 5. Events, replayed in transcript order so the probe/check boundary is
-	// exact: quizzes BEFORE the first plan_init output are probes. Refs are
-	// chat-scoped (a quiz toolCallId or taught:<treeNodeId>) — a ref already
-	// in the ledger means this turn was folded before.
+	// 4. Events, rebuilt deterministically in transcript order so the
+	//    probe/check boundary is exact: quizzes BEFORE the first plan_init
+	//    output are probes. Refs are chat-scoped (a quiz toolCallId or
+	//    taught:<treeNodeId>) — a ref already in the ledger means this turn
+	//    was extracted before. The findings' attribution supplies the
+	//    nodeId, nothing else.
 	const seenRefs = new Set<string>();
 	for (const event of ledgers.readEvents()) {
 		if (event.chatId === chatId && event.ref) seenRefs.add(event.ref);
@@ -223,7 +221,7 @@ export function emitChat(chatId: string, deps: GraphDeps): EmissionStats {
 		if (part.name === "plan_init" && part.state === "output-available") planSeen = true;

 		if (part.name === "quiz" && part.state === "output-available") {
-			const event = quizEvent(part, chatId, mapping, planSeen, now);
+			const event = quizEvent(part, chatId, mapping, findings, planSeen, now);
 			if (event?.ref !== undefined && !seenRefs.has(event.ref)) {
 				ledgers.appendEvent(event);
 				seenRefs.add(event.ref);
@@ -237,6 +235,7 @@ export function emitChat(chatId: string, deps: GraphDeps): EmissionStats {
 			if (result?.ok && typeof input?.id === "string") {
 				const ref = `taught:${input.id}`;
 				if (!seenRefs.has(ref)) {
+					// mark_done is tagged with the tree node id — deterministic.
 					ledgers.appendEvent({
 						nodeId: mapping.get(input.id) ?? null,
 						kind: "taught",
@@ -251,16 +250,18 @@ export function emitChat(chatId: string, deps: GraphDeps): EmissionStats {
 		}
 	}

-	// 6. Checkpoint: this chat is folded through its current updatedAt.
+	// 5. Checkpoint: this chat is extracted through its current updatedAt.
 	ledgers.appendCheckpoint({ chatId, foldedAt: record.updatedAt });
 	return stats;
 }

-/** Build the ledger event for one answered quiz tool part (or null to skip). */
+/** Build the ledger event for one answered quiz tool part (or null to skip).
+ *  Deterministic in everything except the attribution-supplied nodeId. */
 function quizEvent(
 	part: ToolPartView,
 	chatId: string,
 	mapping: Map<string, string>,
+	findings: SessionFindings,
 	planSeen: boolean,
 	now: string,
 ): LedgerEvent | null {
@@ -275,9 +276,13 @@ function quizEvent(
 	else if (answer.correct === true) kind = planSeen ? "check-correct" : "probe-correct";
 	else kind = planSeen ? "check-incorrect" : "probe-miss";

+	// Tagged (input.nodeId) resolves deterministically through the mapping;
+	// attribution is the LLM judgment call for UNTAGGED probes only — the
+	// vision's open probe-attribution question, resolved retroactively.
 	const input = part.input as { nodeId?: unknown } | undefined;
-	const treeNodeId = typeof input?.nodeId === "string" ? input.nodeId : null;
-	const nodeId = treeNodeId !== null ? (mapping.get(treeNodeId) ?? null) : null;
+	const tagged = typeof input?.nodeId === "string" ? mapping.get(input.nodeId) : undefined;
+	const attributedId = findings.attribution.quiz[part.toolCallId];
+	const nodeId = tagged ?? (attributedId ? (mapping.get(attributedId) ?? null) : null);

 	const event: LedgerEvent = {
 		nodeId,
@@ -287,11 +292,39 @@ function quizEvent(
 		ref: part.toolCallId,
 	};
 	if (typeof answer.note === "string" && answer.note.length > 0) event.note = answer.note;
-	// The question text is the attribution input for untagged/unmapped probes.
+	// The question text is the attribution input for unattributed probes.
 	if (nodeId === null && typeof answer.questionText === "string") event.question = answer.questionText;
 	return event;
 }

+// ─── the trigger helper (v1: every N quiz answers — swappable seam) ──────────
+
+/**
+ * How many ANSWERED quiz parts in this chat are not yet in the ledger — i.e.
+ * new evidence since the last extraction. The v1 trigger (turn-service)
+ * fires the pipeline when this crosses EXTRACTION.everyNQuizAnswers. Every
+ * answered quiz always becomes an event (attribution only decides nodeId),
+ * so ref-not-in-ledger is a correct "not yet processed" marker.
+ */
+export function pendingQuizAnswers(chatId: string, deps: GraphDeps): number {
+	const record = deps.chatStore.getChat(chatId);
+	if (!record) return 0;
+	const seenRefs = new Set<string>();
+	for (const event of deps.graphStore.readEvents()) {
+		if (event.chatId === chatId && event.ref) seenRefs.add(event.ref);
+	}
+	let count = 0;
+	for (const part of toolParts(record.messages)) {
+		if (part.name !== "quiz" || part.state !== "output-available") continue;
+		const out = part.output;
+		if (typeof out !== "object" || out === null) continue;
+		const answer = out as Partial<QuestionAnswerData>;
+		if (answer.kind !== "quiz" || answer.dismissed === true) continue; // dismissed = not answered
+		if (!seenRefs.has(part.toolCallId)) count++;
+	}
+	return count;
+}
+
 // ─── derivation (reads only — everything below is recomputable) ─────────────

 export interface DerivedGraph {
@@ -417,33 +450,3 @@ export function graphSnapshot(deps: GraphDeps): GraphSnapshot {

 	return { nodes, edges, nodeStates, sessions, nodeSessions };
 }
-
-// ─── catch-up ────────────────────────────────────────────────────────────────
-
-/**
- * Fold every chat whose updatedAt is newer than its checkpoint (partial
- * sessions deposit value — vision §1 "frequent updates"). Called at the top
- * of GET /api/graph; the chat-handler onFinish hook covers the live path.
- */
-export function foldStaleChats(deps: GraphDeps): number {
-	const checkpoints = new Map<string, string>();
-	for (const checkpoint of deps.graphStore.readCheckpoints()) {
-		const previous = checkpoints.get(checkpoint.chatId);
-		if (previous === undefined || checkpoint.foldedAt > previous) {
-			checkpoints.set(checkpoint.chatId, checkpoint.foldedAt);
-		}
-	}
-
-	let folded = 0;
-	for (const chat of deps.chatStore.listChats()) {
-		const foldedAt = checkpoints.get(chat.id);
-		if (foldedAt !== undefined && chat.updatedAt <= foldedAt) continue;
-		try {
-			emitChat(chat.id, deps);
-			folded++;
-		} catch (err) {
-			console.error(`[graph] fold failed for chat ${chat.id}:`, err);
-		}
-	}
-	return folded;
-}
@@ -1,9 +1,10 @@
 import { createAgentUIStreamResponse, type UIMessage } from "ai";
 import { createTeacherAgent, type TeacherAgent } from "../agent/teacher-agent.ts";
 import { createJobRunner } from "../agent/subagents/runner.ts";
+import { createExtractionPipeline } from "../agent/subagents/extraction.ts";
 import { createTeacherTools } from "../agent/tools.ts";
 import { stores } from "./stores.ts";
-import { buildTeacherContext, foldTurnIntoGraph, persistTurn, spawnTurnJobs } from "./turn-service.ts";
+import { buildTeacherContext, persistTurn, spawnTurnJobs, triggerExtraction } from "./turn-service.ts";

 /**
 * One chat turn: run the teacher agent over the client's message list and
@@ -18,6 +19,7 @@ import { buildTeacherContext, foldTurnIntoGraph, persistTurn, spawnTurnJobs } fr
 */

 const jobRunner = createJobRunner(stores);
+const extraction = createExtractionPipeline(stores);

 export async function handleChatRequest(req: Request, agent?: TeacherAgent): Promise<Response> {
 	let body: { chatId?: unknown; messages?: unknown };
@@ -46,7 +48,7 @@ export async function handleChatRequest(req: Request, agent?: TeacherAgent): Pro
 		onFinish: ({ messages, isAborted }) => {
 			persistTurn(stores.chatStore, chatId, messages as UIMessage[], isAborted);
 			spawnTurnJobs(jobRunner, chatId, messages as UIMessage[]);
-			foldTurnIntoGraph(stores, chatId);
+			triggerExtraction(stores, extraction, chatId);
 		},
 	});
 }
@@ -1,15 +1,13 @@
 import type { GraphSnapshot } from "../core/contracts.ts";
-import { foldStaleChats, graphSnapshot } from "../core/graph.ts";
+import { graphSnapshot } from "../core/graph.ts";
 import { stores } from "./stores.ts";

 /**
- * Server-side graph entry for the home view (GET /api/graph): fold any chat
- * whose updatedAt outstrips its emission checkpoint first (partial sessions
- * deposit value), then return the snapshot. Today the fold is the naive-union
- * emission; the extraction-pipeline stage replaces the writer behind this
- * same read surface.
+ * Server-side graph entry for the home view (GET /api/graph). A PURE READ:
+ * the extraction pipeline (the sole graph writer) is triggered by quiz
+ * answers in sessions (turn-service), never from here — the home view
+ * reflects whatever the pipeline has merged so far.
 */
 export function graphView(): GraphSnapshot {
-	foldStaleChats(stores);
 	return graphSnapshot(stores);
 }
@@ -7,10 +7,11 @@
 *   buildTeacherContext — graph → instructions block (the W8 read channel)
 *   persistTurn        — the canonical state write (client + agent messages)
 *   spawnTurnJobs      — the async sub-agent protocol kick-off
- *   foldTurnIntoGraph  — session → graph consolidation (the write channel;
- *                        the extraction-pipeline stage swaps this body for
- *                        the extractor/auditor/merger trigger behind the
- *                        same seam)
+ *   triggerExtraction  — the graph write channel: fires the extraction
+ *                        pipeline (the sole graph writer) once a chat
+ *                        crosses the v1 quiz-answer threshold. The trigger
+ *                        mechanism is a v1 placeholder — this seam is the
+ *                        ONLY thing to swap when a better one lands.
 *
 * Logs are the minimal-observability baseline: one line per failure, tagged
 * [chat <id>] — structured logging lands behind these seams later.
@@ -18,9 +19,11 @@

 import type { UIMessage } from "ai";
 import { buildGraphContext } from "../core/graph-context.ts";
-import { emitChat, getGraph, type GraphDeps } from "../core/graph.ts";
+import { getGraph, pendingQuizAnswers } from "../core/graph.ts";
 import type { ChatStore, GraphStore } from "../core/ports.ts";
+import { EXTRACTION } from "../../config.ts";
 import type { JobRunner } from "../agent/subagents/runner.ts";
+import type { ExtractionPipeline } from "../agent/subagents/extraction.ts";

 /**
 * W8 (vision §1 "the graph as session context"): the learner's knowledge
@@ -66,16 +69,30 @@ export function spawnTurnJobs(jobRunner: JobRunner, chatId: string, messages: UI
 }

 /**
- * W4: fold this chat into the knowledge-graph ledgers (tree nodes / edges,
- * quiz outcomes, taught events) — synchronous but tiny (append-only JSONL),
- * and errors never touch the response. Skipped under tests: scripted chats
- * are fixtures, not knowledge.
+ * The graph write channel (vision §1 construction): the extraction pipeline
+ * (extractor → auditor → merger — the sole graph writer) runs ASYNC once a
+ * chat accumulates EXTRACTION.everyNQuizAnswers new answered quizzes since
+ * its last checkpoint. Fire-and-forget — never blocks the turn; failures
+ * are logged by the pipeline and leave the checkpoint untouched, so the
+ * next trigger re-runs the whole thing.
+ *
+ * v1 placeholder trigger, deliberately isolated in this one seam: swap
+ * here when a better mechanism lands — the pipeline is trigger-agnostic.
+ * Skipped under tests: scripted chats are fixtures, not knowledge (the
+ * pipeline itself is tested directly with mock models).
 */
-export function foldTurnIntoGraph(deps: GraphDeps, chatId: string): void {
+export function triggerExtraction(
+	deps: { chatStore: ChatStore; graphStore: GraphStore },
+	extraction: ExtractionPipeline,
+	chatId: string,
+): void {
 	if (process.env.NODE_ENV === "test") return;
 	try {
-		emitChat(chatId, deps);
+		if (pendingQuizAnswers(chatId, deps) < EXTRACTION.everyNQuizAnswers) return;
+		void extraction.runExtraction(chatId).catch((err) => {
+			console.error(`[chat ${chatId}] extraction pipeline failed:`, err);
+		});
 	} catch (err) {
-		console.error(`[chat ${chatId}] graph emission failed:`, err);
+		console.error(`[chat ${chatId}] extraction trigger failed:`, err);
 	}
 }
@@ -6,8 +6,8 @@
 *                                      session asserting the same (from,to,type)
 *                                      appends another line; reads merge)
 *   events.jsonl    LedgerEvent      — quiz outcomes + taught events
- *   mappings.jsonl  NodeMapping      — the naive-union identity map:
- *                                      (chatId, treeNodeId) → graphNodeId
+ *   mappings.jsonl  NodeMapping      — the session→graph identity map:
+ *                                      (chatId, sourceId) → graphNodeId
 *   sessions.jsonl  SessionCheckpoint— emission checkpoints (foldedAt)
 *
 * Implements core/ports.ts GraphStore. data/graph/* is exclusively this
@@ -105,7 +105,7 @@ const isLedgerEvent = (v: unknown): v is LedgerEvent =>
 	isString(v.at);

 const isNodeMapping = (v: unknown): v is NodeMapping =>
-	isRecord(v) && isString(v.chatId) && isString(v.treeNodeId) && isString(v.graphNodeId);
+	isRecord(v) && isString(v.chatId) && isString(v.sourceId) && isString(v.graphNodeId);

 const isCheckpoint = (v: unknown): v is SessionCheckpoint =>
 	isRecord(v) && isString(v.chatId) && isString(v.foldedAt);