extraction cleanup: events ledger dies, markdown source view, auditor-as-linker (2026-09-27 rulings)

The graph becomes raw reasoning chains only. Quiz outcomes and
understanding observations are no longer formalized; assessment
moves to the future practice module.

Events ledger removed end-to-end: LedgerEvent/LedgerEventKind/
UnderstandingLevel/ExtractionAttribution/AuditResult.dropped and
snapshot nodeObservations deleted from contracts; quizEvent builder
and the events step of mergeFindings gone; appendEvent/readEvents
dropped from the port and store; LEDGER_FILES loses events.jsonl
with a one-time purge of a stale file next to the nodes ledger;
graph-deletion and the delete dialog lose event purge + quiz-answer
impact counts; GraphSessionProposition becomes a pure birth list.

Whole-session markdown source view: new core/session-markdown.ts is
the shared formatting engine (id-anchored speaker blocks, quiz/
recall/question blocks, tool one-liners) — the extractor's prompt
view and the future user-facing export render from one engine.
Coverage fingerprints deliberately stay on the old line format
(content-hash semantics; re-formatting never re-opens covered
history). The whole session goes to the extractor — no phase cut.

Auditor refocused as the identity linker: never sees the source;
explores the graph via new auditor-specific tools in
agent/tools/graph.ts (find — glob over name/statement/aliases;
list_sessions; list_propositions with full statement text; no
timestamps) instead of an injected full-graph dump; equivalence
bias — paraphrases merge, only genuinely distinct propositions stay
apart; no drop channel, the extractor is fully trusted; may draw
bridging edges (typed follows/supports) between existing nodes and
the batch, resolved by the merger with chat-scoped message-less
provenance. Agentic loop, maxTurns 16.

UI: the node detail card gains an edges section above the sessions
list (follows-from / supported-by / what-follows / what-it-supports,
rendered only when present, click-to-select neighbors); the
understanding-note chips are gone.

Prober list_propositions reworded to what a session covered —
names only, no levels. docs/architecture.md + README updated to
the as-built pipeline. Gates: typecheck, 1059/1059, lint, build.
This commit is contained in:
amosblomqvist
2026-09-28 16:54:44 +03:00
parent cd09c12b56
commit 1b50013551
22 changed files with 909 additions and 1064 deletions
+1 -1
View File
@@ -9,7 +9,7 @@ import type { ThinkingLevel } from "@earendil-works/pi-agent-core";
import type { LhMessage } from "../native-messages.ts";
import type { Phase3 } from "../phases.ts";
import type { PlanNodeStatus, PlanTree } from "./plan.ts";
import type { EmissionStats, UnderstandingLevel } from "./graph.ts";
import type { EmissionStats } from "./graph.ts";
// ─── Chat records (the session store's on-disk format) ───────────────────────
+43 -98
View File
@@ -5,10 +5,11 @@
* 2026-09-02 redesign): every node is a TRUE proposition; edges are follows
* (derivable) and supports (grounds without entailing); provenance is
* message-granular; extraction is free-form and post-hoc, fully decoupled
* from the session's plan tree; the event ledger records quiz outcomes plus
* the extractor's per-proposition understanding observations. Anything not
* yet decided (the state model, strength derivation, understanding
* treatment, selection strategies) is deliberately NOT here — transitional
* from the session's plan tree. The graph is an extraction of the reasoning
* chains from all teaching sessions (and imported notes) — NOTHING else:
* no quiz outcomes, no understanding assessments (2026-09-27 cleanup —
* understanding becomes practice-module territory). Anything not
* yet decided (the state model, strength derivation, selection strategies) is deliberately NOT here — transitional
* stubs carry the renderer until those land.
*
* Part of the domain-split contracts (src/core/contracts/): each domain file
@@ -31,13 +32,13 @@ export interface EmissionStats {
nodes: number;
/** New edge assertions appended. */
edges: number;
/** Ledger events rebuilt (quiz outcomes + observations). */
events: number;
}
/** One provenance point — message-granular (DECIDED 2026-09-02): nodes and
* edges record every (chat, message) where they came up, deep-linkable to
* the exact transcript point. */
* the exact transcript point. Auditor-drawn bridging edges carry the chat
* but NO message anchor (nothing in the transcript asserts them — the
* auditor is the asserter); messageId is "" there. */
export interface ProvenancePoint {
/** The chat this point anchors to — or a notes source id ("src_…"):
* imports carry no message anchors (provenance is the file itself,
@@ -50,11 +51,10 @@ export interface ProvenancePoint {
* (DECIDED 2026-09-13): personal notes in ARBITRARY formats run through the
* same extractor → auditor → merger pipeline, with the file itself as the
* provenance anchor. There is no chat: the detail card shows the file path
* as a STATIC entry (not a link), and understanding observations still
* apply (judged from the author's own voice in the notes). The run's usage
* log lives here, mirroring ChatRecord.extractions. Ids are "src_" + hash
* of the file path — deterministic, so re-importing the same file re-runs
* extraction against the same source id (mappings/events dedupe by it). */
* as a STATIC entry (not a link). The run's usage log lives here,
* mirroring ChatRecord.extractions. Ids are "src_" + hash of the file
* path — deterministic, so re-importing the same file re-runs extraction
* against the same source id (mappings dedupe by it). */
export interface NotesSource {
id: string;
filePath: string;
@@ -149,39 +149,9 @@ export interface CoverageEntry {
foldedAt: string;
}
/**
* Event kinds (DECIDED 2026-09-02): quiz outcomes only — correct / miss /
* dont-know — plus `observation`, the extractor's per-proposition
* understanding note joining the ledger as a session-scoped observation
* (named-word scale; treatment pending the state-model ideation). No
* `taught` events at all; no misconception kind — a misconception is a
* TRUE correction in the graph, and the wrongness lives here, in the
* ledger, until the state model exists.
*/
export type LedgerEventKind = "correct" | "miss" | "dont-know" | "observation";
export interface LedgerEvent {
/** null = not yet attributed. */
nodeId: string | null;
kind: LedgerEventKind;
/** The chat that produced this evidence — or a notes source id
* ("src_…") for imported-notes observations. */
chatId: string;
at: string;
/** Idempotency key: the quiz part's toolCallId, or `obs:<sourceId>` for
* understanding observations — re-emission of the same chat must never
* double-record an event. */
ref?: string;
/** The learner's quiz note (diagnostic gold), or the understanding
* LEVEL word for observations — a session-relative judgment. */
note?: string;
/** The quiz question text, when nodeId is null (later attribution input). */
question?: string;
}
/**
* Store format (src/store/graph.ts): JSONL ledgers at config.LEDGER_FILES
* (nodes/edges/events/mappings/sessions), all append-only.
* (nodes/edges/mappings/sessions), all append-only.
*
* nodes.jsonl one line per node — the FIRST line for an id defines
* name/statement/createdAt; later lines for the
@@ -198,27 +168,23 @@ export interface LedgerEvent {
* (never extract the same turns twice).
*
* The EXTRACTION PIPELINE is the sole graph writer (extractor → auditor →
* merger), tree-blind and post-hoc: the extractor reads only the transcript
* and proposes propositions/edges/quiz-attribution with verbatim evidence;
* the auditor sees the extractor's output PLUS the current graph, owns
* faithfulness (evidence check) and identity (link-or-create: point each
* proposition at the existing node it is, or let it be created — recording
* the session's phrasings as aliases); the merger (core/graph.ts,
* deterministic) applies the verdicts and appends.
* merger), tree-blind and post-hoc (2026-09-27 cleanup — clear separation
* of work): the extractor is the ONLY stage that sees the raw source
* (whole session as readable markdown, or raw notes) and extracts the
* reasoning chains — propositions (lonely relevant ones included) plus
* follows/supports edges, with verbatim evidence; the auditor NEVER sees
* the source — its job is preparing the merge: identity verdicts
* (link-or-create, merging equivalent forms of the same proposition) via
* graph-exploration tools (find / list_sessions / list_propositions), and
* it may draw BRIDGING edges from existing graph nodes to the new
* propositions; the merger (core/graph.ts, deterministic) applies the
* verdicts and appends. Quiz events and understanding observations are
* GONE — the pipeline is the only session→graph bridge, and it carries
* propositions + edges, nothing else.
*/
// ─── Extraction findings (extractor output → audit → merger input) ───────────
/**
* The extractor's standardized per-session understanding scale (DECIDED
* 2026-09-08): a CLOSED named-word list — no numbers, no prose, no
* justifications (a justification, if the agent wants one, is dropped;
* it is not stored). The level is session- and extraction-run-relative
* judgment about the learner, shown next to its session — never a node
* property.
*/
export type UnderstandingLevel = "solid" | "shaky" | "accepted" | "resisted" | "not yet engaged";
/** One proposed proposition — a node candidate (or a re-encounter of one). */
export interface ExtractedProposition {
/** Extractor-local id ("p1", …) referenced by edges + attribution. */
@@ -231,7 +197,8 @@ export interface ExtractedProposition {
/** The minimal proposition — a true, self-contained, world-facing
* statement, at full precision. No learner / teacher / session
* references: misconceptions are stated as the true correction, and
* wrongness lives in the event ledger, never in the node. */
* wrongness is never recorded in the graph (2026-09-27: the event
* ledger is gone). */
statement: string;
/** Verbatim phrasings the session used. */
aliases: string[];
@@ -239,17 +206,17 @@ export interface ExtractedProposition {
messages: string[];
/** Verbatim transcript quotes backing the proposition. */
evidence: string[];
/** The extractor's understanding level for this proposition IN THIS
* session — one word from the UnderstandingLevel scale. Joins the
* ledger as an observation event (note = the level word); treatment
* pending the state-model ideation. */
understanding?: UnderstandingLevel;
}
/** One proposed relation between two propositions. */
/** One proposed relation. Endpoints are extractor-local proposition ids
* ("p1", …); the AUDITOR may also emit bridging edges whose from/to is an
* EXISTING graph node id (connecting the new content to what the graph
* already holds) — the merger resolves both. */
export interface ExtractedEdge {
/** Proposition id that is followed / supported. */
/** Proposition id (or, for auditor-drawn bridges, a graph node id)
* that is followed / supported. */
from: string;
/** Proposition id (or graph node id) that follows / is supported. */
to: string;
type: EdgeType;
/** The message ids where this relation was asserted — provenance. */
@@ -258,26 +225,10 @@ export interface ExtractedEdge {
evidence: string[];
}
/**
* The extractor's quiz attribution — its ONLY say about quiz events. Event
* kinds, timestamps, idempotency refs, notes and question texts are all
* rebuilt deterministically by the merger from the transcript's quiz tool
* parts; the LLM supplies exactly the judgment call: WHICH proposition each
* quiz tested. Tree-blind, so EVERY quiz attributes here (there is no tree
* mapping to lean on) — retroactively against the whole transcript.
*/
export interface ExtractionAttribution {
/** quiz toolCallId → proposition id; null / missing = unattributed (the
* event lands with nodeId null + the question text, a later-
* attribution input). */
quiz: Record<string, string | null>;
}
/** The extractor's output shape. */
/** The extractor's output shape: the raw reasoning chains. */
export interface SessionFindings {
propositions: ExtractedProposition[];
edges: ExtractedEdge[];
attribution: ExtractionAttribution;
}
/** The auditor's verdict on one proposition: the findings, plus `linkTo` —
@@ -286,20 +237,19 @@ export interface AuditedProposition extends ExtractedProposition {
/** The existing GraphNode id this proposition IS — the auditor's
* identity verdict. On link, the session's phrasings (statement,
* aliases) join the node's aliases and the provenance unions.
* Conservative: when unsure, leave it out — duplicates over wrong
* merges. */
* EQUIVALENCE BIAS (2026-09-27): paraphrases, rewordings, and
* compressions of the same claim MERGE — do not create a duplicate
* because the wording differs. Only genuinely distinct propositions
* stay apart. */
linkTo?: string;
}
/** The auditor's output: the verified findings (with identity verdicts),
* plus the drops (the audit trail of what was rejected and why). */
/** The auditor's output: the findings with identity verdicts, plus any
* bridging edges drawn to existing graph nodes. No drop channel — the
* auditor never rejects; the extractor is fully trusted (2026-09-27). */
export interface AuditResult {
propositions: AuditedProposition[];
edges: ExtractedEdge[];
attribution: ExtractionAttribution;
/** Dropped candidates: proposition id, edge "from→to", or attribution
* toolCallId — with the reason. */
dropped: { ref: string; reason: string }[];
}
/** The GET /api/graph payload — everything the home view's PURE renderer needs. */
@@ -328,11 +278,6 @@ export interface GraphSnapshot {
fileSources: Record<string, string>;
/** nodeId → chatIds that touched the node (click-through provenance). */
nodeSessions: Record<string, string[]>;
/** nodeId → chatId → that session's understanding observation (the
* extractor's per-session named-word note; session-relative data,
* shown next to the session list — the confidence history). Multiple
* notes for one (node, chat) collapse to the latest. */
nodeObservations: Record<string, Record<string, string>>;
/** Structural communities (Louvain over follows+supports), size ≥ 3,
* joined with persisted LLM labelings. Purely structural — never
* depends on layout. */
+8 -35
View File
@@ -13,17 +13,16 @@
*
* preserve-graph the chat's extraction bookkeeping dies — its coverage
* rows and its legacy checkpoint lines — and NOTHING else:
* nodes, edges, events, mappings stay byte-identical.
* nodes, edges, mappings stay byte-identical.
* The graph keeps the session's contributions; the
* snapshot simply no longer resolves the dead chatId to
* a title (nodeSessions/nodeObservations filter unknown
* ids at read time — no UI ghosting).
* a title (nodeSessions filters unknown ids at read
* time — no UI ghosting).
*
* delete-nodes "this session never happened": every trace of the
* chat leaves the ledgers — its provenance points on
* every node and edge line, its link lines (their
* phrasings die with them), its quiz events and
* observations, its mapping lines (the audit trail), its
* phrasings die with them), its mapping lines (the audit trail), its
* coverage/checkpoint rows — and orphans die: a node
* whose references all came from this chat disappears
* entirely, pruned from the positions and embedding
@@ -58,7 +57,6 @@ import type {
CoverageEntry,
GraphEdge,
GraphNode,
LedgerEvent,
NodeMapping,
SessionCheckpoint,
} from "./contracts.ts";
@@ -76,8 +74,6 @@ export interface ChatDeletionStats {
nodesRemoved: number;
/** Edge assertion LINES removed (empty-provenance or dead endpoint). */
edgesRemoved: number;
/** Events removed (quiz outcomes + observations). */
eventsRemoved: number;
/** Mapping lines removed (the chat's audit trail). */
mappingsRemoved: number;
/** Coverage rows removed. */
@@ -89,7 +85,6 @@ export interface ChatDeletionStats {
const EMPTY_STATS: ChatDeletionStats = {
nodesRemoved: 0,
edgesRemoved: 0,
eventsRemoved: 0,
mappingsRemoved: 0,
coverageRemoved: 0,
checkpointsRemoved: 0,
@@ -126,7 +121,6 @@ export function deleteChatTraces(graphStore: GraphStore, chatId: string, mode: C
// ── delete-nodes: the full purge ──────────────────────────────────────
const nodeLines = graphStore.readNodes();
const edgeLines = graphStore.readEdges();
const events = graphStore.readEvents();
const mappings = graphStore.readMappings();
// Does the chat touch the graph at all? (A never-extracted chat still
@@ -136,7 +130,6 @@ export function deleteChatTraces(graphStore: GraphStore, chatId: string, mode: C
const touched =
nodeLines.some((line) => line.provenance.some((pt) => pt.chatId === chatId)) ||
edgeLines.some((line) => line.provenance.some((pt) => pt.chatId === chatId)) ||
events.some((event) => event.chatId === chatId) ||
mappings.some((m) => m.chatId === chatId);
if (!touched) {
if (stats.coverageRemoved > 0 || stats.checkpointsRemoved > 0) {
@@ -149,7 +142,7 @@ export function deleteChatTraces(graphStore: GraphStore, chatId: string, mode: C
// remains — another chat/src's provenance point on any of its lines,
// a surviving mapping line (notes-front-door nodes carry EMPTY
// provenance: the file IS the anchor, recorded through the mapping
// line), or a surviving event naming it. (Edge endpoints are deliberately
// line). (Edge endpoints are deliberately
// NOT survival references: an edge's asserting session always left
// provenance on its endpoints, so endpoint-only liveness cannot
// occur in a consistent ledger — and counting it would keep nodes
@@ -163,9 +156,6 @@ export function deleteChatTraces(graphStore: GraphStore, chatId: string, mode: C
for (const mapping of mappings) {
if (mapping.chatId !== chatId) live.add(mapping.graphNodeId);
}
for (const event of events) {
if (event.chatId !== chatId && event.nodeId !== null) live.add(event.nodeId);
}
// 2. Node lines. Per-line (each line is ONE extraction run's append, so
// per-session alias provenance exists at line granularity):
@@ -219,19 +209,16 @@ export function deleteChatTraces(graphStore: GraphStore, chatId: string, mode: C
outEdges.push(points.length === line.provenance.length ? line : { ...line, provenance: points });
}
// 4. Events, mappings: the chat's rows die outright.
const outEvents = events.filter((event) => event.chatId !== chatId);
// 4. Mappings: the chat's rows die outright.
const outMappings = mappings.filter((m) => m.chatId !== chatId);
stats.nodesRemoved = [...new Set(nodeLines.map((line) => line.id))].filter((id) => !live.has(id)).length;
stats.edgesRemoved = edgeLines.length - outEdges.length;
stats.eventsRemoved = events.length - outEvents.length;
stats.mappingsRemoved = mappings.length - outMappings.length;
graphStore.rewriteLedgers({
nodes: outNodes,
edges: outEdges,
events: outEvents,
mappings: outMappings,
coverage,
checkpoints,
@@ -264,8 +251,6 @@ export function deleteChatTraces(graphStore: GraphStore, chatId: string, mode: C
*
* - every line of the node (create + link lines)
* - every EDGE touching it (from or to), whatever session asserted it
* - every EVENT naming it (quiz outcomes + observations), from every
* session — the evidence history dies with the thing it was about
* - every mapping line that anchored it (the audit trail; for a
* notes-born node the file IS the anchor — its mapping line is the
* only one)
@@ -287,20 +272,16 @@ export function deleteChatTraces(graphStore: GraphStore, chatId: string, mode: C
export interface NodeDeletionStats {
/** Edge assertion LINES removed (every edge touching the node). */
edgesRemoved: number;
/** Events removed (quiz outcomes + observations naming the node). */
eventsRemoved: number;
/** Mapping lines removed (the node's audit trail). */
mappingsRemoved: number;
}
/** The confirm dialog's pre-flight facts (the GET on the same node id):
* what full death would take with it. sessions is the deduped set of
* chats/notes sources that reference the node — provenance points,
* mapping lines, and events all count. */
* chats/notes sources that reference the node — provenance points and
* mapping lines count. */
export interface NodeDeletionImpact {
edges: number;
quizEvents: number;
observations: number;
sessions: string[];
}
@@ -313,7 +294,6 @@ function nodeLinesOf(graphStore: GraphStore, nodeId: string): GraphNode[] {
export function nodeDeletionImpact(graphStore: GraphStore, nodeId: string): NodeDeletionImpact | null {
if (nodeLinesOf(graphStore, nodeId).length === 0) return null;
const edges = graphStore.readEdges().filter((line) => line.from === nodeId || line.to === nodeId);
const events = graphStore.readEvents().filter((event) => event.nodeId === nodeId);
const sessions = new Set<string>();
for (const line of nodeLinesOf(graphStore, nodeId)) {
for (const pt of line.provenance) sessions.add(pt.chatId);
@@ -321,11 +301,8 @@ export function nodeDeletionImpact(graphStore: GraphStore, nodeId: string): Node
for (const mapping of graphStore.readMappings()) {
if (mapping.graphNodeId === nodeId) sessions.add(mapping.chatId);
}
for (const event of events) sessions.add(event.chatId);
return {
edges: edges.length,
quizEvents: events.filter((event) => event.kind !== "observation").length,
observations: events.filter((event) => event.kind === "observation").length,
sessions: [...sessions].sort(),
};
}
@@ -341,11 +318,9 @@ export function deleteNodeTraces(graphStore: GraphStore, nodeId: string): NodeDe
if (nodeLines.length === 0) return null;
const edgeLines = graphStore.readEdges();
const events = graphStore.readEvents();
const mappings = graphStore.readMappings();
const outEdges = edgeLines.filter((line) => line.from !== nodeId && line.to !== nodeId);
const outEvents = events.filter((event) => event.nodeId !== nodeId);
const outMappings = mappings.filter((m) => m.graphNodeId !== nodeId);
// Only the touched ledgers: the node's own lines always die (its file
@@ -354,7 +329,6 @@ export function deleteNodeTraces(graphStore: GraphStore, nodeId: string): NodeDe
graphStore.rewriteLedgers({
nodes: outNodes,
...(outEdges.length !== edgeLines.length ? { edges: outEdges } : {}),
...(outEvents.length !== events.length ? { events: outEvents } : {}),
...(outMappings.length !== mappings.length ? { mappings: outMappings } : {}),
});
@@ -367,7 +341,6 @@ export function deleteNodeTraces(graphStore: GraphStore, nodeId: string): NodeDe
return {
edgesRemoved: edgeLines.length - outEdges.length,
eventsRemoved: events.length - outEvents.length,
mappingsRemoved: mappings.length - outMappings.length,
};
}
+73 -245
View File
@@ -2,45 +2,45 @@
* The knowledge graph: merger (the deterministic final stage of the
* extraction pipeline — vision §1 construction) + derivation (reads).
*
* The extraction pipeline is the SOLE graph writer, tree-blind and post-hoc:
* The extraction pipeline is the SOLE graph writer, tree-blind and post-hoc
* (2026-09-27 cleanup — clear separation of work):
*
* extractor (LLM) → SessionFindings: propositions, edges, quiz
* attribution, per-proposition understanding notes —
* each with verbatim evidence, from the transcript ALONE
* auditor (LLM) → AuditResult: the verified subset, with the IDENTITY
* VERDICT per proposition (linkTo an existing node, or
* create), and the drops for the log
* extractor (LLM) → SessionFindings: the raw reasoning chains —
* propositions (lonely relevant ones included) and
* follows/supports edges, each with verbatim evidence,
* from the source ALONE (whole session as markdown,
* or raw notes). Fully trusted: no stage re-checks it.
* auditor (LLM) → AuditResult: identity verdicts (linkTo an existing
* node, or create) via graph-exploration tools, plus
* any BRIDGING edges it draws between existing graph
* nodes and the new propositions. Never sees the
* source; never drops.
* merger (HERE) → mergeFindings(): deterministic ledger appends
*
* The merger is deliberately the ONLY stage that touches the ledgers, and it
* is pure code: the LLM supplies exactly the judgment calls (what the
* propositions are, what follows/supports what, which proposition a quiz
* tested, whether a proposition IS an existing node), while event
* kinds/timestamps/refs/provenance — anything the model could silently
* mis-copy — are rebuilt deterministically from the transcript's tool parts
* and the message ids in the findings. Idempotent + incremental: refs are
* chat-scoped idempotency keys, edges dedupe by (from,to,type)-already-
* asserted-by-this-chat, mapping lines dedupe by (chat, sourceId), and
* sessions.jsonl checkpoints which chats have been extracted (never the
* same turns twice).
* propositions are, what follows/supports what, whether a proposition IS an
* existing node), while timestamps/provenance — anything the model could
* silently mis-copy — are rebuilt deterministically. Idempotent +
* incremental: edges dedupe by (from,to,type)-already-asserted-by-this-chat,
* mapping lines dedupe by (chat, sourceId), and coverage entries /
* sessions.jsonl checkpoints which messages and chats have been extracted
* (never the same turns twice). No events: the events ledger is GONE
* (2026-09-27) — the graph carries propositions + edges, nothing else.
*
* Identity across re-encounters is the AUDITOR's judgment (link-or-create),
* not string matching: the conservative v1 exact-statement merge was
* replaced by the 2026-09-02 redesign. A re-extracted session's
* propositions link to the existing nodes; the link appends a node line
* whose phrasings join `aliases` and whose provenance unions (getGraph
* merges lines by id).
* Identity across re-encounters is the AUDITOR's judgment (link-or-create
* with an equivalence bias — paraphrases merge), not string matching.
* A re-extracted session's propositions link to the existing nodes; the
* link appends a node line whose phrasings join `aliases` and whose
* provenance unions (getGraph merges lines by id).
*
* Derivation below (getGraph / deriveNodeStates / graphSnapshot) is reads
* only — everything is recomputed from the ledgers. Every function takes
* the storage ports (core/ports.ts) as explicit parameters — pure core, no
* adapter imports; the composition root (src/server/stores.ts) wires the fs
* adapters in.
* Derivation below (getGraph / graphSnapshot) is reads only — everything
* is recomputed from the ledgers. Every function takes the storage ports
* (core/ports.ts) as explicit parameters — pure core, no adapter imports;
* the composition root (src/server/stores.ts) wires the fs adapters in.
*/
import { randomBytes } from "node:crypto";
import type { LhMessage } from "./native-messages.ts";
import type { ToolResultMessage } from "@earendil-works/pi-ai";
import type { CoverageEntry } from "./contracts/graph.ts";
import type {
AuditResult,
@@ -50,13 +50,9 @@ import type {
GraphNode,
ProvenancePoint,
GraphSnapshot,
LedgerEvent,
LedgerEventKind,
UnderstandingLevel,
} from "./contracts.ts";
import { MIN_CLUSTER_SIZE } from "./clusters.ts";
import { birthMappingIndex, birthSessionOf, birthSessionPartition } from "./session-clusters.ts";
import type { QuestionAnswerData } from "./question-core.ts";
import type { ChatStore, GraphStore } from "./ports.ts";
/** The storage ports every graph operation needs, passed explicitly. */
@@ -71,71 +67,22 @@ function newGraphNodeId(): string {
}
/** What one merge run merges FROM (the caller supplies it — the merger never
* reads the chat store): a session's transcript + its coverage checkpoint,
* or — for the format-agnostic notes front door (2026-09-13) — `messages:
* null`: raw notes carry no tool parts, so no quiz events are rebuilt (quiz
* outcomes inside notes surface only through the extractor's understanding
* observations), and the whole-file checkpoint stamps the import.
* reads the chat store): the coverage checkpoint for a session batch, or —
* for the format-agnostic notes front door (2026-09-13) — no coverage at
* all: the whole-file checkpoint stamps the import.
*
* Coverage (vision-extraction-coverage.md, settled 2026-09-15): a chat merge
* passes the BATCH it extracted — the caller computed the fingerprints
* before the run, from the same snapshot the merger sees — and step 4
* appends one CoverageEntry per message on success ("covered" means
* exactly this merge happened). `batchTag` prefixes the observation refs so
* re-runs of the SAME batch dedupe while a re-opened (mutated) message's
* fresh understanding observations land as new events; the notes path
* passes neither and keeps the whole-file checkpoint. */
* before the run, from the same snapshot the merger sees — and the final
* step appends one CoverageEntry per message on success ("covered" means
* exactly this merge happened). The notes path passes neither and keeps
* the whole-file checkpoint. */
export interface MergeSource {
messages: LhMessage[] | null;
foldedAt: string;
/** The batch being merged (chat path): one entry per message in
* `messages`, same order — undefined (notes path / coverage-less
* callers) → the legacy whole-chat checkpoint is written instead. */
/** The batch being merged (chat path): one entry per message in the
* extracted batch — undefined (notes path / coverage-less callers) →
* the legacy whole-chat checkpoint is written instead. */
coverage?: CoverageEntry[];
/** The batch's deterministic tag (core/coverage.ts batchTag) — obs refs. */
batchTag?: string;
}
// ─── transcript scanning (shared by the merger + trigger) ─────────────────────
/** A flattened, order-preserving view of every tool call (+ its result
* state) in a native transcript. Wave D: the pairing is native — calls
* come from assistant content blocks; a result map built from the
* toolResult messages carries the outcome (a dangling call is
* input-available, an error result is output-error). */
interface ToolPartView {
name: string;
toolCallId: string;
state: string;
input: unknown;
output: unknown;
}
function toolParts(messages: LhMessage[]): ToolPartView[] {
const results = new Map<string, ToolResultMessage>();
for (const m of messages) {
if ((m as { role?: string }).role === "toolResult") {
results.set((m as ToolResultMessage).toolCallId, m as ToolResultMessage);
}
}
const out: ToolPartView[] = [];
for (const m of messages) {
if ((m as { role?: string }).role !== "assistant") continue;
const content = (m as { content?: unknown }).content;
if (!Array.isArray(content)) continue;
for (const block of content as Array<{ type?: string; id?: string; name?: string; arguments?: unknown }>) {
if (block.type !== "toolCall" || typeof block.id !== "string") continue;
const result = results.get(block.id);
out.push({
name: block.name ?? "",
toolCallId: block.id,
state: result ? (result.isError ? "output-error" : "output-available") : "input-available",
input: block.arguments,
output: result && !result.isError ? result.details : undefined,
});
}
}
return out;
}
/** (chatId, messageIds) → provenance points; order-preserving, deduped. */
@@ -167,21 +114,14 @@ const samePoint = (a: ProvenancePoint, b: ProvenancePoint): boolean =>
* Mapping lines record each verdict: one per (chat, sourceId). Idempotent:
* a proposition id already mapped for this chat reuses its resolution.
*
* Events are rebuilt deterministically from the transcript: quiz outcomes
* (correct / miss / dont-know; kinds + timestamps + refs from the tool
* parts, nodeId from the extractor's attribution through this run's
* mapping) plus one observation event per proposition carrying an
* understanding note (ref `obs:<sourceId>`). No taught events, ever.
*
* `chatId` is the run's source key: a real chat, or a notes source id
* ("src_…") for the format-agnostic import path — same ledgers, same
* idempotency, one graph. The `source` parameter carries the transcript
* being merged (null messages = raw notes: no quiz events) and the
* checkpoint stamp; the merger itself is chat-store-free.
* idempotency, one graph. The `source` parameter carries the checkpoint
* stamp; the merger itself is chat-store-free.
*/
export function mergeFindings(chatId: string, audit: AuditResult, deps: GraphDeps, source: MergeSource): EmissionStats {
const { graphStore: ledgers } = deps;
const stats: EmissionStats = { nodes: 0, edges: 0, events: 0 };
const stats: EmissionStats = { nodes: 0, edges: 0 };
// 1. Resolution: this run's propositions → graph node ids. Existing
// mapping lines for THIS chat are loaded first — the audit trail doubles
@@ -238,63 +178,41 @@ export function mergeFindings(chatId: string, audit: AuditResult, deps: GraphDep
}
}
// 2. Edges. Resolve endpoints through the mapping; dedupe by
// (from,to,type) already asserted by THIS chat (an existing assertion
// by another chat stays untouched — provenance unions at read time).
// 2. Edges. Resolve endpoints through the mapping — with BRIDGING
// (2026-09-27): an endpoint the extractor never proposed but that
// names an EXISTING graph node is the auditor drawing a connection
// between the graph it explored and this batch's propositions. Such
// edges carry chat-scoped provenance with NO message anchor (nothing
// in the transcript asserts them — the auditor is the asserter);
// transcript-articulated edges keep message-granular provenance.
// Dedupe by (from,to,type) already asserted by THIS chat (an existing
// assertion by another chat stays untouched — provenance unions at
// read time).
const assertedByChat = new Set<string>();
for (const line of ledgers.readEdges()) {
const key = `${line.from}\u0000${line.to}\u0000${line.type}`;
if (line.provenance.some((pt) => pt.chatId === chatId)) assertedByChat.add(key);
}
for (const edge of audit.edges) {
const from = mapping.get(edge.from);
const to = mapping.get(edge.to);
if (!from || !to || from === to) continue; // unmapped endpoint — skip
const resolve = (endpoint: string): string | undefined =>
mapping.get(endpoint) ?? (existingIds.has(endpoint) ? endpoint : undefined);
const from = resolve(edge.from);
const to = resolve(edge.to);
if (!from || !to || from === to) continue; // unresolvable endpoint — skip
const bridged = !mapping.has(edge.from) || !mapping.has(edge.to);
const key = `${from}\u0000${to}\u0000${edge.type}`;
if (assertedByChat.has(key)) continue;
ledgers.appendEdge({ from, to, type: edge.type, provenance: pointsOf(chatId, edge.messages) });
ledgers.appendEdge({
from,
to,
type: edge.type,
provenance: bridged ? [{ chatId, messageId: "" }] : pointsOf(chatId, edge.messages),
});
assertedByChat.add(key);
stats.edges++;
}
// 3. Events, rebuilt deterministically in transcript order. Refs are
// chat-scoped idempotency keys (quiz toolCallId, obs:<sourceId>) —
// a ref already in the ledger means this evidence was extracted before.
const seenRefs = new Set<string>();
for (const event of ledgers.readEvents()) {
if (event.chatId === chatId && event.ref) seenRefs.add(event.ref);
}
for (const part of source.messages ? toolParts(source.messages) : []) {
if (part.name !== "quiz" || part.state !== "output-available") continue;
const event = quizEvent(part, chatId, audit, mapping, now);
if (event?.ref !== undefined && !seenRefs.has(event.ref)) {
ledgers.appendEvent(event);
seenRefs.add(event.ref);
stats.events++;
}
}
// Understanding observations: one per proposition carrying a note,
// resolved to the node the verdict mapped it to. Refs carry the batch
// tag when there is one: run-local source ids ("p1"…) shift position
// between runs, so a bare obs:p1 could never both dedupe a re-run of
// the same batch AND let a re-opened message's fresh level through.
for (const p of audit.propositions) {
if (p.understanding === undefined || p.understanding.length === 0) continue;
const ref = source.batchTag !== undefined ? `obs:${source.batchTag}:${p.id}` : `obs:${p.id}`;
if (seenRefs.has(ref)) continue;
ledgers.appendEvent({
nodeId: mapping.get(p.id) ?? null,
kind: "observation",
chatId,
at: now,
ref,
note: p.understanding,
});
seenRefs.add(ref);
stats.events++;
}
// 4. Checkpoint — THE COVERAGE DISCIPLINE (vision-extraction-coverage.md):
// 3. Checkpoint — THE COVERAGE DISCIPLINE (vision-extraction-coverage.md):
// only now, after every append succeeded, is the batch covered. The
// chat path appends one CoverageEntry per message (per-batch atomic —
// an abort or failure mid-run leaves earlier batches covered, this
@@ -308,50 +226,11 @@ export function mergeFindings(chatId: string, audit: AuditResult, deps: GraphDep
return stats;
}
/** Build the ledger event for one answered quiz tool part (or null to skip).
* Deterministic in everything except the attribution-supplied nodeId. */
function quizEvent(
part: ToolPartView,
chatId: string,
audit: AuditResult,
mapping: Map<string, string>,
now: string,
): LedgerEvent | null {
const out = part.output;
if (typeof out !== "object" || out === null) return null;
const answer = out as Partial<QuestionAnswerData>;
if (answer.kind !== "quiz") return null; // ask questions are not graded
if (answer.dismissed === true) return null; // no answer — no evidence
let kind: LedgerEventKind;
if (answer.dontKnow === true) kind = "dont-know";
else if (answer.correct === true) kind = "correct";
else kind = "miss";
// Tree-blind: EVERY quiz attributes through the extractor's judgment —
// the vision's retroactive attribution, with no tree mapping to lean on.
const attributedId = audit.attribution.quiz[part.toolCallId];
const nodeId = attributedId ? (mapping.get(attributedId) ?? null) : null;
const event: LedgerEvent = {
nodeId,
kind,
chatId,
at: typeof answer.answeredAt === "string" ? answer.answeredAt : now,
ref: part.toolCallId,
};
if (typeof answer.note === "string" && answer.note.length > 0) event.note = answer.note;
// The question text is the attribution input for unattributed quizzes.
if (nodeId === null && typeof answer.questionText === "string") event.question = answer.questionText;
return event;
}
// ─── derivation (reads only — everything below is recomputable) ─────────────
export interface DerivedGraph {
nodes: GraphNode[];
edges: GraphEdge[];
events: LedgerEvent[];
}
/**
@@ -396,14 +275,14 @@ export function getGraph(graphStore: GraphStore): DerivedGraph {
merged.push(edge);
}
}
return { nodes: [...byId.values()], edges: merged, events: graphStore.readEvents() };
return { nodes: [...byId.values()], edges: merged };
}
// ─── the /api/graph payload ──────────────────────────────────────────────────
/** Everything the home graph view needs, in one fetch (shape: contracts). */
export function graphSnapshot(deps: GraphDeps): GraphSnapshot {
const { nodes, edges, events } = getGraph(deps.graphStore);
const { nodes, edges } = getGraph(deps.graphStore);
const nodeChats = new Map<string, Set<string>>();
const touch = (nodeId: string, chatId: string): void => {
@@ -424,9 +303,6 @@ export function graphSnapshot(deps: GraphDeps): GraphSnapshot {
touch(edge.to, pt.chatId);
}
}
for (const event of events) {
if (event.nodeId !== null) touch(event.nodeId, event.chatId);
}
// Labels for every source the graph can point back at: real chats
// (title, deep-linkable) and imported notes files (path, static — the
@@ -453,27 +329,6 @@ export function graphSnapshot(deps: GraphDeps): GraphSnapshot {
if (ids.length > 0) nodeSessions[nodeId] = ids;
}
// Understanding observations per (node, chat) — the extractor's
// session-relative note (latest wins within a chat). Read-time join for
// the detail card: the confidence history next to the session list.
const nodeObservations: Record<string, Record<string, string>> = {};
const latestObs = new Map<string, { at: string; note: string; index: number }>();
events.forEach((event, index) => {
if (event.kind !== "observation" || event.nodeId === null) return;
if (!event.note) return;
const key = `${event.nodeId}\u0000${event.chatId}`;
const current = latestObs.get(key);
if (!current || event.at > current.at || (event.at === current.at && index > current.index)) {
latestObs.set(key, { at: event.at, note: event.note, index });
}
});
for (const [key, { note }] of latestObs) {
const sep = key.indexOf("\u0000");
const nodeId = key.slice(0, sep);
const chatId = key.slice(sep + 1);
(nodeObservations[nodeId] ??= {})[chatId] = note;
}
// BIRTH-SESSION CLUSTERS (wave 7, owner-locked 2026-09-23): embeddings
// are dead — a cluster is the SESSION THAT CREATED ITS PROPOSITIONS,
// derived on read from provenance (core/session-clusters.ts). The label
@@ -502,7 +357,6 @@ export function graphSnapshot(deps: GraphDeps): GraphSnapshot {
sessions,
fileSources,
nodeSessions,
nodeObservations,
clusters,
...(Object.keys(positions).length > 0 ? { positions } : {}),
};
@@ -577,48 +431,22 @@ export function graphSessionIndex(deps: GraphDeps): GraphSessionEntry[] {
);
}
/** One birth proposition of one session, for the prober's per-session
* view. `understanding` is the extractor's latest observation for
* (node, session) — the named-word scale (solid/shaky/accepted/resisted/
* not yet engaged) — and `evaluatedAt` is THAT event's own timestamp
* (owner ruling 2026-09-23: quiz outcomes are evidence in the ledger,
* but they never move the evaluation date — the level's event does). */
/** One birth proposition of one session — WHAT WAS COVERED there (2026-09-27
* cleanup: no understanding levels; assessment is practice-module
* territory and nothing writes it until that module exists). */
export interface GraphSessionProposition {
nodeId: string;
/** The compact handle — the same name the canvas shows. */
name: string;
understanding: UnderstandingLevel | null;
/** The `at` of the event that minted `understanding`; null when the
* session never evaluated this proposition. */
evaluatedAt: string | null;
}
/** The propositions BORN in one session, with the session's latest
* understanding of each. Latest observation wins by `at`, then by
* ledger order (the same tie rule the snapshot's nodeObservations
* uses). Members sorted by node id — deterministic. */
/** The propositions BORN in one session — what that session first
* articulated. Members sorted by node id — deterministic. */
export function graphSessionPropositions(deps: GraphDeps, sessionId: string): GraphSessionProposition[] {
const mappingFallback = birthMappingIndex(deps.graphStore.readMappings());
const born = getGraph(deps.graphStore).nodes.filter((node) => birthSessionOf(node, mappingFallback) === sessionId);
const latest = new Map<string, { at: string; note: string; index: number }>();
deps.graphStore.readEvents().forEach((event, index) => {
if (event.kind !== "observation" || event.nodeId === null || event.chatId !== sessionId) return;
if (!event.note) return;
const current = latest.get(event.nodeId);
if (!current || event.at > current.at || (event.at === current.at && index > current.index)) {
latest.set(event.nodeId, { at: event.at, note: event.note, index });
}
});
return born
.map((node) => {
const observation = latest.get(node.id);
return {
nodeId: node.id,
name: node.name,
understanding: (observation?.note as UnderstandingLevel | undefined) ?? null,
evaluatedAt: observation?.at ?? null,
};
})
.map((node) => ({ nodeId: node.id, name: node.name }))
.sort((a, b) => (a.nodeId < b.nodeId ? -1 : 1));
}
-4
View File
@@ -18,7 +18,6 @@ import type {
GraphNode,
JobKind,
JobRecord,
LedgerEvent,
ExtractionRunUsage,
NodeMapping,
PlanTree,
@@ -84,8 +83,6 @@ export interface GraphStore {
readEdges(): GraphEdge[];
/** Append one quiz outcome / understanding observation. */
appendEvent(event: LedgerEvent): void;
readEvents(): LedgerEvent[];
/** Append one identity-map line (chat's tree node → graph node). */
appendMapping(mapping: NodeMapping): void;
@@ -130,7 +127,6 @@ export interface GraphStore {
rewriteLedgers(next: {
nodes?: GraphNode[];
edges?: GraphEdge[];
events?: LedgerEvent[];
mappings?: NodeMapping[];
coverage?: CoverageEntry[];
checkpoints?: SessionCheckpoint[];
+126
View File
@@ -0,0 +1,126 @@
/**
* Session → markdown — the ONE formatting engine (owner ruling 2026-09-27).
*
* The extractor's view of a teaching session and the future user-facing
* session export share this engine: a session is rendered as clean,
* readable markdown — speaker headers, quiz/check-in blocks, tool notes —
* with the message-id anchors preserved (extraction provenance depends on
* the extractor reporting the `[id]` tags it saw; `anchors: false` mints
* the same document without them, for the user export).
*
* Pure core: native messages in (LhMessage), string out. Content parity
* with the coverage line builder (core/coverage.ts messageViewLines) is
* deliberate — every message that produces view lines produces a markdown
* block, in the same order — but the formats are SEPARATE by design
* (2026-09-27): coverage fingerprints hash the line format and stay stable
* (no mass re-extraction when presentation changes); markdown is
* presentation. A failed tool call renders nothing (retry context, not
* learning evidence); custom markers are invisible, same as the view.
*/
import type { LhMessage } from "./native-messages.ts";
import type { ToolResultMessage } from "@earendil-works/pi-ai";
/** The user-facing text blocks of a native message (string or text parts). */
function textOf(content: unknown): string[] {
if (typeof content === "string") return content.trim() ? [content] : [];
if (!Array.isArray(content)) return [];
return content
.filter((c): c is { type: "text"; text: string } => (c as { type?: string }).type === "text")
.map((c) => c.text)
.filter((t) => t.trim().length > 0);
}
/** One tool result's learner-visible outcome, rendered as body lines. */
function toolResultBody(result: ToolResultMessage): { heading: string; body: string[] } {
const out = result.details as
| Partial<{
questionText: string;
correct: boolean;
dontKnow: boolean;
note: string;
text: string;
answers: string[];
other: string;
dismissed: boolean;
}>
| undefined;
if (result.toolName === "quiz" || result.toolName === "explain") {
const heading = result.toolName === "quiz" ? "Quiz" : "Recall check";
const question = typeof out?.questionText === "string" && out.questionText ? out.questionText : "(no question text)";
const body = [`**Q:** ${question}`];
if (out?.dismissed === true) body.push("The learner skipped this check.");
else if (out?.dontKnow === true) body.push("The learner did not know.");
else if (out?.correct === true) body.push("The learner answered correctly.");
else if (out?.correct === false) body.push("The learner answered incorrectly.");
else if (typeof out?.text === "string" && out.text.trim()) body.push(`The learner answered in his own words: ${out.text.trim()}`);
if (typeof out?.note === "string" && out.note) body.push(`*(his thinking: ${out.note})*`);
return { heading, body };
}
if (result.toolName === "ask_user_question") {
const question = typeof out?.questionText === "string" && out.questionText ? out.questionText : "(no question text)";
const body = [`**Q:** ${question}`];
const chosen = [
...(Array.isArray(out?.answers) ? out!.answers! : []),
...(typeof out?.other === "string" && out.other.trim() ? [out.other.trim()] : []),
];
if (typeof out?.text === "string" && out.text.trim()) body.push(`The learner answered: ${out.text.trim()}`);
else if (chosen.length > 0) body.push(`The learner chose: ${chosen.join(", ")}`);
else if (out?.dismissed === true) body.push("The learner skipped this question.");
if (typeof out?.note === "string" && out.note) body.push(`*(his thinking: ${out.note})*`);
return { heading: "Question", body };
}
// Parity with the line builder: other clean tool results are one line.
return { heading: "Tool", body: [`*${result.toolName} result delivered*`] };
}
/**
* ONE message as a markdown block — `null` when the message renders
* nothing (a failed tool call, a bare marker, an empty text part set).
* `mid` is the provenance anchor shown when `anchors` is on.
*/
export function messageMarkdown(message: LhMessage, mid: string, anchors = true): string | null {
const role = (message as { role?: string }).role;
const anchor = anchors ? ` · ${mid}` : "";
const parts: string[] = [];
if (role === "user") {
for (const text of textOf((message as { content: unknown }).content)) {
parts.push(`### Learner${anchor}\n\n${text.trim()}`);
}
} else if (role === "assistant") {
for (const text of textOf((message as { content: unknown }).content)) {
parts.push(`### Teacher${anchor}\n\n${text.trim()}`);
}
} else if (role === "toolResult") {
const result = message as ToolResultMessage;
if (result.isError) return null; // retry context, not learning evidence
const { heading, body } = toolResultBody(result);
parts.push(`### ${heading}${anchor}\n\n${body.join("\n\n")}`);
}
// custom markers are invisible, same as the extraction view.
return parts.length > 0 ? parts.join("\n\n") : null;
}
/**
* A whole session (or one extraction BATCH — same engine, subset of
* messages) as one markdown document. Blocks render in message order;
* messages that render nothing are skipped silently. The optional title
* mints the document header (`# <title>`).
*/
export function sessionMarkdown(
messages: ReadonlyArray<LhMessage>,
opts: { title?: string; anchors?: boolean } = {},
): string {
const blocks: string[] = [];
if (opts.title !== undefined && opts.title.length > 0) blocks.push(`# ${opts.title}`);
for (const message of messages) {
const block = messageMarkdown(message, (message as { id?: string }).id ?? "", opts.anchors ?? true);
if (block !== null) blocks.push(block);
}
return blocks.join("\n\n");
}