The background extraction pipeline is invisible on the home surface —
essential observability per the user's directive: the graph view must
show what is currently running (extracting / auditing / merging) and
for which chat.
- extraction.ts: module-level registry of in-flight runs (chatId,
phase, startedAt), updated at each phase transition in runExtraction
and cleared in finally (done or failed). extractionStatuses() is the
read; one process, one registry.
- GET /api/graph/status: thin transport over the registry, chat titles
joined from the chat store.
- graph-view: polls every 2s and shows one chip per running pipeline
(spinner + phase + title + elapsed seconds, bottom-left); when a run
finishes (some -> none) the graph refetches, so merges land live.
Status is cosmetic — poll errors never surface.
- Unit test drives a mocked pipeline run and observes the registry
mid-flight: 'extracting' during the extractor's generate, 'auditing'
during the auditor's, empty after.
With real extracted data, full-sentence statements as canvas labels proved
unworkable: 10-line label blocks, illegible overprint clusters, bottom-edge
clipping. Per the user's 2026-09-08 decision:
- GraphNode/ExtractedProposition gain `name` — a very short, glanceable
label (glossary-headword style: "Will to power"), presentational only:
not unique, not identity, never precise. Set at creation; link-appends
never change it (first line defines; labels stay stable across sessions).
- `statement` keeps its full precision and becomes the identity anchor;
the detail card shows it under the name. `description` is dropped.
- Extractor coins the name (freshest session context); auditor carries it
through on create, rewording only clearly unglanceable names; linked
propositions keep the existing node's name.
- Renderer: canvas label = name; search matches name + statement;
detail card = name (bold) + statement (muted).
- Store guard requires name (pre-pivot node lines are skipped as legacy);
vision.md node-schema bullet + amendments table updated (2026-09-08).
The minimal implementation of the 2026-09-02 graph vision (docs/vision.md §1):
- contracts/graph.ts: every node is a TRUE proposition (statement is both
rendering label and identity key; required description; no kind, no name).
Edges are follows (derivable) / supports (grounds without entailing) with
message-granular provenance points {chatId, messageId}. Events reduce to
correct/miss/dont-know plus observation (understanding notes); no taught
events, no misconception kind.
- extraction.ts: extractor is TREE-BLIND (transcript only, no plan tree) and
free-form; auditor sees findings + the current graph, owns faithfulness and
identity (link-or-create verdict per proposition, recorded as linkTo);
transcriptView tags every line with its message id (provenance anchors).
- core/graph.ts merger: applies verdicts deterministically — linkTo appends
a link line (phrasings join aliases, provenance unions; reads merge by id),
absent creates; NO string matching anywhere. Quiz events rebuilt from tool
parts with attribution-only nodeId resolution; understanding notes become
observation events (obs:<sourceId>); mapping lines are the verdict audit
trail and make re-merges idempotent.
- deriveNodeStates: transitional — latest quiz event wins (known/missed/
dont-know, else candidate); taught unreachable until the state-model
ideation lands.
- graph-context.ts + 50-graph-context.md: follows chains, hotspot grouping,
state vocabulary updated for the new event kinds.
- graph-view.tsx: node label is the statement; detail shows statement +
description.
- tests: merger verdicts (link, no-string-match, cross-chat assertions),
tree-blind pipeline end-to-end, follows vocabulary in context tests.
276/276 green.
Schema: GraphNode + ExtractedProposition carry 'name' — a short 2-6 word
header, the rendering handle. Node theory made explicit: a node is an
objective world-facing fact; the learner's relationship to it lives
entirely in the event ledger (state labels are derived, never stored).
Extractor + auditor instructions enforce it — misconceptions state the
wrong idea itself as a claim, never 'the learner believes Y'. Legacy
ledger lines without a name derive one from the statement on read.
graph-view: labels use the name; the detail panel shows name as header
and statement as body.
The knowledge graph's write channel is now the pivot's target
architecture (vision §1 construction):
- src/agent/subagents/extraction.ts — the pipeline factory
(createExtractionPipeline({ chatStore, graphStore, extractor?,
auditor? }); models env LH_EXTRACT_MODEL ?? LH_MODEL):
* transcriptView(): the compact view both LLM stages see — final plan
tree with node ids, conversation verbatim, QUIZ lines with outcomes,
mark_done lines; job tool parts are one-liners, plan ops fold into
the tree line.
* Extractor: transcript → SessionFindings (propositions incl.
misconceptions and prose-only supports edges, attribution for
UNTAGGED probes, all with verbatim evidence) via one zod-validated
submit_findings call.
* Auditor: transcript + findings → the verified subset + drops (the
faithfulness net) via submit_review.
- src/core/graph.ts — emitChat/foldStaleChats/replayFinalTree retire;
mergeFindings() replaces them as the pipeline's deterministic apply
stage: exact/alias identity resolution (one mapping line per (chat,
graph node)), edge dedupe, and event rebuild FROM THE TRANSCRIPT's tool
parts — kinds/timestamps/refs/notes deterministic, the LLM's only event
say being WHICH proposition an untagged probe tested (retroactive —
the vision's open question #3, resolved structurally: tagged events
resolve deterministically; attribution rescues untagged and
tagged-but-unmapped ones; unattributable land null + question text).
pendingQuizAnswers() is the trigger helper.
- Trigger (turn-service.ts triggerExtraction, replacing the old
foldTurnIntoGraph body): once a chat accumulates
EXTRACTION.everyNQuizAnswers (LH_EXTRACT_N, default 3) new answered
quizzes, fire the pipeline async — single-flight per chat, failure
checkpoints nothing and re-runs on the next trigger. Deliberately a v1
placeholder isolated in ONE seam, trigger-agnostic pipeline.
- GET /api/graph is now a pure read (foldStaleChats gone).
- NodeMapping.treeNodeId → sourceId (the mapping key is now the writer's
local proposition id — tree node when tree-derived); store strict-read
schema updated. Local ledger data regenerates through the pipeline.
- Tests: graph.test.ts rewritten around mergeFindings (identity,
attribution paths incl. ghost-rescue, idempotency, cross-session
provenance, trigger counting), graph-context.test.ts merge section
converted to findings-driven, NEW extraction.test.ts — transcriptView
units + the full pipeline with mock models (audit drops, failure
checkpoints nothing, single-flight). 195/195, tsc/eslint/build clean.
Probe attribution (vision open question): probes attach retroactively by
the extractor against the whole transcript; a pre-tree probe maps to a
proposition the tree only articulates later. Fuzzy/embedding merge
adjudication stays deferred (conservative exact/alias only).
amosblomqvist