Extraction pipeline: extractor → auditor → merger — the sole graph writer (stage 5)

The knowledge graph's write channel is now the pivot's target
architecture (vision §1 construction):

- src/agent/subagents/extraction.ts — the pipeline factory
  (createExtractionPipeline({ chatStore, graphStore, extractor?,
  auditor? }); models env LH_EXTRACT_MODEL ?? LH_MODEL):
  * transcriptView(): the compact view both LLM stages see — final plan
    tree with node ids, conversation verbatim, QUIZ lines with outcomes,
    mark_done lines; job tool parts are one-liners, plan ops fold into
    the tree line.
  * Extractor: transcript → SessionFindings (propositions incl.
    misconceptions and prose-only supports edges, attribution for
    UNTAGGED probes, all with verbatim evidence) via one zod-validated
    submit_findings call.
  * Auditor: transcript + findings → the verified subset + drops (the
    faithfulness net) via submit_review.
- src/core/graph.ts — emitChat/foldStaleChats/replayFinalTree retire;
  mergeFindings() replaces them as the pipeline's deterministic apply
  stage: exact/alias identity resolution (one mapping line per (chat,
  graph node)), edge dedupe, and event rebuild FROM THE TRANSCRIPT's tool
  parts — kinds/timestamps/refs/notes deterministic, the LLM's only event
  say being WHICH proposition an untagged probe tested (retroactive —
  the vision's open question #3, resolved structurally: tagged events
  resolve deterministically; attribution rescues untagged and
  tagged-but-unmapped ones; unattributable land null + question text).
  pendingQuizAnswers() is the trigger helper.
- Trigger (turn-service.ts triggerExtraction, replacing the old
  foldTurnIntoGraph body): once a chat accumulates
  EXTRACTION.everyNQuizAnswers (LH_EXTRACT_N, default 3) new answered
  quizzes, fire the pipeline async — single-flight per chat, failure
  checkpoints nothing and re-runs on the next trigger. Deliberately a v1
  placeholder isolated in ONE seam, trigger-agnostic pipeline.
- GET /api/graph is now a pure read (foldStaleChats gone).
- NodeMapping.treeNodeId → sourceId (the mapping key is now the writer's
  local proposition id — tree node when tree-derived); store strict-read
  schema updated. Local ledger data regenerates through the pipeline.
- Tests: graph.test.ts rewritten around mergeFindings (identity,
  attribution paths incl. ghost-rescue, idempotency, cross-session
  provenance, trigger counting), graph-context.test.ts merge section
  converted to findings-driven, NEW extraction.test.ts — transcriptView
  units + the full pipeline with mock models (audit drops, failure
  checkpoints nothing, single-flight). 195/195, tsc/eslint/build clean.

Probe attribution (vision open question): probes attach retroactively by
the extractor against the whole transcript; a pre-tree probe maps to a
proposition the tree only articulates later. Fuzzy/embedding merge
adjudication stays deferred (conservative exact/alias only).
This commit is contained in:
Amos
2026-08-31 22:36:41 +03:00
parent 59e1fee850
commit af90fadcda
12 changed files with 1168 additions and 320 deletions
+3 -3
View File
@@ -6,8 +6,8 @@
* session asserting the same (from,to,type)
* appends another line; reads merge)
* events.jsonl LedgerEvent — quiz outcomes + taught events
* mappings.jsonl NodeMapping — the naive-union identity map:
* (chatId, treeNodeId) → graphNodeId
* mappings.jsonl NodeMapping — the session→graph identity map:
* (chatId, sourceId) → graphNodeId
* sessions.jsonl SessionCheckpoint— emission checkpoints (foldedAt)
*
* Implements core/ports.ts GraphStore. data/graph/* is exclusively this
@@ -105,7 +105,7 @@ const isLedgerEvent = (v: unknown): v is LedgerEvent =>
isString(v.at);
const isNodeMapping = (v: unknown): v is NodeMapping =>
isRecord(v) && isString(v.chatId) && isString(v.treeNodeId) && isString(v.graphNodeId);
isRecord(v) && isString(v.chatId) && isString(v.sourceId) && isString(v.graphNodeId);
const isCheckpoint = (v: unknown): v is SessionCheckpoint =>
isRecord(v) && isString(v.chatId) && isString(v.foldedAt);