Notes front door: format-agnostic import into the graph pipeline

The same extractor→auditor→merger pipeline now runs over raw personal
notes in arbitrary formats (vision §1, DECIDED 2026-09-13):

- NotesSource records (data/graph/sources.json, deterministic id from
  path hash → idempotent re-imports) play the chatId role in mappings,
  checkpoints, and observation events; usage lands on the source record
  (appendSourceUsage, idempotent per startedAt, kept across upserts).
- mergeFindings is chat-store-free: callers pass MergeSource
  {messages, foldedAt} — sessions pass the chat record, notes pass
  messages:null (no quiz-event rebuild; a deterministic rebuild needs
  structured quiz tool parts).
- Second extractor prompt (NOTES_EXTRACTOR_INSTRUCTIONS) for general-
  purpose extraction; understanding judges the AUTHOR's grasp. Auditor
  wording input-agnostic. extract/audit share one generic runSubmitAgent.
- graphSnapshot.fileSources (srcId→filePath); notes sources render as
  static path entries in the detail card (FileTextIcon, never a link).
- One shared pipeline instance (exported from chat-handler): same
  single-flight registry, live status chips (label = file path), usage
  accounting, and true abort for both front doors.
- Transport: POST /api/graph/import (multipart, 3-wide worker pool) +
  Import notes… button in the graph view; scripts/import-notes.ts for
  batch runs (concurrency 4).
- real-data-extraction/ gitignored (raw import sources never committed).

355/355 tests; typecheck, lint, build green.
This commit is contained in:
amosblomqvist
2026-09-13 02:08:55 +03:00
parent b769d32bc6
commit 543cf33789
16 changed files with 961 additions and 114 deletions
+30
View File
@@ -20,6 +20,8 @@
/** Two relations only (DECIDED 2026-09-02): follows = b is derivable from a;
* supports = a grounds b without entailing it. No scalars, no encoded
* strength — strength is derived at read time from topology. */
import type { ExtractionRunUsage } from "./chat.ts";
export type EdgeType = "follows" | "supports";
/** What one extraction run merged into the graph ledgers — the merger's
@@ -37,10 +39,32 @@ export interface EmissionStats {
* edges record every (chat, message) where they came up, deep-linkable to
* the exact transcript point. */
export interface ProvenancePoint {
/** The chat this point anchors to — or a notes source id ("src_…"):
* imports carry no message anchors (provenance is the file itself,
* recorded through the mapping line + the source record). */
chatId: string;
messageId: string;
}
/** One imported notes file — the format-agnostic front door's bookkeeping
* (DECIDED 2026-09-13): personal notes in ARBITRARY formats run through the
* same extractor → auditor → merger pipeline, with the file itself as the
* provenance anchor. There is no chat: the detail card shows the file path
* as a STATIC entry (not a link), and understanding observations still
* apply (judged from the author's own voice in the notes). The run's usage
* log lives here, mirroring ChatRecord.extractions. Ids are "src_" + hash
* of the file path — deterministic, so re-importing the same file re-runs
* extraction against the same source id (mappings/events dedupe by it). */
export interface NotesSource {
id: string;
filePath: string;
/** ISO — when this file was imported (the checkpoint's foldedAt). */
importedAt: string;
/** The run's usage log — same append-only discipline as
* ChatRecord.extractions, idempotent per startedAt. */
extractions?: ExtractionRunUsage[];
}
export interface GraphNode {
/** Opaque stable id, store-assigned: "n_" + nanoid-like. */
id: string;
@@ -107,6 +131,8 @@ export interface LedgerEvent {
/** null = not yet attributed. */
nodeId: string | null;
kind: LedgerEventKind;
/** The chat that produced this evidence — or a notes source id
* ("src_…") for imported-notes observations. */
chatId: string;
at: string;
/** Idempotency key: the quiz part's toolCallId, or `obs:<sourceId>` for
@@ -249,6 +275,10 @@ export interface GraphSnapshot {
edges: GraphEdge[];
/** chatId → title, for every chat that touched the graph. */
sessions: Record<string, string>;
/** srcId → file path, for every IMPORTED NOTES file that touched the
* graph (the format-agnostic front door): a static provenance entry —
* the detail card shows the path as plain text, not a link. */
fileSources: Record<string, string>;
/** nodeId → chatIds that touched the node (click-through provenance). */
nodeSessions: Record<string, string[]>;
/** nodeId → chatId → that session's understanding observation (the