Notes front door: format-agnostic import into the graph pipeline

The same extractor→auditor→merger pipeline now runs over raw personal
notes in arbitrary formats (vision §1, DECIDED 2026-09-13):

- NotesSource records (data/graph/sources.json, deterministic id from
  path hash → idempotent re-imports) play the chatId role in mappings,
  checkpoints, and observation events; usage lands on the source record
  (appendSourceUsage, idempotent per startedAt, kept across upserts).
- mergeFindings is chat-store-free: callers pass MergeSource
  {messages, foldedAt} — sessions pass the chat record, notes pass
  messages:null (no quiz-event rebuild; a deterministic rebuild needs
  structured quiz tool parts).
- Second extractor prompt (NOTES_EXTRACTOR_INSTRUCTIONS) for general-
  purpose extraction; understanding judges the AUTHOR's grasp. Auditor
  wording input-agnostic. extract/audit share one generic runSubmitAgent.
- graphSnapshot.fileSources (srcId→filePath); notes sources render as
  static path entries in the detail card (FileTextIcon, never a link).
- One shared pipeline instance (exported from chat-handler): same
  single-flight registry, live status chips (label = file path), usage
  accounting, and true abort for both front doors.
- Transport: POST /api/graph/import (multipart, 3-wide worker pool) +
  Import notes… button in the graph view; scripts/import-notes.ts for
  batch runs (concurrency 4).
- real-data-extraction/ gitignored (raw import sources never committed).

355/355 tests; typecheck, lint, build green.
This commit is contained in:
amosblomqvist
2026-09-13 02:08:55 +03:00
parent b769d32bc6
commit 543cf33789
16 changed files with 961 additions and 114 deletions
+62
View File
@@ -0,0 +1,62 @@
/**
* Import raw notes files through the format-agnostic front door (vision §1,
* 2026-09-13) — every *.md / *.txt file in a directory becomes one
* extraction run (extractor → auditor → merger) over its raw text. Run
* from this worktree only — data never crosses trees:
*
* npx tsx --env-file=.env.local scripts/import-notes.ts <dir> [concurrency]
*
* Default dir: real-data-extraction/learn-sessions (the prototype
* learning-session exports — deliberately NOT imported as chats: they run
* through the notes path, raw). Runs in this process: the dev server's
* status chips do NOT see them (registry is per-process) — the dev server
* sees runs triggered through /api/graph/import.
*/
import * as fs from "node:fs";
import * as path from "node:path";
import { createExtractionPipeline } from "../src/agent/subagents/extraction.ts";
import { getGraph } from "../src/core/graph.ts";
import { stores } from "../src/server/stores.ts";
async function main() {
const dir = process.argv[2] ?? "real-data-extraction/learn-sessions";
const limit = Number(process.argv[3] ?? 4);
const files = fs
.readdirSync(dir)
.filter((f) => f.endsWith(".md") || f.endsWith(".markdown") || f.endsWith(".txt"))
.sort();
console.log(`[import-notes] ${files.length} note files from ${dir} (concurrency ${limit})`);
const pipeline = createExtractionPipeline(stores);
const queue = [...files];
const totals = { nodes: 0, edges: 0, events: 0, failed: 0 };
const workers = Array.from({ length: Math.min(limit, files.length) }, async () => {
for (;;) {
const file = queue.shift();
if (!file) return;
const filePath = path.join(dir, file);
const t0 = Date.now();
try {
const stats = await pipeline.importNotes({ filePath, text: fs.readFileSync(filePath, "utf-8") });
totals.nodes += stats.nodes;
totals.edges += stats.edges;
totals.events += stats.events;
console.log(
`[import-notes] ${file}: ${stats.nodes} nodes, ${stats.edges} edges, ${stats.events} events in ${((Date.now() - t0) / 1000).toFixed(1)}s`,
);
} catch (err) {
totals.failed++;
console.error(`[import-notes] ${file} FAILED:`, err);
}
}
});
await Promise.all(workers);
const graph = getGraph(stores.graphStore);
console.log(
`[import-notes] done: +${totals.nodes} nodes, +${totals.edges} edges, +${totals.events} events, ${totals.failed} failed — ` +
`graph now ${graph.nodes.length} nodes, ${graph.edges.length} edges, ${graph.events.length} events`,
);
}
main();