Notes front door: format-agnostic import into the graph pipeline
The same extractor→auditor→merger pipeline now runs over raw personal
notes in arbitrary formats (vision §1, DECIDED 2026-09-13):
- NotesSource records (data/graph/sources.json, deterministic id from
path hash → idempotent re-imports) play the chatId role in mappings,
checkpoints, and observation events; usage lands on the source record
(appendSourceUsage, idempotent per startedAt, kept across upserts).
- mergeFindings is chat-store-free: callers pass MergeSource
{messages, foldedAt} — sessions pass the chat record, notes pass
messages:null (no quiz-event rebuild; a deterministic rebuild needs
structured quiz tool parts).
- Second extractor prompt (NOTES_EXTRACTOR_INSTRUCTIONS) for general-
purpose extraction; understanding judges the AUTHOR's grasp. Auditor
wording input-agnostic. extract/audit share one generic runSubmitAgent.
- graphSnapshot.fileSources (srcId→filePath); notes sources render as
static path entries in the detail card (FileTextIcon, never a link).
- One shared pipeline instance (exported from chat-handler): same
single-flight registry, live status chips (label = file path), usage
accounting, and true abort for both front doors.
- Transport: POST /api/graph/import (multipart, 3-wide worker pool) +
Import notes… button in the graph view; scripts/import-notes.ts for
batch runs (concurrency 4).
- real-data-extraction/ gitignored (raw import sources never committed).
355/355 tests; typecheck, lint, build green.
This commit is contained in:
@@ -0,0 +1,62 @@
|
||||
/**
|
||||
* Import raw notes files through the format-agnostic front door (vision §1,
|
||||
* 2026-09-13) — every *.md / *.txt file in a directory becomes one
|
||||
* extraction run (extractor → auditor → merger) over its raw text. Run
|
||||
* from this worktree only — data never crosses trees:
|
||||
*
|
||||
* npx tsx --env-file=.env.local scripts/import-notes.ts <dir> [concurrency]
|
||||
*
|
||||
* Default dir: real-data-extraction/learn-sessions (the prototype
|
||||
* learning-session exports — deliberately NOT imported as chats: they run
|
||||
* through the notes path, raw). Runs in this process: the dev server's
|
||||
* status chips do NOT see them (registry is per-process) — the dev server
|
||||
* sees runs triggered through /api/graph/import.
|
||||
*/
|
||||
import * as fs from "node:fs";
|
||||
import * as path from "node:path";
|
||||
import { createExtractionPipeline } from "../src/agent/subagents/extraction.ts";
|
||||
import { getGraph } from "../src/core/graph.ts";
|
||||
import { stores } from "../src/server/stores.ts";
|
||||
|
||||
async function main() {
|
||||
const dir = process.argv[2] ?? "real-data-extraction/learn-sessions";
|
||||
const limit = Number(process.argv[3] ?? 4);
|
||||
const files = fs
|
||||
.readdirSync(dir)
|
||||
.filter((f) => f.endsWith(".md") || f.endsWith(".markdown") || f.endsWith(".txt"))
|
||||
.sort();
|
||||
console.log(`[import-notes] ${files.length} note files from ${dir} (concurrency ${limit})`);
|
||||
|
||||
const pipeline = createExtractionPipeline(stores);
|
||||
const queue = [...files];
|
||||
const totals = { nodes: 0, edges: 0, events: 0, failed: 0 };
|
||||
const workers = Array.from({ length: Math.min(limit, files.length) }, async () => {
|
||||
for (;;) {
|
||||
const file = queue.shift();
|
||||
if (!file) return;
|
||||
const filePath = path.join(dir, file);
|
||||
const t0 = Date.now();
|
||||
try {
|
||||
const stats = await pipeline.importNotes({ filePath, text: fs.readFileSync(filePath, "utf-8") });
|
||||
totals.nodes += stats.nodes;
|
||||
totals.edges += stats.edges;
|
||||
totals.events += stats.events;
|
||||
console.log(
|
||||
`[import-notes] ${file}: ${stats.nodes} nodes, ${stats.edges} edges, ${stats.events} events in ${((Date.now() - t0) / 1000).toFixed(1)}s`,
|
||||
);
|
||||
} catch (err) {
|
||||
totals.failed++;
|
||||
console.error(`[import-notes] ${file} FAILED:`, err);
|
||||
}
|
||||
}
|
||||
});
|
||||
await Promise.all(workers);
|
||||
|
||||
const graph = getGraph(stores.graphStore);
|
||||
console.log(
|
||||
`[import-notes] done: +${totals.nodes} nodes, +${totals.edges} edges, +${totals.events} events, ${totals.failed} failed — ` +
|
||||
`graph now ${graph.nodes.length} nodes, ${graph.edges.length} edges, ${graph.events.length} events`,
|
||||
);
|
||||
}
|
||||
|
||||
main();
|
||||
Reference in New Issue
Block a user