Notes front door: format-agnostic import into the graph pipeline

The same extractor→auditor→merger pipeline now runs over raw personal
notes in arbitrary formats (vision §1, DECIDED 2026-09-13):

- NotesSource records (data/graph/sources.json, deterministic id from
  path hash → idempotent re-imports) play the chatId role in mappings,
  checkpoints, and observation events; usage lands on the source record
  (appendSourceUsage, idempotent per startedAt, kept across upserts).
- mergeFindings is chat-store-free: callers pass MergeSource
  {messages, foldedAt} — sessions pass the chat record, notes pass
  messages:null (no quiz-event rebuild; a deterministic rebuild needs
  structured quiz tool parts).
- Second extractor prompt (NOTES_EXTRACTOR_INSTRUCTIONS) for general-
  purpose extraction; understanding judges the AUTHOR's grasp. Auditor
  wording input-agnostic. extract/audit share one generic runSubmitAgent.
- graphSnapshot.fileSources (srcId→filePath); notes sources render as
  static path entries in the detail card (FileTextIcon, never a link).
- One shared pipeline instance (exported from chat-handler): same
  single-flight registry, live status chips (label = file path), usage
  accounting, and true abort for both front doors.
- Transport: POST /api/graph/import (multipart, 3-wide worker pool) +
  Import notes… button in the graph view; scripts/import-notes.ts for
  batch runs (concurrency 4).
- real-data-extraction/ gitignored (raw import sources never committed).

355/355 tests; typecheck, lint, build green.
This commit is contained in:
amosblomqvist
2026-09-13 02:08:55 +03:00
parent b769d32bc6
commit 543cf33789
16 changed files with 961 additions and 114 deletions
+52
View File
@@ -0,0 +1,52 @@
/**
* POST /api/graph/import — the format-agnostic front door's transport
* (vision §1, 2026-09-13): raw personal-notes files in, one extraction run
* per file through the SAME pipeline the sessions use. Multipart form data
* (field "files", any text format — the extractor is format-agnostic); the
* pipeline mints a deterministic source id per file path, persists the
* source record (the file's provenance + the run's usage), and runs
* extractor → auditor → merger over the raw text.
*
* Small concurrency cap: the model calls are the slow part, and a burst of
* 25 simultaneous runs would trip provider rate limits — a 3-wide worker
* pool keeps the run registry (the dock's status chips) and the ledgers
* healthy. The dock's chips + graph refetch track progress live through
* /api/graph/status, as with session runs.
*/
import { extraction } from "@/src/server";
import type { EmissionStats } from "@/src/core";
export const dynamic = "force-dynamic";
interface ImportResult {
filePath: string;
ok: boolean;
stats?: EmissionStats;
error?: string;
}
export async function POST(request: Request) {
const form = await request.formData();
const files = form.getAll("files").filter((f): f is File => f instanceof File);
if (files.length === 0) {
return Response.json({ results: [] as ImportResult[], error: "no files" }, { status: 400 });
}
const results: ImportResult[] = [];
const queue = [...files];
const workers = Array.from({ length: Math.min(3, files.length) }, async () => {
for (;;) {
const file = queue.shift();
if (!file) return;
try {
const text = await file.text();
const stats = await extraction.importNotes({ filePath: file.name, text });
results.push({ filePath: file.name, ok: true, stats });
} catch (err) {
results.push({ filePath: file.name, ok: false, error: err instanceof Error ? err.message : String(err) });
}
}
});
await Promise.all(workers);
return Response.json({ results });
}
+3 -1
View File
@@ -12,7 +12,9 @@ export const dynamic = "force-dynamic";
export async function GET() {
const runs = extractionStatuses().map((run) => ({
chatId: run.chatId,
title: chatStore.getChat(run.chatId)?.title ?? run.chatId,
// The run's own label (the file path for notes imports) wins; chat
// runs fall back to the chat store's title.
title: run.label ?? chatStore.getChat(run.chatId)?.title ?? run.chatId,
phase: run.phase,
startedAt: run.startedAt,
// Live per-stage usage (the money channel): model + per-step CallUsage,