amosblomqvist
d1b46132b9
Part 1 — the ledger + remnants interlude (master-dispatched, owner ruling 2026-09-23: nothing configures the graph beyond chat models): - The embedding ledger surface is gone: readEmbeddings/writeEmbeddings ports, the EmbeddingsLedger contract, EMBEDDINGS_FILE (config, store impls, graphRevision source list), and graph-deletion's vector-pruning step. On-disk data files stay untouched (inert history). - The extraction embedder seam dies wholesale: ExtractionPipelineDeps drops the embedder dep (chat-handler + backfill/import-notes stubs go with it), the 'embedding' phase leaves ExtractionPhase, the usage capture/persist helpers die, and stages.embedding leaves ExtractionRunUsage (historical records keep their shape — reads are lenient). Extraction is two stages again, like the local-embedder era. - Consumer-less remnants: latestQuizOutcomes + QUIZ_EVENT_KINDS, clusterContentId, matchClusterLabelings + Community + ClusterLabeling, the readClusterLabels/appendClusterLabel port seams + store impls + LEDGER_FILES.clusterLabels. NamingRunUsage/readNamingRuns STAY (standing ruling — the cost pill renders historical naming spend); clusterHue + MIN_CLUSTER_SIZE stay (live). - The 'labeler' role dies: BACKGROUND_MODEL_ROLES, ROLE_SLOT, ModelRole, the catalog pin projections, and the UI roster strings. RolePins keys shrink with the union. The notes birth fallback (owner-approved 2026-09-23, surfaced by the interlude's live smoke: production notes imports create provenance-EMPTY nodes (importNotes passes messages: null — the file's anchor is the mapping line), so the provenance-only birth rule left every imported file unclustered and invisible to the prober's session list, against the locked file-birth ruling. A provenance-empty node now births under its FIRST mapping line's chatId (ledger order); only notes-born nodes ever hit the fallback (chat nodes always carry provenance; a re-encountered node's link-append re-opens the provenance rule). Deletion-safe: mapping lines of deleted chats are stripped alongside their provenance. Live-verified on port 3004: a fresh notes import clusters under its src_ id with the file path as the label, end to end with no embedder. 845/845 tests, typecheck/lint/build green. EOF )
62 lines
2.5 KiB
TypeScript
62 lines
2.5 KiB
TypeScript
/**
|
|
* Import raw notes files through the format-agnostic front door (vision §1,
|
|
* 2026-09-13) — every *.md / *.txt file in a directory becomes one
|
|
* extraction run (extractor → auditor → merger) over its raw text. Run
|
|
* from this worktree only — data never crosses trees:
|
|
*
|
|
* npx tsx --env-file=.env.local scripts/import-notes.ts <dir> [concurrency]
|
|
*
|
|
* Default dir: real-data-extraction/learn-sessions (the prototype
|
|
* learning-session exports — deliberately NOT imported as chats: they run
|
|
* through the notes path, raw). Runs in this process: the dev server's
|
|
* status chips do NOT see them (registry is per-process) — the dev server
|
|
* sees runs triggered through /api/graph/import.
|
|
*/
|
|
import * as fs from "node:fs";
|
|
import * as path from "node:path";
|
|
import { createExtractionPipeline } from "../src/agent/subagents/extraction.ts";
|
|
import { getGraph } from "../src/core/graph.ts";
|
|
import { stores } from "../src/server/stores.ts";
|
|
|
|
async function main() {
|
|
const dir = process.argv[2] ?? "real-data-extraction/learn-sessions";
|
|
const limit = Number(process.argv[3] ?? 4);
|
|
const files = fs
|
|
.readdirSync(dir)
|
|
.filter((f) => f.endsWith(".md") || f.endsWith(".markdown") || f.endsWith(".txt"))
|
|
.sort();
|
|
console.log(`[import-notes] ${files.length} note files from ${dir} (concurrency ${limit})`);
|
|
|
|
const pipeline = createExtractionPipeline(stores);
|
|
const queue = [...files];
|
|
const totals = { nodes: 0, edges: 0, events: 0, failed: 0 };
|
|
const workers = Array.from({ length: Math.min(limit, files.length) }, async () => {
|
|
for (;;) {
|
|
const file = queue.shift();
|
|
if (!file) return;
|
|
const filePath = path.join(dir, file);
|
|
const t0 = Date.now();
|
|
try {
|
|
const stats = await pipeline.importNotes({ filePath, text: fs.readFileSync(filePath, "utf-8") });
|
|
totals.nodes += stats.nodes;
|
|
totals.edges += stats.edges;
|
|
totals.events += stats.events;
|
|
console.log(
|
|
`[import-notes] ${file}: ${stats.nodes} nodes, ${stats.edges} edges, ${stats.events} events in ${((Date.now() - t0) / 1000).toFixed(1)}s`,
|
|
);
|
|
} catch (err) {
|
|
totals.failed++;
|
|
console.error(`[import-notes] ${file} FAILED:`, err);
|
|
}
|
|
}
|
|
});
|
|
await Promise.all(workers);
|
|
|
|
const graph = getGraph(stores.graphStore);
|
|
console.log(
|
|
`[import-notes] done: +${totals.nodes} nodes, +${totals.edges} edges, +${totals.events} events, ${totals.failed} failed — ` +
|
|
`graph now ${graph.nodes.length} nodes, ${graph.edges.length} edges, ${graph.events.length} events`,
|
|
);
|
|
}
|
|
|
|
main(); |