amosblomqvist
c86430e8bf
The owner's wave-7 ruling: embeddings die entirely (git history is the
stash); clustering is birth-session derived (chunk A, 950ddfd), labels
are session names, and the prober's graph tools replace injection.
- settings: GraphSettings collapses to { enabled } — embedding,
embeddingModel, clustering, labeling all die. Migration story: stale
record keys are UNREAD and dropped by the sanitizer's reconstruction;
the next graph-section PUT (replace-whole) persists the collapsed
shape. The settings-panel GraphSection renders one master row.
- deleted: src/server/embeddings.ts (adapters, resolveEmbeddingPortFor,
embedNewNodes, embeddingPort, embeddingModelCatalog), the
GET /api/models/embeddings route, scripts/embed-graph.ts, tests for
all three, and the /api/models/embeddings picker surfaces in the UI.
- prices: EMBEDDING_PRICES overlay + callCost fallback die. cost-summary:
the embedding stage fold dies — back to three stages (extraction/
audit/naming) per the owner ruling; historical stages.embedding spend
in old run records is simply not displayed.
- ports: EmbeddingPort/EmbeddingResult/EmbeddingTask die. The embeddings
LEDGER surface (readEmbeddings/writeEmbeddings + EmbeddingsLedger +
EMBEDDINGS_FILE + graph-deletion's vector pruning) STAYS — inert data,
kept only so graph-context.ts compiles until Teacher's pass removes
it; the seam dies in the follow-up interlude.
- chat-handler (approved cross-lane unwiring, wave-6 precedent): the
extraction pipeline's embedder becomes a documented no-op (nothing
embedded, nothing spent, ledger untouched) until Teacher's pass
removes the phase; the graph-context injection dispatch gate, the
graphContext deps wiring, the setContextRespawn seam, and
turn-service's triggerGraphContext are gone — injection is dead
code pending Teacher's file deletions.
- scripts: backfill-graph/import-notes pass the same no-op embedder;
replay-layout derives clusters from birthSessionPartition (the same
partition the real settle uses) instead of the embeddings ledger.
- settings-body: parseGraphSettingsBody reads the master switch only —
old-shape keys in a body are ignored, not rejected; bad shapes 400.
- next.config: the @huggingface tracing-exclusion guards die with the
last reference.
Cross-lane flags: extraction.ts/graph-context.ts keep stale embed
comments + the embed phase until Teacher's full pass (their parallel
interlude owns those files — zero overlap kept by design);
semantic-clusters.ts + its test stay one more round (extraction.test.ts,
Teacher's file, still imports it — they delete it with the namer cases);
ChatSettings.graphContextTokens + the graph-context-control UI stay
until Teacher+UI.
Gates: tsc clean, eslint clean, 885/885 tests, build compiles.
Live-verified on the port-3004 dev tree: PUT /api/settings/graph with
dead keys collapses on write, bad shapes 400, GET reads {enabled: true},
/api/models/embeddings 404s.
69 lines
2.9 KiB
TypeScript
69 lines
2.9 KiB
TypeScript
/**
|
|
* Import raw notes files through the format-agnostic front door (vision §1,
|
|
* 2026-09-13) — every *.md / *.txt file in a directory becomes one
|
|
* extraction run (extractor → auditor → merger) over its raw text. Run
|
|
* from this worktree only — data never crosses trees:
|
|
*
|
|
* npx tsx --env-file=.env.local scripts/import-notes.ts <dir> [concurrency]
|
|
*
|
|
* Default dir: real-data-extraction/learn-sessions (the prototype
|
|
* learning-session exports — deliberately NOT imported as chats: they run
|
|
* through the notes path, raw). Runs in this process: the dev server's
|
|
* status chips do NOT see them (registry is per-process) — the dev server
|
|
* sees runs triggered through /api/graph/import.
|
|
*/
|
|
import * as fs from "node:fs";
|
|
import * as path from "node:path";
|
|
import { createExtractionPipeline } from "../src/agent/subagents/extraction.ts";
|
|
import { clusterNamingSettled } from "../src/agent/subagents/cluster-namer.ts";
|
|
import { getGraph } from "../src/core/graph.ts";
|
|
import { stores } from "../src/server/stores.ts";
|
|
|
|
async function main() {
|
|
const dir = process.argv[2] ?? "real-data-extraction/learn-sessions";
|
|
const limit = Number(process.argv[3] ?? 4);
|
|
const files = fs
|
|
.readdirSync(dir)
|
|
.filter((f) => f.endsWith(".md") || f.endsWith(".markdown") || f.endsWith(".txt"))
|
|
.sort();
|
|
console.log(`[import-notes] ${files.length} note files from ${dir} (concurrency ${limit})`);
|
|
|
|
const pipeline = createExtractionPipeline({ ...stores, embedder: async () => ({ embedded: 0, total: 0, skipped: "embeddings removed (wave 7)" }) });
|
|
const queue = [...files];
|
|
const totals = { nodes: 0, edges: 0, events: 0, failed: 0 };
|
|
const workers = Array.from({ length: Math.min(limit, files.length) }, async () => {
|
|
for (;;) {
|
|
const file = queue.shift();
|
|
if (!file) return;
|
|
const filePath = path.join(dir, file);
|
|
const t0 = Date.now();
|
|
try {
|
|
const stats = await pipeline.importNotes({ filePath, text: fs.readFileSync(filePath, "utf-8") });
|
|
totals.nodes += stats.nodes;
|
|
totals.edges += stats.edges;
|
|
totals.events += stats.events;
|
|
console.log(
|
|
`[import-notes] ${file}: ${stats.nodes} nodes, ${stats.edges} edges, ${stats.events} events in ${((Date.now() - t0) / 1000).toFixed(1)}s`,
|
|
);
|
|
} catch (err) {
|
|
totals.failed++;
|
|
console.error(`[import-notes] ${file} FAILED:`, err);
|
|
}
|
|
}
|
|
});
|
|
await Promise.all(workers);
|
|
|
|
// The post-merge cluster naming is fire-and-forget inside the pipeline;
|
|
// settle it before the process exits so labels actually land.
|
|
await clusterNamingSettled();
|
|
|
|
const graph = getGraph(stores.graphStore);
|
|
const labels = stores.graphStore.readClusterLabels().length;
|
|
console.log(
|
|
`[import-notes] done: +${totals.nodes} nodes, +${totals.edges} edges, +${totals.events} events, ${totals.failed} failed — ` +
|
|
`graph now ${graph.nodes.length} nodes, ${graph.edges.length} edges, ${graph.events.length} events, ` +
|
|
`${labels} named clusters`,
|
|
);
|
|
}
|
|
|
|
main(); |