amosblomqvist
f5edc7f0a5
Louvain only ever found the reasoning chains (the graph is disconnected by nature — every connected component became its own cluster) and edge-orphans could never join anything. Clusters are now RELATED TOPICS: propositions grouped by meaning. - Embedder (src/server/embeddings.ts): in-process transformers.js v4 running EmbeddingGemma-300M q8 (MTEB 68.4, ~300MB one-time download), MRL-truncated to 512d, "title: none | text:" prefix on 100% of inputs, L2-normalized so cosine = dot. Model/dim mismatch voids the ledger (full re-embed). Injected at the composition root — the agent layer never imports the server layer (architecture law). - Ledger (data/graph/embeddings.json): model + dim + vectors + the persisted cluster partition. Reclustering runs at the END of every embedding batch (post-merge hook, both front doors) — never per read (the agglomerative pass is ~1.5s at 338 nodes). - Clustering (src/core/semantic-clusters.ts): hand-rolled average-linkage agglomerative over cosine (npm HDBSCAN is an abandoned alpha). Stragglers stay singletons by default; attachStragglers is the all-in toggle. Threshold 0.60 swept on the real graph: 26 topics, 74/90 edge-orphans land in real topics, 10 stragglers. - Consumers (snapshot, settle, namer, replay) read the persisted partition — one cluster concept everywhere; core-survival label matching unchanged (20/26 topics kept labels). - Louvain retired: detectCommunities + graphology* deps deleted (graphology-louvain had leaked into the client bundle since the clusters wave); scripts/bench-graph.ts (dead FA2 bench) removed. - scripts/embed-graph.ts: backfill/cold-start (338 nodes in 150s including the one-time model download).
24 lines
956 B
TypeScript
24 lines
956 B
TypeScript
/**
|
|
* Embed the whole graph (backfill / cold start): one-time pass that
|
|
* fills the embedding ledger for every node — the same embedNewNodes the
|
|
* post-merge hook runs, just with an empty (or stale-model) ledger so
|
|
* every node is "new". First run downloads the model weights (~300MB)
|
|
* to the local HF cache; subsequent runs embed only what's missing.
|
|
*
|
|
* npx tsx scripts/embed-graph.ts
|
|
*/
|
|
import { graphStore } from "../src/store/graph.ts";
|
|
import { embedNewNodes, EMBEDDING_MODEL, EMBEDDING_DIM } from "../src/server/embeddings.ts";
|
|
|
|
async function main() {
|
|
const t0 = Date.now();
|
|
console.log(`[embed] model: ${EMBEDDING_MODEL} (dim ${EMBEDDING_DIM}, q8)`);
|
|
const { embedded, total } = await embedNewNodes(graphStore);
|
|
console.log(`[embed] embedded ${embedded} nodes → ledger holds ${total} vectors (${((Date.now() - t0) / 1000).toFixed(1)}s)`);
|
|
}
|
|
|
|
main().catch((err) => {
|
|
console.error("[embed] failed:", err);
|
|
process.exit(1);
|
|
});
|