Semantic topic clusters: local embeddings replace Louvain
Louvain only ever found the reasoning chains (the graph is disconnected by nature — every connected component became its own cluster) and edge-orphans could never join anything. Clusters are now RELATED TOPICS: propositions grouped by meaning. - Embedder (src/server/embeddings.ts): in-process transformers.js v4 running EmbeddingGemma-300M q8 (MTEB 68.4, ~300MB one-time download), MRL-truncated to 512d, "title: none | text:" prefix on 100% of inputs, L2-normalized so cosine = dot. Model/dim mismatch voids the ledger (full re-embed). Injected at the composition root — the agent layer never imports the server layer (architecture law). - Ledger (data/graph/embeddings.json): model + dim + vectors + the persisted cluster partition. Reclustering runs at the END of every embedding batch (post-merge hook, both front doors) — never per read (the agglomerative pass is ~1.5s at 338 nodes). - Clustering (src/core/semantic-clusters.ts): hand-rolled average-linkage agglomerative over cosine (npm HDBSCAN is an abandoned alpha). Stragglers stay singletons by default; attachStragglers is the all-in toggle. Threshold 0.60 swept on the real graph: 26 topics, 74/90 edge-orphans land in real topics, 10 stragglers. - Consumers (snapshot, settle, namer, replay) read the persisted partition — one cluster concept everywhere; core-survival label matching unchanged (20/26 topics kept labels). - Louvain retired: detectCommunities + graphology* deps deleted (graphology-louvain had leaked into the client bundle since the clusters wave); scripts/bench-graph.ts (dead FA2 bench) removed. - scripts/embed-graph.ts: backfill/cold-start (338 nodes in 150s including the one-time model download).
This commit is contained in:
@@ -13,7 +13,6 @@
|
||||
* playground mode baked into the rest state); omit it for the free layout.
|
||||
*/
|
||||
import { graphStore } from "../src/store/graph.ts";
|
||||
import { detectCommunities } from "../src/core/clusters.ts";
|
||||
// --depth=vertical|radial: bake the playground depth mode into the ledger.
|
||||
const depthArg = process.argv.find((a) => a.startsWith("--depth="));
|
||||
const depthMode: "off" | "vertical" | "radial" =
|
||||
@@ -33,6 +32,12 @@ function main() {
|
||||
nodes.push(line);
|
||||
}
|
||||
const edges = graphStore.readEdges();
|
||||
// The semantic-topic source: the embedding ledger (run
|
||||
// scripts/embed-graph.ts first if missing/stale).
|
||||
const embeddings = graphStore.readEmbeddings();
|
||||
if (Object.keys(embeddings.vectors).length === 0) {
|
||||
console.warn("[replay] embedding ledger is empty — clusters will be absent (run scripts/embed-graph.ts)");
|
||||
}
|
||||
|
||||
// Batches: each imported file, in real import order (sources.json
|
||||
// importedAt). A node belongs to the file whose mapping first claims it
|
||||
@@ -79,8 +84,12 @@ function main() {
|
||||
const before = new Map(positions);
|
||||
const batchNodes = nodes.filter((n) => delivered.has(n.id));
|
||||
positions = incrementalLayout(batchNodes, activeEdges, positions, {
|
||||
// The same partition the real settle would see for this batch.
|
||||
clusters: detectCommunities(batchNodes, activeEdges).communities,
|
||||
// The same partition the real settle would see for this batch:
|
||||
// the persisted semantic topics, filtered to delivered nodes
|
||||
// (never recomputed — the agglomerative pass is ~1.5s at scale).
|
||||
clusters: embeddings.clusters
|
||||
.map((c) => ({ members: c.members.filter((id) => delivered.has(id)) }))
|
||||
.filter((c) => c.members.length > 0),
|
||||
...(depthMode === "off"
|
||||
? {}
|
||||
: {
|
||||
|
||||
Reference in New Issue
Block a user