Part 1 — the ledger + remnants interlude (master-dispatched, owner ruling
2026-09-23: nothing configures the graph beyond chat models):
- The embedding ledger surface is gone: readEmbeddings/writeEmbeddings
ports, the EmbeddingsLedger contract, EMBEDDINGS_FILE (config, store
impls, graphRevision source list), and graph-deletion's vector-pruning
step. On-disk data files stay untouched (inert history).
- The extraction embedder seam dies wholesale: ExtractionPipelineDeps
drops the embedder dep (chat-handler + backfill/import-notes stubs go
with it), the 'embedding' phase leaves ExtractionPhase, the usage
capture/persist helpers die, and stages.embedding leaves
ExtractionRunUsage (historical records keep their shape — reads are
lenient). Extraction is two stages again, like the local-embedder era.
- Consumer-less remnants: latestQuizOutcomes + QUIZ_EVENT_KINDS,
clusterContentId, matchClusterLabelings + Community + ClusterLabeling,
the readClusterLabels/appendClusterLabel port seams + store impls +
LEDGER_FILES.clusterLabels. NamingRunUsage/readNamingRuns STAY
(standing ruling — the cost pill renders historical naming spend);
clusterHue + MIN_CLUSTER_SIZE stay (live).
- The 'labeler' role dies: BACKGROUND_MODEL_ROLES, ROLE_SLOT, ModelRole,
the catalog pin projections, and the UI roster strings. RolePins keys
shrink with the union.
The notes birth fallback (owner-approved 2026-09-23, surfaced by the
interlude's live smoke: production notes imports create provenance-EMPTY
nodes (importNotes passes messages: null — the file's anchor is the
mapping line), so the provenance-only birth rule left every imported
file unclustered and invisible to the prober's session list, against the
locked file-birth ruling. A provenance-empty node now births under its
FIRST mapping line's chatId (ledger order); only notes-born nodes ever
hit the fallback (chat nodes always carry provenance; a re-encountered
node's link-append re-opens the provenance rule). Deletion-safe: mapping
lines of deleted chats are stripped alongside their provenance.
Live-verified on port 3004: a fresh notes import clusters under its
src_ id with the file path as the label, end to end with no embedder.
845/845 tests, typecheck/lint/build green.
EOF
)
Louvain only ever found the reasoning chains (the graph is disconnected
by nature — every connected component became its own cluster) and
edge-orphans could never join anything. Clusters are now RELATED TOPICS:
propositions grouped by meaning.
- Embedder (src/server/embeddings.ts): in-process transformers.js v4
running EmbeddingGemma-300M q8 (MTEB 68.4, ~300MB one-time download),
MRL-truncated to 512d, "title: none | text:" prefix on 100% of inputs,
L2-normalized so cosine = dot. Model/dim mismatch voids the ledger
(full re-embed). Injected at the composition root — the agent layer
never imports the server layer (architecture law).
- Ledger (data/graph/embeddings.json): model + dim + vectors + the
persisted cluster partition. Reclustering runs at the END of every
embedding batch (post-merge hook, both front doors) — never per read
(the agglomerative pass is ~1.5s at 338 nodes).
- Clustering (src/core/semantic-clusters.ts): hand-rolled average-linkage
agglomerative over cosine (npm HDBSCAN is an abandoned alpha). Stragglers
stay singletons by default; attachStragglers is the all-in toggle.
Threshold 0.60 swept on the real graph: 26 topics, 74/90 edge-orphans
land in real topics, 10 stragglers.
- Consumers (snapshot, settle, namer, replay) read the persisted
partition — one cluster concept everywhere; core-survival label
matching unchanged (20/26 topics kept labels).
- Louvain retired: detectCommunities + graphology* deps deleted
(graphology-louvain had leaked into the client bundle since the
clusters wave); scripts/bench-graph.ts (dead FA2 bench) removed.
- scripts/embed-graph.ts: backfill/cold-start (338 nodes in 150s
including the one-time model download).
The renderer grows a live d3 sim (createGraphSimulation, shared with
the server settle): pointerdown on a node pins fx/fy and pointer capture
(preventDefault kills the compat mousedown, so d3-zoom never pans); the
first CLICK_SLOP-exceeding move reheats to alphaTarget 0.3; release
un-pins and the sim anneals (release-and-relax) — alpha < 0.005 settles
it, a 250ms debounced onCommit PUTs the rest state. Verified live: 123fps
during a 320ms drag (regions rebuilding every 3rd frame), 85-tick settle,
then zero idle frames. A slop-free press stays a click (select, no reheat,
no commit). PUT /api/graph/positions sanitizes (finite coords, known ids)
and if the graph grew mid-drag runs ONE incremental settle with the
client's arrangement as anchors (50-id PUT test: anchored drift 0.15).
Cluster regions swap convex hulls for a low-res world-space SPLAT bitmap:
per-member radial-gradient discs (alpha 0.2) + intra-cluster edges as
round-cap strokes (Minkowski dilation — concave shapes finally render),
upscaled with bilinear smoothing (the upscale IS the blur, no ctx.filter:
Safari). One tiny blit per frame replaces 32 clusters x 4 offset
Path2D polygons re-rasterized at DPR 2 (the old GPU coil-whine path);
during the live sim the bitmap rebuilds every 3rd frame. cluster-geometry.ts
retires convexHull/offsetPolygon/traceRoundedPath (tests follow; clusterHue
stays). Region labels stay (centroid + clusterHue, inverse-gated on zoom).
Three problems from the first cut (user report 2026-09-13), all rooted
and fixed:
1. Boundaries reshaping on zoom — an inward offset normal. In y-down
screen coordinates, monotone chain yields clockwise-on-screen hulls,
whose OUTWARD normal is (dy, -dx); the code used (-dy, dx), so the
14px padding pulled every hull vertex 14-20px INWARD. Zoomed in this
mostly passed; zoomed out, once a cluster projected smaller than ~2x
the pad, vertices crossed the centroid and the polygon self-
intersected — spiky garbage re-forming every frame. Geometry moved to
components/cluster-geometry.ts (exported, unit-tested) with the
corrected normal; tests pin the direction (offset moves every vertex
AWAY from the centroid; stays sane when the hull is smaller than the
pad; hull orientation asserted clockwise-on-screen).
2. GPU ~100% + audible coil whine on any interaction — the hull redraw
was hooked to sigma's afterRender, which fires for EVERY render
including ~a dozen hover-animation frames per hover, and each redraw
rasterized ~32 fills through ctx.filter blur(12px), a GPU-heavy
full-canvas filter. Now: redraws happen ONLY on camera updates and
resizes (hover leaves the hull canvas untouched — verified live),
and the blur is replaced with layered low-alpha fills (soft edge at
a fraction of the cost, no canvas filter at all).
3. Cluster labels cluttering the view — a persisted 'Cluster labels'
checkbox in the rendering settings (hulls stay, text toggles).
377/377 tests, typecheck, lint, build green. Live-verified: hover does
not repaint hulls, zoom does; far zoom-out yields 3 contiguous tinted
regions (no fragmentation); toggle clears/returns labels and persists.
Core-survival labelings (vision section 1, DECIDED 2026-09-13):
ClusterLabeling records in a new append-only ledger
(data/graph/cluster-labels.jsonl), immutable, id = clu_ + sha256 of the
sorted core so re-naming races and re-runs dedupe at the store seam.
matchClusterLabelings (core/clusters.ts, pure): a community keeps a
record's label while at least half its core survives in it; both sides
single-successor (a split keeps one name; greedy by overlap, ties to
the newer record).
The namer (src/agent/subagents/cluster-namer.ts): ONE singular cheap LLM
call (new 'labeler' role, glm-flash/low) names every new/changed
community at once, given member names + statements (capped at 12
members per cluster). refreshClusterLabels runs POST-EXTRACTION
(user-decided) from both front doors, fire-and-forget — naming failure
never fails the run; refreshes serialize behind a chain so parallel
imports never mint duplicate labels. import-notes.ts awaits
clusterNamingSettled before exit.
Payload: GraphSnapshot.clusters (>= MIN_CLUSTER_SIZE 3) joins detection
with labelings at read time — claimed clusters carry the record's label
and id; unlabeled ones carry a member-hash id and null label. Detection
is deterministic so no memo in v1 (ms at current scale; revisit at
vault scale).
Tests (372/372): matcher half-rule/split/merge/greedy/tie cases; naming
end-to-end through importNotes (one call, one record, claim on
re-refresh), naming failure non-fatal; snapshot derivation with
labeling join; 11 test pipelines now pin an explicit labeler seam.
detectCommunities(nodes, edges) in src/core/clusters.ts (new dep
graphology-communities-louvain): Louvain modularity over the undirected,
unweighted follows+supports structure (user-decided 2026-09-13). The
random-walk traversal runs on an inlined seeded mulberry32 so the same
graph always yields the same partition; community indices are still
unstable across edits, which is exactly why labelings (phase B) key to
sorted MEMBER SETS. follows+supports between the same pair collapse to
one edge; self-loops and dangling edges are skipped defensively; no
min-size filter here (the >=3 render threshold is a presentation rule).
Tests: two-cliques-plus-bridge split, determinism, singleton/empty
graphs, pair collapse, defensive skips, member sorting. 363/363.
Real-data smoke over the current notes graph: 338 nodes / 211 edges ->
138 communities, modularity 0.95, and the big ones are exactly the
learn-session topics (databases, differential forms, sales, servers,
vim, copulas...). 32 communities clear the >=3 render threshold.
amosblomqvist