cf0ec4a2f014bd8e1dbac21609741c5f4decedba
8
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 amosblomqvist
|
8355fbabdb |
delete the mappings ledger — per-job in-memory resolution + notes fingerprint guard
The persisted (chatId, sourceId)→node table was a shared namespace across independent chunk/batch jobs: each chunk's extractor mints local ids fresh (p1..pN), so later chunks collided with earlier ones — propositions silently skipped as 'already resolved' (coverage checkpointed anyway: permanent silent loss) and edges resolved onto the wrong nodes (the 'everything links to everything' mush). Owner ruling 2026-09-29: mergeFindings resolves THIS job's ids in memory, nothing persists between jobs; cross-job identity is the auditor's linkTo verdicts alone. - contracts: NodeMapping dies; NotesSource.contentFp — sha256 of the text at the last run, the re-import guard (unchanged re-import = TRUE no-op, no model calls, no usage; changed text re-runs; legacy unstamped sources re-run once) - merger: job-local resolution map, in-submit duplicate ids skip once, edges resolve extractor-local ids only - ports/store: appendMapping/readMappings, LEDGER_FILES.mappings, rewriteLedgers.mappings gone (stale data/graph/mappings.jsonl just stops being read) - session-clusters: birthSessionOf is provenance-only — the legacy mapping-line fallback for provenance-empty notes nodes dies - deletion: no mapping rows to strip; stats lose mappingsRemoved - replay-layout batches nodes by first provenance point - graph.test pins the new independence semantics (re-merge re-creates; duplicate ids within one submit resolve once); extraction tests pin per-chunk merges + the fingerprint guard |
||
|
amosblomqvist
|
d1b46132b9 |
Wave 7 close-out: the embedding ledger seam dies; notes birth via mapping lines
Part 1 — the ledger + remnants interlude (master-dispatched, owner ruling 2026-09-23: nothing configures the graph beyond chat models): - The embedding ledger surface is gone: readEmbeddings/writeEmbeddings ports, the EmbeddingsLedger contract, EMBEDDINGS_FILE (config, store impls, graphRevision source list), and graph-deletion's vector-pruning step. On-disk data files stay untouched (inert history). - The extraction embedder seam dies wholesale: ExtractionPipelineDeps drops the embedder dep (chat-handler + backfill/import-notes stubs go with it), the 'embedding' phase leaves ExtractionPhase, the usage capture/persist helpers die, and stages.embedding leaves ExtractionRunUsage (historical records keep their shape — reads are lenient). Extraction is two stages again, like the local-embedder era. - Consumer-less remnants: latestQuizOutcomes + QUIZ_EVENT_KINDS, clusterContentId, matchClusterLabelings + Community + ClusterLabeling, the readClusterLabels/appendClusterLabel port seams + store impls + LEDGER_FILES.clusterLabels. NamingRunUsage/readNamingRuns STAY (standing ruling — the cost pill renders historical naming spend); clusterHue + MIN_CLUSTER_SIZE stay (live). - The 'labeler' role dies: BACKGROUND_MODEL_ROLES, ROLE_SLOT, ModelRole, the catalog pin projections, and the UI roster strings. RolePins keys shrink with the union. The notes birth fallback (owner-approved 2026-09-23, surfaced by the interlude's live smoke: production notes imports create provenance-EMPTY nodes (importNotes passes messages: null — the file's anchor is the mapping line), so the provenance-only birth rule left every imported file unclustered and invisible to the prober's session list, against the locked file-birth ruling. A provenance-empty node now births under its FIRST mapping line's chatId (ledger order); only notes-born nodes ever hit the fallback (chat nodes always carry provenance; a re-encountered node's link-append re-opens the provenance rule). Deletion-safe: mapping lines of deleted chats are stripped alongside their provenance. Live-verified on port 3004: a fresh notes import clusters under its src_ id with the file path as the label, end to end with no embedder. 845/845 tests, typecheck/lint/build green. EOF ) |
||
|
amosblomqvist
|
0fcffdffaf |
Birth-session clustering + the prober graph-tools data seam (wave 7 chunk A)
Embeddings are dying (chunk B removes the machinery); the clusters they
used to carry become the session that created them — derived on read.
- New src/core/session-clusters.ts: birthSessionPartition groups merged
nodes by their FIRST SURVIVING provenance point's chatId (owner-locked
2026-09-23). Deletion is self-healing by construction: stripped-birth
survivors re-attribute to their earliest remaining contributor, and
empty-provenance nodes stay unclustered. Notes imports are birth-sessions
too (owner ruling) — their label is the file path.
- graphSnapshot: clusters = the birth partition filtered >= MIN_CLUSTER_SIZE
(the hull presentation rule), id = session id, label = session name (chat
title / file path; a deleted record renders unlabeled). The persisted
cluster-labelings join dies; no LLM naming pass needed anymore.
- settleGraphLayout derives the same partition from readNodes — drops its
readEmbeddings dep. One cluster concept everywhere.
- Prober data seam (core/graph.ts, pure ports-driven like the snapshot):
graphSessionIndex (every birth session: id, name, chat|file kind,
createdAt/updatedAt, birth-proposition count, recency-ordered) and
graphSessionPropositions (a session's birth propositions with the
extractor's latest understanding level and THAT event's own timestamp —
owner ruling: quiz outcomes are ledger evidence, they never move the
evaluation date). Teacher's tool handlers consume these in-process.
- New agent.graphTools subsection { enabled?: boolean } (owner ruling:
the toggle gates prober tool REGISTRATION — agent capability, agent
section, its own subsection, NOT the tool-toggles map; absent = ON).
Lenient sanitizer, strict settings-body parser, passthrough resolution,
and graphToolsEnabled(effective) = graph.enabled && toggle for Teacher's
registry wiring.
Gates: tsc clean, eslint clean, 923/923 tests, next build compiles.
Live-verified on the worktree dev data: 10 nodes → 2 clusters labeled
with their chat titles; session index + propositions carry levels and
their own timestamps.
|
||
|
amosblomqvist
|
652a25e382 |
Orphan discs in regions + orphan exemption from depth anchoring
Two follow-ups on the semantic-cluster wave, both user-reported: 1. Region coloring: edge-only dilation (the anti-double-counting rule from the Louvain era) left 114 of 338 cluster members completely uncolored — semantic clusters contain members that share no intra-cluster edge with anyone (90 edge-orphans + isolated micro-topics), and 20 of 45 clusters painted nothing at all. Fix: a disc (radius = dilation, same as a round cap) for exactly the members no intra-cluster edge stroke reached. Every member is painted by its edges' caps OR one disc, never both — the no-double-counting rule survives. Verified live: all 74 orphans-in-clusters sit on tinted region pixels. 2. Depth modes (ORPHAN RULE): a node with no edges has no dependency structure, so depth-0 stamping put 90 unconnected propositions on the vertical baseline / radial base ring — meaning they don't have. Orphans are now exempt from the depth force and get the regular view's unprojected forces instead (full center gravity + full cluster pull). Structured-node projections unchanged (x-only gravity in vertical, none in radial; tangential-only cluster pull in radial). Gravity in depth modes is a custom force with EXACTLY d3's forceX/forceY math (strength × alpha — the first draft forgot the alpha and shifted the settle). Verified live: radial orphans follow their topics instead of the base ring; vertical orphans spread 1006 world units (6/90 near the baseline band vs 90/90 before). |
||
|
amosblomqvist
|
f5edc7f0a5 |
Semantic topic clusters: local embeddings replace Louvain
Louvain only ever found the reasoning chains (the graph is disconnected by nature — every connected component became its own cluster) and edge-orphans could never join anything. Clusters are now RELATED TOPICS: propositions grouped by meaning. - Embedder (src/server/embeddings.ts): in-process transformers.js v4 running EmbeddingGemma-300M q8 (MTEB 68.4, ~300MB one-time download), MRL-truncated to 512d, "title: none | text:" prefix on 100% of inputs, L2-normalized so cosine = dot. Model/dim mismatch voids the ledger (full re-embed). Injected at the composition root — the agent layer never imports the server layer (architecture law). - Ledger (data/graph/embeddings.json): model + dim + vectors + the persisted cluster partition. Reclustering runs at the END of every embedding batch (post-merge hook, both front doors) — never per read (the agglomerative pass is ~1.5s at 338 nodes). - Clustering (src/core/semantic-clusters.ts): hand-rolled average-linkage agglomerative over cosine (npm HDBSCAN is an abandoned alpha). Stragglers stay singletons by default; attachStragglers is the all-in toggle. Threshold 0.60 swept on the real graph: 26 topics, 74/90 edge-orphans land in real topics, 10 stragglers. - Consumers (snapshot, settle, namer, replay) read the persisted partition — one cluster concept everywhere; core-survival label matching unchanged (20/26 topics kept labels). - Louvain retired: detectCommunities + graphology* deps deleted (graphology-louvain had leaked into the client bundle since the clusters wave); scripts/bench-graph.ts (dead FA2 bench) removed. - scripts/embed-graph.ts: backfill/cold-start (338 nodes in 150s including the one-time model download). |
||
|
amosblomqvist
|
3a1c18b9de |
Inference-depth anchoring playground: vertical + radial modes
The graph's follows/supports edges encode derivation (premise -> conclusion); the layout can now express that rank spatially, as a session-only playground toggle (nothing promoted yet): - inferenceDepths: longest chain from a base node, CYCLE-SAFE by SCC condensation (Tarjan, iterative) — the DAG property is typical, not guaranteed; cycle members share their component's depth (mutual support = equal footing). O(V+E), pure, deterministic. - DepthAnchor force: per-node target scalar — vertical mode rows (baseline at the anchor centroid, arrows up), radial mode rings (foundations center, outer boundary = edge of knowledge). The rank is computed, not simulated: pairwise up-forces can't give baselines. - Mode projections: gravity x-only in vertical / dropped in radial; cluster pull x-only in vertical / tangential-only in radial; the anchor re-blend preserves user arrangement in the FREE dimension only (x / angle) — meaning owns the constrained axis, memory owns the free one. - setPhysicsTuning reheats (alpha 1): toggling a mode re-forms the graph live and commits the new rest state; drag release is springy along the constrained axis (node returns toward its level/ring). - Sweep on the real graph: strength 0.8 = 215/216 arrows correct with organic sag; 1.5 = crisp rows. 174 base nodes, max chain depth 7, zero cycles in current data. replay-layout --depth=<mode> bakes a mode into the ledger for rest-state experiments. |
||
|
amosblomqvist
|
975a78a304 |
Cluster gravity: per-cluster centroid pull force in the engine
The 'virtual node' realized as a centroid force: two allocation-free O(n) passes per tick (membership resolved once at sim creation; Float64Array sums). Measured indistinguishable from force-off at 338 nodes (271ms vs 272ms per 400-tick settle). - createGraphSimulation: SimOptions.clusterPull + linkStrength + velocityDecay (physics controls plumbing); degree-scaled link stiffness so linkStrength=1 is exactly d3's default - settleGraphLayout split into layout-settle.ts (it computes the Louvain partition for the pull); layout.ts stays browser-barrel-pure - barrel no longer re-exports clusters.ts: graphology + louvain are fully out of the client bundle (verified against built chunks) - replay passes the per-batch partition; CLUSTER_PULL_STRENGTH=0.15 tuned by sweep: cluster spread 63->36 world units, centroid separation and total span unchanged (no blob merge, no collapse) |
||
|
amosblomqvist
|
0c14ea425d |
Option A: incremental anchored force layout, persisted server-side
The layout now EVOLVES instead of reshuffling on every extraction. A pure
core module (src/core/layout.ts) warm-starts forceAtlas2 from the positions
ledger (data/graph/positions.json, read-modify-write like sources.json,
updatedAt stamp for future live-physics conflict handling), re-blends every
previously-positioned node toward its anchor each iteration (retention 0.85
- old placements stay recognizable, strong forces still win), seeds new
nodes near their positioned neighbors, and seeds orphans on the rim outside
the core so they read as unconnected knowledge.
- Port/store: GraphStore.readPositions/writePositions; snapshot carries
positions when the ledger exists (optional - fresh graphs unchanged).
- Extraction pipeline settles synchronously after every merge (chat and
notes paths), before usage persist - the next /api/graph fetch already
reflects new nodes.
- View: the ledger IS the layout; the local one-shot FA2 pass survives
only as a fallback when no ledger exists.
- scripts/replay-layout.ts: simulates organic growth by replaying the
current 338-node graph one imported file at a time (real import order,
edges activate when both endpoints exist). Real run: 338 positions in
0.3s, 90-degree-0 orphans collecting at mean r=338 vs connected r=194.
- Also restores the cluster region fills (hull background colors) that
|