451caba1b1db3b6077dbe99bbb20c1250004062f
28
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 amosblomqvist
|
8355fbabdb |
delete the mappings ledger — per-job in-memory resolution + notes fingerprint guard
The persisted (chatId, sourceId)→node table was a shared namespace across independent chunk/batch jobs: each chunk's extractor mints local ids fresh (p1..pN), so later chunks collided with earlier ones — propositions silently skipped as 'already resolved' (coverage checkpointed anyway: permanent silent loss) and edges resolved onto the wrong nodes (the 'everything links to everything' mush). Owner ruling 2026-09-29: mergeFindings resolves THIS job's ids in memory, nothing persists between jobs; cross-job identity is the auditor's linkTo verdicts alone. - contracts: NodeMapping dies; NotesSource.contentFp — sha256 of the text at the last run, the re-import guard (unchanged re-import = TRUE no-op, no model calls, no usage; changed text re-runs; legacy unstamped sources re-run once) - merger: job-local resolution map, in-submit duplicate ids skip once, edges resolve extractor-local ids only - ports/store: appendMapping/readMappings, LEDGER_FILES.mappings, rewriteLedgers.mappings gone (stale data/graph/mappings.jsonl just stops being read) - session-clusters: birthSessionOf is provenance-only — the legacy mapping-line fallback for provenance-empty notes nodes dies - deletion: no mapping rows to strip; stats lose mappingsRemoved - replay-layout batches nodes by first provenance point - graph.test pins the new independence semantics (re-merge re-creates; duplicate ids within one submit resolve once); extraction tests pin per-chunk merges + the fingerprint guard |
||
|
amosblomqvist
|
3d9d20794c |
Auditor submits links ONLY — the edges argument is gone (owner, 2026-09-28 evening)
Hypothesis under test by the owner: coherent extractions turn to mush after merging because the auditor minted too many bridging edges of its own — inference, not identity, and out of the auditor's scope. The graph now connects across sources solely through identified equivalences; relations between propositions come from the extractor alone. - auditSchema drops 'edges' (submit_review = links only); the submit-seam required-field doctrine reworded per tool - AUDITOR_INSTRUCTIONS: the BRIDGING EDGES section dies; deliverable is the identity verdicts only - AuditVerdicts drops edges; ExtractedEdge endpoints are extractor-local ids only; mergeFindings stops consuming verdict edges and its resolve() drops the raw-node-id branch (dead — the extractor never sees graph ids) - extractor untouched: submit_findings keeps propositions + follows/ supports edges, required as before - tests: 17 submit fixtures + 9 typed verdict literals slimmed to links - docs: architecture.md division-of-trust + pipeline diagram updated |
||
|
amosblomqvist
|
1f5e21a674 | teacher-prose-only extraction view: coverage, fingerprints, tokens, and batches derive from the teacher's text blocks alone (owner 2026-09-28) | ||
|
amosblomqvist
|
f11b1664dc |
verdicts-only auditor: submit_review is {links, bridging edges} — the findings echo dies
Owner ruling 2026-09-28: the auditor has no control over the extractor's output and never alters it, so re-emitting the full findings per submit was pure payload cost — the retry collapse the schema slim attacked, halved again. submit_review is now links (local proposition id -> existing graph node; EQUIVALENCE BIAS carried in the linkTo description) + bridging edges ONLY; the merger joins extractor findings + verdicts. mergeFindings(chatId, findings, verdicts, ...) — propositions come from findings verbatim, links join by id, edges emit extractor's then auditor's through one dedup-preserving emitter. The old echo form hid an accidental drop channel: an auditor that omitted a proposition from its echo silently dropped it from the merge. Verdicts-only removes it structurally — extractor output always fully merges (tests updated to full-merge expectations). AuditResult/AuditedProposition -> IdentityLink/AuditVerdicts; audit() logs links/bridges; batch view renders anchorless like transcriptView; auditor prompt rewritten to the verdicts-only deliverable; architecture.md division-of-trust updated. |
||
|
amosblomqvist
|
bd9694b0e4 |
slim the submit schemas: evidence and messages die; stringified-payload heal at the pre-validation seam
OpenRouter log diagnosis (owner, 2026-09-28): extraction time was
dominated by tool-validation retries. The failing call had the whole
propositions array emitted as ONE JSON-STRING ('propositions.0: must
be object'), a truncated placeholder artifact inside it, and
word-salad statements late in the array — payload-size collapse.
Field audit: 'evidence' was never persisted anywhere and its
faithfulness-checking consumer died with the fully-trusted ruling;
'messages' fed messageId granularity that NOTHING downstream reads,
and the model sent [] anyway (chat-level provenance in practice).
'aliases' stays (optional now): the auditor's find tool globs them.
Schema: propositions = {id, name, statement, aliases?}; edges =
{from, to, type}. Prompts lose the evidence/messages doctrine and
[id]-anchor guidance, gain a compact faithfulness line;
transcriptView renders anchorless. The auditor echo instruction
drops the evidence/messages mention, so its payload halves too.
Heal: healStringifiedArrays on the submit tool's prepareArguments
(pi-agent-core's sanctioned pre-TypeBox shim) parses a stringified
propositions/edges back into arrays — the LaTeX-restore doctrine:
heal weak-model wire behavior at the seam. Test drives the real
agent-loop seam with a stringified submit.
|
||
|
amosblomqvist
|
1b50013551 |
extraction cleanup: events ledger dies, markdown source view, auditor-as-linker (2026-09-27 rulings)
The graph becomes raw reasoning chains only. Quiz outcomes and understanding observations are no longer formalized; assessment moves to the future practice module. Events ledger removed end-to-end: LedgerEvent/LedgerEventKind/ UnderstandingLevel/ExtractionAttribution/AuditResult.dropped and snapshot nodeObservations deleted from contracts; quizEvent builder and the events step of mergeFindings gone; appendEvent/readEvents dropped from the port and store; LEDGER_FILES loses events.jsonl with a one-time purge of a stale file next to the nodes ledger; graph-deletion and the delete dialog lose event purge + quiz-answer impact counts; GraphSessionProposition becomes a pure birth list. Whole-session markdown source view: new core/session-markdown.ts is the shared formatting engine (id-anchored speaker blocks, quiz/ recall/question blocks, tool one-liners) — the extractor's prompt view and the future user-facing export render from one engine. Coverage fingerprints deliberately stay on the old line format (content-hash semantics; re-formatting never re-opens covered history). The whole session goes to the extractor — no phase cut. Auditor refocused as the identity linker: never sees the source; explores the graph via new auditor-specific tools in agent/tools/graph.ts (find — glob over name/statement/aliases; list_sessions; list_propositions with full statement text; no timestamps) instead of an injected full-graph dump; equivalence bias — paraphrases merge, only genuinely distinct propositions stay apart; no drop channel, the extractor is fully trusted; may draw bridging edges (typed follows/supports) between existing nodes and the batch, resolved by the merger with chat-scoped message-less provenance. Agentic loop, maxTurns 16. UI: the node detail card gains an edges section above the sessions list (follows-from / supported-by / what-follows / what-it-supports, rendered only when present, click-to-select neighbors); the understanding-note chips are gone. Prober list_propositions reworded to what a session covered — names only, no levels. docs/architecture.md + README updated to the as-built pipeline. Gates: typecheck, 1059/1059, lint, build. |
||
|
amosblomqvist
|
f3003d357a |
Wave D: native AgentEvent wire + client session rewrite (SDK purged)
Server: ui-stream.ts is verbatim pi AgentEvent passthrough (SSE data lines + [DONE]); only app-level events are the loop-level error and turn-settled terminal; onError carries raw provider text (owner ruling). Client: useChat + the AI-SDK transport replaced by components/session-core.ts (SessionModel: 409 identity-retry, optimistic c_ appends, absorbed() adoption, needsContinuation, shared consume for POST + reattach) and the session.ts hook; the golden-master grouping canon moved client-side into core/view.ts (projectView); scanners read the native record (phaseAt3Lh, danglingQuestionOf, graph toolParts, plan-blocks, runner, extraction). Coverage: the native message is the unit (fork 3) with the call+result inseparability bond rule; view-line-less messages (markers, error results) are invisible to coverage and never checkpointed. Deletes: chat-transport, projection, ui-message, their tests (ported to tests/session-core.test.ts), and the ai + @ai-sdk/react dependencies — the Vercel SDK is fully out of the tree. Wire pins in chat-flow rewritten onto the AgentEvent dialect (eventsOf parser). Gates: tsc clean, 994/994 tests, eslint clean, build green; live smoke on PORT=3004 (verbatim wire → native record). |
||
|
amosblomqvist
|
7ab47d4fcb |
Wave B: the native message record — input events, projection, adapter death
The chat record is now pi-native agent history (LhMessage[]), the AI SDK's
UIMessage transcript contract is gone from the server, and every client turn
arrives as ONE explicit input event.
Core (new):
- src/core/native-messages.ts — the LhMessage taxonomy (user/assistant/
toolResult + lhMarker customs: verdict, probe-handoff, subagent-steer,
subagent-abort, plan-block), constructors, and toLlmMessages — the single
model-facing seam (convertToLlm): strips ids, flattens markers to user
text, drops UI-only plan-blocks, synthesizes INTERRUPTED results for
dangling aborted calls, never throws.
- src/core/projection.ts — projectNative: the one-way native → UIMessage
projection (consecutive assistant+toolResult runs group into one
assistant view message; toolResults attach backward by toolCallId;
markers project to data parts; plan-blocks splice after the last
successful plan_init). Every existing consumer reads through it.
- src/core/input-events.ts — the ChatInput union (text/answer/verdict/
continue), parseChatInput validation, and inputAppends (answer targets
the dangling question call or 400s; verdict = optional note user
message + native verdict marker).
Server:
- chat-handler.ts — the turn root takes {chatId, input}; write-ahead
persists input appends + claimed steers before the turn; claimSteers
coalesces the whole queue with the handler's steerDelivered filter as
the dedupe seam; teacher-vs-prober branch on hasProbeDone.
- turn-service.ts — persistTurn stamps ids, drops clean question
placeholder toolResults (the record ends at the unanswered call),
emits plan-block markers idempotently (plan-blocks.ts rewrite), and
persists abort partials.
- ui-stream.ts — PiTurnSpec takes the native base; the wire dialect is
unchanged. jobs.ts steers are native markers (steer_<jobId> dedup ids);
waitForIdle reads danglingQuestionOf. teacher-agent.ts wires
convertToLlm: toLlmMessages. record-adapter.ts DELETED wholesale.
- GET /api/chats/[id] serves messages: projectNative(record.messages).
Client:
- chat-transport.ts — the 409 retry is IDENTITY (re-POST the same event
body after the turn settles — the event was never persisted
pre-acceptance; the merge logic is gone). The transport consumes the
stashed input event via prepareSendMessagesRequest and throws loudly on
a missing stash.
- chat.tsx — every send call site stashes exactly one ChatInput event
(text, answer, verdict, continue) before invoking the SDK send.
Tests: full native migration (chat-flow rewritten to input events and
native record pins; jobs/extraction/demos/graph/usage/plan-blocks/
probe-done/reattach-transport converted; record-adapter.test.ts deleted;
chat-transport tests pin the identity retry). 982/982, typecheck, lint,
build all green.
|
||
|
amosblomqvist
|
73027f49bf |
Cost channel cull: the cluster-naming fold dies — two stages, two channels
Off UI's d651f93 flag (displayCostTotals already isolates the display to extraction + audit; the naming stage row and naming runs were filtered out client-side). This culls the server side of the same ruling. THE STANDING RULING, REVERSED BY ITS OWN PREMISE: NamingRunUsage / readNamingRuns / appendNamingUsage were kept at the wave-7 namer kill ONLY because cost-summary's HISTORICAL naming stage read them. With the fold culled, that premise dies — so the whole ledger read path goes too, following the wave-7 embeddings precedent chain: consumer-less ledger seams are deleted, and the on-disk naming-runs.jsonl stays INERT (real trees keep their historical spend bytes; they are simply not displayed anywhere — the same treatment as historical stages.embedding spend). - core/cost-summary.ts: two channels in (chat + notes front doors), two stages out; RecentRun.kind collapses to "extraction" (labels field gone); CostSummaryInput.naming gone; totalDollars = extraction + audit. - ports/store/contracts/config: appendNamingUsage + readNamingRuns ports, store impls + guards, the NamingRunUsage interface, and LEDGER_FILES.namingRuns die. graphRevision no longer counts the file (a smaller revision value is harmless — reads never compare). - components/graph-view.tsx (mechanical, flagged): the defensive .filter(run.kind !== "naming") guards a kind that no longer exists. - Tests: cost-summary's both-channels fold pins; the naming fixtures and the appendNamingUsage store pin die with the seams. |
||
|
amosblomqvist
|
2e9fd01c0c |
Extraction-UX data layer: the chunk budget becomes a graph setting; the chat view gets a per-session coverage read
Owner-locked extraction-UX redesign (dispatch 2026-09-24), part 1 (data
layer; the UI chunk follows).
THRESHOLD -> SETTING
- config.ts's EXTRACTION.chunkTokens dies; the default moves into
FACTORY_SETTINGS.graph.extractionChunkTokens (4096, the old constant's
value). GraphSettings gains the optional field, Required at resolution
via the standard ?? FACTORY pattern — pre-redesign records resolve the
default cleanly.
- FLOOR 500 (the implementer's call, documented on the field): every run
pays the graph-carrying prompt overhead out of the write channel's
money — a smaller budget fires penny-batches that cost more per
extracted token than the content is worth. The sanitizer CLAMPS up
(bd68486 clamp-never-reject discipline: fractions floor, junk drops);
the settings PUT REJECTS below-floor with the rationale in the message.
No ceiling: a large budget just delays extraction — the user's stated
cost/thoroughness tradeoff, not a misconfiguration to police.
- The trigger (turn-service.triggerExtraction) takes the budget as a
parameter — chat-handler passes effective.graph.extractionChunkTokens,
the fresh per-turn read, same as every other gate.
- The pipeline's BATCH side threads through the same number:
ExtractionPipelineDeps gains an OPTIONAL getChunkTokens getter with the
factory default (Teacher's 24 test fixtures stay untouched; the two
call-site lines inside extraction.ts are the only cross-band edits,
flagged). chat-handler + backfill/import-notes wire the live
effectiveSettings — trigger and batches share one budget, always.
PER-SESSION COVERAGE READ
- sessionCoverage(chatId) (server/chats.ts): the chat view's ring-meter
block — totalMessages/coveredCount/uncoveredCount/coveredTokens/
uncoveredTokens (the same coverageOf derivation /api/graph/coverage
serves, scoped to one chat), running (chat-kind extraction registered
for this id), and lastFoldedAt (the chat's newest coverage row's fold
— a cheap single pass; null before any fold). GET /api/chats/[id]
carries it as coverage: {...}; unknown chat -> null (the route 404s
first).
- /api/graph/coverage (the all-sessions list) compiles untouched — the
per-session read is a scoping of it, not a supersession; the UI's
sidebar/ring chunk decides which surface it consumes.
Tests: factory/sanitizer pins (clamp-up, fraction-floor, junk-drop),
resolution pins (absent -> 4096, stored 2000 wins), settings-body pins
(valid/absent/fraction/below-floor 400/junk 400), sessionCoverage pins
(derived split, newest-fold scoping, cross-session isolation, running via
the registry, null on unknown chat). 893/893.
|
||
|
amosblomqvist
|
2d133f1133 |
Single-node deletion: full death by node id (node interlude, owner-locked)
The remedy for undeletable nodes (preserve-graph orphans from session deletion; notes-import nodes with no chat behind them): DELETE /api/graph/nodes/[id], full-death semantics — the user pointing at the node IS the intent; references from other sessions do NOT preserve it. - core/graph-deletion.ts deleteNodeTraces: the node's every line, every edge touching it, every quiz/observation event naming it, its mapping lines, and its resting position — one synchronous atomic rewriteLedgers call; only the ledgers that changed rewrite (mtimes stay honest). Blast radius is exactly the rows naming the node id; notes source records stay (deleting a proposition is not deleting the file). Derived data self-heals on read: the birth cluster shrinks below MIN_CLUSTER_SIZE and stops rendering. - nodeDeletionImpact (GET /api/graph/nodes/[id]): the confirm dialog's pre-flight facts — edge count, quiz/observation history, referencing sessions. The snapshot's edges + nodeSessions cover part of this, but the raw event history was nowhere on the client — hence the read. - Guards (server/graph.ts deleteGraphNode), decided + documented: NO turn/jobs gates (they never write graph ledgers; the orchestration is synchronous end to end, nothing interleaves). THE EXTRACTION GATE: 409 while ANY pipeline run is registered — a mid-flight run holds pre-deletion findings and could re-reference the dying node. The run is never aborted (its spend belongs to another session — outside this deletion's blast radius; the UI retries once the dock drains. A run started after the deletion is legitimate re-learning: fresh ledgers, fresh node id, and mergeFindings' existingIds guard ignores linkTo ids that no longer exist — no dangling edges, no resurrection. Live-verified on dev data: impact read, full-death stats, honest 404 on repeat, and the file cluster fell 3→2 and unrendered itself. 859/859 tests, typecheck/lint/build green. EOF ) |
||
|
amosblomqvist
|
d1b46132b9 |
Wave 7 close-out: the embedding ledger seam dies; notes birth via mapping lines
Part 1 — the ledger + remnants interlude (master-dispatched, owner ruling 2026-09-23: nothing configures the graph beyond chat models): - The embedding ledger surface is gone: readEmbeddings/writeEmbeddings ports, the EmbeddingsLedger contract, EMBEDDINGS_FILE (config, store impls, graphRevision source list), and graph-deletion's vector-pruning step. On-disk data files stay untouched (inert history). - The extraction embedder seam dies wholesale: ExtractionPipelineDeps drops the embedder dep (chat-handler + backfill/import-notes stubs go with it), the 'embedding' phase leaves ExtractionPhase, the usage capture/persist helpers die, and stages.embedding leaves ExtractionRunUsage (historical records keep their shape — reads are lenient). Extraction is two stages again, like the local-embedder era. - Consumer-less remnants: latestQuizOutcomes + QUIZ_EVENT_KINDS, clusterContentId, matchClusterLabelings + Community + ClusterLabeling, the readClusterLabels/appendClusterLabel port seams + store impls + LEDGER_FILES.clusterLabels. NamingRunUsage/readNamingRuns STAY (standing ruling — the cost pill renders historical naming spend); clusterHue + MIN_CLUSTER_SIZE stay (live). - The 'labeler' role dies: BACKGROUND_MODEL_ROLES, ROLE_SLOT, ModelRole, the catalog pin projections, and the UI roster strings. RolePins keys shrink with the union. The notes birth fallback (owner-approved 2026-09-23, surfaced by the interlude's live smoke: production notes imports create provenance-EMPTY nodes (importNotes passes messages: null — the file's anchor is the mapping line), so the provenance-only birth rule left every imported file unclustered and invisible to the prober's session list, against the locked file-birth ruling. A provenance-empty node now births under its FIRST mapping line's chatId (ledger order); only notes-born nodes ever hit the fallback (chat nodes always carry provenance; a re-encountered node's link-append re-opens the provenance rule). Deletion-safe: mapping lines of deleted chats are stripped alongside their provenance. Live-verified on port 3004: a fresh notes import clusters under its src_ id with the file path as the label, end to end with no embedder. 845/845 tests, typecheck/lint/build green. EOF ) |
||
|
amosblomqvist
|
0fcffdffaf |
Birth-session clustering + the prober graph-tools data seam (wave 7 chunk A)
Embeddings are dying (chunk B removes the machinery); the clusters they
used to carry become the session that created them — derived on read.
- New src/core/session-clusters.ts: birthSessionPartition groups merged
nodes by their FIRST SURVIVING provenance point's chatId (owner-locked
2026-09-23). Deletion is self-healing by construction: stripped-birth
survivors re-attribute to their earliest remaining contributor, and
empty-provenance nodes stay unclustered. Notes imports are birth-sessions
too (owner ruling) — their label is the file path.
- graphSnapshot: clusters = the birth partition filtered >= MIN_CLUSTER_SIZE
(the hull presentation rule), id = session id, label = session name (chat
title / file path; a deleted record renders unlabeled). The persisted
cluster-labelings join dies; no LLM naming pass needed anymore.
- settleGraphLayout derives the same partition from readNodes — drops its
readEmbeddings dep. One cluster concept everywhere.
- Prober data seam (core/graph.ts, pure ports-driven like the snapshot):
graphSessionIndex (every birth session: id, name, chat|file kind,
createdAt/updatedAt, birth-proposition count, recency-ordered) and
graphSessionPropositions (a session's birth propositions with the
extractor's latest understanding level and THAT event's own timestamp —
owner ruling: quiz outcomes are ledger evidence, they never move the
evaluation date). Teacher's tool handlers consume these in-process.
- New agent.graphTools subsection { enabled?: boolean } (owner ruling:
the toggle gates prober tool REGISTRATION — agent capability, agent
section, its own subsection, NOT the tool-toggles map; absent = ON).
Lenient sanitizer, strict settings-body parser, passthrough resolution,
and graphToolsEnabled(effective) = graph.enabled && toggle for Teacher's
registry wiring.
Gates: tsc clean, eslint clean, 923/923 tests, next build compiles.
Live-verified on the worktree dev data: 10 nodes → 2 clusters labeled
with their chat titles; session index + propositions carry levels and
their own timestamps.
|
||
|
amosblomqvist
|
a6338511d3 |
feat(graph): two-mode session deletion — the first destructive ledger operation
DELETE /api/chats/[id]?mode=preserve-graph|delete-nodes (mode required): - preserve-graph: only extraction bookkeeping dies (coverage rows, legacy checkpoints) — graph ledgers byte-untouched - delete-nodes: the chat's ledger traces die. Node survival = surviving provenance point / mapping line (notes imports carry empty provenance) / surviving event. The creator's aliases + provenance are stripped from a surviving node's create line (owner ruling 2026-09-23: name/statement/createdAt are the shared identity); orphan nodes, their edges, events, mappings, positions, embedding vectors and cluster members die; cluster labels stay (immutable) Gates: 404 unknown, 409 turn_in_flight / jobs_running (with jobIds), 503 extraction_winding_down after a 10s abort-and-poll; the purge is one sync block (ledgers → jobs → chat record last). rewriteLedgers is atomic (tmp + rename). graphRevision (max mtime over graph data files) rides the activity feed + the response as the change stamp. Anti-resurrection: the chat pipeline re-checks chat liveness at each merge — a run that slips past the wind-down aborts with nothing merged or checkpointed; its partial spend dies with the chat. |
||
|
amosblomqvist
|
3e263a18cc |
Graph pipeline cost metrics: capture naming spend, fold all channels, cost pill + panel
- Cluster namer bills the money channel now: per-step usage + model captured via onStep, persisted as NamingRunUsage records to data/graph/naming-runs.jsonl (idempotent per startedAt; failed/aborted runs persist too — money spent either way). callUsage promoted to extraction.ts module scope for reuse. - core/cost-summary.ts: the pure fold behind the pill — chat runs, notes-source runs, and naming runs into stage totals (extraction / audit / naming) + newest-first recent runs. Dollars INFERRED read-time via prices.ts, never stored; unpriced models excluded-but-counted (render '—', never a silent zero); tokens always sum. - GET /api/graph/costs (src/server graphCosts composition, thin route). - Graph view: bottom-left cost pill (cumulative dollars headline) + expandable breakdown panel (stage rows with tokens secondary, recent runs with time/title/merged/cost); fetched on mount, refetched when a pipeline run finishes; merged into one bottom-left stack with the running-pipeline pills. - Tests: pure-fold suite, naming capture wiring (full importNotes -> naming path), store idempotency + strict reads. |
||
|
amosblomqvist
|
2678f04e7a |
feat: extraction coverage — message-granular ledger + token-batched runs (d132373 vision)
THE MESSAGE IS THE UNIT OF COVERAGE (vision-extraction-coverage.md): a per-chat append-only ledger (coverage.jsonl) records which messages the pipeline has extracted, by content fingerprint over exactly what the extraction view sees (one shared line builder — core/coverage.ts); the uncovered remainder is DERIVED at read time, never dual-tracked. A quiz answer merging into the asking message re-opens only that message; plan-block data-part attach never re-opens anything. - The trigger (turn-service) replaces every-3-quiz-answers with uncovered-tokens >= EXTRACTION.chunkTokens (4096) — quiz-less sessions extract too, content size is the honest signal. - runExtraction is now chunked: token-budgeted batches of the oldest uncovered consecutive messages (a tool call + result never split), each merged batch checkpointing its messages' coverage atomically — an abort or error loses at most the in-flight batch; merged batches stay covered. - Observation refs are batch-tagged (obs:<batchTag>:<sourceId>): re-runs of the same batch dedupe; a re-opened message's fresh understanding lands as a new event. Fixes the old obs:p1 position-collision. - New API: GET /api/graph/coverage (exact per-session derived coverage + running flags, most-uncovered first) and POST /api/graph/coverage/extract (manual run for one session, 409 while in flight). - Graph view chrome: the coverage button opens a panel with every session's covered/uncovered tokens and a per-session Extract button — the manual answer to aborted runs, errors, and dormant tails. Renders even when the graph is empty: an empty graph with uncovered sessions is exactly when it matters. Polls only while open. - Notes imports keep the whole-file checkpoint path (coverage undefined). - Tests: pure-core coverage suite (11), chunked-pipeline suite (3: multi-batch + no-op, abort-keeps-earlier-batches + remainder re-run, re-open alone + data-part invisibility), panel render suite (5); pendingQuizAnswers and its tests deleted as dead code. 486/486. Live-smoked on real GLM: manual extract of "teach me modal realism" ran one batch, wrote 4 coverage entries, panel flipped to "1.3k covered · complete" with the button disabled, graph refetched (nodes 11), usage persisted with real stats. |
||
|
amosblomqvist
|
f5edc7f0a5 |
Semantic topic clusters: local embeddings replace Louvain
Louvain only ever found the reasoning chains (the graph is disconnected by nature — every connected component became its own cluster) and edge-orphans could never join anything. Clusters are now RELATED TOPICS: propositions grouped by meaning. - Embedder (src/server/embeddings.ts): in-process transformers.js v4 running EmbeddingGemma-300M q8 (MTEB 68.4, ~300MB one-time download), MRL-truncated to 512d, "title: none | text:" prefix on 100% of inputs, L2-normalized so cosine = dot. Model/dim mismatch voids the ledger (full re-embed). Injected at the composition root — the agent layer never imports the server layer (architecture law). - Ledger (data/graph/embeddings.json): model + dim + vectors + the persisted cluster partition. Reclustering runs at the END of every embedding batch (post-merge hook, both front doors) — never per read (the agglomerative pass is ~1.5s at 338 nodes). - Clustering (src/core/semantic-clusters.ts): hand-rolled average-linkage agglomerative over cosine (npm HDBSCAN is an abandoned alpha). Stragglers stay singletons by default; attachStragglers is the all-in toggle. Threshold 0.60 swept on the real graph: 26 topics, 74/90 edge-orphans land in real topics, 10 stragglers. - Consumers (snapshot, settle, namer, replay) read the persisted partition — one cluster concept everywhere; core-survival label matching unchanged (20/26 topics kept labels). - Louvain retired: detectCommunities + graphology* deps deleted (graphology-louvain had leaked into the client bundle since the clusters wave); scripts/bench-graph.ts (dead FA2 bench) removed. - scripts/embed-graph.ts: backfill/cold-start (338 nodes in 150s including the one-time model download). |
||
|
amosblomqvist
|
f0267a5fd9 |
Clusters phases B+C: labeling records, the naming call, snapshot payload
Core-survival labelings (vision section 1, DECIDED 2026-09-13): ClusterLabeling records in a new append-only ledger (data/graph/cluster-labels.jsonl), immutable, id = clu_ + sha256 of the sorted core so re-naming races and re-runs dedupe at the store seam. matchClusterLabelings (core/clusters.ts, pure): a community keeps a record's label while at least half its core survives in it; both sides single-successor (a split keeps one name; greedy by overlap, ties to the newer record). The namer (src/agent/subagents/cluster-namer.ts): ONE singular cheap LLM call (new 'labeler' role, glm-flash/low) names every new/changed community at once, given member names + statements (capped at 12 members per cluster). refreshClusterLabels runs POST-EXTRACTION (user-decided) from both front doors, fire-and-forget — naming failure never fails the run; refreshes serialize behind a chain so parallel imports never mint duplicate labels. import-notes.ts awaits clusterNamingSettled before exit. Payload: GraphSnapshot.clusters (>= MIN_CLUSTER_SIZE 3) joins detection with labelings at read time — claimed clusters carry the record's label and id; unlabeled ones carry a member-hash id and null label. Detection is deterministic so no memo in v1 (ms at current scale; revisit at vault scale). Tests (372/372): matcher half-rule/split/merge/greedy/tie cases; naming end-to-end through importNotes (one call, one record, claim on re-refresh), naming failure non-fatal; snapshot derivation with labeling join; 11 test pipelines now pin an explicit labeler seam. |
||
|
amosblomqvist
|
543cf33789 |
Notes front door: format-agnostic import into the graph pipeline
The same extractor→auditor→merger pipeline now runs over raw personal
notes in arbitrary formats (vision §1, DECIDED 2026-09-13):
- NotesSource records (data/graph/sources.json, deterministic id from
path hash → idempotent re-imports) play the chatId role in mappings,
checkpoints, and observation events; usage lands on the source record
(appendSourceUsage, idempotent per startedAt, kept across upserts).
- mergeFindings is chat-store-free: callers pass MergeSource
{messages, foldedAt} — sessions pass the chat record, notes pass
messages:null (no quiz-event rebuild; a deterministic rebuild needs
structured quiz tool parts).
- Second extractor prompt (NOTES_EXTRACTOR_INSTRUCTIONS) for general-
purpose extraction; understanding judges the AUTHOR's grasp. Auditor
wording input-agnostic. extract/audit share one generic runSubmitAgent.
- graphSnapshot.fileSources (srcId→filePath); notes sources render as
static path entries in the detail card (FileTextIcon, never a link).
- One shared pipeline instance (exported from chat-handler): same
single-flight registry, live status chips (label = file path), usage
accounting, and true abort for both front doors.
- Transport: POST /api/graph/import (multipart, 3-wide worker pool) +
Import notes… button in the graph view; scripts/import-notes.ts for
batch runs (concurrency 4).
- real-data-extraction/ gitignored (raw import sources never committed).
355/355 tests; typecheck, lint, build green.
|
||
|
amosblomqvist
|
e944afebfc |
Understanding levels: closed five-word scale, schema-enforced
The extractor's per-session understanding notes were free-form strings;
the model freely appended em-dash justifications ('shaky — the learner
could state the headline distinction but calls the machinery behind it
foggy'), which overflowed the detail card's session rows and squeezed
the session names.
The note becomes a CLOSED named-word scale — solid / shaky / accepted /
resisted / not yet engaged (DECIDED 2026-09-08, no numbers) — enforced
by the zod enum in propositionSchema: a wrong-shaped note is rejected
at parse time and the agent retries. Justifications are dropped, not
stored (user decision: not necessary). Extractor and auditor
instructions updated to demand exactly one word.
Renderer: the session row keeps the session NAME as the main thing
(flex-1, truncates) and the level chip is bounded (max-w-28,
whitespace-nowrap, truncate) — overflow impossible even for legacy
prose notes in old ledgers.
Vision doc: §1 extractor bullet + event-ledger line updated; state
model line notes the five-class derivation was removed; amendments
table row added.
|
||
|
amosblomqvist
|
0fccbbe2ad |
Detail card: per-session understanding notes next to the session list
The extractor's per-proposition understanding observations (named-word scale: solid/shaky/accepted/...) were ledger data since the redesign, with no surface. This exposes them at read time and shows each session's note as a small chip on its session row in the node detail card — the per-session confidence history the user asked for, one observation per (node, chat), latest wins within a chat. - GraphSnapshot.nodeObservations: nodeId -> chatId -> note, built in graphSnapshot() from observation events (pure read-time join). - Sessions list rows: link + observation chip. - Session-relative judgments stay session-scoped: they render next to their session, never as node state (the state model stays pending). |
||
|
amosblomqvist
|
54f88f37e1 |
Strip the transitional node-state classes (2026-09-08)
The five-class NodeStateClass (taught/known/missed/dont-know/candidate) was transitional scaffelage from the 2026-09-02 redesign, deriving state by latest-quiz-wins — exactly what the philosophy rejects: understanding is inferred at read time from the real state model (still pending ideation), never encoded as quiz-derived classes. - contracts: NodeStateClass deleted; GraphSnapshot no longer carries nodeStates (objective graph + sessions only). - core: deriveNodeStates replaced by latestQuizOutcomes — the raw event fact (nodeId -> latest quiz kind), consumed ONLY by the teacher's graph-as-context selection, never by the home renderer. - graph-context: hotspot selection re-keyed to outcomes — groups are now 'Hotspots (last answer wrong or don-t-know) / Answered correctly / Never probed'. Same diagnostic selection, no state taxonomy. - graph-view: nodes render uniformly (accent color, size 8); the panel is search-only; the detail card's state badge is gone. Quiz outcomes remain ledger data for the future state model. Verified live (headless): uniform discs (single color/size, zero state pixels), search-only panel, card = name + statement + sessions, no console errors. |
||
|
amosblomqvist
|
bdd9cac2b7 |
Node schema pivot: short name (label) + full-precision statement; description dropped
With real extracted data, full-sentence statements as canvas labels proved unworkable: 10-line label blocks, illegible overprint clusters, bottom-edge clipping. Per the user's 2026-09-08 decision: - GraphNode/ExtractedProposition gain `name` — a very short, glanceable label (glossary-headword style: "Will to power"), presentational only: not unique, not identity, never precise. Set at creation; link-appends never change it (first line defines; labels stay stable across sessions). - `statement` keeps its full precision and becomes the identity anchor; the detail card shows it under the name. `description` is dropped. - Extractor coins the name (freshest session context); auditor carries it through on create, rewording only clearly unglanceable names; linked propositions keep the existing node's name. - Renderer: canvas label = name; search matches name + statement; detail card = name (bold) + statement (muted). - Store guard requires name (pre-pivot node lines are skipped as legacy); vision.md node-schema bullet + amendments table updated (2026-09-08). |
||
|
amosblomqvist
|
b07c907ffd |
Graph redesign: true propositions, follows/supports, auditor-owned identity (vision §1 minimal implementation)
The minimal implementation of the 2026-09-02 graph vision (docs/vision.md §1):
- contracts/graph.ts: every node is a TRUE proposition (statement is both
rendering label and identity key; required description; no kind, no name).
Edges are follows (derivable) / supports (grounds without entailing) with
message-granular provenance points {chatId, messageId}. Events reduce to
correct/miss/dont-know plus observation (understanding notes); no taught
events, no misconception kind.
- extraction.ts: extractor is TREE-BLIND (transcript only, no plan tree) and
free-form; auditor sees findings + the current graph, owns faithfulness and
identity (link-or-create verdict per proposition, recorded as linkTo);
transcriptView tags every line with its message id (provenance anchors).
- core/graph.ts merger: applies verdicts deterministically — linkTo appends
a link line (phrasings join aliases, provenance unions; reads merge by id),
absent creates; NO string matching anywhere. Quiz events rebuilt from tool
parts with attribution-only nodeId resolution; understanding notes become
observation events (obs:<sourceId>); mapping lines are the verdict audit
trail and make re-merges idempotent.
- deriveNodeStates: transitional — latest quiz event wins (known/missed/
dont-know, else candidate); taught unreachable until the state-model
ideation lands.
- graph-context.ts + 50-graph-context.md: follows chains, hotspot grouping,
state vocabulary updated for the new event kinds.
- graph-view.tsx: node label is the statement; detail shows statement +
description.
- tests: merger verdicts (link, no-string-match, cross-chat assertions),
tree-blind pipeline end-to-end, follows vocabulary in context tests.
276/276 green.
|
||
|
Amos
|
2e4e43b8c4 |
Graph nodes gain a name; nodes are objective facts (no learner/session references)
Schema: GraphNode + ExtractedProposition carry 'name' — a short 2-6 word header, the rendering handle. Node theory made explicit: a node is an objective world-facing fact; the learner's relationship to it lives entirely in the event ledger (state labels are derived, never stored). Extractor + auditor instructions enforce it — misconceptions state the wrong idea itself as a claim, never 'the learner believes Y'. Legacy ledger lines without a name derive one from the statement on read. graph-view: labels use the name; the detail panel shows name as header and statement as body. |
||
|
Amos
|
af90fadcda |
Extraction pipeline: extractor → auditor → merger — the sole graph writer (stage 5)
The knowledge graph's write channel is now the pivot's target
architecture (vision §1 construction):
- src/agent/subagents/extraction.ts — the pipeline factory
(createExtractionPipeline({ chatStore, graphStore, extractor?,
auditor? }); models env LH_EXTRACT_MODEL ?? LH_MODEL):
* transcriptView(): the compact view both LLM stages see — final plan
tree with node ids, conversation verbatim, QUIZ lines with outcomes,
mark_done lines; job tool parts are one-liners, plan ops fold into
the tree line.
* Extractor: transcript → SessionFindings (propositions incl.
misconceptions and prose-only supports edges, attribution for
UNTAGGED probes, all with verbatim evidence) via one zod-validated
submit_findings call.
* Auditor: transcript + findings → the verified subset + drops (the
faithfulness net) via submit_review.
- src/core/graph.ts — emitChat/foldStaleChats/replayFinalTree retire;
mergeFindings() replaces them as the pipeline's deterministic apply
stage: exact/alias identity resolution (one mapping line per (chat,
graph node)), edge dedupe, and event rebuild FROM THE TRANSCRIPT's tool
parts — kinds/timestamps/refs/notes deterministic, the LLM's only event
say being WHICH proposition an untagged probe tested (retroactive —
the vision's open question #3, resolved structurally: tagged events
resolve deterministically; attribution rescues untagged and
tagged-but-unmapped ones; unattributable land null + question text).
pendingQuizAnswers() is the trigger helper.
- Trigger (turn-service.ts triggerExtraction, replacing the old
foldTurnIntoGraph body): once a chat accumulates
EXTRACTION.everyNQuizAnswers (LH_EXTRACT_N, default 3) new answered
quizzes, fire the pipeline async — single-flight per chat, failure
checkpoints nothing and re-runs on the next trigger. Deliberately a v1
placeholder isolated in ONE seam, trigger-agnostic pipeline.
- GET /api/graph is now a pure read (foldStaleChats gone).
- NodeMapping.treeNodeId → sourceId (the mapping key is now the writer's
local proposition id — tree node when tree-derived); store strict-read
schema updated. Local ledger data regenerates through the pipeline.
- Tests: graph.test.ts rewritten around mergeFindings (identity,
attribution paths incl. ghost-rescue, idempotency, cross-session
provenance, trigger counting), graph-context.test.ts merge section
converted to findings-driven, NEW extraction.test.ts — transcriptView
units + the full pipeline with mock models (audit drops, failure
checkpoints nothing, single-flight). 195/195, tsc/eslint/build clean.
Probe attribution (vision open question): probes attach retroactively by
the extractor against the whole transcript; a pre-tree probe maps to a
proposition the tree only articulates later. Fuzzy/embedding merge
adjudication stays deferred (conservative exact/alias only).
|
||
|
Amos
|
1167157c23 |
Storage ports: ChatStore/GraphStore/JobStore + adapters + core inversion (stage 3)
Per-entity semantic ports (src/core/ports.ts), one adapter family
(src/store/*, fs JSON/JSONL), exactly one composition point
(src/server/stores.ts). Dependency direction is now machine-checked:
adapters implement core interfaces; core never imports an adapter.
The inversion:
- core/graph.ts (emitChat/getGraph/graphSnapshot/foldStaleChats) is now
pure — it takes the storage ports as explicit parameters (GraphDeps).
No store imports; newGraphNodeId is local; node id format unchanged.
- core/graph-context.ts buildGraphContext() no longer defaults to the
live ledgers: the chat-handler hands in getGraph(graphStore) per turn.
- store adapters export exactly one object each implementing their port
(chatStore/graphStore/jobStore); bodies otherwise verbatim.
NodeMapping + SessionCheckpoint moved store/graph -> core/contracts
(they ARE contract shapes); CreateJobInit/JobPatch live in ports.
- titleFromPrompt moved store/chats -> core/contracts (pure helper,
planNodeStatus precedent). Store modules now contain persistence only.
- server/stores.ts is the composition root; server/{chats,graph,jobs}.ts
and chat-handler wire from it; agent (plan tools, sub-agent runner)
call the port objects. The shell surface (@/src/server exports) is
unchanged.
Lint law extended: the whole of src/core is now a purity zone (was a
file glob with a graph/graph-context exemption). Verified it fires on
core->store imports; tsc clean, 190/190 tests, eslint clean, prod build
clean (prerender 6/6). Tests pass deps explicitly — graph.test uses the
same fs-adapter pair stores.ts wires in production; graph-context.test
keeps its in-memory ledgers as a GraphStore implementation.
|
||
|
Amos
|
8728573253 | W4: graph store + emission + home graph view |