Model slots wave (2026-09-20), chunk B:
- settings: new `models` section {strong, cheap, vision} slots (each
field independent, same shape as a role override); ROLE_SLOT maps all
nine roles to tiers (strong=prober+teacher, cheap=researcher/
demoBuilder/extractor/auditor/labeler, vision=visualMaker+pdfReader);
resolution chain per role: override → slot → factory pin.
- config: ModelSpec gains provider (all pins openrouter, the BYOK
default); MODEL_PINS in stores.ts now spans all nine roles (the strong
slot's factory rep IS the prober pin).
- engine: piModels() registers EVERY builtin provider catalog;
piEngineForSpec(resolved spec) is the new engine factory (pi-ai
resolves per-provider keys from env itself); piEngineFor stays as the
config-pin fallback. chat-handler resolves the conversation chain
(chat override → strong slot → pin) and injects engineFor into the job
runner, closing the old gap where per-role overrides never reached
job engines.
- /api/models: models federate ONLY keyed providers (providers surface
with keyPresent, env-wins, fresh per call); slots (resolved tier pins)
join rolePins; catalogDefaultsFor inherits the strong slot; the models
settings section gets the same catalog gate as chat.
- pricing: PRICES/CONTEXT_LIMITS uniformly keyed provider:id
(regenerated from ALL pi-ai catalogs, 1354 keys); usage stamps
(TurnUsage, JobUsage, extraction stages) switch to provider:id —
owner decision: no legacy compat, old stamps render unpriced.
- Cluster namer bills the money channel now: per-step usage + model
captured via onStep, persisted as NamingRunUsage records to
data/graph/naming-runs.jsonl (idempotent per startedAt; failed/aborted
runs persist too — money spent either way). callUsage promoted to
extraction.ts module scope for reuse.
- core/cost-summary.ts: the pure fold behind the pill — chat runs,
notes-source runs, and naming runs into stage totals (extraction /
audit / naming) + newest-first recent runs. Dollars INFERRED read-time
via prices.ts, never stored; unpriced models excluded-but-counted
(render '—', never a silent zero); tokens always sum.
- GET /api/graph/costs (src/server graphCosts composition, thin route).
- Graph view: bottom-left cost pill (cumulative dollars headline) +
expandable breakdown panel (stage rows with tokens secondary, recent
runs with time/title/merged/cost); fetched on mount, refetched when a
pipeline run finishes; merged into one bottom-left stack with the
running-pipeline pills.
- Tests: pure-fold suite, naming capture wiring (full importNotes ->
naming path), store idempotency + strict reads.
Live-smoke follow-up the user approved (2026-09-19): the GEB read showed
"78 pages consulted" when the reader genuinely READ ~12 - pdf_search hit
snippets inflated the count. PdfResult now carries both:
- pagesRead: extract/render output entered the loop - the honest count
the badge and brief footer show ("read N pages", "Pages read: ...").
- pagesConsulted: every page whose content entered the loop, search-hit
context lines included - unchanged, still the citation audit set (the
reader's instructions still say "Pages consulted: N, N, ..." and the
footer still lists it when it exceeds the read set).
The unread-finish gate deliberately stays on CONSULTED: search hits are
genuine consultation (anti-fabrication duty), so a search-only brief is
honest and must deliver - pagesRead is a reporting refinement, not a
delivery gate. Pinned by the search-only faux test (read [], consulted
[1], delivered).
UI tolerates pre-split persisted records via ?? fallbacks (old steers
lack pagesRead).
Live dispatch smoke against the Turbopack dev server (master's ask,
2026-09-19) caught two things unit tests cannot - they run unbundled:
- Bundling pdfjs-dist broke its node fake-worker: the chunk-relative
pdf.worker.mjs import 404s inside .next/dev/server/chunks. Fix:
serverExternalPackages + tracing includes for legacy/ + standard_fonts/
(next.config.ts - a clearly-marked seam, same precedent as bcdcd51; the
standalone output now ships the worker and font files, verified).
- Real-world PDFs warned about missing standardFontDataUrl (non-embedded
standard fonts silently fall back) and flooded the log with JBIG2
warnings on scanned files. Fix: standardFontsDir() + verbosity 0 in
both the node ops and the headless canvas shim.
Verified live end-to-end on GEB (802pp, text layer): teacher dispatch,
12-step read with vision renders, brief with accurate page citations,
wake turn folding the resolution (Chapter IX, pp. 268-270) into the
lesson. Scanned/DRM Polya correctly refuses to fabricate citations.
Learner source files, on the teacher's own terms (nothing auto-injects):
- read tool (src/agent/tools/read.ts): pi-read port — path/offset/limit,
2000-line/50KB truncation with use-offset continuation notices, image
files (jpg/png/gif/webp/bmp) returned as native image content parts
(new LhToolImageResult escape hatch in lh-tool.ts). Always-on in both
the prober and teacher registries (user-decided 2026-09).
- pdf-reader sub-agent (JobKind 'pdf', toggleable, ack-and-steer): triage
via pdf_info (pdfjs text layer: per-page text/image/math-density),
targeted pdf_extract, regex pdf_search (hits count as consultation),
and pdf_render — pages painted by pdfjs onto a canvas shim in headless
Chrome (the pdfium viewer doesn't paint headless), screenshot fed back
image-in-the-loop (visual-maker idiom, providerSafePng applied). The
unread-finish gate bounces a zero-consultation brief once. Teacher gets
a text brief with page citations only; renders never leave the loop.
- MODELS.pdfReader = qwen/qwen3.8-flash (user override 2026-09-19: the
look-at-renders workload wants the vision-proven model; catalog listed
glm-5.3-flash as image-capable but qwen is the known-good daily driver).
- UI: PdfCompleteBadge + PdfBriefCard (steer-card, job-card), dispatch
card + activity-dock kind entries (type-forced, flagged for UI review).
- Prompts: 'Learner source files' section in teacher/30-jobs.md; read
line in prober/10-probe-doctrine.md. Doctrine rides tool descriptions
per the self-contained-tools principle.
THE MESSAGE IS THE UNIT OF COVERAGE (vision-extraction-coverage.md): a
per-chat append-only ledger (coverage.jsonl) records which messages the
pipeline has extracted, by content fingerprint over exactly what the
extraction view sees (one shared line builder — core/coverage.ts); the
uncovered remainder is DERIVED at read time, never dual-tracked. A quiz
answer merging into the asking message re-opens only that message;
plan-block data-part attach never re-opens anything.
- The trigger (turn-service) replaces every-3-quiz-answers with
uncovered-tokens >= EXTRACTION.chunkTokens (4096) — quiz-less sessions
extract too, content size is the honest signal.
- runExtraction is now chunked: token-budgeted batches of the oldest
uncovered consecutive messages (a tool call + result never split), each
merged batch checkpointing its messages' coverage atomically — an abort
or error loses at most the in-flight batch; merged batches stay covered.
- Observation refs are batch-tagged (obs:<batchTag>:<sourceId>): re-runs
of the same batch dedupe; a re-opened message's fresh understanding
lands as a new event. Fixes the old obs:p1 position-collision.
- New API: GET /api/graph/coverage (exact per-session derived coverage +
running flags, most-uncovered first) and POST /api/graph/coverage/extract
(manual run for one session, 409 while in flight).
- Graph view chrome: the coverage button opens a panel with every
session's covered/uncovered tokens and a per-session Extract button —
the manual answer to aborted runs, errors, and dormant tails. Renders
even when the graph is empty: an empty graph with uncovered sessions is
exactly when it matters. Polls only while open.
- Notes imports keep the whole-file checkpoint path (coverage undefined).
- Tests: pure-core coverage suite (11), chunked-pipeline suite (3:
multi-batch + no-op, abort-keeps-earlier-batches + remainder re-run,
re-open alone + data-part invisibility), panel render suite (5);
pendingQuizAnswers and its tests deleted as dead code. 486/486.
Live-smoked on real GLM: manual extract of "teach me modal realism" ran
one batch, wrote 4 coverage entries, panel flipped to "1.3k covered ·
complete" with the button disabled, graph refetched (nodes 11), usage
persisted with real stats.
One shot per chat, fired at the first turn's onFinish: the session's
topic (its only typed message) is embedded as an EmbeddingGemma retrieval
query; the semantically closest graph propositions are selected under the
session's token budget (pure budget, no similarity threshold — the
decided doctrine) and delivered as a marker-only steer through the wake
pipeline, queuing behind pending questions like every sub-agent result.
- contracts: ChatSettings.graphContextTokens (default 1024, max 4096,
step 128, 0 = off); GraphContextData + graphContextText; the
graph-context JobKind + buildGraphContextSteer (id steer_ctx_<chatId>)
- core/graph-context.ts: brute-force cosine top-K, greedy chars/4 fill
under budget, most-recent UnderstandingLevel fold, count cap 64,
parse-don't-cast at the marker seam
- server: embedQuery (role-asymmetric query format, same warm singleton
— vector-space parity by construction); the runner (startGraphContext-
Injection one-shot gates; runGraphContextJob validates ledger parity);
wake branches (chatIdOf/jobIdOf, finalizeJob done, recoverStaleJobs
re-deliver or contextRespawn); abortJob kills context jobs SILENTLY
(nothing was promised in the lesson); turn-service trigger seam
- adapter: data-graph-context flattens to the canonical block (malformed
parts have no model face)
- UI: the budget slider (home dock + /chat/new draft, the session-data
strip); the quiet SystemLine "~537 · 10 propositions"; dock kind
"Knowledge context"; model-picker PUT now spreads settings (a model
pick can no longer clear the sibling override)
Live-smoked end to end on the real embedder + real GLM: seeded ledger
(10 propositions), session on "modal logic: possible worlds and the
necessity box" — job in the dock, queued behind the prober's question,
delivered on the answer, SystemLine in the transcript, wake turn reads
the block and continues the probe. Gates: tsc, lint, 459/459, build.
Louvain only ever found the reasoning chains (the graph is disconnected
by nature — every connected component became its own cluster) and
edge-orphans could never join anything. Clusters are now RELATED TOPICS:
propositions grouped by meaning.
- Embedder (src/server/embeddings.ts): in-process transformers.js v4
running EmbeddingGemma-300M q8 (MTEB 68.4, ~300MB one-time download),
MRL-truncated to 512d, "title: none | text:" prefix on 100% of inputs,
L2-normalized so cosine = dot. Model/dim mismatch voids the ledger
(full re-embed). Injected at the composition root — the agent layer
never imports the server layer (architecture law).
- Ledger (data/graph/embeddings.json): model + dim + vectors + the
persisted cluster partition. Reclustering runs at the END of every
embedding batch (post-merge hook, both front doors) — never per read
(the agglomerative pass is ~1.5s at 338 nodes).
- Clustering (src/core/semantic-clusters.ts): hand-rolled average-linkage
agglomerative over cosine (npm HDBSCAN is an abandoned alpha). Stragglers
stay singletons by default; attachStragglers is the all-in toggle.
Threshold 0.60 swept on the real graph: 26 topics, 74/90 edge-orphans
land in real topics, 10 stragglers.
- Consumers (snapshot, settle, namer, replay) read the persisted
partition — one cluster concept everywhere; core-survival label
matching unchanged (20/26 topics kept labels).
- Louvain retired: detectCommunities + graphology* deps deleted
(graphology-louvain had leaked into the client bundle since the
clusters wave); scripts/bench-graph.ts (dead FA2 bench) removed.
- scripts/embed-graph.ts: backfill/cold-start (338 nodes in 150s
including the one-time model download).
The 'virtual node' realized as a centroid force: two allocation-free
O(n) passes per tick (membership resolved once at sim creation;
Float64Array sums). Measured indistinguishable from force-off at 338
nodes (271ms vs 272ms per 400-tick settle).
- createGraphSimulation: SimOptions.clusterPull + linkStrength +
velocityDecay (physics controls plumbing); degree-scaled link
stiffness so linkStrength=1 is exactly d3's default
- settleGraphLayout split into layout-settle.ts (it computes the
Louvain partition for the pull); layout.ts stays browser-barrel-pure
- barrel no longer re-exports clusters.ts: graphology + louvain are
fully out of the client bundle (verified against built chunks)
- replay passes the per-batch partition; CLUSTER_PULL_STRENGTH=0.15
tuned by sweep: cluster spread 63->36 world units, centroid
separation and total span unchanged (no blob merge, no collapse)
The layout now EVOLVES instead of reshuffling on every extraction. A pure
core module (src/core/layout.ts) warm-starts forceAtlas2 from the positions
ledger (data/graph/positions.json, read-modify-write like sources.json,
updatedAt stamp for future live-physics conflict handling), re-blends every
previously-positioned node toward its anchor each iteration (retention 0.85
- old placements stay recognizable, strong forces still win), seeds new
nodes near their positioned neighbors, and seeds orphans on the rim outside
the core so they read as unconnected knowledge.
- Port/store: GraphStore.readPositions/writePositions; snapshot carries
positions when the ledger exists (optional - fresh graphs unchanged).
- Extraction pipeline settles synchronously after every merge (chat and
notes paths), before usage persist - the next /api/graph fetch already
reflects new nodes.
- View: the ledger IS the layout; the local one-shot FA2 pass survives
only as a fallback when no ledger exists.
- scripts/replay-layout.ts: simulates organic growth by replaying the
current 338-node graph one imported file at a time (real import order,
edges activate when both endpoints exist). Real run: 338 positions in
0.3s, 90-degree-0 orphans collecting at mean r=338 vs connected r=194.
- Also restores the cluster region fills (hull background colors) that
6692b19 had removed on a misreading of "no glow".
Live-verified: fit zoom derives from the ledger bbox, pickAt aligns with
ledger positions at every probe, hover/click/detail-card work, 408/408
tests, typecheck/lint/build green.
Future live physics stays cheap by design: the same pure function is the
merge-time settler and the future client simulator; a pinned node is just
an anchor with full retention.
Core-survival labelings (vision section 1, DECIDED 2026-09-13):
ClusterLabeling records in a new append-only ledger
(data/graph/cluster-labels.jsonl), immutable, id = clu_ + sha256 of the
sorted core so re-naming races and re-runs dedupe at the store seam.
matchClusterLabelings (core/clusters.ts, pure): a community keeps a
record's label while at least half its core survives in it; both sides
single-successor (a split keeps one name; greedy by overlap, ties to
the newer record).
The namer (src/agent/subagents/cluster-namer.ts): ONE singular cheap LLM
call (new 'labeler' role, glm-flash/low) names every new/changed
community at once, given member names + statements (capped at 12
members per cluster). refreshClusterLabels runs POST-EXTRACTION
(user-decided) from both front doors, fire-and-forget — naming failure
never fails the run; refreshes serialize behind a chain so parallel
imports never mint duplicate labels. import-notes.ts awaits
clusterNamingSettled before exit.
Payload: GraphSnapshot.clusters (>= MIN_CLUSTER_SIZE 3) joins detection
with labelings at read time — claimed clusters carry the record's label
and id; unlabeled ones carry a member-hash id and null label. Detection
is deterministic so no memo in v1 (ms at current scale; revisit at
vault scale).
Tests (372/372): matcher half-rule/split/merge/greedy/tie cases; naming
end-to-end through importNotes (one call, one record, claim on
re-refresh), naming failure non-fatal; snapshot derivation with
labeling join; 11 test pipelines now pin an explicit labeler seam.
Diagnosis from the real 25-file notes import (user-pasted session logs):
almost every run hit 'Validation failed for tool submit_findings/review:
edges must have required properties edges' — the model (glm flash, low
thinking) omits top-level fields when they're empty, and pi-ai's
structural JSON-Schema validation runs BEFORE the zod layer, so each
omission cost one of the extractor's maxTurns 4 (full payload
regeneration) and occasionally burned out the run entirely (the two
FAILED imports).
The model is technically wrong, but the schema provokes it:
required-but-usually-empty arrays are a known weak-model anti-pattern.
- lhTool's model-facing declaration now converts with io: "input"
(the declaration describes what the MODEL sends): .default() fields
drop out of required. No-op for teacher tools (none use defaults).
- findingsSchema.edges / auditSchema.dropped: .default([]) — omission
means "none"; attribution: .default({quiz:{}}) for quiz-free sessions.
- Prompts hardened in all three agents: include every top-level field,
never omit because empty.
Structural probe: validateToolArguments now passes the exact pasted
failure shape; zod re-parse in execute applies the defaults. 355/355
tests, typecheck, lint, build green. (lh-tool.ts is the shared engine
seam — flagged to master: io:"input" is the correct mode for every
model-facing declaration, teacher path verified unaffected.)
The same extractor→auditor→merger pipeline now runs over raw personal
notes in arbitrary formats (vision §1, DECIDED 2026-09-13):
- NotesSource records (data/graph/sources.json, deterministic id from
path hash → idempotent re-imports) play the chatId role in mappings,
checkpoints, and observation events; usage lands on the source record
(appendSourceUsage, idempotent per startedAt, kept across upserts).
- mergeFindings is chat-store-free: callers pass MergeSource
{messages, foldedAt} — sessions pass the chat record, notes pass
messages:null (no quiz-event rebuild; a deterministic rebuild needs
structured quiz tool parts).
- Second extractor prompt (NOTES_EXTRACTOR_INSTRUCTIONS) for general-
purpose extraction; understanding judges the AUTHOR's grasp. Auditor
wording input-agnostic. extract/audit share one generic runSubmitAgent.
- graphSnapshot.fileSources (srcId→filePath); notes sources render as
static path entries in the detail card (FileTextIcon, never a link).
- One shared pipeline instance (exported from chat-handler): same
single-flight registry, live status chips (label = file path), usage
accounting, and true abort for both front doors.
- Transport: POST /api/graph/import (multipart, 3-wide worker pool) +
Import notes… button in the graph view; scripts/import-notes.ts for
batch runs (concurrency 4).
- real-data-extraction/ gitignored (raw import sources never committed).
355/355 tests; typecheck, lint, build green.
The Vercel AI SDK survives only as Band D (useChat client + UIMessage
record types); @openrouter/ai-sdk-provider is uninstalled.
- src/agent/models.ts: PiEngine {model, streamFn} via createModels +
openrouterProvider subpath; thinkingLevelFor maps config effort none->off
- src/agent/lh-tool.ts: neutral LhTool (zod authoring, optional execute,
clientQuestion flag); toPiTool converts via z.toJSONSchema and re-runs
zod inside execute so refinements ride the error-result retry path
- src/server/pi/record-adapter.ts: UIMessage records -> pi Messages
(dangling tool calls get synthesized interrupted results)
- src/server/pi/ui-stream.ts: SSE bridge emitting AI-SDK-shaped chunks so
the useChat client parses unchanged (continuation id rule, question
tools never emit output, tool-input-error for bad authoring)
- teacher-agent.ts: stateless per-turn pi Agents; shouldStopAfterTurn
stops only when every question-tool call resolved cleanly (errors retry)
- chat-handler.ts: buildAgent(outgoing) factory seam (stateless doctrine);
assertValidUiMessages owns the retired SDK validation gate
- sub-agents via shared runSubagent (pi-run.ts); extraction stages stop
after a clean submit; visual-maker keeps its hand-rolled loop over
direct streamSimple calls; aborted model calls bill no usage
- tests: all model seams ported to pi-ai faux provider (scripted
FauxResponseSteps, agent factories); handleChatRequest second param
is now an agent factory
Gates: tsc, lint, build, 305/305. Live GLM smoke (disposable driver,
deleted): prober streaming + persistence, question seam, turn-2
continuation same message id, per-turn real token usage - ALL PASS.
Data wiped at the pivot (chats/graph/jobs, .gitkeeps kept).
Extraction is real spend and background work; the dock needs to see
and stop it. User decisions: persist all cost data (chat record),
per-stage usage granularity, and TRUE cancellation for abort.
USAGE CAPTURE (live + persisted):
- extraction.ts: per-call onStepFinish captures each billed step's
usage (SDK LanguageModelUsage -> CallUsage: input/output/cache
read+write/reasoning) into the registry's live per-stage usage
(extractor/auditor; the merger is deterministic — no model call).
- /api/graph/status runs now carry usage (model + steps per stage);
cost/context folds are the consumer's (core/prices.ts — nothing
stores dollars).
- ChatRecord.extractions: append-only per-run usage log (mirrors the
turns log): startedAt key, at, durationMs, merge stats when the
merger ran, stages. Persisted on success AND failure — a failed run
still spent money. Idempotent per startedAt; the list payload
(ChatSummary) never leaks it.
- EmissionStats moved to contracts/graph.ts (its contract home);
core/graph.ts re-exports for existing importers.
- Verified end-to-end against the real pipeline (backfill one chat):
per-step token counts, reasoning tokens, and the auditor's growing
prompt-cache reads captured and persisted.
TRUE ABORT (the dock's abort button, user decision):
- RunningExtraction owns one AbortController per run, shared across
phases; abortExtraction(chatId) fires it and returns whether a run
was aborted (idempotent no-op otherwise). The pending model call
receives the signal — the provider cancels the request, spend
stops. throwIfAborted() at phase boundaries ends runs aborted in
sync gaps.
- Abort is a deliberate stop, not a failure: same ledger discipline
(nothing merged, nothing checkpointed, partial usage persisted) but
the run reports zero emissions instead of throwing. Exported via
src/server/observability.ts for master's /api/activity/abort route.
- Tests: mid-run abort (hanging mock that rejects on the signal),
live-accumulation + persisted-record assertions, store-seam
idempotency, and the summary-payload leak guard.
Wave 4 (agent-side observability + control), reconciled with the dock
contract (master relay 2026-09):
- abortJob in src/server/jobs.ts, exported through the server barrel for
POST /api/activity/abort (master's thin route): fires the job's
AbortController (real kill — the in-flight model call aborts, not just
the next loop check), drops any queued result steer, flips the record
to error "Aborted by learner" + activity "aborted" synchronously, and
queues a one-line abort steer (buildAbortSteer) through the same wake
pipeline so the teacher's promise doesn't hang in the lesson. Shares
the result steer's deterministic id space, so abort + racing result
can never both deliver (steerDelivered dedupe).
- Abort registry (src/agent/subagents/aborts.ts) on globalThis, mirroring
the wake pipeline's module-state pattern; runner registers/unregisters
per run, so the route's module instance sees the loop's controller.
- Runner usage capture: LoopContext (abortSignal + onUsage) threads into
all three kind loops; per-step usage folds via accumulateJobUsage into
JobRecord.usage (model from config MODELS, absent-never-zero), flushed
in runJob's finally so an aborted job's money record survives the kill.
- Contracts: JobUsage token fields optional + lastStep (most recent
call's accounting, for the dock's per-blob context line vs
CONTEXT_LIMITS); SubagentResultMeta.aborted; AbortSteerMessage;
buildAbortSteer; SteerMessage union generalized.
- demo-builder/visual-maker accept abortSignal + onUsage (visual folds
generateText usage per iteration).
- stillRunning/finishJob/jobContext guards: an aborted job is never
overwritten by the runner (terminal no matter what the wake does);
failJob and the pump's finalizeJob keep the flip.
- UI: abort steer renders a quiet aborted chip (job-aborted-badge).
- Prompt 20-jobs.md: one line about aborted jobs.
Tests: accumulateJobUsage folds (undefined-preservation, lastStep),
runner usage capture, real-kill abort (sync flip, silent unwind,
abort steer wakes), usage-so-far survives the kill, aborting a
delivering job drops the queued result steer, outcome reasons,
abort-chip render. 330/330, typecheck/lint/build green.
The extractor's per-session understanding notes were free-form strings;
the model freely appended em-dash justifications ('shaky — the learner
could state the headline distinction but calls the machinery behind it
foggy'), which overflowed the detail card's session rows and squeezed
the session names.
The note becomes a CLOSED named-word scale — solid / shaky / accepted /
resisted / not yet engaged (DECIDED 2026-09-08, no numbers) — enforced
by the zod enum in propositionSchema: a wrong-shaped note is rejected
at parse time and the agent retries. Justifications are dropped, not
stored (user decision: not necessary). Extractor and auditor
instructions updated to demand exactly one word.
Renderer: the session row keeps the session NAME as the main thing
(flex-1, truncates) and the level chip is bounded (max-w-28,
whitespace-nowrap, truncate) — overflow impossible even for legacy
prose notes in old ledgers.
Vision doc: §1 extractor bullet + event-ledger line updated; state
model line notes the five-class derivation was removed; amendments
table row added.
The background extraction pipeline is invisible on the home surface —
essential observability per the user's directive: the graph view must
show what is currently running (extracting / auditing / merging) and
for which chat.
- extraction.ts: module-level registry of in-flight runs (chatId,
phase, startedAt), updated at each phase transition in runExtraction
and cleared in finally (done or failed). extractionStatuses() is the
read; one process, one registry.
- GET /api/graph/status: thin transport over the registry, chat titles
joined from the chat store.
- graph-view: polls every 2s and shows one chip per running pipeline
(spinner + phase + title + elapsed seconds, bottom-left); when a run
finishes (some -> none) the graph refetches, so merges land live.
Status is cosmetic — poll errors never surface.
- Unit test drives a mocked pipeline run and observes the registry
mid-flight: 'extracting' during the extractor's generate, 'auditing'
during the auditor's, empty after.
With real extracted data, full-sentence statements as canvas labels proved
unworkable: 10-line label blocks, illegible overprint clusters, bottom-edge
clipping. Per the user's 2026-09-08 decision:
- GraphNode/ExtractedProposition gain `name` — a very short, glanceable
label (glossary-headword style: "Will to power"), presentational only:
not unique, not identity, never precise. Set at creation; link-appends
never change it (first line defines; labels stay stable across sessions).
- `statement` keeps its full precision and becomes the identity anchor;
the detail card shows it under the name. `description` is dropped.
- Extractor coins the name (freshest session context); auditor carries it
through on create, rewording only clearly unglanceable names; linked
propositions keep the existing node's name.
- Renderer: canvas label = name; search matches name + statement;
detail card = name (bold) + statement (muted).
- Store guard requires name (pre-pivot node lines are skipped as legacy);
vision.md node-schema bullet + amendments table updated (2026-09-08).
The minimal implementation of the 2026-09-02 graph vision (docs/vision.md §1):
- contracts/graph.ts: every node is a TRUE proposition (statement is both
rendering label and identity key; required description; no kind, no name).
Edges are follows (derivable) / supports (grounds without entailing) with
message-granular provenance points {chatId, messageId}. Events reduce to
correct/miss/dont-know plus observation (understanding notes); no taught
events, no misconception kind.
- extraction.ts: extractor is TREE-BLIND (transcript only, no plan tree) and
free-form; auditor sees findings + the current graph, owns faithfulness and
identity (link-or-create verdict per proposition, recorded as linkTo);
transcriptView tags every line with its message id (provenance anchors).
- core/graph.ts merger: applies verdicts deterministically — linkTo appends
a link line (phrasings join aliases, provenance unions; reads merge by id),
absent creates; NO string matching anywhere. Quiz events rebuilt from tool
parts with attribution-only nodeId resolution; understanding notes become
observation events (obs:<sourceId>); mapping lines are the verdict audit
trail and make re-merges idempotent.
- deriveNodeStates: transitional — latest quiz event wins (known/missed/
dont-know, else candidate); taught unreachable until the state-model
ideation lands.
- graph-context.ts + 50-graph-context.md: follows chains, hotspot grouping,
state vocabulary updated for the new event kinds.
- graph-view.tsx: node label is the statement; detail shows statement +
description.
- tests: merger verdicts (link, no-string-match, cross-chat assertions),
tree-blind pipeline end-to-end, follows vocabulary in context tests.
276/276 green.
The §5 baseline had per-step lines, starts, drops, kept ratios, and
merge stats — but no durations (a background pipeline's cost profile
was invisible) and no failure line: a mid-pipeline throw left a
hanging 'pipeline start' in the server log with nothing recording
what failed or when. Now the extractor and auditor each log their
elapsed ms on completion, the merged line carries the pipeline total,
and runExtraction logs a failure line (with elapsed) before
rethrowing. Log shape only — no behavior, contract, or control-flow
change.
The job tools (research/demo/visual) switch from client-side turn-breaking
calls to pi's model: execute resolves IMMEDIATELY with a fire-and-forget
ack, the teacher's turn continues, and the result arrives later as an
injected user-role steer message that wakes the teacher in a new
server-side turn.
- contracts.ts: THE ASYNC PROTOCOL rewritten (ack now, steer later); steer
types (SubagentSteerMessage, metadata.subagentResult + data-subagent-
artifact part), pure buildSubagentSteer/steerDelivered/isSubagentFailure
- tools/jobs.ts: jobTools(ctx) factory — server-side execute creates the
job (deduped by toolCallId), fires spawn with the dispatch-time input
(the part isn't persisted yet at execute — a transcript scan would race)
- runner.ts: runJob(jobId, input?) builds the steer and hands it to the
injected deliver seam; pipeline owns the done/error flip (a polling
client re-syncs once, at the end); failures still steer
- server/jobs.ts (new pipeline): per-chat serialized queues on globalThis,
idle gates (turn liveness + pending question, 1.5s grace), 409 retry
with backoff, idempotent per jobId, stale-job recovery sweep (re-deliver
or re-spawn); wake turns run through the same handleChatRequest
composition via setTurnRunner injection
- chat-handler.ts: wires runner+deliver+tools ctx; drops spawnTurnJobs
from onFinish; teacher default registry = question tools only
- plan.ts: PlanToolContext split out (plan tools need no job wiring)
- visual-maker: edit_svg added (pi parity — exact-match single
replacement, fuzzy context on a miss); researcher: pi's kept/dropped
source accounting restored
- prompts 20-jobs/40-demos: async discipline rewritten (announce and keep
teaching, no waiting/polling/fabricating, parallel dispatch, teach with
the artifact when it arrives)
- tests: jobs.test.ts rewritten (dispatch ack mid-turn, runner steers,
wake pipeline: deferral/409/idempotency/failure), demos.test.tsx runner
tests ported to runJob + fake deliver; 222 green
FLAGGED surgical touches outside the stream's owned files, for master
review at merge: src/core/contracts.ts (protocol doc + steer types),
src/core/ui-message.ts (job tool + data-part types), src/server/
chat-handler.ts (wiring), src/server/turn-service.ts (spawnTurnJobs
retirement), src/agent/teacher-agent.ts (default tools), src/agent/tools/
plan.ts (context type split).
Verified live on PORT=3003 with real models: teacher dispatched research
mid-turn, researcher ran real searches/fetches, steer injected on quiz
answer, wake turn wove the brief in citing its sources, job flipped done;
a client/wake race correctly 409'd.
Schema: GraphNode + ExtractedProposition carry 'name' — a short 2-6 word
header, the rendering handle. Node theory made explicit: a node is an
objective world-facing fact; the learner's relationship to it lives
entirely in the event ledger (state labels are derived, never stored).
Extractor + auditor instructions enforce it — misconceptions state the
wrong idea itself as a claim, never 'the learner believes Y'. Legacy
ledger lines without a name derive one from the statement on read.
graph-view: labels use the name; the detail panel shows name as header
and statement as body.
MODELS in config.ts is now THE single tuning table: model id + OpenRouter
reasoning effort for teacher/researcher/demoBuilder/visualMaker/extractor/
auditor. .env.local is secrets-only (LH_* model vars removed; all sub-agents
previously had NO reasoning setting -> provider-default unbounded thinking,
the extraction token burn). One factory (src/agent/models.ts modelFor) wires
all six call sites; test injection unchanged. EXTRACTION.everyNQuizAnswers
moved to a config literal too. Plus the §5 one-line-per-seam logging in the
extraction pipeline: pipeline start, extractor/auditor start + per-step,
audit kept, merged — previously silent until completion, which made live
runs look like they never fired.
The knowledge graph's write channel is now the pivot's target
architecture (vision §1 construction):
- src/agent/subagents/extraction.ts — the pipeline factory
(createExtractionPipeline({ chatStore, graphStore, extractor?,
auditor? }); models env LH_EXTRACT_MODEL ?? LH_MODEL):
* transcriptView(): the compact view both LLM stages see — final plan
tree with node ids, conversation verbatim, QUIZ lines with outcomes,
mark_done lines; job tool parts are one-liners, plan ops fold into
the tree line.
* Extractor: transcript → SessionFindings (propositions incl.
misconceptions and prose-only supports edges, attribution for
UNTAGGED probes, all with verbatim evidence) via one zod-validated
submit_findings call.
* Auditor: transcript + findings → the verified subset + drops (the
faithfulness net) via submit_review.
- src/core/graph.ts — emitChat/foldStaleChats/replayFinalTree retire;
mergeFindings() replaces them as the pipeline's deterministic apply
stage: exact/alias identity resolution (one mapping line per (chat,
graph node)), edge dedupe, and event rebuild FROM THE TRANSCRIPT's tool
parts — kinds/timestamps/refs/notes deterministic, the LLM's only event
say being WHICH proposition an untagged probe tested (retroactive —
the vision's open question #3, resolved structurally: tagged events
resolve deterministically; attribution rescues untagged and
tagged-but-unmapped ones; unattributable land null + question text).
pendingQuizAnswers() is the trigger helper.
- Trigger (turn-service.ts triggerExtraction, replacing the old
foldTurnIntoGraph body): once a chat accumulates
EXTRACTION.everyNQuizAnswers (LH_EXTRACT_N, default 3) new answered
quizzes, fire the pipeline async — single-flight per chat, failure
checkpoints nothing and re-runs on the next trigger. Deliberately a v1
placeholder isolated in ONE seam, trigger-agnostic pipeline.
- GET /api/graph is now a pure read (foldStaleChats gone).
- NodeMapping.treeNodeId → sourceId (the mapping key is now the writer's
local proposition id — tree node when tree-derived); store strict-read
schema updated. Local ledger data regenerates through the pipeline.
- Tests: graph.test.ts rewritten around mergeFindings (identity,
attribution paths incl. ghost-rescue, idempotency, cross-session
provenance, trigger counting), graph-context.test.ts merge section
converted to findings-driven, NEW extraction.test.ts — transcriptView
units + the full pipeline with mock models (audit drops, failure
checkpoints nothing, single-flight). 195/195, tsc/eslint/build clean.
Probe attribution (vision open question): probes attach retroactively by
the extractor against the whole transcript; a pre-tree probe maps to a
proposition the tree only articulates later. Fuzzy/embedding merge
adjudication stays deferred (conservative exact/alias only).
The chat turn is now composed, not implemented, by chat-handler.ts:
- src/server/turn-service.ts: one named, error-contained seam per turn
responsibility — buildTeacherContext (W8 read channel), persistTurn
(canonical state write), spawnTurnJobs (async protocol kick-off),
foldTurnIntoGraph (session→graph consolidation; the extraction-pipeline
stage swaps this body behind the same seam). Each takes its deps
explicitly; failures log one [chat <id>] line and never touch the
response.
- chat-handler.ts wires: storage ports (stores), the job runner, the
teacher tools (ctx = chatId + injected ChatStore), and the services.
It no longer contains any logic beyond request validation.
Agent layer now receives dependencies instead of importing them — a new
lint zone enforces it (src/agent never imports src/server or src/store;
verified to fire both ways):
- runner.ts: createJobRunner({ chatStore, jobStore }) factory returning
the spawn hook + the three job runners; the model-override test seam
stays module-level. Bodies otherwise verbatim.
- plan tools: persist through the ctx-injected ChatStore
(TeacherToolContext = { chatId, chatStore }).
- teacher-agent's dependency-free default registry is now the true
client-side set: questionTools + jobTools (jobTools was ctx-parameter-
shaped but never used its ctx — now honestly context-free). Production
always passes createTeacherTools({ chatId, chatStore }); tests that
drive a bare mock model get the client-side set unchanged.
Zero behavior change: tsc clean, 190/190 tests, eslint clean, production
build clean. Tests inject the same fs ports the chat-handler does.
Per-entity semantic ports (src/core/ports.ts), one adapter family
(src/store/*, fs JSON/JSONL), exactly one composition point
(src/server/stores.ts). Dependency direction is now machine-checked:
adapters implement core interfaces; core never imports an adapter.
The inversion:
- core/graph.ts (emitChat/getGraph/graphSnapshot/foldStaleChats) is now
pure — it takes the storage ports as explicit parameters (GraphDeps).
No store imports; newGraphNodeId is local; node id format unchanged.
- core/graph-context.ts buildGraphContext() no longer defaults to the
live ledgers: the chat-handler hands in getGraph(graphStore) per turn.
- store adapters export exactly one object each implementing their port
(chatStore/graphStore/jobStore); bodies otherwise verbatim.
NodeMapping + SessionCheckpoint moved store/graph -> core/contracts
(they ARE contract shapes); CreateJobInit/JobPatch live in ports.
- titleFromPrompt moved store/chats -> core/contracts (pure helper,
planNodeStatus precedent). Store modules now contain persistence only.
- server/stores.ts is the composition root; server/{chats,graph,jobs}.ts
and chat-handler wire from it; agent (plan tools, sub-agent runner)
call the port objects. The shell surface (@/src/server exports) is
unchanged.
Lint law extended: the whole of src/core is now a purity zone (was a
file glob with a graph/graph-context exemption). Verified it fires on
core->store imports; tsc clean, 190/190 tests, eslint clean, prod build
clean (prerender 6/6). Tests pass deps explicitly — graph.test uses the
same fs-adapter pair stores.ts wires in production; graph-context.test
keeps its in-memory ledgers as a GraphStore implementation.
- src/store/jobs.ts: per-job JSON store in data/jobs (create/get/update/list,
dedup by toolCallId)
- src/agent/tools/jobs.ts: research tool — client-side (no execute), turn ends
at the call; input per contracts ResearchToolInput
- src/agent/tools/web.ts: server-side web_search (Google CSE) + web_fetch
(Readability/Turndown/linkedom, Jina fallback) — errors return as strings,
never thrown into the agent loop. Deps: @mozilla/readability, linkedom,
turndown (+@types/turndown dev)
- src/agent/subagents/researcher.ts: isolated ToolLoopAgent (web tools only,
12-step cap), researcher prompt ported from pi
- src/agent/subagents/runner.ts: spawnPendingJobs hook + runResearchJob —
fire-and-forget, activity line per tool call, fulfills the part in the
LATEST chat record (output-available) before flipping job status; failure
still fulfills the part so the teacher learns the outcome
- src/server/chat-handler.ts: onFinish spawn hook after saveMessages
- app/api/jobs/route.ts: GET ?chatId= polling endpoint
- components/artifacts/job-card.tsx + registry entry: polling activity card
(2s) → syncFromServer on finish; research-brief card (markdown + sources)
- src/agent/prompts/20-jobs.md: research doctrine for the teacher
- tests/jobs.test.ts: store CRUD, spawn hook via scripted mock teacher,
runner completion/failure with fake model + stubbed fetch
amosblomqvist