Louvain only ever found the reasoning chains (the graph is disconnected
by nature — every connected component became its own cluster) and
edge-orphans could never join anything. Clusters are now RELATED TOPICS:
propositions grouped by meaning.
- Embedder (src/server/embeddings.ts): in-process transformers.js v4
running EmbeddingGemma-300M q8 (MTEB 68.4, ~300MB one-time download),
MRL-truncated to 512d, "title: none | text:" prefix on 100% of inputs,
L2-normalized so cosine = dot. Model/dim mismatch voids the ledger
(full re-embed). Injected at the composition root — the agent layer
never imports the server layer (architecture law).
- Ledger (data/graph/embeddings.json): model + dim + vectors + the
persisted cluster partition. Reclustering runs at the END of every
embedding batch (post-merge hook, both front doors) — never per read
(the agglomerative pass is ~1.5s at 338 nodes).
- Clustering (src/core/semantic-clusters.ts): hand-rolled average-linkage
agglomerative over cosine (npm HDBSCAN is an abandoned alpha). Stragglers
stay singletons by default; attachStragglers is the all-in toggle.
Threshold 0.60 swept on the real graph: 26 topics, 74/90 edge-orphans
land in real topics, 10 stragglers.
- Consumers (snapshot, settle, namer, replay) read the persisted
partition — one cluster concept everywhere; core-survival label
matching unchanged (20/26 topics kept labels).
- Louvain retired: detectCommunities + graphology* deps deleted
(graphology-louvain had leaked into the client bundle since the
clusters wave); scripts/bench-graph.ts (dead FA2 bench) removed.
- scripts/embed-graph.ts: backfill/cold-start (338 nodes in 150s
including the one-time model download).
The graph's follows/supports edges encode derivation (premise ->
conclusion); the layout can now express that rank spatially, as a
session-only playground toggle (nothing promoted yet):
- inferenceDepths: longest chain from a base node, CYCLE-SAFE by
SCC condensation (Tarjan, iterative) — the DAG property is typical,
not guaranteed; cycle members share their component's depth (mutual
support = equal footing). O(V+E), pure, deterministic.
- DepthAnchor force: per-node target scalar — vertical mode rows
(baseline at the anchor centroid, arrows up), radial mode rings
(foundations center, outer boundary = edge of knowledge). The rank
is computed, not simulated: pairwise up-forces can't give baselines.
- Mode projections: gravity x-only in vertical / dropped in radial;
cluster pull x-only in vertical / tangential-only in radial; the
anchor re-blend preserves user arrangement in the FREE dimension
only (x / angle) — meaning owns the constrained axis, memory owns
the free one.
- setPhysicsTuning reheats (alpha 1): toggling a mode re-forms the
graph live and commits the new rest state; drag release is springy
along the constrained axis (node returns toward its level/ring).
- Sweep on the real graph: strength 0.8 = 215/216 arrows correct with
organic sag; 1.5 = crisp rows. 174 base nodes, max chain depth 7,
zero cycles in current data. replay-layout --depth=<mode> bakes a
mode into the ledger for rest-state experiments.
User-facing live-sim controls (settings panel, Physics section):
node repulsion, edge attraction, edge length, central pull, cluster
pull, friction — session-only number inputs feeding createGraphSimulation
(the SAME engine the server settle runs, so the feel IS the physics).
Changing a value rebuilds the sim from current positions; drag pins
(fx/fy live on the node objects) survive; a mid-settle change re-anneals.
The graph-view FA2 bootstrap fallback is rewritten on incrementalLayout
(d3-force): the last graphology import in the shell is gone.
The 'virtual node' realized as a centroid force: two allocation-free
O(n) passes per tick (membership resolved once at sim creation;
Float64Array sums). Measured indistinguishable from force-off at 338
nodes (271ms vs 272ms per 400-tick settle).
- createGraphSimulation: SimOptions.clusterPull + linkStrength +
velocityDecay (physics controls plumbing); degree-scaled link
stiffness so linkStrength=1 is exactly d3's default
- settleGraphLayout split into layout-settle.ts (it computes the
Louvain partition for the pull); layout.ts stays browser-barrel-pure
- barrel no longer re-exports clusters.ts: graphology + louvain are
fully out of the client bundle (verified against built chunks)
- replay passes the per-batch partition; CLUSTER_PULL_STRENGTH=0.15
tuned by sweep: cluster spread 63->36 world units, centroid
separation and total span unchanged (no blob merge, no collapse)
User feedback round 1 on the splat regions:
- Double counting: per-node gradient discs stacked with the edge-stroke
dilation wherever nodes sat on edges (hub hot spots). A cluster is its
CONNECTED skeleton — the region is now the dilated intra-cluster edge
network ONLY (round caps make line-end points discs), per the user's
definition. No per-node discs at all.
- The 'blur' was no blur: bilinear upscale of a 256px bitmap produced
tent-ramp chunkiness at zoom (the low-quality-JPEG look). Now a REAL
blur: 3 box-blur rounds on the premultiplied bitmap (separable running
sum, pure + unit-tested; premultiply keeps halos out of transparent
space). The upscale now has nothing left to make chunky.
- Live dev controls: number inputs (not sliders) in the settings panel —
dilation (screen px at fit), blur (bitmap px), bitmap resolution,
edge alpha. Session-only React state, never persisted; the renderer's
sanitizeRegionTuning makes bad/empty values safe. Verified end-to-end
through the real input (dilation 28 → 60 took effect).
- Premultiply un-recovery bug caught live: c = pm/a*255, not pm/a (the
first version rounded every color to black; regions rendered as
dark blobs with alpha 185).
- Region canvas: willReadFrequently (Chrome keeps it on the CPU for the
per-rebuild getImageData); defaults dilation 55 → 28 after measuring
screen coverage (63% → 55%).
- Perf: 120fps sustained through a drag with blur + rebuild-every-3rd-
frame (45 frames / 360ms measured), zero idle frames after settle.
The renderer grows a live d3 sim (createGraphSimulation, shared with
the server settle): pointerdown on a node pins fx/fy and pointer capture
(preventDefault kills the compat mousedown, so d3-zoom never pans); the
first CLICK_SLOP-exceeding move reheats to alphaTarget 0.3; release
un-pins and the sim anneals (release-and-relax) — alpha < 0.005 settles
it, a 250ms debounced onCommit PUTs the rest state. Verified live: 123fps
during a 320ms drag (regions rebuilding every 3rd frame), 85-tick settle,
then zero idle frames. A slop-free press stays a click (select, no reheat,
no commit). PUT /api/graph/positions sanitizes (finite coords, known ids)
and if the graph grew mid-drag runs ONE incremental settle with the
client's arrangement as anchors (50-id PUT test: anchored drift 0.15).
Cluster regions swap convex hulls for a low-res world-space SPLAT bitmap:
per-member radial-gradient discs (alpha 0.2) + intra-cluster edges as
round-cap strokes (Minkowski dilation — concave shapes finally render),
upscaled with bilinear smoothing (the upscale IS the blur, no ctx.filter:
Safari). One tiny blit per frame replaces 32 clusters x 4 offset
Path2D polygons re-rasterized at DPR 2 (the old GPU coil-whine path);
during the live sim the bitmap rebuilds every 3rd frame. cluster-geometry.ts
retires convexHull/offsetPolygon/traceRoundedPath (tests follow; clusterHue
stays). Region labels stay (centroid + clusterHue, inverse-gated on zoom).
The layout now EVOLVES instead of reshuffling on every extraction. A pure
core module (src/core/layout.ts) warm-starts forceAtlas2 from the positions
ledger (data/graph/positions.json, read-modify-write like sources.json,
updatedAt stamp for future live-physics conflict handling), re-blends every
previously-positioned node toward its anchor each iteration (retention 0.85
- old placements stay recognizable, strong forces still win), seeds new
nodes near their positioned neighbors, and seeds orphans on the rim outside
the core so they read as unconnected knowledge.
- Port/store: GraphStore.readPositions/writePositions; snapshot carries
positions when the ledger exists (optional - fresh graphs unchanged).
- Extraction pipeline settles synchronously after every merge (chat and
notes paths), before usage persist - the next /api/graph fetch already
reflects new nodes.
- View: the ledger IS the layout; the local one-shot FA2 pass survives
only as a fallback when no ledger exists.
- scripts/replay-layout.ts: simulates organic growth by replaying the
current 338-node graph one imported file at a time (real import order,
edges activate when both endpoints exist). Real run: 338 positions in
0.3s, 90-degree-0 orphans collecting at mean r=338 vs connected r=194.
- Also restores the cluster region fills (hull background colors) that
6692b19 had removed on a misreading of "no glow".
Live-verified: fit zoom derives from the ledger bbox, pickAt aligns with
ledger positions at every probe, hover/click/detail-card work, 408/408
tests, typecheck/lint/build green.
Future live physics stays cheap by design: the same pure function is the
merge-time settler and the future client simulator; a pinned node is just
an anchor with full retention.
- LABEL_GAP 6 -> 2 (user: labels sit way too far below their nodes).
- Hover glow is now PROPORTIONAL to the disc — outer radius = node radius
x 1.7 (GLOW_SCALE), matching the blur-a-copy-of-the-disc mental model,
instead of the old constant +6px pad that read huge on small nodes and
thin on big ones. GLOW_ALPHA 0.45 -> 0.3.
- Hover-OUT fade was broken by construction: draw() gated the glow on
hoverId, which nulls instantly on leave, so the 180ms fade-out ran
invisibly and the glow popped off in one frame. New hoverNode field
outlives hoverId through the outgoing fade (cleared when the fade
completes); verified live: 8 distinct frames over ~200ms then stable.
- Sliders rescaled per spec: ALL five are now log-scaled 0-100 with the
default dead-center at 50 (ranges chosen as default/4..default*4:
node/arrow/label size 0.5-8, edge width 0.25-4, labelZoom 0.0625-4),
coarse step 5 (21 stops; labels-until-zoom was "way too granular" at
201). Generic toSliderPosition/fromSliderPosition replace the
labelZoom-only pair; sanitize clamps follow the same range constants;
size sliders display compact values ("2", "1.6", "0.75").
EOF
The defaults were calibrated in the screen-fixed era; world-scaling renders
sizes at setting x k/kFit, so at the default label gate (2x fit) everything
read ~2x too big ("labels way too large even at the minimum"). Fit-zoom
normalizes layout scale away, so the fix belongs in sizes, not layout:
spreading nodes further apart in world units changes nothing at fit.
- Defaults: nodeSize 6 -> 4, arrowLength 5 -> 4, labelSize 12 -> 7.
- Clamps/slider ranges widened at the low end: nodeSize 1-14 (step 0.5),
edgeWidth 0.25-4 (step 0.25), arrowLength 1-15 (step 0.5),
labelSize 3-24 (step 0.5).
- GLOW_PAD 9 -> 6 (halo dominated small discs), REGION_LABEL_SIZE 18 -> 14.
- Tests updated for the new clamp bounds plus a low-end pass-through case
(403/403). The saved render-settings.json still shadows defaults by
design; the local copy was reset so the new defaults showed immediately.
Five refinements to the hand-rolled renderer, all user-directed:
1. Settings are SERVER-persisted (data/render-settings.json via new
GET/PUT /api/graph/render-settings) - one source of truth for every
browser, replacing localStorage. The client loads once after mount,
saves debounced on user changes, and never lets a slow GET clobber a
faster user (or mounting PUT defaults over the server copy). The
shape/defaults/sanitizer moved to src/core/render-settings.ts - pure,
shared by client, server store, and tests; every disk/network value
passes through sanitizeRenderSettings.
2. New "Label size" control (default 12px - the previous hardcoded size).
Font, wrap width, and line height all scale; verified live: +76k px
with the label gate open.
3. "Labels until zoom" slider reworked: log-scaled over 0.0625-4 so the
useful band (0.25-0.5 camera-ratio units) sits at 33%/50% of the
track instead of crammed at the far left of the old linear 0.25-4
slider. Display units are normalized: the default 0.5 reads as "1x",
0.25 as "0.5x". The underlying labelZoom semantics are unchanged,
so existing values survive.
4. Smooth wheel zoom: d3-zoom instant per-notch jumps (x1.2, bang) are
replaced by an eased glide - wheel events accumulate a cursor-anchored
target scale and each tick eases the camera toward it (exponential,
120ms constant, frame-rate independent). Pan/pinch stay d3-owned; a
pan cancels a glide in progress. Verified live: 8-frame glide.
5. Resize blanking fixed (graph vanished for the duration of a window
drag, reappearing ~300ms after). Cause: every resize event
reallocated the canvas backing store - which CLEARS it - and the
redraw was deferred to rAF, which live resizes starve via relayout,
so the canvas sat wiped for the whole drag. Fix: the ResizeObserver
callback now redraws SYNCHRONOUSLY after reallocating (verified via
a probe RO firing in the same delivery batch: canvas fully painted).
kFit (the label gate 1x reference) is also recomputed on resize from
the now-cached world bbox - the camera itself still persists.
tests/render-settings.test.ts: 10 tests pinning the sanitizer (garbage/
partial/clamps, legacy no-labelSize records) and the slider mapping
(default dead-center at 500, monotonic, exact round-trips, display
units). 402/402 green; typecheck, lint, build pass.
Implements Option B from docs/research/graph-rendering.md. sigma.js v3
(WebGL) is replaced by one canvas, one rAF loop with a dirty flag, and
d3-zoom for the camera — eliminating the entire bug class from the
sigma-era bolt-on architecture:
- Hull/node desync during pan (two independent render loops) is now
structurally impossible: hull fills, dashed hull strokes, edges with
arrowheads, node discs, region labels, node labels, and hover glow
are all drawn in ONE frame under ONE camera.
- The settings-change canvas-sizing race (everything shifting up-left)
is gone: settings changes only redraw; the canvas and camera persist
(verified live: camera identical to 15 decimals across a slider drag).
- The GPU floor is gone: idle schedules NO frames at all (verified via
pixel-identity over 600ms idle); pan redraws exactly once per frame.
components/graph-canvas.ts: GraphCanvasRenderer class plus pure helpers
(zoomRatioOf, labelAlpha, fitTransform, arrowGeometry, pickNearest).
The sigma camera-ratio convention (1 = fit, smaller = zoomed in) is kept
verbatim so persisted labelZoom settings survive the swap; node labels
and cluster region labels cross-fade over the same LABELS_FADE_SPAN
(0.5, unchanged). Picking is a nearest-node distance check with the
10px pad that replaces the old padded-picking vertex shader. Hull math
stays in components/cluster-geometry.ts, reused as-is. No ctx.filter,
ever.
graph-view.tsx keeps every feature: fetch/refetch on runs-empty, FA2
run-then-freeze layout, localStorage render settings + sanitizers,
search filtering with edge drops and card-close, detail card with
flipped anchoring and per-source provenance, settings panel, import
button, pipeline status chips. Only the renderer plumbing changed.
tests/graph-canvas.test.ts pins the pure math: ratio direction, label
gate + inverse complement (alphas sum to 1), fit transform padding,
arrowhead tip-on-disc-edge and symmetry, picking pad/ties/null cases.
Live-verified end-to-end on the real 338-node graph (dev server
port 3002): single canvas, paint at fit, hover cursor+glow+fade, click
card open/anchored/close, stage click, search filter to zero with hint,
label gate opens via slider and zoom, cluster-labels toggle, zoom-to-
cursor 0.56px anchor drift, camera persistence across settings, zero
idle repaints. sigma dependency removed from package.json; d3-zoom and
d3-selection added.
Note: the settings feed effect needed isEmpty in its deps — without it
the renderer mounted with settings=null and every draw early-returned
(blank canvas); caught live, not by typecheck.
Three problems from the first cut (user report 2026-09-13), all rooted
and fixed:
1. Boundaries reshaping on zoom — an inward offset normal. In y-down
screen coordinates, monotone chain yields clockwise-on-screen hulls,
whose OUTWARD normal is (dy, -dx); the code used (-dy, dx), so the
14px padding pulled every hull vertex 14-20px INWARD. Zoomed in this
mostly passed; zoomed out, once a cluster projected smaller than ~2x
the pad, vertices crossed the centroid and the polygon self-
intersected — spiky garbage re-forming every frame. Geometry moved to
components/cluster-geometry.ts (exported, unit-tested) with the
corrected normal; tests pin the direction (offset moves every vertex
AWAY from the centroid; stays sane when the hull is smaller than the
pad; hull orientation asserted clockwise-on-screen).
2. GPU ~100% + audible coil whine on any interaction — the hull redraw
was hooked to sigma's afterRender, which fires for EVERY render
including ~a dozen hover-animation frames per hover, and each redraw
rasterized ~32 fills through ctx.filter blur(12px), a GPU-heavy
full-canvas filter. Now: redraws happen ONLY on camera updates and
resizes (hover leaves the hull canvas untouched — verified live),
and the blur is replaced with layered low-alpha fills (soft edge at
a fraction of the cost, no canvas filter at all).
3. Cluster labels cluttering the view — a persisted 'Cluster labels'
checkbox in the rendering settings (hulls stay, text toggles).
377/377 tests, typecheck, lint, build green. Live-verified: hover does
not repaint hulls, zoom does; far zoom-out yields 3 contiguous tinted
regions (no fragmentation); toggle clears/returns labels and persists.
Core-survival labelings (vision section 1, DECIDED 2026-09-13):
ClusterLabeling records in a new append-only ledger
(data/graph/cluster-labels.jsonl), immutable, id = clu_ + sha256 of the
sorted core so re-naming races and re-runs dedupe at the store seam.
matchClusterLabelings (core/clusters.ts, pure): a community keeps a
record's label while at least half its core survives in it; both sides
single-successor (a split keeps one name; greedy by overlap, ties to
the newer record).
The namer (src/agent/subagents/cluster-namer.ts): ONE singular cheap LLM
call (new 'labeler' role, glm-flash/low) names every new/changed
community at once, given member names + statements (capped at 12
members per cluster). refreshClusterLabels runs POST-EXTRACTION
(user-decided) from both front doors, fire-and-forget — naming failure
never fails the run; refreshes serialize behind a chain so parallel
imports never mint duplicate labels. import-notes.ts awaits
clusterNamingSettled before exit.
Payload: GraphSnapshot.clusters (>= MIN_CLUSTER_SIZE 3) joins detection
with labelings at read time — claimed clusters carry the record's label
and id; unlabeled ones carry a member-hash id and null label. Detection
is deterministic so no memo in v1 (ms at current scale; revisit at
vault scale).
Tests (372/372): matcher half-rule/split/merge/greedy/tie cases; naming
end-to-end through importNotes (one call, one record, claim on
re-refresh), naming failure non-fatal; snapshot derivation with
labeling join; 11 test pipelines now pin an explicit labeler seam.
detectCommunities(nodes, edges) in src/core/clusters.ts (new dep
graphology-communities-louvain): Louvain modularity over the undirected,
unweighted follows+supports structure (user-decided 2026-09-13). The
random-walk traversal runs on an inlined seeded mulberry32 so the same
graph always yields the same partition; community indices are still
unstable across edits, which is exactly why labelings (phase B) key to
sorted MEMBER SETS. follows+supports between the same pair collapse to
one edge; self-loops and dangling edges are skipped defensively; no
min-size filter here (the >=3 render threshold is a presentation rule).
Tests: two-cliques-plus-bridge split, determinism, singleton/empty
graphs, pair collapse, defensive skips, member sorting. 363/363.
Real-data smoke over the current notes graph: 338 nodes / 211 edges ->
138 communities, modularity 0.95, and the big ones are exactly the
learn-session topics (databases, differential forms, sales, servers,
vim, copulas...). 32 communities clear the >=3 render threshold.
The same extractor→auditor→merger pipeline now runs over raw personal
notes in arbitrary formats (vision §1, DECIDED 2026-09-13):
- NotesSource records (data/graph/sources.json, deterministic id from
path hash → idempotent re-imports) play the chatId role in mappings,
checkpoints, and observation events; usage lands on the source record
(appendSourceUsage, idempotent per startedAt, kept across upserts).
- mergeFindings is chat-store-free: callers pass MergeSource
{messages, foldedAt} — sessions pass the chat record, notes pass
messages:null (no quiz-event rebuild; a deterministic rebuild needs
structured quiz tool parts).
- Second extractor prompt (NOTES_EXTRACTOR_INSTRUCTIONS) for general-
purpose extraction; understanding judges the AUTHOR's grasp. Auditor
wording input-agnostic. extract/audit share one generic runSubmitAgent.
- graphSnapshot.fileSources (srcId→filePath); notes sources render as
static path entries in the detail card (FileTextIcon, never a link).
- One shared pipeline instance (exported from chat-handler): same
single-flight registry, live status chips (label = file path), usage
accounting, and true abort for both front doors.
- Transport: POST /api/graph/import (multipart, 3-wide worker pool) +
Import notes… button in the graph view; scripts/import-notes.ts for
batch runs (concurrency 4).
- real-data-extraction/ gitignored (raw import sources never committed).
355/355 tests; typecheck, lint, build green.
- Option rows: items-start → items-center — the icon span's top-misalign
came from the row alignment; now marks center across the row.
- The literal ✓/✗ look (owner preference over the round-3 lucide set)
re-rendered as hand-rolled SVG with thick rounded strokes (strokeWidth
3, round caps/joins) — the glyph shape, as a true vector, sized like
the row text. Test pins updated to data-mark=check|x.
- ✓/✗ in the answered card: replaced literal glyphs with lucide CheckIcon/
XIcon (strokeWidth 2.5) inside a fixed size-4 flex box — text elements
never carried the right size/vertical alignment, vectors do. Test pins
updated (lucide-check/lucide-x classes).
- All user-typed INPUT text now matches markdown output: lh-prose class
replaces the earlier font-prose (family-only) band-aid on the composer,
question-dock text/thinking/Other fields, home topic input, model search
box, and answered-card 'your thinking'/free-text records. lh-prose sets
family+size (16.5px) so placeholders inherit correctly.
Live-driving the real browser caught it (the user's report: "they don't
work as buttons"): the stats row sits OUTSIDE the composer shell's
pointer-events-auto div, inside the sticky scroll wrapper's
pointer-events-none — every control on the row was unclickable, including
the pre-existing stats-detail popover button. The row now re-enables
pointer-events itself.
Also, per the user's sizing note: bar text 10px → 11px with a touch more
breathing room, and the model + thinking controls gain quiet pill
affordance (rounded-full border, card bg, ▾ chevron) — still muted by
design, but visibly buttons.
Verified live end-to-end in the browser: click opens the model menu ABOVE
(366 rows, search autofocused), "qwen flash" narrows to 5, picking
qwen3.8-flash persists + flips the label, the thinking menu shows exactly
that model's supported levels (Off/Minimal/Low/Medium/High), Medium
persists on the record via PUT, labels update. Test session settings
restored after the drive. Gates 350/350.
The model-control wave (user-decided design, 2026-09-12): ONE controller
for the conversation agents (prober/teacher are functionally one agent);
background sub-agents stay config-pinned. pi-ai owns the registry — the
shipped 366-model OpenRouter catalog (cost, context, modality,
thinkingLevelMap), getSupportedThinkingLevels/clampThinkingLevel for the
per-model thinking truth, and streamSimple's automatic clamp. No second
registry to drift out of sync.
Seams:
- config.ts: ReasoningEffort retired for pi's ThinkingLevel
(off|minimal|low|medium|high|xhigh|max). GLM roles go honest at "low" —
pi's catalog says GLM-5.3-flash accepts exactly low|high|max, and the old
"minimal" was silently clamped to low by streamSimple since the swap;
nothing changes behaviorally. visualMaker → qwen/qwen3.8-flash (fixes
the dead qwen/qwen3.8-max slug — the catalog only ships -0902; also the
user's cost pick, 0.15/0.47 per-1M).
- src/agent/models.ts: piEngineFor(role, settings?) — provider+id resolved
from the catalog, thinking level from settings ?? config, CLAMPED to the
model's supported list (a settings change between models can never send
an invalid level). PiEngine gains `level` (observability/tests).
- ChatRecord.settings {model?: {provider, id}, thinkingLevel?} — written
by PUT /api/chats/[id]/settings (replace-whole, no updatedAt churn),
read by chat-handler at turn composition: agents are stateless per turn,
so the change is next-turn-effective by construction. The turns log
still records the ACTUAL billed model (the honest witness).
- GET /api/models: the configured-provider catalog as client-safe DTOs
(core/contracts/models.ts) + config defaults; a second provider joins
the picker by registering it in piModels() — nothing else changes.
- PUT settings validation gate (src/server/model-catalog.ts): unknown
provider/model or unsupported level → 400 carrying the supported list,
never a silent clamp.
- scripts/sync-pricing.ts: regenerated PRICES/CONTEXT_LIMITS from pi-ai's
shipped openrouter.json (offline; same source as the engine + picker).
All 366 models priced; glm's stale 0.15/0.5 → pi's 0.075/0.25, limit
1310720 → 1048576 — cost is inferred at read time, so history re-prices.
- UI (components/model-picker.tsx): stats-bar right side becomes model +
thinking buttons; menus open ABOVE; fuzzy search over "provider id
name" (core/fuzzy.ts — pi's /model pattern); model rows show name/id/
price/context; the thinking menu lists exactly the chosen model's
supported levels (GLM: low|high|max — no dead options, no "off").
Model switch drops an incompatible pinned level (falls back to config,
which clamps honestly) rather than carrying a 400.
Tests (350/350, +15): catalog projection + defaults, the settings gate,
pi thinking truth pinned against upgrades (glm low|high|max; minimal→low;
off→low — GLM cannot disable thinking), the engine factory's
override/fallback/clamp/throw matrix, fuzzy ranking, store round-trip,
and the re-priced folds (usage, stats bar, activity dock).
Live smoke (disposable driver, deleted): catalog route 366 + defaults;
glm+minimal → 400 + supported; turn 1 billed qwen/qwen3.8-flash under
the override; mid-session switch flipped turn 2 to glm; clear-overrides
restored the config pin. ALL CHECKS PASS on real OpenRouter.
The custom-message scheme (user-decided 2026-09-12): system-authored
user-role messages are NOT human messages — they carry marker data parts
only, the adapter owns their model-facing wording, and the UI renders
them under control. Nothing fake is ever persisted as human speech.
The inventory (all migrated):
- probe→teacher handoff wake: was a text-bearing user message rendering
as a fake user bubble; now data-probe-handoff marker-only — the
adapter flattens it to HANDOFF_TEXT (core/contracts/chat.ts), the UI
renders a quiet 'Preparing your plan…' system line.
- plan accept/decline: was boilerplate text ('The plan is accepted…',
'I'd like the plan revised…') masquerading as learner speech — the
decline wasn't even marked; now data-verdict {accepted, at} on its own
message, adapter flattens to verdictText(), UI renders the quiet
verdict marker ('✓ Plan accepted' / '↻ Plan revision requested').
sendVerdict sends TWO REAL MESSAGES when the learner typed a note
(the note as a genuine user message, then the verdict) in ONE turn
via setMessages + sendMessage() — no 409 dance, no double teacher turn.
- sub-agent steers: the abort steer gains its data-subagent-aborted
marker part; the UI keys steer rendering off PARTS (artifact/aborted),
not metadata — metadata stays as pump mechanics only.
- record-adapter: systemTextOf() flattens marker-only messages; steers
keep their text (payload — briefs are content, not boilerplate).
phases.ts: plan→teach advances on data-verdict{accepted:true} only;
learner text alone can never sanction teaching (pinned). data-plan-
accepted retired, no back-compat (user-decided: wiped-data doctrine).
Pinned: record-adapter.test.ts (verdict/handoff flattening, note+verdict
order, steer payload kept, UI-only parts skipped), phases.test.ts
(verdict marker semantics, out-of-order, text-never-fakes), jobs.test.ts
(abort marker part), demos.test.tsx (badge renders off the part, not
metadata).
Live GLM smoke (disposable driver, deleted): probe→conclude→wake→plan→
decline-with-note (revised, still plan)→accept-with-note (teach) — ALL
CHECKS PASS, including marker-only handoff/verdict on the wire and zero
fake human speech in the transcript.
Gates: tsc, lint, build, 335/335.
The architectural answer to the validation-crash question: validation IS
first at the model-facing boundary (pi validates arguments before
execute; failures return to the model same-turn — all three live
incidents retried successfully). But the same bytes also exist UNVALIDATED
by design in two other places: the live wire streams raw tool-input
deltas DURING generation, and the transcript keeps the raw input of
REJECTED calls (honesty + the replay invariant). The consumers were
supposed to trust part STATES — but that trust was enforced by
TypeScript casts ('part.input as X'), which are compile-time lies with
no runtime check behind them. That is the hole both crashes came
through, and it should not have been possible.
The structural rule now: model-authored INPUT is consumed only through
parsers running the SAME zod schema the model faced — boundary-valid ⇔
consumer-parseable, mechanically. Server/learner-authored OUTPUT stays
trusted (our code authors it).
- src/core/tool-parts.ts (new): parseQuizInput (display parse over the
shared quizInputBase — strips the teacher-only nodeId, accepts both
prober and teacher boundary shapes), parseAskInput, parseProbeDoneInput.
- contracts/probe.ts: probeDoneInputSchema is now core-owned zod — the
probe_done tool declares it (single source of truth), the seam runs
the literal same schema.
- question-dock.tsx: the dock parses its input at entry; unparsable
renders the quiet malformed-question fallback, never a crash.
- probe-done-card.tsx: the input preview goes through the parse; the
old state reasoning and shape guards collapse into it.
Pinned in tests/tool-parts.test.ts: both exact live crash shapes
(stringified quiz options, stringified assessment), boundary-parity
cases for all three schemas, nodeId stripping, mid-stream partials;
render-cards pins the dock fallback markup.
Gates: tsc, lint, build, 328/328.
Two live bugs from the modal-realism stress runs:
1. Multi-select ask: picking the custom 'Other…' row cleared every
already-picked option, and picking any option cleared a typed custom
answer — single-select displacement logic applied to both modes. The
dock's selection is now a pure state machine in question-core
(toggleOption/toggleOther over DockSelection, unit-tested): single
displaces (unchanged), multi coexists — neither channel clears the
other; the typed text always survives.
2. ProbeDoneCard crashed on a REJECTED probe_done call: the model
stringified 'assessment' (the same double-encoding habit as the
options bug), zod rejected the call, the model retried — but the
card's input-preview fallback trusted the errored part's unvalidated
input shape and died on assessment.strands.map. The preview now reads
input only while the call is pending or streaming (guarded with
Array.isArray); an output-error part renders a quiet
malformed-and-retried note instead of the garbage.
Pinned: question-core.test.ts (3 dock-selection machine cases),
render-cards.test.tsx (4 probe-done card cases incl. the exact live
stringified shape, plus the multi-select ask markup pin).
Gates: tsc, lint, build, 315/315.
Live observation (modal-realism run): the teacher batched two quizzes in
one turn; the dock showed the SECOND question first, so the answering
order was reversed vs the saved record and the transcript cards.
Cause: findPending scanned parts BACKWARD within a message, so with two
input-available quizzes it picked the last one first. No overwriting was
involved — parts are created in call order on both wire and record; the
scan direction alone reversed the interaction order. (pi executes a
parallel batch with starts + result messages in call order; the
execution-end order is completion-racy, which also allowed a millisecond
dock-subject flip mid-turn.)
Fix A — src/core/pending.ts: parts now scan FORWARD within a message
(messages stay newest-first): the FIRST unanswered question is the dock
subject; the second appears only after the first is answered.
Fix B — src/agent/lh-tool.ts: question tools opt into sequential
execution (pi runs the batch in call order when any called tool opts
in), so tool-input-available chunks always arrive in call order and the
dock subject can never flip mid-turn. Question executes are instant.
Pinned in chat-flow.test.ts: a parallel two-quiz batch asserts wire
order (A's available chunk before B's), part persistence order, and the
dock selection sequence (A first, then B after A is answered).
Gates: tsc, lint, build, 307/307.
Live crash (modal-realism session): GLM double-encoded the quiz options
array as a JSON STRING inside the tool-call arguments. The bridge emitted
tool-input-available at toolcall_end — BEFORE validation runs — so the
client briefly saw the malformed quiz as a pending question and the dock
crashed in normalizeOptions ((string).map). The later tool-input-error
chunk could not revive the tree.
The old SDK gated this with InvalidToolInputError: validation ran before
the input-available chunk. The bridge now holds the part in
input-streaming until tool_execution_end and only then emits
tool-input-available (clean) or tool-input-error (invalid, structural
JSON Schema failure or the zod re-run inside execute).
Server-side behavior was already correct (the record shows the error
part + a successful model retry) — this is purely the wire-ordering hole.
Pinned in chat-flow.test.ts: the refinement-layer retry (correctAnswer
mismatch) and the exact live structural shape (string options) both
assert the invalid call never becomes input-available on the wire.
Gates: tsc, lint, build, 306/306.
The Vercel AI SDK survives only as Band D (useChat client + UIMessage
record types); @openrouter/ai-sdk-provider is uninstalled.
- src/agent/models.ts: PiEngine {model, streamFn} via createModels +
openrouterProvider subpath; thinkingLevelFor maps config effort none->off
- src/agent/lh-tool.ts: neutral LhTool (zod authoring, optional execute,
clientQuestion flag); toPiTool converts via z.toJSONSchema and re-runs
zod inside execute so refinements ride the error-result retry path
- src/server/pi/record-adapter.ts: UIMessage records -> pi Messages
(dangling tool calls get synthesized interrupted results)
- src/server/pi/ui-stream.ts: SSE bridge emitting AI-SDK-shaped chunks so
the useChat client parses unchanged (continuation id rule, question
tools never emit output, tool-input-error for bad authoring)
- teacher-agent.ts: stateless per-turn pi Agents; shouldStopAfterTurn
stops only when every question-tool call resolved cleanly (errors retry)
- chat-handler.ts: buildAgent(outgoing) factory seam (stateless doctrine);
assertValidUiMessages owns the retired SDK validation gate
- sub-agents via shared runSubagent (pi-run.ts); extraction stages stop
after a clean submit; visual-maker keeps its hand-rolled loop over
direct streamSimple calls; aborted model calls bill no usage
- tests: all model seams ported to pi-ai faux provider (scripted
FauxResponseSteps, agent factories); handleChatRequest second param
is now an agent factory
Gates: tsc, lint, build, 305/305. Live GLM smoke (disposable driver,
deleted): prober streaming + persistence, question seam, turn-2
continuation same message id, per-turn real token usage - ALL PASS.
Data wiped at the pivot (chats/graph/jobs, .gitkeeps kept).
Two user-reported problems from the muscle-protein-synthesis session,
both diagnosed from the live data (chat_mtxeml9e_6118e59f).
The freeze: after probe_done, the open session never showed the
teacher's wake turn — only navigating away and back loaded it. Not an
agent-switch problem: the server-side handoff worked perfectly
(write-ahead persist, liveness registry, the sidebar's busy dot saw the
wake the whole time). The gap was client attach: the only mid-session
wake detector the open page has is the jobs poll, and probe-phase
sessions have zero job traffic, so that poll is OFF; the post-turn
adoption poll read turnInFlight=false at ~0ms (the wake fires at
~0.75-1.2s, after the pump's grace), adopted, and stopped. Fix: the
post-turn adoption effect now keeps a short WATCH WINDOW after adopting
(~4s, 8x500ms) — if turnInFlight flips true, it hands off to the
existing wake-adoption poll (busy shimmer, adopt at finish).
The mermaid failures: four failed plan_init calls, all caused by ONE
hidden rule — underscores are forbidden in node ids (mps_translation,
net_balance, per_meal). The id regex rejected '_' BEFORE the slug check,
so the specific, actionable message never fired; the generic 'can't read
the edge — expected a[title] --> b[title]' lied (the syntax was exactly
that), and the teacher spent three attempts hunting punctuation it
suspected (colons, commas, multi-word labels) — visible in its own
reasoning trail. Per the user's principle — enforce only the slugs,
free everything else: underscores are first-class slug chars; the id
CANDIDATE class is broad so every bad id reaches the specific slug
message; chained edges (a --> b --> c) expand to pairs; ==>, -.-> are
edge synonyms; any direction header accepted (we layout ourselves);
id["title"] quote pairs are stripped. The tool description, zod
describe, and the teacher's planning prompt now state the one strict
rule instead of the old subset liturgy.
Verified: tsc, 305/305 (the exact muscle-protein mermaid is now a
passing test), lint, build; real-browser puppeteer smoke on live GLM —
the teacher's wake turn appeared in the open page ~50s after the probe
concluded with ZERO navigation, verdict gate + first teach check +
plan window all live in-page (the single console 404 is the app's
missing favicon, environmental noise).
The accept-plan UI lands (docs/probe-teach-split.md §4, formerly the
interim rule): presenting the plan no longer moves the phase — the
learner's ACCEPT does. While the presented plan awaits its verdict, the
composer's send square becomes two stacked buttons (green Accept plan /
red Decline); typed text rides the verdict as his note, Enter keeps
sending plain side questions (phase unmoved), decline asks for a
revision and the buttons stay.
The seam: accept sends a user message carrying a data-plan-accepted
part (PlanAcceptedData in contracts/chat.ts, TeacherUIDataParts
'plan-accepted'); phaseAt3 reads it as the plan→teach event (plan_init
demoted to presentation-only); convertToModelMessages drops data parts,
so the teacher sees only the text. Mid-lesson rewrites need no fresh
verdict — acceptance happens once, at the first presentation.
The teacher's planning prompt now ends the turn at presentation and
waits for the verdict (never teaches into an unaccepted plan).
Gates: tsc clean, 304/304 (phases tests rewritten to the acceptance
seam incl. decline/side-question/failed-init cases; composer verdict
render tests), lint, build. Live GLM smoke ALL PASS: phase pinned at
'plan' after plan_init, presentation turn carried NO check, accept
click flipped the summary to 'teach', then tagged checks moved the map
(current=rates, visited=composite); zero console errors.
The successor of the injected plan blocks (docs/probe-teach-split.md §6):
always on once a tree exists, a zoomed-in window (2:1, ~2¼ node widths)
centered on the CURRENT node, re-centering by pure derivation as tagged
checks move the lesson — statuses derive live from the transcript on
every render (derivePlanStatuses), glow on current, fallback focus on
the lesson's first node, click opens the existing full-DAG popup.
Bottom-left, mirroring the activity dock's bottom-right; hidden on
narrow viewports; the learner's visual polish pass comes later.
PlanDiagram gains a viewBoxOverride (exported); latestTree exported.
Tests: viewBox centering/clamping (pure) + render (hidden pre-plan,
windowed viewBox, glowing current, starting fallback).
Emission runs at persistTurn — after the whole turn is composed — so
appending the block buried it under the same turn's teaching prose and
quiz calls (user-observed in chat_mtwgb9lj: the DAG appeared only after
the first check). Splice it in immediately after the (last) successful
plan_init part instead. Two positional tests added.
Diagnosis (2026-09-11 user testing, minimal-effort GLM): sessions stuck on
'plan' forever. Two causes: (1) the three-phase view read the stored tree's
currentNodeId, which nothing ever writes post author-once (2026-09-03) — a
structural dead seam; (2) the teacher stochastically left quiz.nodeId
untagged, and done-by-quiz-result semantics would have falsely completed
nodes anyway.
The redesign (docs/probe-teach-split.md) abolishes derived occupancy in
favor of explicit events:
- Two agents over one transcript: createProberAgent (quiz without a
nodeId field, ask, probe_done) and createTeacherAgent (full toolset,
plan_init onward). The chat-handler construction branch keys on a
probe_done part in the transcript. MODELS.prober added.
- probe_done: server-side seam tool — carries the learner assessment
(strands + free-form edge read), refuses while a question is open or
the probe is already concluded, and queues a handoff steer through the
generalized wake pipeline (steers now dedupe by id, not jobId; job
finalization only for job steers). The teacher auto-wakes and authors
the plan; the learner sees the assessment via the probe-done card.
- phases: probe ends at a concluded probe_done, plan ends at a
successful plan_init — strictly sequential, no skipping, no legacy
compatibility (data wiped at the pivot per doctrine).
- Plan map = position, not completion: derivePlanStatuses reads only
tagged quiz CALLS (state/outcome feed nothing); current = latest
tagged node with consecutive duplicates collapsed, visited = earlier
distinct stays (a revisit stays in both — current wins at render).
PlanTree is structure-only; currentNodeId/doneNodeIds and the dead
mutation helpers are gone. No fallback current.
- Prompts split per agent (prompts/prober/, prompts/teacher/ over the
trimmed shared base); graph-context injection removed entirely
(shelved, docs/probe-teach-split.md §5); knowledge-graph context no
longer reaches either agent.
- Plan blocks are full-only snapshots at plan_init (neighborhood
machinery retired; the persistent DAG window is the future UI wave);
usage capture unchanged.
Gates: tsc clean, 294/294 tests (phases/plan-tree/plan-blocks/question-
tools rewritten to the new semantics; probe-done + handoff-wake tests
added), lint, build. Live GLM smoke PASS: prober concluded with an honest
assessment, teacher auto-woke, authored a 10-node plan with the block
attached, then tagged checks on slope-instinct → secant derived current/
visited exactly as designed; grading honest both ways (driver off-by-one
proved a wrong answer grades false).
Live smoke exposed two bugs in the turn usage log:
- onFinish messages carry no ids (withMessageIds fills copies inside
persistTurn), so fresh turns keyed messageId "" — provenance lost.
- The SDK coalesces continuation turns into ONE response message (a whole
probe conversation lands in a single assistant message), so per-messageId
idempotency swallowed every continuation turn's usage.
TurnUsage gains id: string (unique per request, the append idempotency
key); messageId stays as transcript provenance, taken from the id-filled
list. appendTurnUsage dedupes by id; continuation turns with the same
messageId each keep their usage.
- src/core/plan-blocks.ts: pure emission builder — full block on successful
plan_init, neighborhood block when the derived plan state moved since the
newest existing block; the as-of state reads from the block's frozen
snapshot (transcript quiz parts mutate in place on answer, so
re-derivation is post-mutation blind). Idempotent by construction.
- turn-service persistTurn enriches with withPlanBlocks at the canonical
write; chat.tsx adopts the record once the server agrees the turn is
done (endTurn flips only after persist).
- chat-handler: per-request usage collector via agent-level onStepFinish
(StepResult.usage), appended at onFinish to ChatRecord.turns via
appendTurnUsage, idempotent per response message id.
Two-part root cause (found by the lh-ui stream + master together): (1)
src/store/jobs.ts updateJob unconditionally overwrote updatedAt with now(),
silently clobbering any explicit patch timestamp — JobPatch types it as
patchable but the adapter ignored it, so order-sensitive tests rode on
same-millisecond tie-breaking (failed ~4/5 in isolation). The store now
honors patch.updatedAt ?? now() — semantics-preserving for every existing
caller, which never passed updatedAt explicitly. (2) the activity-feed
test's finally wiped the ENTIRE shared data/jobs dir — with node:test
running test files as concurrent processes, that deletes other processes'
live fixtures; cleanup is now scoped to the test's own created ids.
The plan now lives IN the chat (plan blocks, wave-4 item 3) and background
processes live in the activity dock (item 1), so the right sidebar's
tree panel + sub-agent fleet column is retired from the chat view:
- delete components/tree-panel.tsx (three-mode rail/list/graph panel) and
components/sub-agents-panel.tsx; their graph interaction machinery lives
on in plan-block.tsx and fleet observability in activity-dock.tsx.
- app/layout.tsx drops TreePanelModeProvider (the mode no longer exists).
- chat.tsx drops the xl:flex right column; the middle column is the view.
- tests: render-tree-panel.test.tsx deleted; the sub-agents panel suite
removed from demos.test.tsx (dispatch cards' jobs-context stays — the
central poll drives transcript dispatch cards).
- components/plan-block.tsx: hand-built SVG (user decision) — deterministic
layered layout (Kahn topo + longest-path depth, rows centered, authoring
tie-break), bezier edges, the retired panel's status color language.
- PlanBlock renders data-plan-block parts AS CARRIED (frozen snapshots,
never re-derived): kind full = borderless, viewBox = layout extent so
nodes hug the column edges, natural height; kind neighborhood = own
small labeled 'Plan fragment' container with the current node glowing
(pulsing primary halo).
- PlanPopup: full-screen overlay with the LIVE plan — latest tree from the
transcript, statuses derived at click time via derivePlanStatuses
(contracts/plan.ts). Esc / backdrop / X close; drag pan, wheel
zoom-to-cursor, zoom buttons; fit-floor semantics harvested from the
retired tree panel (zoom floor = the fit, pan locks at it).
- chat.tsx: data-plan-block case in the part switch + popup mount.
- src/core/ui-message.ts: TeacherUIDataParts gains "plan-block": PlanBlockData
(additive, type-only — lane touch flagged to master).
- tests/render-plan-block.test.tsx: pure layout (layering, centering,
dangling-edge tolerance), wrap, frozen-snapshot render, live derivation.
Extraction is real spend and background work; the dock needs to see
and stop it. User decisions: persist all cost data (chat record),
per-stage usage granularity, and TRUE cancellation for abort.
USAGE CAPTURE (live + persisted):
- extraction.ts: per-call onStepFinish captures each billed step's
usage (SDK LanguageModelUsage -> CallUsage: input/output/cache
read+write/reasoning) into the registry's live per-stage usage
(extractor/auditor; the merger is deterministic — no model call).
- /api/graph/status runs now carry usage (model + steps per stage);
cost/context folds are the consumer's (core/prices.ts — nothing
stores dollars).
- ChatRecord.extractions: append-only per-run usage log (mirrors the
turns log): startedAt key, at, durationMs, merge stats when the
merger ran, stages. Persisted on success AND failure — a failed run
still spent money. Idempotent per startedAt; the list payload
(ChatSummary) never leaks it.
- EmissionStats moved to contracts/graph.ts (its contract home);
core/graph.ts re-exports for existing importers.
- Verified end-to-end against the real pipeline (backfill one chat):
per-step token counts, reasoning tokens, and the auditor's growing
prompt-cache reads captured and persisted.
TRUE ABORT (the dock's abort button, user decision):
- RunningExtraction owns one AbortController per run, shared across
phases; abortExtraction(chatId) fires it and returns whether a run
was aborted (idempotent no-op otherwise). The pending model call
receives the signal — the provider cancels the request, spend
stops. throwIfAborted() at phase boundaries ends runs aborted in
sync gaps.
- Abort is a deliberate stop, not a failure: same ledger discipline
(nothing merged, nothing checkpointed, partial usage persisted) but
the run reports zero emissions instead of throwing. Exported via
src/server/observability.ts for master's /api/activity/abort route.
- Tests: mid-run abort (hanging mock that rejects on the signal),
live-accumulation + persisted-record assertions, store-seam
idempotency, and the summary-payload leak guard.
Wave 4 (agent-side observability + control), reconciled with the dock
contract (master relay 2026-09):
- abortJob in src/server/jobs.ts, exported through the server barrel for
POST /api/activity/abort (master's thin route): fires the job's
AbortController (real kill — the in-flight model call aborts, not just
the next loop check), drops any queued result steer, flips the record
to error "Aborted by learner" + activity "aborted" synchronously, and
queues a one-line abort steer (buildAbortSteer) through the same wake
pipeline so the teacher's promise doesn't hang in the lesson. Shares
the result steer's deterministic id space, so abort + racing result
can never both deliver (steerDelivered dedupe).
- Abort registry (src/agent/subagents/aborts.ts) on globalThis, mirroring
the wake pipeline's module-state pattern; runner registers/unregisters
per run, so the route's module instance sees the loop's controller.
- Runner usage capture: LoopContext (abortSignal + onUsage) threads into
all three kind loops; per-step usage folds via accumulateJobUsage into
JobRecord.usage (model from config MODELS, absent-never-zero), flushed
in runJob's finally so an aborted job's money record survives the kill.
- Contracts: JobUsage token fields optional + lastStep (most recent
call's accounting, for the dock's per-blob context line vs
CONTEXT_LIMITS); SubagentResultMeta.aborted; AbortSteerMessage;
buildAbortSteer; SteerMessage union generalized.
- demo-builder/visual-maker accept abortSignal + onUsage (visual folds
generateText usage per iteration).
- stillRunning/finishJob/jobContext guards: an aborted job is never
overwritten by the runner (terminal no matter what the wake does);
failJob and the pump's finalizeJob keep the flip.
- UI: abort steer renders a quiet aborted chip (job-aborted-badge).
- Prompt 20-jobs.md: one line about aborted jobs.
Tests: accumulateJobUsage folds (undefined-preservation, lastStep),
runner usage capture, real-kill abort (sync flip, silent unwind,
abort steer wakes), usage-so-far survives the kill, aborting a
delivering job drops the queued result steer, outcome reasons,
abort-chip render. 330/330, typecheck/lint/build green.
One quiet row beneath the composer/question shell (not flush to the
screen bottom; static height, outside the morphing shell):
- LEFT: context window + session cost, clickable → per-turn detail
popover (context, total, unpriced count, one row per turn with
model/steps/cost). All folds from core/prices.ts — sessionCost,
contextWindow, turnCost — no cost math in components.
- RIGHT: model + "Thinking —" — a placeholder slot for the coming
model/thinking-level CONTROL (later work with its contract changes);
display-only today, model read from the last turn's usage,
pretty-printed (Glm 5.3 Flash).
- Data: the record's turns log, seeded on mount, refreshed when the
local stream settles, kept fresh by the existing record re-syncs.
No turns yet → "—" everywhere; unpriceable turns never fold into a
fake $0 (the doctrine).
- formatUsd/formatTokens moved to components/usage-format.ts (shared
with the activity dock); prettyModel joins them, pure + tested.
One reserved corner of the shell, one blob per running BACKGROUND
process — session sub-agents (research/demo/visual) and global
extraction runs. Teaching turns are deliberately NOT here: the dock
exists to surface what the learner can't otherwise see; a running
teaching turn is already fully visible where it happens (user decision).
- Data: ONE poll of GET /api/activity (jobs across ALL chats,
title-joined, oldest-first + extraction runs), folded client-side
with core/prices.ts (jobCost). Adaptive interval — 2.5s while
anything runs, 10s idle, 4× slower on hidden tabs; a failed fetch
keeps the last feed (no flicker to empty on a blip).
- Per blob: kind icon, elapsed (ticking), cost, activity line, and an
explicit END on the blob's face. Click → detail popover (context
line, model, steps, open-session link). The context line is the LAST
model call's tokens vs the model's limit (JobUsage.lastStep — pending
agents-stream contract; "—" until it populates), NEVER a cumulative
sum. Absent data renders "—", never zero (the doctrine).
- Header: total running cost (sum of priced blobs, "—" when nothing
is priced). Narrow viewports collapse to a count badge → drawer.
- END posts to /api/activity/abort {kind, id} — the backend is being
composed by master over the streams' mechanisms; until it exists a
click locks the button and the popover says "Abort not available
yet" (never a dead control).
- Idle renders NOTHING. Mounted once in the root layout; every view
gets it for free.
toBlobs / formatElapsed / formatUsd / formatTokens are pure + tested;
SSR render tests pin blobs, the "—" doctrine, the face END, and the
priced total.
The dock is visible in every view, so it needs every running process —
sub-agent jobs across ALL chats (not just the open one) plus the global
extraction runs — in one poll. /api/jobs stays chat-scoped and untouched:
it is the transcript machinery's poll and carries turnInFlight for the
re-attach seam (chat-transport), the seam we spent three live rounds
stabilizing; making it polymorphic would fork its response shape by mode
and put regression risk on exactly the wrong path. The dock is a different
consumer with a different shape (cross-chat, running-only, cost-bearing),
served by a thin route over server/observability.ts:
- activityFeed(): running jobs from every chat (title-joined, oldest-first
on last activity) + extraction runs + busyChats (chats with a teacher
turn in flight — the biggest spender; data made available, rendering is
a dock decision).
- GET /api/activity — the dock's single poll everywhere.
- Cost math stays client-side (core/prices.ts folds); raw usage rides the
records and the transport stays thin.
Decided with the user over the polymorphic-/api/jobs and
separate-all-jobs-endpoint alternatives.
The 2026-09 observability wave's type-level foundation, landed before the
streams so nothing renders against guessed shapes:
- contracts/chat.ts: CallUsage + TurnUsage (per-model-call token accounting,
one entry per teacher/wake turn keyed by messageId) and ChatRecord.turns —
a DELIBERATELY parallel append-only usage log, not per-message metadata:
the client re-POSTs its message list every turn and its copies never
carry server-stamped fields, so anything inside messages gets clobbered
by the next persist (the wave-3 merge-rule lesson).
- contracts/chat.ts: PlanBlockData — the persisted data-plan-block part
(full DAG on plan_init, one-level neighborhood on plan updates, snapshot
semantics so history renders what was true then).
- contracts/jobs.ts: JobUsage + JobRecord.usage — a sub-agent run's
accumulated accounting, written by the runner.
- core/ports.ts + store/chats.ts: ChatStore.appendTurnUsage — idempotent
per messageId (abort replays / wake retries append once); listChats no
longer leaks turns into the list payload.
- core/prices.ts: PRICES + CONTEXT_LIMITS (generated from OpenRouter's
published pricing via scripts/sync-pricing.ts) + pure folds: callCost,
turnCost, sessionCost, contextWindow, jobCost. Unknown model or absent
fields render as absent, never zero.
- server/observability.ts: barrel-approved surface for the transport layer
(fixes the import-law violation in the graph status route that slipped
through the wave-3 merge verification).
Capture wiring (agent onStepFinish -> persist seam, plan-block emission)
lands next as master seam work; the streams start against these shapes.
Wave 3: the plan protocol collapses to one re-callable mermaid plan_init
(seven mutation tools deleted by design; statuses derive from quiz checks;
live quiz-nodeId validation with mid-turn retry). The steer-starvation arc:
steers deliver coalesced with the learner's answer (claimSteers), persist
dedupes same-id continuation copies, jobs settle-sync defers while a turn
is busy. Research steers gain the collapsed full-brief affordance. Model
reasoning effort set to minimal across roles.
With continuation coalescing, a job flips done at CLAIM time — while the
learner's own answer turn is still streaming. The poller's settle re-sync
fired on the flip, swapping in a mid-turn record snapshot mid-stream; the
steer after the assistant message broke the SDK's in-place continuation
merge, and the streaming response landed as a SECOND message with the same
id — duplicate React keys in the live transcript (the user's live test hit
it; the record itself stayed clean).
A settle during a busy turn (local stream OR server turnInFlight) now marks
a pending sync; the first idle poll tick adopts the finished record. This
also fixes the latent multi-job race: steer B settling while steer A's wake
turn is in flight would have grabbed the same mid-turn snapshot.
Out-of-lane surgical touch (flagged for master review): the wave-1 poller
block in components/chat.tsx.
amosblomqvist