Extraction is real spend and background work; the dock needs to see
and stop it. User decisions: persist all cost data (chat record),
per-stage usage granularity, and TRUE cancellation for abort.
USAGE CAPTURE (live + persisted):
- extraction.ts: per-call onStepFinish captures each billed step's
usage (SDK LanguageModelUsage -> CallUsage: input/output/cache
read+write/reasoning) into the registry's live per-stage usage
(extractor/auditor; the merger is deterministic — no model call).
- /api/graph/status runs now carry usage (model + steps per stage);
cost/context folds are the consumer's (core/prices.ts — nothing
stores dollars).
- ChatRecord.extractions: append-only per-run usage log (mirrors the
turns log): startedAt key, at, durationMs, merge stats when the
merger ran, stages. Persisted on success AND failure — a failed run
still spent money. Idempotent per startedAt; the list payload
(ChatSummary) never leaks it.
- EmissionStats moved to contracts/graph.ts (its contract home);
core/graph.ts re-exports for existing importers.
- Verified end-to-end against the real pipeline (backfill one chat):
per-step token counts, reasoning tokens, and the auditor's growing
prompt-cache reads captured and persisted.
TRUE ABORT (the dock's abort button, user decision):
- RunningExtraction owns one AbortController per run, shared across
phases; abortExtraction(chatId) fires it and returns whether a run
was aborted (idempotent no-op otherwise). The pending model call
receives the signal — the provider cancels the request, spend
stops. throwIfAborted() at phase boundaries ends runs aborted in
sync gaps.
- Abort is a deliberate stop, not a failure: same ledger discipline
(nothing merged, nothing checkpointed, partial usage persisted) but
the run reports zero emissions instead of throwing. Exported via
src/server/observability.ts for master's /api/activity/abort route.
- Tests: mid-run abort (hanging mock that rejects on the signal),
live-accumulation + persisted-record assertions, store-seam
idempotency, and the summary-payload leak guard.
The dock is visible in every view, so it needs every running process —
sub-agent jobs across ALL chats (not just the open one) plus the global
extraction runs — in one poll. /api/jobs stays chat-scoped and untouched:
it is the transcript machinery's poll and carries turnInFlight for the
re-attach seam (chat-transport), the seam we spent three live rounds
stabilizing; making it polymorphic would fork its response shape by mode
and put regression risk on exactly the wrong path. The dock is a different
consumer with a different shape (cross-chat, running-only, cost-bearing),
served by a thin route over server/observability.ts:
- activityFeed(): running jobs from every chat (title-joined, oldest-first
on last activity) + extraction runs + busyChats (chats with a teacher
turn in flight — the biggest spender; data made available, rendering is
a dock decision).
- GET /api/activity — the dock's single poll everywhere.
- Cost math stays client-side (core/prices.ts folds); raw usage rides the
records and the transport stays thin.
Decided with the user over the polymorphic-/api/jobs and
separate-all-jobs-endpoint alternatives.
The 2026-09 observability wave's type-level foundation, landed before the
streams so nothing renders against guessed shapes:
- contracts/chat.ts: CallUsage + TurnUsage (per-model-call token accounting,
one entry per teacher/wake turn keyed by messageId) and ChatRecord.turns —
a DELIBERATELY parallel append-only usage log, not per-message metadata:
the client re-POSTs its message list every turn and its copies never
carry server-stamped fields, so anything inside messages gets clobbered
by the next persist (the wave-3 merge-rule lesson).
- contracts/chat.ts: PlanBlockData — the persisted data-plan-block part
(full DAG on plan_init, one-level neighborhood on plan updates, snapshot
semantics so history renders what was true then).
- contracts/jobs.ts: JobUsage + JobRecord.usage — a sub-agent run's
accumulated accounting, written by the runner.
- core/ports.ts + store/chats.ts: ChatStore.appendTurnUsage — idempotent
per messageId (abort replays / wake retries append once); listChats no
longer leaks turns into the list payload.
- core/prices.ts: PRICES + CONTEXT_LIMITS (generated from OpenRouter's
published pricing via scripts/sync-pricing.ts) + pure folds: callCost,
turnCost, sessionCost, contextWindow, jobCost. Unknown model or absent
fields render as absent, never zero.
- server/observability.ts: barrel-approved surface for the transport layer
(fixes the import-law violation in the graph status route that slipped
through the wave-3 merge verification).
Capture wiring (agent onStepFinish -> persist seam, plan-block emission)
lands next as master seam work; the streams start against these shapes.
The background extraction pipeline is invisible on the home surface —
essential observability per the user's directive: the graph view must
show what is currently running (extracting / auditing / merging) and
for which chat.
- extraction.ts: module-level registry of in-flight runs (chatId,
phase, startedAt), updated at each phase transition in runExtraction
and cleared in finally (done or failed). extractionStatuses() is the
read; one process, one registry.
- GET /api/graph/status: thin transport over the registry, chat titles
joined from the chat store.
- graph-view: polls every 2s and shows one chip per running pipeline
(spinner + phase + title + elapsed seconds, bottom-left); when a run
finishes (some -> none) the graph refetches, so merges land live.
Status is cosmetic — poll errors never surface.
- Unit test drives a mocked pipeline run and observes the registry
mid-flight: 'extracting' during the extractor's generate, 'auditing'
during the auditor's, empty after.
chat.tsx woven by hand: UI's session-surface layout (transcript column +
right sidebar as full-height siblings, h-10 header with PhaseChip) wrapped
in sub-agents' JobsContext.Provider; the message map carries both the
steer ternary (steers bypass the Message wrapper — the model-facing glue
text never renders) and UI's PhaseDivider emission (steers are verified
phase-boundary no-ops). Right sidebar = sub-agents' fleet panel column
(SubAgentsPanel above TreePanel, xl-only) carried onto UI's skeleton.
contracts.ts auto-merged (ChatSummary.phase/pendingQuestion + ChatListItem,
additive above the async-protocol section).
Smoke-verified on the merged tree: steer renders bare as its artifact
badge with no learner bubble, no rendered glue text, and no spurious
phase divider at the steer boundary.
The persisted mode (lh:tree-panel-mode) was read inside TreePanel — a
per-page component that remounts on every session navigation. First
paint rendered the SSR-safe "list" default, then the mount effect
applied the stored mode: with the animated frame, opening a session
visibly opened the panel and then closed it. The mode now lives in a
root-layout provider (the SidebarProvider pattern) and survives
navigations untouched — the panel mounts already in the right mode.
The default stays "list" so SSR and first client render agree; the
stored preference applies once, on provider mount.
The pi-style ack protocol (previous commit) left the client describing the
old world: artifact cards read the ack string as a result ("could not be
displayed"), the only job poller rendered on input-available parts that no
longer exist (so wake turns landed invisibly until re-navigation), steers
rendered as learner bubbles with no SVG anywhere, and server-persisted
assistant messages carried empty SDK ids (duplicate React keys).
- persistTurn (turn-service): fill empty message ids at the canonical write
— server-only messages (last turn's, wake turns') now key uniquely;
client-authored ids untouched.
- chat.tsx: ONE central /api/jobs poll (fast while jobs run, slow discovery
cadence, off for job-less sessions) drives every job surface, re-syncs
once when a job settles (the record already holds the steer + wake
response at flip time), and adopts server-side wake turns via the same
turnInFlight machinery as returning mid-turn. Steers branch to the
artifact card — never a bubble. Composer no longer gates on jobs.
- /api/jobs now reports turnInFlight alongside the job list.
- dispatch-card: the tool-call box — live activity by toolCallId while
running, quiet "delivered"/"failed" one-liner after; the ack is visible
observability, the artifact lives in the steer.
- steer-card: renders the delivered artifact by kind (research brief /
sanitized SVG / sandboxed demo iframe) from the data-subagent-artifact
part; result renderers extracted and exported from the artifact cards.
- sub-agents-panel: right-sidebar section above the plan tree listing
running sub-agents (kind, elapsed, live activity) — vanishes when idle.
- tests: dispatch/steer/panel render tests re-target the new protocol;
persistTurn id-hygiene suite. 229/229 green, build green.
Out-of-lane touches for master review: chat.tsx + components/** (cleared
this wave), app/api/jobs/route.ts, src/server/turn-service.ts, contracts
protocol doc §2. Live-verified on :3003 (full loop, SSR DOM checked).
Redesign per review feedback: the colored phase strips were too busy —
removed. One small dot now carries two channels:
- COLOR = phase (sky/violet/emerald, same system as chip + dividers)
- FILL + MOTION = what's owed:
- pulsing = teacher is working (the only state that deserves motion)
- solid = a question awaits the learner's answer — the dominant
resting state of real sessions, so deliberately calm
- hollow = idle, nothing owed
The amber attention channel is gone entirely (it would read as a
phantom fourth phase). The meta line now leads with the useful fact:
"Waiting for your answer · 5m ago" / "Teacher is working…" — the
message count only shows when idle. Ordering stays pure recency.
- SessionRow (extracted, exported for tests): leftmost phase strip
(bookmark tick, per-phase color: probe/plan/teach) + status dot left of
the title — pulsing amber attention dot when an answer is owed, quiet
breathing dot while the teacher works, nothing when it's the learner's
turn. A pending answer outranks 'working'.
- globals.css: phase + attention color tokens (one system shared with the
upcoming header chip and dividers) and the two dot animations, both
disabled under prefers-reduced-motion.
- SessionsSidebar polls /api/chats every 2.5s while open (payload is tiny
now that busy/phase/pendingQuestion come from the list) and pauses while
the tab is hidden; fetch-on-navigation preserved.
- core/phases.ts: Phase3 (probe/plan/teach) + pure derivations —
phaseOfTree (point-in-time, from the stored plan tree: no nodes → probe,
no occupied node → plan, occupied → teach) and phaseAt3 (monotonic
transcript fold for the dividers: failed plan_init never marks a seam;
phase only advances).
- contracts.ts: ChatSummary gains record-derived phase + pendingQuestion;
ChatListItem adds liveness-merged busy.
- store/chats.ts listChats computes the record-derived fields (tree +
pending question scan).
- api/chats route merges busy = turn running || running background job.
Lane crossing granted by master: contracts/store/api (schema arbitration).
Data-only inference: pre-plan planning reads as probe until the plan lands.
The bug: answering a quiz and navigating away mid-turn, then returning,
re-prompted the same question. Root causes: (1) the answer was durable only
at onFinish — every mid-turn view read the stale pre-answer record;
(2) the server had no notion of a turn in flight; (3) remount treated the
stale record as current, with no re-attach path. Worse, the stale dock let
the quiz be answered again, racing two turns into a last-writer-wins
saveMessages clobber.
Fix (four pieces):
- Write-ahead persist in chat-handler: the request body IS the client's
canonical state — saved to disk BEFORE the stream starts. The answer is
durable the instant the request arrives; crashes recover via the existing
mount continuation.
- Turn liveness registry (src/server/turn-liveness.ts): in-memory chatId →
in-flight map with a TTL leak guard; beginTurn rejects concurrent turns
(409) so one turn ever owns a chat. endTurn runs in onFinish AFTER
persistTurn, so pollers that see not-in-flight always read a finished
record; a turn-start failure clears the registry (no lockout).
- GET /api/chats/[id] reports turnInFlight; the chat page passes
initiallyInFlight to the client.
- Client re-attach (components/chat.tsx): while in flight, busy shimmer +
composer gated + mount continuation suppressed; poll /api/chats/[id] until
the turn finishes, then syncFromServer adopts the completed transcript.
Stop stays local-stream only (a server turn has no client request to
abort).
Verified by scripted-model tests: concurrent turn 409 + registry lifecycle,
write-ahead durability mid-turn, failed-turn lockout guard.
The shell now imports through exactly two barrels, enforced by
import/no-restricted-paths (npm run lint):
components/** -> "@/src/core" (pure core API, browser-safe)
app/** -> "@/src/core" + "@/src/server" (server entries)
Zone violations are lint errors with architecture-pointer messages; the
pure-core modules (contracts, question-core, plan-tree, phases, pending,
math-delims, ui-message) may never import agent/server/store.
Supporting moves (direction: IO layers -> core):
- core/index.ts: the core public API barrel
- core/ui-message.ts: moved from src/agent (it was shell-facing all along)
- quiz/ask input zod schemas moved agent/tools -> core/question-core
(the question contract, alongside the grading logic)
- ChatRecord/ChatSummary moved store/chats -> core/contracts (single schema)
- GraphSnapshot/NodeStateClass moved core/graph -> core/contracts (the
/api/graph payload IS a contract; the pure renderer needs only these)
- src/server/{chats,graph,jobs}.ts + index.ts: thin server entries for the
transport layer; GET /api/graph's fold side-effect now lives behind
graphView() instead of inline in the route
Gates: tsc clean, 190/190 tests, eslint clean (law verified to fire on
store/agent/subpath violations), production build clean.
(Repo repair en route: a git object (app/chat/[id] tree) was missing from
the object store — reconstructed byte-exactly via mktree from its blob;
the broken node_modules that stalled the build was cleanly reinstalled.)
- src/core/math-delims.ts: pure preprocessing for the \(...\)/\[...\] math
contract — balanced delimiters rewrite to the two-dollar forms remark-math
parses ($ is never math). Skips code fences/spans, escaped delimiters, and
leaves unbalanced (mid-stream) delimiters untouched. Display math renders
as a block: mid-paragraph gets blank-line padding; indented (list items)
degrades to inline so list continuations don't half-parse.
- message.tsx: transform in the render path (children-memoized); math plugin
now singleDollarTextMath: false — the currency hazard dies structurally.
remark-breaks via the cjk remarkPluginsAfter slot (the bare remarkPlugins
prop would replace the [remark-gfm, codeMeta] defaults).
- mermaid: dark theme + sans font via Streamdown's mermaid prop; dark-styled
error fallback (components/mermaid-error.tsx) with error, source in
<details>, and retry.
- globals.css (appended): @source for streamdown dist — its internal UI
chrome (code blocks, mermaid containers, tables) was entirely unstyled
because Tailwind v4 never scans node_modules; --color-sidebar → card; wide
mermaid SVGs capped to the column; .katex-display scrolls.
- tests/render-markdown.test.tsx: 26 tests (transform unit + render). The
math describe block in render-cards.test.tsx is updated to the new
contract ($...$ now renders as literal text — pinned).
- src/store/jobs.ts: per-job JSON store in data/jobs (create/get/update/list,
dedup by toolCallId)
- src/agent/tools/jobs.ts: research tool — client-side (no execute), turn ends
at the call; input per contracts ResearchToolInput
- src/agent/tools/web.ts: server-side web_search (Google CSE) + web_fetch
(Readability/Turndown/linkedom, Jina fallback) — errors return as strings,
never thrown into the agent loop. Deps: @mozilla/readability, linkedom,
turndown (+@types/turndown dev)
- src/agent/subagents/researcher.ts: isolated ToolLoopAgent (web tools only,
12-step cap), researcher prompt ported from pi
- src/agent/subagents/runner.ts: spawnPendingJobs hook + runResearchJob —
fire-and-forget, activity line per tool call, fulfills the part in the
LATEST chat record (output-available) before flipping job status; failure
still fulfills the part so the teacher learns the outcome
- src/server/chat-handler.ts: onFinish spawn hook after saveMessages
- app/api/jobs/route.ts: GET ?chatId= polling endpoint
- components/artifacts/job-card.tsx + registry entry: polling activity card
(2s) → syncFromServer on finish; research-brief card (markdown + sources)
- src/agent/prompts/20-jobs.md: research doctrine for the teacher
- tests/jobs.test.ts: store CRUD, spawn hook via scripted mock teacher,
runner completion/failure with fake model + stubbed fetch
- Chat now remounts per session (key={chat.id}): fresh StickToBottom,
fresh DockOverlay (no cross-session morph from the previous shell
height), fresh useChat.
- Conversation initial="instant" (was "smooth"): the mount scroll to
bottom is a jump, not a slide.
- DockOverlay skips the pending-question scroll/expand effect for the
session's INITIAL pending question (dock starts expanded; the initial
scroll positions us) — previously it fired a second smooth
scrollToBottom right after the library's initial scroll (the visible
'pause, then the quiz pops and pushes everything up').
- The dock-variant fade/rise no longer plays for the session's opening
variant; reserved for mid-session transitions.
Verified live: client-side switch settles at d=1px from the bottom within
one frame of content swap, dock open from the first frame, both
directions (quiz session, plain session).
- Animation: the blob's shell morphs height (WAAPI, 180ms) on every
variant change — chat ↔ question ↔ collapsed bar — plus a quick
fade/rise (160ms) on the swapped content. Gotcha pinned in code: never
measure the shell while a morph is in flight (getBoundingClientRect
reads the ANIMATED value and corrupts the next morph's start height);
cancel running morphs before measuring, refresh settled height only
when no morph is running.
- Gradient-over-scrollbar fixed structurally: the dock is no longer an
absolute overlay on the conversation root — it's sticky bottom-0 INSIDE
the scroller (in flow). The transcript scrolls fully clear of the dock,
the fade never touches the scrollbar lane, and the pb-hack clearances
are gone.
- New question while at the bottom scrolls to reveal the dock; scrolled
up → collapsed bar, no yank.
- Thin dark native scrollbar app-wide (8px webkit + scrollbar-width:thin,
dark thumb, transparent track). Note: use-stick-to-bottom sets
scrollbar-gutter: stable both-edges on its scroller — the lane is
outside the content flow by design.
- 'Writing a question…' transcript shimmer while a tool call streams its
input (same slot as 'Thinking…'). Protocol pinned by a test asserting
tool-input-start precedes tool-input-available; scripted mock now
streams tool input as deltas like real providers do. (OpenRouter/
Anthropic often delivers the whole input in ONE delta — then the dock
appears instantly and no indicator is needed or shown.)
Verified live: grow 54→349px and shrink 349→54px fire with correct
from/to; collapse/expand intact; gradient stops before the gutter;
Thinking… observed in-page; 58/58 tests.
Sidebar (replaces the modal drawer):
- Inline, toggleable, no backdrop/focus theft — can stay open while working
- Open state persists across pages (provider in root layout) and reloads
(localStorage; starts closed server-side to avoid hydration mismatch,
stored value applied post-mount; default open on first run)
- Hamburger in page headers only when closed; close button in sidebar header
Floating composer blob (per reference screenshot):
- Floats over the transcript (absolute overlay), persists at bottom while
scrolling, content fades behind it (gradient mask); pb-44 scroll clearance
- Concentric corner radii: blob rounded-2xl(16px) − p-1.5(6px) = send
rounded-[10px] = 25% of the 40px button; CornerDownLeft icon per reference
- Stop moved from header into the blob (busy → square icon)
- prompt-input.tsx + 7 orphan ui files pruned (attachments/command palette
unused); hand-rolled Composer is ~100 lines and render-tested
Pending-question morph (one text field on screen, never two):
- Quiz/ask-select pending → blob becomes the note field (send hidden, hint
line above); card's internal note input removed; Submit consumes it
- Ask free-text pending → blob becomes the answer field; its send submits
via the card's registered submit api (ref bridge); Enter works
- Chat owns the single text state; card clears it on submit
- canSend for answer mode mirrors the card predicate directly — reading the
ref during render lagged one commit (found in browser testing)
Widths: one shared column (max-w-3xl + px-4) for transcript, cards, and
composer — all three measured pixel-identical (left 416, width 736).
Assistant Message max-w-full/MessageContent w-full (was max-w-[95%]/w-fit —
the 'cards are a bit shorter' bug).
User messages render plain text again — no math pipeline on learner-authored
text (currency hazard accepted for the model only, guarded by the prompt).
Tests: 58 pass (5 new composer pins; card tests updated to controlled
props). Verified live: note flow, answer flow, sidebar persistence, float,
fade, zero console errors — plus the live model spontaneously honoring the
$ currency escape contract in a continuation.
Layout:
- Home = knowledge-graph placeholder (main view) with new-session input
docked below; session list moved out of home into a hamburger drawer
- SessionDrawer: slide-over session navigation (backdrop/ESC close, active
highlight, fetches /api/chats on open) on both home and chat headers
- Chat header loses the back button — the drawer is the navigation
Math (inline $ was silently not rendering):
- @streamdown/math defaults singleDollarTextMath:false — now
createMathPlugin({ singleDollarTextMath: true }) in message + reasoning
- katex.min.css imported in layout (streamdown does not inject plugin
styles; display math only looked right via browser MathML fallback)
- QuizCard/AnsweredCard route question/options/explanation through
MessageResponse; user chat bubbles render markdown+math too
- Currency hazard: micromark-extension-math@3.1 lacks pandoc's
closing-digit rule, so bare '$5 and $10' parses as math — defense is at
the source: system prompt now instructs escaped \$5 / 'USD' for money
Hydration fix (React #418):
- QuizCard shuffled options with Math.random() — server/client orders
diverged. shuffleOptions now takes an injectable RNG; QuizCard passes
seededRandom(toolCallId) (mulberry32/FNV-1a in question-core). Same
question → same stable order across reloads; still varies per question.
Tests: 53 pass (5 new math-rendering pins incl. fenced-vs-one-line $$
behavior and the escaped-currency contract). tsc + next build clean.
Verified live in browser: inline/display math in text + cards, drawer
open/navigate/active-highlight/ESC, no console errors, no font 404s.
The teaching agent no longer runs on pi. Stack:
- ToolLoopAgent (ai@7) + OpenRouter (OPENROUTER_API_KEY), model via LH_MODEL,
default anthropic/claude-sonnet-4.5 with reasoning tokens
- quiz/ask_user_question = client-side tools (no server execute): the call
lands as a typed tool part in input-available state; the card grades via
src/core/question-core.ts; addToolOutput sends the verdict; the teacher's
reaction continues the same assistant message
- resume-mid-quiz is free: the pending tool part persists in the transcript
- persistence: one JSON per chat (data/chats/), full UIMessage transcripts,
saved onFinish (also on abort)
- UI: AI Elements (owned copy-paste components, pruned to the five in use)
+ button-based QuizCard/AnsweredCard (ask answers never graded)
- pi fully removed: TUI, extensions, Fastify+WS server, bridge, custom
protocol all deleted (recoverable from git history)
- tests: question-core units carry over; chat-flow integration drives
handleChatRequest with MockLanguageModelV4 (48 tests, zero tokens);
render smoke tests guard the cards (renderToStaticMarkup)
Carried over unchanged: teacher system prompt, question-core, schema/
ledger/derive/extract pipeline, data model.
amosblomqvist