b18294948d40bb2035fb96e6339016366f07dc14
64
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 amosblomqvist
|
b18294948d |
feat(keys): LH_IGNORE_ENV_KEYS — the fresh-user switch + dev-fresh launcher
The owner needs exact new-user simulation as a one-command launch (repeatable first-run BYOK testing). Option A, owner-approved: env-hiding switch + launcher script. Fix 1 — LH_IGNORE_ENV_KEYS=1 in the keys seam: when set, every env key is invisible app-wide; only auth.json counts. The mechanism is boot-time DELETION, not seam-local filtering: pi-ai resolves provider keys from process.env NATIVELY at every request (verified — auth/context.js's defaultProviderAuthContext reads live env, no boot snapshot), so deleting the managed vars from process.env in initAuthEnv is the ONE mechanism that hides keys from every consumer — model engines included, plus the three direct process.env readers (visual-maker, pdf-reader, openRouterApiKey). realEnv stays EMPTY under the switch (nothing counts as env), and the fill-gap still reflects auth.json keys so a fresh user can add keys mid-session and engines see them with no restart. keyValue reads file-only under the switch as belt-and-suspenders (a post-boot env reintroduction stays invisible to the seam). Unset switch = today's behavior, byte-for-byte (server-keys.test.ts pins it). Fix 2 — scripts/dev-fresh.sh (npm run dev:fresh): fresh throwaway LH_DATA_DIR (.fresh-data/, wiped each launch, --keep preserves), and — the subtle bit — LH_AUTH_FILE redirected INTO the fresh dir: dev's AUTH_FILE default is the repo-root auth.json, which may symlink the real keys; the fresh instance must never see it. Sets the switch, dedicated port (default 3005, --port overrideable, standing assignments 3000-3004 refused), prints URL + data dir, safe to rerun. .fresh-data/ gitignored. Test: tests/fresh-env-keys.test.ts (own process — the switch is boot-time): the engine-path deletion asserted directly, the env/file collision resolved to the file, status rows report file sources only, mid-session saveKey/deleteKey round-trip, idempotent re-init, and the unset-switch env-wins read seam live-checked. Live-verified on 3005 via the launcher: zero federated models, all providers keyPresent false, no present keys, no auth.json/settings.json in the fresh dir; then mid-session ZAI save → 7 zai models federate instantly, source 'file' — while .env.local's real keys stayed invisible throughout. Gates: typecheck, lint, 863/863, build. |
||
|
amosblomqvist
|
9387418864 |
feat(models): clamp-on-write for pinned thinking levels + DEFAULT_THINKING_ORDER in the core barrel
Fix 1 — clamp, never reject (owner ruling, interlude 2026-09-22): a model swap with a pinned thinking level must JUST WORK. The owner hit the write-side 400 live (Claude Fable 5.1/xhigh → GLM 5.3 flash: "z-ai/glm-5.3-flash does not support thinking level xhigh"). The supported-level verdicts in validateChatSettings and validateModelsSettings are DEAD — the WRITE clamps the pin to the new model's nearest supported level (pi's clampThinkingLevel, the same translation piEngineForSpec applies at runtime) and stores the clamped result; the server is the truth — the record shows the clamped level after write. New pure helpers in model-catalog.ts, applied by ALL THREE write routes (section, defaults, per-chat — one verdict seam, so every surface that used the old 400 gets the clamp uniformly): - clampModelsLevels: every slot's level clamps against the SLOT'S OWN model (the parser already requires a model on each slot write). - clampChatLevel: the body's own model, else the record's resolved strong slot (the chain's fallback); no model anywhere → rides unchecked (runtime clamps at engine build; the turn gate refuses the model-less chat anyway). Fix 2 — DEFAULT_THINKING_ORDER moves from config.ts to core/settings (core barrel, AUTH_KEYS precedent — one source of truth, no drift). The UI imports it to auto-populate a thinking level on FRESH model picks; the ordering has no other client role. Legacy unset records keep the runtime defaultThinkingLevel fallback at engine build untouched. Import sites updated: agent/models.ts, tests. Verified live on 3004: the owner's exact case stores max (not 400); a bare chat-level write clamps to low against the strong slot's glm; unknown provider/model still 400s with the provider list. Tests: the level-verdict suites became clamp suites (the owner's case pinned verbatim: xhigh→max, minimal→low, off→low, supported pass-through, level-less slots ride); the machine-aware expectBareLevelVerdict chain is retired — clampChatLevel takes the strong slot as an explicit param, so the tests are pure again. Gates: typecheck, lint, 839/839, build. |
||
|
amosblomqvist
|
bfa906e2f8 |
feat(models): BYOK all the way down — factory pins deleted, per-need refusal gates
Owner decision 2026-09-22: 'it doesn't make sense to have any default
model configured anywhere; the application is bring your own keys.'
config.ts: MODELS deleted; ModelRole becomes an explicit 9-role union;
DEFAULT_THINKING_ORDER (medium, low, high, minimal, xhigh, max) bakes
the unset-level default — first entry the model supports; explicitly
set levels keep pi's native nearest-clamp.
core/settings-resolution: the factory layer dissolves — the chain is
role override → slot → NOTHING. resolveSettings(record) takes no
factory. Empty chains resolve to null/undefined; a thinkingLevel-only
slot is inert (stored, never run). resolveConversationSpec: chat.model
override over the strong slot, null when both empty. resolveRoleSpec
spans all nine roles for the job/extraction engines.
Refusal envelope (core/settings.ts): modelsNotConfigured(missing) —
{error, code: 'models_not_configured', missing: [slot tiers]} +
ModelsNotConfiguredError with .envelope. Per-need gates, each honest:
- chat-handler: universal turn gate (new + continuing turns) returns
503 before beginTurn — no liveness, no write-ahead persist, no graph
trigger before the gate fires.
- runner.runJob: unresolvable engine lands the job as error with the
envelope wording; no phantom loop, no result steer.
- extraction: refuseIfStagesUnconfigured throws before any stage
(turn-service info-skips so coverage retries next turn; the import
route surfaces per-file code+missing); labeler-less runs skip
cluster naming with a console line and still merge.
Write side: slot entries must carry a model — level-only slot writes
are 400s (absent slot key = clear). piEngineForSpec resolves unset
thinking levels via DEFAULT_THINKING_ORDER; piEngineFor now throws
honestly; thinkingLevelFor/modelIdFor deleted; teacher-agent, pi-run,
visual-maker, pdf-reader all read engine.level.
model-catalog: defaults/slots/rolePins null when the chain is empty
(contracts nullable) — no phantom ids in payloads. Stores drop
MODEL_PINS entirely.
Tests: pin-leaners converted to explicit injection (per-chat model
pins on scripted chats — exactly what the setup flow writes; faux
engines for agent-construction tests). New coverage: the 503 gate
envelope, the runner refusal, the extraction refusal + labeler skip.
Gates: typecheck, lint, 815/815, build.
|
||
|
amosblomqvist
|
f6626f1703 |
Fix wave chunk A: no Default option, composer = strong slot, catalog revalidation
Item 1 — no Default in the slot selectors (owner, fix wave): a "default" model is not a thing in this app. Picking Default used to delete the slot and re-arm the first-run gate — incoherent. Now the model menus carry NO clear row at all (a slot holds a pick or reads "—"), the thinking menus carry a "—" row (menu-none) that clears only the thinking pin, and the hint line is the roster only — the resolved factory pin is never named. Master's amendment confirmed: the factory-pin hint is SUPERSEDED entirely (Platform removes the pins). Item 2 — the composer control IS the strong slot (owner: "one and the same"): ModelControl (model-picker.tsx) reads the raw models section (GET /api/settings) and the federated catalog, and writes PUT /api/settings/models via composeSlotModel — the same seam the settings Agent section and SetupDialog use. Props retired: chatId, settings, onSaved, fallbackModelId, noDefaultModel — the control is self-sufficient and identical on every surface (composer, home dock, /chat/new). The per-chat model PUT is gone from the UI; legacy per-chat overrides keep resolving server-side. StatsBar/DockOverlay plumbing slimmed accordingly; chat.tsx drops its per-chat settings state and initialSettings prop. Item 3 — the composer catalog cache no longer freezes per page load: fetchCatalog(force) invalidates, and every model-menu open revalidates in the background (stale-while-revalidate — cached list shows instantly, the fresh federation swaps in). Owner's repro verified live: a key saved mid-session widens the menu without a reload. Item 4 — stale doc comments fixed: model-catalog.ts's "OpenRouter catalog"/"multi-provider future" (Platform's file — mechanical, flagged for review) and model-picker's retired "config-pinned"/ per-chat-PUT narrative. Live-verified on 3001: unpicked slots read "—" everywhere; model menus have no clear row; the "—" thinking row clears the pin but keeps the model; composer pick writes strong and preserves cheap/ vision siblings; thinking pill writes strong.thinkingLevel; groq key added mid-session appears in the open menu without reload. Caught in live verification: the record fetch was mapping the WHOLE settings record into the models section (picks never wrote) — fixed to read body.settings.models. Gates: typecheck, lint, 807/807, build. |
||
|
amosblomqvist
|
d85d8a58a0 |
fix(tools): wire the searchAvailable gate + research/pdf in the probe phase
Fix wave (2026-09-20), two owner-approved fixes: Fix 1 — the searchAvailable gate was dead code on the live wiring path: createTeacherTools RECEIVED ctx.searchAvailable but dropped it when building the inner JobToolContext, so jobTools' research gate always took its ?? true default (jobTools-direct tests passed; the production path never saw the signal). Threaded through — the gate now reads the live server signal on the teacher registry too. Fix 2 — the prober registry gains the research + pdf dispatch tools (owner ruling, 2026-09-20): these two are phase-omnipresent — the prober can verify facts or read learner PDFs while assessing. demo/visual stay teach-phase-only (they need the lesson's plan context). Same gating as the teacher side: toggle-filtered, research subject to the search gate (honestly absent when no engine is live — the prober's dispatch closes over the SAME jobTools builder, one gate, one semantics). probe_done stays always-on-by-construction; the learner-fs set untouched. - ProberToolContext (extends ProbeDoneContext with the job wiring); createProberAgent takes it; chat-handler injects jobStore + spawn + searchAvailable into the prober ctx. - TeacherToolContext.spawn's stale input union fixed (PdfInput was missing from the TYPE — runtime already passed it). - Tests: the threaded-gate describe in web-search.test.ts pins the LIVE factory path both ways (teacher + prober); question-tools' registry key-sets gain research/pdf; the prober toggle test pins the research:false path. Gates: typecheck, 807/807 tests, lint, build — all green. |
||
|
amosblomqvist
|
372d7f9975 |
feat(extraction): stage engines resolve the slot chain — engineFor on the pipeline
Chunk B, item 3 of the search port + slots wave (2026-09-20): the flagged gap closed — ExtractionModels deps were engine VALUES, keeping extractor/auditor/labeler config-pin-bound while the job runner went slot-native. The pipeline now takes the runner's exact treatment: - ExtractionPipelineDeps.engineFor: (role) => PiEngine — resolved FRESH per stage call, so a settings PUT between stages lands. Direct test values (extractor/auditor/labeler) remain the priority seam, checked FIRST; absent values fall to the chain; absent chain → config pins (unchanged). - All six stage call sites (chat + notes extract/audit, the cluster namer x2) resolve through stageEngine(role, direct). - chat-handler's extraction composition injects the full chain — piEngineForSpec(resolveRoleSpec(role, readSettings(), MODEL_PINS)) — same wiring as the job runner. Chunk B, item 4 (session-level slot-anchored model picks): VERIFIED already complete from Platform's model-slots wave — the conversation engine chain (chat-handler) anchors on effective.models.strong, the picker's default surface (catalogDefaultsFor) falls back to strong, both pinned in tests. No phase split (user decision, 2026-09-20). Capability-filtered pickers live in the settings UI (main 343be10). Tests: engineFor fills the stages end-to-end (both propositions merged through the chain, consulted in pipeline order) + the direct-value priority pin (engineFor that throws when consulted). Gates: typecheck, 795/795 tests, lint, build — all green. |
||
|
amosblomqvist
|
ed264c936d |
feat(search): web_search onto the injected port — Google CSE retired, research honestly disabled
Chunk A of the search port wave (2026-09-20), per master's assignment: - web.ts: the Google Custom Search backend is gone — web_search now runs on the core search port (SearchProvider), injected as a dependency (import law: the tool never imports server code). createWebTools(search) resolves the LIVE provider fresh per call; null = the honest unavailability string (no silent fallback, owner decision). web_fetch unchanged (needs no engine). - runner: JobRunnerDeps.searchProvider — the server composition threaded into the researcher; jobs already in flight when a key disappears still fail honestly through the same seam, never crash. - jobTools: the research dispatch tool DROPS from the teacher's registry when ctx.searchAvailable() is false (owner decision: registry-level drop — the model never sees the tool, no doomed dispatch, no error card). Absent seam = available (tests). chat-handler injects server/searchAvailable + searchProvider. - tests: stubFetch (the Google wire stub) retired; jobs.test.ts threads a recording stub provider through all four research runners and asserts the scripted query hit the port; new web-search.test.ts covers the factory contract (null → honest string, hits, error strings, fresh resolution, web_fetch independence) + the registry drop matrix. Gates: typecheck, 793/793 tests, lint, build — all green. |
||
|
amosblomqvist
|
f9e3a4c43d |
feat(models): strong/cheap/vision model slots + multi-provider engine federation
Model slots wave (2026-09-20), chunk B:
- settings: new `models` section {strong, cheap, vision} slots (each
field independent, same shape as a role override); ROLE_SLOT maps all
nine roles to tiers (strong=prober+teacher, cheap=researcher/
demoBuilder/extractor/auditor/labeler, vision=visualMaker+pdfReader);
resolution chain per role: override → slot → factory pin.
- config: ModelSpec gains provider (all pins openrouter, the BYOK
default); MODEL_PINS in stores.ts now spans all nine roles (the strong
slot's factory rep IS the prober pin).
- engine: piModels() registers EVERY builtin provider catalog;
piEngineForSpec(resolved spec) is the new engine factory (pi-ai
resolves per-provider keys from env itself); piEngineFor stays as the
config-pin fallback. chat-handler resolves the conversation chain
(chat override → strong slot → pin) and injects engineFor into the job
runner, closing the old gap where per-role overrides never reached
job engines.
- /api/models: models federate ONLY keyed providers (providers surface
with keyPresent, env-wins, fresh per call); slots (resolved tier pins)
join rolePins; catalogDefaultsFor inherits the strong slot; the models
settings section gets the same catalog gate as chat.
- pricing: PRICES/CONTEXT_LIMITS uniformly keyed provider:id
(regenerated from ALL pi-ai catalogs, 1354 keys); usage stamps
(TurnUsage, JobUsage, extraction stages) switch to provider:id —
owner decision: no legacy compat, old stamps render unpriced.
|
||
|
amosblomqvist
|
f1373d5de5 |
feat(search): Tavily + Brave behind a search-provider port; Google CSE retires
Search port + model slots wave, chunk A (owner design, 2026-09-20):
- core/search.ts: SearchHit + SearchProvider port + SearchStatus —
pure, browser-safe; adapters arrive as injected dependencies.
- server/search.ts: Tavily + Brave adapters (wire shapes VERIFIED
against docs.tavily.com's OpenAPI + Brave's dashboard docs — Tavily
auth is the Bearer header, not the legacy body api_key; Brave is
X-Subscription-Token) + engine resolution per owner decisions: a key
ALONE activates its engine (no engine choice needed), explicit
agent.search.engine pins the preference (both keys: Tavily first),
a pinned engine with a missing key is honestly unavailable — no
silent fallback, no default engine.
- Curated keys catalog: GOOGLE_SEARCH_API_KEY/GOOGLE_CSE_ID retire
(CSE closed to new customers, dies Jan 2027 — stored values become
preserved unknown vars); TAVILY_API_KEY (tvly- shape) + BRAVE_API_KEY
(length guard) join. UI keys-section TOOL_IDS now derives from the
catalog — cannot drift.
- GET /api/search: {available, engine, reason} — the honest status for
the settings gray-out; researcher-disable semantics ride the same
signal (Teacher stream rewires the tool onto the port).
- agent.search settings field (schema + sanitizer + strict parser);
web.ts's WebSearchResult is now the core SearchHit (re-exported).
|
||
|
amosblomqvist
|
ec38f91a2b |
feat(keys): free-form model-provider keys — pi-ai provider table on /api/keys
Provider-keys interlude (owner spec, 2026-09-19): any pi-compatible
provider's key is manageable, not just the curated set.
GET /api/keys now also returns providers: {id, envVar, label}[] — every
pi-ai provider that takes a plain api-key env var (33 today), derived
from pi-ai's own provider→env-var table. No-drift mechanism: the table
is generated from pi-ai (builtinProviders + findEnvKeys over a
truthy-for-every-var proxy env) and pinned by
tests/auth-provider-keys.test.ts, which re-derives the truth at test
time and asserts EXACT equality — a pi-ai upgrade fails the gate loudly.
Owner-spec exclusions: github-copilot (OAuth), amazon-bedrock (AWS),
google-vertex (ADC), azure-openai-responses (base URL), the cloudflare
pair; anthropic keeps ANTHROPIC_API_KEY only. OPENROUTER_API_KEY
graduates from the curated catalog to the provider table (dedupe — the
UI pins it always-visible); curated stays Deepgram + the Google search
pair. Statuses: curated + present vars + unknown file vars — absent
provider keys never spam rows; the dropdown is discovery.
Write side: PUT/DELETE accept provider env vars (generic truncation
guard — trim, no whitespace, ≥12; family checks stay on the curated
three). Present-in-env provider vars surface as status rows so
env-wins/shadow semantics stay honest.
|
||
|
amosblomqvist
|
a3b6a495d3 | Merge branch 'feat/edge-glyph' | ||
|
amosblomqvist
|
a828d9b6d9 |
feat(models): rolePins on GET /api/models — the background roles' resolved pins
Settings interlude (owner request, 2026-09-19): the route now also returns
rolePins: Record<BackgroundModelRole, {model: {provider, id},
thinkingLevel}> — the six background roles' RESOLVED models (the user's
agent-models override falling back to the config.ts MODELS pin), so the
settings UI shows each role row's 'Default: <actual pinned model>' instead
of a generic config caption. Same resolution the agent factories use
(effectiveSettings), so the caption can never drift from what runs.
Pure projection (rolePinsFor) + the fs wrapper (catalogRolePins); pins
tested against config defaults and model/thinkingLevel-only overrides.
BACKGROUND_MODEL_PINS exported from stores.ts for the test (was local).
|
||
|
amosblomqvist
|
3e263a18cc |
Graph pipeline cost metrics: capture naming spend, fold all channels, cost pill + panel
- Cluster namer bills the money channel now: per-step usage + model captured via onStep, persisted as NamingRunUsage records to data/graph/naming-runs.jsonl (idempotent per startedAt; failed/aborted runs persist too — money spent either way). callUsage promoted to extraction.ts module scope for reuse. - core/cost-summary.ts: the pure fold behind the pill — chat runs, notes-source runs, and naming runs into stage totals (extraction / audit / naming) + newest-first recent runs. Dollars INFERRED read-time via prices.ts, never stored; unpriced models excluded-but-counted (render '—', never a silent zero); tokens always sum. - GET /api/graph/costs (src/server graphCosts composition, thin route). - Graph view: bottom-left cost pill (cumulative dollars headline) + expandable breakdown panel (stage rows with tokens secondary, recent runs with time/title/merged/cost); fetched on mount, refetched when a pipeline run finishes; merged into one bottom-left stack with the running-pipeline pills. - Tests: pure-fold suite, naming capture wiring (full importNotes -> naming path), store idempotency + strict reads. |
||
|
amosblomqvist
|
cbaf6df454 |
feat(keys): first-class BYOK key management — store, catalog, /api/keys
The auth/keys wave (2026-09-19): the server now owns the flat auth.json map directly (LH_AUTH_FILE from main.cjs; repo-root auth.json in dev, gitignored). The store reflects itself into the child's env at boot with fill-gap precedence — real env (incl. .env.local in dev) always wins — so every consumer keeps its plain process.env reads and saved keys take effect immediately, no restart; main.cjs's spawn-time injection retired (an env copy would shadow every settings-page change). Curated 4-key catalog (OpenRouter, Deepgram, Google search pair) with shape-only validation (no live pings — owner decision), 400s on bad pastes, last-4+length masking (plaintext never leaves the store), unknown file vars preserved + surfaced. Routes: GET /api/keys, PUT/DELETE /api/keys/:id. Tests: catalog/store/composition, with the env-sanitizing prologue (a dev shell may export real keys; tests use throwaway values only). |
||
|
amosblomqvist
|
524bf6024a |
feat(prompts): composeInstructions honors overrides — the P2 segment reconciliation
The other half-wired seam: Platform's settings 'agent.prompts' switches had no consumer in the agent layer, and their effectivePromptSegments mirrored the PRE-sources-v1 loader — the shared/ fragments dir (the learner-files doctrine) was invisible to it, so a naive delegation would have silently dropped the segment from both agents' prompts. - src/core/prompt-segments.ts (NEW, pure): the mirror-layout path grammar + segment types, defined exactly once; core stays zero-IO. - src/agent/prompt-segments.ts (NEW, the CANONICAL discovery): base + shared/ + per-agent fragments, override resolution, allSegmentPaths. Composition is an agent concern; the agent layer reads its own tree (the architecture law pins agent and store apart — the first direct refactor attempt tripped the enforced import/no-restricted-paths zone, hence the three-layer split). - src/store/prompt-overrides.ts: slimmed to the override-file CRUD behind the core port; shares the core grammar. Its doc no longer claims to be the canonical mirror. - teacher-agent: composeInstructions(kind, switches?) composes through effectivePromptSegments — a segment whose switch is ON and whose override file exists composes the override text. Switches-less calls use the cached pure-source composition (byte-identical to the old loader, pinned by test); AgentBuildOptions.promptSwitches threads effective.agent.prompts per turn, composing fresh so override-file writes flip the very next turn. - server barrel: re-exports the agent module's resolution for the settings routes (server→agent, the lawful direction). Tests: byte-identical source pin, override swap-in (on/off/file-absent), shared/ overridability in both agents, agent-construction threading, updated segment-list pins with shared/ in composition order. This file keeps owning PROMPTS_OVERRIDE_DIR (parallel-test discipline). Gates: 671/671, typecheck, lint, build. |
||
|
amosblomqvist
|
6c42393e42 |
feat(tools): the P2 toggle bridge — toolToggles first-class, definitive vocabulary
Reconciliation of the diverged tool-toggles seam (2026-09-19, master's P2): Platform injected the resolved delta as an ad-hoc intersection on AgentBuildOptions while the factories' enabled? param sat unread — the seam was inert on both sides. - teacher-agent: AgentBuildOptions gains toolToggles?: EnabledTools; both constructors thread it to the factories (createTeacherTools / createProberTools). An explicitly injected opts.tools registry is never filtered — the caller owns it. - Vocabulary ADJUDICATED with the user (2026-09-19): TOGGLEABLE_TOOLS is the definitive list — the job tools only (research/demo/visual/pdf). quiz/ask_user_question stay always-on forever (the prober's flow is graded-quiz brackets -> probe_done; quiz-off collapses assessment and teaching doctrine is built on graded verification — the wedge danger); the learner-fs set, plan tools, and probe_done also always-on. - settings-body (write side): unknown tool keys reject with 400 + the supported list, mirroring the prompt-switch validation; the read-side sanitizer stays lenient for forward-compat. - chat-handler: the intersection is gone (field now first-class) and the seam comment states the adjudicated vocabulary. Gates: typecheck, 28 new/updated tests green (tool-toggles-bridge, question-tools, settings-body), chat-flow + settings suites green. |
||
|
amosblomqvist
|
cb4fe25c89 |
wave 2 (platform stream): settings storage & plumbing foundation (P2)
- src/core/settings.ts: versioned DELTA record schema + sanitizer (unknown keys preserved)
— general/chat/agent{models,tools,prompts}/appearance/graph sections
- src/core/settings-resolution.ts: pure resolveSettings (factory defaults → user delta),
extraction/graph-context channel predicates
- src/store/settings.ts + prompt-overrides.ts: SettingsStore/PromptOverrideStore adapters
wired through stores.ts (effectiveSettings() fresh per call); lazy migration of
default-settings.json into record.chat
- app/api/settings/** routes (record, section bodies, prompt overrides per segment)
- chat-handler: graph gates at trigger call-sites + toolToggles injection seam
- config.ts: SETTINGS_FILE + PROMPTS_OVERRIDE_DIR paths
587/587 tests; typecheck/lint/build clean
|
||
|
amosblomqvist
|
487f5d4cea |
wave 1 (platform stream): background-observability accuracy + owner-driven follow-ups
- observability: liveness-gated stale-job recovery, extraction run registry on globalThis with explicit 'embedding' phase, /api/jobs force-dynamic, client refresh nudges at every seam - headless: providerSafePng canonicalizer (multi-IDAT Chrome PNGs → single-IDAT) fixing qwen 400s - composer: Enter routes through pure enterAction — accepts a pending plan verdict (owner decision) - markdown chrome: session-scoped mermaid overlay + md-blocks component set + pan-zoom math, corner-copy (clipboard grant in electron main), math/table/code block fixes, heading weight 900 543/543 tests; typecheck/lint/build clean |
||
|
amosblomqvist
|
bd3086194f |
feat: no baked-in default model — the new-session controls ARE the default settings page
Owner reframe (2026-09-18): the model + thinking + context controls on
the home graph view and /chat/new are not a draft overlay on a baked-in
default — they ARE the settings page for the default model, and the
user's last choice IS the default. BYOK first boot has no keys and
possibly no OpenRouter, so there is NO default to show: the controls
render 'no model' ('—') until the first pick.
The flash was the deeper symptom: the surfaces rendered from client
state that started empty and fetched truth afterward, so every visit
flashed the config pin (GLM/low — a dev artifact) for a few frames.
Fix at the root: both pages are now force-dynamic server components
that read the record (readDefaultSettings → initialSettings) and hand
it to the client — the first frame shows truth. Dynamic routes aren't
cached client-side (staleTimes.dynamic = 0), so every navigation
re-reads the record; the useDefaultSettings hook keeps only save-side
state (optimistic set + PUT). The GET-on-mount and its touch-guard are
gone.
ModelControl gains noDefaultModel (the default surfaces only): unset
settings render no model/level, never the catalog pin; the thinking
menu no-ops until a model exists. Chat surfaces keep the catalog
fallback — the engine runs the config pin when a session record has no
override, so the control keeps showing what will actually run.
Renamed draft→default throughout: data/default-settings.json,
GET/PUT /api/settings/defaults, useDefaultSettings. New sessions
still inherit the default on creation exactly as before.
Live-verified: first-boot 'no model' with zero flash, picker +
Shift+Tab cycle, values held across session→/chat/new→home roundtrips
and a reload (10×100ms samples, never the old default), zero console
errors.
|
||
|
amosblomqvist
|
a94a108e74 |
feat: draft settings persistence — shared model/thinking/context record for the new-session surfaces
Both new-session pages (the home graph view's topic dock and /chat/new)
held their model + thinking + knowledge-context choices in plain
useState(undefined) — every navigation reset them to config defaults.
Now both surfaces read/write ONE server-side record
(data/draft-settings.json via GET/PUT /api/settings/draft, the
render-settings pattern): whatever the learner last left the controls on
is what every later visit shows. useDraftSettings loads on mount with a
touch-guard (a slow GET never clobbers a fast choice), saves
optimistically. Session creation still carries the choice into the new
chat exactly as before; per-session settings are untouched.
The PUT validation is the exact per-chat contract — parseSettingsBody
extracted verbatim from app/api/chats/[id]/settings and shared by both
routes (unknown model/level → 400 with the supported list, never a
silent clamp). 400s on garbage, {} = the clear-overrides write.
Live-verified: Cmd+K+Enter model pick, Shift+Tab thinking cycle, context
slider — all held across home → session → home → /chat/new, and carried
into a created session's record.
|
||
|
amosblomqvist
|
597b3f43e1 |
fix: wake-turn steer delivered twice — claimSteers re-claimed the steer the pump's request already carried
The wake pump fires its turn with the steer INSIDE the request body but dequeues the queue entry only after the turn returns 200 — so the entry is still queued when the handler claims. The record doesn't carry the steer yet (write-ahead persist hasn't run), so claimSteers claimed it a SECOND time and the handler appended it: outgoing = [...body, ...claimed] held two copies of the same deterministic steer id, and the write-ahead persist saved that for the whole wake turn. The client's adoption poll rendered both -> React duplicate-key warning (steer_ctx_<chatId>). withMessageIds deduped only at onFinish, so the final record looked clean. Fix: claimSteers(chatId, incoming) — a queued steer whose id is already in the turn's incoming list is being delivered BY this very turn: finalized and resolved, never appended again. |
||
|
amosblomqvist
|
7902984bd6 |
feat: Electron desktop wrapper — LearnHarness.app (macOS arm64)
The web app becomes a desktop app without changing shape: the Electron
shell spawns the Next standalone server on a loopback port and points a
sandboxed window at it. All app logic stays in the Next server.
Bands:
1. LH_DATA_DIR seam (config.ts): PATHS.data/chats/graph derive from
process.env.LH_DATA_DIR (else repo data/, unchanged in dev). New
PATHS.modelCache routes the transformers.js embedding-model cache
(~300MB) to <data>/models when the seam is set — the default cache
inside node_modules is read-only in a packaged bundle. All existing
fs writes flow through PATHS; teacherPrompt stays ROOT-anchored
(read-only asset, ships inside the bundle by design).
tests/data-dir-seam.test.ts covers set/unset via child process.
2. Standalone build (next.config.ts + scripts/build-server.mjs):
output 'standalone'; outputFileTracingExcludes data/**/* keeps the
repo's real session data out of the bundle (verified: an unexcluded
scratch build traced 34 files / 716K of real chats + ledgers).
Two empirically-verified tracer gaps required outputFileTracingIncludes:
pi-ai's provider catalog (JSON import attributes — 39 files, 500s on
/api/models without them) and onnxruntime-node's sibling dylib
(libonnxruntime.dlopen via @rpath). build-server.mjs copies public/
and .next/static per docs, refuses a leaked data/ dir, and asserts
the runtime-read prompt .md assets are traced.
3. Electron shell (electron/main.cjs, no preload):
- single-instance lock (two instances would fight over the data dir);
- free port via listen(0); spawn via process.execPath +
ELECTRON_RUN_AS_NODE=1 with PORT/HOSTNAME=127.0.0.1/LH_DATA_DIR;
- BYOK: userData/auth.json, flat {VAR: value} JSON, mode 0600,
merged into the child env only when absent from the real env
(auth UI is the settings milestone; file is hand-editable);
- poll until the server answers (no ready event), then open a
contextIsolation+sandbox window; deny-by-default permission
handler (media allowed only from the loopback origin — mic/STT
groundwork); will-navigate pinned to origin, window.open denied
with external links to the system browser; macOS Edit-menu roles;
- quit path SIGTERM→3s→SIGKILL + exit/SIGHUP nets; child stdout
mirrored to userData/logs/main.log;
- dev mode via LH_DEV_URL: no spawn, points at next dev (HMR).
4. Packaging (electron-builder.yml + build/entitlements.mac.plist):
files = electron shell only; the whole .next payload ships via
extraResources from .next (filter standalone/**/*) — NOT from
.next/standalone directly: electron-builder hard-excludes a copy's
root node_modules and the default pattern is a literal match, so
node_modules/ and .next/ were silently dropped and the server died
with 'Cannot find module next'. Asar holds no native code.
Unsigned (identity: null) until a Developer ID cert exists; hardened
runtime + audio-input entitlements laid now for the STT milestone.
desktop:dev / desktop:dist scripts (electron 44, electron-builder 26).
Live smoke of the packaged .app: server up on a random loopback port,
all API routes 200, a real GLM chat turn answered end-to-end via
auth.json BYOK, chat persisted under userData/data, second instance
exited via the lock, no orphaned server child after quit.
Gates: typecheck, eslint, 500/500 tests (+2 data-dir-seam), build.
|
||
|
amosblomqvist
|
2678f04e7a |
feat: extraction coverage — message-granular ledger + token-batched runs (d132373 vision)
THE MESSAGE IS THE UNIT OF COVERAGE (vision-extraction-coverage.md): a per-chat append-only ledger (coverage.jsonl) records which messages the pipeline has extracted, by content fingerprint over exactly what the extraction view sees (one shared line builder — core/coverage.ts); the uncovered remainder is DERIVED at read time, never dual-tracked. A quiz answer merging into the asking message re-opens only that message; plan-block data-part attach never re-opens anything. - The trigger (turn-service) replaces every-3-quiz-answers with uncovered-tokens >= EXTRACTION.chunkTokens (4096) — quiz-less sessions extract too, content size is the honest signal. - runExtraction is now chunked: token-budgeted batches of the oldest uncovered consecutive messages (a tool call + result never split), each merged batch checkpointing its messages' coverage atomically — an abort or error loses at most the in-flight batch; merged batches stay covered. - Observation refs are batch-tagged (obs:<batchTag>:<sourceId>): re-runs of the same batch dedupe; a re-opened message's fresh understanding lands as a new event. Fixes the old obs:p1 position-collision. - New API: GET /api/graph/coverage (exact per-session derived coverage + running flags, most-uncovered first) and POST /api/graph/coverage/extract (manual run for one session, 409 while in flight). - Graph view chrome: the coverage button opens a panel with every session's covered/uncovered tokens and a per-session Extract button — the manual answer to aborted runs, errors, and dormant tails. Renders even when the graph is empty: an empty graph with uncovered sessions is exactly when it matters. Polls only while open. - Notes imports keep the whole-file checkpoint path (coverage undefined). - Tests: pure-core coverage suite (11), chunked-pipeline suite (3: multi-batch + no-op, abort-keeps-earlier-batches + remainder re-run, re-open alone + data-part invisibility), panel render suite (5); pendingQuizAnswers and its tests deleted as dead code. 486/486. Live-smoked on real GLM: manual extract of "teach me modal realism" ran one batch, wrote 4 coverage entries, panel flipped to "1.3k covered · complete" with the button disabled, graph refetched (nodes 11), usage persisted with real stats. |
||
|
amosblomqvist
|
31978f3fc3 |
fix: background-delivery ground rules — the dock is the source of truth, delivered content shows the moment it persists
The owner's three-part test of the context-injection flow surfaced
timing/visibility bugs, fixed as standing ground rules for ALL background
processes (owner rules, 2026-09-15):
1. THE JOB STARTS AT SEND. triggerGraphContext moves from onFinish to
right after the write-ahead persist in handleChatRequest: its only
input is the first typed message (the topic), just made durable — so
embedding + cosine scan run CONCURRENT with the prober's turn and
the dock blob appears the moment the session starts, not after turn
one finishes. Extraction stays at onFinish (it needs answers
accumulated DURING the turn).
2. DELIVERED CONTENT SHOWS MID-TURN. Steers inserted server-side
(claimed at turn start, persisted write-ahead) used to wait for the
next turn's completion to appear — the full record sync mid-stream
is forbidden (it breaks the SDK's in-place continuation merge), so
adoption was deferred to idle. New append-only seam: core
mergeAdopted() inserts the record's new message ids in canonical
order (in-flight streaming tail kept last) and never rewrites a
local message; any other shape bails to the full-sync-at-idle net.
Wired at three points, no new standing polls:
- turn-start window: three bounded record fetches (400ms/1s/2s)
after a send catch the claimed steer while the turn streams;
self-terminating, survives the submitted->streaming flip.
- jobs-poll settle: mid-turn settles adopt append-only NOW, the
owed full sync still runs at idle.
- wake-adoption poll: each tick adopts persisted wake content, so
steers show at the wake turn's START, not its finish.
The dock stays honest with zero gap: jobs hold "running" while
queued behind a pending question ("delivering context"), then flip
done at claim/delivery — visible from start to delivery, and the
turn-start nudge (useActivity().refresh) clears it within ~150ms.
3. THE MARKER IS CLICKABLE (observability). The knowledge-context
SystemLine becomes GraphContextMarker: quiet line, popover with the
delivered propositions IN DELIVERY ORDER (most similar first — the
order the budget filled), each with its folded understanding level;
Esc / outside pointerdown close. The part data was already the audit
trail — now it's readable at the seam where it landed.
Tests: mergeAdopted unit pins (6), marker render pins (3) — 468/468,
tsc, eslint, build all green; live-smoked end-to-end: blob at send,
marker at t+500ms mid-turn, popover order + levels, instant dock clear.
|
||
|
amosblomqvist
|
1d45d77303 |
feat: graph context injection — the session's prior knowledge, steered in at session start (vision-context-injection.md)
One shot per chat, fired at the first turn's onFinish: the session's topic (its only typed message) is embedded as an EmbeddingGemma retrieval query; the semantically closest graph propositions are selected under the session's token budget (pure budget, no similarity threshold — the decided doctrine) and delivered as a marker-only steer through the wake pipeline, queuing behind pending questions like every sub-agent result. - contracts: ChatSettings.graphContextTokens (default 1024, max 4096, step 128, 0 = off); GraphContextData + graphContextText; the graph-context JobKind + buildGraphContextSteer (id steer_ctx_<chatId>) - core/graph-context.ts: brute-force cosine top-K, greedy chars/4 fill under budget, most-recent UnderstandingLevel fold, count cap 64, parse-don't-cast at the marker seam - server: embedQuery (role-asymmetric query format, same warm singleton — vector-space parity by construction); the runner (startGraphContext- Injection one-shot gates; runGraphContextJob validates ledger parity); wake branches (chatIdOf/jobIdOf, finalizeJob done, recoverStaleJobs re-deliver or contextRespawn); abortJob kills context jobs SILENTLY (nothing was promised in the lesson); turn-service trigger seam - adapter: data-graph-context flattens to the canonical block (malformed parts have no model face) - UI: the budget slider (home dock + /chat/new draft, the session-data strip); the quiet SystemLine "~537 · 10 propositions"; dock kind "Knowledge context"; model-picker PUT now spreads settings (a model pick can no longer clear the sibling override) Live-smoked end to end on the real embedder + real GLM: seeded ledger (10 propositions), session on "modal logic: possible worlds and the necessity box" — job in the dock, queued behind the prober's question, delivered on the answer, SystemLine in the transcript, wake turn reads the block and continues the probe. Gates: tsc, lint, 459/459, build. |
||
|
amosblomqvist
|
f5edc7f0a5 |
Semantic topic clusters: local embeddings replace Louvain
Louvain only ever found the reasoning chains (the graph is disconnected by nature — every connected component became its own cluster) and edge-orphans could never join anything. Clusters are now RELATED TOPICS: propositions grouped by meaning. - Embedder (src/server/embeddings.ts): in-process transformers.js v4 running EmbeddingGemma-300M q8 (MTEB 68.4, ~300MB one-time download), MRL-truncated to 512d, "title: none | text:" prefix on 100% of inputs, L2-normalized so cosine = dot. Model/dim mismatch voids the ledger (full re-embed). Injected at the composition root — the agent layer never imports the server layer (architecture law). - Ledger (data/graph/embeddings.json): model + dim + vectors + the persisted cluster partition. Reclustering runs at the END of every embedding batch (post-merge hook, both front doors) — never per read (the agglomerative pass is ~1.5s at 338 nodes). - Clustering (src/core/semantic-clusters.ts): hand-rolled average-linkage agglomerative over cosine (npm HDBSCAN is an abandoned alpha). Stragglers stay singletons by default; attachStragglers is the all-in toggle. Threshold 0.60 swept on the real graph: 26 topics, 74/90 edge-orphans land in real topics, 10 stragglers. - Consumers (snapshot, settle, namer, replay) read the persisted partition — one cluster concept everywhere; core-survival label matching unchanged (20/26 topics kept labels). - Louvain retired: detectCommunities + graphology* deps deleted (graphology-louvain had leaked into the client bundle since the clusters wave); scripts/bench-graph.ts (dead FA2 bench) removed. - scripts/embed-graph.ts: backfill/cold-start (338 nodes in 150s including the one-time model download). |
||
|
amosblomqvist
|
975a78a304 |
Cluster gravity: per-cluster centroid pull force in the engine
The 'virtual node' realized as a centroid force: two allocation-free O(n) passes per tick (membership resolved once at sim creation; Float64Array sums). Measured indistinguishable from force-off at 338 nodes (271ms vs 272ms per 400-tick settle). - createGraphSimulation: SimOptions.clusterPull + linkStrength + velocityDecay (physics controls plumbing); degree-scaled link stiffness so linkStrength=1 is exactly d3's default - settleGraphLayout split into layout-settle.ts (it computes the Louvain partition for the pull); layout.ts stays browser-barrel-pure - barrel no longer re-exports clusters.ts: graphology + louvain are fully out of the client bundle (verified against built chunks) - replay passes the per-batch partition; CLUSTER_PULL_STRENGTH=0.15 tuned by sweep: cluster spread 63->36 world units, centroid separation and total span unchanged (no blob merge, no collapse) |
||
|
amosblomqvist
|
05712037b2 |
Live physics drag + splat cluster regions
The renderer grows a live d3 sim (createGraphSimulation, shared with the server settle): pointerdown on a node pins fx/fy and pointer capture (preventDefault kills the compat mousedown, so d3-zoom never pans); the first CLICK_SLOP-exceeding move reheats to alphaTarget 0.3; release un-pins and the sim anneals (release-and-relax) — alpha < 0.005 settles it, a 250ms debounced onCommit PUTs the rest state. Verified live: 123fps during a 320ms drag (regions rebuilding every 3rd frame), 85-tick settle, then zero idle frames. A slop-free press stays a click (select, no reheat, no commit). PUT /api/graph/positions sanitizes (finite coords, known ids) and if the graph grew mid-drag runs ONE incremental settle with the client's arrangement as anchors (50-id PUT test: anchored drift 0.15). Cluster regions swap convex hulls for a low-res world-space SPLAT bitmap: per-member radial-gradient discs (alpha 0.2) + intra-cluster edges as round-cap strokes (Minkowski dilation — concave shapes finally render), upscaled with bilinear smoothing (the upscale IS the blur, no ctx.filter: Safari). One tiny blit per frame replaces 32 clusters x 4 offset Path2D polygons re-rasterized at DPR 2 (the old GPU coil-whine path); during the live sim the bitmap rebuilds every 3rd frame. cluster-geometry.ts retires convexHull/offsetPolygon/traceRoundedPath (tests follow; clusterHue stays). Region labels stay (centroid + clusterHue, inverse-gated on zoom). |
||
|
amosblomqvist
|
ecadededd9 |
World-space rendering: geometry cached once, zoom pays only rasterization
Fixes the zoom frame-rate collapse and rebuilds draw() around a single camera transform, per the user's Obsidian-style sizing proposal. Bug fix first: the eased zoom leaked a queued rAF callback per frame -- advanceZoom applies each step through d3-zoom, whose zoom event called markDirty() from inside the tick loop, scheduling an extra draw while the loop scheduled its own. Draws multiplied per displayed frame; sustained wheel ran at ~20fps against a 120Hz display. markDirty now skips scheduling while inTick (verified: 62 frames/500ms during a 28x zoom burst, matching pan). World-space model: every draw call now runs in graph coordinates under one setTransform(k). Sizes are defined as screen-px-at-fit and converted once per geometry build (/ kFit), so nodes, node labels, arrowheads and the hover halo scale with the graph as you zoom (0.25x painted area at half zoom, exactly k^2) while edge widths and hull borders stay viewer-relative hairlines (/ k). Cluster hull geometry (convex hull + three offset fills + solid boundary -- dashes dropped for now) is built as cached Path2Ds per data/visibility/settings change; camera moves do zero geometry math, making zoom per-frame cost identical to pan. Label wrapping runs once per build, not per frame. Region labels stay viewer-relative (18px at any zoom) as UI chrome. Picking radius scales with the world-sized discs. Glide feel: ZOOM_EASE_MS 120 -> 45, convergence 0.1% -> 0.2%; a notch now settles in ~260ms instead of ~750ms. Resize still paints synchronously. Also fixes barrel-law lint violations from the settings wave (import from @/src/core and @/src/server, not deep module paths). |
||
|
amosblomqvist
|
543cf33789 |
Notes front door: format-agnostic import into the graph pipeline
The same extractor→auditor→merger pipeline now runs over raw personal
notes in arbitrary formats (vision §1, DECIDED 2026-09-13):
- NotesSource records (data/graph/sources.json, deterministic id from
path hash → idempotent re-imports) play the chatId role in mappings,
checkpoints, and observation events; usage lands on the source record
(appendSourceUsage, idempotent per startedAt, kept across upserts).
- mergeFindings is chat-store-free: callers pass MergeSource
{messages, foldedAt} — sessions pass the chat record, notes pass
messages:null (no quiz-event rebuild; a deterministic rebuild needs
structured quiz tool parts).
- Second extractor prompt (NOTES_EXTRACTOR_INSTRUCTIONS) for general-
purpose extraction; understanding judges the AUTHOR's grasp. Auditor
wording input-agnostic. extract/audit share one generic runSubmitAgent.
- graphSnapshot.fileSources (srcId→filePath); notes sources render as
static path entries in the detail card (FileTextIcon, never a link).
- One shared pipeline instance (exported from chat-handler): same
single-flight registry, live status chips (label = file path), usage
accounting, and true abort for both front doors.
- Transport: POST /api/graph/import (multipart, 3-wide worker pool) +
Import notes… button in the graph view; scripts/import-notes.ts for
batch runs (concurrency 4).
- real-data-extraction/ gitignored (raw import sources never committed).
355/355 tests; typecheck, lint, build green.
|
||
|
amosblomqvist
|
de0c286311 |
feat: model + thinking control — the stats bar's picker, on pi-ai's catalog
The model-control wave (user-decided design, 2026-09-12): ONE controller
for the conversation agents (prober/teacher are functionally one agent);
background sub-agents stay config-pinned. pi-ai owns the registry — the
shipped 366-model OpenRouter catalog (cost, context, modality,
thinkingLevelMap), getSupportedThinkingLevels/clampThinkingLevel for the
per-model thinking truth, and streamSimple's automatic clamp. No second
registry to drift out of sync.
Seams:
- config.ts: ReasoningEffort retired for pi's ThinkingLevel
(off|minimal|low|medium|high|xhigh|max). GLM roles go honest at "low" —
pi's catalog says GLM-5.3-flash accepts exactly low|high|max, and the old
"minimal" was silently clamped to low by streamSimple since the swap;
nothing changes behaviorally. visualMaker → qwen/qwen3.8-flash (fixes
the dead qwen/qwen3.8-max slug — the catalog only ships -0902; also the
user's cost pick, 0.15/0.47 per-1M).
- src/agent/models.ts: piEngineFor(role, settings?) — provider+id resolved
from the catalog, thinking level from settings ?? config, CLAMPED to the
model's supported list (a settings change between models can never send
an invalid level). PiEngine gains `level` (observability/tests).
- ChatRecord.settings {model?: {provider, id}, thinkingLevel?} — written
by PUT /api/chats/[id]/settings (replace-whole, no updatedAt churn),
read by chat-handler at turn composition: agents are stateless per turn,
so the change is next-turn-effective by construction. The turns log
still records the ACTUAL billed model (the honest witness).
- GET /api/models: the configured-provider catalog as client-safe DTOs
(core/contracts/models.ts) + config defaults; a second provider joins
the picker by registering it in piModels() — nothing else changes.
- PUT settings validation gate (src/server/model-catalog.ts): unknown
provider/model or unsupported level → 400 carrying the supported list,
never a silent clamp.
- scripts/sync-pricing.ts: regenerated PRICES/CONTEXT_LIMITS from pi-ai's
shipped openrouter.json (offline; same source as the engine + picker).
All 366 models priced; glm's stale 0.15/0.5 → pi's 0.075/0.25, limit
1310720 → 1048576 — cost is inferred at read time, so history re-prices.
- UI (components/model-picker.tsx): stats-bar right side becomes model +
thinking buttons; menus open ABOVE; fuzzy search over "provider id
name" (core/fuzzy.ts — pi's /model pattern); model rows show name/id/
price/context; the thinking menu lists exactly the chosen model's
supported levels (GLM: low|high|max — no dead options, no "off").
Model switch drops an incompatible pinned level (falls back to config,
which clamps honestly) rather than carrying a 400.
Tests (350/350, +15): catalog projection + defaults, the settings gate,
pi thinking truth pinned against upgrades (glm low|high|max; minimal→low;
off→low — GLM cannot disable thinking), the engine factory's
override/fallback/clamp/throw matrix, fuzzy ranking, store round-trip,
and the re-priced folds (usage, stats bar, activity dock).
Live smoke (disposable driver, deleted): catalog route 366 + defaults;
glm+minimal → 400 + supported; turn 1 billed qwen/qwen3.8-flash under
the override; mid-session switch flipped turn 2 to glm; clear-overrides
restored the config pin. ALL CHECKS PASS on real OpenRouter.
|
||
|
amosblomqvist
|
740293c361 |
feat: custom message types for system turns — wake and verdict messages
The custom-message scheme (user-decided 2026-09-12): system-authored
user-role messages are NOT human messages — they carry marker data parts
only, the adapter owns their model-facing wording, and the UI renders
them under control. Nothing fake is ever persisted as human speech.
The inventory (all migrated):
- probe→teacher handoff wake: was a text-bearing user message rendering
as a fake user bubble; now data-probe-handoff marker-only — the
adapter flattens it to HANDOFF_TEXT (core/contracts/chat.ts), the UI
renders a quiet 'Preparing your plan…' system line.
- plan accept/decline: was boilerplate text ('The plan is accepted…',
'I'd like the plan revised…') masquerading as learner speech — the
decline wasn't even marked; now data-verdict {accepted, at} on its own
message, adapter flattens to verdictText(), UI renders the quiet
verdict marker ('✓ Plan accepted' / '↻ Plan revision requested').
sendVerdict sends TWO REAL MESSAGES when the learner typed a note
(the note as a genuine user message, then the verdict) in ONE turn
via setMessages + sendMessage() — no 409 dance, no double teacher turn.
- sub-agent steers: the abort steer gains its data-subagent-aborted
marker part; the UI keys steer rendering off PARTS (artifact/aborted),
not metadata — metadata stays as pump mechanics only.
- record-adapter: systemTextOf() flattens marker-only messages; steers
keep their text (payload — briefs are content, not boilerplate).
phases.ts: plan→teach advances on data-verdict{accepted:true} only;
learner text alone can never sanction teaching (pinned). data-plan-
accepted retired, no back-compat (user-decided: wiped-data doctrine).
Pinned: record-adapter.test.ts (verdict/handoff flattening, note+verdict
order, steer payload kept, UI-only parts skipped), phases.test.ts
(verdict marker semantics, out-of-order, text-never-fakes), jobs.test.ts
(abort marker part), demos.test.tsx (badge renders off the part, not
metadata).
Live GLM smoke (disposable driver, deleted): probe→conclude→wake→plan→
decline-with-note (revised, still plan)→accept-with-note (teach) — ALL
CHECKS PASS, including marker-only handoff/verdict on the wire and zero
fake human speech in the transcript.
Gates: tsc, lint, build, 335/335.
|
||
|
amosblomqvist
|
6af7a38684 |
fix(bridge): gate tool-input-available on clean execution — invalid tool inputs never reach the dock
Live crash (modal-realism session): GLM double-encoded the quiz options array as a JSON STRING inside the tool-call arguments. The bridge emitted tool-input-available at toolcall_end — BEFORE validation runs — so the client briefly saw the malformed quiz as a pending question and the dock crashed in normalizeOptions ((string).map). The later tool-input-error chunk could not revive the tree. The old SDK gated this with InvalidToolInputError: validation ran before the input-available chunk. The bridge now holds the part in input-streaming until tool_execution_end and only then emits tool-input-available (clean) or tool-input-error (invalid, structural JSON Schema failure or the zod re-run inside execute). Server-side behavior was already correct (the record shows the error part + a successful model retry) — this is purely the wire-ordering hole. Pinned in chat-flow.test.ts: the refinement-layer retry (correctAnswer mismatch) and the exact live structural shape (string options) both assert the invalid call never becomes input-available on the wire. Gates: tsc, lint, build, 306/306. |
||
|
amosblomqvist
|
2d717fd932 |
Swap the server engine to pi-ai/pi-agent-core (Band B+C of docs/pi-engine-swap.md)
The Vercel AI SDK survives only as Band D (useChat client + UIMessage
record types); @openrouter/ai-sdk-provider is uninstalled.
- src/agent/models.ts: PiEngine {model, streamFn} via createModels +
openrouterProvider subpath; thinkingLevelFor maps config effort none->off
- src/agent/lh-tool.ts: neutral LhTool (zod authoring, optional execute,
clientQuestion flag); toPiTool converts via z.toJSONSchema and re-runs
zod inside execute so refinements ride the error-result retry path
- src/server/pi/record-adapter.ts: UIMessage records -> pi Messages
(dangling tool calls get synthesized interrupted results)
- src/server/pi/ui-stream.ts: SSE bridge emitting AI-SDK-shaped chunks so
the useChat client parses unchanged (continuation id rule, question
tools never emit output, tool-input-error for bad authoring)
- teacher-agent.ts: stateless per-turn pi Agents; shouldStopAfterTurn
stops only when every question-tool call resolved cleanly (errors retry)
- chat-handler.ts: buildAgent(outgoing) factory seam (stateless doctrine);
assertValidUiMessages owns the retired SDK validation gate
- sub-agents via shared runSubagent (pi-run.ts); extraction stages stop
after a clean submit; visual-maker keeps its hand-rolled loop over
direct streamSimple calls; aborted model calls bill no usage
- tests: all model seams ported to pi-ai faux provider (scripted
FauxResponseSteps, agent factories); handleChatRequest second param
is now an agent factory
Gates: tsc, lint, build, 305/305. Live GLM smoke (disposable driver,
deleted): prober streaming + persistence, question seam, turn-2
continuation same message id, per-turn real token usage - ALL PASS.
Data wiped at the pivot (chats/graph/jobs, .gitkeeps kept).
|
||
|
amosblomqvist
|
5653c77a29 |
feat: probe/teach agent split — explicit phase markers, position-only plan map
Diagnosis (2026-09-11 user testing, minimal-effort GLM): sessions stuck on 'plan' forever. Two causes: (1) the three-phase view read the stored tree's currentNodeId, which nothing ever writes post author-once (2026-09-03) — a structural dead seam; (2) the teacher stochastically left quiz.nodeId untagged, and done-by-quiz-result semantics would have falsely completed nodes anyway. The redesign (docs/probe-teach-split.md) abolishes derived occupancy in favor of explicit events: - Two agents over one transcript: createProberAgent (quiz without a nodeId field, ask, probe_done) and createTeacherAgent (full toolset, plan_init onward). The chat-handler construction branch keys on a probe_done part in the transcript. MODELS.prober added. - probe_done: server-side seam tool — carries the learner assessment (strands + free-form edge read), refuses while a question is open or the probe is already concluded, and queues a handoff steer through the generalized wake pipeline (steers now dedupe by id, not jobId; job finalization only for job steers). The teacher auto-wakes and authors the plan; the learner sees the assessment via the probe-done card. - phases: probe ends at a concluded probe_done, plan ends at a successful plan_init — strictly sequential, no skipping, no legacy compatibility (data wiped at the pivot per doctrine). - Plan map = position, not completion: derivePlanStatuses reads only tagged quiz CALLS (state/outcome feed nothing); current = latest tagged node with consecutive duplicates collapsed, visited = earlier distinct stays (a revisit stays in both — current wins at render). PlanTree is structure-only; currentNodeId/doneNodeIds and the dead mutation helpers are gone. No fallback current. - Prompts split per agent (prompts/prober/, prompts/teacher/ over the trimmed shared base); graph-context injection removed entirely (shelved, docs/probe-teach-split.md §5); knowledge-graph context no longer reaches either agent. - Plan blocks are full-only snapshots at plan_init (neighborhood machinery retired; the persistent DAG window is the future UI wave); usage capture unchanged. Gates: tsc clean, 294/294 tests (phases/plan-tree/plan-blocks/question- tools rewritten to the new semantics; probe-done + handoff-wake tests added), lint, build. Live GLM smoke PASS: prober concluded with an honest assessment, teacher auto-woke, authored a 10-node plan with the block attached, then tagged checks on slope-instinct → secant derived current/ visited exactly as designed; grading honest both ways (driver off-by-one proved a wrong answer grades false). |
||
|
amosblomqvist
|
741ff416d6 |
Fix turn-usage keying: idempotent per request turn id, not messageId
Live smoke exposed two bugs in the turn usage log: - onFinish messages carry no ids (withMessageIds fills copies inside persistTurn), so fresh turns keyed messageId "" — provenance lost. - The SDK coalesces continuation turns into ONE response message (a whole probe conversation lands in a single assistant message), so per-messageId idempotency swallowed every continuation turn's usage. TurnUsage gains id: string (unique per request, the append idempotency key); messageId stays as transcript provenance, taken from the id-filled list. appendTurnUsage dedupes by id; continuation turns with the same messageId each keep their usage. |
||
|
amosblomqvist
|
f4abad3f15 |
Master seam: plan-block emission + turn usage capture
- src/core/plan-blocks.ts: pure emission builder — full block on successful plan_init, neighborhood block when the derived plan state moved since the newest existing block; the as-of state reads from the block's frozen snapshot (transcript quiz parts mutate in place on answer, so re-derivation is post-mutation blind). Idempotent by construction. - turn-service persistTurn enriches with withPlanBlocks at the canonical write; chat.tsx adopts the record once the server agrees the turn is done (endTurn flips only after persist). - chat-handler: per-request usage collector via agent-level onStepFinish (StepResult.usage), appended at onFinish to ChatRecord.turns via appendTurnUsage, idempotent per response message id. |
||
|
amosblomqvist
|
37c4638feb |
Merge feat/graph-rendering: wave 4 — extraction usage capture (live + persisted), true abort
# Conflicts: # src/server/observability.ts |
||
|
amosblomqvist
|
6bf44b821c | Merge feat/sub-agents: wave 4 — job abort (real kill + abort steer), job usage capture, JobUsage.lastStep | ||
|
amosblomqvist
|
9174cde55e |
Extraction observability: usage capture, live status payload, true abort (Wave 4)
Extraction is real spend and background work; the dock needs to see and stop it. User decisions: persist all cost data (chat record), per-stage usage granularity, and TRUE cancellation for abort. USAGE CAPTURE (live + persisted): - extraction.ts: per-call onStepFinish captures each billed step's usage (SDK LanguageModelUsage -> CallUsage: input/output/cache read+write/reasoning) into the registry's live per-stage usage (extractor/auditor; the merger is deterministic — no model call). - /api/graph/status runs now carry usage (model + steps per stage); cost/context folds are the consumer's (core/prices.ts — nothing stores dollars). - ChatRecord.extractions: append-only per-run usage log (mirrors the turns log): startedAt key, at, durationMs, merge stats when the merger ran, stages. Persisted on success AND failure — a failed run still spent money. Idempotent per startedAt; the list payload (ChatSummary) never leaks it. - EmissionStats moved to contracts/graph.ts (its contract home); core/graph.ts re-exports for existing importers. - Verified end-to-end against the real pipeline (backfill one chat): per-step token counts, reasoning tokens, and the auditor's growing prompt-cache reads captured and persisted. TRUE ABORT (the dock's abort button, user decision): - RunningExtraction owns one AbortController per run, shared across phases; abortExtraction(chatId) fires it and returns whether a run was aborted (idempotent no-op otherwise). The pending model call receives the signal — the provider cancels the request, spend stops. throwIfAborted() at phase boundaries ends runs aborted in sync gaps. - Abort is a deliberate stop, not a failure: same ledger discipline (nothing merged, nothing checkpointed, partial usage persisted) but the run reports zero emissions instead of throwing. Exported via src/server/observability.ts for master's /api/activity/abort route. - Tests: mid-run abort (hanging mock that rejects on the signal), live-accumulation + persisted-record assertions, store-seam idempotency, and the summary-payload leak guard. |
||
|
amosblomqvist
|
17ed37a471 |
feat: abort sub-agent jobs (real kill) + runner usage capture into JobRecord.usage
Wave 4 (agent-side observability + control), reconciled with the dock contract (master relay 2026-09): - abortJob in src/server/jobs.ts, exported through the server barrel for POST /api/activity/abort (master's thin route): fires the job's AbortController (real kill — the in-flight model call aborts, not just the next loop check), drops any queued result steer, flips the record to error "Aborted by learner" + activity "aborted" synchronously, and queues a one-line abort steer (buildAbortSteer) through the same wake pipeline so the teacher's promise doesn't hang in the lesson. Shares the result steer's deterministic id space, so abort + racing result can never both deliver (steerDelivered dedupe). - Abort registry (src/agent/subagents/aborts.ts) on globalThis, mirroring the wake pipeline's module-state pattern; runner registers/unregisters per run, so the route's module instance sees the loop's controller. - Runner usage capture: LoopContext (abortSignal + onUsage) threads into all three kind loops; per-step usage folds via accumulateJobUsage into JobRecord.usage (model from config MODELS, absent-never-zero), flushed in runJob's finally so an aborted job's money record survives the kill. - Contracts: JobUsage token fields optional + lastStep (most recent call's accounting, for the dock's per-blob context line vs CONTEXT_LIMITS); SubagentResultMeta.aborted; AbortSteerMessage; buildAbortSteer; SteerMessage union generalized. - demo-builder/visual-maker accept abortSignal + onUsage (visual folds generateText usage per iteration). - stillRunning/finishJob/jobContext guards: an aborted job is never overwritten by the runner (terminal no matter what the wake does); failJob and the pump's finalizeJob keep the flip. - UI: abort steer renders a quiet aborted chip (job-aborted-badge). - Prompt 20-jobs.md: one line about aborted jobs. Tests: accumulateJobUsage folds (undefined-preservation, lastStep), runner usage capture, real-kill abort (sync flip, silent unwind, abort steer wakes), usage-so-far survives the kill, aborting a delivering job drops the queued result steer, outcome reasons, abort-chip render. 330/330, typecheck/lint/build green. |
||
|
amosblomqvist
|
a023231edd | Activity feed: drop busyChats — dock is background processes only (user decision 2026-09-11) | ||
|
amosblomqvist
|
f8ba5b1628 |
Activity feed: one dock-shaped GET /api/activity (cross-chat processes)
The dock is visible in every view, so it needs every running process — sub-agent jobs across ALL chats (not just the open one) plus the global extraction runs — in one poll. /api/jobs stays chat-scoped and untouched: it is the transcript machinery's poll and carries turnInFlight for the re-attach seam (chat-transport), the seam we spent three live rounds stabilizing; making it polymorphic would fork its response shape by mode and put regression risk on exactly the wrong path. The dock is a different consumer with a different shape (cross-chat, running-only, cost-bearing), served by a thin route over server/observability.ts: - activityFeed(): running jobs from every chat (title-joined, oldest-first on last activity) + extraction runs + busyChats (chats with a teacher turn in flight — the biggest spender; data made available, rendering is a dock decision). - GET /api/activity — the dock's single poll everywhere. - Cost math stays client-side (core/prices.ts folds); raw usage rides the records and the transport stays thin. Decided with the user over the polymorphic-/api/jobs and separate-all-jobs-endpoint alternatives. |
||
|
amosblomqvist
|
83a00800e9 |
Observability contracts: usage accounting, plan blocks, price table (Phase 0)
The 2026-09 observability wave's type-level foundation, landed before the streams so nothing renders against guessed shapes: - contracts/chat.ts: CallUsage + TurnUsage (per-model-call token accounting, one entry per teacher/wake turn keyed by messageId) and ChatRecord.turns — a DELIBERATELY parallel append-only usage log, not per-message metadata: the client re-POSTs its message list every turn and its copies never carry server-stamped fields, so anything inside messages gets clobbered by the next persist (the wave-3 merge-rule lesson). - contracts/chat.ts: PlanBlockData — the persisted data-plan-block part (full DAG on plan_init, one-level neighborhood on plan updates, snapshot semantics so history renders what was true then). - contracts/jobs.ts: JobUsage + JobRecord.usage — a sub-agent run's accumulated accounting, written by the runner. - core/ports.ts + store/chats.ts: ChatStore.appendTurnUsage — idempotent per messageId (abort replays / wake retries append once); listChats no longer leaks turns into the list payload. - core/prices.ts: PRICES + CONTEXT_LIMITS (generated from OpenRouter's published pricing via scripts/sync-pricing.ts) + pure folds: callCost, turnCost, sessionCost, contextWindow, jobCost. Unknown model or absent fields render as absent, never zero. - server/observability.ts: barrel-approved surface for the transport layer (fixes the import-law violation in the graph status route that slipped through the wave-3 merge verification). Capture wiring (agent onStepFinish -> persist seam, plan-block emission) lands next as master seam work; the streams start against these shapes. |
||
|
amosblomqvist
|
91ae2e137d |
fix: dedupe same-id continuation copies in persistTurn (coalescing follow-up)
With a steer coalesced into a quiz-answer continuation, the steer sits AFTER the assistant message being continued, so the SDK's onFinish list carries the input copy AND the extended response copy as two entries with the SAME id — persistTurn saved both, the jobs poller's syncFromServer imported the dup, and React keyed a single message twice (the user's live test hit it). withMessageIds now dedupes by id keeping the LAST occurrence — the SDK's final merge state (a superset of the earlier copy) — which also yields the right semantic shape: [user, steer, assistant(extended)]. Out-of-lane surgical touch (flagged for master review): turn-service.ts. Squashed the live session's already-corrupted record by hand (same dedupe). |
||
|
amosblomqvist
|
e33b83ca2d |
fix: steer-delivery starvation — continuation coalescing (claimSteers)
The bug (seen live): a teacher that ends every turn with a new pending quiz keeps the wake pipeline's idle gates permanently closed — the steer sits in 'delivering result' forever while the learner answers check after check. - claimSteers (server/jobs.ts): the chat-handler claims every queued, undelivered steer the instant a client turn wins the turn registry, and appends it after the learner's answer — conversion-safe, the answered quiz carries its output. Every answer flushes pending steers immediately. - The pump keeps its post-idle duplicate-guard (re-check steerDelivered before firing), so a claim racing the pump converges to exactly one delivery either way. - The stale-job sweep now skips chats with a turn in flight (a healthy but slow wake no longer gets falsely re-queued as 'recovering'). - Out-of-lane surgical touch (flagged for master review): the 3-line handleChatRequest seam (claim after beginTurn, persist+stream 'outgoing'). - Verified live end-to-end on the stuck session: answering the pending quiz fired the steer + wake turn within a second; job flipped done/delivered. |
||
|
amosblomqvist
|
2f8cf71f52 |
Wave 2: 409 merge-retry transport, dead pendingJob code, evidence-based timing, prompt-size guard
- components/chat-transport.ts (NEW): createChatTransport(chatId) — the chat.tsx seam is now a one-line swap. Its fetch turns a 409 on POST /api/chat (a learner send racing the wake pipeline's ~1s adoption window) into: wait out the in-flight turn (poll turnInFlight, 30s cap), refetch the record, MERGE the outgoing messages onto the fresh server list, re-POST. A bare retry would clobber the steer + wake messages via the write-ahead persist (steerDelivered makes that loss permanent) — merging is the whole point. Unit-tested: pass-through, wait+merge+reorder, dedupe by id, repeated-409 loop, bounded give-up. - src/core/pending.ts: findPendingJobCall deleted (dead since wave 1 — job tools resolve server-side with an ack, nothing pends client-side). JOB_TOOLS kept (runner.ts still uses it). Header doc rewritten. - src/server/jobs.ts: GRACE_MS 1500 → 750, with the evidence in the comment: the race window is only the learner's answer→POST flight (sendAutomaticallyWhen dispatches synchronously; ~10ms localhost, <300ms realistic RTT — 750 is 2.5× worst case) and misses self-heal via the pump's gate recheck, the 409 backoff ramp, and the new transport merge-retry. POLL_MS=400 kept (sampling granularity only). - tests/jobs.test.ts: prompt-size guard — every fragment ≤8k chars, composed prompt ≤60k — the README-leak class (815 bytes ≈ 200 tokens per teacher turn before wave 1's fix) can't silently recur. Gates: typecheck, 279/279 tests, lint, build — all green. |
||
|
amosblomqvist
|
2d7b004fda |
fix: live sub-agent observability — dispatch cards, steer artifacts, sidebar panel
The pi-style ack protocol (previous commit) left the client describing the
old world: artifact cards read the ack string as a result ("could not be
displayed"), the only job poller rendered on input-available parts that no
longer exist (so wake turns landed invisibly until re-navigation), steers
rendered as learner bubbles with no SVG anywhere, and server-persisted
assistant messages carried empty SDK ids (duplicate React keys).
- persistTurn (turn-service): fill empty message ids at the canonical write
— server-only messages (last turn's, wake turns') now key uniquely;
client-authored ids untouched.
- chat.tsx: ONE central /api/jobs poll (fast while jobs run, slow discovery
cadence, off for job-less sessions) drives every job surface, re-syncs
once when a job settles (the record already holds the steer + wake
response at flip time), and adopts server-side wake turns via the same
turnInFlight machinery as returning mid-turn. Steers branch to the
artifact card — never a bubble. Composer no longer gates on jobs.
- /api/jobs now reports turnInFlight alongside the job list.
- dispatch-card: the tool-call box — live activity by toolCallId while
running, quiet "delivered"/"failed" one-liner after; the ack is visible
observability, the artifact lives in the steer.
- steer-card: renders the delivered artifact by kind (research brief /
sanitized SVG / sandboxed demo iframe) from the data-subagent-artifact
part; result renderers extracted and exported from the artifact cards.
- sub-agents-panel: right-sidebar section above the plan tree listing
running sub-agents (kind, elapsed, live activity) — vanishes when idle.
- tests: dispatch/steer/panel render tests re-target the new protocol;
persistTurn id-hygiene suite. 229/229 green, build green.
Out-of-lane touches for master review: chat.tsx + components/** (cleared
this wave), app/api/jobs/route.ts, src/server/turn-service.ts, contracts
protocol doc §2. Live-verified on :3003 (full loop, SSR DOM checked).
|
||
|
Amos
|
99fa3f0847 |
Pi-style ack-and-steer sub-agent protocol: dispatch mid-turn, steer via server wake turn
The job tools (research/demo/visual) switch from client-side turn-breaking calls to pi's model: execute resolves IMMEDIATELY with a fire-and-forget ack, the teacher's turn continues, and the result arrives later as an injected user-role steer message that wakes the teacher in a new server-side turn. - contracts.ts: THE ASYNC PROTOCOL rewritten (ack now, steer later); steer types (SubagentSteerMessage, metadata.subagentResult + data-subagent- artifact part), pure buildSubagentSteer/steerDelivered/isSubagentFailure - tools/jobs.ts: jobTools(ctx) factory — server-side execute creates the job (deduped by toolCallId), fires spawn with the dispatch-time input (the part isn't persisted yet at execute — a transcript scan would race) - runner.ts: runJob(jobId, input?) builds the steer and hands it to the injected deliver seam; pipeline owns the done/error flip (a polling client re-syncs once, at the end); failures still steer - server/jobs.ts (new pipeline): per-chat serialized queues on globalThis, idle gates (turn liveness + pending question, 1.5s grace), 409 retry with backoff, idempotent per jobId, stale-job recovery sweep (re-deliver or re-spawn); wake turns run through the same handleChatRequest composition via setTurnRunner injection - chat-handler.ts: wires runner+deliver+tools ctx; drops spawnTurnJobs from onFinish; teacher default registry = question tools only - plan.ts: PlanToolContext split out (plan tools need no job wiring) - visual-maker: edit_svg added (pi parity — exact-match single replacement, fuzzy context on a miss); researcher: pi's kept/dropped source accounting restored - prompts 20-jobs/40-demos: async discipline rewritten (announce and keep teaching, no waiting/polling/fabricating, parallel dispatch, teach with the artifact when it arrives) - tests: jobs.test.ts rewritten (dispatch ack mid-turn, runner steers, wake pipeline: deferral/409/idempotency/failure), demos.test.tsx runner tests ported to runJob + fake deliver; 222 green FLAGGED surgical touches outside the stream's owned files, for master review at merge: src/core/contracts.ts (protocol doc + steer types), src/core/ui-message.ts (job tool + data-part types), src/server/ chat-handler.ts (wiring), src/server/turn-service.ts (spawnTurnJobs retirement), src/agent/teacher-agent.ts (default tools), src/agent/tools/ plan.ts (context type split). Verified live on PORT=3003 with real models: teacher dispatched research mid-turn, researcher ran real searches/fetches, steer injected on quiz answer, wake turn wove the brief in citing its sources, job flipped done; a client/wake race correctly 409'd. |
||
|
Amos
|
7ab444ead3 |
Fix turn-liveness registry across Next dev module instances (globalThis)
In Next dev (Turbopack), each route compiles as its own entry — a module-level Map in turn-liveness.ts was a DIFFERENT instance in the page render, GET /api/chats/[id], and POST /api/chat. The chat route held the live turn (hence the 409), but the page render read an empty registry: initiallyInFlight=false → mount continuation fired a send → 409 → the 'Something went wrong' banner on returning mid-turn. The registry now lives on globalThis (the standard Next singleton channel) — one Map per process regardless of module instances. Semantics unchanged (TTL leak guard, begin/end round trip). Browser-verified end-to-end (headless tester): answer quiz → navigate away → return mid-turn → graded card (no re-prompt), 'Thinking…' shimmer, composer gated, no error banner; teacher response auto-delivered when the background turn finished. Polling GET /api/chats/[id] observed working. |