4bea0b46dff5081844f947363b40b197f2c121a5
141
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 amosblomqvist
|
f3003d357a |
Wave D: native AgentEvent wire + client session rewrite (SDK purged)
Server: ui-stream.ts is verbatim pi AgentEvent passthrough (SSE data lines + [DONE]); only app-level events are the loop-level error and turn-settled terminal; onError carries raw provider text (owner ruling). Client: useChat + the AI-SDK transport replaced by components/session-core.ts (SessionModel: 409 identity-retry, optimistic c_ appends, absorbed() adoption, needsContinuation, shared consume for POST + reattach) and the session.ts hook; the golden-master grouping canon moved client-side into core/view.ts (projectView); scanners read the native record (phaseAt3Lh, danglingQuestionOf, graph toolParts, plan-blocks, runner, extraction). Coverage: the native message is the unit (fork 3) with the call+result inseparability bond rule; view-line-less messages (markers, error results) are invisible to coverage and never checkpointed. Deletes: chat-transport, projection, ui-message, their tests (ported to tests/session-core.test.ts), and the ai + @ai-sdk/react dependencies — the Vercel SDK is fully out of the tree. Wire pins in chat-flow rewritten onto the AgentEvent dialect (eventsOf parser). Gates: tsc clean, 994/994 tests, eslint clean, build green; live smoke on PORT=3004 (verbatim wire → native record). |
||
|
amosblomqvist
|
7ab47d4fcb |
Wave B: the native message record — input events, projection, adapter death
The chat record is now pi-native agent history (LhMessage[]), the AI SDK's
UIMessage transcript contract is gone from the server, and every client turn
arrives as ONE explicit input event.
Core (new):
- src/core/native-messages.ts — the LhMessage taxonomy (user/assistant/
toolResult + lhMarker customs: verdict, probe-handoff, subagent-steer,
subagent-abort, plan-block), constructors, and toLlmMessages — the single
model-facing seam (convertToLlm): strips ids, flattens markers to user
text, drops UI-only plan-blocks, synthesizes INTERRUPTED results for
dangling aborted calls, never throws.
- src/core/projection.ts — projectNative: the one-way native → UIMessage
projection (consecutive assistant+toolResult runs group into one
assistant view message; toolResults attach backward by toolCallId;
markers project to data parts; plan-blocks splice after the last
successful plan_init). Every existing consumer reads through it.
- src/core/input-events.ts — the ChatInput union (text/answer/verdict/
continue), parseChatInput validation, and inputAppends (answer targets
the dangling question call or 400s; verdict = optional note user
message + native verdict marker).
Server:
- chat-handler.ts — the turn root takes {chatId, input}; write-ahead
persists input appends + claimed steers before the turn; claimSteers
coalesces the whole queue with the handler's steerDelivered filter as
the dedupe seam; teacher-vs-prober branch on hasProbeDone.
- turn-service.ts — persistTurn stamps ids, drops clean question
placeholder toolResults (the record ends at the unanswered call),
emits plan-block markers idempotently (plan-blocks.ts rewrite), and
persists abort partials.
- ui-stream.ts — PiTurnSpec takes the native base; the wire dialect is
unchanged. jobs.ts steers are native markers (steer_<jobId> dedup ids);
waitForIdle reads danglingQuestionOf. teacher-agent.ts wires
convertToLlm: toLlmMessages. record-adapter.ts DELETED wholesale.
- GET /api/chats/[id] serves messages: projectNative(record.messages).
Client:
- chat-transport.ts — the 409 retry is IDENTITY (re-POST the same event
body after the turn settles — the event was never persisted
pre-acceptance; the merge logic is gone). The transport consumes the
stashed input event via prepareSendMessagesRequest and throws loudly on
a missing stash.
- chat.tsx — every send call site stashes exactly one ChatInput event
(text, answer, verdict, continue) before invoking the SDK send.
Tests: full native migration (chat-flow rewritten to input events and
native record pins; jobs/extraction/demos/graph/usage/plan-blocks/
probe-done/reattach-transport converted; record-adapter.test.ts deleted;
chat-transport tests pin the identity retry). 982/982, typecheck, lint,
build all green.
|
||
|
amosblomqvist
|
b87a6925af |
Wave A: pi-native tool layer — delete LhTool/toPiTool, TypeBox everywhere, terminate on questions
- src/agent/lh-tool.ts deleted: the zod-wrapped LhTool + toPiTool bridge (z.toJSONSchema per call, clientQuestion flag, hand-rolled result mapping) is gone. Tools are pi-agent-core native AgentTools now. - New src/agent/tool-execute.ts owns what the wrapper added: the abort guard (runWithAbortGuard: run signal + 30s per-call timeout + 250ms grace, 'Operation aborted' / timeout wording preserved), ToolInputError (the retryable '- problem at path' format), result mapping (string verbatim / object JSON / image parts, details rides the record's tool part output), and questionResult() — placeholder + terminate:true (owner Option A: belt of terminate + slim makeQuestionStopper with the clean-question rule; maxTurns deleted from the teaching turn entirely per owner). - All 11 tool files + tools.ts registries converted to TypeBox parameters (typebox@1.3.7, pinned to pi-agent-core's copy). zod refinements (quizInputProblems, askOptionProblems, live-nodeId) moved from schemas into execute — the model sees the same retryable error shape. - question-core + contracts/probe TypeBox-authored (single source shared with the UI); tool-parts parse seam runs the boundary's full chain (Convert → Check → Clean → refinement), fixing the live-smoke lockout: pi's boundary Value.Converts correctAnswer: 17 → '17' but the record keeps raw args, so a parse without Convert rejected a posed quiz (invisible pending question, turn ended). Pinned with the real smoke emission shape. - Subagents converted: demo-builder/researcher AgentTool-native, visual-maker/pdf-reader declaration-only pi-ai Tools with local execute fns, extraction submit schemas re-authored in TypeBox (owner option 2: zod defaults → Type.Optional + fill at the submit seam). - Accepted delta, flagged: TypeBox-native schemas skip pi's JSON-Schema primitive coercion fast path... (boundary still Value.Converts), and prober-quiz unknown keys ride through (Value.Clean strips at display). - Tests: 988 passing (was 967 at the recon baseline); tool-abort and read-tool rewritten against the native seams; question-core/question- tools/tool-parts/demos/web-search/graph-tools/plan-tree/probe-done/ tool-toggles converted to Value.Check/execute pins. Live smoke (PORT=3004, real model): native ls turn streams end-to-end; a real quiz pose (first attempt missing correctAnswer → tool-input-error → model retry → posed) terminates the turn cleanly and persists pending. |
||
|
amosblomqvist
|
b07cb313f0 |
delete the getApiKey hook — pi-ai resolves keys natively per provider (owner, 2026-09-26)
baseAgentOptions (teacher-agent.ts) and runSubagent (subagents/pi-run.ts) passed getApiKey: () => openRouterApiKey(). pi-agent-core resolves (config.getApiKey ? await config.getApiKey(provider) : undefined) || config.apiKey (agent-loop.js:191) — a provided hook OVERRIDES pi-ai's native per-provider env resolution, and this one ignored its provider arg: every non-OpenRouter provider was sent the OpenRouter key and 401'd (owner repro: openai and anthropic direct). Deletion restores the architecture's stated intent — the keys seam reflects file-saved keys into env and pi-ai reads them per request. openRouterApiKey() export kept (now consumer-less; visual-maker/ pdf-reader read process.env.OPENROUTER_API_KEY directly for their own OpenRouter-specific fetch paths — correct, untouched). Pinned by tests/api-key-hook.test.ts: built teacher + prober expose NO getApiKey hook. Gates: typecheck 0, 987/987, lint clean, build green. |
||
|
amosblomqvist
|
e89d4c2e23 |
serial questioning + maximal formatting (owner ruling, 2026-09-26)
The prober was planning perfect 7-question batches instead of probing. Three
surfaces now say the same thing — one question, adapt, next question:
- prober/10 rewritten to ~2.6k (from 5.6k): serial discipline leads (ask,
wait, choose the next question from the answer); 'How your questions
arrive' deleted (owner: does nothing); 'You cannot get there by thinking'
deleted (owner: didn't reduce thinking); 'never plan a batch' and the
first-message-length line dropped per owner — no overcorrection, no
strong-instruction inflation. Binary search compressed to four moves.
- tool descriptions: the async-era fossil 'Ask exactly one question per
tool call; after calling, end your turn and wait' — which permitted
multi-call batches — swapped for 'ONE question per turn: ... his answer
determines the next' in all four places (prober quiz + ask descriptions,
teacher quiz description, both zod question-field describes).
- shared/40-formatting: markdown/LaTeX coverage now explicitly names quiz
questions, options, and explanations ('in prose and in quiz questions
alike') — the prober wasn't using LaTeX in questions.
Prober composed prompt ~13.2k -> ~10.0k chars. Pins preserved verbatim
('measurement should be efficient', 'the read tool can inspect it directly').
Gates: typecheck 0, 986/986, lint clean, build green.
|
||
|
amosblomqvist
|
c25ee44bfd | formatting chunk: from restraint to maximal use — documents, not chat messages (owner ruling, 2026-09-26) | ||
|
amosblomqvist
|
18c84b8a93 |
prober rescoped to a tight five-step process; method moves out of the shared base; logistics chunks compressed
- base keeps intro + the four philosophy paragraphs + accuracy (9.0k -> 3.0k); the click/arc/hedge paragraphs and Principles i/ii move verbatim to teacher/05-method.md (teacher-only — the prober never teaches) - prober/10 rewritten as one process doc: read map -> make the goal concrete (questions, not orientation) -> reason out strands -> bracket with quiz -> probe_done; teaching-in-brief compressed into the opening, probe-done folded into step 5; prober/20 and prober/30 deleted - shared logistics chunks compressed per owner ruling 2026-09-26: question-tools 5.1k -> 2.3k (option-construction procedure kept whole), learner-files 771 -> 374, formatting 1.8k -> 549 - prober composed prompt 25.9k -> 13.2k chars (~6.5k -> ~3.3k tok, -49%); teacher 36.6k -> 32.8k (method content identical, order shifted) |
||
|
amosblomqvist
|
369fe497ae | question-tool semantics: the async-era 'end turn and wait' doctrine dies — tools are ordinary tools now (owner ruling, 2026-09-26) | ||
|
amosblomqvist
|
d1542cdcbf | quiz argument order: explanation before correctAnswer — reasoning precedes the pick (owner ruling, 2026-09-25) | ||
|
amosblomqvist
|
e4049e4bf8 |
Key-presence refusal at every gate: a pinned model is not enough
Owner-diagnosed: the turn gate checked only model coordinates, so a pinned slot whose provider key was deleted passed the gate and died mid-stream masked to 'An error occurred.' The key is now gated where the model is — by what would RUN (env-first keyValue truth, so an env-shadowed key passes). - core/settings.ts: the provider_key_missing envelope family (code, provider, slots, human line naming the provider from AUTH_PROVIDER_KEYS labels) + ProviderKeyMissingError + isProviderKeyRefusal matcher. - chat-handler turn gate: refuses 503 before beginTurn when the resolved conversation's provider key is gone (slots: [strong]). - Job runner: same check via an injected dep (agent layer cannot read the key store) — a keyless engine refuses the spawn through the existing failure-shape convention, no phantom run. - Extraction pipeline: refuseIfStagesUnready refuses BEFORE any batch (first offender: extractor before auditor); the trigger skips info- level, /api/graph/import surfaces the 503 envelope per file. - Tests: 8 new pins (turn gate both ways, runner refusal, pipeline first-offender + pass-through, envelope factory, matcher); the three turn-driving suites seed OPENROUTER_API_KEY like any real setup. - Doc fix: stale EXTRACTION.chunkTokens comment in the manual-extract route (left over from eed4b4e). Known residual (pre-existing, flagged for master): the manual extract route logs BOTH config refusals as console.error (fire-and-forget path); the coverage ledger keeps everything uncovered either way. |
||
|
amosblomqvist
|
401a21e83f | teaching methodology: theory doc + step-unit teaching loop + minimal-tree planning + click-as-collapse philosophy (owner theory, 2026-09-24) | ||
|
amosblomqvist
|
2e9fd01c0c |
Extraction-UX data layer: the chunk budget becomes a graph setting; the chat view gets a per-session coverage read
Owner-locked extraction-UX redesign (dispatch 2026-09-24), part 1 (data
layer; the UI chunk follows).
THRESHOLD -> SETTING
- config.ts's EXTRACTION.chunkTokens dies; the default moves into
FACTORY_SETTINGS.graph.extractionChunkTokens (4096, the old constant's
value). GraphSettings gains the optional field, Required at resolution
via the standard ?? FACTORY pattern — pre-redesign records resolve the
default cleanly.
- FLOOR 500 (the implementer's call, documented on the field): every run
pays the graph-carrying prompt overhead out of the write channel's
money — a smaller budget fires penny-batches that cost more per
extracted token than the content is worth. The sanitizer CLAMPS up
(bd68486 clamp-never-reject discipline: fractions floor, junk drops);
the settings PUT REJECTS below-floor with the rationale in the message.
No ceiling: a large budget just delays extraction — the user's stated
cost/thoroughness tradeoff, not a misconfiguration to police.
- The trigger (turn-service.triggerExtraction) takes the budget as a
parameter — chat-handler passes effective.graph.extractionChunkTokens,
the fresh per-turn read, same as every other gate.
- The pipeline's BATCH side threads through the same number:
ExtractionPipelineDeps gains an OPTIONAL getChunkTokens getter with the
factory default (Teacher's 24 test fixtures stay untouched; the two
call-site lines inside extraction.ts are the only cross-band edits,
flagged). chat-handler + backfill/import-notes wire the live
effectiveSettings — trigger and batches share one budget, always.
PER-SESSION COVERAGE READ
- sessionCoverage(chatId) (server/chats.ts): the chat view's ring-meter
block — totalMessages/coveredCount/uncoveredCount/coveredTokens/
uncoveredTokens (the same coverageOf derivation /api/graph/coverage
serves, scoped to one chat), running (chat-kind extraction registered
for this id), and lastFoldedAt (the chat's newest coverage row's fold
— a cheap single pass; null before any fold). GET /api/chats/[id]
carries it as coverage: {...}; unknown chat -> null (the route 404s
first).
- /api/graph/coverage (the all-sessions list) compiles untouched — the
per-session read is a scoping of it, not a supersession; the UI's
sidebar/ring chunk decides which surface it consumes.
Tests: factory/sanitizer pins (clamp-up, fraction-floor, junk-drop),
resolution pins (absent -> 4096, stored 2000 wins), settings-body pins
(valid/absent/fraction/below-floor 400/junk 400), sessionCoverage pins
(derived split, newest-fold scoping, cross-session isolation, running via
the registry, null on unknown chat). 893/893.
|
||
|
amosblomqvist
|
b9397fa783 | formatting chunk: renderer-awareness opener — full markdown set available, content determines presentation (LaTeX/mermaid verbatim) | ||
|
amosblomqvist
|
d75867246d | prompt rechunk: neutral base (philosophy + principles + accuracy), shared voice/mechanics/files/formatting fragments, phase-owned question methodology, teacher 30-60 renumbering | ||
|
amosblomqvist
|
d1b46132b9 |
Wave 7 close-out: the embedding ledger seam dies; notes birth via mapping lines
Part 1 — the ledger + remnants interlude (master-dispatched, owner ruling 2026-09-23: nothing configures the graph beyond chat models): - The embedding ledger surface is gone: readEmbeddings/writeEmbeddings ports, the EmbeddingsLedger contract, EMBEDDINGS_FILE (config, store impls, graphRevision source list), and graph-deletion's vector-pruning step. On-disk data files stay untouched (inert history). - The extraction embedder seam dies wholesale: ExtractionPipelineDeps drops the embedder dep (chat-handler + backfill/import-notes stubs go with it), the 'embedding' phase leaves ExtractionPhase, the usage capture/persist helpers die, and stages.embedding leaves ExtractionRunUsage (historical records keep their shape — reads are lenient). Extraction is two stages again, like the local-embedder era. - Consumer-less remnants: latestQuizOutcomes + QUIZ_EVENT_KINDS, clusterContentId, matchClusterLabelings + Community + ClusterLabeling, the readClusterLabels/appendClusterLabel port seams + store impls + LEDGER_FILES.clusterLabels. NamingRunUsage/readNamingRuns STAY (standing ruling — the cost pill renders historical naming spend); clusterHue + MIN_CLUSTER_SIZE stay (live). - The 'labeler' role dies: BACKGROUND_MODEL_ROLES, ROLE_SLOT, ModelRole, the catalog pin projections, and the UI roster strings. RolePins keys shrink with the union. The notes birth fallback (owner-approved 2026-09-23, surfaced by the interlude's live smoke: production notes imports create provenance-EMPTY nodes (importNotes passes messages: null — the file's anchor is the mapping line), so the provenance-only birth rule left every imported file unclustered and invisible to the prober's session list, against the locked file-birth ruling. A provenance-empty node now births under its FIRST mapping line's chatId (ledger order); only notes-born nodes ever hit the fallback (chat nodes always carry provenance; a re-encountered node's link-append re-opens the provenance rule). Deletion-safe: mapping lines of deleted chats are stripped alongside their provenance. Live-verified on port 3004: a fresh notes import clusters under its src_ id with the file path as the label, end to end with no embedder. 845/845 tests, typecheck/lint/build green. EOF ) |
||
|
amosblomqvist
|
6b13583e96 |
wave 7: the prober reads its map — graph-context injection dies whole
The owner's zero-configuration ruling (2026-09-23): nothing configures the graph beyond chat models. The prior-knowledge injection — its budget dial, its runner, its marker part, its dock/dispatch entries — is removed across all bands; the prober now reads the learner's persistent knowledge map on demand through two gated tools instead: - list_sessions: every birth session, names + kinds + counts + dates (graphSessionIndex, Platform's core seam) - list_propositions(session): what was born there + the session's own latest understanding level + that evaluation's date (graphSessionPropositions) — names/levels/dating ONLY, no statement text (owner ruling: the map's organization is the prober's business, its prose is not) Registered via createProberTools when chat-handler passes graphStore (the graphToolsEnabled gate decided server-side, per-turn fresh); the prober doctrine gains the map-review nudge at session start. Also: runner's JobKind special-casing collapses (KIND_ROLE/KIND_RUNNERS are plain records again), semantic-clusters.ts dies with its test, the embedder dep's doc tells the wave-7 stub story, and ports.ts's readClusterLabels/readNamingRuns surface stays untouched for Platform's ledger-port interlude. 848/848 tests, typecheck/lint/build green. |
||
|
amosblomqvist
|
078635e61c |
feat(extraction): kill the dead LLM cluster-namer — session names own the labels now
Wave 7 chunk A (950ddfd) replaced semantic clustering with birth-session clusters labeled by session names — the LLM namer became dead code, but extraction still called it, minting naming runs nobody reads on every merge. Real model spend wasted per extraction. Removed (agent band): - src/agent/subagents/cluster-namer.ts — the naming sub-agent, deleted whole (refreshClusterLabels, clusterNamingSettled, the naming loop). - extraction.ts: both post-merge garnish blocks (chat + notes paths), the labeler test seam on ExtractionModels, and the import. The labeler role's only consumer is gone; the 'labeler' entry in BACKGROUND_MODEL_ROLES/ROLE_SLOT (core band) is now consumer-less — flagged to Platform, not crossed. - scripts/import-notes.ts: the settle-before-exit line + the named- clusters count in the summary. - tests: the cluster-naming describe (naming + failure-non-fatal tests) and the dead CLUSTER_* fixtures; the labeler-less BYOK test repurposed as the kill pin — a merge mints NO naming runs and NO label records. NOT touched (flagged, outside the agent band): matchClusterLabelings (core/clusters.ts), clusterContentId (core/graph.ts — now consumer- less), the appendClusterLabel/readClusterLabels port seams + store impls + ClusterLabeling type (store/core band), tests/clusters.test.ts's matchClusterLabelings describe. Cost-summary's naming stage untouched per Platform's ruling (historical spend display reads readNamingRuns, which stays live). Gates: typecheck, 921/921, lint, build. |
||
|
amosblomqvist
|
683cb9fd12 |
docs(extraction): catch the capture comments up to usage-on-error (02bd1b0)
Platform's partial-spend interlude made embedNewNodes fail BY RESULT: landed vectors + partial usage ride the return value instead of a throw. The extraction-side doc comments still claimed 'failed embeds contribute nothing' — now they state the pay-once truth: skipped (unconfigured) embeds carry no usage, failed ones carry the landed batches' steps, and a mid-run failure leaves a three-stage run's partial embedding spend on the ledger. The call-site comment says clustering runs with what LANDED (not 'without the new nodes'), and recordEmbedFailure's doc marks the catch belt-and-suspenders for injected embedders that still throw (production no longer does). |
||
|
amosblomqvist
|
d755ae82b7 |
fix(graph-context): honest wording for the unconfigured-embedder failure
Wave-6 review (Teacher stream): the injection runner's null-embedder branch failed with 'graph.embeddingModel is null', but the resolver collapses BOTH an unset model AND a set model whose provider key vanished into null — a key race mid-session would have reported a false 'null' setting. The message now names both causes honestly. Pinned by a new failure-mode test (null resolver → honest error, never a phantom query). Review verdicts (no changes needed): the session-deletion guard sits at every chat-merge boundary (per batch, fresh lookup, rides the existing AbortError path — nothing merged, nothing checkpointed, registry cleaned in the finally); persistRunUsage's deleted-chat skip composes with appendSourceUsage's vanished-source no-op so no usage record dangles; notes sources have no deletion surface so the notes path needs no guard. The embedding tail's non-fatal contract holds (failure → recordEmbedFailure → no stage, extractor/auditor usage still persists through the catch). |
||
|
amosblomqvist
|
0a4a9a11de |
BYOK provider embeddings (wave 6 chunk B): kill local transformers
Replace the @huggingface/transformers local embedder (~316MB ONNX cache) with an EmbeddingPort seam: graph.embeddingModel (nullable "provider:id" slug) selects an OpenRouter/OpenAI/Gemini adapter, resolved fresh per call from settings + the auth store so key rotations land immediately. - src/server/embeddings.ts rewritten: slug grammar, injected-fetch adapters (OpenRouter input_type, Gemini taskType batching, OpenAI index-ordered rows), honest resolution (null = skip-with-info, set model with missing key = honest error), embedNewNodes with model-change re-embed wipe + ghost pruning + usage steps, federated catalog (OpenRouter /api/v1/embeddings/models — VERIFIED live, the inverse path 404s; OpenAI text-embedding- filter; Gemini embedContent filter). - GET /api/models/embeddings: federated catalog, empty when no keys. - Extraction tail records stages.embedding usage into the run's money channel; EMBEDDING_PRICES overlay in core/prices.ts (outside the sync-generated block), callCost falls back to it; cost-summary folds the fourth stage; graph-context injection resolves its embedder per run, fails honestly when unconfigured, records query tokens. - Settings: GraphSettings.embeddingModel (null factory), section parser validates slug shape (id may contain colons — OpenRouter :free suffixes, live catalog fact). - Removal: @huggingface/transformers uninstalled, PATHS.modelCache and onnx tracing entries deleted, @huggingface tracing-excludes guard added, stale sharp/libvips comments updated to historical. Live-verified on port 3004: catalog federates 36 rows (33 OpenRouter + 3 OpenAI), embedNewNodes embedded 10 nodes via the free liquid/lfm-2.5 model (dim 1024, 435 prompt tokens, 1 call, $0), settings PUT rejects bad slugs with 400. Gates: typecheck, lint, 895/895 tests, build. |
||
|
amosblomqvist
|
a6338511d3 |
feat(graph): two-mode session deletion — the first destructive ledger operation
DELETE /api/chats/[id]?mode=preserve-graph|delete-nodes (mode required): - preserve-graph: only extraction bookkeeping dies (coverage rows, legacy checkpoints) — graph ledgers byte-untouched - delete-nodes: the chat's ledger traces die. Node survival = surviving provenance point / mapping line (notes imports carry empty provenance) / surviving event. The creator's aliases + provenance are stripped from a surviving node's create line (owner ruling 2026-09-23: name/statement/createdAt are the shared identity); orphan nodes, their edges, events, mappings, positions, embedding vectors and cluster members die; cluster labels stay (immutable) Gates: 404 unknown, 409 turn_in_flight / jobs_running (with jobIds), 503 extraction_winding_down after a 10s abort-and-poll; the purge is one sync block (ledgers → jobs → chat record last). rewriteLedgers is atomic (tmp + rename). graphRevision (max mtime over graph data files) rides the activity feed + the response as the change stamp. Anti-resurrection: the chat pipeline re-checks chat liveness at each merge — a run that slips past the wind-down aborts with nothing merged or checkpointed; its partial spend dies with the chat. |
||
|
amosblomqvist
|
9387418864 |
feat(models): clamp-on-write for pinned thinking levels + DEFAULT_THINKING_ORDER in the core barrel
Fix 1 — clamp, never reject (owner ruling, interlude 2026-09-22): a model swap with a pinned thinking level must JUST WORK. The owner hit the write-side 400 live (Claude Fable 5.1/xhigh → GLM 5.3 flash: "z-ai/glm-5.3-flash does not support thinking level xhigh"). The supported-level verdicts in validateChatSettings and validateModelsSettings are DEAD — the WRITE clamps the pin to the new model's nearest supported level (pi's clampThinkingLevel, the same translation piEngineForSpec applies at runtime) and stores the clamped result; the server is the truth — the record shows the clamped level after write. New pure helpers in model-catalog.ts, applied by ALL THREE write routes (section, defaults, per-chat — one verdict seam, so every surface that used the old 400 gets the clamp uniformly): - clampModelsLevels: every slot's level clamps against the SLOT'S OWN model (the parser already requires a model on each slot write). - clampChatLevel: the body's own model, else the record's resolved strong slot (the chain's fallback); no model anywhere → rides unchecked (runtime clamps at engine build; the turn gate refuses the model-less chat anyway). Fix 2 — DEFAULT_THINKING_ORDER moves from config.ts to core/settings (core barrel, AUTH_KEYS precedent — one source of truth, no drift). The UI imports it to auto-populate a thinking level on FRESH model picks; the ordering has no other client role. Legacy unset records keep the runtime defaultThinkingLevel fallback at engine build untouched. Import sites updated: agent/models.ts, tests. Verified live on 3004: the owner's exact case stores max (not 400); a bare chat-level write clamps to low against the strong slot's glm; unknown provider/model still 400s with the provider list. Tests: the level-verdict suites became clamp suites (the owner's case pinned verbatim: xhigh→max, minimal→low, off→low, supported pass-through, level-less slots ride); the machine-aware expectBareLevelVerdict chain is retired — clampChatLevel takes the strong slot as an explicit param, so the tests are pure again. Gates: typecheck, lint, 839/839, build. |
||
|
amosblomqvist
|
bb7804bd7f |
docs(prompts): the prober's dispatch tools get their grounded mention
The fix wave (4e1f545) put research + pdf in the prober's registry per the owner ruling, but the doctrine never mentioned them. Owner chose (2026-09-22) a brief grounded mention over schema-only discovery: the paragraph extending 'read his actual material' now covers pdf (same grounding for his PDF material, no copy-pasting) and research (sparingly, only when pinning a fact would genuinely sharpen a probe question — probing is measurement, not exploration, and a research brief is never a reason to stall the questions). |
||
|
amosblomqvist
|
4d5b38b540 |
fix(runner): the BYOK refusal steers — a dispatched job is never stranded
The BYOK refusal path (engineFor came up empty) only flipped the job record to error and returned: no steer. But the teacher had already dispatched — the ack promised a result as a new message — so the lesson stalled at the dock card with the turn over and the teacher waiting forever on a brief that would never arrive. Every other failed job steers (the research/demo/visual/pdf catch paths all deliver the kind's failure shape through finishJob); the refusal now follows the SAME convention: - record the honest error (envelope wording + missing tier) WITHOUT flipping status — finalizeJob owns the terminal flip after delivery; - build the kind's failure result (Research failed:/FAILED:/PDF failed: shapes the isSubagentFailure heuristics read) and steer it through finishJob, so the wake pipeline resumes the lesson and the teacher reads the refusal and tells the learner. Also cleaned the stale engineFor doc (the model-slots-era block still said 'factory-pin chain ... absent → config pins' — two comments had accumulated on the field; merged into one BYOK-accurate block). Platform's refusal test pinned the old no-steer semantics; updated to assert the new convention (steer delivered, record flipped by finalizeJob after). The other piEngineFor callers (teacher-agent, pi-run, visual-maker, pdf-reader) are deliberate honest-thrower fallbacks (opts.engine ?? piEngineFor(role)) — missed-injection detectors, kept as-is. |
||
|
amosblomqvist
|
bfa906e2f8 |
feat(models): BYOK all the way down — factory pins deleted, per-need refusal gates
Owner decision 2026-09-22: 'it doesn't make sense to have any default
model configured anywhere; the application is bring your own keys.'
config.ts: MODELS deleted; ModelRole becomes an explicit 9-role union;
DEFAULT_THINKING_ORDER (medium, low, high, minimal, xhigh, max) bakes
the unset-level default — first entry the model supports; explicitly
set levels keep pi's native nearest-clamp.
core/settings-resolution: the factory layer dissolves — the chain is
role override → slot → NOTHING. resolveSettings(record) takes no
factory. Empty chains resolve to null/undefined; a thinkingLevel-only
slot is inert (stored, never run). resolveConversationSpec: chat.model
override over the strong slot, null when both empty. resolveRoleSpec
spans all nine roles for the job/extraction engines.
Refusal envelope (core/settings.ts): modelsNotConfigured(missing) —
{error, code: 'models_not_configured', missing: [slot tiers]} +
ModelsNotConfiguredError with .envelope. Per-need gates, each honest:
- chat-handler: universal turn gate (new + continuing turns) returns
503 before beginTurn — no liveness, no write-ahead persist, no graph
trigger before the gate fires.
- runner.runJob: unresolvable engine lands the job as error with the
envelope wording; no phantom loop, no result steer.
- extraction: refuseIfStagesUnconfigured throws before any stage
(turn-service info-skips so coverage retries next turn; the import
route surfaces per-file code+missing); labeler-less runs skip
cluster naming with a console line and still merge.
Write side: slot entries must carry a model — level-only slot writes
are 400s (absent slot key = clear). piEngineForSpec resolves unset
thinking levels via DEFAULT_THINKING_ORDER; piEngineFor now throws
honestly; thinkingLevelFor/modelIdFor deleted; teacher-agent, pi-run,
visual-maker, pdf-reader all read engine.level.
model-catalog: defaults/slots/rolePins null when the chain is empty
(contracts nullable) — no phantom ids in payloads. Stores drop
MODEL_PINS entirely.
Tests: pin-leaners converted to explicit injection (per-chat model
pins on scripted chats — exactly what the setup flow writes; faux
engines for agent-construction tests). New coverage: the 503 gate
envelope, the runner refusal, the extraction refusal + labeler skip.
Gates: typecheck, lint, 815/815, build.
|
||
|
amosblomqvist
|
d85d8a58a0 |
fix(tools): wire the searchAvailable gate + research/pdf in the probe phase
Fix wave (2026-09-20), two owner-approved fixes: Fix 1 — the searchAvailable gate was dead code on the live wiring path: createTeacherTools RECEIVED ctx.searchAvailable but dropped it when building the inner JobToolContext, so jobTools' research gate always took its ?? true default (jobTools-direct tests passed; the production path never saw the signal). Threaded through — the gate now reads the live server signal on the teacher registry too. Fix 2 — the prober registry gains the research + pdf dispatch tools (owner ruling, 2026-09-20): these two are phase-omnipresent — the prober can verify facts or read learner PDFs while assessing. demo/visual stay teach-phase-only (they need the lesson's plan context). Same gating as the teacher side: toggle-filtered, research subject to the search gate (honestly absent when no engine is live — the prober's dispatch closes over the SAME jobTools builder, one gate, one semantics). probe_done stays always-on-by-construction; the learner-fs set untouched. - ProberToolContext (extends ProbeDoneContext with the job wiring); createProberAgent takes it; chat-handler injects jobStore + spawn + searchAvailable into the prober ctx. - TeacherToolContext.spawn's stale input union fixed (PdfInput was missing from the TYPE — runtime already passed it). - Tests: the threaded-gate describe in web-search.test.ts pins the LIVE factory path both ways (teacher + prober); question-tools' registry key-sets gain research/pdf; the prober toggle test pins the research:false path. Gates: typecheck, 807/807 tests, lint, build — all green. |
||
|
amosblomqvist
|
372d7f9975 |
feat(extraction): stage engines resolve the slot chain — engineFor on the pipeline
Chunk B, item 3 of the search port + slots wave (2026-09-20): the flagged gap closed — ExtractionModels deps were engine VALUES, keeping extractor/auditor/labeler config-pin-bound while the job runner went slot-native. The pipeline now takes the runner's exact treatment: - ExtractionPipelineDeps.engineFor: (role) => PiEngine — resolved FRESH per stage call, so a settings PUT between stages lands. Direct test values (extractor/auditor/labeler) remain the priority seam, checked FIRST; absent values fall to the chain; absent chain → config pins (unchanged). - All six stage call sites (chat + notes extract/audit, the cluster namer x2) resolve through stageEngine(role, direct). - chat-handler's extraction composition injects the full chain — piEngineForSpec(resolveRoleSpec(role, readSettings(), MODEL_PINS)) — same wiring as the job runner. Chunk B, item 4 (session-level slot-anchored model picks): VERIFIED already complete from Platform's model-slots wave — the conversation engine chain (chat-handler) anchors on effective.models.strong, the picker's default surface (catalogDefaultsFor) falls back to strong, both pinned in tests. No phase split (user decision, 2026-09-20). Capability-filtered pickers live in the settings UI (main 343be10). Tests: engineFor fills the stages end-to-end (both propositions merged through the chain, consulted in pipeline order) + the direct-value priority pin (engineFor that throws when consulted). Gates: typecheck, 795/795 tests, lint, build — all green. |
||
|
amosblomqvist
|
ed264c936d |
feat(search): web_search onto the injected port — Google CSE retired, research honestly disabled
Chunk A of the search port wave (2026-09-20), per master's assignment: - web.ts: the Google Custom Search backend is gone — web_search now runs on the core search port (SearchProvider), injected as a dependency (import law: the tool never imports server code). createWebTools(search) resolves the LIVE provider fresh per call; null = the honest unavailability string (no silent fallback, owner decision). web_fetch unchanged (needs no engine). - runner: JobRunnerDeps.searchProvider — the server composition threaded into the researcher; jobs already in flight when a key disappears still fail honestly through the same seam, never crash. - jobTools: the research dispatch tool DROPS from the teacher's registry when ctx.searchAvailable() is false (owner decision: registry-level drop — the model never sees the tool, no doomed dispatch, no error card). Absent seam = available (tests). chat-handler injects server/searchAvailable + searchProvider. - tests: stubFetch (the Google wire stub) retired; jobs.test.ts threads a recording stub provider through all four research runners and asserts the scripted query hit the port; new web-search.test.ts covers the factory contract (null → honest string, hits, error strings, fresh resolution, web_fetch independence) + the registry drop matrix. Gates: typecheck, 793/793 tests, lint, build — all green. |
||
|
amosblomqvist
|
f9e3a4c43d |
feat(models): strong/cheap/vision model slots + multi-provider engine federation
Model slots wave (2026-09-20), chunk B:
- settings: new `models` section {strong, cheap, vision} slots (each
field independent, same shape as a role override); ROLE_SLOT maps all
nine roles to tiers (strong=prober+teacher, cheap=researcher/
demoBuilder/extractor/auditor/labeler, vision=visualMaker+pdfReader);
resolution chain per role: override → slot → factory pin.
- config: ModelSpec gains provider (all pins openrouter, the BYOK
default); MODEL_PINS in stores.ts now spans all nine roles (the strong
slot's factory rep IS the prober pin).
- engine: piModels() registers EVERY builtin provider catalog;
piEngineForSpec(resolved spec) is the new engine factory (pi-ai
resolves per-provider keys from env itself); piEngineFor stays as the
config-pin fallback. chat-handler resolves the conversation chain
(chat override → strong slot → pin) and injects engineFor into the job
runner, closing the old gap where per-role overrides never reached
job engines.
- /api/models: models federate ONLY keyed providers (providers surface
with keyPresent, env-wins, fresh per call); slots (resolved tier pins)
join rolePins; catalogDefaultsFor inherits the strong slot; the models
settings section gets the same catalog gate as chat.
- pricing: PRICES/CONTEXT_LIMITS uniformly keyed provider:id
(regenerated from ALL pi-ai catalogs, 1354 keys); usage stamps
(TurnUsage, JobUsage, extraction stages) switch to provider:id —
owner decision: no legacy compat, old stamps render unpriced.
|
||
|
amosblomqvist
|
f1373d5de5 |
feat(search): Tavily + Brave behind a search-provider port; Google CSE retires
Search port + model slots wave, chunk A (owner design, 2026-09-20):
- core/search.ts: SearchHit + SearchProvider port + SearchStatus —
pure, browser-safe; adapters arrive as injected dependencies.
- server/search.ts: Tavily + Brave adapters (wire shapes VERIFIED
against docs.tavily.com's OpenAPI + Brave's dashboard docs — Tavily
auth is the Bearer header, not the legacy body api_key; Brave is
X-Subscription-Token) + engine resolution per owner decisions: a key
ALONE activates its engine (no engine choice needed), explicit
agent.search.engine pins the preference (both keys: Tavily first),
a pinned engine with a missing key is honestly unavailable — no
silent fallback, no default engine.
- Curated keys catalog: GOOGLE_SEARCH_API_KEY/GOOGLE_CSE_ID retire
(CSE closed to new customers, dies Jan 2027 — stored values become
preserved unknown vars); TAVILY_API_KEY (tvly- shape) + BRAVE_API_KEY
(length guard) join. UI keys-section TOOL_IDS now derives from the
catalog — cannot drift.
- GET /api/search: {available, engine, reason} — the honest status for
the settings gray-out; researcher-disable semantics ride the same
signal (Teacher stream rewires the tool onto the port).
- agent.search settings field (schema + sanitizer + strict parser);
web.ts's WebSearchResult is now the core SearchHit (re-exported).
|
||
|
amosblomqvist
|
3e263a18cc |
Graph pipeline cost metrics: capture naming spend, fold all channels, cost pill + panel
- Cluster namer bills the money channel now: per-step usage + model captured via onStep, persisted as NamingRunUsage records to data/graph/naming-runs.jsonl (idempotent per startedAt; failed/aborted runs persist too — money spent either way). callUsage promoted to extraction.ts module scope for reuse. - core/cost-summary.ts: the pure fold behind the pill — chat runs, notes-source runs, and naming runs into stage totals (extraction / audit / naming) + newest-first recent runs. Dollars INFERRED read-time via prices.ts, never stored; unpriced models excluded-but-counted (render '—', never a silent zero); tokens always sum. - GET /api/graph/costs (src/server graphCosts composition, thin route). - Graph view: bottom-left cost pill (cumulative dollars headline) + expandable breakdown panel (stage rows with tokens secondary, recent runs with time/title/merged/cost); fetched on mount, refetched when a pipeline run finishes; merged into one bottom-left stack with the running-pipeline pills. - Tests: pure-fold suite, naming capture wiring (full importNotes -> naming path), store idempotency + strict reads. |
||
|
amosblomqvist
|
524bf6024a |
feat(prompts): composeInstructions honors overrides — the P2 segment reconciliation
The other half-wired seam: Platform's settings 'agent.prompts' switches had no consumer in the agent layer, and their effectivePromptSegments mirrored the PRE-sources-v1 loader — the shared/ fragments dir (the learner-files doctrine) was invisible to it, so a naive delegation would have silently dropped the segment from both agents' prompts. - src/core/prompt-segments.ts (NEW, pure): the mirror-layout path grammar + segment types, defined exactly once; core stays zero-IO. - src/agent/prompt-segments.ts (NEW, the CANONICAL discovery): base + shared/ + per-agent fragments, override resolution, allSegmentPaths. Composition is an agent concern; the agent layer reads its own tree (the architecture law pins agent and store apart — the first direct refactor attempt tripped the enforced import/no-restricted-paths zone, hence the three-layer split). - src/store/prompt-overrides.ts: slimmed to the override-file CRUD behind the core port; shares the core grammar. Its doc no longer claims to be the canonical mirror. - teacher-agent: composeInstructions(kind, switches?) composes through effectivePromptSegments — a segment whose switch is ON and whose override file exists composes the override text. Switches-less calls use the cached pure-source composition (byte-identical to the old loader, pinned by test); AgentBuildOptions.promptSwitches threads effective.agent.prompts per turn, composing fresh so override-file writes flip the very next turn. - server barrel: re-exports the agent module's resolution for the settings routes (server→agent, the lawful direction). Tests: byte-identical source pin, override swap-in (on/off/file-absent), shared/ overridability in both agents, agent-construction threading, updated segment-list pins with shared/ in composition order. This file keeps owning PROMPTS_OVERRIDE_DIR (parallel-test discipline). Gates: 671/671, typecheck, lint, build. |
||
|
amosblomqvist
|
6c42393e42 |
feat(tools): the P2 toggle bridge — toolToggles first-class, definitive vocabulary
Reconciliation of the diverged tool-toggles seam (2026-09-19, master's P2): Platform injected the resolved delta as an ad-hoc intersection on AgentBuildOptions while the factories' enabled? param sat unread — the seam was inert on both sides. - teacher-agent: AgentBuildOptions gains toolToggles?: EnabledTools; both constructors thread it to the factories (createTeacherTools / createProberTools). An explicitly injected opts.tools registry is never filtered — the caller owns it. - Vocabulary ADJUDICATED with the user (2026-09-19): TOGGLEABLE_TOOLS is the definitive list — the job tools only (research/demo/visual/pdf). quiz/ask_user_question stay always-on forever (the prober's flow is graded-quiz brackets -> probe_done; quiz-off collapses assessment and teaching doctrine is built on graded verification — the wedge danger); the learner-fs set, plan tools, and probe_done also always-on. - settings-body (write side): unknown tool keys reject with 400 + the supported list, mirroring the prompt-switch validation; the read-side sanitizer stays lenient for forward-compat. - chat-handler: the intersection is gone (field now first-class) and the seam comment states the adjudicated vocabulary. Gates: typecheck, 28 new/updated tests green (tool-toggles-bridge, question-tools, settings-body), chat-flow + settings suites green. |
||
|
amosblomqvist
|
36c5d19437 |
fix(tools): the abort cascade — kill running tool calls, honest activity labels
Root cause of the live-test bugs (2026-09-19): pi passes every run's
AbortSignal to tool execute, but toPiTool dropped it — agent.abort()
only landed at the next step boundary, so a long local grep held its
turn hostage until the dev server's memory restart killed the run
mid-tool, freezing the record at its write-ahead state (the 'went
back in time' rollback).
- lh-tool: execute ctx gains signal (parent + timeout combined); every
call races a 30s system-level timeout with a 250ms cooperative grace —
abort kills a stuck tool but lets a poll-the-signal tool deliver its
partial results. Pre-aborted signals never fire listeners: checked
upfront. toolTimeoutMs is the test seam (pdfModelOverride idiom).
- fs-shared: walkFiles returns {timedOut, aborted} instead of throwing —
abort yields partial results.
- grep: the scan phase was unbounded (the walk's 10s deadline only
bounded collection) — same budget now checked per file; find/grep
check the signal and return partials with a notice.
- pending/chat: streamingToolActivity names the ACTUAL running tool —
a part sits input-streaming through its whole execution, so the old
shimmer called minutes-long greps 'Writing a question…'.
Tests: toPiTool guard (forward, abort, timeout, grace, question
bypass, ToolInputError passthrough), walk mid-abort, find/grep
pre-abort + mid-run abort, activity labels. Gates: 617/617.
|
||
|
amosblomqvist
|
1b9122c88b |
fix(pdf): split pagesRead from pagesConsulted — the honest badge count
Live-smoke follow-up the user approved (2026-09-19): the GEB read showed
"78 pages consulted" when the reader genuinely READ ~12 - pdf_search hit
snippets inflated the count. PdfResult now carries both:
- pagesRead: extract/render output entered the loop - the honest count
the badge and brief footer show ("read N pages", "Pages read: ...").
- pagesConsulted: every page whose content entered the loop, search-hit
context lines included - unchanged, still the citation audit set (the
reader's instructions still say "Pages consulted: N, N, ..." and the
footer still lists it when it exceeds the read set).
The unread-finish gate deliberately stays on CONSULTED: search hits are
genuine consultation (anti-fabrication duty), so a search-only brief is
honest and must deliver - pagesRead is a reporting refinement, not a
delivery gate. Pinned by the search-only faux test (read [], consulted
[1], delivered).
UI tolerates pre-split persisted records via ?? fallbacks (old steers
lack pagesRead).
|
||
|
amosblomqvist
|
7a3e19e95f |
feat(sources v1): the agent locates files itself — ls/find/grep + tool-call cards
User pivot (2026-09-19): no attachments UI at all. Pasting paths is native for technical users and alien for everyone else - so the agent gets the tools to find what the learner merely NAMES. The learner communicates in messages; the agent locates and reads. Zero cross-stream seams: paths already live in tool parts, so the sources "list" is the transcript. - ls/find/grep: pi-verbatim schemas, descriptions, and output formats (fd's glob semantics, rg's path:line: match format, pi's truncation notices), ported to pure Node - no binaries, no shell-out, read-only by construction. Deltas from pi: home-anchored resolution (the harness has no cwd), .gitignore replaced by hard noise exclusions (node_modules/.git/Library/.Trash), 10s walk timeout with partial results. - read: description swapped to pi-verbatim (doctrine moved to the prompt); relative paths now resolve against the learner's home like the rest. - prompts/shared/05-learner-files.md: one minimal doctrine fragment composed into BOTH agents (composeInstructions grew a shared/ dir, before the agent's own fragments). teacher/30-jobs.md keeps only the pdf-dispatch addendum. - ToolCallCard: the generic quiet tool-call card the user specced - collapsed one-liner (tool name + headline argument), expandable to full input/output, errors red, image payloads never render. Registered for read/ls/find/grep. - ui-message.ts: TeacherUITools entries for the four tools (mechanical). - tests: fs-tools suite (glob semantics, pi formats, exclusions, limits), registry/card renders, shared-fragment composition pins, tool-registry key updates, read re-anchor. |
||
|
amosblomqvist
|
2780166904 |
fix(pdf): make pdfjs survive bundling + real-PDF font fidelity (live smoke)
Live dispatch smoke against the Turbopack dev server (master's ask, 2026-09-19) caught two things unit tests cannot - they run unbundled: - Bundling pdfjs-dist broke its node fake-worker: the chunk-relative pdf.worker.mjs import 404s inside .next/dev/server/chunks. Fix: serverExternalPackages + tracing includes for legacy/ + standard_fonts/ (next.config.ts - a clearly-marked seam, same precedent as bcdcd51; the standalone output now ships the worker and font files, verified). - Real-world PDFs warned about missing standardFontDataUrl (non-embedded standard fonts silently fall back) and flooded the log with JBIG2 warnings on scanned files. Fix: standardFontsDir() + verbosity 0 in both the node ops and the headless canvas shim. Verified live end-to-end on GEB (802pp, text layer): teacher dispatch, 12-step read with vision renders, brief with accurate page citations, wake turn folding the resolution (Chapter IX, pp. 268-270) into the lesson. Scanned/DRM Polya correctly refuses to fabricate citations. |
||
|
amosblomqvist
|
9a90dd51d9 |
feat(T2b): sources v0 — read tool + pdf-reader sub-agent
Learner source files, on the teacher's own terms (nothing auto-injects): - read tool (src/agent/tools/read.ts): pi-read port — path/offset/limit, 2000-line/50KB truncation with use-offset continuation notices, image files (jpg/png/gif/webp/bmp) returned as native image content parts (new LhToolImageResult escape hatch in lh-tool.ts). Always-on in both the prober and teacher registries (user-decided 2026-09). - pdf-reader sub-agent (JobKind 'pdf', toggleable, ack-and-steer): triage via pdf_info (pdfjs text layer: per-page text/image/math-density), targeted pdf_extract, regex pdf_search (hits count as consultation), and pdf_render — pages painted by pdfjs onto a canvas shim in headless Chrome (the pdfium viewer doesn't paint headless), screenshot fed back image-in-the-loop (visual-maker idiom, providerSafePng applied). The unread-finish gate bounces a zero-consultation brief once. Teacher gets a text brief with page citations only; renders never leave the loop. - MODELS.pdfReader = qwen/qwen3.8-flash (user override 2026-09-19: the look-at-renders workload wants the vision-proven model; catalog listed glm-5.3-flash as image-capable but qwen is the known-good daily driver). - UI: PdfCompleteBadge + PdfBriefCard (steer-card, job-card), dispatch card + activity-dock kind entries (type-forced, flagged for UI review). - Prompts: 'Learner source files' section in teacher/30-jobs.md; read line in prober/10-probe-doctrine.md. Doctrine rides tool descriptions per the self-contained-tools principle. |
||
|
amosblomqvist
|
cab1632c74 |
feat(T2a): tool-toggle seam + phase-based question principles
Part 1 — tool availability (the settings seam, machinery generic): - core/pending.ts: TOGGLEABLE_TOOLS (today: research, demo, visual — the model-facing tool names), EnabledTools = the JSON-native seam shape Partial<Record<string, boolean>> (absent key = enabled, so old records behave exactly as today), filterEnabledTools shared helper. quiz/ask_user_question are always-on FOR NOW (user-decided); the filtering is generic so future toggleables (question-tool variants, the read tool, …) need no factory change. - createProberTools/createTeacherTools take enabled?: EnabledTools; the composition root (chat-handler, Platform's lane this wave) passes the resolved settings through. probe_done is re-added after the filter — always-on by construction (it ends the phase; a prober that can never conclude is a lockout). - Prompts stay STATIC under toggles this wave (user-decided); standing principle recorded: tools ship self-contained usage guidance in their descriptions; system prompts reference only always-on tools. Part 2 — phase-based question-writing principles (user-locked wording, drafted with the user over three iterations): - NEW teacher/21-question-principles.md (composes after 20-teaching): answerable-by-reasoning-from-the-session (the fairness rule that filters unanswerable questions without forbidding visible answers), never-just-reciting-your-words-back, PREFER problems that apply the knowledge (preference, not hard rule — honest recall is the fallback, not the habit), catch-confusion-early-and-nameable (wrong options as real misconceptions), answer-is-signal-not-verdict. - prober/10-probe-doctrine.md: 'no rush' retired; quick-measurement directive in — every question moves an edge (bracket, narrow, or characterize); a bracketed strand is done. Tests: TOGGLEABLE_TOOLS pin, prober/teacher filtering (absent = full, false drops exactly that tool, probe_done survives), prompt pins (the principles compose verbatim into the teacher prompt, the no-rush framing is gone from the prober). 547/547, typecheck/lint/build green. |
||
|
amosblomqvist
|
487f5d4cea |
wave 1 (platform stream): background-observability accuracy + owner-driven follow-ups
- observability: liveness-gated stale-job recovery, extraction run registry on globalThis with explicit 'embedding' phase, /api/jobs force-dynamic, client refresh nudges at every seam - headless: providerSafePng canonicalizer (multi-IDAT Chrome PNGs → single-IDAT) fixing qwen 400s - composer: Enter routes through pure enterAction — accepts a pending plan verdict (owner decision) - markdown chrome: session-scoped mermaid overlay + md-blocks component set + pan-zoom math, corner-copy (clipboard grant in electron main), math/table/code block fixes, heading weight 900 543/543 tests; typecheck/lint/build clean |
||
|
amosblomqvist
|
2678f04e7a |
feat: extraction coverage — message-granular ledger + token-batched runs (d132373 vision)
THE MESSAGE IS THE UNIT OF COVERAGE (vision-extraction-coverage.md): a per-chat append-only ledger (coverage.jsonl) records which messages the pipeline has extracted, by content fingerprint over exactly what the extraction view sees (one shared line builder — core/coverage.ts); the uncovered remainder is DERIVED at read time, never dual-tracked. A quiz answer merging into the asking message re-opens only that message; plan-block data-part attach never re-opens anything. - The trigger (turn-service) replaces every-3-quiz-answers with uncovered-tokens >= EXTRACTION.chunkTokens (4096) — quiz-less sessions extract too, content size is the honest signal. - runExtraction is now chunked: token-budgeted batches of the oldest uncovered consecutive messages (a tool call + result never split), each merged batch checkpointing its messages' coverage atomically — an abort or error loses at most the in-flight batch; merged batches stay covered. - Observation refs are batch-tagged (obs:<batchTag>:<sourceId>): re-runs of the same batch dedupe; a re-opened message's fresh understanding lands as a new event. Fixes the old obs:p1 position-collision. - New API: GET /api/graph/coverage (exact per-session derived coverage + running flags, most-uncovered first) and POST /api/graph/coverage/extract (manual run for one session, 409 while in flight). - Graph view chrome: the coverage button opens a panel with every session's covered/uncovered tokens and a per-session Extract button — the manual answer to aborted runs, errors, and dormant tails. Renders even when the graph is empty: an empty graph with uncovered sessions is exactly when it matters. Polls only while open. - Notes imports keep the whole-file checkpoint path (coverage undefined). - Tests: pure-core coverage suite (11), chunked-pipeline suite (3: multi-batch + no-op, abort-keeps-earlier-batches + remainder re-run, re-open alone + data-part invisibility), panel render suite (5); pendingQuizAnswers and its tests deleted as dead code. 486/486. Live-smoked on real GLM: manual extract of "teach me modal realism" ran one batch, wrote 4 coverage entries, panel flipped to "1.3k covered · complete" with the button disabled, graph refetched (nodes 11), usage persisted with real stats. |
||
|
amosblomqvist
|
1d45d77303 |
feat: graph context injection — the session's prior knowledge, steered in at session start (vision-context-injection.md)
One shot per chat, fired at the first turn's onFinish: the session's topic (its only typed message) is embedded as an EmbeddingGemma retrieval query; the semantically closest graph propositions are selected under the session's token budget (pure budget, no similarity threshold — the decided doctrine) and delivered as a marker-only steer through the wake pipeline, queuing behind pending questions like every sub-agent result. - contracts: ChatSettings.graphContextTokens (default 1024, max 4096, step 128, 0 = off); GraphContextData + graphContextText; the graph-context JobKind + buildGraphContextSteer (id steer_ctx_<chatId>) - core/graph-context.ts: brute-force cosine top-K, greedy chars/4 fill under budget, most-recent UnderstandingLevel fold, count cap 64, parse-don't-cast at the marker seam - server: embedQuery (role-asymmetric query format, same warm singleton — vector-space parity by construction); the runner (startGraphContext- Injection one-shot gates; runGraphContextJob validates ledger parity); wake branches (chatIdOf/jobIdOf, finalizeJob done, recoverStaleJobs re-deliver or contextRespawn); abortJob kills context jobs SILENTLY (nothing was promised in the lesson); turn-service trigger seam - adapter: data-graph-context flattens to the canonical block (malformed parts have no model face) - UI: the budget slider (home dock + /chat/new draft, the session-data strip); the quiet SystemLine "~537 · 10 propositions"; dock kind "Knowledge context"; model-picker PUT now spreads settings (a model pick can no longer clear the sibling override) Live-smoked end to end on the real embedder + real GLM: seeded ledger (10 propositions), session on "modal logic: possible worlds and the necessity box" — job in the dock, queued behind the prober's question, delivered on the answer, SystemLine in the transcript, wake turn reads the block and continues the probe. Gates: tsc, lint, 459/459, build. |
||
|
amosblomqvist
|
f5edc7f0a5 |
Semantic topic clusters: local embeddings replace Louvain
Louvain only ever found the reasoning chains (the graph is disconnected by nature — every connected component became its own cluster) and edge-orphans could never join anything. Clusters are now RELATED TOPICS: propositions grouped by meaning. - Embedder (src/server/embeddings.ts): in-process transformers.js v4 running EmbeddingGemma-300M q8 (MTEB 68.4, ~300MB one-time download), MRL-truncated to 512d, "title: none | text:" prefix on 100% of inputs, L2-normalized so cosine = dot. Model/dim mismatch voids the ledger (full re-embed). Injected at the composition root — the agent layer never imports the server layer (architecture law). - Ledger (data/graph/embeddings.json): model + dim + vectors + the persisted cluster partition. Reclustering runs at the END of every embedding batch (post-merge hook, both front doors) — never per read (the agglomerative pass is ~1.5s at 338 nodes). - Clustering (src/core/semantic-clusters.ts): hand-rolled average-linkage agglomerative over cosine (npm HDBSCAN is an abandoned alpha). Stragglers stay singletons by default; attachStragglers is the all-in toggle. Threshold 0.60 swept on the real graph: 26 topics, 74/90 edge-orphans land in real topics, 10 stragglers. - Consumers (snapshot, settle, namer, replay) read the persisted partition — one cluster concept everywhere; core-survival label matching unchanged (20/26 topics kept labels). - Louvain retired: detectCommunities + graphology* deps deleted (graphology-louvain had leaked into the client bundle since the clusters wave); scripts/bench-graph.ts (dead FA2 bench) removed. - scripts/embed-graph.ts: backfill/cold-start (338 nodes in 150s including the one-time model download). |
||
|
amosblomqvist
|
975a78a304 |
Cluster gravity: per-cluster centroid pull force in the engine
The 'virtual node' realized as a centroid force: two allocation-free O(n) passes per tick (membership resolved once at sim creation; Float64Array sums). Measured indistinguishable from force-off at 338 nodes (271ms vs 272ms per 400-tick settle). - createGraphSimulation: SimOptions.clusterPull + linkStrength + velocityDecay (physics controls plumbing); degree-scaled link stiffness so linkStrength=1 is exactly d3's default - settleGraphLayout split into layout-settle.ts (it computes the Louvain partition for the pull); layout.ts stays browser-barrel-pure - barrel no longer re-exports clusters.ts: graphology + louvain are fully out of the client bundle (verified against built chunks) - replay passes the per-batch partition; CLUSTER_PULL_STRENGTH=0.15 tuned by sweep: cluster spread 63->36 world units, centroid separation and total span unchanged (no blob merge, no collapse) |
||
|
amosblomqvist
|
0c14ea425d |
Option A: incremental anchored force layout, persisted server-side
The layout now EVOLVES instead of reshuffling on every extraction. A pure
core module (src/core/layout.ts) warm-starts forceAtlas2 from the positions
ledger (data/graph/positions.json, read-modify-write like sources.json,
updatedAt stamp for future live-physics conflict handling), re-blends every
previously-positioned node toward its anchor each iteration (retention 0.85
- old placements stay recognizable, strong forces still win), seeds new
nodes near their positioned neighbors, and seeds orphans on the rim outside
the core so they read as unconnected knowledge.
- Port/store: GraphStore.readPositions/writePositions; snapshot carries
positions when the ledger exists (optional - fresh graphs unchanged).
- Extraction pipeline settles synchronously after every merge (chat and
notes paths), before usage persist - the next /api/graph fetch already
reflects new nodes.
- View: the ledger IS the layout; the local one-shot FA2 pass survives
only as a fallback when no ledger exists.
- scripts/replay-layout.ts: simulates organic growth by replaying the
current 338-node graph one imported file at a time (real import order,
edges activate when both endpoints exist). Real run: 338 positions in
0.3s, 90-degree-0 orphans collecting at mean r=338 vs connected r=194.
- Also restores the cluster region fills (hull background colors) that
|
||
|
amosblomqvist
|
f0267a5fd9 |
Clusters phases B+C: labeling records, the naming call, snapshot payload
Core-survival labelings (vision section 1, DECIDED 2026-09-13): ClusterLabeling records in a new append-only ledger (data/graph/cluster-labels.jsonl), immutable, id = clu_ + sha256 of the sorted core so re-naming races and re-runs dedupe at the store seam. matchClusterLabelings (core/clusters.ts, pure): a community keeps a record's label while at least half its core survives in it; both sides single-successor (a split keeps one name; greedy by overlap, ties to the newer record). The namer (src/agent/subagents/cluster-namer.ts): ONE singular cheap LLM call (new 'labeler' role, glm-flash/low) names every new/changed community at once, given member names + statements (capped at 12 members per cluster). refreshClusterLabels runs POST-EXTRACTION (user-decided) from both front doors, fire-and-forget — naming failure never fails the run; refreshes serialize behind a chain so parallel imports never mint duplicate labels. import-notes.ts awaits clusterNamingSettled before exit. Payload: GraphSnapshot.clusters (>= MIN_CLUSTER_SIZE 3) joins detection with labelings at read time — claimed clusters carry the record's label and id; unlabeled ones carry a member-hash id and null label. Detection is deterministic so no memo in v1 (ms at current scale; revisit at vault scale). Tests (372/372): matcher half-rule/split/merge/greedy/tie cases; naming end-to-end through importNotes (one call, one record, claim on re-refresh), naming failure non-fatal; snapshot derivation with labeling join; 11 test pipelines now pin an explicit labeler seam. |
||
|
amosblomqvist
|
c3cc55c585 |
Extraction tool calls: tolerate omitted-empty fields (weak-model trap)
Diagnosis from the real 25-file notes import (user-pasted session logs):
almost every run hit 'Validation failed for tool submit_findings/review:
edges must have required properties edges' — the model (glm flash, low
thinking) omits top-level fields when they're empty, and pi-ai's
structural JSON-Schema validation runs BEFORE the zod layer, so each
omission cost one of the extractor's maxTurns 4 (full payload
regeneration) and occasionally burned out the run entirely (the two
FAILED imports).
The model is technically wrong, but the schema provokes it:
required-but-usually-empty arrays are a known weak-model anti-pattern.
- lhTool's model-facing declaration now converts with io: "input"
(the declaration describes what the MODEL sends): .default() fields
drop out of required. No-op for teacher tools (none use defaults).
- findingsSchema.edges / auditSchema.dropped: .default([]) — omission
means "none"; attribution: .default({quiz:{}}) for quiz-free sessions.
- Prompts hardened in all three agents: include every top-level field,
never omit because empty.
Structural probe: validateToolArguments now passes the exact pasted
failure shape; zod re-parse in execute applies the defaults. 355/355
tests, typecheck, lint, build green. (lh-tool.ts is the shared engine
seam — flagged to master: io:"input" is the correct mode for every
model-facing declaration, teacher path verified unaffected.)
|
||
|
amosblomqvist
|
543cf33789 |
Notes front door: format-agnostic import into the graph pipeline
The same extractor→auditor→merger pipeline now runs over raw personal
notes in arbitrary formats (vision §1, DECIDED 2026-09-13):
- NotesSource records (data/graph/sources.json, deterministic id from
path hash → idempotent re-imports) play the chatId role in mappings,
checkpoints, and observation events; usage lands on the source record
(appendSourceUsage, idempotent per startedAt, kept across upserts).
- mergeFindings is chat-store-free: callers pass MergeSource
{messages, foldedAt} — sessions pass the chat record, notes pass
messages:null (no quiz-event rebuild; a deterministic rebuild needs
structured quiz tool parts).
- Second extractor prompt (NOTES_EXTRACTOR_INSTRUCTIONS) for general-
purpose extraction; understanding judges the AUTHOR's grasp. Auditor
wording input-agnostic. extract/audit share one generic runSubmitAgent.
- graphSnapshot.fileSources (srcId→filePath); notes sources render as
static path entries in the detail card (FileTextIcon, never a link).
- One shared pipeline instance (exported from chat-handler): same
single-flight registry, live status chips (label = file path), usage
accounting, and true abort for both front doors.
- Transport: POST /api/graph/import (multipart, 3-wide worker pool) +
Import notes… button in the graph view; scripts/import-notes.ts for
batch runs (concurrency 4).
- real-data-extraction/ gitignored (raw import sources never committed).
355/355 tests; typecheck, lint, build green.
|
||
|
amosblomqvist
|
de0c286311 |
feat: model + thinking control — the stats bar's picker, on pi-ai's catalog
The model-control wave (user-decided design, 2026-09-12): ONE controller
for the conversation agents (prober/teacher are functionally one agent);
background sub-agents stay config-pinned. pi-ai owns the registry — the
shipped 366-model OpenRouter catalog (cost, context, modality,
thinkingLevelMap), getSupportedThinkingLevels/clampThinkingLevel for the
per-model thinking truth, and streamSimple's automatic clamp. No second
registry to drift out of sync.
Seams:
- config.ts: ReasoningEffort retired for pi's ThinkingLevel
(off|minimal|low|medium|high|xhigh|max). GLM roles go honest at "low" —
pi's catalog says GLM-5.3-flash accepts exactly low|high|max, and the old
"minimal" was silently clamped to low by streamSimple since the swap;
nothing changes behaviorally. visualMaker → qwen/qwen3.8-flash (fixes
the dead qwen/qwen3.8-max slug — the catalog only ships -0902; also the
user's cost pick, 0.15/0.47 per-1M).
- src/agent/models.ts: piEngineFor(role, settings?) — provider+id resolved
from the catalog, thinking level from settings ?? config, CLAMPED to the
model's supported list (a settings change between models can never send
an invalid level). PiEngine gains `level` (observability/tests).
- ChatRecord.settings {model?: {provider, id}, thinkingLevel?} — written
by PUT /api/chats/[id]/settings (replace-whole, no updatedAt churn),
read by chat-handler at turn composition: agents are stateless per turn,
so the change is next-turn-effective by construction. The turns log
still records the ACTUAL billed model (the honest witness).
- GET /api/models: the configured-provider catalog as client-safe DTOs
(core/contracts/models.ts) + config defaults; a second provider joins
the picker by registering it in piModels() — nothing else changes.
- PUT settings validation gate (src/server/model-catalog.ts): unknown
provider/model or unsupported level → 400 carrying the supported list,
never a silent clamp.
- scripts/sync-pricing.ts: regenerated PRICES/CONTEXT_LIMITS from pi-ai's
shipped openrouter.json (offline; same source as the engine + picker).
All 366 models priced; glm's stale 0.15/0.5 → pi's 0.075/0.25, limit
1310720 → 1048576 — cost is inferred at read time, so history re-prices.
- UI (components/model-picker.tsx): stats-bar right side becomes model +
thinking buttons; menus open ABOVE; fuzzy search over "provider id
name" (core/fuzzy.ts — pi's /model pattern); model rows show name/id/
price/context; the thinking menu lists exactly the chosen model's
supported levels (GLM: low|high|max — no dead options, no "off").
Model switch drops an incompatible pinned level (falls back to config,
which clamps honestly) rather than carrying a 400.
Tests (350/350, +15): catalog projection + defaults, the settings gate,
pi thinking truth pinned against upgrades (glm low|high|max; minimal→low;
off→low — GLM cannot disable thinking), the engine factory's
override/fallback/clamp/throw matrix, fuzzy ranking, store round-trip,
and the re-priced folds (usage, stats bar, activity dock).
Live smoke (disposable driver, deleted): catalog route 366 + defaults;
glm+minimal → 400 + supported; turn 1 billed qwen/qwen3.8-flash under
the override; mid-session switch flipped turn 2 to glm; clear-overrides
restored the config pin. ALL CHECKS PASS on real OpenRouter.
|
||
|
amosblomqvist
|
4ba594bab5 |
fix(arch): parse, don't cast — the UI parse seam for model-authored input
The architectural answer to the validation-crash question: validation IS
first at the model-facing boundary (pi validates arguments before
execute; failures return to the model same-turn — all three live
incidents retried successfully). But the same bytes also exist UNVALIDATED
by design in two other places: the live wire streams raw tool-input
deltas DURING generation, and the transcript keeps the raw input of
REJECTED calls (honesty + the replay invariant). The consumers were
supposed to trust part STATES — but that trust was enforced by
TypeScript casts ('part.input as X'), which are compile-time lies with
no runtime check behind them. That is the hole both crashes came
through, and it should not have been possible.
The structural rule now: model-authored INPUT is consumed only through
parsers running the SAME zod schema the model faced — boundary-valid ⇔
consumer-parseable, mechanically. Server/learner-authored OUTPUT stays
trusted (our code authors it).
- src/core/tool-parts.ts (new): parseQuizInput (display parse over the
shared quizInputBase — strips the teacher-only nodeId, accepts both
prober and teacher boundary shapes), parseAskInput, parseProbeDoneInput.
- contracts/probe.ts: probeDoneInputSchema is now core-owned zod — the
probe_done tool declares it (single source of truth), the seam runs
the literal same schema.
- question-dock.tsx: the dock parses its input at entry; unparsable
renders the quiet malformed-question fallback, never a crash.
- probe-done-card.tsx: the input preview goes through the parse; the
old state reasoning and shape guards collapse into it.
Pinned in tests/tool-parts.test.ts: both exact live crash shapes
(stringified quiz options, stringified assessment), boundary-parity
cases for all three schemas, nodeId stripping, mid-stream partials;
render-cards pins the dock fallback markup.
Gates: tsc, lint, build, 328/328.
|