d6b229865a38faf2a2bc0e0eee3157b02f1c06c7
8
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 amosblomqvist
|
543cf33789 |
Notes front door: format-agnostic import into the graph pipeline
The same extractor→auditor→merger pipeline now runs over raw personal
notes in arbitrary formats (vision §1, DECIDED 2026-09-13):
- NotesSource records (data/graph/sources.json, deterministic id from
path hash → idempotent re-imports) play the chatId role in mappings,
checkpoints, and observation events; usage lands on the source record
(appendSourceUsage, idempotent per startedAt, kept across upserts).
- mergeFindings is chat-store-free: callers pass MergeSource
{messages, foldedAt} — sessions pass the chat record, notes pass
messages:null (no quiz-event rebuild; a deterministic rebuild needs
structured quiz tool parts).
- Second extractor prompt (NOTES_EXTRACTOR_INSTRUCTIONS) for general-
purpose extraction; understanding judges the AUTHOR's grasp. Auditor
wording input-agnostic. extract/audit share one generic runSubmitAgent.
- graphSnapshot.fileSources (srcId→filePath); notes sources render as
static path entries in the detail card (FileTextIcon, never a link).
- One shared pipeline instance (exported from chat-handler): same
single-flight registry, live status chips (label = file path), usage
accounting, and true abort for both front doors.
- Transport: POST /api/graph/import (multipart, 3-wide worker pool) +
Import notes… button in the graph view; scripts/import-notes.ts for
batch runs (concurrency 4).
- real-data-extraction/ gitignored (raw import sources never committed).
355/355 tests; typecheck, lint, build green.
|
||
|
amosblomqvist
|
2d717fd932 |
Swap the server engine to pi-ai/pi-agent-core (Band B+C of docs/pi-engine-swap.md)
The Vercel AI SDK survives only as Band D (useChat client + UIMessage
record types); @openrouter/ai-sdk-provider is uninstalled.
- src/agent/models.ts: PiEngine {model, streamFn} via createModels +
openrouterProvider subpath; thinkingLevelFor maps config effort none->off
- src/agent/lh-tool.ts: neutral LhTool (zod authoring, optional execute,
clientQuestion flag); toPiTool converts via z.toJSONSchema and re-runs
zod inside execute so refinements ride the error-result retry path
- src/server/pi/record-adapter.ts: UIMessage records -> pi Messages
(dangling tool calls get synthesized interrupted results)
- src/server/pi/ui-stream.ts: SSE bridge emitting AI-SDK-shaped chunks so
the useChat client parses unchanged (continuation id rule, question
tools never emit output, tool-input-error for bad authoring)
- teacher-agent.ts: stateless per-turn pi Agents; shouldStopAfterTurn
stops only when every question-tool call resolved cleanly (errors retry)
- chat-handler.ts: buildAgent(outgoing) factory seam (stateless doctrine);
assertValidUiMessages owns the retired SDK validation gate
- sub-agents via shared runSubagent (pi-run.ts); extraction stages stop
after a clean submit; visual-maker keeps its hand-rolled loop over
direct streamSimple calls; aborted model calls bill no usage
- tests: all model seams ported to pi-ai faux provider (scripted
FauxResponseSteps, agent factories); handleChatRequest second param
is now an agent factory
Gates: tsc, lint, build, 305/305. Live GLM smoke (disposable driver,
deleted): prober streaming + persistence, question seam, turn-2
continuation same message id, per-turn real token usage - ALL PASS.
Data wiped at the pivot (chats/graph/jobs, .gitkeeps kept).
|
||
|
amosblomqvist
|
9174cde55e |
Extraction observability: usage capture, live status payload, true abort (Wave 4)
Extraction is real spend and background work; the dock needs to see and stop it. User decisions: persist all cost data (chat record), per-stage usage granularity, and TRUE cancellation for abort. USAGE CAPTURE (live + persisted): - extraction.ts: per-call onStepFinish captures each billed step's usage (SDK LanguageModelUsage -> CallUsage: input/output/cache read+write/reasoning) into the registry's live per-stage usage (extractor/auditor; the merger is deterministic — no model call). - /api/graph/status runs now carry usage (model + steps per stage); cost/context folds are the consumer's (core/prices.ts — nothing stores dollars). - ChatRecord.extractions: append-only per-run usage log (mirrors the turns log): startedAt key, at, durationMs, merge stats when the merger ran, stages. Persisted on success AND failure — a failed run still spent money. Idempotent per startedAt; the list payload (ChatSummary) never leaks it. - EmissionStats moved to contracts/graph.ts (its contract home); core/graph.ts re-exports for existing importers. - Verified end-to-end against the real pipeline (backfill one chat): per-step token counts, reasoning tokens, and the auditor's growing prompt-cache reads captured and persisted. TRUE ABORT (the dock's abort button, user decision): - RunningExtraction owns one AbortController per run, shared across phases; abortExtraction(chatId) fires it and returns whether a run was aborted (idempotent no-op otherwise). The pending model call receives the signal — the provider cancels the request, spend stops. throwIfAborted() at phase boundaries ends runs aborted in sync gaps. - Abort is a deliberate stop, not a failure: same ledger discipline (nothing merged, nothing checkpointed, partial usage persisted) but the run reports zero emissions instead of throwing. Exported via src/server/observability.ts for master's /api/activity/abort route. - Tests: mid-run abort (hanging mock that rejects on the signal), live-accumulation + persisted-record assertions, store-seam idempotency, and the summary-payload leak guard. |
||
|
amosblomqvist
|
0c12d53fca |
Background pipeline status: registry + /api/graph/status + graph-view chips
The background extraction pipeline is invisible on the home surface — essential observability per the user's directive: the graph view must show what is currently running (extracting / auditing / merging) and for which chat. - extraction.ts: module-level registry of in-flight runs (chatId, phase, startedAt), updated at each phase transition in runExtraction and cleared in finally (done or failed). extractionStatuses() is the read; one process, one registry. - GET /api/graph/status: thin transport over the registry, chat titles joined from the chat store. - graph-view: polls every 2s and shows one chip per running pipeline (spinner + phase + title + elapsed seconds, bottom-left); when a run finishes (some -> none) the graph refetches, so merges land live. Status is cosmetic — poll errors never surface. - Unit test drives a mocked pipeline run and observes the registry mid-flight: 'extracting' during the extractor's generate, 'auditing' during the auditor's, empty after. |
||
|
amosblomqvist
|
bdd9cac2b7 |
Node schema pivot: short name (label) + full-precision statement; description dropped
With real extracted data, full-sentence statements as canvas labels proved unworkable: 10-line label blocks, illegible overprint clusters, bottom-edge clipping. Per the user's 2026-09-08 decision: - GraphNode/ExtractedProposition gain `name` — a very short, glanceable label (glossary-headword style: "Will to power"), presentational only: not unique, not identity, never precise. Set at creation; link-appends never change it (first line defines; labels stay stable across sessions). - `statement` keeps its full precision and becomes the identity anchor; the detail card shows it under the name. `description` is dropped. - Extractor coins the name (freshest session context); auditor carries it through on create, rewording only clearly unglanceable names; linked propositions keep the existing node's name. - Renderer: canvas label = name; search matches name + statement; detail card = name (bold) + statement (muted). - Store guard requires name (pre-pivot node lines are skipped as legacy); vision.md node-schema bullet + amendments table updated (2026-09-08). |
||
|
amosblomqvist
|
b07c907ffd |
Graph redesign: true propositions, follows/supports, auditor-owned identity (vision §1 minimal implementation)
The minimal implementation of the 2026-09-02 graph vision (docs/vision.md §1):
- contracts/graph.ts: every node is a TRUE proposition (statement is both
rendering label and identity key; required description; no kind, no name).
Edges are follows (derivable) / supports (grounds without entailing) with
message-granular provenance points {chatId, messageId}. Events reduce to
correct/miss/dont-know plus observation (understanding notes); no taught
events, no misconception kind.
- extraction.ts: extractor is TREE-BLIND (transcript only, no plan tree) and
free-form; auditor sees findings + the current graph, owns faithfulness and
identity (link-or-create verdict per proposition, recorded as linkTo);
transcriptView tags every line with its message id (provenance anchors).
- core/graph.ts merger: applies verdicts deterministically — linkTo appends
a link line (phrasings join aliases, provenance unions; reads merge by id),
absent creates; NO string matching anywhere. Quiz events rebuilt from tool
parts with attribution-only nodeId resolution; understanding notes become
observation events (obs:<sourceId>); mapping lines are the verdict audit
trail and make re-merges idempotent.
- deriveNodeStates: transitional — latest quiz event wins (known/missed/
dont-know, else candidate); taught unreachable until the state-model
ideation lands.
- graph-context.ts + 50-graph-context.md: follows chains, hotspot grouping,
state vocabulary updated for the new event kinds.
- graph-view.tsx: node label is the statement; detail shows statement +
description.
- tests: merger verdicts (link, no-string-match, cross-chat assertions),
tree-blind pipeline end-to-end, follows vocabulary in context tests.
276/276 green.
|
||
|
Amos
|
2e4e43b8c4 |
Graph nodes gain a name; nodes are objective facts (no learner/session references)
Schema: GraphNode + ExtractedProposition carry 'name' — a short 2-6 word header, the rendering handle. Node theory made explicit: a node is an objective world-facing fact; the learner's relationship to it lives entirely in the event ledger (state labels are derived, never stored). Extractor + auditor instructions enforce it — misconceptions state the wrong idea itself as a claim, never 'the learner believes Y'. Legacy ledger lines without a name derive one from the statement on read. graph-view: labels use the name; the detail panel shows name as header and statement as body. |
||
|
Amos
|
af90fadcda |
Extraction pipeline: extractor → auditor → merger — the sole graph writer (stage 5)
The knowledge graph's write channel is now the pivot's target
architecture (vision §1 construction):
- src/agent/subagents/extraction.ts — the pipeline factory
(createExtractionPipeline({ chatStore, graphStore, extractor?,
auditor? }); models env LH_EXTRACT_MODEL ?? LH_MODEL):
* transcriptView(): the compact view both LLM stages see — final plan
tree with node ids, conversation verbatim, QUIZ lines with outcomes,
mark_done lines; job tool parts are one-liners, plan ops fold into
the tree line.
* Extractor: transcript → SessionFindings (propositions incl.
misconceptions and prose-only supports edges, attribution for
UNTAGGED probes, all with verbatim evidence) via one zod-validated
submit_findings call.
* Auditor: transcript + findings → the verified subset + drops (the
faithfulness net) via submit_review.
- src/core/graph.ts — emitChat/foldStaleChats/replayFinalTree retire;
mergeFindings() replaces them as the pipeline's deterministic apply
stage: exact/alias identity resolution (one mapping line per (chat,
graph node)), edge dedupe, and event rebuild FROM THE TRANSCRIPT's tool
parts — kinds/timestamps/refs/notes deterministic, the LLM's only event
say being WHICH proposition an untagged probe tested (retroactive —
the vision's open question #3, resolved structurally: tagged events
resolve deterministically; attribution rescues untagged and
tagged-but-unmapped ones; unattributable land null + question text).
pendingQuizAnswers() is the trigger helper.
- Trigger (turn-service.ts triggerExtraction, replacing the old
foldTurnIntoGraph body): once a chat accumulates
EXTRACTION.everyNQuizAnswers (LH_EXTRACT_N, default 3) new answered
quizzes, fire the pipeline async — single-flight per chat, failure
checkpoints nothing and re-runs on the next trigger. Deliberately a v1
placeholder isolated in ONE seam, trigger-agnostic pipeline.
- GET /api/graph is now a pure read (foldStaleChats gone).
- NodeMapping.treeNodeId → sourceId (the mapping key is now the writer's
local proposition id — tree node when tree-derived); store strict-read
schema updated. Local ledger data regenerates through the pipeline.
- Tests: graph.test.ts rewritten around mergeFindings (identity,
attribution paths incl. ghost-rescue, idempotency, cross-session
provenance, trigger counting), graph-context.test.ts merge section
converted to findings-driven, NEW extraction.test.ts — transcriptView
units + the full pipeline with mock models (audit drops, failure
checkpoints nothing, single-flight). 195/195, tsc/eslint/build clean.
Probe attribution (vision open question): probes attach retroactively by
the extractor against the whole transcript; a pre-tree probe maps to a
proposition the tree only articulates later. Fuzzy/embedding merge
adjudication stays deferred (conservative exact/alias only).
|