learning-harness

A standalone learning harness: a teacher agent (Next.js + Vercel AI SDK, models via OpenRouter), graded quiz/ask interactions, a formal session plan tree the teacher authors and edits live, and a persistent cross-session knowledge graph of what the learner knows — the graph is the app's home page.

Not a chat app with a graph bolted on: sessions are probe→teach lessons, the plan tree is a structured, agent-editable object, quiz outcomes reach the graph through the extraction pipeline as ledger events, and the graph feeds back as the teacher's context so probing compounds across sessions.

• Living design doc (all decisions): docs/vision.md
• Build plan + wave/gate results: docs/plan.md
• docs/architecture.md is the canonical architecture spec — layers, ports, the import law, the single-writer rule, the two graph channels.

Commands

npm install          # once

npm run dev          # http://localhost:3000
npm run build        # production build
npm start            # serve the production build

npm test             # all tests (node:test via tsx; deterministic, no LLM)
npm run lint         # eslint — includes the import law (architecture §1)
npm run typecheck    # tsc --noEmit

npx tsx scripts/bench-graph.ts   # force-layout benchmark (synthetic graphs)

Environment (.env.local)

Secrets and machine-specific paths only. Model choice and thinking budgets are product tuning, not environment — they live in config.ts → MODELS (teacher, researcher, demoBuilder, visualMaker, extractor, auditor; each with an OpenRouter model id + reasoning effort). The extraction trigger threshold is there too (EXTRACTION.everyNQuizAnswers).

Variable	Purpose
OPENROUTER_API_KEY	required — all LLM calls go through OpenRouter
GOOGLE_SEARCH_API_KEY + GOOGLE_CSE_ID	researcher's web_search (Google Custom Search)
CHROME_PATH	optional explicit Chrome path for headless verification (puppeteer-core uses system Chrome; nothing is downloaded)

Architecture

Five layers, dependency arrows pointing down only, the import law machine-enforced as four ESLint zones (shell → barrels only; core and agent never import below themselves). The full spec: docs/architecture.md.

graph TB
    subgraph SESSION_UI["Browser · session page (app/chat/[id])"]
        TR["Transcript — streamdown markdown,<br/>KaTeX \\(..\\)/\\[..\\], mermaid,<br/>probe→teach phase divider"]
        DOCK["QuestionDock / AnsweredCard<br/>quiz graded client-side<br/>(question-core)"]
        TREE["TreePanel — plan tree,<br/>topo order, done/current/upcoming"]
        ART["Artifact cards — research brief,<br/>demo in sandboxed iframe, SVG"]
        COMP["Composer — gated by question<br/>or turn; jobs run in parallel"]
    end

    subgraph HOME_UI["Browser · home page (app/)"]
        GV["GraphView — sigma.js + forceAtlas2,<br/>run-then-freeze, node state colors"]
    end

    subgraph ROUTES["Next.js route handlers (app/api)"]
        RCHAT["POST /api/chat"]
        RCHATS["/api/chats · /api/chats/[id]"]
        RGRAPH["GET /api/graph"]
        RJOBS["GET /api/jobs?chatId"]
    end

    subgraph TURN["Turn composition (src/server)"]
        CH["chat-handler.ts — turn composition root<br/>(services: context · persist · spawn · extract)"]
    end

    subgraph TEACHER["Teacher agent (src/agent)"]
        TA["ToolLoopAgent<br/>system-prompt.md + prompts/*.md fragments<br/>+ graph-context block"]
        QT["quiz · ask_user_question<br/>(client-side tools, no execute)"]
        PT["plan_* tools (server-side,<br/>mutate the chat's tree)"]
        JT["research · demo · visual<br/>(server-executed, ack mid-turn)"]
    end

    subgraph SUBS["Background sub-agents (src/agent/subagents) + wake pipeline (src/server/jobs.ts)"]
        WAKE["wake queue — serialized per chat,<br/>turn-free + quiz-aware, server turn"]
        RES["researcher<br/>web_search + web_fetch"]
        DEMO["demo-builder<br/>write → headless-run → fix"]
        VIS["visual-maker<br/>write → render → look"]
        EXT["extraction pipeline<br/>extractor → auditor → merger<br/>(the sole graph writer)"]
    end

    subgraph STORE["File stores (src/store adapters → data/, never in git)"]
        CHATS[("data/chats/*.json<br/>UIMessages + plan tree")]
        JOBS[("data/jobs/*.json")]
        LEDG[("data/graph/*.jsonl<br/>nodes/edges/events/mappings/sessions")]
    end

    subgraph PIPE["Graph core (src/core) — merger + derivation"]
        MERGE["graph.ts mergeFindings<br/>deterministic apply: identity,<br/>event rebuild, dedupe, checkpoint"]
        GCTX["graph-context.ts<br/>hotspots + 1-hop when large"]
        SNAP["graphSnapshot (pure read)"]
    end

    TR & COMP & DOCK --> RCHAT
    GV --> RGRAPH
    ART --> RJOBS
    RCHATS --> CHATS
    TREE -.reads plan tool outputs.- TR

    RCHAT --> CH
    CH --> TA
    TA --> QT & PT & JT
    PT --> CHATS
    CH -->|"onFinish: saveMessages"| CHATS
    JT -->|"spawn(job, input) — fire-and-forget"| RES & DEMO & VIS
    CH -->|"onFinish: every N quiz answers<br/>(turn-service seam)"| EXT

    RES & DEMO & VIS --> JOBS
    RES & DEMO & VIS -->|"steer message via `deliver`"| WAKE
    WAKE -->|"server-side teacher turn<br/>(adopted client-side via turnInFlight)"| CH

    EXT -->|"audit-passed findings"| MERGE
    MERGE --> LEDG
    LEDG --> GCTX -->|"appended to instructions"| TA
    LEDG --> SNAP --> RGRAPH

Core invariant: the extraction pipeline (extractor → auditor → merger) is the sole graph writer. The LLM stages propose and verify (propositions, edges, probe attribution — each with verbatim evidence); the deterministic merger owns every ledger append and rebuilds anything the model could silently mis-copy (event kinds, timestamps, idempotency refs) from the transcript. The teacher stays a teacher; everything below the ledgers is derived.

Layout

app/                            Next.js App Router (UI + API route handlers)
components/                     session + home UI, question dock, tree panel,
                                graph view, artifact renderers (registry: artifacts/)
src/
  agent/
    system-prompt.md            base teacher behavior (the teach methodology)
    prompts/*.md                composed fragments (plan tree, jobs, phases,
                                demos, graph context) — one per workstream
    teacher-agent.ts            ToolLoopAgent bootstrap (model injectable for tests)
    tools.ts                    per-request tool registry
    tools/plan.ts               plan_init / add_node / … / set_current / mark_done
    tools/web.ts                web_search (Google CSE) + web_fetch (Readability)
    tools/jobs.ts               research / demo / visual tool surfaces
    subagents/                  researcher · demo-builder · visual-maker,
                                extraction pipeline (extractor/auditor + prompts),
                                runner (async protocol), headless Chrome helper
  server/chat-handler.ts       one chat turn: compose agent + services, stream
  server/turn-service.ts       turn services: teacher context · persist ·
                                extraction trigger (the v1 quiz-answer
                                threshold — one swappable seam)
  server/jobs.ts               the sub-agent wake pipeline: serialized per-chat
                                queue, steer delivery, server-side wake turns
  server/stores.ts              THE composition root — the one place adapters wire in
  server/{chats,graph,jobs}.ts  route entry points: chat list, pure-read
                                graph snapshot, job polling
  server/index.ts              the server barrel (the only server import app/ may use)
  core/contracts.ts             THE contracts file — types + protocol docs
  core/ports.ts                 storage ports — ChatStore · GraphStore · JobStore
                                (core code depends on these types only)
  core/                         question-core, plan-tree, phases, pending,
                                graph (merger/derivation), graph-context, math delims
                                — all pure: storage arrives as parameters
  store/                        fs adapters implementing the ports: chats ·
                                graph ledgers · jobs (plain fs, no framework)
scripts/bench-graph.ts          sigma/forceAtlas2 layout benchmark
data/                           chats/ · graph/ · jobs/ — runtime state, git-ignored

Key mechanisms

Client-side question tools. quiz and ask_user_question have no server execute: the tool call streams to the browser in input-available state, the learner answers in the dock, the client grades via question-core and returns the verdict with addToolOutput — which fires the SDK continuation so the teacher resumes mid-turn. A pending quiz part persisted in the chat record is exactly what makes resume mid-question free.

The formal plan tree. The teacher authors the session's dependency tree with schema-validated tools (plan_init, plan_add_node, … plan_set_current, plan_mark_done). The tree persists in the chat record; the tree panel renders from the latest plan tool output in the transcript. Quiz calls can tag a nodeId, so outcomes feed the graph per-node.

The async sub-agent protocol (ack-and-steer). research / demo / visual are server-executed tools: the call resolves immediately mid-turn with a fire-and-forget ack and the teacher keeps teaching; a background runner works in parallel (the job store tracks a live activity line; the client polls /api/jobs). On completion the runner builds a steer message — a user-role UIMessage whose text part is the model-facing result (data parts are dropped by provider conversion) and whose data-subagent-artifact part carries the full result for the client — and the wake pipeline (server/jobs.ts) delivers it: a per-chat serialized queue waits for the turn to end and no question to be pending, then fires a server-side teacher turn that weaves the result into the lesson. The client adopts the wake turn through the same turnInFlight machinery as returning mid-stream (busy shimmer, composer gate) and renders steers as artifacts — never a bubble. Sub-agents verify, never one-shot: the demo-builder headlessly executes its HTML in system Chrome before delivery; the visual-maker renders each SVG and looks at it (needs a vision-capable model — MODELS.visualMaker in config.ts).

The knowledge graph (the two channels). The write channel: the extraction pipeline — extractor (LLM: propositions, edges, probe attribution, each with verbatim evidence) → auditor (LLM: drops anything unbacked) → merger (mergeFindings, deterministic: exact/alias identity resolution, event rebuild from the transcript's tool parts, dedupe, checkpoint). Triggered once a chat accumulates EXTRACTION.everyNQuizAnswers (default 3, config.ts) new answered quizzes — a v1 placeholder trigger isolated in one seam (turn-service.ts), swappable without touching the pipeline. Failure checkpoints nothing; the next trigger re-runs. The read channel: the home view derives node states from the ledgers (GET /api/graph is a pure read — forceAtlas2 run-then-freeze, no idle simulation), and at session start the derived graph is rendered into a compact markdown block appended to the teacher's instructions (buildGraphContext) — whole graph while small, hotspots + 1-hop neighborhood past 150 nodes.

Testing

All deterministic, no LLM (models are injected as MockLanguageModelV4 where an agent loop is under test):

1. Pure core units — plan-tree ops, question grading, phases, graph merger + trigger helper, graph-context selection, math delimiters.
2. Server flows — chat-handler driven directly with a scripted mock model: persistence, job dispatch, steer delivery, the wake pipeline (409 retry, quiz deferral, idempotency).
3. Sub-agent loops — researcher/demo-builder/visual-maker with mock models + fake headless API; the extraction pipeline end-to-end with mock models (extractor → auditor → in-memory ledgers).
4. Render tests — transcript, cards, dock, composer, tree panel via React server-side rendering.

Lineage

The teaching methodology comes from the Obsidian-based POC (~/Desktop/Obsidian/vault/π/learn; its knowledge graph design.md is still canonical for the epistemic model). The harness was originally built on the pi agent SDK, then pivoted to standalone Next.js + Vercel AI SDK (see the amendments table in docs/vision.md).

Data

Nothing under data/ is ever tracked by git. Chats, graph ledgers, and jobs are plain files (JSON/JSONL) — inspect them freely; they are the canonical records.