learnharness/README.md

# learning-harness

A standalone learning harness: a **teacher agent** (Next.js + Vercel AI SDK, models via OpenRouter), graded quiz/ask interactions, a formal **session plan tree** the teacher authors and edits live, and a persistent cross-session **knowledge graph** of what the learner knows — the graph is the app's home page.

Not a chat app with a graph bolted on: sessions are probe→teach lessons, the plan tree is a structured, agent-editable object, quiz outcomes reach the graph through the extraction pipeline as ledger events, and the graph feeds back as the teacher's context so probing compounds across sessions.

- Living design doc (all decisions): [`docs/vision.md`](docs/vision.md)
- Build plan + wave/gate results: [`docs/plan.md`](docs/plan.md)
- [`docs/architecture.md`](docs/architecture.md) is the **canonical architecture spec** — layers, ports, the import law, the single-writer rule, the two graph channels.

## Commands

```bash
npm install          # once

npm run dev          # http://localhost:3000
npm run build        # production build
npm start            # serve the production build

npm test             # all tests (node:test via tsx; deterministic, no LLM)
npm run lint         # eslint — includes the import law (architecture §1)
npm run typecheck    # tsc --noEmit

npx tsx scripts/bench-graph.ts   # force-layout benchmark (synthetic graphs)
```

## Environment (`.env.local`)

Secrets and machine-specific paths only. Model choice and thinking budgets are product tuning, not environment — they live in **`config.ts` → `MODELS`** (teacher, researcher, demoBuilder, visualMaker, extractor, auditor; each with an OpenRouter model id + reasoning `effort`). The extraction trigger threshold is there too (`EXTRACTION.everyNQuizAnswers`).

| Variable | Purpose |
|---|---|
| `OPENROUTER_API_KEY` | required — all LLM calls go through OpenRouter |
| `GOOGLE_SEARCH_API_KEY` + `GOOGLE_CSE_ID` | researcher's `web_search` (Google Custom Search) |
| `CHROME_PATH` | optional explicit Chrome path for headless verification (puppeteer-core uses system Chrome; nothing is downloaded) |

## Architecture

Five layers, dependency arrows pointing down only, the import law machine-enforced as four ESLint zones (shell → barrels only; core and agent never import below themselves). The full spec: [`docs/architecture.md`](docs/architecture.md).

```mermaid
graph TB
    subgraph SESSION_UI["Browser · session page (app/chat/[id])"]
        TR["Transcript — streamdown markdown,<br/>KaTeX \\(..\\)/\\[..\\], mermaid,<br/>probe→teach phase divider"]
        DOCK["QuestionDock / AnsweredCard<br/>quiz graded client-side<br/>(question-core)"]
        TREE["TreePanel — plan tree,<br/>topo order, done/current/upcoming"]
        ART["Artifact cards — research brief,<br/>demo in sandboxed iframe, SVG"]
        COMP["Composer — gated while a<br/>question or job is pending"]
    end

    subgraph HOME_UI["Browser · home page (app/)"]
        GV["GraphView — sigma.js + forceAtlas2,<br/>run-then-freeze, node state colors"]
    end

    subgraph ROUTES["Next.js route handlers (app/api)"]
        RCHAT["POST /api/chat"]
        RCHATS["/api/chats · /api/chats/[id]"]
        RGRAPH["GET /api/graph"]
        RJOBS["GET /api/jobs?chatId"]
    end

    subgraph TURN["Turn composition (src/server)"]
        CH["chat-handler.ts — turn composition root<br/>(services: context · persist · spawn · extract)"]
    end

    subgraph TEACHER["Teacher agent (src/agent)"]
        TA["ToolLoopAgent<br/>system-prompt.md + prompts/*.md fragments<br/>+ graph-context block"]
        QT["quiz · ask_user_question<br/>(client-side tools, no execute)"]
        PT["plan_* tools (server-side,<br/>mutate the chat's tree)"]
        JT["research · demo · visual<br/>(client-side tools, job protocol)"]
    end

    subgraph SUBS["Background sub-agents (src/agent/subagents)"]
        RES["researcher<br/>web_search + web_fetch"]
        DEMO["demo-builder<br/>write → headless-run → fix"]
        VIS["visual-maker<br/>write → render → look"]
        EXT["extraction pipeline<br/>extractor → auditor → merger<br/>(the sole graph writer)"]
    end

    subgraph STORE["File stores (src/store adapters → data/, never in git)"]
        CHATS[("data/chats/*.json<br/>UIMessages + plan tree")]
        JOBS[("data/jobs/*.json")]
        LEDG[("data/graph/*.jsonl<br/>nodes/edges/events/mappings/sessions")]
    end

    subgraph PIPE["Graph core (src/core) — merger + derivation"]
        MERGE["graph.ts mergeFindings<br/>deterministic apply: identity,<br/>event rebuild, dedupe, checkpoint"]
        GCTX["graph-context.ts<br/>hotspots + 1-hop when large"]
        SNAP["graphSnapshot (pure read)"]
    end

    TR & COMP & DOCK --> RCHAT
    GV --> RGRAPH
    ART --> RJOBS
    RCHATS --> CHATS
    TREE -.reads plan tool outputs.- TR

    RCHAT --> CH
    CH --> TA
    TA --> QT & PT & JT
    PT --> CHATS
    CH -->|"onFinish: saveMessages"| CHATS
    CH -->|"onFinish: spawnPendingJobs"| RES & DEMO & VIS
    CH -->|"onFinish: every N quiz answers<br/>(turn-service seam)"| EXT

    RES & DEMO & VIS --> JOBS
    RES & DEMO & VIS -->|"rewrite tool part<br/>→ output-available"| CHATS

    EXT -->|"audit-passed findings"| MERGE
    MERGE --> LEDG
    LEDG --> GCTX -->|"appended to instructions"| TA
    LEDG --> SNAP --> RGRAPH
```

**Core invariant:** the extraction pipeline (extractor → auditor → merger) is the sole graph writer. The LLM stages propose and verify (propositions, edges, probe attribution — each with verbatim evidence); the deterministic merger owns every ledger append and rebuilds anything the model could silently mis-copy (event kinds, timestamps, idempotency refs) from the transcript. The teacher stays a teacher; everything below the ledgers is derived.

## Layout

```
app/                            Next.js App Router (UI + API route handlers)
components/                     session + home UI, question dock, tree panel,
                                graph view, artifact renderers (registry: artifacts/)
src/
  agent/
    system-prompt.md            base teacher behavior (the teach methodology)
    prompts/*.md                composed fragments (plan tree, jobs, phases,
                                demos, graph context) — one per workstream
    teacher-agent.ts            ToolLoopAgent bootstrap (model injectable for tests)
    tools.ts                    per-request tool registry
    tools/plan.ts               plan_init / add_node / … / set_current / mark_done
    tools/web.ts                web_search (Google CSE) + web_fetch (Readability)
    tools/jobs.ts               research / demo / visual tool surfaces
    subagents/                  researcher · demo-builder · visual-maker,
                                extraction pipeline (extractor/auditor + prompts),
                                runner (async protocol), headless Chrome helper
  server/chat-handler.ts       one chat turn: compose agent + services, stream
  server/turn-service.ts       turn services: teacher context · persist ·
                                spawn jobs · extraction trigger (the v1
                                quiz-answer threshold — one swappable seam)
  server/stores.ts              THE composition root — the one place adapters wire in
  server/{chats,graph,jobs}.ts  route entry points: chat list, pure-read
                                graph snapshot, job polling
  server/index.ts              the server barrel (the only server import app/ may use)
  core/contracts.ts             THE contracts file — types + protocol docs
  core/ports.ts                 storage ports — ChatStore · GraphStore · JobStore
                                (core code depends on these types only)
  core/                         question-core, plan-tree, phases, pending,
                                graph (merger/derivation), graph-context, math delims
                                — all pure: storage arrives as parameters
  store/                        fs adapters implementing the ports: chats ·
                                graph ledgers · jobs (plain fs, no framework)
scripts/bench-graph.ts          sigma/forceAtlas2 layout benchmark
data/                           chats/ · graph/ · jobs/ — runtime state, git-ignored
```

## Key mechanisms

**Client-side question tools.** `quiz` and `ask_user_question` have no server `execute`: the tool call streams to the browser in `input-available` state, the learner answers in the dock, the client grades via `question-core` and returns the verdict with `addToolOutput` — which fires the SDK continuation so the teacher resumes mid-turn. A pending quiz part persisted in the chat record is exactly what makes **resume mid-question free**.

**The formal plan tree.** The teacher authors the session's dependency tree with schema-validated tools (`plan_init`, `plan_add_node`, … `plan_set_current`, `plan_mark_done`). The tree persists in the chat record; the tree panel renders from the latest plan tool output in the transcript. Quiz calls can tag a `nodeId`, so outcomes feed the graph per-node.

**The async sub-agent protocol.** `research` / `demo` / `visual` are also client-side tools: the server turn ends at the call, `onFinish` spawns a background runner (fire-and-forget), the job store tracks a live activity line, the client polls `/api/jobs` while parts are pending, and on completion the runner rewrites the tool part in the chat record to `output-available` — the client re-syncs, the continuation fires, the teacher receives the result as a normal tool output. Sub-agents **verify, never one-shot**: the demo-builder headlessly executes its HTML in system Chrome before delivery; the visual-maker renders each SVG and looks at it (needs a vision-capable model — `MODELS.visualMaker` in config.ts).

**The knowledge graph (the two channels).** The write channel: the extraction pipeline — extractor (LLM: propositions, edges, probe attribution, each with verbatim evidence) → auditor (LLM: drops anything unbacked) → merger (`mergeFindings`, deterministic: exact/alias identity resolution, event rebuild from the transcript's tool parts, dedupe, checkpoint). Triggered once a chat accumulates `EXTRACTION.everyNQuizAnswers` (default 3, config.ts) new answered quizzes — a v1 placeholder trigger isolated in one seam (`turn-service.ts`), swappable without touching the pipeline. Failure checkpoints nothing; the next trigger re-runs. The read channel: the home view derives node states from the ledgers (`GET /api/graph` is a pure read — forceAtlas2 run-then-freeze, no idle simulation), and at session start the derived graph is rendered into a compact markdown block appended to the teacher's instructions (`buildGraphContext`) — whole graph while small, hotspots + 1-hop neighborhood past 150 nodes.

## Testing

All deterministic, no LLM (models are injected as `MockLanguageModelV4` where an agent loop is under test):

1. **Pure core units** — plan-tree ops, question grading, phases, graph merger + trigger helper, graph-context selection, math delimiters.
2. **Server flows** — `chat-handler` driven directly with a scripted mock model: persistence, job spawning, part rewriting, continuation.
3. **Sub-agent loops** — researcher/demo-builder/visual-maker with mock models + fake headless API; the extraction pipeline end-to-end with mock models (extractor → auditor → in-memory ledgers).
4. **Render tests** — transcript, cards, dock, composer, tree panel via React server-side rendering.

## Lineage

The teaching methodology comes from the Obsidian-based POC (`~/Desktop/Obsidian/vault/π/learn`; its `knowledge graph design.md` is still canonical for the epistemic model). The harness was originally built on the pi agent SDK, then pivoted to standalone Next.js + Vercel AI SDK (see the amendments table in `docs/vision.md`).

## Data

Nothing under `data/` is ever tracked by git. Chats, graph ledgers, and jobs are plain files (JSON/JSONL) — inspect them freely; they are the canonical records.