Understanding levels: closed five-word scale, schema-enforced

The extractor's per-session understanding notes were free-form strings;
the model freely appended em-dash justifications ('shaky — the learner
could state the headline distinction but calls the machinery behind it
foggy'), which overflowed the detail card's session rows and squeezed
the session names.

The note becomes a CLOSED named-word scale — solid / shaky / accepted /
resisted / not yet engaged (DECIDED 2026-09-08, no numbers) — enforced
by the zod enum in propositionSchema: a wrong-shaped note is rejected
at parse time and the agent retries. Justifications are dropped, not
stored (user decision: not necessary). Extractor and auditor
instructions updated to demand exactly one word.

Renderer: the session row keeps the session NAME as the main thing
(flex-1, truncates) and the level chip is bounded (max-w-28,
whitespace-nowrap, truncate) — overflow impossible even for legacy
prose notes in old ledgers.

Vision doc: §1 extractor bullet + event-ledger line updated; state
model line notes the five-class derivation was removed; amendments
table row added.
This commit is contained in:
amosblomqvist
2026-09-09 02:06:32 +03:00
parent 5f0a1ca30b
commit e944afebfc
4 changed files with 35 additions and 20 deletions
+4 -3
View File
@@ -43,15 +43,15 @@ Every node records every **(chat, message) point** where it came up; every edge
Three stages, one pipeline — the sole graph writer:
- **Extractor (LLM) — session-only scope.** Sees ONLY the session transcript (tree-blind, with message ids). Reports freely, with verbatim evidence: propositions from anywhere in the session (one reasoning step each; prose counts as much as anything the teacher formalized), edges as the transcript articulates them, message-granular provenance, retroactive quiz attribution, and per proposition an **understanding note** — how strongly the learner accepted/understood it in this session, a small named-word scale, no numbers. The notes are data for the future state model; their treatment is TBD.
- **Extractor (LLM) — session-only scope.** Sees ONLY the session transcript (tree-blind, with message ids). Reports freely, with verbatim evidence: propositions from anywhere in the session (one reasoning step each; prose counts as much as anything the teacher formalized), edges as the transcript articulates them, message-granular provenance, retroactive quiz attribution, and per proposition an **understanding level** — how strongly the learner accepted/understood it in this session, a CLOSED named-word scale (`solid` / `shaky` / `accepted` / `resisted` / `not yet engaged`; 2026-09-08: schema-enforced — exactly one word, no numbers, no justifications, which are dropped rather than stored). The levels are data for the future state model; their treatment is TBD.
- **Auditor (LLM) — the global-graph scope.** Sees the extractor's output AND the existing graph. Two jobs: the faithfulness net (every candidate verified against verbatim evidence — unchanged), and the **identity judgment**: link each proposition to an existing node (recording the session's phrasing as an alias) or create it new. Conservative: duplicates over wrong merges.
- **Merger (deterministic).** Append-only ledger writes; quiz event kinds/refs rebuilt mechanically from quiz tool parts. Every judgment is an LLM's; everything mechanical stays code.
**Event ledger (amended):** quiz outcomes are rebuilt deterministically from quiz tool parts (correct / miss / don't-know, plus the learner's note). **There is no `taught` event** — no marking-things-done. The extractor's per-proposition understanding notes join the ledger as session-scoped observations. Identity is no longer exact-string matching — it is the auditor's judgment, with aliases as the phrasing record.
**Event ledger (amended):** quiz outcomes are rebuilt deterministically from quiz tool parts (correct / miss / don't-know, plus the learner's note). **There is no `taught` event** — no marking-things-done. The extractor's per-proposition understanding levels join the ledger as session-scoped observations (one closed-scale word per node per session). Identity is no longer exact-string matching — it is the auditor's judgment, with aliases as the phrasing record.
### Understanding is derived (amended 2026-09-02)
Known vs. understood, event ledger as source of truth, derived state recomputed on read — carried over from the design doc in spirit. The state model itself is now an explicit design discussion: **node states should be session-independent and computed from the depth and strength of dependencies** (the `follows`/`supports` topology) plus quiz events and understanding notes. That redesign is pending ideation; the current five-class derivation (taught / known / missed / don't-know / candidate) is transitional. The anchored calibration rubric stays parked until real data.
Known vs. understood, event ledger as source of truth, derived state recomputed on read — carried over from the design doc in spirit. The state model itself is now an explicit design discussion: **node states should be session-independent and computed from the depth and strength of dependencies** (the `follows`/`supports` topology) plus quiz events and understanding notes. That redesign is pending ideation; the transitional five-class derivation (taught / known / missed / don't-know / candidate) was **removed** (2026-09-08) — the graph view renders state-neutral nodes until the state model is designed. The anchored calibration rubric stays parked until real data.
### Rendering (DECIDED — research report 2026-08-22)
@@ -224,6 +224,7 @@ The teacher's context window and attention are precious — it has the lesson to
| Misconception nodes as first-class structure | No node `kind`: every node is a true proposition; misconceptions enter as true corrections, wrongness lives in the event ledger |
| `requires` edges | `follows` / `supports` — derivability vs. grounding; strength derived at read time, never stored |
| `taught` events (marking things done) | Dropped; the extractor records per-proposition understanding notes (named-word scale) as session-scoped observations instead |
| Understanding scale standardized (2026-09-08) | The note becomes a closed five-word scale (`solid`/`shaky`/`accepted`/`resisted`/`not yet engaged`), enforced by the extraction schema — no numbers, no prose; agent justifications are dropped |
| Chat-granular node/edge provenance | Message-granular (chat, message) provenance on nodes and edges — the exact derivation points |
| `name` + `statement` on nodes | `statement` (minimal proposition, the rendering label) + `description` (required elaboration) + `aliases` (verbatim phrasings) |
| `statement` + `description` (2026-09-02) | `name` (very short label, the rendering handle) + `statement` (full precision, identity anchor, detail card); `description` dropped — full-sentence canvas labels cluttered the graph (2026-09-08, seeing real extracted data) |