amosblomqvist
e6f52d3ad1
Vision-assessment records the 2026-10-06 thinking round: propositions as the unit, forecast semantics (mastery = P(correct unaided retrieval)), the Beta-with-decay estimator (BKT rejected: binary-only, guess-less, never forgets, unidentifiable at our volume; the weighted average completed with prior + recency + uncertainty), the e-table over outcome x rung x missClass, uniform 1/k credit (optional missed-field upgrade unruled), no new outcome enum values, PRACTICE-ONLY perimeter (ruled — quiz nodes are session-scoped plan objects, no honest mapping), mastery computed-on-read over the already durable problem/conclude calls (no stored ledger, graph untouched). Release-0.3 captures the owner's polish brain-dump (graph rendering toggle separate from extraction + open home-view question, question keybinds/VIM, session naming, markdown + graph exports, graph defaults, compaction dialog Esc law, keyboard audit) and the schema-lock position with the persistent surface inventory and pre-release gap list (schemaVersion stamps, pi-ai pins).
82 lines
8.1 KiB
Markdown
82 lines
8.1 KiB
Markdown
# Vision — Assessment Levels (Mastery)
|
||
|
||
Started 2026-10-06, thinking round (owner + assistant). **Deliberately unscheduled**: first release (0.3.0) comes first; this is the post-release arc. It resolves the "aggregation TBD" left open in vision-practice.md's assessment-record model (2026-09-27 rulings) — and inherits that round's law: *understanding is inferred, never encoded*.
|
||
|
||
The surface for all of this is the future **data view** (still undesigned), precisely because these are computed estimates, not truths. Nothing here writes to the graph (the 2026-09-28 cleansing ruling stands: propositions + edges only, no understanding fields).
|
||
|
||
## The unit: propositions (ruled 2026-10-06)
|
||
|
||
Mastery is assessed **per proposition**. Not per chain, not per session, not per learner-skill.
|
||
|
||
- The infrastructure already speaks propositions: practice problems tag target proposition ids, validated whole-graph at pose time (the round-9 mastery-trace contract, ids-first).
|
||
- The worry "knows all propositions ≠ knows the chain" is real but is an **instrument concern, not a model concern**: composition-mode problems *are* chain assessments (you cannot pass one without executing the inference), and their evidence already lands on member propositions. A chain whose propositions are all green but which never appears in a composition item is a problem-generation coverage gap.
|
||
- **Chain readiness is derived on read, never stored**: min (or product) over premise+conclusion mastery. For selection and display only. No second ledger.
|
||
|
||
## What the number means: forecast semantics (ruled 2026-10-06)
|
||
|
||
> **Mastery(p) = our current estimate of the probability that the learner can correctly exercise proposition p in a fresh, unaided retrieval.**
|
||
|
||
We do not claim to observe understanding; we forecast performance. This makes the number honest (it is a probability, presented with its uncertainty, in the data view — never a verdict), naturally 0..1, and directly consumable: the doctrine's ~85%-success-per-retrieval target becomes selection arithmetic (pick propositions whose forecast sits in the ~0.85 band, decaying ones, or agent-flagged `retest`).
|
||
|
||
It also gives the parked Brier idea its precise home: each new observation *scores* the pre-update forecast — (forecast − outcome)². "Are our numbers honest?" becomes measurable; a calibration panel is a natural data-view neighbor.
|
||
|
||
Terminology: **mastery** is the display word. The precise phrase is *estimated recall reliability*; the definition carries the precision.
|
||
|
||
## The estimator: a Beta belief with decay (agreed direction 2026-10-06)
|
||
|
||
The owner's two candidates — BKT per proposition, or a weighted average of outcomes — resolve as: **BKT is the wrong tool; the weighted average is the right skeleton, completed with a prior, time decay, and uncertainty.** That completion makes it a Bayesian model anyway.
|
||
|
||
Why BKT fails here:
|
||
- **Binary observations.** Our items are graded by the agent's rubric (outcome × rung × missClass — the richest signal we have); binarizing throws most of it away.
|
||
- **The guess parameter is fictitious** — the model sees the learner's thinking. Slip survives only as careless/execution misses, which missClass already *names* rather than smooths as noise.
|
||
- **Identifiability.** BKT wants p-init/p-learn/p-slip per skill; we will have ~1–10 observations per proposition. Per-proposition parameters cannot be fit; global parameters degenerate BKT into an EMA with ceremony.
|
||
- **BKT never forgets** (p-known is monotone). Mastery must decay with disuse.
|
||
|
||
Three layers:
|
||
|
||
**1 — the evidence scalar.** Each *concluded* problem becomes one number e ∈ [0,1]: "fraction of unaided mastery demonstrated." The agent's rubric, numerized — a tunable table, not physics:
|
||
|
||
| outcome | e (approx.) |
|
||
|---|---|
|
||
| solved, no hint | ~0.95 (never 1.0 — one clean solve is not certainty) |
|
||
| solvedWithHints | ~0.85 → ~0.3 as rung climbs R1→R5 |
|
||
| movedOn | ~0.25 |
|
||
| surrendered | ~0.1 |
|
||
| dismissed | **excluded entirely** — nothing was demonstrated, in either direction |
|
||
|
||
missClass modulates within a cell: *careless*/*execution* misses are failures not about the proposition (discount the negative evidence); *cue* and *structural* are the real negative signal; *discrimination* sits between.
|
||
|
||
**2 — the belief.** Per proposition a Beta(α, β); each observation adds fractional success/failure counts weighted by:
|
||
- **target share 1/k** — uniform distribution across the problem's targets (correct, not just convenient: a k-target composition item genuinely spreads its evidence; counting it k full observations would inflate confidence k×);
|
||
- **recency** — exp(−Δt/τ), τ in weeks (forgetting; old evidence fades toward the prior);
|
||
- later, if the perimeter ever widens: an instrument weight.
|
||
|
||
Posterior mean α/(α+β) is the displayed mastery; the spread is the honest error band (`0.62 ± wide · ~3 effective observations · last practiced 6d ago` — two data points visibly ≠ twenty). New propositions start at a weak prior (order Beta(2,5) ≈ 0.29 — propositions come from the learner's own material; low concentration so 2–3 observations move it decisively). Every knob (the e-table, τ, the prior) is a number in one place, tunable from field data without touching records or prompts.
|
||
|
||
**3 — derived, never stored.** Chain readiness from member mastery (above).
|
||
|
||
## Credit assignment (default ruled; the upgrade open)
|
||
|
||
Default: uniform 1/k. The finer detail the owner wanted has a cheap shape, **unruled**: an optional `missed: [propositionIds]` field on the conclude call — the agent names which target(s) the miss was about, validated against the pose's own targets (round-9 discipline). Negative evidence then concentrates where it belongs; the other targets take fuller positive credit; "one third of a failure" becomes "the failure has an address." Rejected: per-submission grading (the conclude is the graded unit; grading every thinking message over-weights long problems and doubles agent work) and string-mining the verdict prose (drift).
|
||
|
||
## Outcome levels: no new enum values (ruled 2026-10-06)
|
||
|
||
The five-value outcome enum stays. outcome × rung × missClass already spans the space; felt missing resolution ("solved with a nudge" vs "solved after a walk-through") is carried by the rung. Resolution belongs in the scoring table (tunable in an afternoon); enum growth taxes every future conclude call — weak models, weak schemas (the GLM incident's lesson).
|
||
|
||
## The perimeter: practice sessions only (ruled 2026-10-06)
|
||
|
||
**Practice sessions are the sole source of assessment data.** Teaching sessions never contribute, even though they contain proposition-shaped evidence (quizzes, explain free-recall, predict/check). Reasons: clear boundaries and a single truth per construct; teaching is an entirely different purpose (active, just-taught material vs cold retrieval); and concretely, the nodes quiz questions reference are session-scoped plan objects, *not* graph propositions — no honest mapping exists (the 2026-09-27 guardrail). The evidence record still carries a `source` field internally, but only practice writes it.
|
||
|
||
## Storage: nothing new, on principle (ruled 2026-10-06)
|
||
|
||
The raw evidence is already durable: problem calls carry targets + mode; conclude calls carry outcome + rung + missClass; both live in append-only chat records under honest activation. **The assessment trace is a join over what exists; mastery is a computed view** (like graphSnapshot) — cached only if it ever gets slow. If the practice agent later needs mastery as a selection input, it reads the computed view at session start. No stored ledger, no drift with the records, graph untouched.
|
||
|
||
## Open questions (unruled)
|
||
|
||
- The prior's center: Beta(2,5)-ish, or higher for propositions from the learner's own documents?
|
||
- The `missed` field on conclude: adopt or not?
|
||
- τ (the decay half-life): weeks — set from field data once it exists.
|
||
- The e-table's exact cells: calibrate after the first real corpus of concluded problems.
|
||
- Whether mastery ever feeds the practice agent's *live* problem selection during a session (the arc's second half — adaptive selection) or only at session start.
|
||
- The data view's presentation of all of this (its own design round).
|