Understanding levels: closed five-word scale, schema-enforced
The extractor's per-session understanding notes were free-form strings;
the model freely appended em-dash justifications ('shaky — the learner
could state the headline distinction but calls the machinery behind it
foggy'), which overflowed the detail card's session rows and squeezed
the session names.
The note becomes a CLOSED named-word scale — solid / shaky / accepted /
resisted / not yet engaged (DECIDED 2026-09-08, no numbers) — enforced
by the zod enum in propositionSchema: a wrong-shaped note is rejected
at parse time and the agent retries. Justifications are dropped, not
stored (user decision: not necessary). Extractor and auditor
instructions updated to demand exactly one word.
Renderer: the session row keeps the session NAME as the main thing
(flex-1, truncates) and the level chip is bounded (max-w-28,
whitespace-nowrap, truncate) — overflow impossible even for legacy
prose notes in old ledgers.
Vision doc: §1 extractor bullet + event-ledger line updated; state
model line notes the five-class derivation was removed; amendments
table row added.
This commit is contained in:
+4
-3
@@ -43,15 +43,15 @@ Every node records every **(chat, message) point** where it came up; every edge
|
|||||||
|
|
||||||
Three stages, one pipeline — the sole graph writer:
|
Three stages, one pipeline — the sole graph writer:
|
||||||
|
|
||||||
- **Extractor (LLM) — session-only scope.** Sees ONLY the session transcript (tree-blind, with message ids). Reports freely, with verbatim evidence: propositions from anywhere in the session (one reasoning step each; prose counts as much as anything the teacher formalized), edges as the transcript articulates them, message-granular provenance, retroactive quiz attribution, and per proposition an **understanding note** — how strongly the learner accepted/understood it in this session, a small named-word scale, no numbers. The notes are data for the future state model; their treatment is TBD.
|
- **Extractor (LLM) — session-only scope.** Sees ONLY the session transcript (tree-blind, with message ids). Reports freely, with verbatim evidence: propositions from anywhere in the session (one reasoning step each; prose counts as much as anything the teacher formalized), edges as the transcript articulates them, message-granular provenance, retroactive quiz attribution, and per proposition an **understanding level** — how strongly the learner accepted/understood it in this session, a CLOSED named-word scale (`solid` / `shaky` / `accepted` / `resisted` / `not yet engaged`; 2026-09-08: schema-enforced — exactly one word, no numbers, no justifications, which are dropped rather than stored). The levels are data for the future state model; their treatment is TBD.
|
||||||
- **Auditor (LLM) — the global-graph scope.** Sees the extractor's output AND the existing graph. Two jobs: the faithfulness net (every candidate verified against verbatim evidence — unchanged), and the **identity judgment**: link each proposition to an existing node (recording the session's phrasing as an alias) or create it new. Conservative: duplicates over wrong merges.
|
- **Auditor (LLM) — the global-graph scope.** Sees the extractor's output AND the existing graph. Two jobs: the faithfulness net (every candidate verified against verbatim evidence — unchanged), and the **identity judgment**: link each proposition to an existing node (recording the session's phrasing as an alias) or create it new. Conservative: duplicates over wrong merges.
|
||||||
- **Merger (deterministic).** Append-only ledger writes; quiz event kinds/refs rebuilt mechanically from quiz tool parts. Every judgment is an LLM's; everything mechanical stays code.
|
- **Merger (deterministic).** Append-only ledger writes; quiz event kinds/refs rebuilt mechanically from quiz tool parts. Every judgment is an LLM's; everything mechanical stays code.
|
||||||
|
|
||||||
**Event ledger (amended):** quiz outcomes are rebuilt deterministically from quiz tool parts (correct / miss / don't-know, plus the learner's note). **There is no `taught` event** — no marking-things-done. The extractor's per-proposition understanding notes join the ledger as session-scoped observations. Identity is no longer exact-string matching — it is the auditor's judgment, with aliases as the phrasing record.
|
**Event ledger (amended):** quiz outcomes are rebuilt deterministically from quiz tool parts (correct / miss / don't-know, plus the learner's note). **There is no `taught` event** — no marking-things-done. The extractor's per-proposition understanding levels join the ledger as session-scoped observations (one closed-scale word per node per session). Identity is no longer exact-string matching — it is the auditor's judgment, with aliases as the phrasing record.
|
||||||
|
|
||||||
### Understanding is derived (amended 2026-09-02)
|
### Understanding is derived (amended 2026-09-02)
|
||||||
|
|
||||||
Known vs. understood, event ledger as source of truth, derived state recomputed on read — carried over from the design doc in spirit. The state model itself is now an explicit design discussion: **node states should be session-independent and computed from the depth and strength of dependencies** (the `follows`/`supports` topology) plus quiz events and understanding notes. That redesign is pending ideation; the current five-class derivation (taught / known / missed / don't-know / candidate) is transitional. The anchored calibration rubric stays parked until real data.
|
Known vs. understood, event ledger as source of truth, derived state recomputed on read — carried over from the design doc in spirit. The state model itself is now an explicit design discussion: **node states should be session-independent and computed from the depth and strength of dependencies** (the `follows`/`supports` topology) plus quiz events and understanding notes. That redesign is pending ideation; the transitional five-class derivation (taught / known / missed / don't-know / candidate) was **removed** (2026-09-08) — the graph view renders state-neutral nodes until the state model is designed. The anchored calibration rubric stays parked until real data.
|
||||||
|
|
||||||
### Rendering (DECIDED — research report 2026-08-22)
|
### Rendering (DECIDED — research report 2026-08-22)
|
||||||
|
|
||||||
@@ -224,6 +224,7 @@ The teacher's context window and attention are precious — it has the lesson to
|
|||||||
| Misconception nodes as first-class structure | No node `kind`: every node is a true proposition; misconceptions enter as true corrections, wrongness lives in the event ledger |
|
| Misconception nodes as first-class structure | No node `kind`: every node is a true proposition; misconceptions enter as true corrections, wrongness lives in the event ledger |
|
||||||
| `requires` edges | `follows` / `supports` — derivability vs. grounding; strength derived at read time, never stored |
|
| `requires` edges | `follows` / `supports` — derivability vs. grounding; strength derived at read time, never stored |
|
||||||
| `taught` events (marking things done) | Dropped; the extractor records per-proposition understanding notes (named-word scale) as session-scoped observations instead |
|
| `taught` events (marking things done) | Dropped; the extractor records per-proposition understanding notes (named-word scale) as session-scoped observations instead |
|
||||||
|
| Understanding scale standardized (2026-09-08) | The note becomes a closed five-word scale (`solid`/`shaky`/`accepted`/`resisted`/`not yet engaged`), enforced by the extraction schema — no numbers, no prose; agent justifications are dropped |
|
||||||
| Chat-granular node/edge provenance | Message-granular (chat, message) provenance on nodes and edges — the exact derivation points |
|
| Chat-granular node/edge provenance | Message-granular (chat, message) provenance on nodes and edges — the exact derivation points |
|
||||||
| `name` + `statement` on nodes | `statement` (minimal proposition, the rendering label) + `description` (required elaboration) + `aliases` (verbatim phrasings) |
|
| `name` + `statement` on nodes | `statement` (minimal proposition, the rendering label) + `description` (required elaboration) + `aliases` (verbatim phrasings) |
|
||||||
| `statement` + `description` (2026-09-02) | `name` (very short label, the rendering handle) + `statement` (full precision, identity anchor, detail card); `description` dropped — full-sentence canvas labels cluttered the graph (2026-09-08, seeing real extracted data) |
|
| `statement` + `description` (2026-09-02) | `name` (very short label, the rendering handle) + `statement` (full precision, identity anchor, detail card); `description` dropped — full-sentence canvas labels cluttered the graph (2026-09-08, seeing real extracted data) |
|
||||||
|
|||||||
@@ -118,11 +118,12 @@ const propositionSchema = z.object({
|
|||||||
messages: messagesField,
|
messages: messagesField,
|
||||||
evidence,
|
evidence,
|
||||||
understanding: z
|
understanding: z
|
||||||
.string()
|
.enum(["solid", "shaky", "accepted", "resisted", "not yet engaged"])
|
||||||
.optional()
|
.optional()
|
||||||
.describe(
|
.describe(
|
||||||
'One note about THIS session\'s evidence of the learner\'s grasp of this proposition, on a named-word scale ' +
|
"ONE word — the learner's grasp of this proposition in THIS session, from the fixed scale " +
|
||||||
'("solid", "shaky", "accepted", "resisted", "not yet engaged") — no numbers, no predictions.',
|
'"solid" | "shaky" | "accepted" | "resisted" | "not yet engaged". Nothing else: no numbers, ' +
|
||||||
|
"no justification, no explanation.",
|
||||||
),
|
),
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -180,8 +181,9 @@ quiz outcomes carry that the learner believed the wrong version.
|
|||||||
- One reasoning step per proposition: never a topic blob, never a sub-step.
|
- One reasoning step per proposition: never a topic blob, never a sub-step.
|
||||||
- "messages": the [id] tags of every message where the proposition came up — provenance, report it.
|
- "messages": the [id] tags of every message where the proposition came up — provenance, report it.
|
||||||
- "aliases": verbatim phrasings this session used for the same proposition.
|
- "aliases": verbatim phrasings this session used for the same proposition.
|
||||||
- "understanding": ONE named-word-scale note on the evidence of the learner's grasp, for this session
|
- "understanding": ONE word from the fixed scale — "solid", "shaky", "accepted", "resisted", or
|
||||||
only ("solid", "shaky", "accepted", "resisted", "not yet engaged"). No numbers, no predictions.
|
"not yet engaged" — judging the learner's grasp of the proposition in THIS session. Strictly one
|
||||||
|
word, no numbers, no justification, no explanation (a wrong-shaped note is rejected by the schema).
|
||||||
|
|
||||||
2. EDGES between propositions:
|
2. EDGES between propositions:
|
||||||
- "follows": to is DERIVABLE from from (understanding from makes understanding to achievable).
|
- "follows": to is DERIVABLE from from (understanding from makes understanding to achievable).
|
||||||
@@ -256,8 +258,9 @@ FAITHFULNESS — verify every candidate against the transcript:
|
|||||||
direction-swapped or speculative edges.
|
direction-swapped or speculative edges.
|
||||||
- ATTRIBUTION: the QUIZ line's question must genuinely test the attributed proposition. Drop wrong
|
- ATTRIBUTION: the QUIZ line's question must genuinely test the attributed proposition. Drop wrong
|
||||||
attributions to null, or re-point them to the proposition the question actually tests.
|
attributions to null, or re-point them to the proposition the question actually tests.
|
||||||
- UNDERSTANDING notes: verify against the quiz outcomes and the session's evidence; reword to a named-word
|
- UNDERSTANDING levels: verify against the quiz outcomes and the session's evidence; each note must be
|
||||||
scale or drop.
|
exactly one word from the fixed scale ("solid", "shaky", "accepted", "resisted", "not yet engaged") —
|
||||||
|
reword or drop anything else. No justifications.
|
||||||
|
|
||||||
IDENTITY — link-or-create, per proposition:
|
IDENTITY — link-or-create, per proposition:
|
||||||
- You see the current graph (every node: id, name, statement, aliases). For each verified proposition,
|
- You see the current graph (every node: id, name, statement, aliases). For each verified proposition,
|
||||||
|
|||||||
@@ -102,7 +102,8 @@ export interface LedgerEvent {
|
|||||||
* understanding observations — re-emission of the same chat must never
|
* understanding observations — re-emission of the same chat must never
|
||||||
* double-record an event. */
|
* double-record an event. */
|
||||||
ref?: string;
|
ref?: string;
|
||||||
/** The learner's quiz note — diagnostic gold (vision §1 calibration). */
|
/** The learner's quiz note (diagnostic gold), or the understanding
|
||||||
|
* LEVEL word for observations — a session-relative judgment. */
|
||||||
note?: string;
|
note?: string;
|
||||||
/** The quiz question text, when nodeId is null (later attribution input). */
|
/** The quiz question text, when nodeId is null (later attribution input). */
|
||||||
question?: string;
|
question?: string;
|
||||||
@@ -138,6 +139,16 @@ export interface LedgerEvent {
|
|||||||
|
|
||||||
// ─── Extraction findings (extractor output → audit → merger input) ───────────
|
// ─── Extraction findings (extractor output → audit → merger input) ───────────
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The extractor's standardized per-session understanding scale (DECIDED
|
||||||
|
* 2026-09-08): a CLOSED named-word list — no numbers, no prose, no
|
||||||
|
* justifications (a justification, if the agent wants one, is dropped;
|
||||||
|
* it is not stored). The level is session- and extraction-run-relative
|
||||||
|
* judgment about the learner, shown next to its session — never a node
|
||||||
|
* property.
|
||||||
|
*/
|
||||||
|
export type UnderstandingLevel = "solid" | "shaky" | "accepted" | "resisted" | "not yet engaged";
|
||||||
|
|
||||||
/** One proposed proposition — a node candidate (or a re-encounter of one). */
|
/** One proposed proposition — a node candidate (or a re-encounter of one). */
|
||||||
export interface ExtractedProposition {
|
export interface ExtractedProposition {
|
||||||
/** Extractor-local id ("p1", …) referenced by edges + attribution. */
|
/** Extractor-local id ("p1", …) referenced by edges + attribution. */
|
||||||
@@ -158,11 +169,11 @@ export interface ExtractedProposition {
|
|||||||
messages: string[];
|
messages: string[];
|
||||||
/** Verbatim transcript quotes backing the proposition. */
|
/** Verbatim transcript quotes backing the proposition. */
|
||||||
evidence: string[];
|
evidence: string[];
|
||||||
/** The extractor's understanding note for this proposition IN THIS
|
/** The extractor's understanding level for this proposition IN THIS
|
||||||
* session — a named-word scale ("solid", "shaky", "accepted",
|
* session — one word from the UnderstandingLevel scale. Joins the
|
||||||
* "resisted"), no numbers. Joins the ledger as an observation event;
|
* ledger as an observation event (note = the level word); treatment
|
||||||
* treatment pending the state-model ideation. */
|
* pending the state-model ideation. */
|
||||||
understanding?: string;
|
understanding?: UnderstandingLevel;
|
||||||
}
|
}
|
||||||
|
|
||||||
/** One proposed relation between two propositions. */
|
/** One proposed relation between two propositions. */
|
||||||
|
|||||||
+4
-4
@@ -172,7 +172,7 @@ function fixtureAudit(): AuditResult {
|
|||||||
aliases: ["alpha feel"],
|
aliases: ["alpha feel"],
|
||||||
messages: ["a1", "a3"],
|
messages: ["a1", "a3"],
|
||||||
evidence: ["fixture evidence"],
|
evidence: ["fixture evidence"],
|
||||||
understanding: "solid after the check",
|
understanding: "solid",
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
id: "p2",
|
id: "p2",
|
||||||
@@ -295,7 +295,7 @@ describe("the merger (mergeFindings)", () => {
|
|||||||
const observation = byRef.get("obs:p1")!;
|
const observation = byRef.get("obs:p1")!;
|
||||||
assert.equal(observation.kind, "observation");
|
assert.equal(observation.kind, "observation");
|
||||||
assert.equal(observation.nodeId, idByStatement.get("Alpha intuition"));
|
assert.equal(observation.nodeId, idByStatement.get("Alpha intuition"));
|
||||||
assert.equal(observation.note, "solid after the check");
|
assert.equal(observation.note, "solid");
|
||||||
|
|
||||||
// The dismissed quiz and the ask question became nothing.
|
// The dismissed quiz and the ask question became nothing.
|
||||||
assert.equal([...byRef.keys()].filter((r) => r === "call-6" || r === "call-7").length, 0);
|
assert.equal([...byRef.keys()].filter((r) => r === "call-6" || r === "call-7").length, 0);
|
||||||
@@ -380,9 +380,9 @@ describe("the merger (mergeFindings)", () => {
|
|||||||
assert.deepEqual(snap.nodeSessions[idByStatement.get("Gamma application")!], [chat1.id]);
|
assert.deepEqual(snap.nodeSessions[idByStatement.get("Gamma application")!], [chat1.id]);
|
||||||
|
|
||||||
// Understanding observations: (node → chat → the extractor's note).
|
// Understanding observations: (node → chat → the extractor's note).
|
||||||
// Alpha carries chat1's "solid after the check"; chat2's q1 has no note.
|
// Alpha carries chat1's "solid" level; chat2's q1 has no level.
|
||||||
const alphaObs = snap.nodeObservations[idByStatement.get("Alpha intuition")!] ?? {};
|
const alphaObs = snap.nodeObservations[idByStatement.get("Alpha intuition")!] ?? {};
|
||||||
assert.equal(alphaObs[chat1.id], "solid after the check");
|
assert.equal(alphaObs[chat1.id], "solid");
|
||||||
assert.equal(alphaObs[chat2Id], undefined);
|
assert.equal(alphaObs[chat2Id], undefined);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user