quiz nodeId required (owner ruling 2026-09-26): no untagged teach quizzes — boundary rejects, model retries tagged; off-map checks route to ask_user_question; anti-rush doctrine in schema+prompt; docs+comments updated

This commit is contained in:
amosblomqvist
2026-09-27 02:05:51 +03:00
parent 8c0f0f7fbf
commit d6d7c09e2e
9 changed files with 72 additions and 35 deletions
+5 -4
View File
@@ -79,10 +79,11 @@ That is the wrong signal and is **abolished**. The new contract:
regardless of individual check outcomes (a failed check followed by a regardless of individual check outcomes (a failed check followed by a
re-teach and a move-on is still a visited node). re-teach and a move-on is still a visited node).
- **Failed checks move nothing.** Results no longer feed derivation at all. - **Failed checks move nothing.** Results no longer feed derivation at all.
- **Untagged checks are legal ad-hoc checks** — they don't move the map - **Untagged checks no longer exist** (owner ruling 2026-09-26): `nodeId`
(quick comprehension checks don't need nodes). Not every check needs a is required on the teach quiz schema — an untagged call is rejected at the
node. The prompt stresses tagging when teaching a map node: an untagged boundary and the model retries tagged. Off-map checks (quick comprehension
stream leaves the map frozen (lost signal). checks that don't fit a node) route through ask_user_question. The map can
never freeze from a lost tagging signal again.
- **No fallback current.** Before the first tagged check nothing is - **No fallback current.** Before the first tagged check nothing is
current. No derived-from-prereqs guess. current. No derived-from-prereqs guess.
- Present-but-invalid ids are still rejected at the call boundary (existing - Present-but-invalid ids are still rejected at the call boundary (existing
+1 -1
View File
@@ -26,7 +26,7 @@ You write the map with `plan_init` (a mermaid `graph TD` block). It is a **map,
The map is yours to rewrite at your own initiative: re-call `plan_init` whenever it's wrong — a detour node for a question worth exploring, a repair node for a caught misconception, prune what proves unnecessary. A map that can't change is a wrong map. The map is yours to rewrite at your own initiative: re-call `plan_init` whenever it's wrong — a detour node for a question worth exploring, a repair node for a caught misconception, prune what proves unnecessary. A map that can't change is a wrong map.
**Tag a quiz with `nodeId` whenever it's about a map node** — that's what moves the map: its live position is simply the latest tagged node, so asking about the next node IS how the lesson gets there. A tag is a position, not a verdict — nodes are never marked finished, a node takes as many questions as it deserves, and you move on by asking about the next one. Untagged questions are ad-hoc checks — legal, but they move nothing. **Every quiz carries `nodeId`** (required — a quiz without it is rejected): that's what moves the map. Its live position is simply the latest tagged node, so asking about the next node IS how the lesson gets there. A tag is a position, not a verdict — nodes are never marked finished. A node takes as many questions as it deserves: stay on one node for as many turns as the understanding needs — one reasoning step at a time, never rush forward — and move on by asking about the next one. Off-map checks (no plan node fits) use ask_user_question.
## Presenting the plan — always, before any teaching ## Presenting the plan — always, before any teaching
+8 -7
View File
@@ -201,8 +201,8 @@ function nodeIdProblem(ctx: Pick<PlanToolContext, "chatId" | "chatStore">, nodeI
return ( return (
`nodeId "${nodeId}" matches no node in this session's plan map. ` + `nodeId "${nodeId}" matches no node in this session's plan map. ` +
(known.length > 0 (known.length > 0
? `Copy one of your plan_init slugs exactly (${known.slice(0, 5).join(", ")}${known.length > 5 ? ", …" : ""}), or omit nodeId for an ad-hoc check.` ? `Copy one of your plan_init slugs exactly (${known.slice(0, 5).join(", ")}${known.length > 5 ? ", …" : ""}).`
: "No plan map exists yet — omit nodeId (ad-hoc checks only until you call plan_init).") : "No plan map exists yet — call plan_init before posing quizzes.")
); );
} }
return null; return null;
@@ -212,7 +212,9 @@ function nodeIdProblem(ctx: Pick<PlanToolContext, "chatId" | "chatStore">, nodeI
* + the per-session plan-nodeId validation (both run in execute — the * + the per-session plan-nodeId validation (both run in execute — the
* grading contract is untouched). Reads the LIVE chat record, so rewrites * grading contract is untouched). Reads the LIVE chat record, so rewrites
* mid-turn are honored, and with no tree yet every nodeId is rejected. * mid-turn are honored, and with no tree yet every nodeId is rejected.
* Omitting nodeId is legal — an ad-hoc check that moves nothing. */ * nodeId is REQUIRED (owner ruling 2026-09-26): the untagged ad-hoc quiz is
* gone — off-map checks route through ask_user_question, and the teach
* seam (first tagged quiz) is structurally guaranteed. */
export function teachQuizToolFor(ctx: Pick<PlanToolContext, "chatId" | "chatStore">): AgentTool<typeof quizInputObject> { export function teachQuizToolFor(ctx: Pick<PlanToolContext, "chatId" | "chatStore">): AgentTool<typeof quizInputObject> {
return { return {
name: "quiz", name: "quiz",
@@ -222,8 +224,9 @@ export function teachQuizToolFor(ctx: Pick<PlanToolContext, "chatId" | "chatStor
"Unlike ask_user_question (which collects preferences/decisions with no right answer), quiz always has a " + "Unlike ask_user_question (which collects preferences/decisions with no right answer), quiz always has a " +
"correct answer supplied by you, marks the learner's selection right/wrong, reveals the correct answer, and " + "correct answer supplied by you, marks the learner's selection right/wrong, reveals the correct answer, and " +
"can show an explanation. Use it to (1) confirm a taught node actually landed, or (2) probe understanding " + "can show an explanation. Use it to (1) confirm a taught node actually landed, or (2) probe understanding " +
"whenever you're unsure. Set `nodeId` whenever the question targets a map node — the lesson's position on the " + "whenever you're unsure. Set `nodeId` (required) to the plan node this question is about — the lesson's " +
"map derives from those tags; omit it only for quick ad-hoc checks. Options-only: single-select or multi-select, " + "position on the map derives from those tags; a node takes as many questions as it deserves, so don't rush " +
"forward. Off-map checks use ask_user_question. Options-only: single-select or multi-select, " +
"plus an automatic 'I don't know' choice so the learner can signal a genuine gap instead of guessing. " + "plus an automatic 'I don't know' choice so the learner can signal a genuine gap instead of guessing. " +
"An always-present optional note field lets the learner attach a free-text note to ANY answer; it reaches you " + "An always-present optional note field lets the learner attach a free-text note to ANY answer; it reaches you " +
"only when non-empty. No free-text answers — for non-graded questions use ask_user_question instead. " + "only when non-empty. No free-text answers — for non-graded questions use ask_user_question instead. " +
@@ -233,10 +236,8 @@ export function teachQuizToolFor(ctx: Pick<PlanToolContext, "chatId" | "chatStor
executionMode: "sequential", executionMode: "sequential",
execute: async (_toolCallId, input) => { execute: async (_toolCallId, input) => {
const problems = [...quizInputProblems(input)]; const problems = [...quizInputProblems(input)];
if (typeof input.nodeId === "string" && input.nodeId.length > 0) {
const nodeIdProblemText = nodeIdProblem(ctx, input.nodeId); const nodeIdProblemText = nodeIdProblem(ctx, input.nodeId);
if (nodeIdProblemText) problems.push(nodeIdProblemText); if (nodeIdProblemText) problems.push(nodeIdProblemText);
}
if (problems.length > 0) throw new ToolInputError("quiz", problems); if (problems.length > 0) throw new ToolInputError("quiz", problems);
return questionResult(); return questionResult();
}, },
+3 -3
View File
@@ -55,9 +55,9 @@ export type PlanNodeStatus = "current" | "visited" | "upcoming";
// node is the teacher's judgment the previous one is finished, regardless of // node is the teacher's judgment the previous one is finished, regardless of
// check outcomes (a failed check then a move-on is still visited); results // check outcomes (a failed check then a move-on is still visited); results
// feed nothing; there is NO fallback current (before the first tagged check // feed nothing; there is NO fallback current (before the first tagged check
// nothing is current). Untagged checks are legal ad-hoc checks — they move // nothing is current). Untagged checks no longer exist (nodeId is required
// nothing. NodeIds missing from the current tree are ignored (a rewrite // on the teach quiz, owner ruling 2026-09-26). NodeIds missing from the
// may have dropped the node). // current tree are ignored (a rewrite may have dropped the node).
/** Minimal structural message slice — anything with typed parts (mirrors /** Minimal structural message slice — anything with typed parts (mirrors
* phases.ts: pure, no runtime deps, safe on server, client, and in tests). */ * phases.ts: pure, no runtime deps, safe on server, client, and in tests). */
+3 -2
View File
@@ -17,8 +17,9 @@
* turn, and the learner's next message is a natural reply (a change * turn, and the learner's next message is a natural reply (a change
* request, or the go-ahead). The live indicator LAGS by design (owner * request, or the go-ahead). The live indicator LAGS by design (owner
* accepted): "teach" shows only once the first node's question is posed. * accepted): "teach" shows only once the first node's question is posed.
* A teacher that never tags a quiz would sit in "plan" visually — * A teacher that never tags a quiz can no longer exist — nodeId is
* off-doctrine (tagging is prompt-mandated and nodeId is call-validated). * schema-required at the teach quiz boundary (owner ruling 2026-09-26),
* so the seam signal is structural, not prompt-mandated.
* *
* The former derived-occupancy seam (phase-of-tree via `currentNodeId`) * The former derived-occupancy seam (phase-of-tree via `currentNodeId`)
* is GONE, as is the verdict-marker fold: no backwards compatibility * is GONE, as is the verdict-marker fold: no backwards compatibility
+6 -7
View File
@@ -200,16 +200,15 @@ export const quizInputBase = Type.Object({
// never touches). // never touches).
export const quizInputObject = Type.Object({ export const quizInputObject = Type.Object({
...quizInputBase.properties, ...quizInputBase.properties,
nodeId: Type.Optional( nodeId: Type.String({
Type.String({
description: description:
"Optional plan-tree node id this question is about — must EXACTLY match one of this session's " + "REQUIRED plan-tree node id this question is about — must EXACTLY match one of this session's " +
"plan_init node slugs (validated at call time; a mismatched id is rejected back to you). " + "plan_init node slugs (validated at call time; a mismatched id is rejected back to you). " +
"Tag every question that targets a map node: the lesson's position on the map derives from these tags " + "The lesson's position on the map derives from these tags (the node the latest tagged question is about is " +
"(the node the latest tagged question is about is where the map stands; earlier ones count as visited). " + "where the map stands; earlier ones count as visited). A node takes as many questions as it deserves: stay on " +
"Omit for quick ad-hoc checks that don't target a node.", "one node for as many turns as the understanding needs — one reasoning step at a time, never rush forward. " +
"For off-map checks use ask_user_question.",
}), }),
),
}); });
/** The PROBE-side quiz declaration: the shared authoring contract, no /** The PROBE-side quiz declaration: the shared authoring contract, no
+15
View File
@@ -55,6 +55,7 @@ const QUIZ_INPUT = {
], ],
correctAnswer: "Mercury", correctAnswer: "Mercury",
explanation: "Mercury is the innermost planet.", explanation: "Mercury is the innermost planet.",
nodeId: "planet-order",
}; };
/** Turn 1: a bit of reasoning, then the quiz tool call. Turn 2: plain text. /** Turn 1: a bit of reasoning, then the quiz tool call. Turn 2: plain text.
@@ -187,6 +188,18 @@ function post(chatId: string, input: unknown): Request {
function createConfiguredChat(title: string) { function createConfiguredChat(title: string) {
const chat = chatStore.createChat(title); const chat = chatStore.createChat(title);
chatStore.saveSettings(chat.id, { model: { provider: "openrouter", id: "z-ai/glm-5.3-flash" } }); chatStore.saveSettings(chat.id, { model: { provider: "openrouter", id: "z-ai/glm-5.3-flash" } });
// nodeId is required on teach quizzes (owner ruling 2026-09-26) — the
// scripted lessons quiz against this seeded tree.
chatStore.saveTree(chat.id, {
nodes: [
{ id: "planet-order", title: "Planet order" },
{ id: "genealogy", title: "Genealogy of religion" },
{ id: "indexicals", title: "Indexical 'actual'" },
{ id: "modal-realism", title: "Modal realism" },
],
edges: [],
version: 1,
});
return chat; return chat;
} }
@@ -560,6 +573,7 @@ describe("teacher chat flow (scripted model)", () => {
// same turn. This pins that whole mechanism (the live Nietzsche bug). // same turn. This pins that whole mechanism (the live Nietzsche bug).
const BAD_QUIZ_INPUT = { const BAD_QUIZ_INPUT = {
question: "What is Nietzsche's genealogical claim about religion?", question: "What is Nietzsche's genealogical claim about religion?",
nodeId: "genealogy",
options: [{ label: "A cultural diagnosis" }, { label: "Atheism as a conclusion" }], options: [{ label: "A cultural diagnosis" }, { label: "Atheism as a conclusion" }],
correctAnswer: "cultural-diagnosis", // the exact live failure shape: a slug, no match correctAnswer: "cultural-diagnosis", // the exact live failure shape: a slug, no match
explanation: "Genealogy diagnoses, it does not convert.", explanation: "Genealogy diagnoses, it does not convert.",
@@ -635,6 +649,7 @@ describe("teacher chat flow (scripted model)", () => {
// question, and the model's retry must become the pending quiz. // question, and the model's retry must become the pending quiz.
const STRINGY_QUIZ_INPUT = { const STRINGY_QUIZ_INPUT = {
question: "Lewis's indexical analysis of 'actual' — which is it?", question: "Lewis's indexical analysis of 'actual' — which is it?",
nodeId: "indexicals",
options: JSON.stringify([ options: JSON.stringify([
{ label: "Like 'I'", value: "indexical" }, { label: "Like 'I'", value: "indexical" },
{ label: "Like a title", value: "unique" }, { label: "Like a title", value: "unique" },
+1
View File
@@ -73,6 +73,7 @@ describe("quizInputSchema / quizInputProblems (the authoring contract)", () => {
options: [{ label: "Venus" }, { label: "Mercury" }, { label: "Mars" }], options: [{ label: "Venus" }, { label: "Mercury" }, { label: "Mars" }],
correctAnswer: "Mercury", correctAnswer: "Mercury",
explanation: "Mercury is the innermost planet.", explanation: "Mercury is the innermost planet.",
nodeId: "planet-order",
}; };
it("accepts a valid single-select quiz (label as correctAnswer)", () => { it("accepts a valid single-select quiz (label as correctAnswer)", () => {
+25 -6
View File
@@ -25,7 +25,7 @@ import {
} from "../src/agent/tools.ts"; } from "../src/agent/tools.ts";
import { TOGGLEABLE_TOOLS } from "../src/core/pending.ts"; import { TOGGLEABLE_TOOLS } from "../src/core/pending.ts";
import { jobStore } from "../src/store/jobs.ts"; import { jobStore } from "../src/store/jobs.ts";
import { probeQuizInputSchema } from "../src/core/question-core.ts"; import { probeQuizInputSchema, quizInputObject } from "../src/core/question-core.ts";
import { chatStore } from "../src/store/chats.ts"; import { chatStore } from "../src/store/chats.ts";
import { PATHS } from "../config.ts"; import { PATHS } from "../config.ts";
@@ -57,6 +57,7 @@ const validQuiz = {
], ],
correctAnswer: "4", correctAnswer: "4",
explanation: "Arithmetic.", explanation: "Arithmetic.",
nodeId: "arith",
}; };
import type { AgentTool } from "@earendil-works/pi-agent-core"; import type { AgentTool } from "@earendil-works/pi-agent-core";
@@ -119,15 +120,33 @@ describe("teachQuizToolFor: the dynamic plan-nodeId validation (position signal)
assert.match(msg, /arith/); assert.match(msg, /arith/);
}); });
it("no tree yet: ANY nodeId is rejected — omit it for ad-hoc checks", async () => { it("no tree yet: ANY nodeId is rejected — quizzes wait for plan_init", async () => {
const chatId = newChat("NodeId: no tree"); const chatId = newChat("NodeId: no tree");
const tool = teachQuizToolFor({ chatId, chatStore }); const tool = teachQuizToolFor({ chatId, chatStore });
const res = await boundaryParse(tool, { ...validQuiz, nodeId: "anything" }); const res = await boundaryParse(tool, { ...validQuiz, nodeId: "anything" });
assert.equal(res.success, false); assert.equal(res.success, false);
assert.match(problemLines(res.error).at(-1)!, /No plan map exists yet/); assert.match(problemLines(res.error).at(-1)!, /No plan map exists yet — call plan_init/);
// …while an untagged ad-hoc check is fine });
const plain = await boundaryParse(tool, validQuiz);
assert.equal(plain.success, true); it("missing nodeId is rejected at the boundary (owner ruling 2026-09-26: no untagged quizzes)", async () => {
const chatId = newChat("NodeId: required");
chatStore.saveTree(chatId, {
nodes: [{ id: "arith", title: "Arithmetic" }],
edges: [],
version: 1,
});
const tool = teachQuizToolFor({ chatId, chatStore });
const { nodeId: _dropped, ...untagged } = validQuiz;
const res = await boundaryParse(tool, untagged);
assert.equal(res.success, false, "an untagged quiz must be rejected back to the model");
// The structural pin: the SCHEMA itself names the missing field (pi's
// boundary formats Value.Errors as the tool-input-error the model sees).
const schemaErrors = [...Value.Errors(quizInputObject, untagged)].map((e) => e.message);
assert.match(
schemaErrors.join("\n"),
/nodeId/,
"the schema rejection names nodeId so the model can retry tagged",
);
}); });
it("reads the LIVE record — a plan_init mutation mid-turn is honored immediately", async () => { it("reads the LIVE record — a plan_init mutation mid-turn is honored immediately", async () => {