quiz nodeId required (owner ruling 2026-09-26): no untagged teach quizzes — boundary rejects, model retries tagged; off-map checks route to ask_user_question; anti-rush doctrine in schema+prompt; docs+comments updated
This commit is contained in:
@@ -79,10 +79,11 @@ That is the wrong signal and is **abolished**. The new contract:
|
|||||||
regardless of individual check outcomes (a failed check followed by a
|
regardless of individual check outcomes (a failed check followed by a
|
||||||
re-teach and a move-on is still a visited node).
|
re-teach and a move-on is still a visited node).
|
||||||
- **Failed checks move nothing.** Results no longer feed derivation at all.
|
- **Failed checks move nothing.** Results no longer feed derivation at all.
|
||||||
- **Untagged checks are legal ad-hoc checks** — they don't move the map
|
- **Untagged checks no longer exist** (owner ruling 2026-09-26): `nodeId`
|
||||||
(quick comprehension checks don't need nodes). Not every check needs a
|
is required on the teach quiz schema — an untagged call is rejected at the
|
||||||
node. The prompt stresses tagging when teaching a map node: an untagged
|
boundary and the model retries tagged. Off-map checks (quick comprehension
|
||||||
stream leaves the map frozen (lost signal).
|
checks that don't fit a node) route through ask_user_question. The map can
|
||||||
|
never freeze from a lost tagging signal again.
|
||||||
- **No fallback current.** Before the first tagged check nothing is
|
- **No fallback current.** Before the first tagged check nothing is
|
||||||
current. No derived-from-prereqs guess.
|
current. No derived-from-prereqs guess.
|
||||||
- Present-but-invalid ids are still rejected at the call boundary (existing
|
- Present-but-invalid ids are still rejected at the call boundary (existing
|
||||||
|
|||||||
@@ -26,7 +26,7 @@ You write the map with `plan_init` (a mermaid `graph TD` block). It is a **map,
|
|||||||
|
|
||||||
The map is yours to rewrite at your own initiative: re-call `plan_init` whenever it's wrong — a detour node for a question worth exploring, a repair node for a caught misconception, prune what proves unnecessary. A map that can't change is a wrong map.
|
The map is yours to rewrite at your own initiative: re-call `plan_init` whenever it's wrong — a detour node for a question worth exploring, a repair node for a caught misconception, prune what proves unnecessary. A map that can't change is a wrong map.
|
||||||
|
|
||||||
**Tag a quiz with `nodeId` whenever it's about a map node** — that's what moves the map: its live position is simply the latest tagged node, so asking about the next node IS how the lesson gets there. A tag is a position, not a verdict — nodes are never marked finished, a node takes as many questions as it deserves, and you move on by asking about the next one. Untagged questions are ad-hoc checks — legal, but they move nothing.
|
**Every quiz carries `nodeId`** (required — a quiz without it is rejected): that's what moves the map. Its live position is simply the latest tagged node, so asking about the next node IS how the lesson gets there. A tag is a position, not a verdict — nodes are never marked finished. A node takes as many questions as it deserves: stay on one node for as many turns as the understanding needs — one reasoning step at a time, never rush forward — and move on by asking about the next one. Off-map checks (no plan node fits) use ask_user_question.
|
||||||
|
|
||||||
## Presenting the plan — always, before any teaching
|
## Presenting the plan — always, before any teaching
|
||||||
|
|
||||||
|
|||||||
+8
-7
@@ -201,8 +201,8 @@ function nodeIdProblem(ctx: Pick<PlanToolContext, "chatId" | "chatStore">, nodeI
|
|||||||
return (
|
return (
|
||||||
`nodeId "${nodeId}" matches no node in this session's plan map. ` +
|
`nodeId "${nodeId}" matches no node in this session's plan map. ` +
|
||||||
(known.length > 0
|
(known.length > 0
|
||||||
? `Copy one of your plan_init slugs exactly (${known.slice(0, 5).join(", ")}${known.length > 5 ? ", …" : ""}), or omit nodeId for an ad-hoc check.`
|
? `Copy one of your plan_init slugs exactly (${known.slice(0, 5).join(", ")}${known.length > 5 ? ", …" : ""}).`
|
||||||
: "No plan map exists yet — omit nodeId (ad-hoc checks only until you call plan_init).")
|
: "No plan map exists yet — call plan_init before posing quizzes.")
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
return null;
|
return null;
|
||||||
@@ -212,7 +212,9 @@ function nodeIdProblem(ctx: Pick<PlanToolContext, "chatId" | "chatStore">, nodeI
|
|||||||
* + the per-session plan-nodeId validation (both run in execute — the
|
* + the per-session plan-nodeId validation (both run in execute — the
|
||||||
* grading contract is untouched). Reads the LIVE chat record, so rewrites
|
* grading contract is untouched). Reads the LIVE chat record, so rewrites
|
||||||
* mid-turn are honored, and with no tree yet every nodeId is rejected.
|
* mid-turn are honored, and with no tree yet every nodeId is rejected.
|
||||||
* Omitting nodeId is legal — an ad-hoc check that moves nothing. */
|
* nodeId is REQUIRED (owner ruling 2026-09-26): the untagged ad-hoc quiz is
|
||||||
|
* gone — off-map checks route through ask_user_question, and the teach
|
||||||
|
* seam (first tagged quiz) is structurally guaranteed. */
|
||||||
export function teachQuizToolFor(ctx: Pick<PlanToolContext, "chatId" | "chatStore">): AgentTool<typeof quizInputObject> {
|
export function teachQuizToolFor(ctx: Pick<PlanToolContext, "chatId" | "chatStore">): AgentTool<typeof quizInputObject> {
|
||||||
return {
|
return {
|
||||||
name: "quiz",
|
name: "quiz",
|
||||||
@@ -222,8 +224,9 @@ export function teachQuizToolFor(ctx: Pick<PlanToolContext, "chatId" | "chatStor
|
|||||||
"Unlike ask_user_question (which collects preferences/decisions with no right answer), quiz always has a " +
|
"Unlike ask_user_question (which collects preferences/decisions with no right answer), quiz always has a " +
|
||||||
"correct answer supplied by you, marks the learner's selection right/wrong, reveals the correct answer, and " +
|
"correct answer supplied by you, marks the learner's selection right/wrong, reveals the correct answer, and " +
|
||||||
"can show an explanation. Use it to (1) confirm a taught node actually landed, or (2) probe understanding " +
|
"can show an explanation. Use it to (1) confirm a taught node actually landed, or (2) probe understanding " +
|
||||||
"whenever you're unsure. Set `nodeId` whenever the question targets a map node — the lesson's position on the " +
|
"whenever you're unsure. Set `nodeId` (required) to the plan node this question is about — the lesson's " +
|
||||||
"map derives from those tags; omit it only for quick ad-hoc checks. Options-only: single-select or multi-select, " +
|
"position on the map derives from those tags; a node takes as many questions as it deserves, so don't rush " +
|
||||||
|
"forward. Off-map checks use ask_user_question. Options-only: single-select or multi-select, " +
|
||||||
"plus an automatic 'I don't know' choice so the learner can signal a genuine gap instead of guessing. " +
|
"plus an automatic 'I don't know' choice so the learner can signal a genuine gap instead of guessing. " +
|
||||||
"An always-present optional note field lets the learner attach a free-text note to ANY answer; it reaches you " +
|
"An always-present optional note field lets the learner attach a free-text note to ANY answer; it reaches you " +
|
||||||
"only when non-empty. No free-text answers — for non-graded questions use ask_user_question instead. " +
|
"only when non-empty. No free-text answers — for non-graded questions use ask_user_question instead. " +
|
||||||
@@ -233,10 +236,8 @@ export function teachQuizToolFor(ctx: Pick<PlanToolContext, "chatId" | "chatStor
|
|||||||
executionMode: "sequential",
|
executionMode: "sequential",
|
||||||
execute: async (_toolCallId, input) => {
|
execute: async (_toolCallId, input) => {
|
||||||
const problems = [...quizInputProblems(input)];
|
const problems = [...quizInputProblems(input)];
|
||||||
if (typeof input.nodeId === "string" && input.nodeId.length > 0) {
|
|
||||||
const nodeIdProblemText = nodeIdProblem(ctx, input.nodeId);
|
const nodeIdProblemText = nodeIdProblem(ctx, input.nodeId);
|
||||||
if (nodeIdProblemText) problems.push(nodeIdProblemText);
|
if (nodeIdProblemText) problems.push(nodeIdProblemText);
|
||||||
}
|
|
||||||
if (problems.length > 0) throw new ToolInputError("quiz", problems);
|
if (problems.length > 0) throw new ToolInputError("quiz", problems);
|
||||||
return questionResult();
|
return questionResult();
|
||||||
},
|
},
|
||||||
|
|||||||
@@ -55,9 +55,9 @@ export type PlanNodeStatus = "current" | "visited" | "upcoming";
|
|||||||
// node is the teacher's judgment the previous one is finished, regardless of
|
// node is the teacher's judgment the previous one is finished, regardless of
|
||||||
// check outcomes (a failed check then a move-on is still visited); results
|
// check outcomes (a failed check then a move-on is still visited); results
|
||||||
// feed nothing; there is NO fallback current (before the first tagged check
|
// feed nothing; there is NO fallback current (before the first tagged check
|
||||||
// nothing is current). Untagged checks are legal ad-hoc checks — they move
|
// nothing is current). Untagged checks no longer exist (nodeId is required
|
||||||
// nothing. NodeIds missing from the current tree are ignored (a rewrite
|
// on the teach quiz, owner ruling 2026-09-26). NodeIds missing from the
|
||||||
// may have dropped the node).
|
// current tree are ignored (a rewrite may have dropped the node).
|
||||||
|
|
||||||
/** Minimal structural message slice — anything with typed parts (mirrors
|
/** Minimal structural message slice — anything with typed parts (mirrors
|
||||||
* phases.ts: pure, no runtime deps, safe on server, client, and in tests). */
|
* phases.ts: pure, no runtime deps, safe on server, client, and in tests). */
|
||||||
|
|||||||
+3
-2
@@ -17,8 +17,9 @@
|
|||||||
* turn, and the learner's next message is a natural reply (a change
|
* turn, and the learner's next message is a natural reply (a change
|
||||||
* request, or the go-ahead). The live indicator LAGS by design (owner
|
* request, or the go-ahead). The live indicator LAGS by design (owner
|
||||||
* accepted): "teach" shows only once the first node's question is posed.
|
* accepted): "teach" shows only once the first node's question is posed.
|
||||||
* A teacher that never tags a quiz would sit in "plan" visually —
|
* A teacher that never tags a quiz can no longer exist — nodeId is
|
||||||
* off-doctrine (tagging is prompt-mandated and nodeId is call-validated).
|
* schema-required at the teach quiz boundary (owner ruling 2026-09-26),
|
||||||
|
* so the seam signal is structural, not prompt-mandated.
|
||||||
*
|
*
|
||||||
* The former derived-occupancy seam (phase-of-tree via `currentNodeId`)
|
* The former derived-occupancy seam (phase-of-tree via `currentNodeId`)
|
||||||
* is GONE, as is the verdict-marker fold: no backwards compatibility
|
* is GONE, as is the verdict-marker fold: no backwards compatibility
|
||||||
|
|||||||
@@ -200,16 +200,15 @@ export const quizInputBase = Type.Object({
|
|||||||
// never touches).
|
// never touches).
|
||||||
export const quizInputObject = Type.Object({
|
export const quizInputObject = Type.Object({
|
||||||
...quizInputBase.properties,
|
...quizInputBase.properties,
|
||||||
nodeId: Type.Optional(
|
nodeId: Type.String({
|
||||||
Type.String({
|
|
||||||
description:
|
description:
|
||||||
"Optional plan-tree node id this question is about — must EXACTLY match one of this session's " +
|
"REQUIRED plan-tree node id this question is about — must EXACTLY match one of this session's " +
|
||||||
"plan_init node slugs (validated at call time; a mismatched id is rejected back to you). " +
|
"plan_init node slugs (validated at call time; a mismatched id is rejected back to you). " +
|
||||||
"Tag every question that targets a map node: the lesson's position on the map derives from these tags " +
|
"The lesson's position on the map derives from these tags (the node the latest tagged question is about is " +
|
||||||
"(the node the latest tagged question is about is where the map stands; earlier ones count as visited). " +
|
"where the map stands; earlier ones count as visited). A node takes as many questions as it deserves: stay on " +
|
||||||
"Omit for quick ad-hoc checks that don't target a node.",
|
"one node for as many turns as the understanding needs — one reasoning step at a time, never rush forward. " +
|
||||||
|
"For off-map checks use ask_user_question.",
|
||||||
}),
|
}),
|
||||||
),
|
|
||||||
});
|
});
|
||||||
|
|
||||||
/** The PROBE-side quiz declaration: the shared authoring contract, no
|
/** The PROBE-side quiz declaration: the shared authoring contract, no
|
||||||
|
|||||||
@@ -55,6 +55,7 @@ const QUIZ_INPUT = {
|
|||||||
],
|
],
|
||||||
correctAnswer: "Mercury",
|
correctAnswer: "Mercury",
|
||||||
explanation: "Mercury is the innermost planet.",
|
explanation: "Mercury is the innermost planet.",
|
||||||
|
nodeId: "planet-order",
|
||||||
};
|
};
|
||||||
|
|
||||||
/** Turn 1: a bit of reasoning, then the quiz tool call. Turn 2: plain text.
|
/** Turn 1: a bit of reasoning, then the quiz tool call. Turn 2: plain text.
|
||||||
@@ -187,6 +188,18 @@ function post(chatId: string, input: unknown): Request {
|
|||||||
function createConfiguredChat(title: string) {
|
function createConfiguredChat(title: string) {
|
||||||
const chat = chatStore.createChat(title);
|
const chat = chatStore.createChat(title);
|
||||||
chatStore.saveSettings(chat.id, { model: { provider: "openrouter", id: "z-ai/glm-5.3-flash" } });
|
chatStore.saveSettings(chat.id, { model: { provider: "openrouter", id: "z-ai/glm-5.3-flash" } });
|
||||||
|
// nodeId is required on teach quizzes (owner ruling 2026-09-26) — the
|
||||||
|
// scripted lessons quiz against this seeded tree.
|
||||||
|
chatStore.saveTree(chat.id, {
|
||||||
|
nodes: [
|
||||||
|
{ id: "planet-order", title: "Planet order" },
|
||||||
|
{ id: "genealogy", title: "Genealogy of religion" },
|
||||||
|
{ id: "indexicals", title: "Indexical 'actual'" },
|
||||||
|
{ id: "modal-realism", title: "Modal realism" },
|
||||||
|
],
|
||||||
|
edges: [],
|
||||||
|
version: 1,
|
||||||
|
});
|
||||||
return chat;
|
return chat;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -560,6 +573,7 @@ describe("teacher chat flow (scripted model)", () => {
|
|||||||
// same turn. This pins that whole mechanism (the live Nietzsche bug).
|
// same turn. This pins that whole mechanism (the live Nietzsche bug).
|
||||||
const BAD_QUIZ_INPUT = {
|
const BAD_QUIZ_INPUT = {
|
||||||
question: "What is Nietzsche's genealogical claim about religion?",
|
question: "What is Nietzsche's genealogical claim about religion?",
|
||||||
|
nodeId: "genealogy",
|
||||||
options: [{ label: "A cultural diagnosis" }, { label: "Atheism as a conclusion" }],
|
options: [{ label: "A cultural diagnosis" }, { label: "Atheism as a conclusion" }],
|
||||||
correctAnswer: "cultural-diagnosis", // the exact live failure shape: a slug, no match
|
correctAnswer: "cultural-diagnosis", // the exact live failure shape: a slug, no match
|
||||||
explanation: "Genealogy diagnoses, it does not convert.",
|
explanation: "Genealogy diagnoses, it does not convert.",
|
||||||
@@ -635,6 +649,7 @@ describe("teacher chat flow (scripted model)", () => {
|
|||||||
// question, and the model's retry must become the pending quiz.
|
// question, and the model's retry must become the pending quiz.
|
||||||
const STRINGY_QUIZ_INPUT = {
|
const STRINGY_QUIZ_INPUT = {
|
||||||
question: "Lewis's indexical analysis of 'actual' — which is it?",
|
question: "Lewis's indexical analysis of 'actual' — which is it?",
|
||||||
|
nodeId: "indexicals",
|
||||||
options: JSON.stringify([
|
options: JSON.stringify([
|
||||||
{ label: "Like 'I'", value: "indexical" },
|
{ label: "Like 'I'", value: "indexical" },
|
||||||
{ label: "Like a title", value: "unique" },
|
{ label: "Like a title", value: "unique" },
|
||||||
|
|||||||
@@ -73,6 +73,7 @@ describe("quizInputSchema / quizInputProblems (the authoring contract)", () => {
|
|||||||
options: [{ label: "Venus" }, { label: "Mercury" }, { label: "Mars" }],
|
options: [{ label: "Venus" }, { label: "Mercury" }, { label: "Mars" }],
|
||||||
correctAnswer: "Mercury",
|
correctAnswer: "Mercury",
|
||||||
explanation: "Mercury is the innermost planet.",
|
explanation: "Mercury is the innermost planet.",
|
||||||
|
nodeId: "planet-order",
|
||||||
};
|
};
|
||||||
|
|
||||||
it("accepts a valid single-select quiz (label as correctAnswer)", () => {
|
it("accepts a valid single-select quiz (label as correctAnswer)", () => {
|
||||||
|
|||||||
@@ -25,7 +25,7 @@ import {
|
|||||||
} from "../src/agent/tools.ts";
|
} from "../src/agent/tools.ts";
|
||||||
import { TOGGLEABLE_TOOLS } from "../src/core/pending.ts";
|
import { TOGGLEABLE_TOOLS } from "../src/core/pending.ts";
|
||||||
import { jobStore } from "../src/store/jobs.ts";
|
import { jobStore } from "../src/store/jobs.ts";
|
||||||
import { probeQuizInputSchema } from "../src/core/question-core.ts";
|
import { probeQuizInputSchema, quizInputObject } from "../src/core/question-core.ts";
|
||||||
import { chatStore } from "../src/store/chats.ts";
|
import { chatStore } from "../src/store/chats.ts";
|
||||||
import { PATHS } from "../config.ts";
|
import { PATHS } from "../config.ts";
|
||||||
|
|
||||||
@@ -57,6 +57,7 @@ const validQuiz = {
|
|||||||
],
|
],
|
||||||
correctAnswer: "4",
|
correctAnswer: "4",
|
||||||
explanation: "Arithmetic.",
|
explanation: "Arithmetic.",
|
||||||
|
nodeId: "arith",
|
||||||
};
|
};
|
||||||
|
|
||||||
import type { AgentTool } from "@earendil-works/pi-agent-core";
|
import type { AgentTool } from "@earendil-works/pi-agent-core";
|
||||||
@@ -119,15 +120,33 @@ describe("teachQuizToolFor: the dynamic plan-nodeId validation (position signal)
|
|||||||
assert.match(msg, /arith/);
|
assert.match(msg, /arith/);
|
||||||
});
|
});
|
||||||
|
|
||||||
it("no tree yet: ANY nodeId is rejected — omit it for ad-hoc checks", async () => {
|
it("no tree yet: ANY nodeId is rejected — quizzes wait for plan_init", async () => {
|
||||||
const chatId = newChat("NodeId: no tree");
|
const chatId = newChat("NodeId: no tree");
|
||||||
const tool = teachQuizToolFor({ chatId, chatStore });
|
const tool = teachQuizToolFor({ chatId, chatStore });
|
||||||
const res = await boundaryParse(tool, { ...validQuiz, nodeId: "anything" });
|
const res = await boundaryParse(tool, { ...validQuiz, nodeId: "anything" });
|
||||||
assert.equal(res.success, false);
|
assert.equal(res.success, false);
|
||||||
assert.match(problemLines(res.error).at(-1)!, /No plan map exists yet/);
|
assert.match(problemLines(res.error).at(-1)!, /No plan map exists yet — call plan_init/);
|
||||||
// …while an untagged ad-hoc check is fine
|
});
|
||||||
const plain = await boundaryParse(tool, validQuiz);
|
|
||||||
assert.equal(plain.success, true);
|
it("missing nodeId is rejected at the boundary (owner ruling 2026-09-26: no untagged quizzes)", async () => {
|
||||||
|
const chatId = newChat("NodeId: required");
|
||||||
|
chatStore.saveTree(chatId, {
|
||||||
|
nodes: [{ id: "arith", title: "Arithmetic" }],
|
||||||
|
edges: [],
|
||||||
|
version: 1,
|
||||||
|
});
|
||||||
|
const tool = teachQuizToolFor({ chatId, chatStore });
|
||||||
|
const { nodeId: _dropped, ...untagged } = validQuiz;
|
||||||
|
const res = await boundaryParse(tool, untagged);
|
||||||
|
assert.equal(res.success, false, "an untagged quiz must be rejected back to the model");
|
||||||
|
// The structural pin: the SCHEMA itself names the missing field (pi's
|
||||||
|
// boundary formats Value.Errors as the tool-input-error the model sees).
|
||||||
|
const schemaErrors = [...Value.Errors(quizInputObject, untagged)].map((e) => e.message);
|
||||||
|
assert.match(
|
||||||
|
schemaErrors.join("\n"),
|
||||||
|
/nodeId/,
|
||||||
|
"the schema rejection names nodeId so the model can retry tagged",
|
||||||
|
);
|
||||||
});
|
});
|
||||||
|
|
||||||
it("reads the LIVE record — a plan_init mutation mid-turn is honored immediately", async () => {
|
it("reads the LIVE record — a plan_init mutation mid-turn is honored immediately", async () => {
|
||||||
|
|||||||
Reference in New Issue
Block a user