sorokin.oleg.edu/learnharness
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 6 Wiki Activity

quiz nodeId required (owner ruling 2026-09-26): no untagged teach quizzes — boundary rejects, model retries tagged; off-map checks route to ask_user_question; anti-rush doctrine in schema+prompt; docs+comments updated

Browse Source
This commit is contained in:
amosblomqvist
2026-09-27 02:05:51 +03:00
parent 8c0f0f7fbf
commit d6d7c09e2e
9 changed files with 72 additions and 35 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/agent/prompts/teacher/10-planning.md
+1 -1
View File
@@ -26,7 +26,7 @@ You write the map with `plan_init` (a mermaid `graph TD` block). It is a **map,
The map is yours to rewrite at your own initiative: re-call `plan_init` whenever it's wrong — a detour node for a question worth exploring, a repair node for a caught misconception, prune what proves unnecessary. A map that can't change is a wrong map.
**Tag a quiz with `nodeId` whenever it's about a map node** — that's what moves the map: its live position is simply the latest tagged node, so asking about the next node IS how the lesson gets there. A tag is a position, not a verdict — nodes are never marked finished, a node takes as many questions as it deserves, and you move on by asking about the next one. Untagged questions are ad-hoc checks — legal, but they move nothing.
**Every quiz carries `nodeId`** (required — a quiz without it is rejected): that's what moves the map. Its live position is simply the latest tagged node, so asking about the next node IS how the lesson gets there. A tag is a position, not a verdict — nodes are never marked finished. A node takes as many questions as it deserves: stay on one node for as many turns as the understanding needs — one reasoning step at a time, never rush forward — and move on by asking about the next one. Off-map checks (no plan node fits) use ask_user_question.
## Presenting the plan — always, before any teaching
src/agent/tools.ts
+10 -9
View File
@@ -201,8 +201,8 @@ function nodeIdProblem(ctx: Pick<PlanToolContext, "chatId" | "chatStore">, nodeI
return (
`nodeId "${nodeId}" matches no node in this session's plan map. ` +
(known.length > 0
? `Copy one of your plan_init slugs exactly (${known.slice(0, 5).join(", ")}${known.length > 5 ? ", …" : ""}), or omit nodeId for an ad-hoc check.`
: "No plan map exists yet — omit nodeId (ad-hoc checks only until you call plan_init).")
? `Copy one of your plan_init slugs exactly (${known.slice(0, 5).join(", ")}${known.length > 5 ? ", …" : ""}).`
: "No plan map exists yet — call plan_init before posing quizzes.")
);
}
return null;
@@ -212,7 +212,9 @@ function nodeIdProblem(ctx: Pick<PlanToolContext, "chatId" | "chatStore">, nodeI
* + the per-session plan-nodeId validation (both run in execute — the
* grading contract is untouched). Reads the LIVE chat record, so rewrites
* mid-turn are honored, and with no tree yet every nodeId is rejected.
* Omitting nodeId is legal — an ad-hoc check that moves nothing. */
* nodeId is REQUIRED (owner ruling 2026-09-26): the untagged ad-hoc quiz is
* gone — off-map checks route through ask_user_question, and the teach
* seam (first tagged quiz) is structurally guaranteed. */
export function teachQuizToolFor(ctx: Pick<PlanToolContext, "chatId" | "chatStore">): AgentTool<typeof quizInputObject> {
return {
name: "quiz",
@@ -222,8 +224,9 @@ export function teachQuizToolFor(ctx: Pick<PlanToolContext, "chatId" | "chatStor
"Unlike ask_user_question (which collects preferences/decisions with no right answer), quiz always has a " +
"correct answer supplied by you, marks the learner's selection right/wrong, reveals the correct answer, and " +
"can show an explanation. Use it to (1) confirm a taught node actually landed, or (2) probe understanding " +
"whenever you're unsure. Set `nodeId` whenever the question targets a map node — the lesson's position on the " +
"map derives from those tags; omit it only for quick ad-hoc checks. Options-only: single-select or multi-select, " +
"whenever you're unsure. Set `nodeId` (required) to the plan node this question is about — the lesson's " +
"position on the map derives from those tags; a node takes as many questions as it deserves, so don't rush " +
"forward. Off-map checks use ask_user_question. Options-only: single-select or multi-select, " +
"plus an automatic 'I don't know' choice so the learner can signal a genuine gap instead of guessing. " +
"An always-present optional note field lets the learner attach a free-text note to ANY answer; it reaches you " +
"only when non-empty. No free-text answers — for non-graded questions use ask_user_question instead. " +
@@ -233,10 +236,8 @@ export function teachQuizToolFor(ctx: Pick<PlanToolContext, "chatId" | "chatStor
executionMode: "sequential",
execute: async (_toolCallId, input) => {
const problems = [...quizInputProblems(input)];
if (typeof input.nodeId === "string" && input.nodeId.length > 0) {
const nodeIdProblemText = nodeIdProblem(ctx, input.nodeId);
if (nodeIdProblemText) problems.push(nodeIdProblemText);
}
const nodeIdProblemText = nodeIdProblem(ctx, input.nodeId);
if (nodeIdProblemText) problems.push(nodeIdProblemText);
if (problems.length > 0) throw new ToolInputError("quiz", problems);
return questionResult();
},
src/core/contracts/plan.ts
+3 -3
View File
@@ -55,9 +55,9 @@ export type PlanNodeStatus = "current" | "visited" | "upcoming";
// node is the teacher's judgment the previous one is finished, regardless of
// check outcomes (a failed check then a move-on is still visited); results
// feed nothing; there is NO fallback current (before the first tagged check
// nothing is current). Untagged checks are legal ad-hoc checks — they move
// nothing. NodeIds missing from the current tree are ignored (a rewrite
// may have dropped the node).
// nothing is current). Untagged checks no longer exist (nodeId is required
// on the teach quiz, owner ruling 2026-09-26). NodeIds missing from the
// current tree are ignored (a rewrite may have dropped the node).
/** Minimal structural message slice — anything with typed parts (mirrors
* phases.ts: pure, no runtime deps, safe on server, client, and in tests). */
src/core/phases.ts
+3 -2
View File
@@ -17,8 +17,9 @@
* turn, and the learner's next message is a natural reply (a change
* request, or the go-ahead). The live indicator LAGS by design (owner
* accepted): "teach" shows only once the first node's question is posed.
* A teacher that never tags a quiz would sit in "plan" visually —
* off-doctrine (tagging is prompt-mandated and nodeId is call-validated).
* A teacher that never tags a quiz can no longer exist — nodeId is
* schema-required at the teach quiz boundary (owner ruling 2026-09-26),
* so the seam signal is structural, not prompt-mandated.
*
* The former derived-occupancy seam (phase-of-tree via `currentNodeId`)
* is GONE, as is the verdict-marker fold: no backwards compatibility
src/core/question-core.ts
+9 -10
View File
@@ -200,16 +200,15 @@ export const quizInputBase = Type.Object({
// never touches).
export const quizInputObject = Type.Object({
...quizInputBase.properties,
nodeId: Type.Optional(
Type.String({
description:
"Optional plan-tree node id this question is about — must EXACTLY match one of this session's " +
"plan_init node slugs (validated at call time; a mismatched id is rejected back to you). " +
"Tag every question that targets a map node: the lesson's position on the map derives from these tags " +
"(the node the latest tagged question is about is where the map stands; earlier ones count as visited). " +
"Omit for quick ad-hoc checks that don't target a node.",
}),
),
nodeId: Type.String({
description:
"REQUIRED plan-tree node id this question is about — must EXACTLY match one of this session's " +
"plan_init node slugs (validated at call time; a mismatched id is rejected back to you). " +
"The lesson's position on the map derives from these tags (the node the latest tagged question is about is " +
"where the map stands; earlier ones count as visited). A node takes as many questions as it deserves: stay on " +
"one node for as many turns as the understanding needs — one reasoning step at a time, never rush forward. " +
"For off-map checks use ask_user_question.",
}),
});
/** The PROBE-side quiz declaration: the shared authoring contract, no