free-recall explain tool: the quizzing-mechanic switch (owner 2026-09-27)
Sycophancy mitigation as a structural commitment: the model writes its full correctAnswer (plus explanation, in that order) BEFORE the learner sees the question; the commitment rides the persisted call args, is revealed on submit, and grades his free text in the model's next reply — the client never grades free recall. - tools.explain setting (segmented 'Quizzing — Multiple choice | Free recall' in Agent settings' Tools group): free arms explain and drops quiz for BOTH agents (prober variant carries no nodeId, same pattern as the probe quiz); mechanics keys never pass filterEnabledTools — quiz stays unwritable (the 2026-09-19 wedge ruling stands) - doctrine lives pi-natively in the tool descriptions (owner ruling — no conditional system-prompt chunks): explain carries the commitment rule; the MC option-building craft moved out of the prompt into the quiz description; shared/20-question-tools.md slims to the quiz-agnostic core - display seams: parseExplainInput (LaTeX-debris restoration included), dock text-mode with an 'I don't know' row (kbd row 0), AnsweredCard reveal (The answer / Why, no verdict chips), [Explain answer] format - phases/plan seams recognize both quizzing tools' nodeId tags; explain answers deliberately feed no ledger evidence (no client grading to key on — a future ruling can parse the model's grade) 1049/1049 tests, typecheck/lint/build green.
This commit is contained in:
@@ -1,6 +1,6 @@
|
||||
# The probe
|
||||
|
||||
Your whole job is two measurements: **what he needs to learn** (his goal — `ask_user_question`) and **what he already knows** (a detailed map of his exact understanding — `quiz`). The whole lesson will be planned from your map — it must be precise.
|
||||
Your whole job is two measurements: **what he needs to learn** (his goal — `ask_user_question`) and **what he already knows** (a detailed map of his exact understanding — the quizzing tool: `quiz` or `explain`, whichever the learner’s mechanic armed). The whole lesson will be planned from your map — it must be precise.
|
||||
|
||||
Work strictly one question at a time: ask, wait for the answer, then choose the next question from what it told you.
|
||||
|
||||
@@ -16,7 +16,7 @@ Work strictly one question at a time: ask, wait for the answer, then choose the
|
||||
|
||||
From the goal, what will the lesson depend on? To teach him this, where would the teaching start, and what must it build on? Each dependency is a strand — bound by relevance, nothing more.
|
||||
|
||||
## 4. Binary-search every strand — `quiz`
|
||||
## 4. Binary-search every strand — the quizzing tool
|
||||
|
||||
The edge of a strand is found by bracketing: something he reliably gets **right** (a floor) and something he gets **wrong** or doesn't know (a ceiling). Narrow the bracket one question at a time:
|
||||
|
||||
|
||||
@@ -1,28 +1,14 @@
|
||||
## The question tools
|
||||
|
||||
Two instruments. `quiz` only for questions with a known correct answer, `ask_user_question` for everything else.
|
||||
Two instruments. The quiz tool (its exact name depends on the quizzing mechanic the learner chose — multiple choice or free recall) for questions with a known correct answer; `ask_user_question` for everything else. Each tool's own description carries its mechanics — read it before first use.
|
||||
|
||||
### `quiz`
|
||||
### Quiz checks — shared doctrine
|
||||
|
||||
Asks a question, grades instantly, shows ✓/✗ + the correct answer + explanation to both of you. Single- or multi-select.
|
||||
|
||||
- `correctAnswer` (required) is the option's `value` string — self-checking, never a position number. Multi-select: array of values, graded as an exact-set match.
|
||||
- `explanation` (required) says why the correct answer is correct. It comes BEFORE `correctAnswer` in the call by design: reason it out first, then commit the answer the reasoning supports.
|
||||
- An "I don't know" option is added automatically — provide only the real, gradable options (≥2), never your own opt-out. A dontKnow result is a genuine gap, not a wrong answer.
|
||||
- Answers may carry a free-text `note` from the always-present note field — read it; it says what he was thinking.
|
||||
- Options display shuffled by default; `shuffle: false` only when order is meaningful (e.g. an 'All/None of the above' that must stay last). `multiSelect: true` only when several options are correct.
|
||||
- `explanation` (required) says why the correct answer is correct. It comes BEFORE `correctAnswer` in the call by design: reason it out first, then commit to the answer the reasoning supports.
|
||||
- An "I don't know" choice is always available to the learner — never author your own opt-out. A dontKnow result is a genuine gap, not a wrong answer.
|
||||
- Answers may carry a free-text `note` — read it; it says what he was thinking.
|
||||
- ONE question per turn: ask, then end your turn — his answer determines the next.
|
||||
|
||||
### `ask_user_question`
|
||||
|
||||
Asks a single question and waits. He can always pick "Other" and type. If you recommend an option, list it first with "(Recommended)" appended. Multiple questions → multiple calls.
|
||||
|
||||
### Building quiz options
|
||||
|
||||
The giveaway is never the distractors' wrongness — it's asymmetry: the correct option carrying its reasoning while the others are bare, or key-term bolding that flags it. Build evenness in, don't audit it in afterwards:
|
||||
|
||||
1. Every option is a bare claim — zero "why" in any of them; all reasoning goes in `explanation`, shown only after he answers.
|
||||
2. Write the correct claim first, then mutate it into each distractor: what would someone holding a specific misconception — or confusing it with a specific neighbour — claim? Same skeleton, same grain, same register: parallelism by construction.
|
||||
3. Each distractor is a real error he might actually make (so his pick is diagnostic) yet unambiguously wrong.
|
||||
4. No asymmetric bolding — bold the parallel term in every option or in none.
|
||||
|
||||
If you can tell the right one cold without knowing the material, regenerate — don't patch.
|
||||
|
||||
@@ -26,7 +26,7 @@ You write the map with `plan_init` (a mermaid `graph TD` block). It is a **map,
|
||||
|
||||
The map is yours to rewrite at your own initiative: re-call `plan_init` whenever it's wrong — a detour node for a question worth exploring, a repair node for a caught misconception, prune what proves unnecessary. A map that can't change is a wrong map.
|
||||
|
||||
**Every quiz carries `nodeId`** (required — a quiz without it is rejected): that's what moves the map. Its live position is simply the latest tagged node, so asking about the next node IS how the lesson gets there. A tag is a position, not a verdict — nodes are never marked finished. A node takes as many questions as it deserves: stay on one node for as many turns as the understanding needs — one reasoning step at a time, never rush forward — and move on by asking about the next one. Off-map checks (no plan node fits) use ask_user_question.
|
||||
**Every quizzing call carries `nodeId`** (required — the quizzing tool is one of `quiz` or `explain` depending on the mechanic the learner chose; a call without it is rejected): that's what moves the map. Its live position is simply the latest tagged node, so asking about the next node IS how the lesson gets there. A tag is a position, not a verdict — nodes are never marked finished. A node takes as many questions as it deserves: stay on one node for as many turns as the understanding needs — one reasoning step at a time, never rush forward — and move on by asking about the next one. Off-map checks (no plan node fits) use ask_user_question.
|
||||
|
||||
## Presenting the plan — always, before any teaching
|
||||
|
||||
|
||||
@@ -5,5 +5,5 @@ Every check exists to answer one question: did the idea actually **take** — no
|
||||
- **Answerable by reasoning from the session.** Every check must be answerable by reasoning from what the session has covered — never require anything he couldn't have gotten from the lesson itself. A check he has no route to is not a check, it's a coin flip.
|
||||
- **Never just reciting your words back.** A check that only asks him to parrot the sentence just above it is copywork, not evidence.
|
||||
- **Prefer problems that apply the knowledge.** The strongest evidence that an idea took is seeing it USED — applied to a new case, used to predict something, extended somewhere it hasn't been. Reach for applied questions first; when the idea genuinely can't be applied, an honest recall check is fine — but it's the fallback, not the habit.
|
||||
- **Catch confusion early, and make it nameable.** Design the wrong options as real misconceptions — each the answer he'd give under a specific wrong model. Then a miss TELLS you which wrong model he holds; diagnose and repair before building anything on top of it.
|
||||
- **Catch confusion early, and make it nameable.** Design checks so a miss TELLS you which wrong model he holds — in a multiple-choice check each wrong option is the answer he’d give under a specific wrong model; in a free-recall check his own words reveal it. Diagnose and repair before building anything on top of it.
|
||||
- **His answer is signal, not verdict.** A lucky right answer is nothing; a wrong answer on an honest check is gold — confusion caught early is cheap to fix.
|
||||
@@ -203,7 +203,8 @@ function baseAgentOptions(kind: AgentKind, opts: AgentBuildOptions, tools: Agent
|
||||
}
|
||||
|
||||
/**
|
||||
* The PROBER (agent A): quiz (no nodeId) + ask + probe_done, lean on
|
||||
* The PROBER (agent A): the quizzing tool (quiz, or explain under free
|
||||
* recall — no nodeId either way) + ask + probe_done, lean on
|
||||
* probing doctrine. Its whole job is bracketing the learner's edge and
|
||||
* concluding with the assessment (probe_done) — which also queues the
|
||||
* handoff wake (see tools/probe.ts) and is the transcript seam the
|
||||
|
||||
+114
-6
@@ -4,6 +4,9 @@ import { questionResult, serverExecute, ToolInputError } from "./tool-execute.ts
|
||||
import {
|
||||
askInputSchema,
|
||||
askOptionProblems,
|
||||
explainInputBase,
|
||||
explainInputObject,
|
||||
explainInputProblems,
|
||||
parseOptionsIfString,
|
||||
probeQuizInputSchema,
|
||||
quizInputObject,
|
||||
@@ -92,6 +95,32 @@ export const askUserQuestionTool: AgentTool<typeof askInputSchema> = {
|
||||
|
||||
// ─── the prober's question tools (no nodeId — no map exists yet) ─────────────
|
||||
|
||||
/** The option-building craft (owner 2026-09-27: moved OUT of the system
|
||||
* prompt into the tool description — pi-native doctrine travels with
|
||||
* the armed tool). Shared verbatim by both quiz variants. */
|
||||
const QUIZ_OPTION_CRAFT =
|
||||
"Building options — the giveaway is never the distractors' wrongness, it's asymmetry: the correct option " +
|
||||
"carrying its reasoning while the others are bare, or key-term bolding that flags it. Build evenness in: " +
|
||||
"(1) every option is a bare claim — zero 'why' in any of them; all reasoning goes in explanation, shown only " +
|
||||
"after he answers; (2) write the correct claim first, then mutate it into each distractor — what would someone " +
|
||||
"holding a specific misconception claim? Same skeleton, same grain, same register; (3) each distractor is a real " +
|
||||
"error he might actually make (his pick is diagnostic) yet unambiguously wrong; (4) no asymmetric bolding. If " +
|
||||
"you can tell the right one cold without knowing the material, regenerate — don't patch.";
|
||||
|
||||
/** The explain doctrine (owner 2026-09-27): free-recall quizzing — the
|
||||
* sycophancy mitigation is the commitment. Carried by the DESCRIPTION
|
||||
* (pi-native ruling), not the system prompt. Shared by both variants;
|
||||
* the teach variant appends the nodeId tagging paragraph. */
|
||||
const EXPLAIN_DOCTRINE =
|
||||
"Ask the learner to EXPLAIN something in his own words — a graded check with NO options: he types a free-text " +
|
||||
"answer. Unlike ask_user_question (no right answer), explain always has one. THE COMMITMENT RULE: write " +
|
||||
"explanation (your reasoning) first, then correctAnswer — the FULL correct answer, committed before he ever sees " +
|
||||
"the question. When his answer returns, grade it strictly against your committed answer in your reply: never " +
|
||||
"revise the commitment after seeing his words, and never mark a wrong answer right to be kind — a near-miss is " +
|
||||
"partial understanding; name exactly what he got and what he missed. He sees the committed answer and the " +
|
||||
"explanation on submit. An 'I don't know' choice is always available to him — a genuine gap, not a wrong answer. " +
|
||||
"ONE question per turn: call a single explain, then end your turn — his answer determines the next.";
|
||||
|
||||
export const probeQuizTool: AgentTool<typeof probeQuizInputSchema> = {
|
||||
name: "quiz",
|
||||
label: "quiz",
|
||||
@@ -105,7 +134,7 @@ export const probeQuizTool: AgentTool<typeof probeQuizInputSchema> = {
|
||||
"signal a genuine gap instead of guessing. An always-present optional note field lets the learner attach a " +
|
||||
"free-text note to ANY answer; it reaches you only when non-empty. No free-text answers — for non-graded " +
|
||||
"questions use ask_user_question instead. ONE question per turn: call a single quiz, then end your turn — " +
|
||||
"his answer determines the next question to ask.",
|
||||
"his answer determines the next question to ask. " + QUIZ_OPTION_CRAFT,
|
||||
parameters: probeQuizInputSchema,
|
||||
prepareArguments: prepareQuestionArguments,
|
||||
executionMode: "sequential",
|
||||
@@ -116,6 +145,48 @@ export const probeQuizTool: AgentTool<typeof probeQuizInputSchema> = {
|
||||
},
|
||||
};
|
||||
|
||||
/** The teacher's free-recall variant (nodeId-validated, same seam as the
|
||||
* teach quiz — phases/plan recognize BOTH tool names). */
|
||||
export function teachExplainToolFor(
|
||||
ctx: Pick<PlanToolContext, "chatId" | "chatStore">,
|
||||
): AgentTool<typeof explainInputObject> {
|
||||
return {
|
||||
name: "explain",
|
||||
label: "explain",
|
||||
description:
|
||||
EXPLAIN_DOCTRINE +
|
||||
" Set `nodeId` (required) to the plan node this question is about — the lesson's position on the map derives " +
|
||||
"from those tags; a node takes as many questions as it deserves, so don't rush forward. Off-map checks use " +
|
||||
"ask_user_question.",
|
||||
parameters: explainInputObject,
|
||||
prepareArguments: prepareQuestionArguments,
|
||||
executionMode: "sequential",
|
||||
execute: async (_toolCallId, input) => {
|
||||
const problems = [...explainInputProblems(input)];
|
||||
const nodeIdProblemText = nodeIdProblem(ctx, input.nodeId);
|
||||
if (nodeIdProblemText) problems.push(nodeIdProblemText);
|
||||
if (problems.length > 0) throw new ToolInputError("explain", problems);
|
||||
return questionResult();
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
/** The prober's free-recall variant (no nodeId — same pattern as the
|
||||
* probe quiz). */
|
||||
export const probeExplainTool: AgentTool<typeof explainInputBase> = {
|
||||
name: "explain",
|
||||
label: "explain",
|
||||
description: EXPLAIN_DOCTRINE,
|
||||
parameters: explainInputBase,
|
||||
prepareArguments: prepareQuestionArguments,
|
||||
executionMode: "sequential",
|
||||
execute: async (_toolCallId, input) => {
|
||||
const problems = explainInputProblems(input);
|
||||
if (problems.length > 0) throw new ToolInputError("explain", problems);
|
||||
return questionResult();
|
||||
},
|
||||
};
|
||||
|
||||
/** The prober's registry context: the probe's own needs (probe_done —
|
||||
* ProbeDoneContext) PLUS the job-dispatch wiring, since the owner ruling
|
||||
* (2026-09-20) puts research + pdf in the probe phase's registry too. */
|
||||
@@ -156,10 +227,20 @@ export function createProberTools(
|
||||
spawn: ctx.spawn,
|
||||
searchAvailable: ctx.searchAvailable,
|
||||
});
|
||||
// Quizzing mechanics (owner 2026-09-27): free-recall swaps the probe
|
||||
// quiz for the probe explain — one quizzing tool at a time, whichever
|
||||
// the settings arm. ask stays always-on.
|
||||
const mechanics = quizMechanics(enabled);
|
||||
// The armed quizzing tool, widened to the registry's heterogeneous
|
||||
// AgentTool record BEFORE the filter (a conditional spread of two
|
||||
// differently-typed literals makes the whole object's type a union —
|
||||
// unusable as Record<string, AgentTool<any>> downstream).
|
||||
const quizzing: Record<string, AgentTool<any>> =
|
||||
mechanics === "free" ? { explain: probeExplainTool } : { quiz: probeQuizTool };
|
||||
return {
|
||||
...filterEnabledTools(
|
||||
{
|
||||
quiz: probeQuizTool,
|
||||
...quizzing,
|
||||
ask_user_question: askUserQuestionTool,
|
||||
// The learner-filesystem set is always-on (user-decided 2026-09,
|
||||
// sources v1): the prober inspects and LOCATES learner-referenced
|
||||
@@ -184,7 +265,7 @@ export function createProberTools(
|
||||
: {}),
|
||||
...done,
|
||||
},
|
||||
enabled,
|
||||
withoutMechanics(enabled),
|
||||
),
|
||||
...done,
|
||||
};
|
||||
@@ -230,7 +311,8 @@ export function teachQuizToolFor(ctx: Pick<PlanToolContext, "chatId" | "chatStor
|
||||
"plus an automatic 'I don't know' choice so the learner can signal a genuine gap instead of guessing. " +
|
||||
"An always-present optional note field lets the learner attach a free-text note to ANY answer; it reaches you " +
|
||||
"only when non-empty. No free-text answers — for non-graded questions use ask_user_question instead. " +
|
||||
"ONE question per turn: call a single quiz, then end your turn — his answer determines the next.",
|
||||
"ONE question per turn: call a single quiz, then end your turn — his answer determines the next. " +
|
||||
QUIZ_OPTION_CRAFT,
|
||||
parameters: quizInputObject,
|
||||
prepareArguments: prepareQuestionArguments,
|
||||
executionMode: "sequential",
|
||||
@@ -269,13 +351,39 @@ export interface TeacherToolContext extends PlanToolContext {
|
||||
* future toggleables (question-tool variants, …) need no change here.
|
||||
* Absent/undefined = enabled — tests and old records see today's set.
|
||||
*/
|
||||
/** The quizzing mechanics (owner 2026-09-27): `tools.explain === true` in
|
||||
* the resolved settings ⇒ free-recall checks (explain armed, quiz dropped)
|
||||
* — anything else ⇒ multiple choice, exactly today's set. Mechanics-
|
||||
* managed, NOT individually toggleable: quiz/explain never pass through
|
||||
* filterEnabledTools (the 2026-09-19 wedge ruling stands — a quiz-less
|
||||
* prober collapses its assessment flow). */
|
||||
export function quizMechanics(enabled?: EnabledTools): "choice" | "free" {
|
||||
return enabled?.explain === true ? "free" : "choice";
|
||||
}
|
||||
|
||||
/** Strip the mechanics keys so filterEnabledTools can never drop the
|
||||
* quizzing tools (the derivation above is their only switch). */
|
||||
function withoutMechanics(enabled?: EnabledTools): EnabledTools | undefined {
|
||||
if (!enabled) return undefined;
|
||||
const { explain: _e, quiz: _q, ...rest } = enabled;
|
||||
return rest;
|
||||
}
|
||||
|
||||
export function createTeacherTools(
|
||||
ctx: TeacherToolContext,
|
||||
enabled?: EnabledTools,
|
||||
): Record<string, AgentTool<any>> {
|
||||
// Quizzing mechanics (owner 2026-09-27): free-recall swaps the teach
|
||||
// quiz for the teach explain — one quizzing tool at a time, whichever
|
||||
// the settings arm. ask stays always-on.
|
||||
const mechanics = quizMechanics(enabled);
|
||||
// Same widening as the prober's (see above) — the conditional spread of
|
||||
// two differently-typed literals collapses into an unusable union type.
|
||||
const quizzing: Record<string, AgentTool<any>> =
|
||||
mechanics === "free" ? { explain: teachExplainToolFor(ctx) } : { quiz: teachQuizToolFor(ctx) };
|
||||
return filterEnabledTools(
|
||||
{
|
||||
quiz: teachQuizToolFor(ctx),
|
||||
...quizzing,
|
||||
ask_user_question: askUserQuestionTool,
|
||||
// The learner-filesystem set is always-on (user-decided 2026-09,
|
||||
// sources v1): the teacher locates and reads learner-referenced
|
||||
@@ -292,6 +400,6 @@ export function createTeacherTools(
|
||||
// reaches jobTools' research gate.
|
||||
...jobTools({ chatId: ctx.chatId, jobStore: ctx.jobStore, spawn: ctx.spawn, searchAvailable: ctx.searchAvailable }),
|
||||
},
|
||||
enabled,
|
||||
withoutMechanics(enabled),
|
||||
);
|
||||
}
|
||||
|
||||
@@ -85,7 +85,9 @@ export function derivePlanStatuses(tree: PlanTree, messages: readonly PartsCarri
|
||||
// duplicates collapsed (several checks on one node = one stay on it)
|
||||
for (const m of messages) {
|
||||
for (const part of m.parts) {
|
||||
if (part.type !== "tool-quiz") continue;
|
||||
// Either quizzing tool (quiz/explain) tags position — the mechanic
|
||||
// is a settings switch; the fold recognizes both (owner 2026-09-27).
|
||||
if (part.type !== "tool-quiz" && part.type !== "tool-explain") continue;
|
||||
const nodeId = (part.input as { nodeId?: unknown } | undefined)?.nodeId;
|
||||
if (typeof nodeId !== "string" || !ids.has(nodeId)) continue; // untagged/unknown — moves nothing
|
||||
if (runs.at(-1) !== nodeId) runs.push(nodeId);
|
||||
@@ -130,7 +132,7 @@ export function derivePlanStatusesLh(tree: PlanTree, messages: readonly LhMessag
|
||||
const content = (m as { content?: unknown }).content;
|
||||
if (!Array.isArray(content)) continue;
|
||||
for (const block of content as Array<{ type?: string; name?: string; arguments?: unknown }>) {
|
||||
if (block.type !== "toolCall" || block.name !== "quiz") continue;
|
||||
if (block.type !== "toolCall" || (block.name !== "quiz" && block.name !== "explain")) continue;
|
||||
const nodeId = (block.arguments as { nodeId?: unknown } | undefined)?.nodeId;
|
||||
if (typeof nodeId !== "string" || !ids.has(nodeId)) continue;
|
||||
if (runs.at(-1) !== nodeId) runs.push(nodeId);
|
||||
|
||||
+8
-1
@@ -39,7 +39,7 @@ export interface PendingToolCall {
|
||||
}
|
||||
|
||||
/** Tools that render in the question dock and gate the composer. */
|
||||
export const QUESTION_TOOLS = ["quiz", "ask_user_question"] as const;
|
||||
export const QUESTION_TOOLS = ["quiz", "ask_user_question", "explain"] as const;
|
||||
|
||||
/** Tools fulfilled by background sub-agent jobs (W2/W6). */
|
||||
export const JOB_TOOLS = ["research", "demo", "visual"] as const;
|
||||
@@ -62,6 +62,13 @@ export const JOB_TOOLS = ["research", "demo", "visual"] as const;
|
||||
* 2026-09). */
|
||||
export const TOGGLEABLE_TOOLS = ["research", "demo", "visual", "pdf"] as const;
|
||||
|
||||
/** The keys the settings WRITE side accepts in agent.tools: the plain
|
||||
* toggleables PLUS the quizzing-mechanics switch (`explain: true` arms
|
||||
* free-recall quizzing and drops the MC quiz — see quizMechanics in
|
||||
* agent/tools.ts; owner 2026-09-27). `quiz` itself stays unwritable:
|
||||
* mechanics-managed, never individually toggleable. */
|
||||
export const WRITABLE_TOOL_KEYS = [...TOGGLEABLE_TOOLS, "explain"] as const;
|
||||
|
||||
/** The resolved settings → factory seam shape (Platform wires chat-handler
|
||||
* to pass this through from the settings record; storage and defaults are
|
||||
* Platform's lane). JSON-native: a partial record keyed by tool name.
|
||||
|
||||
+8
-2
@@ -83,7 +83,13 @@ function advance(
|
||||
if (phase === "probe" && part.type === "tool-probe_done" && part.state === "output-available") {
|
||||
phase = "plan";
|
||||
}
|
||||
if (phase === "plan" && part.type === "tool-quiz" && taggedNodeId(part.input)) {
|
||||
// plan → teach via the first nodeId-tagged quizzing CALL — quiz OR
|
||||
// explain, whichever mechanic the settings armed (both carry nodeId).
|
||||
if (
|
||||
phase === "plan" &&
|
||||
(part.type === "tool-quiz" || part.type === "tool-explain") &&
|
||||
taggedNodeId(part.input)
|
||||
) {
|
||||
phase = "teach";
|
||||
}
|
||||
}
|
||||
@@ -136,7 +142,7 @@ export function phaseAt3Lh(messages: readonly LhMessage[], upToIndex?: number):
|
||||
for (const block of content) {
|
||||
if (
|
||||
block.type === "toolCall" &&
|
||||
block.name === "quiz" &&
|
||||
(block.name === "quiz" || block.name === "explain") &&
|
||||
taggedNodeId(block.arguments)
|
||||
) {
|
||||
phase = "teach";
|
||||
|
||||
@@ -217,6 +217,68 @@ export const quizInputObject = Type.Object({
|
||||
* it ran at the call boundary, where quizInputProblems runs now). */
|
||||
export const probeQuizInputSchema = quizInputBase;
|
||||
|
||||
// ─── the explain tool (free-recall quizzing, owner 2026-09-27) ─────────────
|
||||
//
|
||||
// Quiz mechanics WITHOUT options: the learner answers in his own words.
|
||||
// The SYCOPHANCY MITIGATION is structural — the model commits to the full
|
||||
// correct answer BEFORE posing (the commitment rides the persisted call
|
||||
// args, is revealed to the learner on submit, and is restated in the answer
|
||||
// result the model grades against; the client never grades free text).
|
||||
// Mechanics doctrine lives in the TOOL DESCRIPTIONS (owner ruling
|
||||
// 2026-09-27: pi-native — doctrine travels with the armed tool), not in
|
||||
// the system prompt.
|
||||
|
||||
export const explainInputBase = Type.Object({
|
||||
question: Type.String({
|
||||
description:
|
||||
"The single question to ask. ONE question per turn: ask, then end your turn — the answer determines the next.",
|
||||
}),
|
||||
explanation: Type.String({
|
||||
description:
|
||||
"REQUIRED. Explanation revealed AFTER the learner answers (shown however he did). It is written BEFORE " +
|
||||
"correctAnswer by design: reason it out first, then commit to the answer the reasoning supports.",
|
||||
}),
|
||||
correctAnswer: Type.String({
|
||||
description:
|
||||
"REQUIRED. The full correct answer in your own words — COMMITTED before the learner ever sees the question. " +
|
||||
"The commitment is immutable: when his answer returns, grade it strictly against this text — never revise the " +
|
||||
"commitment after seeing his words, and never mark a wrong answer right to be kind. Both this and the explanation " +
|
||||
"are shown to him on submit; your next reply grades the nuance of what he wrote.",
|
||||
}),
|
||||
details: Type.Optional(
|
||||
Type.String({ description: "Optional extra context or instructions shown under the question." }),
|
||||
),
|
||||
});
|
||||
|
||||
/** The TEACH-side explain object: base + the plan-position field (same
|
||||
* semantics as the quiz's nodeId — the seam recognizes BOTH tools). */
|
||||
export const explainInputObject = Type.Object({
|
||||
...explainInputBase.properties,
|
||||
nodeId: Type.String({
|
||||
description:
|
||||
"REQUIRED plan-tree node id this question is about — must EXACTLY match one of this session's " +
|
||||
"plan_init node slugs (validated at call time; a mismatched id is rejected back to you). " +
|
||||
"The lesson's position on the map derives from these tags (the node the latest tagged question is about is " +
|
||||
"where the map stands; earlier ones count as visited). A node takes as many questions as it deserves: stay on " +
|
||||
"one node for as many turns as the understanding needs — one reasoning step at a time, never rush forward. " +
|
||||
"For off-map checks use ask_user_question.",
|
||||
}),
|
||||
});
|
||||
|
||||
/** The explain authoring contract: the commitment must exist. Empty
|
||||
* result = an explain the dock can arm and the model can grade against. */
|
||||
export function explainInputProblems(input: { correctAnswer: string }): string[] {
|
||||
if (input.correctAnswer.trim().length === 0) {
|
||||
return ["correctAnswer is required — commit to the full correct answer BEFORE posing the question."];
|
||||
}
|
||||
return [];
|
||||
}
|
||||
|
||||
/** The tool names that carry the quizzing mechanic — the teach seam
|
||||
* (phases), the plan-position fold, and the pending gate all recognize
|
||||
* whichever is armed. */
|
||||
export const QUIZ_TOOL_NAMES = ["quiz", "explain"] as const;
|
||||
|
||||
/** The TEACHER's quiz declaration (base + the position field). */
|
||||
export const quizInputSchema = quizInputObject;
|
||||
|
||||
@@ -249,6 +311,8 @@ export const askInputSchema = Type.Object({
|
||||
|
||||
export type QuizToolInput = Static<typeof quizInputObject>;
|
||||
export type AskToolInput = Static<typeof askInputSchema>;
|
||||
export type ExplainToolInput = Static<typeof explainInputObject>;
|
||||
export type ProbeExplainToolInput = Static<typeof explainInputBase>;
|
||||
|
||||
|
||||
/** A real selected answer, in the order the options were displayed. */
|
||||
@@ -258,7 +322,7 @@ export interface OptionAnswer {
|
||||
index: number; // 1-based display index
|
||||
}
|
||||
|
||||
export type QuestionKind = "quiz" | "ask";
|
||||
export type QuestionKind = "quiz" | "ask" | "explain";
|
||||
export type SelectMode = "single-select" | "multi-select";
|
||||
export type QuestionMode = SelectMode | "text";
|
||||
|
||||
@@ -272,6 +336,8 @@ export interface QuestionData {
|
||||
options: QuizOption[]; // display order (post-shuffle)
|
||||
// quiz only. NEVER send to the client before answering:
|
||||
correctIndices?: number[];
|
||||
// explain only. NEVER send to the client before answering:
|
||||
committedAnswer?: string;
|
||||
explanation?: string;
|
||||
createdAt: string;
|
||||
}
|
||||
@@ -285,11 +351,12 @@ export interface QuestionAnswerData {
|
||||
dontKnow: boolean;
|
||||
answers: OptionAnswer[];
|
||||
other?: string; // ask: "Other" free text
|
||||
text?: string; // ask: text-mode answer
|
||||
text?: string; // ask/explain: text-mode answer
|
||||
note?: string;
|
||||
correct?: boolean; // quiz only
|
||||
correctIndices?: number[]; // quiz only (rendering)
|
||||
explanation?: string; // quiz only (rendering)
|
||||
committed?: string; // explain only (rendering): the model's committed answer
|
||||
explanation?: string; // quiz/explain only (rendering)
|
||||
dismissed?: boolean;
|
||||
answeredAt: string;
|
||||
}
|
||||
@@ -526,12 +593,18 @@ export function buildAnswer(qd: QuestionData, req: AnswerRequest): QuestionAnswe
|
||||
base.explanation = qd.explanation;
|
||||
base.correct = base.dontKnow ? false : isCorrect(base.answers.map((a) => a.index), qd.correctIndices ?? []);
|
||||
}
|
||||
if (qd.kind === "explain") {
|
||||
// No client grading — the model grades in its reply, against the
|
||||
// commitment. Snapshots ride along for the reveal + rendering.
|
||||
base.committed = qd.committedAnswer;
|
||||
base.explanation = qd.explanation;
|
||||
}
|
||||
return base;
|
||||
}
|
||||
|
||||
/** The user message that carries the answer into the agent's context. */
|
||||
export function formatAnswerMessage(ad: QuestionAnswerData): string {
|
||||
const label = ad.kind === "quiz" ? "Quiz answer" : "Answer to your question";
|
||||
const label = ad.kind === "quiz" ? "Quiz answer" : ad.kind === "explain" ? "Explain answer" : "Answer to your question";
|
||||
const head = `[${label}]`;
|
||||
const lines: string[] = [head];
|
||||
|
||||
@@ -553,6 +626,18 @@ export function formatAnswerMessage(ad: QuestionAnswerData): string {
|
||||
lines.push(`Result: ${verdict}`);
|
||||
}
|
||||
lines.push(`Correct: ${correctStr}`);
|
||||
} else if (ad.kind === "explain") {
|
||||
if (ad.dontKnow) {
|
||||
lines.push(
|
||||
`The learner selected "I don't know" — they did NOT attempt an answer (a genuine knowledge gap, not a wrong guess).`,
|
||||
);
|
||||
} else {
|
||||
lines.push(`The learner answered: ${ad.text ?? "(no answer)"}`);
|
||||
}
|
||||
if (ad.committed) {
|
||||
lines.push(`Your committed correct answer (immutable — grade against it, never revise it): ${ad.committed}`);
|
||||
}
|
||||
lines.push("Grade his answer strictly against your committed answer in your reply: name what he got and what he missed.");
|
||||
} else {
|
||||
if (ad.text) {
|
||||
lines.push(`The learner answered: ${ad.text}`);
|
||||
|
||||
+12
-1
@@ -29,7 +29,7 @@
|
||||
|
||||
import { Value } from "typebox/value";
|
||||
import type { TSchema } from "typebox";
|
||||
import { quizInputBase, askInputSchema, askOptionProblems, quizInputProblems, type AskToolInput, type QuizToolInput } from "./question-core.ts";
|
||||
import { quizInputBase, askInputSchema, askOptionProblems, quizInputProblems, explainInputBase, explainInputProblems, type AskToolInput, type QuizToolInput, type ExplainToolInput } from "./question-core.ts";
|
||||
import { restoreMathEscapes } from "./math-delims.ts";
|
||||
import { probeDoneInputSchema, type ProbeDoneInput } from "./contracts/probe.ts";
|
||||
|
||||
@@ -103,6 +103,17 @@ export function parseAskInput(input: unknown): AskToolInput | undefined {
|
||||
return cleaned;
|
||||
}
|
||||
|
||||
/** Explain parse for DISPLAY: the shared authoring base (the teacher's
|
||||
* nodeId is an unknown key Value.Clean strips, same as the quiz seam) —
|
||||
* both explain variants share the same fields, and the commitment
|
||||
* refinement (explainInputProblems) mirrors the boundary chain. */
|
||||
export function parseExplainInput(input: unknown): ExplainToolInput | undefined {
|
||||
const cleaned = boundaryParse<ExplainToolInput>(explainInputBase, restoreMathEscapesDeep(input));
|
||||
if (!cleaned) return undefined;
|
||||
if (explainInputProblems(cleaned).length > 0) return undefined;
|
||||
return cleaned;
|
||||
}
|
||||
|
||||
/** probe_done parse — the core-owned schema the tool itself declares. */
|
||||
export function parseProbeDoneInput(input: unknown): ProbeDoneInput | undefined {
|
||||
return boundaryParse<ProbeDoneInput>(probeDoneInputSchema, input);
|
||||
|
||||
@@ -23,7 +23,7 @@ import type {
|
||||
RoleModelOverride,
|
||||
} from "../core/settings.ts";
|
||||
import { BACKGROUND_MODEL_ROLES, EXTRACTION_CHUNK_TOKENS_FLOOR, MODEL_SLOTS } from "../core/settings.ts";
|
||||
import { TOGGLEABLE_TOOLS } from "../core/pending.ts";
|
||||
import { WRITABLE_TOOL_KEYS } from "../core/pending.ts";
|
||||
|
||||
/** Parsed-or-rejected — `ok: false` carries the 400's error message. */
|
||||
export type SettingsBodyResult<T> =
|
||||
@@ -157,10 +157,11 @@ export function parseAgentSettingsBody(body: unknown): SettingsBodyResult<AgentS
|
||||
if (tools.map) {
|
||||
// Unknown tool keys are rejected at WRITE time (user decision
|
||||
// 2026-09-19): the delta is inert while unread, but once the bridge
|
||||
// honors it a persisted `quiz: false` would genuinely drop the quiz —
|
||||
// and the definitive list is the adjudicated job-tools-only set.
|
||||
// The read-side sanitizer stays lenient for forward-compat.
|
||||
const supported = TOGGLEABLE_TOOLS as readonly string[];
|
||||
// honors it a persisted bogus key would drift. The writable set is
|
||||
// the job tools + the quizzing-mechanics switch (explain, owner
|
||||
// 2026-09-27). The read-side sanitizer stays lenient for
|
||||
// forward-compat.
|
||||
const supported = WRITABLE_TOOL_KEYS as readonly string[];
|
||||
for (const key of Object.keys(tools.map)) {
|
||||
if (!supported.includes(key)) {
|
||||
return { ok: false, error: `unknown tool "${key}" (supported: ${supported.join(", ")})` };
|
||||
|
||||
Reference in New Issue
Block a user