feat(T2a): tool-toggle seam + phase-based question principles

Part 1 — tool availability (the settings seam, machinery generic):
- core/pending.ts: TOGGLEABLE_TOOLS (today: research, demo, visual —
  the model-facing tool names), EnabledTools = the JSON-native seam
  shape Partial<Record<string, boolean>> (absent key = enabled, so old
  records behave exactly as today), filterEnabledTools shared helper.
  quiz/ask_user_question are always-on FOR NOW (user-decided); the
  filtering is generic so future toggleables (question-tool variants,
  the read tool, …) need no factory change.
- createProberTools/createTeacherTools take enabled?: EnabledTools; the
  composition root (chat-handler, Platform's lane this wave) passes the
  resolved settings through. probe_done is re-added after the filter —
  always-on by construction (it ends the phase; a prober that can never
  conclude is a lockout).
- Prompts stay STATIC under toggles this wave (user-decided); standing
  principle recorded: tools ship self-contained usage guidance in their
  descriptions; system prompts reference only always-on tools.

Part 2 — phase-based question-writing principles (user-locked wording,
drafted with the user over three iterations):
- NEW teacher/21-question-principles.md (composes after 20-teaching):
  answerable-by-reasoning-from-the-session (the fairness rule that
  filters unanswerable questions without forbidding visible answers),
  never-just-reciting-your-words-back, PREFER problems that apply the
  knowledge (preference, not hard rule — honest recall is the fallback,
  not the habit), catch-confusion-early-and-nameable (wrong options as
  real misconceptions), answer-is-signal-not-verdict.
- prober/10-probe-doctrine.md: 'no rush' retired; quick-measurement
  directive in — every question moves an edge (bracket, narrow, or
  characterize); a bracketed strand is done.

Tests: TOGGLEABLE_TOOLS pin, prober/teacher filtering (absent = full,
false drops exactly that tool, probe_done survives), prompt pins (the
principles compose verbatim into the teacher prompt, the no-rush
framing is gone from the prober). 547/547, typecheck/lint/build green.
This commit is contained in:
amosblomqvist
2026-09-19 14:06:07 +03:00
parent 487f5d4cea
commit cab1632c74
6 changed files with 158 additions and 15 deletions
@@ -4,7 +4,7 @@ Probing is not a warm-up. It is the whole job of this phase, and it is the highe
## 1a. His current level — use `quiz`. This is a mapping job, not a spot-check.
Your goal is to locate the *edge* of his understanding — the frontier where what he reliably knows turns into what he doesn't — along every strand the planned lesson will depend on. Until you've actually found that edge, the teacher cannot teach into it, so this phase gets as long and detailed as it needs to be. There is no rush.
Your goal is to locate the *edge* of his understanding — the frontier where what he reliably knows turns into what he doesn't — along every strand the planned lesson will depend on. Until you've actually found that edge, the teacher cannot teach into it. And measurement should be efficient: every question you ask should move an edge somewhere — bracket a strand, narrow a bracket, or characterize a miss. No filler questions, no re-asking what his last answer already showed you. When a strand is bracketed, it's done: move to the next.
**The edge is only located when it's bracketed.** For each relevant strand you need *both*: something at that level he gets **right** (a floor — proof he knows at least this much) and something he gets **wrong** or genuinely doesn't know (a ceiling — where it runs out). The edge sits between them. One side alone tells you almost nothing.
@@ -0,0 +1,9 @@
# Question principles — what a teach-phase check must be
Every check exists to answer one question: did the idea actually **take** — not whether he can read it back. The principles:
- **Answerable by reasoning from the session.** Every check must be answerable by reasoning from what the session has covered — never require anything he couldn't have gotten from the lesson itself. A check he has no route to is not a check, it's a coin flip.
- **Never just reciting your words back.** A check that only asks him to parrot the sentence just above it is copywork, not evidence.
- **Prefer problems that apply the knowledge.** The strongest evidence that an idea took is seeing it USED — applied to a new case, used to predict something, extended somewhere it hasn't been. Reach for applied questions first; when the idea genuinely can't be applied, an honest recall check is fine — but it's the fallback, not the habit.
- **Catch confusion early, and make it nameable.** Design the wrong options as real misconceptions — each the answer he'd give under a specific wrong model. Then a miss TELLS you which wrong model he holds; diagnose and repair before building anything on top of it.
- **His answer is signal, not verdict.** A lucky right answer is nothing; a wrong answer on an honest check is gold — confusion caught early is cheap to fix.
+42 -12
View File
@@ -5,6 +5,7 @@ import {
quizInputObject,
quizInputProblems,
} from "../core/question-core.ts";
import { filterEnabledTools, type EnabledTools } from "../core/pending.ts";
import type { JobRecord } from "../core/contracts.ts";
import type { JobStore } from "../core/ports.ts";
import { jobTools, type DemoInput, type ResearchInput, type VisualInput } from "./tools/jobs.ts";
@@ -62,12 +63,28 @@ export const probeQuizTool: LhTool<any, any> = {
schema: probeQuizInputSchema,
};
/** The prober's tool set: probe quiz (no nodeId) + ask + probe_done. */
export function createProberTools(ctx: ProbeDoneContext): Record<string, LhTool<any, any>> {
/** The prober's tool set: probe quiz (no nodeId) + ask + probe_done.
* `enabled` is the resolved settings seam (core/pending.ts): any tool
* explicitly false is dropped from the registry. probe_done stays
* always-on regardless — it is how the phase ENDS; a prober that can
* never conclude is a lockout. */
export function createProberTools(
ctx: ProbeDoneContext,
enabled?: EnabledTools,
): Record<string, LhTool<any, any>> {
// probe_done is re-added AFTER the filter, always-on by construction — it
// is how the phase ENDS; a prober that can never conclude is a lockout.
const done = probeDoneTool(ctx);
return {
quiz: probeQuizTool,
ask_user_question: askUserQuestionTool,
...probeDoneTool(ctx),
...filterEnabledTools(
{
quiz: probeQuizTool,
ask_user_question: askUserQuestionTool,
...done,
},
enabled,
),
...done,
};
}
@@ -131,12 +148,25 @@ export interface TeacherToolContext extends PlanToolContext {
/**
* Per-request teacher tool registry: teach questions (validated nodeId) +
* plan tools + job tools, all closing over the ctx.
*
* `enabled` is the resolved settings seam (core/pending.ts EnabledTools —
* Platform's chat-handler wiring passes it through from settings). Today
* only the sub-agent tools (research/demo/visual) are user-toggleable, but
* the filtering is GENERIC: any name explicitly false is dropped, so
* future toggleables (question-tool variants, …) need no change here.
* Absent/undefined = enabled — tests and old records see today's set.
*/
export function createTeacherTools(ctx: TeacherToolContext): Record<string, LhTool<any, any>> {
return {
quiz: teachQuizToolFor(ctx),
ask_user_question: askUserQuestionTool,
...planTools(ctx),
...jobTools({ chatId: ctx.chatId, jobStore: ctx.jobStore, spawn: ctx.spawn }),
};
export function createTeacherTools(
ctx: TeacherToolContext,
enabled?: EnabledTools,
): Record<string, LhTool<any, any>> {
return filterEnabledTools(
{
quiz: teachQuizToolFor(ctx),
ask_user_question: askUserQuestionTool,
...planTools(ctx),
...jobTools({ chatId: ctx.chatId, jobStore: ctx.jobStore, spawn: ctx.spawn }),
},
enabled,
);
}
+32
View File
@@ -30,6 +30,38 @@ export const QUESTION_TOOLS = ["quiz", "ask_user_question"] as const;
/** Tools fulfilled by background sub-agent jobs (W2/W6). */
export const JOB_TOOLS = ["research", "demo", "visual"] as const;
// ─── Tool availability (2026-09 settings seam) ─────────────────────────────
/** The tools a user may toggle off in settings (Wave 2 T2a, user-decided
* 2026-09-15): for now only the SUB-AGENT dispatch tools — quiz,
* ask_user_question (and the read tool, when it lands) are always on;
* the long-term vision is that everything becomes toggleable as new
* tool variants (e.g. alternative question tools) ship, so the factory
* seam below is GENERIC: it honors any tool name it receives. New
* toggleables join this list when settings exposes them.
* The names are the MODEL-FACING tool names (research, not researcher). */
export const TOGGLEABLE_TOOLS = ["research", "demo", "visual"] as const;
/** The resolved settings → factory seam shape (Platform wires chat-handler
* to pass this through from the settings record; storage and defaults are
* Platform's lane). JSON-native: a partial record keyed by tool name.
* Absent key / undefined = enabled — old records behave exactly as today.
* Any explicit `false` removes that tool from the registry, whatever its
* name — the machinery stays generic for future toggleables. */
export type EnabledTools = Partial<Record<string, boolean>>;
/** Drop every tool the resolved settings mark as disabled. The single
* filtering site every factory shares (prober/teacher). */
export function filterEnabledTools<T>(tools: Record<string, T>, enabled?: EnabledTools): Record<string, T> {
if (!enabled) return tools;
const out: Record<string, T> = {};
for (const [name, tool] of Object.entries(tools)) {
if (enabled[name] === false) continue;
out[name] = tool;
}
return out;
}
function findPending(messages: UIMessage[], names: readonly string[]): PendingToolCall | null {
for (let m = messages.length - 1; m >= 0; m--) {
const msg = messages[m]!;