feat: probe/teach agent split — explicit phase markers, position-only plan map
Diagnosis (2026-09-11 user testing, minimal-effort GLM): sessions stuck on 'plan' forever. Two causes: (1) the three-phase view read the stored tree's currentNodeId, which nothing ever writes post author-once (2026-09-03) — a structural dead seam; (2) the teacher stochastically left quiz.nodeId untagged, and done-by-quiz-result semantics would have falsely completed nodes anyway. The redesign (docs/probe-teach-split.md) abolishes derived occupancy in favor of explicit events: - Two agents over one transcript: createProberAgent (quiz without a nodeId field, ask, probe_done) and createTeacherAgent (full toolset, plan_init onward). The chat-handler construction branch keys on a probe_done part in the transcript. MODELS.prober added. - probe_done: server-side seam tool — carries the learner assessment (strands + free-form edge read), refuses while a question is open or the probe is already concluded, and queues a handoff steer through the generalized wake pipeline (steers now dedupe by id, not jobId; job finalization only for job steers). The teacher auto-wakes and authors the plan; the learner sees the assessment via the probe-done card. - phases: probe ends at a concluded probe_done, plan ends at a successful plan_init — strictly sequential, no skipping, no legacy compatibility (data wiped at the pivot per doctrine). - Plan map = position, not completion: derivePlanStatuses reads only tagged quiz CALLS (state/outcome feed nothing); current = latest tagged node with consecutive duplicates collapsed, visited = earlier distinct stays (a revisit stays in both — current wins at render). PlanTree is structure-only; currentNodeId/doneNodeIds and the dead mutation helpers are gone. No fallback current. - Prompts split per agent (prompts/prober/, prompts/teacher/ over the trimmed shared base); graph-context injection removed entirely (shelved, docs/probe-teach-split.md §5); knowledge-graph context no longer reaches either agent. - Plan blocks are full-only snapshots at plan_init (neighborhood machinery retired; the persistent DAG window is the future UI wave); usage capture unchanged. Gates: tsc clean, 294/294 tests (phases/plan-tree/plan-blocks/question- tools rewritten to the new semantics; probe-done + handoff-wake tests added), lint, build. Live GLM smoke PASS: prober concluded with an honest assessment, teacher auto-woke, authored a 10-node plan with the block attached, then tagged checks on slope-instinct → secant derived current/ visited exactly as designed; grading honest both ways (driver off-by-one proved a wrong answer grades false).
This commit is contained in:
amosblomqvist
+94
-42
@@ -1,84 +1,117 @@
|
||||
import * as fs from "node:fs";
|
||||
import * as path from "node:path";
|
||||
import { ToolLoopAgent, stepCountIs, type LanguageModel, type LanguageModelUsage, type ToolSet } from "ai";
|
||||
import { ToolLoopAgent, stepCountIs, tool, type LanguageModel, type LanguageModelUsage, type ToolSet } from "ai";
|
||||
import { PATHS } from "../../config.ts";
|
||||
import { modelFor } from "./models.ts";
|
||||
import { questionTools } from "./tools.ts";
|
||||
import {
|
||||
askUserQuestionTool,
|
||||
createProberTools,
|
||||
createTeacherTools,
|
||||
type ProbeDoneContext,
|
||||
type TeacherToolContext,
|
||||
} from "./tools.ts";
|
||||
import { quizInputObject, quizInputProblems } from "../core/question-core.ts";
|
||||
|
||||
/**
|
||||
* The teacher agent. Everything pi used to provide (agent loop, tool calling,
|
||||
* streaming, multi-step) is these few lines now.
|
||||
* The two teaching-side agents (docs/probe-teach-split.md): the PROBER and
|
||||
* the TEACHER. Same harness (agent loop, tool calling, streaming,
|
||||
* multi-step — the few lines ToolLoopAgent provides), same transcript, but
|
||||
* different prompts and different tool sets; the chat-handler constructs
|
||||
* one per request off the transcript's seam (probe_done = teacher).
|
||||
*
|
||||
* The model is injectable so tests can pass a MockLanguageModelV4 and drive
|
||||
* Models are injectable so tests can pass a MockLanguageModelV4 and drive
|
||||
* deterministic scripted lessons (see tests/chat-flow.test.ts).
|
||||
*
|
||||
* The optional graphContext (buildGraphContext() in src/core/graph-context.ts,
|
||||
* wired by the chat-handler per request) is appended after the composed
|
||||
* fragments; absent/empty means a new learner — nothing is injected.
|
||||
*
|
||||
* The system prompt is COMPOSED: system-prompt.md is the base, then every
|
||||
* src/agent/prompts/*.md fragment in sorted filename order (workers add
|
||||
* fragments — nobody edits shared sections of the base prompt). Prefix
|
||||
* fragments with a numeric order key: 10-plan-tree.md, 20-jobs.md, …
|
||||
* System prompts are COMPOSED per agent: system-prompt.md (the shared base:
|
||||
* philosophy, accuracy, quiz construction, question tools, formatting) +
|
||||
* every numeric-prefix fragment in that agent's directory
|
||||
* (src/agent/prompts/prober/ or src/agent/prompts/teacher/), in sorted
|
||||
* filename order, joined with `---` separators.
|
||||
*
|
||||
* ONLY numeric-prefix files load — the loader enforces the convention, so
|
||||
* non-fragment documentation (README.md, …) can live in the directory
|
||||
* non-fragment documentation (README.md, …) can live in the directories
|
||||
* without ever leaking into the model's prompt.
|
||||
*/
|
||||
|
||||
/** Which agent's prompt directory to compose. */
|
||||
export type AgentKind = "prober" | "teacher";
|
||||
|
||||
const PROMPTS_DIR = path.join(PATHS.root, "src", "agent", "prompts");
|
||||
|
||||
/** A prompt fragment: numeric order key + .md (e.g. 20-jobs.md). Nothing else loads. */
|
||||
/** A prompt fragment: numeric order key + .md (e.g. 20-teaching.md). Nothing else loads. */
|
||||
const FRAGMENT_RE = /^\d+-[\w.-]+\.md$/;
|
||||
|
||||
/** Exported for tests: assert what composes into the model's prompt. */
|
||||
export function composeInstructions(): string {
|
||||
export function composeInstructions(kind: AgentKind): string {
|
||||
const base = fs.readFileSync(PATHS.teacherPrompt, "utf-8");
|
||||
if (!fs.existsSync(PROMPTS_DIR)) return base;
|
||||
const dir = path.join(PROMPTS_DIR, kind);
|
||||
if (!fs.existsSync(dir)) return base;
|
||||
const fragments = fs
|
||||
.readdirSync(PROMPTS_DIR)
|
||||
.readdirSync(dir)
|
||||
.filter((f) => FRAGMENT_RE.test(f))
|
||||
.sort()
|
||||
.map((f) => fs.readFileSync(path.join(PROMPTS_DIR, f), "utf-8").trim())
|
||||
.map((f) => fs.readFileSync(path.join(dir, f), "utf-8").trim())
|
||||
.filter(Boolean);
|
||||
return [base.trim(), ...fragments].join("\n\n---\n\n");
|
||||
}
|
||||
|
||||
const instructions = composeInstructions();
|
||||
const composedInstructions: Record<AgentKind, string> = {
|
||||
prober: composeInstructions("prober"),
|
||||
teacher: composeInstructions("teacher"),
|
||||
};
|
||||
|
||||
// Money channel (wave 4): the chat-handler passes a per-turn collector;
|
||||
// every model call's usage lands in the chat record's turns log
|
||||
// (ChatStore.appendTurnUsage). The step carries far more than usage —
|
||||
// the narrow param keeps the seam honest about what we consume.
|
||||
export type StepUsageCollector = (step: { usage: LanguageModelUsage }) => void;
|
||||
|
||||
/**
|
||||
* Compose the full instruction string: base prompt + fragments, then — when
|
||||
* the session carries graph context (W8, vision §1 "the graph as session
|
||||
* context") — the auto-generated knowledge-graph block appended after them as
|
||||
* a clearly-delimited section (the 50-graph-context.md fragment teaches the
|
||||
* teacher how to read it). The fragment mechanism itself stays untouched.
|
||||
* The PROBER (agent A): quiz (no nodeId) + ask + probe_done, lean on
|
||||
* probing doctrine. Its whole job is bracketing the learner's edge and
|
||||
* concluding with the assessment (probe_done) — which also queues the
|
||||
* handoff wake (see tools/probe.ts) and is the transcript seam the
|
||||
* handler keys teacher construction on.
|
||||
*/
|
||||
function withGraphContext(graphContext: string | null | undefined): string {
|
||||
if (!graphContext) return instructions;
|
||||
return `${instructions}\n\n---\n\n${graphContext}`;
|
||||
export function createProberAgent(ctx: ProbeDoneContext, model?: LanguageModel, onStepFinish?: StepUsageCollector) {
|
||||
return new ToolLoopAgent({
|
||||
// Model choice + thinking budget live in config.ts (MODELS.prober).
|
||||
// Reasoning tokens stream as reasoning parts into the UI's collapsible block.
|
||||
model: model ?? modelFor("prober"),
|
||||
instructions: composedInstructions.prober,
|
||||
tools: createProberTools(ctx),
|
||||
// Question tools are client-side (a call ends the server turn);
|
||||
// probe_done resolves immediately (it queues the handoff and acks).
|
||||
// stepCountIs is just the safety net.
|
||||
stopWhen: stepCountIs(20),
|
||||
...(onStepFinish ? { onStepFinish } : {}),
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* The TEACHER (agent B): enters at planning (the probe is already
|
||||
* concluded — its assessment is in the transcript), owns plan_init and the
|
||||
* full toolset (teach questions with validated nodeId, the map, research,
|
||||
* demos).
|
||||
*/
|
||||
export function createTeacherAgent(
|
||||
// Optional so tests can construct a bare mock-driven agent (no storage
|
||||
// wiring): without ctx the tools fall back to the validation-free
|
||||
// question set. The handler ALWAYS passes ctx (the composition root).
|
||||
ctx?: TeacherToolContext,
|
||||
model?: LanguageModel,
|
||||
onStepFinish?: StepUsageCollector,
|
||||
// Tests may pass a bare tools set to drive deterministic flows; when
|
||||
// present it overrides the ctx-composed registry.
|
||||
tools?: ToolSet,
|
||||
graphContext?: string | null,
|
||||
// Money channel (wave 4): the chat-handler passes a per-turn collector;
|
||||
// every model call's usage lands in the chat record's turns log
|
||||
// (ChatStore.appendTurnUsage). The step carries far more than usage —
|
||||
// the narrow param keeps the seam honest about what we consume.
|
||||
onStepFinish?: (step: { usage: LanguageModelUsage }) => void,
|
||||
) {
|
||||
return new ToolLoopAgent({
|
||||
// Model choice + thinking budget live in config.ts (MODELS.teacher).
|
||||
// Reasoning tokens stream as reasoning parts into the UI's collapsible block.
|
||||
model: model ?? modelFor("teacher"),
|
||||
instructions: withGraphContext(graphContext),
|
||||
// Tools are per-request (plan tools close over the injected ctx; job
|
||||
// tools close over the job store + spawn seam) — chat-handler passes
|
||||
// createTeacherTools({ chatId, chatStore, jobStore, spawn }). The
|
||||
// dependency-free default is the question-only set (client-side tools,
|
||||
// no storage needed) — used when tests pass a bare mock model.
|
||||
tools: tools ?? { ...questionTools },
|
||||
instructions: composedInstructions.teacher,
|
||||
// Tools are per-request (they close over the injected ctx — chat-
|
||||
// handler passes the composition root's stores + spawn seam).
|
||||
tools: tools ?? (ctx ? createTeacherTools(ctx) : bareQuestionTools()),
|
||||
// Question tools are client-side (a call ends the server turn);
|
||||
// job tools resolve immediately (fire-and-forget ack). stepCountIs is
|
||||
// just the safety net.
|
||||
@@ -87,4 +120,23 @@ export function createTeacherAgent(
|
||||
});
|
||||
}
|
||||
|
||||
/** The validation-free teach question set (tests, bare agents): quiz with
|
||||
* the shared authoring refinement but NO per-session nodeId check (no
|
||||
* record to read), plus ask. */
|
||||
export function bareQuestionTools(): ToolSet {
|
||||
return {
|
||||
quiz: tool({
|
||||
description: "Ask the learner a graded question.",
|
||||
inputSchema: quizInputObject.superRefine((input, add) => {
|
||||
for (const problem of quizInputProblems(input)) {
|
||||
add.addIssue({ code: "custom", message: problem });
|
||||
}
|
||||
}),
|
||||
}),
|
||||
ask_user_question: askUserQuestionTool,
|
||||
};
|
||||
}
|
||||
|
||||
export type ProberAgent = ReturnType<typeof createProberAgent>;
|
||||
export type TeacherAgent = ReturnType<typeof createTeacherAgent>;
|
||||
export type TeachingAgent = ProberAgent | TeacherAgent;
|
||||
Reference in New Issue
Block a user