W6: demo + visual artifacts, verifying builders

This commit is contained in:
Amos
2026-08-22 10:24:02 +03:00
parent ded163aa3f
commit 6f1b35b2c2
9 changed files with 1831 additions and 29 deletions
+45
View File
@@ -0,0 +1,45 @@
## Interactive demos and precise figures — build the instrument, don't describe it
If the concept is DYNAMIC — a parameter to sweep, a process to animate, a distribution to sample, a
system to simulate — build the learner the instrument rather than describing its behavior in prose.
A paragraph about how frequency response changes with damping teaches less than ten seconds of
dragging a slider. You have two artifact tools for this, both asynchronous (same discipline as
research):
### `demo` — interactive teaching instruments
A background sub-agent builds a self-contained interactive demo (single-file HTML: parameter sweeps,
simulations, animations) and **headlessly tests that it actually runs** before it reaches the lesson.
Nothing untested ever appears.
- Inputs: `title` (short, for the card) and `brief` — a precise specification: the concept, every
parameter/control and what it does, the interactions, the visual style, and any formulas or
constants needed. **The builder never sees the lesson** — the brief is its entire world. Include
everything; a vague brief yields a vague demo.
- Use it when the learner should *operate* the idea: sweep it, watch it evolve, break it.
### `visual` — precise static figures
A background sub-agent hand-authors one SVG, renders it, **looks at the render**, and iterates until
the picture is correct. Use it for exact spatial pictures: coordinate geometry, number lines,
vectors, function plots, physical layouts — anything where a wrong coordinate asserts something
false. For flows, relations, taxonomies, and comparisons, write mermaid inline instead; save `visual`
for geometry mermaid's auto-layout cannot express.
- Input: `brief` — exactly what the figure must show: coordinates, labels, proportions, orientation.
One idea per figure. The maker sees nothing else of the lesson.
### The async discipline
The call is asynchronous and **your turn ends at it**, exactly like research:
1. **Say your piece first.** Set up the concept, explain what the learner is about to see and what to
watch for — the artifact arrives later as the tool's output and the lesson resumes from there. A
bare tool call with no surrounding text is a failed turn.
2. **At most one demo or visual per turn.**
3. **When the artifact arrives, TEACH WITH IT.** Reference its controls and elements by name: "drag
the damping slider all the way left and watch the overshoot", "notice the vector at t=2s". An
artifact you never point at is decoration.
4. **Failures arrive as notes** ("FAILED: …") with no artifact — say so plainly, fall back to
explaining in prose, and if the brief was the problem, re-call with a tighter one. Re-calling a
tool produces a new version of the artifact; use that to fix a demo that misbehaves in the field.
+238
View File
@@ -0,0 +1,238 @@
import { ToolLoopAgent, stepCountIs, tool, type LanguageModel, type ToolSet } from "ai";
import { openrouter } from "@openrouter/ai-sdk-provider";
import { z } from "zod";
import type { DemoResult } from "../../core/contracts.ts";
import { getHeadless, type DemoRunResult, type HeadlessApi } from "./headless.ts";
/**
* The demo-builder sub-agent (vision §2 interactive demos + §2.5 "sub-agents
* verify; they never one-shot"): given a brief it writes a self-contained
* single-file HTML/JS demo, TESTS IT HEADLESSLY (run_demo), and iterates
* fix → re-run until clean — before anything reaches the lesson.
*
* Shape: a ToolLoopAgent with two executable tools (write_demo / run_demo) —
* the model is injectable (tests use MockLanguageModelV4) and the headless
* API is injectable (tests use a fake; production uses system Chrome).
*
* Failure semantics (decided in the W6 brief): `tested: true` means "the
* verify loop ran"; on failure the result is { title, html: "", tested: true,
* notes: "FAILED: <reason>" } — the card renders the failure note, no iframe.
* The job wrapper treats html === "" as failure and marks the job error.
*/
const DEFAULT_DEMO_MODEL =
process.env.LH_DEMO_MODEL ?? process.env.LH_MODEL ?? "anthropic/claude-sonnet-4.5";
export interface DemoBuilderInput {
title: string;
brief: string;
}
export interface DemoBuilderOptions {
model?: LanguageModel;
headless?: HeadlessApi;
/** Max agent steps (each write/run/fix is one step). Default 24 ≈ 6 cycles. */
maxSteps?: number;
/** Live activity-line callback (updates the job record). */
onActivity?: (line: string) => void;
}
const INSTRUCTIONS = `You are a demo builder — a specialist sub-agent inside a learning application. You
receive a brief for ONE interactive teaching instrument and deliver a single self-contained HTML document
that runs it. You never see the lesson; the brief is everything you know.
Runtime contract (the document runs in a sandboxed iframe, sandbox="allow-scripts", opaque origin):
- Single file. Vanilla JS + Canvas 2D or SVG. CDN imports (e.g. a plotting or math library) are allowed
when they genuinely help — but prefer dependency-free; you own the failure if a CDN import fails.
- NO localStorage, sessionStorage, or document.cookie — the sandbox makes them throw.
- The parent sizes the iframe to the document height automatically; just build normal flowing layout.
Make the demo fill the width and size its own height sensibly.
Build quality:
- The demo must directly serve the brief: every control named in the brief must exist and do the thing.
- Controls need labels; sliders/inputs need current-value readouts. A learner should operate it unguided.
- Smooth enough interaction; requestAnimationFrame for animation, not setInterval when it matters.
- Dark-mode neutral: pick a clean light background for the demo area so it reads on any card.
THE HOUSE RULE — never deliver untested:
1. Call write_demo with the complete HTML document.
2. Call run_demo. It loads the file in headless Chrome, waits ~2s, and reports console errors, uncaught
errors (window.onerror), and failed requests. Optionally pass a probe: a JS expression evaluated in
the page (e.g. "typeof window.simulate === 'function'" or a state assertion) to check deeper.
3. If run_demo reports problems, FIX the document and call write_demo again, then run_demo again.
Iterate until run_demo reports PASSED.
4. Only then finish. Your final message is 1-3 sentences of NOTES for the teacher (what the demo shows,
its controls, what to try) — NEVER the HTML itself.
Never claim success you did not see: a PASSED run_demo is the only definition of done.`;
interface BuilderState {
html: string | null;
/** The html that the last run_demo tested (null = never tested). */
testedHtml: string | null;
lastResult: DemoRunResult | null;
}
export function createDemoBuilderTools(
state: BuilderState,
headless: HeadlessApi,
onActivity?: (line: string) => void,
): ToolSet {
return {
write_demo: tool({
description:
"Write the complete self-contained HTML document for the demo (overwrites any previous version). " +
"Always follow with run_demo to test it headlessly.",
inputSchema: z.object({
html: z.string().describe("The complete HTML document: <!doctype html>…</html>, all JS inline."),
}),
execute: async ({ html }) => {
if (!html || html.trim().length === 0) return "ERROR: empty document — write the full HTML.";
state.html = html;
onActivity?.(`write_demo: ${(html.length / 1024).toFixed(1)} KB`);
return `Saved (${html.length} bytes). Now call run_demo to test it — never deliver untested.`;
},
}),
run_demo: tool({
description:
"Headlessly execute the current demo in Chrome and report console errors, uncaught errors, and failed requests. " +
"Optionally pass a probe expression to assert on page state.",
inputSchema: z.object({
probe: z
.string()
.optional()
.describe("Optional JS expression evaluated in the page after settling, e.g. \"document.querySelectorAll('input').length\"."),
}),
execute: async ({ probe }) => {
if (!state.html) return "ERROR: nothing to run — call write_demo first.";
const reason = headless.unavailableReason();
if (reason) return `HEADLESS UNAVAILABLE: ${reason}`;
let result: DemoRunResult;
try {
result = await headless.runDemo(state.html, probe ? { probe } : undefined);
} catch (err) {
const msg = err instanceof Error ? err.message : String(err);
onActivity?.("run_demo: browser error");
return `HEADLESS ERROR (retrying may help): ${msg}`;
}
state.testedHtml = state.html;
state.lastResult = result;
onActivity?.(
result.ok ? "run_demo: clean" : `run_demo: ${summarizeCounts(result)}`,
);
return formatRunResult(result);
},
}),
};
}
function summarizeCounts(result: DemoRunResult): string {
const counts = [
result.consoleErrors.length > 0 ? `${result.consoleErrors.length} console error${result.consoleErrors.length === 1 ? "" : "s"}` : null,
result.pageErrors.length > 0 ? `${result.pageErrors.length} page error${result.pageErrors.length === 1 ? "" : "s"}` : null,
result.failedRequests.length > 0 ? `${result.failedRequests.length} failed request${result.failedRequests.length === 1 ? "" : "s"}` : null,
].filter(Boolean);
return counts.length > 0 ? counts.join(", ") : "ok";
}
function formatRunResult(result: DemoRunResult): string {
if (result.ok) {
const probe =
result.probeResult !== undefined
? ` Probe result: ${JSON.stringify(result.probeResult) ?? "undefined"}.`
: "";
return `PASSED: no console errors, no uncaught errors, no failed requests.${probe} If you are satisfied, finish with your notes; otherwise keep improving and re-test.`;
}
const sections: string[] = [`FAILED: ${summarizeCounts(result)}.`];
if (result.consoleErrors.length > 0)
sections.push(`Console errors:\n${result.consoleErrors.slice(0, 10).map((e) => `- ${e}`).join("\n")}`);
if (result.pageErrors.length > 0)
sections.push(`Uncaught errors:\n${result.pageErrors.slice(0, 10).map((e) => `- ${e}`).join("\n")}`);
if (result.failedRequests.length > 0)
sections.push(`Failed requests:\n${result.failedRequests.slice(0, 10).map((r) => `- ${r}`).join("\n")}`);
sections.push("Fix the document (write_demo) and re-test (run_demo).");
return sections.join("\n");
}
/** Short summary of a run for failure notes. */
function failureSummary(result: DemoRunResult | null): string {
if (!result) return "never tested";
const details = [...result.consoleErrors, ...result.pageErrors, ...result.failedRequests].slice(0, 5);
return details.length > 0 ? `${summarizeCounts(result)} — ${details.join(" | ")}` : summarizeCounts(result);
}
export async function runDemoBuilder(input: DemoBuilderInput, opts: DemoBuilderOptions = {}): Promise<DemoResult> {
const headless = opts.headless ?? getHeadless();
const title = input.title.trim() || "Interactive demo";
const brief = input.brief.trim() || "(no brief)";
const notes = (text: string) => (text.trim() ? text.trim() : undefined);
const reason = headless.unavailableReason();
if (reason) {
return { title, html: "", tested: true, notes: `FAILED: ${reason}` };
}
const state: BuilderState = { html: null, testedHtml: null, lastResult: null };
const agent = new ToolLoopAgent({
model: opts.model ?? openrouter(DEFAULT_DEMO_MODEL),
instructions: INSTRUCTIONS,
tools: createDemoBuilderTools(state, headless, opts.onActivity),
stopWhen: stepCountIs(opts.maxSteps ?? 24),
});
const prompt = [
`Demo title: ${title}`,
`Brief (all you know about the lesson — this comes from the teacher, verbatim):`,
brief,
"",
"Build the demo, test it with run_demo, iterate until PASSED, then finish with your notes.",
].join("\n");
let finalText = "";
try {
const result = await agent.generate({
prompt,
onToolExecutionStart: ({ toolCall }) => {
opts.onActivity?.(toolCall.toolName);
},
});
finalText = result.text;
} catch (err) {
const msg = err instanceof Error ? err.message : String(err);
return { title, html: "", tested: true, notes: `FAILED: the builder crashed — ${msg}` };
}
// The model produced nothing.
if (!state.html) {
return {
title,
html: "",
tested: true,
notes: `FAILED: the builder never wrote a demo.${finalText ? ` (${finalText.slice(0, 300)})` : ""}`,
};
}
// The verify gate: the delivered html must have a clean headless run —
// even when the model claims success or forgets the final run_demo.
if (state.testedHtml !== state.html || !state.lastResult?.ok) {
opts.onActivity?.("final verification");
let verify: DemoRunResult;
try {
verify = await headless.runDemo(state.html);
} catch (err) {
const msg = err instanceof Error ? err.message : String(err);
return { title, html: "", tested: true, notes: `FAILED: final verification crashed — ${msg}` };
}
state.lastResult = verify;
if (!verify.ok) {
return {
title,
html: "",
tested: true,
notes: `FAILED: the demo never passed headless verification (${failureSummary(verify)}).`,
};
}
}
return { title, html: state.html, tested: true, notes: notes(finalText) };
}
+220
View File
@@ -0,0 +1,220 @@
import * as fs from "node:fs";
import * as os from "node:os";
import * as path from "node:path";
import { pathToFileURL } from "node:url";
/**
* Shared headless-browser helper for the verifying sub-agents (W6):
* the demo-builder executes demos and captures errors; the visual-maker
* rasterizes SVGs so it can look at them. Both use the SAME system Chrome
* via puppeteer-core — NO bundled chromium is downloaded.
*
* Chrome resolution: process.env.CHROME_PATH first, then the common
* macOS/Linux install paths. If none exists we fall back to puppeteer's
* `channel: "chrome"` at launch time; `available()` is cheap and only knows
* about the explicit candidates, so a channel-only setup reports unavailable
* and the builders refuse to run (the house rule: never skip testing).
*
* Test seam: `headlessOverride.current` replaces the whole API (tests inject
* a fake; nothing here touches a real browser). Builders call `getHeadless()`.
*/
export interface DemoRunResult {
/** console.error / console messages of type "error". */
consoleErrors: string[];
/** Uncaught exceptions (window.onerror / unhandled rejections). */
pageErrors: string[];
/** Network requests that failed (CDN scripts that didn't load, …). */
failedRequests: string[];
/** Value of the optional probe expression, when requested and successful. */
probeResult?: unknown;
/** True when nothing above recorded anything. */
ok: boolean;
}
export interface RunDemoOptions {
/** How long to let the page settle (animations, async CDN loads) — default 2s. */
settleMs?: number;
/** Optional JS expression evaluated in the page after settling; its value
* is returned to the builder so it can assert on internal state. */
probe?: string;
}
export interface RenderSvgOptions {
/** Viewport size — defaults to the SVG's width/height attrs, else 900×700. */
width?: number;
height?: number;
}
export interface HeadlessApi {
/** True when a Chrome executable is resolvable (cheap, no launch). */
available(): boolean;
/** Why not — for error messages. */
unavailableReason(): string | null;
/** Load the HTML in headless Chrome, capture errors, optionally probe. */
runDemo(html: string, opts?: RunDemoOptions): Promise<DemoRunResult>;
/** Rasterize an SVG to a PNG buffer (for verify-by-looking). */
renderSvg(svg: string, opts?: RenderSvgOptions): Promise<Buffer>;
}
/** Test seam — undefined in production. */
export const headlessOverride: { current?: HeadlessApi } = {};
export function getHeadless(): HeadlessApi {
return headlessOverride.current ?? realHeadless;
}
// ─── Chrome resolution ───────────────────────────────────────────────────────
const CHROME_CANDIDATES = [
"/Applications/Google Chrome.app/Contents/MacOS/Google Chrome",
"/Applications/Chromium.app/Contents/MacOS/Chromium",
"/Applications/Microsoft Edge.app/Contents/MacOS/Microsoft Edge",
"/usr/bin/google-chrome",
"/usr/bin/google-chrome-stable",
"/usr/bin/chromium",
"/usr/bin/chromium-browser",
];
export function resolveChromeExecutablePath(): string | null {
const candidates = [process.env.CHROME_PATH, ...CHROME_CANDIDATES].filter(
(p): p is string => typeof p === "string" && p.length > 0,
);
for (const candidate of candidates) {
try {
if (fs.existsSync(candidate)) return candidate;
} catch {
/* not readable — try the next */
}
}
return null;
}
const NO_CHROME_MESSAGE =
"No Chrome/Chromium found for headless testing. Install Google Chrome or set CHROME_PATH to a Chrome executable. " +
"(Verification is mandatory — demos and visuals are never delivered untested.)";
async function launchBrowser(): Promise<import("puppeteer-core").Browser> {
const puppeteer = (await import("puppeteer-core")).default;
const executablePath = resolveChromeExecutablePath();
const browser = await puppeteer.launch({
headless: true,
// Explicit path when we found one; otherwise let puppeteer resolve the
// system Chrome install itself via the "chrome" channel.
...(executablePath ? { executablePath } : { channel: "chrome" as const }),
args: ["--disable-gpu", "--no-sandbox", "--disable-dev-shm-usage"],
});
return browser;
}
/** A temp dir per invocation; removed in finally. */
function tempDir(prefix: string): string {
return fs.mkdtempSync(path.join(os.tmpdir(), prefix));
}
// ─── Demo execution ──────────────────────────────────────────────────────────
async function runDemo(html: string, opts: RunDemoOptions = {}): Promise<DemoRunResult> {
const settleMs = opts.settleMs ?? 2000;
const dir = tempDir("lh-demo-");
const file = path.join(dir, "demo.html");
fs.writeFileSync(file, html, "utf-8");
const consoleErrors: string[] = [];
const pageErrors: string[] = [];
const failedRequests: string[] = [];
let probeResult: unknown;
let probed = false;
const browser = await launchBrowser();
try {
const page = await browser.newPage();
page.on("console", (msg) => {
if (msg.type() === "error") consoleErrors.push(msg.text());
});
page.on("pageerror", (err) => pageErrors.push(err instanceof Error ? err.message : String(err)));
page.on("requestfailed", (req) => failedRequests.push(`${req.url()} (${req.failure()?.errorText ?? "failed"})`));
try {
await page.goto(pathToFileURL(file).href, { waitUntil: "load", timeout: 30_000 });
await new Promise((resolve) => setTimeout(resolve, settleMs));
if (opts.probe) {
try {
probeResult = await page.evaluate(opts.probe);
} catch (err) {
pageErrors.push(`probe failed: ${err instanceof Error ? err.message : String(err)}`);
}
probed = true;
}
} catch (err) {
pageErrors.push(err instanceof Error ? err.message : String(err));
}
} finally {
await browser.close().catch(() => {});
try {
fs.rmSync(dir, { recursive: true, force: true });
} catch {
/* best-effort cleanup */
}
}
return {
consoleErrors,
pageErrors,
failedRequests,
...(probed ? { probeResult } : {}),
ok: consoleErrors.length === 0 && pageErrors.length === 0 && failedRequests.length === 0,
};
}
// ─── SVG rasterization ───────────────────────────────────────────────────────
function svgViewport(svg: string, opts: RenderSvgOptions): { width: number; height: number } {
if (opts.width && opts.height) return { width: opts.width, height: opts.height };
const width = Number(svg.match(/\bwidth\s*=\s*"?(\d+(?:\.\d+)?)/)?.[1]);
const height = Number(svg.match(/\bheight\s*=\s*"?(\d+(?:\.\d+)?)/)?.[1]);
return {
width: Math.ceil(Number.isFinite(width) && width > 0 ? width : 900),
height: Math.ceil(Number.isFinite(height) && height > 0 ? height : 700),
};
}
async function renderSvg(svg: string, opts: RenderSvgOptions = {}): Promise<Buffer> {
const dir = tempDir("lh-visual-");
const file = path.join(dir, "figure.html");
fs.writeFileSync(
file,
`<!doctype html><html><head><meta charset="utf-8"><style>html,body{margin:0;padding:0;background:#fff}svg{display:block}</style></head><body>${svg}</body></html>`,
"utf-8",
);
const browser = await launchBrowser();
try {
const page = await browser.newPage();
const { width, height } = svgViewport(svg, opts);
await page.setViewport({ width, height });
await page.goto(pathToFileURL(file).href, { waitUntil: "load", timeout: 30_000 });
const png = (await page.screenshot({ fullPage: true, type: "png" })) as Buffer;
return png;
} finally {
await browser.close().catch(() => {});
try {
fs.rmSync(dir, { recursive: true, force: true });
} catch {
/* best-effort cleanup */
}
}
}
// ─── the real API ────────────────────────────────────────────────────────────
export const realHeadless: HeadlessApi = {
available() {
return resolveChromeExecutablePath() !== null;
},
unavailableReason() {
return resolveChromeExecutablePath() === null ? NO_CHROME_MESSAGE : null;
},
runDemo,
renderSvg,
};
+122 -16
View File
@@ -1,9 +1,11 @@
import { isToolUIPart, type LanguageModel, type UIMessage } from "ai";
import type { ResearchResult, ResearchToolInput } from "../../core/contracts.ts";
import type { DemoResult, DemoToolInput, JobKind, ResearchResult, ResearchToolInput, VisualResult, VisualToolInput } from "../../core/contracts.ts";
import { JOB_TOOLS } from "../../core/pending.ts";
import { getChat, saveMessages } from "../../store/chats.ts";
import { createJob, findJobByToolCallId, getJob, updateJob } from "../../store/jobs.ts";
import { runDemoBuilder } from "./demo-builder.ts";
import { createResearcherAgent } from "./researcher.ts";
import { runVisualMaker } from "./visual-maker.ts";
/**
* The background job runner + the chat-handler spawn hook (contracts.ts
@@ -11,18 +13,22 @@ import { createResearcherAgent } from "./researcher.ts";
*
* spawnPendingJobs — called from onFinish AFTER saveMessages: finds
* input-available JOB_TOOLS parts in the last assistant message with no
* existing job and fires runResearchJob WITHOUT awaiting (the HTTP
* response is long gone).
* existing job and fires the kind's runner WITHOUT awaiting (the HTTP
* response is long gone). Kind dispatch (research/demo/visual) is a
* self-contained map here, so chat-handler needs no changes for new
* job kinds — anything listed in JOB_TOOLS routes automatically.
*
* runResearchJob — runs the researcher agent loop in the background,
* updating the job's activity line on every tool call, then rewrites the
* tool part in the LATEST chat record to output-available (read-modify-
* write; no awaits between the read and the write). The part is fulfilled
* BEFORE the job status flips, so a client that polls the job and then
* re-syncs always sees the finished part.
* runResearchJob / runDemoJob / runVisualJob — run the sub-agent loop in
* the background, updating the job's activity line on every iteration,
* then rewrite the tool part in the LATEST chat record to
* output-available (read-modify-write; no awaits between the read and
* the write). The part is fulfilled BEFORE the job status flips, so a
* client that polls the job and then re-syncs always sees the finished
* part.
*
* On failure the part is still fulfilled ({brief: "Research failed: …"}) —
* the teacher must learn the outcome — and the job is marked error.
* On failure the part is still fulfilled (DemoResult/VisualResult failure
* shape: empty html/svg + "FAILED: …" notes; research: failure brief) — the
* teacher must learn the outcome — and the job is marked error.
*/
/**
@@ -31,11 +37,20 @@ import { createResearcherAgent } from "./researcher.ts";
* network. Undefined in production.
*/
export const researchModelOverride: { current?: LanguageModel } = {};
export const demoModelOverride: { current?: LanguageModel } = {};
export const visualModelOverride: { current?: LanguageModel } = {};
export interface RunResearchJobOptions {
model?: LanguageModel;
}
/** The kind→runner map: everything in JOB_TOOLS routes through here. */
const KIND_RUNNERS: Record<JobKind, (jobId: string) => Promise<void>> = {
research: (jobId) => runResearchJob(jobId, { model: researchModelOverride.current }),
demo: (jobId) => runDemoJob(jobId, { model: demoModelOverride.current }),
visual: (jobId) => runVisualJob(jobId, { model: visualModelOverride.current }),
};
export function spawnPendingJobs(chatId: string, messages: UIMessage[]): void {
const last = [...messages].reverse().find((m) => m.role === "assistant");
if (!last) return;
@@ -43,11 +58,12 @@ export function spawnPendingJobs(chatId: string, messages: UIMessage[]): void {
if (!isToolUIPart(part) || part.state !== "input-available") continue;
const tool = part.type.slice("tool-".length);
if (!(JOB_TOOLS as readonly string[]).includes(tool)) continue;
if (tool !== "research") continue; // demo/visual runners arrive in W6
const runner = KIND_RUNNERS[tool as JobKind];
if (!runner) continue;
if (findJobByToolCallId(chatId, part.toolCallId)) continue; // already has a job
const job = createJob({ kind: "research", chatId, toolCallId: part.toolCallId });
void runResearchJob(job.id, { model: researchModelOverride.current }).catch((err) => {
console.error(`[job ${job.id}] research runner crashed:`, err);
const job = createJob({ kind: tool as JobKind, chatId, toolCallId: part.toolCallId });
void runner(job.id).catch((err) => {
console.error(`[job ${job.id}] ${tool} runner crashed:`, err);
updateJob(job.id, { status: "error", error: String(err), activity: "failed" });
});
}
@@ -103,6 +119,96 @@ export async function runResearchJob(jobId: string, opts: RunResearchJobOptions
}
}
// ─── demo + visual job wrappers ───────────────────────────────────────────
export interface RunDemoJobOptions {
model?: LanguageModel;
}
/** Failure shape (W6 brief): html === "" with "FAILED: …" notes. */
function isDemoFailure(result: DemoResult): boolean {
return typeof result.html !== "string" || result.html.length === 0;
}
function isVisualFailure(result: VisualResult): boolean {
return typeof result.svg !== "string" || result.svg.length === 0;
}
export async function runDemoJob(jobId: string, opts: RunDemoJobOptions = {}): Promise<void> {
const job = getJob(jobId);
if (!job || job.status !== "running") return;
const chat = getChat(job.chatId);
if (!chat) {
updateJob(jobId, { status: "error", error: `Chat not found: ${job.chatId}`, activity: "failed" });
return;
}
const input = findToolPart(chat.messages, job.toolCallId)?.input as DemoToolInput | undefined;
const title = typeof input?.title === "string" && input.title.trim() ? input.title : "Interactive demo";
const brief = typeof input?.brief === "string" ? input.brief : "";
try {
updateJob(jobId, { activity: "building demo" });
const result = await runDemoBuilder(
{ title, brief },
{
model: opts.model,
onActivity: (line) => updateJob(jobId, { activity: line }),
},
);
fulfillPart(job.chatId, job.toolCallId, result);
if (isDemoFailure(result)) {
updateJob(jobId, { status: "error", error: result.notes ?? "demo build failed", activity: "failed" });
} else {
updateJob(jobId, { status: "done", result });
}
} catch (err) {
const reason = err instanceof Error ? err.message : String(err);
const failure: DemoResult = { title, html: "", tested: true, notes: `FAILED: ${reason}` };
fulfillPart(job.chatId, job.toolCallId, failure);
updateJob(jobId, { status: "error", error: reason, activity: "failed" });
}
}
export interface RunVisualJobOptions {
model?: LanguageModel;
}
export async function runVisualJob(jobId: string, opts: RunVisualJobOptions = {}): Promise<void> {
const job = getJob(jobId);
if (!job || job.status !== "running") return;
const chat = getChat(job.chatId);
if (!chat) {
updateJob(jobId, { status: "error", error: `Chat not found: ${job.chatId}`, activity: "failed" });
return;
}
const input = findToolPart(chat.messages, job.toolCallId)?.input as VisualToolInput | undefined;
const brief = typeof input?.brief === "string" ? input.brief : "";
try {
updateJob(jobId, { activity: "drawing figure" });
const result = await runVisualMaker(
{ brief },
{
model: opts.model,
onActivity: (line) => updateJob(jobId, { activity: line }),
},
);
fulfillPart(job.chatId, job.toolCallId, result);
if (isVisualFailure(result)) {
updateJob(jobId, { status: "error", error: result.notes ?? "visual build failed", activity: "failed" });
} else {
updateJob(jobId, { status: "done", result });
}
} catch (err) {
const reason = err instanceof Error ? err.message : String(err);
const failure: VisualResult = { svg: "", notes: `FAILED: ${reason}` };
fulfillPart(job.chatId, job.toolCallId, failure);
updateJob(jobId, { status: "error", error: reason, activity: "failed" });
}
}
// ─── helpers ─────────────────────────────────────────────────────────────────
function findToolPart(messages: UIMessage[], toolCallId: string) {
@@ -134,7 +240,7 @@ function hostOf(url: string | undefined): string {
* Rewrite the tool part in the LATEST chat record to output-available.
* Synchronous read-modify-write (no awaits) — atomic within the event loop.
*/
function fulfillPart(chatId: string, toolCallId: string, output: ResearchResult): void {
function fulfillPart(chatId: string, toolCallId: string, output: ResearchResult | DemoResult | VisualResult): void {
const chat = getChat(chatId);
if (!chat) return;
let found = false;
+229
View File
@@ -0,0 +1,229 @@
import { generateText, tool, type LanguageModel, type ModelMessage, type ToolResultPart } from "ai";
import { openrouter } from "@openrouter/ai-sdk-provider";
import { z } from "zod";
import type { VisualResult } from "../../core/contracts.ts";
import { getHeadless, type HeadlessApi } from "./headless.ts";
/**
* The visual-maker sub-agent (vision §2.5): authors ONE precise SVG from a
* brief, renders it to a PNG in the SAME headless Chrome, LOOKS at the
* result, and iterates until correct — a direct port of the vault POC's
* svg-maker doctrine ("the one rule that matters: verify by looking").
*
* LOOP SHAPE (decided in the W6 brief): hand-rolled with generateText + a
* messages array, because the loop must feed an IMAGE back to the model after
* each render. write_svg / render_svg are declared WITHOUT execute — when the
* model calls them generateText stops with the tool calls, and this loop
* appends the tool results itself, followed by a user message containing the
* rendered PNG as a file part ("Here is the current render"). ToolLoopAgent's
* execute-tool plumbing can't express that image-in-the-loop shape cleanly.
*
* Doctrine enforcement in code: if the model tries to finish while the
* current SVG has never been rendered (or changed since its last render), the
* loop renders it anyway, shows the image, and asks the model to look before
* finishing. Nothing is delivered without having been looked at.
*
* Failure semantics mirror the demo-builder: { svg: "", notes: "FAILED: …" }.
*/
const DEFAULT_VISUAL_MODEL =
process.env.LH_VISUAL_MODEL ?? process.env.LH_MODEL ?? "anthropic/claude-sonnet-4.5";
export interface VisualMakerInput {
brief: string;
}
export interface VisualMakerOptions {
model?: LanguageModel;
headless?: HeadlessApi;
/** Max model rounds (write/render cycles + final answer). Default 8. */
maxIterations?: number;
/** Live activity-line callback (updates the job record). */
onActivity?: (line: string) => void;
}
const INSTRUCTIONS = `You are a visual maker — a diagram author + renderer for precise, spatial figures. You
receive a brief describing ONE idea that needs exact placement (coordinate geometry, number lines,
vectors, function plots, physical layouts — things auto-layout diagrams cannot express), and you return
ONE clean, correct SVG. You never see the lesson; the brief is everything you know.
You do NOT decide what idea to show — the teacher already decided. Your job is faithful, precise
composition, and above everything CORRECTNESS: the picture must not assert anything false. A right-angle
mark on the wrong corner, a vector pointing the wrong way, a point plotted at the wrong coordinate is a
failure even if it renders cleanly.
Your two tools:
- write_svg({ svg }) — write the complete <svg>…</svg> source (overwrites the previous version).
- render_svg({}) — rasterize the current SVG in headless Chrome. The render arrives as an image in the
message right after the tool result.
THE ONE RULE THAT MATTERS MOST: verify by looking. You are done only when you have LOOKED at the
rendered PNG and confirmed it is true to the brief. Rendering success only proves the SVG parsed; it
says nothing about whether the geometry is right or the picture is readable. When you look:
- Is every coordinate, angle, direction, and proportion actually correct? Re-derive the geometry if unsure.
- Are labels placed clearly, not overlapping lines or each other?
- Is anything clipped by the viewBox, too small to read, or cramped?
- Would a learner instantly read the intended idea from this picture alone?
Workflow: plan the coordinate space (viewBox + margins, nothing touching the edges) → write_svg →
render_svg → LOOK critically → fix (write_svg) and re-render until correct and clean. Never finish
without having looked at a render of your FINAL SVG.
SVG requirements:
- A complete <svg> with explicit width/height AND a matching viewBox.
- White background (<rect> or style), dark strokes, sans-serif labels at generous sizes.
- One idea, fewest elements — sparse and large beats busy and tiny.
- Draw only what the brief specifies; never invent data, values, or shapes to fill space.
- Plain, clean styling: at most one accent color. An explanatory diagram, not art.
Your FINAL message: 1-2 sentences of notes for the teacher (what the figure shows, any caveats) —
NEVER the SVG source itself. If you genuinely cannot make a correct picture of the brief, say exactly
that in your final message and why.`;
export function createVisualTools() {
return {
write_svg: tool({
description: "Write the complete SVG source for the figure (overwrites any previous version).",
inputSchema: z.object({
svg: z.string().describe('Complete <svg xmlns="…">…</svg> markup.'),
}),
}),
render_svg: tool({
description: "Render the current SVG to a PNG and show it to you for inspection. No arguments.",
inputSchema: z.object({}),
}),
};
}
const LOOK_MESSAGE =
"Here is the current render of your SVG — inspect it critically (geometry, labels, clipping, readability):";
const UNLOOKED_FINISH_MESSAGE =
"You finished without looking at a render of your final SVG. Above is its render. If it is correct and clean, reply with just your notes; otherwise call write_svg with fixes and render again.";
export async function runVisualMaker(input: VisualMakerInput, opts: VisualMakerOptions = {}): Promise<VisualResult> {
const headless = opts.headless ?? getHeadless();
const brief = input.brief.trim() || "(no brief)";
const maxIterations = opts.maxIterations ?? 8;
const reason = headless.unavailableReason();
if (reason) {
return { svg: "", notes: `FAILED: ${reason}` };
}
const tools = createVisualTools();
const prompt = [
"Brief for ONE precise figure (from the teacher, verbatim — it is everything you know):",
brief,
"",
"Author the SVG, render it, look at it, iterate until correct, then finish with your notes.",
].join("\n");
let messages: ModelMessage[] = [{ role: "user", content: prompt }];
let currentSvg: string | null = null;
/** The svg shown in the most recent render (null = never rendered). */
let renderedSvg: string | null = null;
let finalText = "";
try {
for (let iteration = 0; iteration < maxIterations; iteration++) {
const result = await generateText({ model: opts.model ?? openrouter(DEFAULT_VISUAL_MODEL), system: INSTRUCTIONS, messages, tools });
messages = [...messages, ...result.response.messages];
const toolCalls = result.toolCalls;
if (toolCalls.length === 0) {
finalText = result.text.trim();
// Doctrine gate: never deliver an unlooked SVG. If the current SVG
// changed since its last render (or was never rendered), show it now
// and make the model look before accepting the finish.
if (currentSvg && renderedSvg !== currentSvg) {
let png: Buffer;
try {
png = await headless.renderSvg(currentSvg);
} catch (err) {
const msg = err instanceof Error ? err.message : String(err);
return { svg: "", notes: `FAILED: rendering the final SVG failed — ${msg}` };
}
opts.onActivity?.("render_svg: inspecting final render");
renderedSvg = currentSvg;
messages = [
...messages,
{ role: "user", content: [{ type: "file", data: png, mediaType: "image/png" }, { type: "text", text: UNLOOKED_FINISH_MESSAGE }] },
];
continue;
}
return finish(currentSvg, finalText);
}
const toolResults: ToolResultPart[] = [];
let renderPng: Buffer | null = null; // appended as the image message AFTER the tool results
for (const call of toolCalls) {
if (call.toolName === "write_svg") {
const svg = (call.input as { svg?: unknown }).svg;
if (typeof svg !== "string" || !svg.includes("<svg")) {
toolResults.push(toolText(call.toolCallId, call.toolName, "ERROR: expected a complete <svg>…</svg> document in the `svg` field."));
continue;
}
currentSvg = svg;
opts.onActivity?.(`write_svg: ${(svg.length / 1024).toFixed(1)} KB`);
toolResults.push(toolText(call.toolCallId, call.toolName, `Saved (${svg.length} bytes). Call render_svg to look at it.`));
} else if (call.toolName === "render_svg") {
if (!currentSvg) {
toolResults.push(toolText(call.toolCallId, call.toolName, "ERROR: nothing to render — call write_svg first."));
continue;
}
try {
const png = await headless.renderSvg(currentSvg);
renderPng = png;
renderedSvg = currentSvg;
opts.onActivity?.("render_svg: inspecting render");
toolResults.push(toolText(call.toolCallId, call.toolName, "Rendered — the PNG follows in the next message. Look at it."));
} catch (err) {
const msg = err instanceof Error ? err.message : String(err);
opts.onActivity?.("render_svg: render error");
toolResults.push(toolText(call.toolCallId, call.toolName, `RENDER ERROR (the SVG may be malformed): ${msg}. Fix it with write_svg and render again.`));
}
} else {
toolResults.push(toolText(call.toolCallId, call.toolName, `ERROR: unknown tool "${call.toolName}". Use write_svg or render_svg.`));
}
}
// Message order: assistant tool-calls → tool results → render image.
messages = [...messages, { role: "tool", content: toolResults }];
if (renderPng) {
messages = [
...messages,
{
role: "user",
content: [
{ type: "text", text: LOOK_MESSAGE },
{ type: "file", data: renderPng, mediaType: "image/png" },
],
},
];
}
}
} catch (err) {
const msg = err instanceof Error ? err.message : String(err);
return { svg: "", notes: `FAILED: the maker crashed — ${msg}` };
}
// Iteration limit: deliver the last written SVG only if it was looked at.
if (currentSvg && renderedSvg === currentSvg) {
return { svg: currentSvg, notes: finalText || "Iteration limit reached — delivering the last verified SVG." };
}
return { svg: "", notes: `FAILED: iteration limit (${maxIterations}) reached without a verified SVG.` };
}
function toolText(toolCallId: string, toolName: string, value: string): ToolResultPart {
return { type: "tool-result", toolCallId, toolName, output: { type: "text", value } };
}
function finish(svg: string | null, finalText: string): VisualResult {
if (!svg) {
return {
svg: "",
notes: `FAILED: the maker never wrote an SVG.${finalText ? ` (${finalText.slice(0, 300)})` : ""}`,
};
}
return { svg, notes: finalText || undefined };
}
+45 -2
View File
@@ -31,16 +31,59 @@ export const researchTool = tool({
inputSchema: researchInputSchema,
});
export const demoInputSchema = z.object({
title: z.string().describe("Short title for the demo card, e.g. 'Damped harmonic oscillator'."),
brief: z
.string()
.describe(
"Precise brief for the builder: the concept, the parameters/controls and what each does, the interactions, and the visual style. " +
"The builder sees NOTHING else of the lesson — include everything it needs, including necessary formulas and constants.",
),
});
export const demoTool = tool({
description:
"Dispatch a background sub-agent that BUILDS and headlessly TESTS an interactive teaching instrument — a single-file HTML " +
"demo (parameter sweep, simulation, animation, sampling) rendered live in the lesson. Use it when the concept is DYNAMIC: " +
"something to sweep, animate, or sample is better experienced than described. " +
"The call is ASYNC — your turn ENDS here, so say everything you want to say first; the tested demo arrives later as this " +
"tool's output and the lesson resumes with it. Write a precise brief: concept, controls, interactions, style.",
inputSchema: demoInputSchema,
});
export const visualInputSchema = z.object({
brief: z
.string()
.describe(
"Precise brief for ONE static figure: exactly what it must show — coordinates, labels, proportions, orientation. " +
"The maker sees NOTHING else of the lesson; a wrong or vague brief produces a wrong figure.",
),
});
export const visualTool = tool({
description:
"Dispatch a background sub-agent that hand-authors ONE precise SVG figure, renders it, LOOKS at the render, and iterates " +
"until correct. Use it for exact spatial pictures auto-layout diagrams can't do: coordinate geometry, number lines, vectors, " +
"function plots, physical layouts. For flows/relations/taxonomies write mermaid inline instead. " +
"The call is ASYNC — your turn ENDS here, so say everything you want to say first; the verified figure arrives later as this " +
"tool's output and the lesson resumes with it.",
inputSchema: visualInputSchema,
});
/**
* Per-request job tool registry. Context-free for now (the runner resolves
* Per-request job tool registry. Context-free (the runner resolves
* everything from the chat record by toolCallId) — the ctx parameter keeps
* the createTeacherTools factory shape stable for W6's demo/visual tools.
* the createTeacherTools factory shape stable.
*/
// eslint-disable-next-line @typescript-eslint/no-unused-vars
export function jobTools(ctx: TeacherToolContext): ToolSet {
return {
research: researchTool,
demo: demoTool,
visual: visualTool,
};
}
export type ResearchInput = z.infer<typeof researchInputSchema>;
export type DemoInput = z.infer<typeof demoInputSchema>;
export type VisualInput = z.infer<typeof visualInputSchema>;