tools: the standard argument heal — every registry, both seams
The whitehead incident (2026-10-09): glm-5.3 emitted quiz type with
quotes baked in ('"predict"') — rejected ten times, byte-identical
retries, the lesson never started — and correctAnswer with the JSON
key-value colon glued on ([':parmenides']). Third GLM wobble incident,
and each prior heal was wired only where its incident landed; this one
landed precisely where the practice heal never reached.
Layer 1 — healEncodedArguments moves to core/argument-heal (the display
seam needs it too — the tool-parts law) with withArgumentHeal +
healToolRecord (agent/tools/heal) wired at EVERY registry: prober,
teacher, practice, bareQuestionTools (it had no prepareArguments at
all), demo-builder, and submit-agent (composing after
healStringifiedArrays; the auditor's extraTools wrapped).
Layer 2 — stripModelValueDebris (core/value-debris): leading/trailing
colons, quotes, backticks, whitespace strip at the MEMBERSHIP seams —
coerceCorrectAnswer (boundary check + display + grading in one),
normalizeQuizType (persisted wobbled records keep their role),
resolveTargetId (copy-syntax on targets).
Layer 3 — schema description hygiene: the quiz type and practice
mode/outcome/rung/missClass values now read as bare words — the quoted
JSON-key style invited quote-copying.
Display mirror — parseQuizInput heals the raw record on read, so a
boundary-healed quiz whose record keeps raw args renders instead of
becoming an invisible pending quiz (learner lockout).
Pins: tests/tool-heal.test.ts (the incident shapes end-to-end through
pi-ai validation, the wiring, the strips, the mirror). The stringy-
options pins flip to boundary-parity per tool-parts' own law; the
chat-flow garbage pin keeps its honest-reject intent with an unhealable
shape. 1471 tests, typecheck, lint green.
This commit is contained in:
amosblomqvist
+10
-10
@@ -689,20 +689,20 @@ describe("teacher chat flow (scripted model)", () => {
|
||||
assert.deepEqual(pending?.input, QUIZ_INPUT);
|
||||
});
|
||||
|
||||
it("a structurally invalid quiz (the live double-encoded options string) never reaches the dock and retries clean", async () => {
|
||||
// The exact live modal-realism crash shape: GLM occasionally serializes
|
||||
// a nested array as a JSON STRING inside the arguments. The structural
|
||||
// layer (JSON Schema: options must be array) rejects the call at the
|
||||
// boundary — it must surface as tool-input-error, never as a pending
|
||||
// question, and the model's retry must become the pending quiz.
|
||||
it("structurally invalid quiz input (unhealable garbage) never reaches the dock and retries clean", async () => {
|
||||
// The two-tier world (owner, 2026-10-09 — the universal argument
|
||||
// heal): a JSON-ENCODED stand-in for the schema shape (the live
|
||||
// string-options habit) now heals at EVERY registry — see
|
||||
// tests/tool-heal.test.ts for that pin — so the structural reject
|
||||
// this test holds is the OTHER tier: garbage no mechanical rule
|
||||
// could rescue (not JSON, not an array) rejects honestly at the
|
||||
// boundary — tool-input-error, never a pending question — and the
|
||||
// model's clean retry becomes the pending quiz.
|
||||
const STRINGY_QUIZ_INPUT = {
|
||||
question: "Lewis's indexical analysis of 'actual' — which is it?",
|
||||
type: "check",
|
||||
nodeId: "indexicals",
|
||||
options: JSON.stringify([
|
||||
{ label: "Like 'I'", value: "indexical" },
|
||||
{ label: "Like a title", value: "unique" },
|
||||
]), // ← the bug: a string, not an array
|
||||
options: "definitely not a JSON array — denim all the way down", // ← unhealable garbage
|
||||
correctAnswer: "indexical",
|
||||
explanation: "Indexical: every world is actual to its own inhabitants.",
|
||||
};
|
||||
|
||||
@@ -16,11 +16,8 @@
|
||||
|
||||
import assert from "node:assert/strict";
|
||||
import { describe, it } from "node:test";
|
||||
import {
|
||||
concludeInputObject,
|
||||
healEncodedArguments,
|
||||
problemInputObject,
|
||||
} from "../src/agent/tools/practice.ts";
|
||||
import { concludeInputObject, problemInputObject } from "../src/agent/tools/practice.ts";
|
||||
import { healEncodedArguments } from "../src/core/argument-heal.ts";
|
||||
import { validateToolArguments } from "@earendil-works/pi-ai";
|
||||
|
||||
/** The incident's exact mode spelling: the string "recall" INCLUDING
|
||||
|
||||
@@ -0,0 +1,168 @@
|
||||
/**
|
||||
* THE WHITEHEAD INCIDENT PINS (owner, 2026-10-09): glm-5.3 (z-ai via
|
||||
* OpenRouter) emitted two argument-wobble shapes in one real session —
|
||||
* quiz `type: "\"predict\""` (the literal with quotes baked in; the
|
||||
* schema's own description displayed the values quoted+colon) rejected
|
||||
* TEN times with byte-identical retries while the lesson never started,
|
||||
* and `correctAnswer: [":parmenides"]` (the option value with the JSON
|
||||
* key-value colon glued on) rejected four times.
|
||||
*
|
||||
* The fixes these pins hold:
|
||||
* 1. THE STANDARD HEAL EVERYWHERE — healToolRecord wraps every registry
|
||||
* (bareQuestionTools carried NO prepareArguments at all), and the
|
||||
* quote-encoded literal heals before TypeBox validation.
|
||||
* 2. THE DEBRIS STRIP AT MEMBERSHIP SEAMS — stripModelValueDebris in
|
||||
* coerceCorrectAnswer (boundary check + grading), normalizeQuizType,
|
||||
* and resolveTargetId.
|
||||
* 3. THE DISPLAY MIRROR — the record keeps RAW arguments, so
|
||||
* parseQuizInput heals the same wobble on read (the tool-parts law:
|
||||
* a healed-at-boundary quiz whose display parse rejects is an
|
||||
* invisible pending quiz and a learner lockout).
|
||||
*/
|
||||
|
||||
import assert from "node:assert/strict";
|
||||
import { describe, it } from "node:test";
|
||||
import { validateToolArguments } from "@earendil-works/pi-ai";
|
||||
import { healEncodedArguments } from "../src/core/argument-heal.ts";
|
||||
import { stripModelValueDebris } from "../src/core/value-debris.ts";
|
||||
import {
|
||||
coerceCorrectAnswer,
|
||||
normalizeQuizType,
|
||||
quizInputObject,
|
||||
quizInputProblems,
|
||||
} from "../src/core/question-core.ts";
|
||||
import { parseQuizInput } from "../src/core/tool-parts.ts";
|
||||
import { resolveTargetId } from "../src/core/chains.ts";
|
||||
import { bareQuestionTools } from "../src/agent/teacher-agent.ts";
|
||||
|
||||
/** The incident's exact encoding: the value wrapped in literal quotes. */
|
||||
function QUOTED(value: string): string {
|
||||
return `"${value}"`;
|
||||
}
|
||||
|
||||
// A complete, well-formed teach-quiz input — the incident's call with
|
||||
// every field valid BUT the wobbled one.
|
||||
function whiteheadQuiz(type: unknown, correctAnswer: unknown) {
|
||||
return {
|
||||
question: "What was Parmenides' claim?",
|
||||
options: [
|
||||
{ label: "Being is changeless and complete", value: "parmenides" },
|
||||
{ label: "Becoming is fundamental", value: "heraclitus" },
|
||||
{ label: "Split world", value: "split" },
|
||||
{ label: "Both unreal", value: "both-illusion" },
|
||||
],
|
||||
explanation: "What truly *is* cannot come to be or pass away — the flux is appearance.",
|
||||
correctAnswer,
|
||||
type,
|
||||
nodeId: "experience-first",
|
||||
};
|
||||
}
|
||||
|
||||
describe("the death-loop shape — quiz type with quotes baked in", () => {
|
||||
it("heals the quote-encoded literal (the ten-retry loop)", () => {
|
||||
const healed = healEncodedArguments({ type: QUOTED("predict") }, quizInputObject) as {
|
||||
type: unknown;
|
||||
};
|
||||
assert.equal(healed.type, "predict");
|
||||
});
|
||||
|
||||
it("END-TO-END: the healed args pass pi-ai validation where the raw threw", () => {
|
||||
const raw = whiteheadQuiz(QUOTED("predict"), ["parmenides"]);
|
||||
assert.throws(() => {
|
||||
validateToolArguments({ name: "quiz", parameters: quizInputObject } as never, {
|
||||
id: "call_x",
|
||||
type: "toolCall",
|
||||
name: "quiz",
|
||||
arguments: raw,
|
||||
} as never);
|
||||
});
|
||||
const validated = validateToolArguments(
|
||||
{ name: "quiz", parameters: quizInputObject } as never,
|
||||
{
|
||||
id: "call_x",
|
||||
type: "toolCall",
|
||||
name: "quiz",
|
||||
arguments: healEncodedArguments(raw, quizInputObject),
|
||||
} as never,
|
||||
);
|
||||
assert.equal((validated as { type: unknown }).type, "predict");
|
||||
});
|
||||
|
||||
it("WIRING: bareQuestionTools (the registry with NO prepareArguments) now heals", () => {
|
||||
const tools = bareQuestionTools();
|
||||
const quiz = tools.quiz;
|
||||
assert.ok(quiz, "the bare quiz must exist");
|
||||
assert.ok(quiz.prepareArguments, "the bare quiz must carry the standard heal");
|
||||
const healed = quiz.prepareArguments!(
|
||||
whiteheadQuiz(QUOTED("predict"), ["parmenides"]),
|
||||
) as { type: unknown };
|
||||
assert.equal(healed.type, "predict");
|
||||
});
|
||||
|
||||
it("WIRING: the 2026-09-26 string-options shape heals through the bare quiz too", () => {
|
||||
// The universal heal covers the ORIGINAL incident's shape as well:
|
||||
// options serialized as a JSON string (then healed only on the
|
||||
// plan-tree quiz path; the bare registries had nothing).
|
||||
const quiz = bareQuestionTools().quiz;
|
||||
assert.ok(quiz?.prepareArguments, "the bare quiz must carry the standard heal");
|
||||
const input = whiteheadQuiz("predict", ["parmenides"]);
|
||||
const stringified = { ...input, options: JSON.stringify(input.options) };
|
||||
const healed = quiz.prepareArguments!(stringified) as { options: unknown };
|
||||
assert.ok(Array.isArray(healed.options), "the JSON string heals to the array");
|
||||
assert.equal(
|
||||
(healed.options as Array<{ value?: string }>)[0]?.value,
|
||||
"parmenides",
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe("the colon shape — correctAnswer with copy-syntax debris", () => {
|
||||
it("stripModelValueDebris: the observed shapes heal, interiors stay", () => {
|
||||
assert.equal(stripModelValueDebris(":parmenides"), "parmenides");
|
||||
assert.equal(stripModelValueDebris(QUOTED("predict")), "predict");
|
||||
assert.equal(stripModelValueDebris(`: "primary-external" `), "primary-external");
|
||||
assert.equal(stripModelValueDebris("`value`"), "value");
|
||||
assert.equal(stripModelValueDebris("a:b"), "a:b"); // interior colon is content
|
||||
assert.equal(stripModelValueDebris('["x"]'), '["x"]'); // interior quotes are content (JSON-array strings)
|
||||
});
|
||||
|
||||
it("coerceCorrectAnswer strips every entry (boundary check + grading seam)", () => {
|
||||
assert.deepEqual(coerceCorrectAnswer([":parmenides"]), ["parmenides"]);
|
||||
assert.deepEqual(coerceCorrectAnswer(QUOTED("primary-external")), ["primary-external"]);
|
||||
});
|
||||
|
||||
it("the incident's exact call passes the semantic boundary (quizInputProblems)", () => {
|
||||
const input = whiteheadQuiz("predict", [":parmenides"]) as Parameters<
|
||||
typeof quizInputProblems
|
||||
>[0];
|
||||
assert.deepEqual(quizInputProblems(input), []);
|
||||
});
|
||||
|
||||
it("normalizeQuizType heals a persisted wobbled record", () => {
|
||||
assert.equal(normalizeQuizType(QUOTED("predict")), "predict");
|
||||
assert.equal(normalizeQuizType('"check"'), "check");
|
||||
assert.equal(normalizeQuizType(undefined), undefined);
|
||||
assert.equal(normalizeQuizType("bogus"), undefined);
|
||||
});
|
||||
|
||||
it("resolveTargetId heals copy-syntax on targets", () => {
|
||||
const nodes = [{ id: "experience-first", name: "Experience first" }];
|
||||
assert.equal(resolveTargetId(":experience-first", nodes), "experience-first");
|
||||
assert.equal(resolveTargetId('"Experience first"', nodes), "experience-first");
|
||||
assert.equal(resolveTargetId("nope", nodes), null);
|
||||
});
|
||||
});
|
||||
|
||||
describe("the display mirror — raw records heal on read", () => {
|
||||
it("parseQuizInput renders the incident's persisted raw record (no invisible quiz)", () => {
|
||||
const raw = whiteheadQuiz(QUOTED("predict"), [":parmenides"]);
|
||||
const parsed = parseQuizInput(raw);
|
||||
assert.ok(parsed, "the wobbled record must parse — else the quiz is invisible");
|
||||
// The display schema's type is DELIBERATELY a lenient string (old
|
||||
// and probe records carry none); the read seam is normalizeQuizType —
|
||||
// which strips the recorded wobble to the intended role.
|
||||
assert.equal(normalizeQuizType(parsed?.type), "predict");
|
||||
// The graded answer resolves to the option despite the recorded colon.
|
||||
assert.deepEqual(parsed ? coerceCorrectAnswer(parsed.correctAnswer) : [], ["parmenides"]);
|
||||
});
|
||||
});
|
||||
@@ -73,9 +73,18 @@ describe("parseQuizInput", () => {
|
||||
assert.ok(parseQuizInput(PROBER_QUIZ));
|
||||
});
|
||||
|
||||
it("the live crash shape — options as a JSON string — does not parse", () => {
|
||||
it("boundary parity: the 2026-09-26 string-options shape heals on read (universal-heal mirror, 2026-10-09)", () => {
|
||||
// The boundary now heals a JSON-encoded options string at EVERY
|
||||
// registry (healToolRecord, owner 2026-10-09 — the bare-quiz sets
|
||||
// carried no prepareArguments at all), and the record keeps the RAW
|
||||
// args — so the display parse heals the same shape, per this file's
|
||||
// own parity law: anything the boundary accepts, the display parse
|
||||
// accepts, or the posed quiz renders nowhere while the turn has
|
||||
// ended = learner lockout.
|
||||
const stringified = { ...PROBER_QUIZ, options: JSON.stringify(PROBER_QUIZ.options) };
|
||||
assert.equal(parseQuizInput(stringified), undefined);
|
||||
const parsed = parseQuizInput(stringified);
|
||||
assert.ok(parsed, "the healed-at-boundary record must parse on read");
|
||||
assert.deepEqual(parsed.options, PROBER_QUIZ.options);
|
||||
});
|
||||
|
||||
it("boundary parity under Value.Convert: a NUMERIC correctAnswer parses (live-smoke lockout fix, 2026-09-26)", () => {
|
||||
|
||||
Reference in New Issue
Block a user