fix: quiet Retrying lines, quiz option descriptions nuked, dividers at the markers

Three owner rulings from the gradient-descent session
(chat_muhskq6p_e8c3600b — 7 failed quiz calls, answers leaked in
option descriptions, planning divider floating at the transcript top):

1. RETRYING LINES (general): any tool call that errors back to the
   agent (validation failure, malformed args — the retry loop) renders
   one quiet "Retrying…" SystemLine instead of silence. Question tools
   previously rendered null, so a flailing quiz loop read as a stall.
   Aborts are terminal ("Operation aborted") and stay silent. Verified
   live: exactly 7 lines render at the 7 failure sites of the session.

2. QUIZ OPTION DESCRIPTIONS: NUKED. The model used them to telegraph
   correctness in every quiz of the session ("that moves uphill — it
   climbs f instead of descending it") — answerable by elimination.
   The field is gone from BOTH quiz schemas (probe + teach); ask keeps
   it (ungraded — nothing to leak). The dock additionally suppresses
   descriptions for quiz kind, so pre-fix records stop leaking too.
   Verified live: the session's pending quiz renders bare labels.

   Plus the diagnosed failure-mode fixes: GLM sometimes emits `options`
   as a JSON-encoded string (6 of 7 failures). prepareArguments hooks
   on all three question tools coerce it BEFORE pi's validation (the
   call lands instead of bouncing); the client coerces again in
   normalizeOptions (pi persists raw call args). Schema descriptions
   hardened ("REQUIRED ... never a JSON-encoded string").

3. DIVIDERS AT THE MARKERS: the probe-handoff SystemLine
   ("Preparing your plan…") IS NOW the Planning divider; the accepted
   verdict IS NOW the Teaching divider ("✓ Plan accepted" line gone) —
   one signal per seam, not two. The fold-based divider block is
   DELETED: Wave D grouping merged the whole probe into one giant
   assistant group, so the inclusive-fold comparison floated the
   divider to the transcript top (reproduced against the real record:
   6 view messages, phase flipped at the giant group). The markers
   render in the right spot by record construction. A DECLINED verdict
   is not a seam — it keeps the "↻ Plan revision requested" line.
   Verified live: plan divider after the probe concluded, teach
   divider after the plan presentation, nothing at the top.

Tests: 1004/1004 — string-coercion pins (normalizeOptions + the
prepareArguments hooks on all three tools, identity fast path),
quiz-schema-has-no-description pin (ask keeps it).
This commit is contained in:
amosblomqvist
2026-09-26 19:08:39 +03:00
parent b84ab1cdc0
commit 9853075331
6 changed files with 186 additions and 44 deletions
+43 -8
View File
@@ -30,19 +30,46 @@ export interface QuizOption {
// ─── tool input schemas (the question contract) ─────────────────────────────
// Parameter names mirror the prompt docs exactly — keep them in sync.
/** One answer option — shared by quiz + ask. */
const optionProps = {
label: Type.String({ description: "Display label for the answer option" }),
/** One quiz answer option. Owner ruling (2026-09-26): NO `description`
* field — the model reliably used it to leak correctness ("that's not the
* gradient", "close but…"), making quizzes answerable by elimination.
* Neutral elaboration belongs in the question text or the post-answer
* explanation; the option is a bare claim. */
const quizOptionProps = {
label: Type.String({ description: "Display label for the answer option — a bare claim, no reasoning" }),
value: Type.Optional(
Type.String({
description: "Optional machine-readable value returned for the option. Defaults to the label.",
}),
),
};
/** One ask answer option — keeps `description`: ask is ungraded preference
* collection, there is no correctness to leak. */
const askOptionProps = {
...quizOptionProps,
description: Type.Optional(
Type.String({ description: "Optional extra detail shown below the option" }),
),
};
/** GLM wobble guard (owner-approved, 2026-09-26): models sometimes emit
* `options` as a JSON-ENCODED STRING instead of an array. Parse it before
* use. Shared by the tools' prepareArguments (server, pre-validation) and
* normalizeOptions (client — the persisted call keeps the raw string, so
* the dock must coerce again). Returns the input untouched when it isn't
* a parseable array — validation then reports honestly. */
export function parseOptionsIfString<T>(options: string | T[] | undefined): unknown {
if (typeof options !== "string") return options;
try {
const parsed: unknown = JSON.parse(options);
if (Array.isArray(parsed)) return parsed;
} catch {
/* not JSON — fall through untouched */
}
return options;
}
// ─── cross-field authoring validation (tool-schema refinements) ──────────
// Enforced at the tool-call boundary (see the superRefines below): an invalid
// question NEVER reaches the learner — the SDK returns the validation error
@@ -130,10 +157,11 @@ export const quizInputBase = Type.Object({
description:
"The single quiz question to ask. ONE question per turn: ask, then end your turn — the answer determines the next question.",
}),
options: Type.Array(Type.Object(optionProps), {
options: Type.Array(Type.Object(quizOptionProps), {
minItems: 2,
description:
"The answer options (2 or more). Options only — there is no free-text mode. " +
"REQUIRED. The answer options (2 or more) as a JSON ARRAY of option objects — never a " +
"JSON-encoded string. Options only — there is no free-text mode. " +
"Give each option a stable `value`; you reference the correct one by that value in correctAnswer. " +
"Never add your own 'I don't know' option — that choice is added for you.",
}),
@@ -205,7 +233,7 @@ export const askInputSchema = Type.Object({
Type.String({ description: "Optional extra context or instructions shown under the question." }),
),
options: Type.Optional(
Type.Array(Type.Object(optionProps), {
Type.Array(Type.Object(askOptionProps), {
minItems: 2,
description:
"Optional multiple-choice options. Omit for free-form text input. " +
@@ -330,10 +358,17 @@ export const QUESTION_ANSWERED_ENTRY = "lh_question_answered";
// ─── options ────────────────────────────────────────────────────────────────
export function normalizeOptions(
options: Array<{ label: string; value?: string; description?: string }> | undefined,
options: string | Array<{ label: string; value?: string; description?: string }> | undefined,
): QuizOption[] {
const seen = new Set<string>();
return (options || [])
// The persisted tool call can carry GLM's string-encoded options —
// parseOptionsIfString is the same coercion the server's
// prepareArguments runs pre-validation (see parseOptionsIfString).
const coerced = parseOptionsIfString(options);
const list = Array.isArray(coerced)
? (coerced as Array<{ label: string; value?: string; description?: string }>)
: [];
return list
.map((option) => ({
label: option.label.trim(),
value: option.value?.trim() || option.label.trim(),