sorokin.oleg.edu/learnharness
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 6 Wiki Activity

fix: quiet Retrying lines, quiz option descriptions nuked, dividers at the markers

Browse Source 
Three owner rulings from the gradient-descent session
(chat_muhskq6p_e8c3600b — 7 failed quiz calls, answers leaked in
option descriptions, planning divider floating at the transcript top):

1. RETRYING LINES (general): any tool call that errors back to the
   agent (validation failure, malformed args — the retry loop) renders
   one quiet "Retrying…" SystemLine instead of silence. Question tools
   previously rendered null, so a flailing quiz loop read as a stall.
   Aborts are terminal ("Operation aborted") and stay silent. Verified
   live: exactly 7 lines render at the 7 failure sites of the session.

2. QUIZ OPTION DESCRIPTIONS: NUKED. The model used them to telegraph
   correctness in every quiz of the session ("that moves uphill — it
   climbs f instead of descending it") — answerable by elimination.
   The field is gone from BOTH quiz schemas (probe + teach); ask keeps
   it (ungraded — nothing to leak). The dock additionally suppresses
   descriptions for quiz kind, so pre-fix records stop leaking too.
   Verified live: the session's pending quiz renders bare labels.

   Plus the diagnosed failure-mode fixes: GLM sometimes emits `options`
   as a JSON-encoded string (6 of 7 failures). prepareArguments hooks
   on all three question tools coerce it BEFORE pi's validation (the
   call lands instead of bouncing); the client coerces again in
   normalizeOptions (pi persists raw call args). Schema descriptions
   hardened ("REQUIRED ... never a JSON-encoded string").

3. DIVIDERS AT THE MARKERS: the probe-handoff SystemLine
   ("Preparing your plan…") IS NOW the Planning divider; the accepted
   verdict IS NOW the Teaching divider ("✓ Plan accepted" line gone) —
   one signal per seam, not two. The fold-based divider block is
   DELETED: Wave D grouping merged the whole probe into one giant
   assistant group, so the inclusive-fold comparison floated the
   divider to the transcript top (reproduced against the real record:
   6 view messages, phase flipped at the giant group). The markers
   render in the right spot by record construction. A DECLINED verdict
   is not a seam — it keeps the "↻ Plan revision requested" line.
   Verified live: plan divider after the probe concluded, teach
   divider after the plan presentation, nothing at the top.

Tests: 1004/1004 — string-coercion pins (normalizeOptions + the
prepareArguments hooks on all three tools, identity fast path),
quiz-schema-has-no-description pin (ask keeps it).
This commit is contained in:
amosblomqvist
2026-09-26 19:08:39 +03:00
parent b84ab1cdc0
commit 9853075331
6 changed files with 186 additions and 44 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/agent/tools.ts
+24
View File
@@ -4,6 +4,7 @@ import { questionResult, serverExecute, ToolInputError } from "./tool-execute.ts
import {
askInputSchema,
askOptionProblems,
parseOptionsIfString,
probeQuizInputSchema,
quizInputObject,
quizInputProblems,
@@ -51,6 +52,26 @@ export type { ProbeDoneContext } from "./tools/probe.ts";
* only.
*/
/** GLM wobble guard (owner-approved, 2026-09-26): models sometimes emit
* `options` as a JSON-encoded string instead of an array; pi runs this
* BEFORE schema validation, so a double-encoded call lands instead of
* bouncing back as a validation error + wasted retry turn. NOTE: pi
* persists the tool call with its RAW arguments — the client coerces
* again in normalizeOptions (question-core), the same parseOptionsIfString.
* Anything unparseable passes through untouched: validation then reports
* honestly and the agent retries with a real error to learn from. */
function prepareQuestionArguments<T extends object>(args: unknown): T {
if (args && typeof args === "object" && !Array.isArray(args)) {
const o = args as { options?: unknown };
if (typeof o.options === "string") {
const parsed = parseOptionsIfString(o.options);
if (parsed !== o.options) return { ...o, options: parsed } as T;
}
return o as T;
}
return args as T;
}
export const askUserQuestionTool: AgentTool<typeof askInputSchema> = {
name: "ask_user_question",
label: "ask_user_question",
@@ -60,6 +81,7 @@ export const askUserQuestionTool: AgentTool<typeof askInputSchema> = {
"or you need an open-ended response. ONE question per turn: call a single ask_user_question, then end your " +
"turn — his answer determines the next.",
parameters: askInputSchema,
prepareArguments: prepareQuestionArguments,
executionMode: "sequential",
execute: async (_toolCallId, input) => {
const problems = askOptionProblems(input.options ?? []);
@@ -85,6 +107,7 @@ export const probeQuizTool: AgentTool<typeof probeQuizInputSchema> = {
"questions use ask_user_question instead. ONE question per turn: call a single quiz, then end your turn — " +
"his answer determines the next question to ask.",
parameters: probeQuizInputSchema,
prepareArguments: prepareQuestionArguments,
executionMode: "sequential",
execute: async (_toolCallId, input) => {
const problems = quizInputProblems(input);
@@ -206,6 +229,7 @@ export function teachQuizToolFor(ctx: Pick<PlanToolContext, "chatId" | "chatStor
"only when non-empty. No free-text answers — for non-graded questions use ask_user_question instead. " +
"ONE question per turn: call a single quiz, then end your turn — his answer determines the next.",
parameters: quizInputObject,
prepareArguments: prepareQuestionArguments,
executionMode: "sequential",
execute: async (_toolCallId, input) => {
const problems = [...quizInputProblems(input)];
src/core/question-core.ts
+43 -8
View File
@@ -30,19 +30,46 @@ export interface QuizOption {
// ─── tool input schemas (the question contract) ─────────────────────────────
// Parameter names mirror the prompt docs exactly — keep them in sync.
/** One answer option — shared by quiz + ask. */
const optionProps = {
label: Type.String({ description: "Display label for the answer option" }),
/** One quiz answer option. Owner ruling (2026-09-26): NO `description`
* field — the model reliably used it to leak correctness ("that's not the
* gradient", "close but…"), making quizzes answerable by elimination.
* Neutral elaboration belongs in the question text or the post-answer
* explanation; the option is a bare claim. */
const quizOptionProps = {
label: Type.String({ description: "Display label for the answer option — a bare claim, no reasoning" }),
value: Type.Optional(
Type.String({
description: "Optional machine-readable value returned for the option. Defaults to the label.",
}),
),
};
/** One ask answer option — keeps `description`: ask is ungraded preference
* collection, there is no correctness to leak. */
const askOptionProps = {
...quizOptionProps,
description: Type.Optional(
Type.String({ description: "Optional extra detail shown below the option" }),
),
};
/** GLM wobble guard (owner-approved, 2026-09-26): models sometimes emit
* `options` as a JSON-ENCODED STRING instead of an array. Parse it before
* use. Shared by the tools' prepareArguments (server, pre-validation) and
* normalizeOptions (client — the persisted call keeps the raw string, so
* the dock must coerce again). Returns the input untouched when it isn't
* a parseable array — validation then reports honestly. */
export function parseOptionsIfString<T>(options: string | T[] | undefined): unknown {
if (typeof options !== "string") return options;
try {
const parsed: unknown = JSON.parse(options);
if (Array.isArray(parsed)) return parsed;
} catch {
/* not JSON — fall through untouched */
}
return options;
}
// ─── cross-field authoring validation (tool-schema refinements) ──────────
// Enforced at the tool-call boundary (see the superRefines below): an invalid
// question NEVER reaches the learner — the SDK returns the validation error
@@ -130,10 +157,11 @@ export const quizInputBase = Type.Object({
description:
"The single quiz question to ask. ONE question per turn: ask, then end your turn — the answer determines the next question.",
}),
options: Type.Array(Type.Object(optionProps), {
options: Type.Array(Type.Object(quizOptionProps), {
minItems: 2,
description:
"The answer options (2 or more). Options only — there is no free-text mode. " +
"REQUIRED. The answer options (2 or more) as a JSON ARRAY of option objects — never a " +
"JSON-encoded string. Options only — there is no free-text mode. " +
"Give each option a stable `value`; you reference the correct one by that value in correctAnswer. " +
"Never add your own 'I don't know' option — that choice is added for you.",
}),
@@ -205,7 +233,7 @@ export const askInputSchema = Type.Object({
Type.String({ description: "Optional extra context or instructions shown under the question." }),
),
options: Type.Optional(
Type.Array(Type.Object(optionProps), {
Type.Array(Type.Object(askOptionProps), {
minItems: 2,
description:
"Optional multiple-choice options. Omit for free-form text input. " +
@@ -330,10 +358,17 @@ export const QUESTION_ANSWERED_ENTRY = "lh_question_answered";
// ─── options ────────────────────────────────────────────────────────────────
export function normalizeOptions(
options: Array<{ label: string; value?: string; description?: string }> | undefined,
options: string | Array<{ label: string; value?: string; description?: string }> | undefined,
): QuizOption[] {
const seen = new Set<string>();
return (options || [])
// The persisted tool call can carry GLM's string-encoded options —
// parseOptionsIfString is the same coercion the server's
// prepareArguments runs pre-validation (see parseOptionsIfString).
const coerced = parseOptionsIfString(options);
const list = Array.isArray(coerced)
? (coerced as Array<{ label: string; value?: string; description?: string }>)
: [];
return list
.map((option) => ({
label: option.label.trim(),
value: option.value?.trim() || option.label.trim(),