fix: parallel question calls are asked, saved, and answered in the model's call order

Live observation (modal-realism run): the teacher batched two quizzes in
one turn; the dock showed the SECOND question first, so the answering
order was reversed vs the saved record and the transcript cards.

Cause: findPending scanned parts BACKWARD within a message, so with two
input-available quizzes it picked the last one first. No overwriting was
involved — parts are created in call order on both wire and record; the
scan direction alone reversed the interaction order. (pi executes a
parallel batch with starts + result messages in call order; the
execution-end order is completion-racy, which also allowed a millisecond
dock-subject flip mid-turn.)

Fix A — src/core/pending.ts: parts now scan FORWARD within a message
(messages stay newest-first): the FIRST unanswered question is the dock
subject; the second appears only after the first is answered.

Fix B — src/agent/lh-tool.ts: question tools opt into sequential
execution (pi runs the batch in call order when any called tool opts
in), so tool-input-available chunks always arrive in call order and the
dock subject can never flip mid-turn. Question executes are instant.

Pinned in chat-flow.test.ts: a parallel two-quiz batch asserts wire
order (A's available chunk before B's), part persistence order, and the
dock selection sequence (A first, then B after A is answered).

Gates: tsc, lint, build, 307/307.
This commit is contained in:
amosblomqvist
2026-09-12 04:37:05 +03:00
parent 6af7a38684
commit 4baf8385be
3 changed files with 70 additions and 1 deletions
+6
View File
@@ -67,6 +67,12 @@ export function toPiTool(def: LhTool<any, any>): AgentTool<any, any> {
label: def.name,
description: def.description,
parameters: z.toJSONSchema(def.schema as z.ZodType) as never,
// Question tools opt into SEQUENTIAL execution: pi runs the whole
// batch in call order when any called tool opts in, so the
// tool-input-available chunks arrive in the model's call order — the
// dock's subject can never flip mid-turn between two parallel
// questions. Question executes are instant; no latency is lost.
executionMode: def.clientQuestion ? "sequential" : "parallel",
execute: async (toolCallId, args) => {
// The authoring check: structural JSON Schema validation already
// ran in the loop's prepare step; this re-run adds the refinements
+8 -1
View File
@@ -5,6 +5,13 @@ import { isToolUIPart, type UIMessage } from "ai";
* blocks the composer — one question at a time, mirroring the teacher's
* prompt contract. Derived client state; the server enforces nothing.
*
* Scan order: MESSAGES newest-first (a later turn's live question supersedes
* an older dangling one), but PARTS forward within a message — the model's
* FIRST unanswered question is the dock subject. Parallel question calls
* (pi executes them in one batch) are thus answered in the same order they
* were asked, saved, and rendered as cards; a backward part scan reversed
* that order (the live modal-realism observation).
*
* Job tool parts (research/demo/visual) never pend client-side — the async
* protocol resolves them server-side with an immediate ack (see
* contracts/jobs.ts "THE ASYNC PROTOCOL"), so job tools are NOT returned
@@ -26,7 +33,7 @@ export const JOB_TOOLS = ["research", "demo", "visual"] as const;
function findPending(messages: UIMessage[], names: readonly string[]): PendingToolCall | null {
for (let m = messages.length - 1; m >= 0; m--) {
const msg = messages[m]!;
for (let p = msg.parts.length - 1; p >= 0; p--) {
for (let p = 0; p < msg.parts.length; p++) {
const part = msg.parts[p]!;
if (isToolUIPart(part) && part.state === "input-available") {
const tool = part.type.slice("tool-".length);
+56
View File
@@ -355,6 +355,62 @@ describe("teacher chat flow (scripted model)", () => {
const pending = findPendingToolCall(saved.messages);
assert.equal(pending?.toolCallId, "call-clean", "the retried quiz is the pending one");
});
it("parallel quiz calls are asked, saved, and answered in the model's call order", async () => {
// The live modal-realism observation: the teacher batched two quizzes
// in one turn, and the dock showed the SECOND one first (the backward
// part scan), so the interaction order was reversed vs the record.
// Pins: (A) findPendingToolCall picks the FIRST unanswered question in
// part order; (B) question tools execute sequentially, so their
// tool-input-available chunks arrive in call order on the wire.
const QUIZ_A = { ...QUIZ_INPUT, question: "Quiz A: which planet is closest to the sun?" };
const QUIZ_B = { ...QUIZ_INPUT, question: "Quiz B: which planet is largest?" };
const { engine } = scriptedTeacher(() => [
() =>
fauxAssistantMessage(
[fauxToolCall("quiz", QUIZ_A, { id: "call-a" }), fauxToolCall("quiz", QUIZ_B, { id: "call-b" })],
{ stopReason: "toolUse" },
),
]);
const chat = chatStore.createChat("Test: parallel quizzes");
createdChats.push(chat.id);
const firstUser = { id: "u1", role: "user", parts: [{ type: "text", text: "quiz me twice" }] };
const body = await drain(await handleChatRequest(post(chat.id, [firstUser]), agentFactory(engine)));
// (B) wire determinism: the batches' input-available chunks in call order.
const availA = body.indexOf('"type":"tool-input-available","toolCallId":"call-a"');
const availB = body.indexOf('"type":"tool-input-available","toolCallId":"call-b"');
assert.ok(availA !== -1 && availB !== -1, "both quizzes became available");
assert.ok(availA < availB, "call A's input became available before call B's (sequential execution)");
// Saved in call order.
const saved = chatStore.getChat(chat.id)!;
const quizIds = saved.messages[1]!.parts
.filter((p) => p.type === "tool-quiz")
.map((p) => (p as { toolCallId: string }).toolCallId);
assert.deepEqual(quizIds, ["call-a", "call-b"], "parts persisted in call order");
// (A) the dock serializes answering in the same order: A first, then B.
const pending = findPendingToolCall(saved.messages);
assert.equal(pending?.toolCallId, "call-a", "the FIRST asked question is the dock subject");
// …the learner answers A (client-side grading seam — state flip only here)…
const answered = saved.messages.map((m) =>
(m as { id?: string }).id === saved.messages[1]!.id
? {
...m,
parts: (m as { parts: unknown[] }).parts.map((p) =>
(p as { toolCallId?: string }).toolCallId === "call-a"
? { ...(p as object), state: "output-available", output: { correct: true } }
: p,
),
}
: m,
);
const next = findPendingToolCall(answered as UIMessage[]);
assert.equal(next?.toolCallId, "call-b", "the SECOND question becomes the dock subject only after the first is answered");
});
});
describe("chat store", () => {