practice — formatting doctrine for generator + grader (owner-approved 2026-09-29)

Prior state: zero formatting guidance anywhere in the agent module —
models wrote prose-only by default.

GENERATOR_DOCTRINE Part III gains the FORMATTING bullet (after the
THE PARAMETERS bullet, before PREVIOUS PROBLEMS): statement, solution,
and rubric render as full Markdown — math always LaTeX (inline $…$,
display $$…$$), code/pseudocode/formal systems in fenced blocks with
a language tag, identifiers inline as `code`, tables for parallel
structure, lists for enumerations; prose-only renderings of
mathematical or structured content are a defect.

GRADER_INSTRUCTIONS gains the feedback-formatting paragraph (before
the OUTPUT section): full Markdown, LaTeX math, fenced code, lists/
tables where they carry structure — teach the boundary in properly
formatted prose, not plain text.

PARTITIONER gets nothing (machine-shaped output).

Tests: substring pins on the exported doctrine constants — the exact
FORMATTING sentence + ordering (after presentation, before previous
problems), the exact grader paragraph + before-OUTPUT ordering.
Architecture.md §7 one clause (the formatting doctrine in the
generator's prompt inventory).

1144/1144, tsc 0, lint 0, build 0.
This commit is contained in:
amosblomqvist
2026-09-29 20:21:12 +03:00
parent 8d39f3717c
commit 25cc2a238a
3 changed files with 47 additions and 3 deletions
+1 -1
View File
@@ -376,7 +376,7 @@ Practice turns the graph into problem sets the learner reasons through out loud.
**The three axes are discretized named levels** (owner, 2026-09-29 — never a number to interpolate; there is no continuum between levels): the **ladder rung** (the main axis — 0 Worked example, 1 Completion, 2 Direct exercise, 3 Composition, 4 Path-finding; the invocation names the rung and the generator builds exactly that CLASS of problem), the **cue distance** (0 Lesson context — the textbook/toy-example shape in the teaching's own framing; 1 Adjacent context — a genuinely new situation built from the teaching's own world; 2 Far context — the edge of the recognizable band, where the proposition must still genuinely apply and be findable by looking), and the **presentation** (Theory — the subject's own terms, or Application — a concrete worldly affair in which the proposition's structure appears in the wild; it dresses the situation, never changes the rung or the cue distance). The names + one-line definitions live in one place (`src/core/practice.ts`'s level tables) and are spoken by everything: the agents' prompts (`renderParams`), the creation dialog's stepped controls, the exam header's levels line. Creation defaults: rung 2 Direct exercise · cue 1 Adjacent context · Theory. The API validates the exact unions — a float between rungs is a 400; legacy float-params sets fail the store validator and are skipped (no migration — `data/practice` is disposable scratch).
**Two-stage generation.** One **partitioner** agent reads the chains plus the three NAMED-LEVEL parameters and submits the true partition; each partition becomes one problem's target. Then **one generator agent per problem**: its prompt carries the problem-engineering framework verbatim (the owner's doctrine — the three distances, the search-to-solution axis, the rung ladder, reverse/discriminate/probe-the-premises, the standing prohibitions), **YOUR TARGET** (the assigned chains echoed verbatim — the echo law: the submit's chain set must equal the assignment exactly), and **THE FULL SET** (the whole selection: outside propositions are supporting material the generator may draw on with minimal use and maximal focus; the never-announce law covers ALL propositions). The submit is commit-before-pose — statement, solution, and rubric arrive together (the same sycophancy mitigation the quiz/explain mechanic uses); a problem never exists without its pre-written answer. Generators run in parallel (a process-wide 3-permit semaphore shared with grading) with per-problem failure isolation: one bad problem errors alone; the set still reaches `ready`. **Dedup across sets**: each generator also sees the problems of the ≤5 most recent sets sharing a node, so situations don't repeat between sets; within a set, parallel generators can't see each other — accepted for V1 (owner ruling).
**Two-stage generation.** One **partitioner** agent reads the chains plus the three NAMED-LEVEL parameters and submits the true partition; each partition becomes one problem's target. Then **one generator agent per problem**: its prompt carries the problem-engineering framework verbatim (the owner's doctrine — the three distances, the search-to-solution axis, the rung ladder, reverse/discriminate/probe-the-premises, the standing prohibitions, the formatting doctrine — full Markdown, LaTeX math, fenced code, tables/lists; prose-only renderings of structured content are a defect), **YOUR TARGET** (the assigned chains echoed verbatim — the echo law: the submit's chain set must equal the assignment exactly), and **THE FULL SET** (the whole selection: outside propositions are supporting material the generator may draw on with minimal use and maximal focus; the never-announce law covers ALL propositions). The submit is commit-before-pose — statement, solution, and rubric arrive together (the same sycophancy mitigation the quiz/explain mechanic uses); a problem never exists without its pre-written answer. Generators run in parallel (a process-wide 3-permit semaphore shared with grading) with per-problem failure isolation: one bad problem errors alone; the set still reaches `ready`. **Dedup across sets**: each generator also sees the problems of the ≤5 most recent sets sharing a node, so situations don't repeat between sets; within a set, parallel generators can't see each other — accepted for V1 (owner ruling).
**Grading is feedback-only.** Submitting answers fires one **grader** job per answered problem: the grader sees the problem, the propositions its chains touch, the pre-written solution and rubric, and the learner's answer — never the full set (minimal, by ruling). It returns a verdict (`solid`/`partial`/`missed`) and feedback that names the failed recall and teaches the boundary. The verdict lands on the problem and the UI reveals it with the solution and rubric; the answer locks. Nothing reaches the graph, no level, no count, anywhere. A grader failure leaves the answer gradeless and re-submittable.
+5 -2
View File
@@ -40,7 +40,7 @@ import { chainCoverage, CUE_LEVELS, LADDER_RUNGS, PRESENTATIONS } from "../../co
// ─── doctrines ───────────────────────────────────────────────────────────────
const GENERATOR_DOCTRINE = `# The problem engineering framework
export const GENERATOR_DOCTRINE = `# The problem engineering framework
## I. Theory
@@ -188,6 +188,7 @@ INPUTS (in the user message):
THE CUE DISTANCE: 0 Lesson context · 1 Adjacent context · 2 Far context — how far the situation sits from the cues the learner has already met: the lesson's own examples AND every previous problem's situation. Far context is the edge of the recognizable band — the proposition must still genuinely apply, and the learner must be able to find it by looking.
THE PRESENTATION: Theory (the subject's own terms — symbols, formal objects, native vocabulary) or Application (a concrete worldly affair in which the proposition's structure appears in the wild). It dresses the situation; it never changes the rung or the cue distance.
Apply Part II step 2: degrade honestly if the target cannot support them — the degradation shows in the problem's own shape, never in meta-commentary.
- FORMATTING: your statement, solution, and rubric are rendered as full Markdown — use everything it offers. Mathematics is always LaTeX: inline as $…$, display as $$…$$ on its own lines. Code, pseudocode, and formal systems live in fenced code blocks with a language tag; identifiers and symbols inline as \`code\`. Use tables for parallel structure and comparisons, lists for enumerations. Prose-only renderings of mathematical or structured content are a defect: if a sentence carries an equation, a derivation, or a piece of code, format it as what it is.
- PREVIOUS PROBLEMS: problems the learner has already answered, with the chains they exercised. Do not rebuild one; keep your situation distant from theirs — cue distance counts previous problems' situations (Part I, "Cue distance").
OUTPUT — your entire deliverable is ONE submit_problem call:
@@ -210,7 +211,7 @@ PARTITION DOCTRINE:
OUTPUT — your entire deliverable is ONE submit_partition call: "partitions", a list of { chains: [ids] }, one entry per problem, together containing every input chain exactly once. No empty entries; no unknown ids; no problem without a real target relationship.`;
const GRADER_INSTRUCTIONS = `You are the grading stage of a learning application's practice system. One practice problem — built to strengthen the learner's recall of specific propositions — has been answered. Your job is FEEDBACK ONLY: verdict + what actually mattered. You are not assessing a person; you are coaching a recall.
export const GRADER_INSTRUCTIONS = `You are the grading stage of a learning application's practice system. One practice problem — built to strengthen the learner's recall of specific propositions — has been answered. Your job is FEEDBACK ONLY: verdict + what actually mattered. You are not assessing a person; you are coaching a recall.
You receive: the problem's statement; its pre-written MODEL SOLUTION and GRADING RUBRIC (committed before the learner ever saw the problem — they are the reference, not you); the propositions (names + statements) the problem was built to exercise; and the learner's ANSWER — often dictated thinking-out-loud prose. Extract the reasoning from the prose; grade the substance, never the form. Do not demand the solution's exact wording — the rubric names what actually matters; nothing else is graded.
@@ -227,6 +228,8 @@ THE VERDICT (exactly one):
- "partial": some of the target came to mind; the answer is materially incomplete or partly wrong.
- "missed": the targeted proposition(s) did not come to mind — the answer is empty or on the wrong track.
Feedback renders as full Markdown. Mathematics is always LaTeX — inline as $…$, display as $$…$$ on its own lines. Code in fenced blocks with a language tag, identifiers inline as \`code\`. Lists and tables wherever they carry structure. Name the failed recall and teach the boundary in properly formatted prose, not plain text.
OUTPUT — your entire deliverable is ONE submit_grade call: "verdict" and "feedback" (non-empty; the boundary-teaching text above).`;
// ─── schemas + fills ─────────────────────────────────────────────────────────
+41
View File
@@ -16,6 +16,10 @@ import {
type ToolCall,
} from "@earendil-works/pi-ai";
import type { PiEngine } from "../src/agent/models.ts";
import {
GENERATOR_DOCTRINE,
GRADER_INSTRUCTIONS,
} from "../src/agent/subagents/practice.ts";
import { createPracticeAgents, renderChains, renderParams, renderPreviousProblems } from "../src/agent/subagents/practice.ts";
import { ModelsNotConfiguredError, ProviderKeyMissingError, type ProviderKeyMissing } from "../src/core/settings.ts";
import type { PracticeParams, ReasoningChain } from "../src/core/contracts.ts";
@@ -283,3 +287,40 @@ describe("refusal gates (BYOK honest refusal, extraction's pattern)", () => {
agents.refuseIfStagesUnready(); // no throw
});
});
// ─── formatting doctrine (owner-approved 2026-09-29) ─────────────────────────
describe("formatting doctrine pins", () => {
it("GENERATOR_DOCTRINE carries the FORMATTING bullet verbatim (Markdown + LaTeX + fenced code + tables/lists)", () => {
assert.ok(
GENERATOR_DOCTRINE.includes(
"- FORMATTING: your statement, solution, and rubric are rendered as full Markdown — use everything it offers. Mathematics is always LaTeX: inline as $…$, display as $$…$$ on its own lines. Code, pseudocode, and formal systems live in fenced code blocks with a language tag; identifiers and symbols inline as `code`. Use tables for parallel structure and comparisons, lists for enumerations. Prose-only renderings of mathematical or structured content are a defect: if a sentence carries an equation, a derivation, or a piece of code, format it as what it is.",
),
"the exact FORMATTING sentence",
);
// it sits after the parameters bullet, before PREVIOUS PROBLEMS
assert.ok(
GENERATOR_DOCTRINE.indexOf("- FORMATTING:") > GENERATOR_DOCTRINE.indexOf("THE PRESENTATION:"),
"after the presentation bullet",
);
assert.ok(
GENERATOR_DOCTRINE.indexOf("- PREVIOUS PROBLEMS:") > GENERATOR_DOCTRINE.indexOf("- FORMATTING:"),
"before PREVIOUS PROBLEMS",
);
});
it("GRADER_INSTRUCTIONS carries the formatting paragraph before the OUTPUT section", () => {
assert.ok(
GRADER_INSTRUCTIONS.includes(
"Feedback renders as full Markdown. Mathematics is always LaTeX — inline as $…$, display as $$…$$ on its own lines. Code in fenced blocks with a language tag, identifiers inline as `code`. Lists and tables wherever they carry structure. Name the failed recall and teach the boundary in properly formatted prose, not plain text.",
),
"the exact grader paragraph",
);
assert.ok(
GRADER_INSTRUCTIONS.indexOf("Feedback renders as full Markdown") < GRADER_INSTRUCTIONS.indexOf("OUTPUT — your entire deliverable is ONE submit_grade call"),
"before the OUTPUT section",
);
});
});