fix: README.md leaked into the teacher's system prompt

composeInstructions loaded EVERY .md in src/agent/prompts — including
README.md, the directory's human documentation, which rode into the
system prompt as its own section (visible in OpenRouter logs, old POC
paths and all).

The loader now enforces the documented convention: only numeric-prefix
fragments (\d+-*.md) compose. README.md stays in the directory as docs
and can never reach the model again. README's rule text updated to
match; composeInstructions exported for tests; regression test asserts
the README never leaks and every numeric fragment composes verbatim.
This commit is contained in:
amosblomqvist
2026-09-02 03:19:03 +03:00
parent 8f8659c8b4
commit 28ccc76594
3 changed files with 29 additions and 4 deletions
+1 -1
View File
@@ -3,7 +3,7 @@
The teacher's system prompt is composed at agent creation: The teacher's system prompt is composed at agent creation:
1. `src/agent/system-prompt.md` — the base (owned by the orchestrator; do not edit from a worker). 1. `src/agent/system-prompt.md` — the base (owned by the orchestrator; do not edit from a worker).
2. Every `*.md` file in this directory, in **sorted filename order**, joined with `---` separators. 2. Every numeric-prefix `*.md` fragment in this directory, in **sorted filename order**, joined with `---` separators. The loader enforces the numeric prefix (`\d*-*.md`) — other files (like this README) never enter the model's prompt.
Rules for adding a fragment: Rules for adding a fragment:
+10 -2
View File
@@ -20,16 +20,24 @@ import { questionTools } from "./tools.ts";
* src/agent/prompts/*.md fragment in sorted filename order (workers add * src/agent/prompts/*.md fragment in sorted filename order (workers add
* fragments — nobody edits shared sections of the base prompt). Prefix * fragments — nobody edits shared sections of the base prompt). Prefix
* fragments with a numeric order key: 10-plan-tree.md, 20-jobs.md, … * fragments with a numeric order key: 10-plan-tree.md, 20-jobs.md, …
*
* ONLY numeric-prefix files load — the loader enforces the convention, so
* non-fragment documentation (README.md, …) can live in the directory
* without ever leaking into the model's prompt.
*/ */
const PROMPTS_DIR = path.join(PATHS.root, "src", "agent", "prompts"); const PROMPTS_DIR = path.join(PATHS.root, "src", "agent", "prompts");
function composeInstructions(): string { /** A prompt fragment: numeric order key + .md (e.g. 20-jobs.md). Nothing else loads. */
const FRAGMENT_RE = /^\d+-[\w.-]+\.md$/;
/** Exported for tests: assert what composes into the model's prompt. */
export function composeInstructions(): string {
const base = fs.readFileSync(PATHS.teacherPrompt, "utf-8"); const base = fs.readFileSync(PATHS.teacherPrompt, "utf-8");
if (!fs.existsSync(PROMPTS_DIR)) return base; if (!fs.existsSync(PROMPTS_DIR)) return base;
const fragments = fs const fragments = fs
.readdirSync(PROMPTS_DIR) .readdirSync(PROMPTS_DIR)
.filter((f) => f.endsWith(".md")) .filter((f) => FRAGMENT_RE.test(f))
.sort() .sort()
.map((f) => fs.readFileSync(path.join(PROMPTS_DIR, f), "utf-8").trim()) .map((f) => fs.readFileSync(path.join(PROMPTS_DIR, f), "utf-8").trim())
.filter(Boolean); .filter(Boolean);
+18 -1
View File
@@ -38,7 +38,7 @@ import {
type SubagentSteerMessage, type SubagentSteerMessage,
type VisualResult, type VisualResult,
} from "../src/core/contracts.ts"; } from "../src/core/contracts.ts";
import { createTeacherAgent } from "../src/agent/teacher-agent.ts"; import { composeInstructions, createTeacherAgent } from "../src/agent/teacher-agent.ts";
import { createTeacherTools } from "../src/agent/tools.ts"; import { createTeacherTools } from "../src/agent/tools.ts";
import { handleChatRequest } from "../src/server/chat-handler.ts"; import { handleChatRequest } from "../src/server/chat-handler.ts";
import { deliverJobResult, setTurnRunner } from "../src/server/jobs.ts"; import { deliverJobResult, setTurnRunner } from "../src/server/jobs.ts";
@@ -687,3 +687,20 @@ describe("persistTurn id hygiene", () => {
assert.equal(new Set(ids).size, ids.length, "ids unique — React keys safe"); assert.equal(new Set(ids).size, ids.length, "ids unique — React keys safe");
}); });
}); });
// ─── prompt composition hygiene (the README leak) ─────────────────────────────
describe("prompt composition", () => {
it("loads ONLY numeric-prefix fragments — README.md never enters the system prompt", () => {
const instructions = composeInstructions();
assert.ok(!instructions.includes("Prompt fragments"), "README content must not leak");
assert.ok(!instructions.includes("Rules for adding a fragment"), "README content must not leak (rules)");
// every numeric fragment DOES compose, verbatim, after the base prompt
const base = fs.readFileSync("src/agent/system-prompt.md", "utf-8").trim();
assert.ok(instructions.startsWith(base), "base prompt first");
for (const f of fs.readdirSync("src/agent/prompts").filter((f) => /^\d+-[\w.-]+\.md$/.test(f)).sort()) {
const body = fs.readFileSync(`src/agent/prompts/${f}`, "utf-8").trim();
assert.ok(instructions.includes(body), `fragment ${f} must compose verbatim`);
}
});
});