practice bug fix — the boot sweep stamped a LIVE run (owner live test 2026-09-29)

Owner report: created a set, 'Interrupted (server restarted)' showed
on list + set views, generation proceeded and overwrote it. The
sweep's 'generating + no registry row' test had a false-positive
mode: it ran at EVERY pipeline instantiation, and an instance could
evaluate against a registry whose fresh row it couldn't observe.

Three guards, per ruling:

1. HEARTBEAT: ProblemSet.updatedAt? (contract + doc) — bumped on
   every store write; createSet stamps it = createdAt; saveSet
   re-stamps (validator tolerates absence for legacy rows).

2. ONCE PER PROCESS: the sweep guards on globalThis
   .__lhPracticeSweepDone — no instance re-evaluates after the
   process's first sweep.

3. GRACE WINDOW: only sets whose heartbeat (updatedAt ?? createdAt)
   is older than 120s are stamped. A live run saves at every phase/
   problem transition, so it is structurally unreachable; a true
   orphan is minutes stale at next boot. One [practice <id>] log
   line per stamped set (the extraction/jobs console.error idiom).

Tests: stale orphan stamped; FRESH generating set untouched (grace);
once-guard (second factory instantiation never re-sweeps); live-row
protection re-armed via resetSweep() test seam; store pins for
updatedAt birth-stamp + re-stamp + legacy tolerance. Architecture.md
sweep sentence updated (once-per-process + grace semantics).

1142/1142, tsc 0, lint 0, build 0.
This commit is contained in:
amosblomqvist
2026-09-29 20:19:46 +03:00
parent e92679254d
commit 8d39f3717c
6 changed files with 116 additions and 14 deletions
+1 -1
View File
@@ -380,7 +380,7 @@ Practice turns the graph into problem sets the learner reasons through out loud.
**Grading is feedback-only.** Submitting answers fires one **grader** job per answered problem: the grader sees the problem, the propositions its chains touch, the pre-written solution and rubric, and the learner's answer — never the full set (minimal, by ruling). It returns a verdict (`solid`/`partial`/`missed`) and feedback that names the failed recall and teaches the boundary. The verdict lands on the problem and the UI reveals it with the solution and rubric; the answer locks. Nothing reaches the graph, no level, no count, anywhere. A grader failure leaves the answer gradeless and re-submittable.
**The jobs pattern — registry, not JobRecords.** Practice runs are NOT Workbench jobs: no `JobStore` records, no steer messages, nothing ever wakes the teacher — a practice run isn't chat-scoped (a set outlives its birth session and has no lesson to re-enter; its result is itself, persisted in its own store and read by its own page). What it shares with the Workbench is the observability surface: a `globalThis` run registry (the module-instance idiom — `__lhPracticeRuns`), one row per set-phase (`setId:generation`, `setId:grading`), feeding the activity dock (GraduationCap rows; abort is set-scoped and idempotent; a boot sweep marks orphaned `partitioning`/`generating` sets as `error "Interrupted (server restarted)"` — the job runner's crash-honesty at set grain). Store writes serialize behind a per-set mutex; the pipeline is the sole writer of `data/practice`.
**The jobs pattern — registry, not JobRecords.** Practice runs are NOT Workbench jobs: no `JobStore` records, no steer messages, nothing ever wakes the teacher — a practice run isn't chat-scoped (a set outlives its birth session and has no lesson to re-enter; its result is itself, persisted in its own store and read by its own page). What it shares with the Workbench is the observability surface: a `globalThis` run registry (the module-instance idiom — `__lhPracticeRuns`), one row per set-phase (`setId:generation`, `setId:grading`), feeding the activity dock (GraduationCap rows; abort is set-scoped and idempotent; a boot sweep marks orphaned `partitioning`/`generating` sets as `error "Interrupted (server restarted)"` — once per process, and only when the set's `updatedAt` heartbeat is >120s stale (a live run saves at every transition and is structurally never stamped; the guard trio closed a live-stamping bug from the first live test) — the job runner's crash-honesty at set grain). Store writes serialize behind a per-set mutex; the pipeline is the sole writer of `data/practice`.
**API + UI.** Creation is GRAPH-DOMAIN, not chat-domain (owner ruling 2026-09-29: "problem sets always take in nodes — graph objects"): `POST /api/practice/sets` takes `{nodeIds, params, title?}` ONLY — no chatId, no clusterId; the UI resolves a picked cluster to its member node ids and passes the cluster's label as the title. `GET /api/graph/clusters` serves the picker: the birth-session partition itself (`birthSessionPartition`, same ordering — size desc, ties by id) — EVERY group regardless of size (owner ruling, same evening: MIN_CLUSTER_SIZE is a hull-drawing rule; practice availability must not inherit a visual threshold), each as `{key, kind: "chat"|"notes", label, nodeIds, nodeCount}`; a cluster whose naming record is gone keeps `label: null` — honest, never guessed. 202 `{id}` with generation fire-and-forget; 400 empty selection (empty intersection with live nodes — the vanished-nodes edge) / bad params; 503 model/key refusal. `GET /api/practice/sets` · `GET/DELETE /api/practice/sets/[id]` (delete aborts live runs first) · `POST /api/practice/sets/[id]/answers` (202; 409 when not ready). Surfaces: `/practice` (the list — status/counts rows, delete confirm, the new-set dialog with the named-level controls (a 5-detent ladder slider, a 3-detent cue slider, a Theory|Application toggle — each showing the level's name + definition) and the cluster picker — the ONLY creation entry for now; the graph-view cluster entry comes later with the free-form selection UI) and `/practice/[id]` (the exam page — a levels line in the header, numbered problems, markdown statements with math healing at the display seam, a textarea + focus-gated dictation bar per problem, one Submit per set, the post-grade reveal). Entry: the sidebar's Practice nav row.
+23 -9
View File
@@ -264,15 +264,29 @@ export interface PracticePipelineDeps extends PracticeModels {
export function createPracticePipeline(deps: PracticePipelineDeps) {
const agents = createPracticeAgents(deps);
// BOOT SWEEP: a persisted partitioning/generating set with no live row
// is a crash/restart orphan (the registry is memory-only) — mark it,
// idempotently. Runs at every pipeline instantiation (dev route
// entries re-run it; live rows are globalThis-shared and skipped).
for (const set of deps.practiceStore.listSets()) {
if (
(set.status === "partitioning" || set.status === "generating") &&
!practiceRuns().has(`${set.id}:generation`)
) {
// BOOT SWEEP (owner bug fix, live test 2026-09-29 — the sweep once
// stamped a LIVE run): a persisted partitioning/generating set with no
// live row AND a heartbeat older than the grace window is a crash/
// restart orphan (the registry is memory-only) — mark it, idempotently.
// THREE guards against stamping the living:
// 1. ONCE PER PROCESS (globalThis flag) — an instance could otherwise
// evaluate against a registry whose fresh row it can't observe
// (dev route bundles, restart paths).
// 2. GRACE WINDOW (120s over updatedAt ?? createdAt) — a live run
// saves at every phase/problem transition, so its heartbeat is
// structurally never 120s stale; a true orphan is minutes stale at
// the next boot.
// 3. Live-row check (globalThis-shared registry) as before.
const sweepFlag = globalThis as { __lhPracticeSweepDone?: boolean };
if (!sweepFlag.__lhPracticeSweepDone) {
sweepFlag.__lhPracticeSweepDone = true;
const staleBefore = Date.now() - 120_000;
for (const set of deps.practiceStore.listSets()) {
if (set.status !== "partitioning" && set.status !== "generating") continue;
if (practiceRuns().has(`${set.id}:generation`)) continue;
const heartbeat = Date.parse(set.updatedAt ?? set.createdAt);
if (Number.isFinite(heartbeat) && heartbeat > staleBefore) continue; // grace
console.error(`[practice ${set.id}] boot sweep: stale ${set.status} set marked interrupted`);
set.status = "error";
set.error = "Interrupted (server restarted)";
deps.practiceStore.saveSet(set);
+6
View File
@@ -163,6 +163,12 @@ export interface ProblemSet {
* default. */
title: string;
createdAt: string;
/** Bumped on every store write — the live-run heartbeat the boot
* sweep's grace window reads (a live run saves at every phase or
* problem transition, so it is structurally never 120s stale).
* Absent on legacy rows (validator tolerates; readers fall back
* to createdAt). */
updatedAt?: string;
params: PracticeParams;
/** The selected node ids — the raw selection record. The chain
* decomposition derives from these + the follows-edges among them at
+4
View File
@@ -133,6 +133,7 @@ function createSet(init: Omit<ProblemSet, "id">): ProblemSet {
const set: ProblemSet = {
...init,
id: `ps_${Date.now().toString(36)}_${crypto.randomUUID().slice(0, 8)}`,
updatedAt: init.updatedAt ?? init.createdAt,
};
writeSetFile(set);
return set;
@@ -164,6 +165,9 @@ function listSets(): ProblemSet[] {
function saveSet(set: ProblemSet): void {
if (!isProblemSet(set)) throw new Error("problem set record failed validation");
// The heartbeat: every write re-stamps (the boot sweep's grace window
// reads this — a live run is structurally never 120s stale).
set.updatedAt = new Date().toISOString();
writeSetFile(set);
}
+62 -4
View File
@@ -234,6 +234,13 @@ function fixture(partial: Partial<Parameters<typeof createPracticePipeline>[0]>
return { practice, deps, make: () => createPracticePipeline(deps) };
}
/** Reset the boot sweep's once-per-process flag — each call simulates a
* fresh process for the sweep's semantics (production sets it once and
* never re-sweeps; tests need controlled re-arming). */
const resetSweep = () => {
delete (globalThis as { __lhPracticeSweepDone?: boolean }).__lhPracticeSweepDone;
};
/** Await a condition (the extraction until() idiom, trimmed). */
async function until(cond: () => boolean, ms = 2000): Promise<void> {
const t0 = Date.now();
@@ -490,11 +497,11 @@ describe("submitAnswers (grading)", () => {
// ─── boot sweep + delete ─────────────────────────────────────────────────────
describe("boot sweep + delete", () => {
it("an orphaned generating set (no live row) is marked interrupted", async () => {
it("a STALE orphan (>120s heartbeat, no live row) is marked interrupted", async () => {
const f = fixture({});
const orphan = f.practice.store.createSet({
title: "orphan",
createdAt: NOW,
createdAt: new Date(Date.now() - 300_000).toISOString(), // 5 min stale
params: PARAMS,
nodeIds: ["b"],
propositions: { b: { name: "B", statement: "s" } },
@@ -502,18 +509,69 @@ describe("boot sweep + delete", () => {
status: "generating",
problems: [{ id: "pp_orphan", chains: ["b"], statement: "", solution: "", rubric: "", status: "generating" }],
});
resetSweep();
f.make(); // the sweep runs at construction
const swept = f.practice.store.getSet(orphan.id)!;
assert.equal(swept.status, "error");
assert.equal(swept.error, "Interrupted (server restarted)");
});
it("a FRESH generating set (no live row) is NOT stamped — the grace window (owner bug fix)", async () => {
const f = fixture({});
const fresh = f.practice.store.createSet({
title: "fresh",
createdAt: new Date().toISOString(), // heartbeat seconds old
params: PARAMS,
nodeIds: ["b"],
propositions: { b: { name: "B", statement: "s" } },
chains: [{ id: "b", premises: ["a"], conclusion: "b" }],
status: "generating",
problems: [{ id: "pp_fresh", chains: ["b"], statement: "", solution: "", rubric: "", status: "generating" }],
});
resetSweep();
f.make();
const read = f.practice.store.getSet(fresh.id)!;
assert.equal(read.status, "generating", "grace: a live run's fresh heartbeat is never stamped");
assert.equal(read.error, undefined);
});
it("once per process: a second factory instantiation never re-sweeps", async () => {
const f = fixture({});
const orphanA = f.practice.store.createSet({
title: "swept",
createdAt: new Date(Date.now() - 300_000).toISOString(),
params: PARAMS,
nodeIds: ["b"],
propositions: { b: { name: "B", statement: "s" } },
chains: [{ id: "b", premises: ["a"], conclusion: "b" }],
status: "generating",
problems: [],
});
resetSweep();
f.make(); // sweeps orphanA, sets the process flag
assert.equal(f.practice.store.getSet(orphanA.id)!.status, "error");
const orphanB = f.practice.store.createSet({
title: "second",
createdAt: new Date(Date.now() - 300_000).toISOString(),
params: PARAMS,
nodeIds: ["b"],
propositions: { b: { name: "B", statement: "s" } },
chains: [{ id: "b", premises: ["a"], conclusion: "b" }],
status: "generating",
problems: [],
});
f.make(); // flag set → NO sweep
assert.equal(f.practice.store.getSet(orphanB.id)!.status, "generating", "the sweep runs once per process");
});
it("a LIVE generating set (registry row) is NOT swept by a second instance", async () => {
const f = fixture({ partitioner: hangingModel(), problemGenerator: echoGenerator(1), grader: echoGrader(1) });
const a = f.make();
const { set, done } = await a.createProblemSet(["b"], PARAMS); // hangs partitioning
f.make(); // second pipeline instance, same process → same registry
assert.equal(f.practice.store.getSet(set.id)!.status, "partitioning");
resetSweep();
f.make(); // second pipeline instance, same process → same registry, sweep RE-ARMED
assert.equal(f.practice.store.getSet(set.id)!.status, "partitioning", "the live row protects it even mid-sweep");
a.abortPractice(set.id);
await done;
});
+20
View File
@@ -42,6 +42,26 @@ const init = (over: Partial<Omit<ProblemSet, "id">> = {}): Omit<ProblemSet, "id"
// ─── round-trip ──────────────────────────────────────────────────────────────
describe("updatedAt — the live-run heartbeat", () => {
it("createSet stamps updatedAt = createdAt; saveSet re-stamps on every write", () => {
const set = practiceStore.createSet(init());
assert.equal(set.updatedAt, set.createdAt, "birth stamp");
const before = new Date().toISOString();
const edited = { ...set, title: "Renamed" };
practiceStore.saveSet(edited);
const read = practiceStore.getSet(set.id)!
assert.ok(read.updatedAt !== undefined && read.updatedAt >= before, "every write re-stamps");
assert.notEqual(read.title, set.title);
});
it("legacy rows without updatedAt still validate (readers fall back to createdAt)", () => {
const set = practiceStore.createSet(init());
const bare = { ...set } as Partial<typeof set>;
delete bare.updatedAt;
assert.ok(isProblemSet({ ...bare, id: set.id }));
});
});
describe("the practice store", () => {
it("createSet assigns a ps_ id, mkdirs, and round-trips through getSet", () => {
const set = practiceStore.createSet(init());