practice bug fix — the boot sweep stamped a LIVE run (owner live test 2026-09-29)
Owner report: created a set, 'Interrupted (server restarted)' showed on list + set views, generation proceeded and overwrote it. The sweep's 'generating + no registry row' test had a false-positive mode: it ran at EVERY pipeline instantiation, and an instance could evaluate against a registry whose fresh row it couldn't observe. Three guards, per ruling: 1. HEARTBEAT: ProblemSet.updatedAt? (contract + doc) — bumped on every store write; createSet stamps it = createdAt; saveSet re-stamps (validator tolerates absence for legacy rows). 2. ONCE PER PROCESS: the sweep guards on globalThis .__lhPracticeSweepDone — no instance re-evaluates after the process's first sweep. 3. GRACE WINDOW: only sets whose heartbeat (updatedAt ?? createdAt) is older than 120s are stamped. A live run saves at every phase/ problem transition, so it is structurally unreachable; a true orphan is minutes stale at next boot. One [practice <id>] log line per stamped set (the extraction/jobs console.error idiom). Tests: stale orphan stamped; FRESH generating set untouched (grace); once-guard (second factory instantiation never re-sweeps); live-row protection re-armed via resetSweep() test seam; store pins for updatedAt birth-stamp + re-stamp + legacy tolerance. Architecture.md sweep sentence updated (once-per-process + grace semantics). 1142/1142, tsc 0, lint 0, build 0.
This commit is contained in:
@@ -380,7 +380,7 @@ Practice turns the graph into problem sets the learner reasons through out loud.
|
|||||||
|
|
||||||
**Grading is feedback-only.** Submitting answers fires one **grader** job per answered problem: the grader sees the problem, the propositions its chains touch, the pre-written solution and rubric, and the learner's answer — never the full set (minimal, by ruling). It returns a verdict (`solid`/`partial`/`missed`) and feedback that names the failed recall and teaches the boundary. The verdict lands on the problem and the UI reveals it with the solution and rubric; the answer locks. Nothing reaches the graph, no level, no count, anywhere. A grader failure leaves the answer gradeless and re-submittable.
|
**Grading is feedback-only.** Submitting answers fires one **grader** job per answered problem: the grader sees the problem, the propositions its chains touch, the pre-written solution and rubric, and the learner's answer — never the full set (minimal, by ruling). It returns a verdict (`solid`/`partial`/`missed`) and feedback that names the failed recall and teaches the boundary. The verdict lands on the problem and the UI reveals it with the solution and rubric; the answer locks. Nothing reaches the graph, no level, no count, anywhere. A grader failure leaves the answer gradeless and re-submittable.
|
||||||
|
|
||||||
**The jobs pattern — registry, not JobRecords.** Practice runs are NOT Workbench jobs: no `JobStore` records, no steer messages, nothing ever wakes the teacher — a practice run isn't chat-scoped (a set outlives its birth session and has no lesson to re-enter; its result is itself, persisted in its own store and read by its own page). What it shares with the Workbench is the observability surface: a `globalThis` run registry (the module-instance idiom — `__lhPracticeRuns`), one row per set-phase (`setId:generation`, `setId:grading`), feeding the activity dock (GraduationCap rows; abort is set-scoped and idempotent; a boot sweep marks orphaned `partitioning`/`generating` sets as `error "Interrupted (server restarted)"` — the job runner's crash-honesty at set grain). Store writes serialize behind a per-set mutex; the pipeline is the sole writer of `data/practice`.
|
**The jobs pattern — registry, not JobRecords.** Practice runs are NOT Workbench jobs: no `JobStore` records, no steer messages, nothing ever wakes the teacher — a practice run isn't chat-scoped (a set outlives its birth session and has no lesson to re-enter; its result is itself, persisted in its own store and read by its own page). What it shares with the Workbench is the observability surface: a `globalThis` run registry (the module-instance idiom — `__lhPracticeRuns`), one row per set-phase (`setId:generation`, `setId:grading`), feeding the activity dock (GraduationCap rows; abort is set-scoped and idempotent; a boot sweep marks orphaned `partitioning`/`generating` sets as `error "Interrupted (server restarted)"` — once per process, and only when the set's `updatedAt` heartbeat is >120s stale (a live run saves at every transition and is structurally never stamped; the guard trio closed a live-stamping bug from the first live test) — the job runner's crash-honesty at set grain). Store writes serialize behind a per-set mutex; the pipeline is the sole writer of `data/practice`.
|
||||||
|
|
||||||
**API + UI.** Creation is GRAPH-DOMAIN, not chat-domain (owner ruling 2026-09-29: "problem sets always take in nodes — graph objects"): `POST /api/practice/sets` takes `{nodeIds, params, title?}` ONLY — no chatId, no clusterId; the UI resolves a picked cluster to its member node ids and passes the cluster's label as the title. `GET /api/graph/clusters` serves the picker: the birth-session partition itself (`birthSessionPartition`, same ordering — size desc, ties by id) — EVERY group regardless of size (owner ruling, same evening: MIN_CLUSTER_SIZE is a hull-drawing rule; practice availability must not inherit a visual threshold), each as `{key, kind: "chat"|"notes", label, nodeIds, nodeCount}`; a cluster whose naming record is gone keeps `label: null` — honest, never guessed. 202 `{id}` with generation fire-and-forget; 400 empty selection (empty intersection with live nodes — the vanished-nodes edge) / bad params; 503 model/key refusal. `GET /api/practice/sets` · `GET/DELETE /api/practice/sets/[id]` (delete aborts live runs first) · `POST /api/practice/sets/[id]/answers` (202; 409 when not ready). Surfaces: `/practice` (the list — status/counts rows, delete confirm, the new-set dialog with the named-level controls (a 5-detent ladder slider, a 3-detent cue slider, a Theory|Application toggle — each showing the level's name + definition) and the cluster picker — the ONLY creation entry for now; the graph-view cluster entry comes later with the free-form selection UI) and `/practice/[id]` (the exam page — a levels line in the header, numbered problems, markdown statements with math healing at the display seam, a textarea + focus-gated dictation bar per problem, one Submit per set, the post-grade reveal). Entry: the sidebar's Practice nav row.
|
**API + UI.** Creation is GRAPH-DOMAIN, not chat-domain (owner ruling 2026-09-29: "problem sets always take in nodes — graph objects"): `POST /api/practice/sets` takes `{nodeIds, params, title?}` ONLY — no chatId, no clusterId; the UI resolves a picked cluster to its member node ids and passes the cluster's label as the title. `GET /api/graph/clusters` serves the picker: the birth-session partition itself (`birthSessionPartition`, same ordering — size desc, ties by id) — EVERY group regardless of size (owner ruling, same evening: MIN_CLUSTER_SIZE is a hull-drawing rule; practice availability must not inherit a visual threshold), each as `{key, kind: "chat"|"notes", label, nodeIds, nodeCount}`; a cluster whose naming record is gone keeps `label: null` — honest, never guessed. 202 `{id}` with generation fire-and-forget; 400 empty selection (empty intersection with live nodes — the vanished-nodes edge) / bad params; 503 model/key refusal. `GET /api/practice/sets` · `GET/DELETE /api/practice/sets/[id]` (delete aborts live runs first) · `POST /api/practice/sets/[id]/answers` (202; 409 when not ready). Surfaces: `/practice` (the list — status/counts rows, delete confirm, the new-set dialog with the named-level controls (a 5-detent ladder slider, a 3-detent cue slider, a Theory|Application toggle — each showing the level's name + definition) and the cluster picker — the ONLY creation entry for now; the graph-view cluster entry comes later with the free-form selection UI) and `/practice/[id]` (the exam page — a levels line in the header, numbered problems, markdown statements with math healing at the display seam, a textarea + focus-gated dictation bar per problem, one Submit per set, the post-grade reveal). Entry: the sidebar's Practice nav row.
|
||||||
|
|
||||||
|
|||||||
@@ -264,15 +264,29 @@ export interface PracticePipelineDeps extends PracticeModels {
|
|||||||
export function createPracticePipeline(deps: PracticePipelineDeps) {
|
export function createPracticePipeline(deps: PracticePipelineDeps) {
|
||||||
const agents = createPracticeAgents(deps);
|
const agents = createPracticeAgents(deps);
|
||||||
|
|
||||||
// BOOT SWEEP: a persisted partitioning/generating set with no live row
|
// BOOT SWEEP (owner bug fix, live test 2026-09-29 — the sweep once
|
||||||
// is a crash/restart orphan (the registry is memory-only) — mark it,
|
// stamped a LIVE run): a persisted partitioning/generating set with no
|
||||||
// idempotently. Runs at every pipeline instantiation (dev route
|
// live row AND a heartbeat older than the grace window is a crash/
|
||||||
// entries re-run it; live rows are globalThis-shared and skipped).
|
// restart orphan (the registry is memory-only) — mark it, idempotently.
|
||||||
for (const set of deps.practiceStore.listSets()) {
|
// THREE guards against stamping the living:
|
||||||
if (
|
// 1. ONCE PER PROCESS (globalThis flag) — an instance could otherwise
|
||||||
(set.status === "partitioning" || set.status === "generating") &&
|
// evaluate against a registry whose fresh row it can't observe
|
||||||
!practiceRuns().has(`${set.id}:generation`)
|
// (dev route bundles, restart paths).
|
||||||
) {
|
// 2. GRACE WINDOW (120s over updatedAt ?? createdAt) — a live run
|
||||||
|
// saves at every phase/problem transition, so its heartbeat is
|
||||||
|
// structurally never 120s stale; a true orphan is minutes stale at
|
||||||
|
// the next boot.
|
||||||
|
// 3. Live-row check (globalThis-shared registry) as before.
|
||||||
|
const sweepFlag = globalThis as { __lhPracticeSweepDone?: boolean };
|
||||||
|
if (!sweepFlag.__lhPracticeSweepDone) {
|
||||||
|
sweepFlag.__lhPracticeSweepDone = true;
|
||||||
|
const staleBefore = Date.now() - 120_000;
|
||||||
|
for (const set of deps.practiceStore.listSets()) {
|
||||||
|
if (set.status !== "partitioning" && set.status !== "generating") continue;
|
||||||
|
if (practiceRuns().has(`${set.id}:generation`)) continue;
|
||||||
|
const heartbeat = Date.parse(set.updatedAt ?? set.createdAt);
|
||||||
|
if (Number.isFinite(heartbeat) && heartbeat > staleBefore) continue; // grace
|
||||||
|
console.error(`[practice ${set.id}] boot sweep: stale ${set.status} set marked interrupted`);
|
||||||
set.status = "error";
|
set.status = "error";
|
||||||
set.error = "Interrupted (server restarted)";
|
set.error = "Interrupted (server restarted)";
|
||||||
deps.practiceStore.saveSet(set);
|
deps.practiceStore.saveSet(set);
|
||||||
|
|||||||
@@ -163,6 +163,12 @@ export interface ProblemSet {
|
|||||||
* default. */
|
* default. */
|
||||||
title: string;
|
title: string;
|
||||||
createdAt: string;
|
createdAt: string;
|
||||||
|
/** Bumped on every store write — the live-run heartbeat the boot
|
||||||
|
* sweep's grace window reads (a live run saves at every phase or
|
||||||
|
* problem transition, so it is structurally never 120s stale).
|
||||||
|
* Absent on legacy rows (validator tolerates; readers fall back
|
||||||
|
* to createdAt). */
|
||||||
|
updatedAt?: string;
|
||||||
params: PracticeParams;
|
params: PracticeParams;
|
||||||
/** The selected node ids — the raw selection record. The chain
|
/** The selected node ids — the raw selection record. The chain
|
||||||
* decomposition derives from these + the follows-edges among them at
|
* decomposition derives from these + the follows-edges among them at
|
||||||
|
|||||||
@@ -133,6 +133,7 @@ function createSet(init: Omit<ProblemSet, "id">): ProblemSet {
|
|||||||
const set: ProblemSet = {
|
const set: ProblemSet = {
|
||||||
...init,
|
...init,
|
||||||
id: `ps_${Date.now().toString(36)}_${crypto.randomUUID().slice(0, 8)}`,
|
id: `ps_${Date.now().toString(36)}_${crypto.randomUUID().slice(0, 8)}`,
|
||||||
|
updatedAt: init.updatedAt ?? init.createdAt,
|
||||||
};
|
};
|
||||||
writeSetFile(set);
|
writeSetFile(set);
|
||||||
return set;
|
return set;
|
||||||
@@ -164,6 +165,9 @@ function listSets(): ProblemSet[] {
|
|||||||
|
|
||||||
function saveSet(set: ProblemSet): void {
|
function saveSet(set: ProblemSet): void {
|
||||||
if (!isProblemSet(set)) throw new Error("problem set record failed validation");
|
if (!isProblemSet(set)) throw new Error("problem set record failed validation");
|
||||||
|
// The heartbeat: every write re-stamps (the boot sweep's grace window
|
||||||
|
// reads this — a live run is structurally never 120s stale).
|
||||||
|
set.updatedAt = new Date().toISOString();
|
||||||
writeSetFile(set);
|
writeSetFile(set);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -234,6 +234,13 @@ function fixture(partial: Partial<Parameters<typeof createPracticePipeline>[0]>
|
|||||||
return { practice, deps, make: () => createPracticePipeline(deps) };
|
return { practice, deps, make: () => createPracticePipeline(deps) };
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** Reset the boot sweep's once-per-process flag — each call simulates a
|
||||||
|
* fresh process for the sweep's semantics (production sets it once and
|
||||||
|
* never re-sweeps; tests need controlled re-arming). */
|
||||||
|
const resetSweep = () => {
|
||||||
|
delete (globalThis as { __lhPracticeSweepDone?: boolean }).__lhPracticeSweepDone;
|
||||||
|
};
|
||||||
|
|
||||||
/** Await a condition (the extraction until() idiom, trimmed). */
|
/** Await a condition (the extraction until() idiom, trimmed). */
|
||||||
async function until(cond: () => boolean, ms = 2000): Promise<void> {
|
async function until(cond: () => boolean, ms = 2000): Promise<void> {
|
||||||
const t0 = Date.now();
|
const t0 = Date.now();
|
||||||
@@ -490,11 +497,11 @@ describe("submitAnswers (grading)", () => {
|
|||||||
// ─── boot sweep + delete ─────────────────────────────────────────────────────
|
// ─── boot sweep + delete ─────────────────────────────────────────────────────
|
||||||
|
|
||||||
describe("boot sweep + delete", () => {
|
describe("boot sweep + delete", () => {
|
||||||
it("an orphaned generating set (no live row) is marked interrupted", async () => {
|
it("a STALE orphan (>120s heartbeat, no live row) is marked interrupted", async () => {
|
||||||
const f = fixture({});
|
const f = fixture({});
|
||||||
const orphan = f.practice.store.createSet({
|
const orphan = f.practice.store.createSet({
|
||||||
title: "orphan",
|
title: "orphan",
|
||||||
createdAt: NOW,
|
createdAt: new Date(Date.now() - 300_000).toISOString(), // 5 min stale
|
||||||
params: PARAMS,
|
params: PARAMS,
|
||||||
nodeIds: ["b"],
|
nodeIds: ["b"],
|
||||||
propositions: { b: { name: "B", statement: "s" } },
|
propositions: { b: { name: "B", statement: "s" } },
|
||||||
@@ -502,18 +509,69 @@ describe("boot sweep + delete", () => {
|
|||||||
status: "generating",
|
status: "generating",
|
||||||
problems: [{ id: "pp_orphan", chains: ["b"], statement: "", solution: "", rubric: "", status: "generating" }],
|
problems: [{ id: "pp_orphan", chains: ["b"], statement: "", solution: "", rubric: "", status: "generating" }],
|
||||||
});
|
});
|
||||||
|
resetSweep();
|
||||||
f.make(); // the sweep runs at construction
|
f.make(); // the sweep runs at construction
|
||||||
const swept = f.practice.store.getSet(orphan.id)!;
|
const swept = f.practice.store.getSet(orphan.id)!;
|
||||||
assert.equal(swept.status, "error");
|
assert.equal(swept.status, "error");
|
||||||
assert.equal(swept.error, "Interrupted (server restarted)");
|
assert.equal(swept.error, "Interrupted (server restarted)");
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it("a FRESH generating set (no live row) is NOT stamped — the grace window (owner bug fix)", async () => {
|
||||||
|
const f = fixture({});
|
||||||
|
const fresh = f.practice.store.createSet({
|
||||||
|
title: "fresh",
|
||||||
|
createdAt: new Date().toISOString(), // heartbeat seconds old
|
||||||
|
params: PARAMS,
|
||||||
|
nodeIds: ["b"],
|
||||||
|
propositions: { b: { name: "B", statement: "s" } },
|
||||||
|
chains: [{ id: "b", premises: ["a"], conclusion: "b" }],
|
||||||
|
status: "generating",
|
||||||
|
problems: [{ id: "pp_fresh", chains: ["b"], statement: "", solution: "", rubric: "", status: "generating" }],
|
||||||
|
});
|
||||||
|
resetSweep();
|
||||||
|
f.make();
|
||||||
|
const read = f.practice.store.getSet(fresh.id)!;
|
||||||
|
assert.equal(read.status, "generating", "grace: a live run's fresh heartbeat is never stamped");
|
||||||
|
assert.equal(read.error, undefined);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("once per process: a second factory instantiation never re-sweeps", async () => {
|
||||||
|
const f = fixture({});
|
||||||
|
const orphanA = f.practice.store.createSet({
|
||||||
|
title: "swept",
|
||||||
|
createdAt: new Date(Date.now() - 300_000).toISOString(),
|
||||||
|
params: PARAMS,
|
||||||
|
nodeIds: ["b"],
|
||||||
|
propositions: { b: { name: "B", statement: "s" } },
|
||||||
|
chains: [{ id: "b", premises: ["a"], conclusion: "b" }],
|
||||||
|
status: "generating",
|
||||||
|
problems: [],
|
||||||
|
});
|
||||||
|
resetSweep();
|
||||||
|
f.make(); // sweeps orphanA, sets the process flag
|
||||||
|
assert.equal(f.practice.store.getSet(orphanA.id)!.status, "error");
|
||||||
|
|
||||||
|
const orphanB = f.practice.store.createSet({
|
||||||
|
title: "second",
|
||||||
|
createdAt: new Date(Date.now() - 300_000).toISOString(),
|
||||||
|
params: PARAMS,
|
||||||
|
nodeIds: ["b"],
|
||||||
|
propositions: { b: { name: "B", statement: "s" } },
|
||||||
|
chains: [{ id: "b", premises: ["a"], conclusion: "b" }],
|
||||||
|
status: "generating",
|
||||||
|
problems: [],
|
||||||
|
});
|
||||||
|
f.make(); // flag set → NO sweep
|
||||||
|
assert.equal(f.practice.store.getSet(orphanB.id)!.status, "generating", "the sweep runs once per process");
|
||||||
|
});
|
||||||
|
|
||||||
it("a LIVE generating set (registry row) is NOT swept by a second instance", async () => {
|
it("a LIVE generating set (registry row) is NOT swept by a second instance", async () => {
|
||||||
const f = fixture({ partitioner: hangingModel(), problemGenerator: echoGenerator(1), grader: echoGrader(1) });
|
const f = fixture({ partitioner: hangingModel(), problemGenerator: echoGenerator(1), grader: echoGrader(1) });
|
||||||
const a = f.make();
|
const a = f.make();
|
||||||
const { set, done } = await a.createProblemSet(["b"], PARAMS); // hangs partitioning
|
const { set, done } = await a.createProblemSet(["b"], PARAMS); // hangs partitioning
|
||||||
f.make(); // second pipeline instance, same process → same registry
|
resetSweep();
|
||||||
assert.equal(f.practice.store.getSet(set.id)!.status, "partitioning");
|
f.make(); // second pipeline instance, same process → same registry, sweep RE-ARMED
|
||||||
|
assert.equal(f.practice.store.getSet(set.id)!.status, "partitioning", "the live row protects it even mid-sweep");
|
||||||
a.abortPractice(set.id);
|
a.abortPractice(set.id);
|
||||||
await done;
|
await done;
|
||||||
});
|
});
|
||||||
|
|||||||
@@ -42,6 +42,26 @@ const init = (over: Partial<Omit<ProblemSet, "id">> = {}): Omit<ProblemSet, "id"
|
|||||||
|
|
||||||
// ─── round-trip ──────────────────────────────────────────────────────────────
|
// ─── round-trip ──────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
describe("updatedAt — the live-run heartbeat", () => {
|
||||||
|
it("createSet stamps updatedAt = createdAt; saveSet re-stamps on every write", () => {
|
||||||
|
const set = practiceStore.createSet(init());
|
||||||
|
assert.equal(set.updatedAt, set.createdAt, "birth stamp");
|
||||||
|
const before = new Date().toISOString();
|
||||||
|
const edited = { ...set, title: "Renamed" };
|
||||||
|
practiceStore.saveSet(edited);
|
||||||
|
const read = practiceStore.getSet(set.id)!
|
||||||
|
assert.ok(read.updatedAt !== undefined && read.updatedAt >= before, "every write re-stamps");
|
||||||
|
assert.notEqual(read.title, set.title);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("legacy rows without updatedAt still validate (readers fall back to createdAt)", () => {
|
||||||
|
const set = practiceStore.createSet(init());
|
||||||
|
const bare = { ...set } as Partial<typeof set>;
|
||||||
|
delete bare.updatedAt;
|
||||||
|
assert.ok(isProblemSet({ ...bare, id: set.id }));
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
describe("the practice store", () => {
|
describe("the practice store", () => {
|
||||||
it("createSet assigns a ps_ id, mkdirs, and round-trips through getSet", () => {
|
it("createSet assigns a ps_ id, mkdirs, and round-trips through getSet", () => {
|
||||||
const set = practiceStore.createSet(init());
|
const set = practiceStore.createSet(init());
|
||||||
|
|||||||
Reference in New Issue
Block a user