Per-chunk jobs: parallel extracts under a 3-slot semaphore, serialized audits (owner 2026-09-28)

Every notes chunk and chat coverage batch is its own job — registry row,
usage record, failure domain. Extracts fan out process-wide behind a
3-permit extract semaphore on globalThis (new 'queued' phase); audits and
merges stay serialized behind the existing FIFO gate ('waiting'), so each
auditor explores the graph after every earlier job's merge.

- ExtractionPhase gains 'queued'; RunningExtraction gains part {i,n};
  the feed shows one row per job with a '· i/N' suffix (status route)
- abortExtraction is source-wide: every job of a fan-out dies together;
  merged jobs keep their coverage checkpoints
- failure isolation at the fan-out (Promise.allSettled + aggregation):
  a failed chunk no longer kills its siblings — the run returns what
  actually merged; job-level aborts log, pre-fan-out errors still throw
- usage ledgers: one record per job (source-keyed), failed jobs bill
  their partial spend
- runJob/aggregateJobs helpers; setPhase/recordStep/extract/audit/
  notesExtract key the registry by jobKey (src#i) — notesExtract's
  signal lookup was the one real bug the new tests caught (source-id
  lookup meant fan-out aborts never reached the model call)
- semaphore is handoff-correct (fresh takes increment; handed-off
  permits don't double-count — an inflated counter deadlocks at 3)
- tests: isolation pins rewritten (stage failure now resolves zeros),
  single-flight/registry tails updated, per-job usage counts, and a new
  describe pinning the fan-out: 3 extracting + 5 queued, parts 1..8,
  source-wide abort, per-job usage on failure, extract peak == 3,
  audit peak == 1
- docs/architecture.md: serialized-audit paragraph rewritten for
  per-chunk jobs; two-front-doors wording aligned
This commit is contained in:
amosblomqvist
2026-09-29 00:34:21 +03:00
parent 3e9a85d60f
commit d047e7b2dc
4 changed files with 463 additions and 155 deletions
+9 -4
View File
@@ -2,8 +2,10 @@
* GET /api/graph/status — the in-flight background extraction runs
* (vision §1: the pipeline status must be visible on the home graph).
* Thin transport over the pipeline's module-level registry: phase
* (extracting / auditing / merging) per chat, oldest first. Titles joined
* from the chat store; a missing chat record still reports the run.
* (queued / extracting / waiting / auditing / merging) per job, oldest
* first — a multi-chunk source shows one row PER CHUNK ("· i/N" suffix,
* per-chunk jobs, owner 2026-09-28). Titles joined from the chat store;
* a missing chat record still reports the run.
*/
import { chatStore, extractionStatuses } from "@/src/server";
@@ -13,8 +15,11 @@ export async function GET() {
const runs = extractionStatuses().map((run) => ({
chatId: run.chatId,
// The run's own label (the file path for notes imports) wins; chat
// runs fall back to the chat store's title.
title: run.label ?? chatStore.getChat(run.chatId)?.title ?? run.chatId,
// runs fall back to the chat store's title. Chunk jobs append their
// part ("· 2/5") — one row per chunk of the fan-out.
title:
(run.label ?? chatStore.getChat(run.chatId)?.title ?? run.chatId) +
(run.part ? ` · ${run.part.i}/${run.part.n}` : ""),
phase: run.phase,
startedAt: run.startedAt,
// Live per-stage usage (the money channel): model + per-step CallUsage,