Per-chunk jobs: parallel extracts under a 3-slot semaphore, serialized audits (owner 2026-09-28)
Every notes chunk and chat coverage batch is its own job — registry row,
usage record, failure domain. Extracts fan out process-wide behind a
3-permit extract semaphore on globalThis (new 'queued' phase); audits and
merges stay serialized behind the existing FIFO gate ('waiting'), so each
auditor explores the graph after every earlier job's merge.
- ExtractionPhase gains 'queued'; RunningExtraction gains part {i,n};
the feed shows one row per job with a '· i/N' suffix (status route)
- abortExtraction is source-wide: every job of a fan-out dies together;
merged jobs keep their coverage checkpoints
- failure isolation at the fan-out (Promise.allSettled + aggregation):
a failed chunk no longer kills its siblings — the run returns what
actually merged; job-level aborts log, pre-fan-out errors still throw
- usage ledgers: one record per job (source-keyed), failed jobs bill
their partial spend
- runJob/aggregateJobs helpers; setPhase/recordStep/extract/audit/
notesExtract key the registry by jobKey (src#i) — notesExtract's
signal lookup was the one real bug the new tests caught (source-id
lookup meant fan-out aborts never reached the model call)
- semaphore is handoff-correct (fresh takes increment; handed-off
permits don't double-count — an inflated counter deadlocks at 3)
- tests: isolation pins rewritten (stage failure now resolves zeros),
single-flight/registry tails updated, per-job usage counts, and a new
describe pinning the fan-out: 3 extracting + 5 queued, parts 1..8,
source-wide abort, per-job usage on failure, extract peak == 3,
audit peak == 1
- docs/architecture.md: serialized-audit paragraph rewritten for
per-chunk jobs; two-front-doors wording aligned
This commit is contained in:
@@ -2,8 +2,10 @@
|
||||
* GET /api/graph/status — the in-flight background extraction runs
|
||||
* (vision §1: the pipeline status must be visible on the home graph).
|
||||
* Thin transport over the pipeline's module-level registry: phase
|
||||
* (extracting / auditing / merging) per chat, oldest first. Titles joined
|
||||
* from the chat store; a missing chat record still reports the run.
|
||||
* (queued / extracting / waiting / auditing / merging) per job, oldest
|
||||
* first — a multi-chunk source shows one row PER CHUNK ("· i/N" suffix,
|
||||
* per-chunk jobs, owner 2026-09-28). Titles joined from the chat store;
|
||||
* a missing chat record still reports the run.
|
||||
*/
|
||||
import { chatStore, extractionStatuses } from "@/src/server";
|
||||
|
||||
@@ -13,8 +15,11 @@ export async function GET() {
|
||||
const runs = extractionStatuses().map((run) => ({
|
||||
chatId: run.chatId,
|
||||
// The run's own label (the file path for notes imports) wins; chat
|
||||
// runs fall back to the chat store's title.
|
||||
title: run.label ?? chatStore.getChat(run.chatId)?.title ?? run.chatId,
|
||||
// runs fall back to the chat store's title. Chunk jobs append their
|
||||
// part ("· 2/5") — one row per chunk of the fan-out.
|
||||
title:
|
||||
(run.label ?? chatStore.getChat(run.chatId)?.title ?? run.chatId) +
|
||||
(run.part ? ` · ${run.part.i}/${run.part.n}` : ""),
|
||||
phase: run.phase,
|
||||
startedAt: run.startedAt,
|
||||
// Live per-stage usage (the money channel): model + per-step CallUsage,
|
||||
|
||||
Reference in New Issue
Block a user