Extraction-UX data layer: the chunk budget becomes a graph setting; the chat view gets a per-session coverage read
Owner-locked extraction-UX redesign (dispatch 2026-09-24), part 1 (data
layer; the UI chunk follows).
THRESHOLD -> SETTING
- config.ts's EXTRACTION.chunkTokens dies; the default moves into
FACTORY_SETTINGS.graph.extractionChunkTokens (4096, the old constant's
value). GraphSettings gains the optional field, Required at resolution
via the standard ?? FACTORY pattern — pre-redesign records resolve the
default cleanly.
- FLOOR 500 (the implementer's call, documented on the field): every run
pays the graph-carrying prompt overhead out of the write channel's
money — a smaller budget fires penny-batches that cost more per
extracted token than the content is worth. The sanitizer CLAMPS up
(bd68486 clamp-never-reject discipline: fractions floor, junk drops);
the settings PUT REJECTS below-floor with the rationale in the message.
No ceiling: a large budget just delays extraction — the user's stated
cost/thoroughness tradeoff, not a misconfiguration to police.
- The trigger (turn-service.triggerExtraction) takes the budget as a
parameter — chat-handler passes effective.graph.extractionChunkTokens,
the fresh per-turn read, same as every other gate.
- The pipeline's BATCH side threads through the same number:
ExtractionPipelineDeps gains an OPTIONAL getChunkTokens getter with the
factory default (Teacher's 24 test fixtures stay untouched; the two
call-site lines inside extraction.ts are the only cross-band edits,
flagged). chat-handler + backfill/import-notes wire the live
effectiveSettings — trigger and batches share one budget, always.
PER-SESSION COVERAGE READ
- sessionCoverage(chatId) (server/chats.ts): the chat view's ring-meter
block — totalMessages/coveredCount/uncoveredCount/coveredTokens/
uncoveredTokens (the same coverageOf derivation /api/graph/coverage
serves, scoped to one chat), running (chat-kind extraction registered
for this id), and lastFoldedAt (the chat's newest coverage row's fold
— a cheap single pass; null before any fold). GET /api/chats/[id]
carries it as coverage: {...}; unknown chat -> null (the route 404s
first).
- /api/graph/coverage (the all-sessions list) compiles untouched — the
per-session read is a scoping of it, not a supersession; the UI's
sidebar/ring chunk decides which surface it consumes.
Tests: factory/sanitizer pins (clamp-up, fraction-floor, junk-drop),
resolution pins (absent -> 4096, stored 2000 wins), settings-body pins
(valid/absent/fraction/below-floor 400/junk 400), sessionCoverage pins
(derived split, newest-fold scoping, cross-session isolation, running via
the registry, null on unknown chat). 893/893.
This commit is contained in:
@@ -317,7 +317,10 @@ export async function handleChatRequest(
|
||||
// pipeline knobs — embedding/clustering/labeling — gate inside the
|
||||
// pipeline and its consumers, later waves, via the same resolved view.)
|
||||
if (extractionEnabled(effective.graph)) {
|
||||
triggerExtraction(stores, extraction, chatId);
|
||||
// Fresh per-turn read of the graph section's chunk budget
|
||||
// (extraction-UX redesign, 2026-09-24): the trigger compares
|
||||
// against it and the pipeline's batches share it.
|
||||
triggerExtraction(stores, extraction, chatId, effective.graph.extractionChunkTokens);
|
||||
}
|
||||
// LAST: a poller that sees not-in-flight must read an already-persisted
|
||||
// record — endTurn only after persistTurn has done its job.
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
import { abortExtraction, extractionStatuses } from "../agent/subagents/extraction.ts";
|
||||
import { coverageOf } from "../core/coverage.ts";
|
||||
import { deleteChatTraces, type ChatDeleteMode, type ChatDeletionStats } from "../core/graph-deletion.ts";
|
||||
import { titleFromPrompt } from "../core/contracts.ts";
|
||||
import type { UIMessage } from "ai";
|
||||
import { graphRevision } from "./graph.ts";
|
||||
import { stores } from "./stores.ts";
|
||||
import { isTurnRunning } from "./turn-liveness.ts";
|
||||
@@ -110,3 +112,47 @@ export async function deleteChatSession(chatId: string, mode: ChatDeleteMode): P
|
||||
stores.chatStore.deleteChat(chatId);
|
||||
return { ok: true, mode, stats, jobsRemoved, graphRevision: graphRevision() };
|
||||
}
|
||||
|
||||
/** One session's coverage read (extraction-UX redesign, owner dispatch
|
||||
* 2026-09-24): the chat view's ring meter + popover data — the same
|
||||
* derivation the all-sessions list serves (/api/graph/coverage), scoped
|
||||
* to one chat. "Uncovered" stays DERIVED per request (fingerprint the
|
||||
* transcript, diff the coverage ledger — never dual-tracked).
|
||||
*
|
||||
* lastFoldedAt: the newest coverage row's fold timestamp — a cheap single
|
||||
* pass over the chat's ledger rows (the ring popover's "last extraction"
|
||||
* line); null before the session's first fold. */
|
||||
export interface SessionCoverage {
|
||||
totalMessages: number;
|
||||
coveredCount: number;
|
||||
uncoveredCount: number;
|
||||
coveredTokens: number;
|
||||
uncoveredTokens: number;
|
||||
/** True while a chat-kind extraction run is registered for this session. */
|
||||
running: boolean;
|
||||
/** ISO of the newest fold in this session's coverage rows; null before any. */
|
||||
lastFoldedAt: string | null;
|
||||
}
|
||||
|
||||
/** The per-session coverage block (see SessionCoverage). Null when the
|
||||
* chat doesn't exist — callers own the 404. */
|
||||
export function sessionCoverage(chatId: string): SessionCoverage | null {
|
||||
const record = stores.chatStore.getChat(chatId);
|
||||
if (!record) return null;
|
||||
const entries = stores.graphStore.readCoverage();
|
||||
const cov = coverageOf(record.messages as UIMessage[], entries, chatId);
|
||||
let lastFoldedAt: string | null = null;
|
||||
for (const entry of entries) {
|
||||
if (entry.chatId !== chatId) continue;
|
||||
if (lastFoldedAt === null || entry.foldedAt > lastFoldedAt) lastFoldedAt = entry.foldedAt;
|
||||
}
|
||||
return {
|
||||
totalMessages: cov.slots.length,
|
||||
coveredCount: cov.coveredCount,
|
||||
uncoveredCount: cov.uncoveredCount,
|
||||
coveredTokens: cov.coveredTokens,
|
||||
uncoveredTokens: cov.uncoveredTokens,
|
||||
running: extractionStatuses().some((run) => run.kind === "chat" && run.chatId === chatId),
|
||||
lastFoldedAt,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -22,7 +22,7 @@ import type {
|
||||
ModelsSettings,
|
||||
RoleModelOverride,
|
||||
} from "../core/settings.ts";
|
||||
import { BACKGROUND_MODEL_ROLES, MODEL_SLOTS } from "../core/settings.ts";
|
||||
import { BACKGROUND_MODEL_ROLES, EXTRACTION_CHUNK_TOKENS_FLOOR, MODEL_SLOTS } from "../core/settings.ts";
|
||||
import { TOGGLEABLE_TOOLS } from "../core/pending.ts";
|
||||
|
||||
/** Parsed-or-rejected — `ok: false` carries the 400's error message. */
|
||||
@@ -283,15 +283,33 @@ export function parseAppearanceSettingsBody(body: unknown): SettingsBodyResult<A
|
||||
export function parseGraphSettingsBody(body: unknown): SettingsBodyResult<GraphSettings> {
|
||||
const raw = asRecord(body);
|
||||
if (!raw) return { ok: false, error: "Invalid JSON body" };
|
||||
// Wave 7 collapse (owner ruling 2026-09-23): the section is the master
|
||||
// switch ONLY. PUT is replace-whole — a body like {enabled: true} IS the
|
||||
// whole section now; the dead rungs' keys in old records are unread
|
||||
// (the sanitizer drops them; see sanitizeGraph).
|
||||
// Wave 7 collapse (owner ruling 2026-09-23) + the extraction-UX
|
||||
// redesign (2026-09-24): the section is the master switch and the
|
||||
// chunk budget. PUT is replace-whole — a body like {enabled: true} IS
|
||||
// the whole section (extractionChunkTokens absent = factory default at
|
||||
// resolution); the dead rungs' keys in old records are unread (the
|
||||
// sanitizer drops them; see sanitizeGraph).
|
||||
const settings: GraphSettings = {};
|
||||
if (raw.enabled !== undefined) {
|
||||
if (typeof raw.enabled !== "boolean") return { ok: false, error: "enabled must be a boolean" };
|
||||
settings.enabled = raw.enabled;
|
||||
}
|
||||
if (raw.extractionChunkTokens !== undefined) {
|
||||
if (typeof raw.extractionChunkTokens !== "number" || !Number.isFinite(raw.extractionChunkTokens)) {
|
||||
return { ok: false, error: "extractionChunkTokens must be a number" };
|
||||
}
|
||||
const floored = Math.floor(raw.extractionChunkTokens);
|
||||
if (floored < EXTRACTION_CHUNK_TOKENS_FLOOR) {
|
||||
// the floor's rationale lives on GraphSettings.extractionChunkTokens:
|
||||
// every run pays the graph-carrying prompt overhead — penny-batches
|
||||
// cost more per extracted token than the content is worth.
|
||||
return {
|
||||
ok: false,
|
||||
error: `extractionChunkTokens must be at least ${EXTRACTION_CHUNK_TOKENS_FLOOR} (smaller batches waste the per-run graph overhead)`,
|
||||
};
|
||||
}
|
||||
settings.extractionChunkTokens = floored;
|
||||
}
|
||||
return { ok: true, settings };
|
||||
}
|
||||
|
||||
|
||||
@@ -19,7 +19,6 @@ import type { UIMessage } from "ai";
|
||||
import { withPlanBlocks } from "../core/plan-blocks.ts";
|
||||
import { coverageOf } from "../core/coverage.ts";
|
||||
import type { ChatStore, GraphStore } from "../core/ports.ts";
|
||||
import { EXTRACTION } from "../../config.ts";
|
||||
import type { ExtractionPipeline } from "../agent/subagents/extraction.ts";
|
||||
import { ModelsNotConfiguredError } from "../core/settings.ts";
|
||||
|
||||
@@ -74,11 +73,14 @@ export function persistTurn(chatStore: ChatStore, chatId: string, messages: UIMe
|
||||
/**
|
||||
* The graph write channel (vision §1 construction): the extraction pipeline
|
||||
* (extractor → auditor → merger — the sole graph writer) runs ASYNC once a
|
||||
* chat's UNCOVERED transcript tokens reach EXTRACTION.chunkTokens
|
||||
* chat's UNCOVERED transcript tokens reach the chunk budget
|
||||
* (vision-extraction-coverage.md, 2026-09-15 — replacing the old
|
||||
* every-3-quiz-answers count: content size is the honest signal, and
|
||||
* quiz-less sessions extract too). The pipeline itself extracts only the
|
||||
* uncovered batches. Fire-and-forget — never blocks the turn; failures are
|
||||
* quiz-less sessions extract too). The budget is the graph section's
|
||||
* extractionChunkTokens setting (extraction-UX redesign, 2026-09-24) —
|
||||
* the composition root passes the per-turn effective read (chat-handler
|
||||
* resolves fresh per turn, same as every other gate). The pipeline
|
||||
* itself extracts only the uncovered batches. Fire-and-forget — never blocks the turn; failures are
|
||||
* logged by the pipeline and leave coverage untouched, so the next trigger
|
||||
* (or the manual extract button) re-runs what's left.
|
||||
*
|
||||
@@ -90,13 +92,14 @@ export function triggerExtraction(
|
||||
deps: { chatStore: ChatStore; graphStore: GraphStore },
|
||||
extraction: ExtractionPipeline,
|
||||
chatId: string,
|
||||
chunkTokens: number,
|
||||
): void {
|
||||
if (process.env.NODE_ENV === "test") return;
|
||||
try {
|
||||
const record = deps.chatStore.getChat(chatId);
|
||||
if (!record) return;
|
||||
const covered = coverageOf(record.messages as UIMessage[], deps.graphStore.readCoverage(), chatId);
|
||||
if (covered.uncoveredTokens < EXTRACTION.chunkTokens) return;
|
||||
if (covered.uncoveredTokens < chunkTokens) return;
|
||||
void extraction.runExtraction(chatId).catch((err) => {
|
||||
// BYOK (owner, 2026-09-22): an unconfigured write channel SKIPS
|
||||
// (the coverage ledger keeps the messages uncovered — the next
|
||||
|
||||
Reference in New Issue
Block a user