feat: extraction coverage — message-granular ledger + token-batched runs (d132373 vision)
THE MESSAGE IS THE UNIT OF COVERAGE (vision-extraction-coverage.md): a per-chat append-only ledger (coverage.jsonl) records which messages the pipeline has extracted, by content fingerprint over exactly what the extraction view sees (one shared line builder — core/coverage.ts); the uncovered remainder is DERIVED at read time, never dual-tracked. A quiz answer merging into the asking message re-opens only that message; plan-block data-part attach never re-opens anything. - The trigger (turn-service) replaces every-3-quiz-answers with uncovered-tokens >= EXTRACTION.chunkTokens (4096) — quiz-less sessions extract too, content size is the honest signal. - runExtraction is now chunked: token-budgeted batches of the oldest uncovered consecutive messages (a tool call + result never split), each merged batch checkpointing its messages' coverage atomically — an abort or error loses at most the in-flight batch; merged batches stay covered. - Observation refs are batch-tagged (obs:<batchTag>:<sourceId>): re-runs of the same batch dedupe; a re-opened message's fresh understanding lands as a new event. Fixes the old obs:p1 position-collision. - New API: GET /api/graph/coverage (exact per-session derived coverage + running flags, most-uncovered first) and POST /api/graph/coverage/extract (manual run for one session, 409 while in flight). - Graph view chrome: the coverage button opens a panel with every session's covered/uncovered tokens and a per-session Extract button — the manual answer to aborted runs, errors, and dormant tails. Renders even when the graph is empty: an empty graph with uncovered sessions is exactly when it matters. Polls only while open. - Notes imports keep the whole-file checkpoint path (coverage undefined). - Tests: pure-core coverage suite (11), chunked-pipeline suite (3: multi-batch + no-op, abort-keeps-earlier-batches + remainder re-run, re-open alone + data-part invisibility), panel render suite (5); pendingQuizAnswers and its tests deleted as dead code. 486/486. Live-smoked on real GLM: manual extract of "teach me modal realism" ran one batch, wrote 4 coverage entries, panel flipped to "1.3k covered · complete" with the button disabled, graph refetched (nodes 11), usage persisted with real stats.
This commit is contained in:
@@ -66,6 +66,7 @@ import Link from "next/link";
|
||||
import { FileTextIcon, Loader2Icon, SearchIcon, SettingsIcon, UploadIcon, WaypointsIcon } from "lucide-react";
|
||||
import { incrementalLayout, type GraphNode, type GraphSnapshot } from "@/src/core";
|
||||
import { GraphCanvasRenderer, type GraphRenderSettings } from "./graph-canvas.ts";
|
||||
import { GraphCoverageControl } from "./graph-coverage.tsx";
|
||||
import {
|
||||
ARROW_LENGTH_RANGE,
|
||||
CHARGE_RANGE,
|
||||
@@ -843,11 +844,15 @@ export function GraphView() {
|
||||
)}
|
||||
|
||||
{/* Legend + filters: search box and per-state toggle rows. */}
|
||||
{/* Top-right controls — three separate floating containers (owner,
|
||||
{/* Top-right controls — separate floating containers (owner,
|
||||
graph-view round 2): the search box, the settings gear (its own
|
||||
collapsible component), and the import button. */}
|
||||
{!isEmpty && (
|
||||
<div className="absolute right-3 top-3 z-10 flex items-center gap-2">
|
||||
collapsible component), the import button, and the coverage
|
||||
control. Coverage renders even when the graph is EMPTY — an
|
||||
empty graph with unextracted sessions is exactly when it
|
||||
matters (extract → the graph gains nodes). */}
|
||||
<div className="absolute right-3 top-3 z-10 flex items-center gap-2">
|
||||
{!isEmpty && (
|
||||
<>
|
||||
{/* Search — its own container (icon + input + no-match note). */}
|
||||
<div className="w-56 rounded-xl border bg-card/85 p-1.5 shadow-lg backdrop-blur">
|
||||
<div className="relative">
|
||||
@@ -893,8 +898,12 @@ export function GraphView() {
|
||||
gear sits at the far right (owner, round 3) so its panel
|
||||
drops at the very right edge, over less of the graph. */}
|
||||
<GraphSettingsControl settings={settings} changeSettings={changeSettings} />
|
||||
</div>
|
||||
)}
|
||||
</>
|
||||
)}
|
||||
{/* Extraction coverage — own container, always present; the
|
||||
panel re-reads when background runs settle (refreshTick). */}
|
||||
<GraphCoverageControl refreshKey={refreshTick} />
|
||||
</div>
|
||||
|
||||
{/* Background pipeline status: what is currently running. */}
|
||||
{pipelineRuns.length > 0 && (
|
||||
|
||||
Reference in New Issue
Block a user