From e95fc2630992b9171fa5a52f922fc5bb5c3663f9 Mon Sep 17 00:00:00 2001 From: amosblomqvist Date: Tue, 15 Sep 2026 17:16:31 +0300 Subject: [PATCH] =?UTF-8?q?docs:=20vision=20for=20extraction=20coverage=20?= =?UTF-8?q?=E2=80=94=20message-granular=20ledger,=20token-batched=20runs?= =?UTF-8?q?=20(2026-09-15=20ideation)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The extraction pipeline stops re-reading whole sessions: coverage is a per-chat covered-fingerprint ledger at MESSAGE granularity (tool call and result never separated — they share a message; quiz events not special — they ride re-opened messages normally), uncovered content is derived at read time (never dual-tracked), the trigger becomes uncovered tokens >= a chunk budget (fixing quiz-less sessions that never extract), and runs checkpoint coverage incrementally per merged batch — abort/error loses at most one in-flight batch. Manual extract-the-rest falls out of the diff. Pointer line under vision.md 'Construction'. --- docs/vision-extraction-coverage.md | 52 ++++++++++++++++++++++++++++++ docs/vision.md | 2 ++ 2 files changed, 54 insertions(+) create mode 100644 docs/vision-extraction-coverage.md diff --git a/docs/vision-extraction-coverage.md b/docs/vision-extraction-coverage.md new file mode 100644 index 0000000..a25f32c --- /dev/null +++ b/docs/vision-extraction-coverage.md @@ -0,0 +1,52 @@ +# Vision — Extraction coverage: message-granular ledger, token-batched runs + +Ideation settled 2026-09-15. Not yet planned, not yet assigned to a wave. This document is the decision record; `vision.md` § "The graph write channel" points here. + +One line: the extraction pipeline stops re-reading whole sessions and starts reading **only what it hasn't covered**. Coverage is tracked per **message** — a covered-fingerprint ledger per chat — runs are batched under a **token budget**, and the uncovered remainder is *derived*, never separately tracked. Abort or error loses at most one in-flight batch; nothing is ever silently left unextracted, and a manual "extract the rest" affordance falls out for free. + +--- + +## The problem (found 2026-09-15, live review) + +The pipeline's emission checkpoint (`SessionCheckpoint {chatId, foldedAt}`) is **whole-chat granularity**. Every run rebuilds the *entire* transcript (`transcriptView` from message 0) for both the extractor and the auditor — so total extraction cost over a session grows quadratically-ish with its length, the auditor re-sees everything, and the checkpoint only answers "has this chat ever folded". + +Worse, the trigger is `EXTRACTION.everyNQuizAnswers` — a count of answered quizzes not yet in the event ledger. Consequences: + +1. **Quiz-less sessions never extract.** A pure discussion session (no answered quizzes) accumulates unextracted knowledge forever. Quiz density is the wrong signal for "is there something to fold". +2. **Coverage holes are invisible.** An aborted or errored run checkpoints nothing (correct discipline) — but the hole it leaves is only detectable by re-counting quiz answers. Content the pipeline hasn't covered is nowhere recorded. Sessions are not granular enough; the unit was wrong. + +## The decision set (DECIDED 2026-09-15) + +**The message is the unit of coverage.** Not the session, not a chunk of text. A coverage ledger per chat records **which messages have been extracted** — append-only, each entry `{fingerprint, tokens, foldedAt}`. + +- *Why not infer coverage procedurally (from provenance):* provenance is a **lower bound** — a message the extractor saw and found nothing in leaves no trace, so "seen and empty" is indistinguishable from "never seen". An explicit record is required. +- *Why not chunk-identity:* if chunks are the recorded unit, one quiz answer merging into a message inside chunk 2 shifts token counts and can cascade boundary changes across every later chunk — mass re-opening. Message fingerprints localize the re-open to exactly the mutated message. + +**Coverage = the covered set; "uncovered" is derived, never tracked.** No dual bookkeeping: at read time, fingerprint the current transcript's messages, diff against the covered set. The complement *is* the coverage view — and the input to any "run extraction on what's missing" affordance. Two ledgers would drift; one ledger plus a deterministic diff cannot. + +**The fingerprint is over the extraction-relevant content**: role + text parts + tool parts *including their outputs* — exactly what `transcriptView` sees. It is stable under cosmetic mutation (plan-block part attach — data parts are invisible to the view) and changes exactly when the view changes. + +- *The quiz-merge wrinkle, handled by the fingerprint:* when the learner answers, the SDK part-merge fills the output **inside the assistant message that asked** — a message the previous run already covered. The fingerprint changes, the message re-opens itself (and nothing else), and the next run re-extracts that one message. The same behavior today's full re-read achieved, bounded to one message. + +**Tool calls and results are never separated (owner rule).** They live in the same message — coverage and batching are message-granular, and runs are consecutive-message groups — so a call and its result land in the same run by construction. No splitting rule is ever needed. + +**Quiz events are not special.** No separate quiz-outcome channel, no deterministic quiz-event pass — quiz parts are messages like any other tool content. They ride re-opened messages through the normal pipeline; the merger's attribution and the existing ref-idempotent appends do the rest. Nothing in the pipeline treats them differently. + +**Trigger: uncovered tokens ≥ the chunk budget.** Replaces `everyNQuizAnswers`. `EXTRACTION.chunkTokens` (a config knob; sizing at build time — the budget is really the transcript share of the extractor's context window, which also carries the graph and instructions; candidate range 2k–8k). Token estimate chars/4, same as the rest of the app. This fixes the quiz-less-session gap for free: text alone fires the trigger. Still fired at `onFinish` — content is settled there. + +**Runs are token-budgeted batches of the oldest uncovered messages.** Group consecutive uncovered messages (by transcript position) into runs under the chunk budget; execute sequentially; **checkpoint coverage incrementally per merged run** — the today-discipline ("nothing merged, nothing checkpointed") becomes per-batch atomic. An abort or mid-run error loses only the in-flight batch; every earlier batch's coverage is already durable, and the remainder stays visibly uncovered. Single-flight per chat unchanged. + +**Cost shape.** Chunking bounds the *transcript* share of each run; the auditor still sees the whole graph (identity needs it) — that share is untouched by this design. + +## What it buys + +- Abort/error never silently loses knowledge: the uncovered set is the standing record of what the pipeline hasn't covered, and it's derivable at any moment, per session. +- Long sessions extract linearly, not quadratically. +- The manual path is trivial: "extract uncovered content" = batch the diff and run. The UX (coverage readouts per session, an extract-now affordance, maybe a graph-view coverage strip) is a later wave — the data model makes all of it cheap. + +## Parked + +- The manual-run UI and coverage readouts (deliberately — the user experience gets its own pass). +- The one-shot context injection's cousin case: an aborted/errored injection is never retried for that session (one-shot ever). Not knowledge loss — a graph read — but if a manual re-run affordance lands for extraction, the injection could share it. +- The notes-import path (`src_` sources): whole-file semantics stay; files are one-shot imports, not growing sessions. +- `SessionCheckpoint`/`sessions.jsonl` migration shape: covered-set lines replace/augment the current whole-chat checkpoint; details at build time. \ No newline at end of file diff --git a/docs/vision.md b/docs/vision.md index 0bb92bd..dc431e0 100644 --- a/docs/vision.md +++ b/docs/vision.md @@ -39,6 +39,8 @@ Every node records every **(chat, message) point** where it came up; every edge ### Construction: free-form extraction, fully decoupled from sessions (DECIDED 2026-09-02 — supersedes live emission) +*Coverage settled 2026-09-15 — see `vision-extraction-coverage.md`: the message is the unit of coverage (a covered-fingerprint ledger per chat), runs are token-budgeted batches of the oldest uncovered messages, and the trigger is uncovered tokens ≥ a chunk budget (replacing everyNQuizAnswers — quiz-less sessions extract too).* + **Second amendment, reversing the 2026-08-22 live-emission decision.** The session's plan tree is the teacher's scaffolding for the session experience (§2); the global graph does not depend on it — structurally or otherwise. What the teacher formalizes is session-relative; the graph must not inherit those choices. (The original design doc's post-hoc stance was right after all.) Three stages, one pipeline — the sole graph writer: