sorokin.oleg.edu/learnharness
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 6 Wiki Activity

teacher-prose-only extraction view: coverage, fingerprints, tokens, and batches derive from the teacher's text blocks alone (owner 2026-09-28)

Browse Source
This commit is contained in:
amosblomqvist
2026-09-28 22:37:18 +03:00
parent c17405a3d7
commit 1f5e21a674
7 changed files with 206 additions and 345 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/agent/subagents/extraction.ts
+28 -23
View File
@@ -26,7 +26,8 @@ import { getGraph, mergeFindings, type EmissionStats, type GraphDeps } from "../
import type { GraphStore } from "../../core/ports.ts";
import { settleGraphLayout } from "../../core/layout-settle.ts";
import { batchRuns, coverageOf, messageViewLines } from "../../core/coverage.ts";
import { sessionMarkdown } from "../../core/session-markdown.ts";
import { teacherProseBlocks } from "../../core/coverage.ts";
import type { LhMessage } from "../../core/native-messages.ts";
import { auditorGraphTools } from "../tools/graph.ts";
/**
@@ -83,22 +84,29 @@ import { auditorGraphTools } from "../tools/graph.ts";
// ─── the source view (what the extractor sees) ──────────────────────────────
/**
* A session rendered as readable markdown — the EXTRACTOR's view (owner
* ruling 2026-09-27): the whole session goes in, formatted by the ONE
* shared session-markdown engine (core/session-markdown.ts — the same
* engine the user-facing session export will use), with the message-id
* anchors preserved (the extractor reports them back per
* proposition/edge — they become provenance). TREE-BLIND by design:
* no plan tree, no mark_done.
*
* Presentation-only: coverage fingerprints hash core/coverage.ts's line
* format, which is SEPARATE and unchanged — re-formatting the view never
* re-opens covered history (the deliberate 2026-09-27 split).
* The extraction view: the session as TEACHER PROSE ONLY (owner ruling,
* 2026-09-28 — supersedes the 2026-09-27 whole-session markdown view):
* plain prose paragraphs under the session title. The shared markdown
* engine (core/session-markdown.ts) keeps the full session format for
* the future user-facing export; the extractor no longer uses it.
*/
/** The whole-record view (used by callers that render a full session;
* the pipeline itself batches). TEACHER PROSE ONLY (owner, 2026-09-28).
*/
export function transcriptView(record: ChatRecord): string {
// anchors OFF (submit-schema slim, 2026-09-28): propositions no longer
// carry message ids, so the [id] anchors are noise for the extractor.
return sessionMarkdown(record.messages, { title: record.title, anchors: false });
return proseView(record.messages, record.title);
}
/** The extraction view under the teacher-prose ruling (owner, 2026-09-28):
* ONLY the teacher agent's text blocks — no learner messages, no
* thinking, no tool calls or results, no quiz blocks. Plain paragraphs
* under the session title; the shared markdown engine stays reserved for
* the future user-facing export. */
function proseView(messages: ReadonlyArray<LhMessage>, title?: string): string {
const prose = teacherProseBlocks(messages);
const head = title && title.length > 0 ? `# ${title}\n\n` : "";
return head + prose.join("\n\n");
}
// ─── the extractor ──────────────────────────────────────────────────────────
@@ -866,15 +874,12 @@ export function createExtractionPipeline(deps: ExtractionPipelineDeps): Extracti
`${covered.uncoveredCount} messages, ${batches.length} batch(es) of ≤${chunkTokens} tok`,
);
for (const batch of batches) {
// The batch's source view: the ONE shared markdown engine
// (owner 2026-09-27) over the batch's messages. Anchors OFF
// (2026-09-28): propositions report no message ids, so the
// [id] anchors are noise — same as transcriptView. The
// fingerprint's line format is separate and unchanged (no
// mass re-extraction).
const view = sessionMarkdown(
// The batch's source view: TEACHER PROSE ONLY (owner,
// 2026-09-28) — plain paragraphs, same selection as the
// coverage fingerprints (messageViewLines parity).
const view = proseView(
batch.map((slot) => slot.message),
{ title: record.title, anchors: false },
record.title,
);
setPhase(chatId, "extracting", "chat");
const findings = await extract(chatId, view);
src/core/coverage.ts
+59 -151
View File
@@ -1,45 +1,31 @@
/**
* Extraction coverage (vision-extraction-coverage.md, settled 2026-09-15) —
* the pure core. THE MESSAGE IS THE UNIT OF COVERAGE (Wave D, owner ruling
* 2026-09-26: the native message, not the old projected group): a per-chat
* append-only ledger records which messages have been extracted, by CONTENT
* FINGERPRINT over exactly what the extraction view sees; "uncovered" is
* DERIVED at read time (fingerprint the current transcript, diff against the
* ledger) — never dual-tracked. Runs are token-budgeted batches of the oldest
* uncovered messages, and THE ONE BATCH RULE (owner): a tool-call message and
* its tool-result message(s) are INSEPARABLE — they always appear in the same
* chunk (a covered partner rides along as context; checkpoints are
* idempotent, so re-marking it is a no-op).
* Extraction coverage — the pure core. THE TEACHER'S PROSE IS THE UNIT
* (owner ruling, 2026-09-28): the extractor sees ONLY the teacher agent's
* text blocks — no learner messages, no thinking, no tool calls or
* results, no quiz blocks. Everything else is INVISIBLE to coverage: it
* neither opens nor joins a run, so answering a quiz or editing a learner
* message triggers nothing. "Covered" is by construction exactly "what
* the extractor saw": the fingerprint (sha256 of a message's view lines)
* and the extractor's prompt share ONE selection — the teacher prose.
*
* The fingerprint and the LLM prompt share ONE line builder —
* messageViewLines — so "covered" and "what the model saw" cannot drift
* apart. The line logic is tree-blind, identical to the extractor's view:
* user/teacher text verbatim, quiz outcomes as one-line summaries, other
* clean tool results as one generic line each, markers invisible (a
* plan-block marker does not re-open a covered message).
* A per-chat append-only ledger records which messages have been
* extracted by CONTENT FINGERPRINT; "uncovered" is DERIVED at read time
* (fingerprint the current transcript, diff against the ledger) — never
* dual-tracked. Runs are token-budgeted batches of the oldest uncovered
* teacher-prose messages: greedy consecutive fill, a message is never
* split, a single over-budget message rides alone. (The old
* inseparability bond — call + results in one chunk — died with the
* ruling: tool results are invisible now, so nothing needs keeping
* together.)
*/
import { createHash } from "node:crypto";
import type { CoverageEntry } from "./contracts/graph.ts";
import type { LhMessage } from "./native-messages.ts";
import type { ToolResultMessage } from "@earendil-works/pi-ai";
export type { CoverageEntry };
/** The quiz outcome line — both the fingerprint and the LLM view see the
* quiz answer, so an answered quiz re-opens (the answer is its own native
* message now; its fingerprint is new the moment it lands). */
function quizLine(result: ToolResultMessage): string {
const out = result.details as
| Partial<{ questionText: string; correct: boolean; dontKnow: boolean; note: string }>
| undefined;
const question = typeof out?.questionText === "string" ? out.questionText : "(no question)";
const outcome = out?.dontKnow === true ? "did not know" : out?.correct === true ? "answered correctly" : "answered incorrectly";
const note = typeof out?.note === "string" && out.note ? ` (note: ${out.note})` : "";
return `QUIZ [${result.toolCallId}] ${question} — the learner ${outcome}${note}`;
}
/** The user-facing text of a native message (string or text blocks). */
function textOf(content: unknown): string[] {
if (typeof content === "string") return content.trim() ? [content] : [];
@@ -52,47 +38,46 @@ function textOf(content: unknown): string[] {
/**
* The extraction-relevant lines of ONE native message — the shared seam
* between transcriptView (the LLM's prompt) and messageFingerprint
* (coverage). Same logic, same order, so a covered message is by
* construction exactly what the extractor saw. `mid` is the provenance tag
* (the message id; the transcript view passes an index fallback for legacy
* id-less messages, coverage passes the id — persisted messages always have
* ids).
* between the extractor's prompt and messageFingerprint (coverage).
* TEACHER PROSE ONLY (owner, 2026-09-28): an assistant message yields one
* line per text block; everything else (user messages, tool results,
* markers) yields NOTHING and is invisible to coverage. `mid` is the
* message id — part of the hash input, so the same prose in two messages
* is two slots (per-message coverage).
*/
export function messageViewLines(message: LhMessage, mid: string): string[] {
const lines: string[] = [];
const role = (message as { role?: string }).role;
if (role === "user") {
for (const text of textOf((message as { content: unknown }).content)) {
lines.push(`LEARNER [${mid}] ${text.trim()}`);
}
} else if (role === "assistant") {
for (const text of textOf((message as { content: unknown }).content)) {
lines.push(`TEACHER [${mid}] ${text.trim()}`);
}
} else if (role === "toolResult") {
const result = message as ToolResultMessage;
if (result.isError) return lines; // a failed call is retry context, not learning evidence
lines.push(
result.toolName === "quiz" ? quizLine(result) : `(${result.toolName} tool result delivered)`,
);
if ((message as { role?: string }).role !== "assistant") return lines;
for (const text of textOf((message as { content: unknown }).content)) {
lines.push(`TEACHER [${mid}] ${text.trim()}`);
}
// custom markers are invisible to the extraction view.
return lines;
}
/**
* The teacher's prose blocks of a transcript, in order — the extractor's
* prompt material. SAME selection as messageViewLines (the parity
* doctrine survives the ruling in this one function pair), rendered
* without the provenance prefix: plain paragraphs.
*/
export function teacherProseBlocks(messages: ReadonlyArray<LhMessage>): string[] {
const out: string[] = [];
for (const message of messages) {
if ((message as { role?: string }).role !== "assistant") continue;
for (const text of textOf((message as { content: unknown }).content)) out.push(text.trim());
}
return out;
}
/** Tokens of a message's view — chars/4, ceil, the app-wide estimate. */
export function viewTokens(lines: string[]): number {
return Math.ceil(lines.join("\n").length / 4);
}
/**
* A message's coverage fingerprint: sha256 of role + its view lines (with
* the message's real id as the provenance tag). Changes exactly when the
* extraction view of the message changes. (Wave D: the unit is the native
* message — an answered quiz is the answer's OWN toolResult message, so
* answering opens exactly that new message; the asking message is pulled
* into the batch by the inseparability rule, not re-opened.)
* A message's coverage fingerprint: sha256 of role + its view lines (the
* teacher prose, with the message id as the tag). Changes exactly when
* the teacher prose of the message changes — nothing else can re-open it.
*/
export function messageFingerprint(message: LhMessage): string {
const h = createHash("sha256");
@@ -122,11 +107,11 @@ export function coveredSet(entries: ReadonlyArray<CoverageEntry>, chatId: string
/** The derived state: which slots are uncovered, plus the token totals.
* "Uncovered" needs no ledger of its own — this diff IS the coverage view.
* `uncovered` is INDEXES into `slots` (the batcher needs positions for the
* inseparability rule). Messages with NO view lines (custom markers, an
* error result) are invisible to coverage entirely: the extractor can
* never see them, so they are neither covered nor uncovered — a marker
* attach never opens (or joins) an extraction run. */
* `uncovered` is INDEXES into `slots` (the batcher batches by position).
* Messages with NO view lines — every learner message, every tool result,
* markers, prose-less assistant turns — are invisible to coverage
* entirely: the extractor can never see them, so they are neither covered
* nor uncovered. */
export function coverageOf(
messages: ReadonlyArray<LhMessage>,
entries: ReadonlyArray<CoverageEntry>,
@@ -166,86 +151,15 @@ export function coverageOf(
};
}
// ─── the inseparability rule (owner, Wave D fork 3) ─────────────────────────
//
// A tool-call message and its tool-result message(s) always appear in the
// same chunk. Bonding is adjacency-honest: results follow their call's
// assistant message contiguously in the native record (the agent loop
// appends the results batch; the question gate guarantees nothing can land
// between a quiz call and its learner answer). A result whose call's
// assistant message has NO other bond partners still pulls that message in
// (the extractor needs the question with its answer).
/** The call ids an assistant message carries. */
function callIdsOf(message: LhMessage): Set<string> {
const out = new Set<string>();
const content = (message as { content?: unknown }).content;
if (!Array.isArray(content)) return out;
for (const block of content as Array<{ type?: string; id?: string }>) {
if (block.type === "toolCall" && typeof block.id === "string") out.add(block.id);
}
return out;
}
/** The contiguous bond-range containing slot i: an assistant-with-calls
* message plus its immediately-following results; a toolResult message
* plus the assistant message that made its call (and that call's sibling
* results). Anything else bonds to itself alone. */
function bondRange(messages: ReadonlyArray<LhMessage>, i: number): [number, number] {
const roleOf = (m: LhMessage | undefined) => (m ? (m as { role?: string }).role : undefined);
const m = messages[i];
if (!m) return [i, i];
if (roleOf(m) === "assistant") {
const calls = callIdsOf(m);
if (calls.size === 0) return [i, i];
let end = i;
while (
roleOf(messages[end + 1]) === "toolResult" &&
calls.has((messages[end + 1] as ToolResultMessage).toolCallId)
) {
end++;
}
return [i, end];
}
if (roleOf(m) === "toolResult") {
let start = i;
while (roleOf(messages[start - 1]) === "toolResult") start--;
const owner = messages[start - 1];
if (
owner &&
roleOf(owner) === "assistant" &&
callIdsOf(owner).has((m as ToolResultMessage).toolCallId)
) {
const [s, e] = bondRange(messages, start - 1);
return [s, Math.max(e, i)];
}
return [start, i]; // orphaned results (no asking message): keep the contiguous result run together
}
return [i, i];
}
/**
* Batch the uncovered messages into token-budgeted RUNS: consecutive slots,
* oldest first, greedy fill — a message is never split, and a bond-range
* (call + results) never splits either (owner fork 3): a selected uncovered
* slot EXPANDS to its whole bond-range, covered partners riding as context
* (checkpoint idempotency makes re-marking them a no-op). A single
* bond-range larger than the budget becomes its own (over-budget) run
* rather than being dropped. Runs are ephemeral — the durable record is
* per-message coverage, checkpointed after each run merges.
* Batch the uncovered teacher-prose messages into token-budgeted RUNS:
* consecutive slots, oldest first, greedy fill — a message is never split.
* A single message larger than the budget becomes its own (over-budget)
* run rather than being dropped. Runs are ephemeral — the durable record
* is per-message coverage, checkpointed after each run merges.
*/
export function batchRuns(slots: ReadonlyArray<CoverageSlot>, uncovered: ReadonlyArray<number>, budget: number): CoverageSlot[][] {
if (budget <= 0 || uncovered.length === 0) return [];
const messages = slots.map((s) => s.message);
// Expand: every uncovered index marks its whole bond-range as needed.
const needed = new Set<number>();
for (const i of uncovered) {
const [start, end] = bondRange(messages, i);
for (let j = start; j <= end; j++) needed.add(j);
}
// Group the needed slots in order, greedy by budget, never splitting a
// bond-range: a range that would overflow starts a new run (and an
// over-budget range rides alone).
const runs: CoverageSlot[][] = [];
let current: CoverageSlot[] = [];
let currentTokens = 0;
@@ -254,17 +168,11 @@ export function batchRuns(slots: ReadonlyArray<CoverageSlot>, uncovered: Readonl
current = [];
currentTokens = 0;
};
for (let i = 0; i < slots.length; i++) {
if (!needed.has(i)) continue;
const [start, end] = bondRange(messages, i);
const rangeTokens = slots.slice(start, end + 1).reduce((t, s) => t + s.tokens, 0);
if (current.length > 0 && currentTokens + rangeTokens > budget) flush();
// The whole range enters together — skip past its end in the outer loop.
for (let j = start; j <= end; j++) {
current.push(slots[j]!);
currentTokens += slots[j]!.tokens;
}
i = end;
for (const i of uncovered) {
const slot = slots[i]!;
if (current.length > 0 && currentTokens + slot.tokens > budget) flush();
current.push(slot);
currentTokens += slot.tokens;
}
flush();
return runs;