digest-aware serialized extraction + chain simplification replace the spine stage
Owner live-test verdict on the spine (2026-09-30): no perceptible gain — the real problems (restatement nodes, shallow re-derivations, same conclusion from many angles) are mint-time issues a post-hoc chain minter cannot fix. This arc moves assembly INTO the extractor and adds the deterministic dedupe the owner specified: - Digest-aware extraction: each stretch's extractor prompt carries THE GRAPH SO FAR (its own source's merged propositions + chains, ids verbatim); build on existing ids instead of restating, lean on established conclusions as premises. Mixed-namespace chain endpoints (fresh p-ids OR existing n_-ids) validated at the submit seam. - Serialized per source: a source's batches/chunks run strictly in order (each extract reads every earlier merge); cross-source runs stay parallel; source-wide abort stops the loop. The spine stage, role, phase, usage stage, cost row, and setup roster are gone. - Core-ideas prompt: goal reframed as uncovering core ideas and showing how everything derives from the foundations; direction upward, prefer lean derivations, never re-derive what the graph already holds. - Chain simplification (merger maintenance): simplifyChains drops provably redundant chains at every merge — prefer-fewest-premises + ancestry-cover ([A,B]⟹X with [X,C]⟹Y covers [A,B,C]⟹Y, transitive); same-key provenance lines drop together; rewriteLedgers under the audit lock; droppedChains counted separately in stats. Chunk budget stays 8192 (owner: keep for now, maybe lower later).
This commit is contained in:
amosblomqvist
+285
-232
@@ -23,7 +23,7 @@ import type {
|
||||
ExtractionStageUsage,
|
||||
SessionFindings,
|
||||
} from "../../core/contracts.ts";
|
||||
import { getGraph, mergeFindings, type EmissionStats, type GraphDeps } from "../../core/graph.ts";
|
||||
import { getGraph, mergeFindings, type DerivedGraph, type EmissionStats, type GraphDeps } from "../../core/graph.ts";
|
||||
import type { GraphStore } from "../../core/ports.ts";
|
||||
import { settleGraphLayout } from "../../core/layout-settle.ts";
|
||||
import { batchRuns, coverageOf, messageViewLines } from "../../core/coverage.ts";
|
||||
@@ -31,38 +31,43 @@ import { teacherProseBlocks } from "../../core/coverage.ts";
|
||||
import type { LhMessage } from "../../core/native-messages.ts";
|
||||
import { auditorGraphTools } from "../tools/graph.ts";
|
||||
import { runSubmitAgent, healStringifiedArrays } from "./submit-agent.ts";
|
||||
import {
|
||||
SPINE_INSTRUCTIONS,
|
||||
NOTES_SPINE_INSTRUCTIONS,
|
||||
SPINE_EMPTY,
|
||||
buildSpinePrompt,
|
||||
combineFindings,
|
||||
runSpine,
|
||||
spineHasDigest,
|
||||
type SpineSubmit,
|
||||
} from "./spine.ts";
|
||||
|
||||
/**
|
||||
* The extraction pipeline (vision §1 construction, 2026-09-02 redesign;
|
||||
* relinked 2026-09-27) — the sole graph writer. Three stages, one
|
||||
* orchestrator:
|
||||
* orchestrator (the spine stage came and went, 2026-09-30: serialized
|
||||
* digest-aware extraction replaced it — the owner's live test showed
|
||||
* the post-hoc assembly pass added complexity without perceptible
|
||||
* gain; the extractor itself is the assembler now):
|
||||
*
|
||||
* extractor (LLM) the ONE stage that sees the raw source (a session
|
||||
* rendered as the teacher's prose, or a notes file
|
||||
* verbatim) → submits propositions + REASONING CHAINS
|
||||
* (owner redesign, 2026-09-30), expanded at the seam to
|
||||
* SessionFindings' follows-edges. TREE-BLIND,
|
||||
* graph-blind, FULLY TRUSTED — nothing downstream
|
||||
* re-checks it.
|
||||
* verbatim, walked in stretches) → submits NEW core
|
||||
* ideas + reasoning chains (owner redesign,
|
||||
* 2026-09-30). DIGEST-AWARE (2026-09-30 evening):
|
||||
* when the source already yielded material, the
|
||||
* prompt carries THE GRAPH SO FAR (that source's
|
||||
* propositions + chains, ids verbatim) and the
|
||||
* extractor builds on it — referencing existing ids
|
||||
* instead of restating, leaning on established
|
||||
* conclusions as premises. Stretches of one source
|
||||
* run SERIALLY (each extract reads every earlier
|
||||
* merge); tree-blind, FULLY TRUSTED — nothing
|
||||
* downstream re-checks it.
|
||||
* auditor (LLM) never sees the source. Its job is IDENTITY: for each
|
||||
* proposition, linkTo an existing graph node or create —
|
||||
* decided by EXPLORING the graph through tools (find /
|
||||
* list_sessions / list_propositions), not by an injected
|
||||
* graph dump (context bloat, owner 2026-09-27). Links are
|
||||
* its ONLY deliverable — it never mints edges (owner
|
||||
* 2026-09-28: the graph connects across sources solely
|
||||
* through identified equivalences).
|
||||
* merger (core) mergeFindings() → deterministic ledger appends
|
||||
* NEW proposition, linkTo an existing graph node or
|
||||
* create — decided by EXPLORING the graph through
|
||||
* tools (find / list_sessions / list_propositions),
|
||||
* not by an injected graph dump (context bloat,
|
||||
* owner 2026-09-27). Links are its ONLY deliverable
|
||||
* — it never mints edges (owner 2026-09-28: the
|
||||
* graph connects across sources solely through
|
||||
* identified equivalences). Input is PROPOSITIONS
|
||||
* only — never chains (owner 2026-09-30).
|
||||
* merger (core) mergeFindings() → deterministic ledger appends +
|
||||
* chain simplification (2026-09-30 evening: the
|
||||
* ancestry-cover / prefer-fewest-premises pass that
|
||||
* strips redundant derivations — core/graph.ts)
|
||||
*
|
||||
* The events ledger is GONE (2026-09-27): quiz outcomes and understanding
|
||||
* observations are no longer formalized in the graph — the graph is purely
|
||||
@@ -126,7 +131,7 @@ function proseView(messages: ReadonlyArray<LhMessage>, title?: string): string {
|
||||
// ─── the extractor ──────────────────────────────────────────────────────────
|
||||
|
||||
const propositionProps = {
|
||||
id: Type.String({ description: 'Local id: "p1", "p2", … — referenced by chains.' }),
|
||||
id: Type.String({ description: 'Local id: "p1", "p2", … — referenced by chains in this call.' }),
|
||||
name: Type.String({
|
||||
description:
|
||||
"A very short label — established terminology, the term the discipline itself uses, a glossary " +
|
||||
@@ -135,7 +140,7 @@ const propositionProps = {
|
||||
}),
|
||||
statement: Type.String({
|
||||
description:
|
||||
"One true, self-contained, world-facing reasoning step at full precision. Never about the " +
|
||||
"One true, self-contained, world-facing core idea at full precision. Never about the " +
|
||||
'learner, the teacher, or the reading: not "the learner understands X" but "X". A misconception ' +
|
||||
"enters as its TRUE correction.",
|
||||
}),
|
||||
@@ -150,13 +155,20 @@ const propositionProps = {
|
||||
// REASONING CHAINS — premises + conclusion — stored AS-IS (chains-primary,
|
||||
// same evening: chains.jsonl is the truth; edges are a derived view, and
|
||||
// supports edges are retired entirely).
|
||||
const endpoint = (role: string) =>
|
||||
Type.String({
|
||||
description:
|
||||
`The id this ${role} — EITHER an existing graph idea id from THE GRAPH SO FAR in the prompt, ` +
|
||||
'verbatim ("n_…"), OR the local id of a proposition in this submission ("p1", …).',
|
||||
});
|
||||
|
||||
const chainSchema = Type.Object({
|
||||
premises: Type.Array(Type.String(), {
|
||||
premises: Type.Array(endpoint("premise uses"), {
|
||||
minItems: 1,
|
||||
description:
|
||||
"Proposition ids used TOGETHER: they jointly entail the conclusion; independent reasons are separate chains.",
|
||||
"Premise ids used TOGETHER: they jointly entail the conclusion; independent reasons are separate chains.",
|
||||
}),
|
||||
conclusion: Type.String({ description: "The proposition id the premises jointly imply." }),
|
||||
conclusion: endpoint("conclusion is"),
|
||||
});
|
||||
|
||||
// Weak-model tolerance (owner, 2026-09-25): the old zod `.default([])` /
|
||||
@@ -166,13 +178,15 @@ const chainSchema = Type.Object({
|
||||
// no zod left in the agent band.
|
||||
const findingsProps = {
|
||||
propositions: Type.Array(Type.Object(propositionProps), {
|
||||
description: "The propositions themselves — every distinct proposition the source's reasoning uses.",
|
||||
description:
|
||||
"Only the NEW core ideas this stretch contributes — ideas the prompt's THE GRAPH SO FAR already " +
|
||||
"holds are referenced by id in chains, never restated here.",
|
||||
}),
|
||||
chains: Type.Array(chainSchema, {
|
||||
description:
|
||||
"Every argument the source makes; conclusions reusable by id as premises of later chains; " +
|
||||
"bare propositions (no argument) ride the propositions list alone; pass [] when none — " +
|
||||
"never omit the field.",
|
||||
"Every argument this stretch makes — including arguments that join new ideas to established " +
|
||||
"ones; conclusions reusable by id as premises of later chains; bare ideas (no argument) ride " +
|
||||
"the propositions list alone; pass [] when none — never omit the field.",
|
||||
}),
|
||||
};
|
||||
|
||||
@@ -226,21 +240,24 @@ function fillFindings(raw: FindingsSubmit): FindingsSubmit {
|
||||
};
|
||||
}
|
||||
|
||||
/** The chain laws (extractor redesign, 2026-09-30) — the LOUD channel:
|
||||
* runSubmitAgent turns a problem string into an error tool result and
|
||||
* the model re-submits within its turns (the practice-proven hook).
|
||||
* Identical (premise-set, conclusion) chains are NOT errors — they
|
||||
* dedupe silently at the merge (chainKey): restating an argument is a
|
||||
* no-op. */
|
||||
function validateFindingsChains(value: FindingsSubmit): string | null {
|
||||
/** The chain laws (extractor redesign, 2026-09-30; digest-aware
|
||||
* 2026-09-30 evening) — the LOUD channel: runSubmitAgent turns a
|
||||
* problem string into an error tool result and the model re-submits
|
||||
* within its turns (the practice-proven hook). The MIXED NAMESPACE: an
|
||||
* endpoint resolves when it is a submitted proposition id OR an
|
||||
* existing graph node id (the extractor continues a partly-extracted
|
||||
* source and references digest ids verbatim). Identical (premise-set,
|
||||
* conclusion) chains are NOT errors — they dedupe silently at the
|
||||
* merge (chainKey): restating an argument is a no-op. */
|
||||
function validateFindingsChains(value: FindingsSubmit, existingIds: ReadonlySet<string>): string | null {
|
||||
const ids = new Set(value.propositions.map((p) => p.id));
|
||||
for (const chain of value.chains) {
|
||||
if (chain.premises.length === 0) {
|
||||
return "chains: premises must name at least one proposition — a proposition with no argument stays in the propositions list alone.";
|
||||
}
|
||||
const unknown = [...chain.premises, chain.conclusion].find((id) => !ids.has(id));
|
||||
const unknown = [...chain.premises, chain.conclusion].find((id) => !ids.has(id) && !existingIds.has(id));
|
||||
if (unknown !== undefined) {
|
||||
return `chains: "${unknown}" is not one of the submitted proposition ids — every premise and conclusion must appear in this call's propositions.`;
|
||||
return `chains: "${unknown}" is neither a submitted proposition id nor an existing idea id from THE GRAPH SO FAR — every premise and conclusion must resolve.`;
|
||||
}
|
||||
if (chain.premises.includes(chain.conclusion)) {
|
||||
return "chains: a conclusion cannot be its own premise.";
|
||||
@@ -277,59 +294,98 @@ function fillAudit(raw: Static<typeof auditSchema>): AuditVerdicts {
|
||||
// 2026-09-28 teacher-prose ruling and is simply omitted.)
|
||||
|
||||
export const SESSION_PREAMBLE =
|
||||
"You are the extraction stage of a learning application. You read one source — a teaching session, " +
|
||||
"rendered as the teacher's prose in reading order — and report what it teaches as findings for the " +
|
||||
"learner's global knowledge graph.";
|
||||
"You are the extraction stage of a learning application. You read one stretch of a source — a teaching " +
|
||||
"session, rendered as the teacher's prose in reading order — and report what it teaches as findings " +
|
||||
"for the learner's global knowledge graph, building on everything already extracted from this source.";
|
||||
|
||||
export const NOTES_PREAMBLE =
|
||||
"You are the extraction stage of a learning application. You read one source — one chunk of the " +
|
||||
"You are the extraction stage of a learning application. You read one stretch of a source — the " +
|
||||
"author's personal notes, rendered in reading order — and report what it teaches as findings for the " +
|
||||
"learner's global knowledge graph.";
|
||||
"learner's global knowledge graph, building on everything already extracted from this source.";
|
||||
|
||||
export const EXTRACTION_CORE = `THE GOAL: extract the source's REASONING CHAINS. The source teaches by making arguments — propositions
|
||||
worked out from other propositions. Your deliverable is twofold: the propositions themselves, and every
|
||||
argument the source makes. Each argument is a chain: premises that JOINTLY entail a conclusion.
|
||||
Entailment is the only link that earns a chain — where the real connection is historical or narrative
|
||||
(who argued what, how a debate or exposition unfolds), there is no chain; keep the pieces as bare
|
||||
propositions or drop them.
|
||||
export const EXTRACTION_CORE = `THE GOAL: uncover the source's CORE IDEAS and show how everything derives from the foundations. The
|
||||
source teaches by arguing — ideas worked out from other ideas, down to a foundation the rest stands on.
|
||||
Your deliverable is twofold: the core ideas themselves, and the chains of reasoning that derive them.
|
||||
Each chain is premises that JOINTLY entail a conclusion. Entailment is the only link that earns a
|
||||
chain — where the real connection is historical or narrative (who argued what, how a debate or
|
||||
exposition unfolds), there is no chain; keep the pieces as bare ideas or drop them. The direction of
|
||||
the graph is upward: foundational ideas at the bottom, derived ideas above them, each chain showing
|
||||
how its conclusion follows from what is more established — do not submit connections that carry no
|
||||
derivation.
|
||||
|
||||
The graph holds TRUE propositions about the world, nothing else — never anything about the learner, the
|
||||
The graph holds TRUE ideas about the world, nothing else — never anything about the learner, the
|
||||
teacher, or this reading: not "the learner understands X" but X. A misconception enters as its TRUE
|
||||
correction. Each statement is one self-contained reasoning step at full precision.
|
||||
correction. Each statement is one self-contained idea at full precision.
|
||||
|
||||
Extract the load-bearing ideas of the subject — what a reference on it would teach — not the full run of
|
||||
the discourse. The graph stores propositions as STEPS OF REASONING: a proposition earns its place only
|
||||
when the source's arguments run through it — examples, notation glosses, and restatements that no
|
||||
argument needs are not steps. Every proposition must stand alone for a reader who never saw this source:
|
||||
names and statements may not lean on the source's own scaffolding (claim numbering, axiom tags, arc
|
||||
labels); a claim that only matters as part of how this document is organized is not knowledge — drop it.
|
||||
Extract the load-bearing ideas of the subject — what a reference on it would teach — not the full run
|
||||
of the discourse. An idea earns its place only when the source's arguments run through it — examples,
|
||||
notation glosses, and restatements that no argument needs are not core ideas. Every idea must stand
|
||||
alone for a reader who never saw this source: names and statements may not lean on the source's own
|
||||
scaffolding (claim numbering, axiom tags, arc labels); a claim that only matters as part of how this
|
||||
document is organized is not knowledge — drop it.
|
||||
|
||||
SUBMIT with one submit_findings call: every proposition with a short name and a precise statement, and
|
||||
every chain as premises + conclusion. Premises within one chain are used TOGETHER — independent arguments
|
||||
for the same conclusion are separate chains. A conclusion may reappear, by id, as a premise of later
|
||||
chains. Include every field even when empty — "chains": [] when the source makes no arguments. The
|
||||
tool's field descriptions carry the details.`;
|
||||
When the prompt carries THE GRAPH SO FAR, you are continuing a source already partly extracted: build
|
||||
on it. Reference an existing idea by its id ("n_…") instead of restating it, and prefer chains that
|
||||
lean on established conclusions as premises — derive new material from what is already derived. Do
|
||||
not re-derive a conclusion the graph already holds unless the route is genuinely different.
|
||||
|
||||
SUBMIT with one submit_findings call: every NEW core idea with a short name and a precise statement,
|
||||
and every chain as premises + conclusion. A chain's premises and conclusion may be ids from this
|
||||
submission OR existing ids from THE GRAPH SO FAR, verbatim. Premises within one chain are used
|
||||
TOGETHER — independent arguments for the same conclusion are separate chains. A conclusion may
|
||||
reappear, by id, as a premise of later chains. A bare idea that no argument uses rides the
|
||||
propositions list alone. Include every field even when empty — "chains": [] when the source makes no
|
||||
arguments. The tool's field descriptions carry the details.`;
|
||||
|
||||
export const EXTRACTOR_INSTRUCTIONS = `${SESSION_PREAMBLE}\n\n${EXTRACTION_CORE}`;
|
||||
export const NOTES_EXTRACTOR_INSTRUCTIONS = `${NOTES_PREAMBLE}\n\n${EXTRACTION_CORE}`;
|
||||
|
||||
// ── the same-source digest (owner, 2026-09-30 evening) ──
|
||||
//
|
||||
// Serialized digest-aware extraction REPLACED the spine stage (owner
|
||||
// ruling after the live test: the post-hoc assembly pass added
|
||||
// complexity without perceptible gain — the extractor itself is the
|
||||
// assembler when it can see what its source has already yielded). Pure
|
||||
// string fold over the derived graph; "" when the source owns nothing
|
||||
// yet (the first stretch runs bare). Ids ride VERBATIM — that is what
|
||||
// the extractor references in its chains (the merger's mixed-namespace
|
||||
// resolution passes them through).
|
||||
export function sourceDigest(graph: DerivedGraph, sourceId: string): string {
|
||||
const nameOf = new Map(graph.nodes.map((n) => [n.id, n.name]));
|
||||
const ours = graph.nodes.filter((n) => n.provenance.some((pt) => pt.chatId === sourceId));
|
||||
const ourChains = graph.chains.filter((c) => c.provenance.some((pt) => pt.chatId === sourceId));
|
||||
if (ours.length === 0 && ourChains.length === 0) return "";
|
||||
const lines: string[] = [];
|
||||
lines.push("THE GRAPH SO FAR — everything already extracted from this source (reference these ids verbatim):");
|
||||
for (const n of ours) lines.push(` ${n.id} | ${n.name} | ${n.statement.replace(/\s+/g, " ")}`);
|
||||
if (ourChains.length > 0) {
|
||||
lines.push("DERIVATIONS ALREADY RECORDED (build on these; do not re-derive them):");
|
||||
for (const c of ourChains) {
|
||||
const names = c.premises.map((p) => nameOf.get(p) ?? p).join(" ∧ ");
|
||||
lines.push(` [${c.premises.join(" + ")} ⇒ ${c.conclusion}] ${names} ⇒ ${nameOf.get(c.conclusion) ?? c.conclusion}`);
|
||||
}
|
||||
}
|
||||
return lines.join("\n");
|
||||
}
|
||||
|
||||
/** The extractor over ONE session transcript (the chat front door). */
|
||||
function runExtractor(
|
||||
engine: PiEngine | undefined,
|
||||
prompt: string,
|
||||
existingIds: ReadonlySet<string>,
|
||||
signal: AbortSignal | undefined,
|
||||
onStep: (usage: Usage, model: string, step: number, finishReason: string, tools: string) => void,
|
||||
): Promise<SessionFindings> {
|
||||
// Chains in (the wire), follows-edges out (the pipeline) — the
|
||||
// translation seam of the 2026-09-30 redesign. Validation runs at the
|
||||
// submit seam (error-back); expansion runs once, after a clean submit.
|
||||
// Chains in (the wire), chains out (the store) — no translation seam
|
||||
// since chains-primary. Validation runs at the submit seam (error-back)
|
||||
// over the MIXED namespace: local ids + existing graph ids.
|
||||
return runSubmitAgent<FindingsSubmit>({
|
||||
role: "extractor",
|
||||
toolName: "submit_findings",
|
||||
toolDescription: "Submit the session findings.",
|
||||
parameters: findingsSchema,
|
||||
fill: fillFindings,
|
||||
validate: validateFindingsChains,
|
||||
validate: (value) => validateFindingsChains(value, existingIds),
|
||||
systemPrompt: EXTRACTOR_INSTRUCTIONS,
|
||||
prompt,
|
||||
engine,
|
||||
@@ -345,6 +401,7 @@ function runExtractor(
|
||||
function runNotesExtractor(
|
||||
engine: PiEngine | undefined,
|
||||
prompt: string,
|
||||
existingIds: ReadonlySet<string>,
|
||||
signal: AbortSignal | undefined,
|
||||
onStep: (usage: Usage, model: string, step: number, finishReason: string, tools: string) => void,
|
||||
): Promise<NotesFindings> {
|
||||
@@ -354,7 +411,7 @@ function runNotesExtractor(
|
||||
toolDescription: "Submit the notes findings.",
|
||||
parameters: findingsSchema,
|
||||
fill: fillFindings,
|
||||
validate: validateFindingsChains,
|
||||
validate: (value) => validateFindingsChains(value, existingIds),
|
||||
systemPrompt: NOTES_EXTRACTOR_INSTRUCTIONS,
|
||||
prompt,
|
||||
engine,
|
||||
@@ -446,7 +503,7 @@ function runAuditor(
|
||||
* audit+merge critical section is queued behind another job's; the
|
||||
* extractor is done but the auditor may not start until the graph is
|
||||
* stable. */
|
||||
export type ExtractionPhase = "queued" | "extracting" | "waiting" | "assembling" | "auditing" | "merging";
|
||||
export type ExtractionPhase = "queued" | "extracting" | "waiting" | "auditing" | "merging";
|
||||
|
||||
/** One in-flight pipeline run. Module-level: one process, one registry.
|
||||
* Exported for the status payload's consumer typing (the dock). */
|
||||
@@ -473,7 +530,6 @@ export interface RunningExtraction {
|
||||
* context-window blobs read this through /api/graph/status). */
|
||||
usage: {
|
||||
extractor?: ExtractionStageUsage;
|
||||
spine?: ExtractionStageUsage;
|
||||
auditor?: ExtractionStageUsage;
|
||||
};
|
||||
/** True-cancellation handle (the dock's abort button): one controller per
|
||||
@@ -543,7 +599,6 @@ export interface ExtractionModels {
|
||||
/** Test seam: a scripted engine per stage — checked FIRST, ahead of the
|
||||
* engineFor chain (production wires no direct values). */
|
||||
extractor?: PiEngine;
|
||||
spine?: PiEngine;
|
||||
auditor?: PiEngine;
|
||||
/** Settings-resolved engine per role — the same slot treatment the
|
||||
* job runner got, read FRESH per stage call (a settings PUT between
|
||||
@@ -622,7 +677,6 @@ export function createExtractionPipeline(deps: ExtractionPipelineDeps): Extracti
|
||||
function refuseIfStagesUnready(): void {
|
||||
const stages: readonly [ModelRole, PiEngine | undefined][] = [
|
||||
["extractor", stageEngine("extractor", deps.extractor)],
|
||||
["spine", stageEngine("spine", deps.spine)],
|
||||
["auditor", stageEngine("auditor", deps.auditor)],
|
||||
];
|
||||
const missing = new Set<ModelSlotName>();
|
||||
@@ -671,7 +725,7 @@ export function createExtractionPipeline(deps: ExtractionPipelineDeps): Extracti
|
||||
}
|
||||
|
||||
/** Accumulate one billed step into the run's registry entry (live). */
|
||||
function recordStep(jobKey: string, stage: "extractor" | "spine" | "auditor", usage: CallUsage, model: string): void {
|
||||
function recordStep(jobKey: string, stage: "extractor" | "auditor", usage: CallUsage, model: string): void {
|
||||
const run = runningExtractions.get(jobKey);
|
||||
if (!run) return; // not running (tests driving stages directly)
|
||||
run.usage[stage] ??= { model, steps: [] };
|
||||
@@ -718,15 +772,28 @@ export function createExtractionPipeline(deps: ExtractionPipelineDeps): Extracti
|
||||
}
|
||||
}
|
||||
|
||||
async function extract(jobKey: string, view: string): Promise<SessionFindings> {
|
||||
async function extract(jobKey: string, sourceId: string, view: string): Promise<SessionFindings> {
|
||||
const signal = runningExtractions.get(jobKey)?.controller.signal;
|
||||
signal?.throwIfAborted(); // an abort during a sync gap ends the job here
|
||||
console.info(`[extraction ${jobKey}] extractor starting (${view.split("\n").length} markdown lines)`);
|
||||
// THE DIGEST (owner, 2026-09-30 evening): the extractor continues a
|
||||
// partly-extracted source — it sees everything this source has
|
||||
// already yielded (propositions + chains, ids verbatim) and builds
|
||||
// on it. Serialized batches make this read the post-merge graph of
|
||||
// every earlier batch (cross-source runs stay parallel — the
|
||||
// digest is source-scoped, so only SAME-source order matters).
|
||||
const graph = getGraph(graphStore);
|
||||
const digest = sourceDigest(graph, sourceId);
|
||||
const prompt = digest === "" ? view : `${digest}\n\n${view}`;
|
||||
console.info(
|
||||
`[extraction ${jobKey}] extractor starting (${view.split("\n").length} markdown lines` +
|
||||
`${digest === "" ? "" : `, digest ${digest.split("\n").length} lines`})`,
|
||||
);
|
||||
const t0 = performance.now();
|
||||
|
||||
const findings = await runExtractor(
|
||||
stageEngine("extractor", deps.extractor),
|
||||
view,
|
||||
prompt,
|
||||
new Set(graph.nodes.map((n) => n.id)),
|
||||
signal,
|
||||
(usage, model, step, finishReason, tools) => {
|
||||
recordStep(jobKey, "extractor", callUsage(usage), model);
|
||||
@@ -737,50 +804,6 @@ export function createExtractionPipeline(deps: ExtractionPipelineDeps): Extracti
|
||||
return findings;
|
||||
}
|
||||
|
||||
/** THE SPINE (owner, 2026-09-30): the assembly stage between extract
|
||||
* and audit, INSIDE the serialized section (it reads the graph-so-far
|
||||
* digest; its output rides the same audit + merge). Skipped without a
|
||||
* call when the source owns nothing yet (first stretch — nothing to
|
||||
* join). Identity stays the auditor's: the spine submits chains +
|
||||
* connective propositions only, and the audit sees the COMBINED
|
||||
* propositions — chunk findings plus connectives — never chains. */
|
||||
async function assemble(
|
||||
jobKey: string,
|
||||
sourceId: string,
|
||||
kind: "chat" | "notes",
|
||||
findings: SessionFindings,
|
||||
): Promise<SessionFindings> {
|
||||
const signal = runningExtractions.get(jobKey)?.controller.signal;
|
||||
signal?.throwIfAborted();
|
||||
const graph = getGraph(graphStore);
|
||||
if (!spineHasDigest(graph, sourceId)) {
|
||||
console.info(`[extraction ${jobKey}] spine skipped — empty digest (first stretch)`);
|
||||
return findings;
|
||||
}
|
||||
console.info(
|
||||
`[extraction ${jobKey}] spine starting (${graph.nodes.length} graph nodes, ` +
|
||||
`${findings.propositions.length} fresh propositions)`,
|
||||
);
|
||||
const t0 = performance.now();
|
||||
const spine: SpineSubmit = await runSpine(
|
||||
stageEngine("spine", deps.spine),
|
||||
kind === "chat" ? SPINE_INSTRUCTIONS : NOTES_SPINE_INSTRUCTIONS,
|
||||
buildSpinePrompt(graph, sourceId, findings),
|
||||
new Set(findings.propositions.map((p) => p.id)),
|
||||
new Set(graph.nodes.map((n) => n.id)),
|
||||
signal,
|
||||
(usage, model, step, finishReason, tools) => {
|
||||
recordStep(jobKey, "spine", callUsage(usage), model);
|
||||
console.info(`[extraction] spine step ${step}: ${finishReason}, tools: ${tools}`);
|
||||
},
|
||||
);
|
||||
console.info(
|
||||
`[extraction ${jobKey}] spine done in ${Math.round(performance.now() - t0)}ms — ` +
|
||||
`${spine.chains.length} chain(s), ${spine.propositions.length} connective(s)`,
|
||||
);
|
||||
return spine === SPINE_EMPTY ? findings : combineFindings(findings, spine);
|
||||
}
|
||||
|
||||
async function audit(jobKey: string, findings: SessionFindings): Promise<AuditVerdicts> {
|
||||
const signal = runningExtractions.get(jobKey)?.controller.signal;
|
||||
signal?.throwIfAborted(); // an abort during the extract→audit gap ends here
|
||||
@@ -849,7 +872,6 @@ export function createExtractionPipeline(deps: ExtractionPipelineDeps): Extracti
|
||||
stats,
|
||||
stages: {
|
||||
extractor: run.usage.extractor ?? { model: "unknown", steps: [] },
|
||||
spine: run.usage.spine ?? { model: "unknown", steps: [] },
|
||||
auditor: run.usage.auditor ?? { model: "unknown", steps: [] },
|
||||
},
|
||||
};
|
||||
@@ -894,6 +916,10 @@ export function createExtractionPipeline(deps: ExtractionPipelineDeps): Extracti
|
||||
if (result.status === "fulfilled") {
|
||||
stats.nodes += result.value.nodes;
|
||||
stats.chains! += result.value.chains ?? 0;
|
||||
// Simplification maintenance, summed like appends — but ABSENT
|
||||
// when nothing dropped (deep-equal pins read cleaner).
|
||||
if (result.value.droppedChains)
|
||||
stats.droppedChains = (stats.droppedChains ?? 0) + result.value.droppedChains;
|
||||
continue;
|
||||
}
|
||||
const err = result.reason;
|
||||
@@ -945,71 +971,82 @@ export function createExtractionPipeline(deps: ExtractionPipelineDeps): Extracti
|
||||
`[extraction ${chatId}] pipeline start — ${covered.uncoveredTokens} uncovered tok, ` +
|
||||
`${covered.uncoveredCount} messages, ${batches.length} batch(es) of ≤${chunkTokens} tok`,
|
||||
);
|
||||
// PER-CHUNK JOBS (owner, 2026-09-28): every batch is its own job —
|
||||
// extracts fan out behind the global 3-slot semaphore ("queued"),
|
||||
// audits serialize through the process-wide gate ("waiting"). A
|
||||
// batch that fails is ISOLATED (its messages stay uncovered —
|
||||
// honestly re-runnable); its siblings keep merging.
|
||||
const settled = await Promise.allSettled(
|
||||
batches.map((batch, i) => {
|
||||
const part = batches.length > 1 ? { i: i + 1, n: batches.length } : undefined;
|
||||
return runJob({ id: chatId, kind: "chat", part }, async (jobKey) => {
|
||||
// The batch's source view: TEACHER PROSE ONLY (owner,
|
||||
// 2026-09-28) — plain paragraphs, same selection as the
|
||||
// coverage fingerprints (messageViewLines parity).
|
||||
const view = proseView(
|
||||
batch.map((slot) => slot.message),
|
||||
record.title,
|
||||
);
|
||||
setPhase(jobKey, "queued", "chat", undefined, part);
|
||||
const findings = await withExtractSlot(jobKey, async () => {
|
||||
setPhase(jobKey, "extracting", "chat", undefined, part);
|
||||
return extract(jobKey, view);
|
||||
});
|
||||
// SERIALIZED AUDIT (owner 2026-09-28): audit + merge are the
|
||||
// graph-mutating critical section — one job at a time,
|
||||
// process-wide, so the auditor explores the graph as it
|
||||
// stands AFTER every earlier job's merge. The spine rides the
|
||||
// same section (owner order, 2026-09-30): extract → spine →
|
||||
// audit → merge.
|
||||
return withAuditLock(jobKey, "chat", undefined, async () => {
|
||||
setPhase(jobKey, "assembling", "chat", undefined, part);
|
||||
const combined = await assemble(jobKey, chatId, "chat", findings);
|
||||
setPhase(jobKey, "auditing", "chat", undefined, part);
|
||||
const auditResult = await audit(jobKey, combined);
|
||||
setPhase(jobKey, "merging", "chat", undefined, part);
|
||||
// SESSION-DELETION GUARD (wave 6): a chat deleted mid-run must never
|
||||
// merge. The deletion route aborts this run first and runs its purge
|
||||
// as one sync block; this closes the last window — a run that slipped
|
||||
// past the wind-down finds the chat gone HERE and rides the existing
|
||||
// abort path (info, not error): nothing merged, nothing checkpointed.
|
||||
if (!chatStore.getChat(chatId)) {
|
||||
const gone = new Error(`chat deleted mid-run: ${chatId}`);
|
||||
gone.name = "AbortError";
|
||||
throw gone;
|
||||
}
|
||||
// The batch's messages, snapshotted when the run computed
|
||||
// coverage — if a message mutates mid-run (a quiz answer
|
||||
// merges in), the stored fingerprint won't match its new
|
||||
// one and it re-opens: honest by construction.
|
||||
const foldedAt = new Date().toISOString();
|
||||
const batchStats = mergeFindings(chatId, combined, auditResult, { chatStore, graphStore }, {
|
||||
foldedAt,
|
||||
coverage: batch
|
||||
.filter((slot) => messageViewLines(slot.message, slot.message.id).length > 0)
|
||||
.map((slot) => ({ chatId, fp: slot.fp, tokens: slot.tokens, foldedAt })),
|
||||
});
|
||||
// The layout ledger evolves with the merge (sync — the
|
||||
// next snapshot fetch must already reflect the new nodes).
|
||||
settleGraphLayout(graphStore);
|
||||
console.info(
|
||||
`[extraction ${jobKey}] merged: ${batchStats.nodes} nodes, ${batchStats.chains ?? 0} chains`,
|
||||
// PER-CHUNK JOBS (owner, 2026-09-28), SERIALIZED PER SOURCE (owner,
|
||||
// 2026-09-30 evening): every batch is its own job — registry row,
|
||||
// because each batch's extractor continues the source's digest
|
||||
// (everything the earlier batches merged). Cross-source runs stay
|
||||
// parallel; the 3-slot extract semaphore still caps concurrent
|
||||
// extracts process-wide ("queued"); audits serialize through the
|
||||
// process-wide gate ("waiting"). A batch that fails is ISOLATED
|
||||
// (its messages stay uncovered — honestly re-runnable); its
|
||||
// siblings keep merging.
|
||||
const settled: PromiseSettledResult<EmissionStats>[] = [];
|
||||
for (const [batchIndex, batch] of batches.entries()) {
|
||||
const part = batches.length > 1 ? { i: batchIndex + 1, n: batches.length } : undefined;
|
||||
try {
|
||||
settled.push({
|
||||
status: "fulfilled",
|
||||
value: await runJob({ id: chatId, kind: "chat", part }, async (jobKey) => {
|
||||
// The batch's source view: TEACHER PROSE ONLY (owner,
|
||||
// 2026-09-28) — plain paragraphs, same selection as the
|
||||
// coverage fingerprints (messageViewLines parity).
|
||||
const view = proseView(
|
||||
batch.map((slot) => slot.message),
|
||||
record.title,
|
||||
);
|
||||
return batchStats;
|
||||
});
|
||||
setPhase(jobKey, "queued", "chat", undefined, part);
|
||||
const findings = await withExtractSlot(jobKey, async () => {
|
||||
setPhase(jobKey, "extracting", "chat", undefined, part);
|
||||
return extract(jobKey, chatId, view);
|
||||
});
|
||||
// SERIALIZED AUDIT (owner, 2026-09-28): audit + merge are the
|
||||
// graph-mutating critical section — one job at a time,
|
||||
// process-wide, so the auditor explores the graph as it
|
||||
// stands AFTER every earlier job's merge.
|
||||
return withAuditLock(jobKey, "chat", undefined, async () => {
|
||||
setPhase(jobKey, "auditing", "chat", undefined, part);
|
||||
const auditResult = await audit(jobKey, findings);
|
||||
setPhase(jobKey, "merging", "chat", undefined, part);
|
||||
// SESSION-DELETION GUARD (wave 6): a chat deleted mid-run must never
|
||||
// merge. The deletion route aborts this run first and runs its purge
|
||||
// as one sync block; this closes the last window — a run that slipped
|
||||
// past the wind-down finds the chat gone HERE and rides the existing
|
||||
// abort path (info, not error): nothing merged, nothing checkpointed.
|
||||
if (!chatStore.getChat(chatId)) {
|
||||
const gone = new Error(`chat deleted mid-run: ${chatId}`);
|
||||
gone.name = "AbortError";
|
||||
throw gone;
|
||||
}
|
||||
// The batch's messages, snapshotted when the run computed
|
||||
// coverage — if a message mutates mid-run (a quiz answer
|
||||
// merges in), the stored fingerprint won't match its new
|
||||
// one and it re-opens: honest by construction.
|
||||
const foldedAt = new Date().toISOString();
|
||||
const batchStats = mergeFindings(chatId, findings, auditResult, { chatStore, graphStore }, {
|
||||
foldedAt,
|
||||
coverage: batch
|
||||
.filter((slot) => messageViewLines(slot.message, slot.message.id).length > 0)
|
||||
.map((slot) => ({ chatId, fp: slot.fp, tokens: slot.tokens, foldedAt })),
|
||||
});
|
||||
// The layout ledger evolves with the merge (sync — the
|
||||
// next snapshot fetch must already reflect the new nodes).
|
||||
settleGraphLayout(graphStore);
|
||||
console.info(
|
||||
`[extraction ${jobKey}] merged: ${batchStats.nodes} nodes, ${batchStats.chains ?? 0} chains`,
|
||||
);
|
||||
return batchStats;
|
||||
});
|
||||
}),
|
||||
});
|
||||
}),
|
||||
);
|
||||
} catch (err) {
|
||||
settled.push({ status: "rejected", reason: err });
|
||||
// A source-wide abort (user abort / session deletion)
|
||||
// ends the WHOLE run: later batches would mint fresh,
|
||||
// un-aborted jobs and hang — stop here. Isolated
|
||||
// FAILURES keep going (siblings still merge).
|
||||
if (err instanceof Error && err.name === "AbortError") break;
|
||||
}
|
||||
}
|
||||
const stats = aggregateJobs(chatId, settled);
|
||||
console.info(
|
||||
`[extraction ${chatId}] merged: ${stats.nodes} nodes, ${stats.chains ?? 0} chains ` +
|
||||
@@ -1070,11 +1107,20 @@ export function createExtractionPipeline(deps: ExtractionPipelineDeps): Extracti
|
||||
const signal = runningExtractions.get(jobKey)?.controller.signal;
|
||||
signal?.throwIfAborted(); // an abort during a sync gap ends the job here
|
||||
const view = notesView(input);
|
||||
console.info(`[extraction ${jobKey}] notes extractor starting (${view.split("\n").length} source lines)`);
|
||||
// Same digest law as the chat door (owner, 2026-09-30 evening): the
|
||||
// chunk extractor continues its own source, ids verbatim.
|
||||
const graph = getGraph(graphStore);
|
||||
const digest = sourceDigest(graph, input.id);
|
||||
const prompt = digest === "" ? view : `${digest}\n\n${view}`;
|
||||
console.info(
|
||||
`[extraction ${jobKey}] notes extractor starting (${view.split("\n").length} source lines` +
|
||||
`${digest === "" ? "" : `, digest ${digest.split("\n").length} lines`})`,
|
||||
);
|
||||
const t0 = performance.now();
|
||||
const findings = await runNotesExtractor(
|
||||
stageEngine("extractor", deps.extractor),
|
||||
view,
|
||||
prompt,
|
||||
new Set(graph.nodes.map((n) => n.id)),
|
||||
signal,
|
||||
(usage, model, step, finishReason, tools) => {
|
||||
recordStep(jobKey, "extractor", callUsage(usage), model);
|
||||
@@ -1107,49 +1153,56 @@ export function createExtractionPipeline(deps: ExtractionPipelineDeps): Extracti
|
||||
console.info(
|
||||
`[extraction ${id}] notes pipeline start (${input.filePath}) — ${chunks.length} chunk(s) of ≤${chunkTokens} tok`,
|
||||
);
|
||||
// PER-CHUNK JOBS (owner, 2026-09-28): every chunk is its own job —
|
||||
// registry row, usage record, failure domain. Extracts fan out in
|
||||
// parallel behind the global 3-slot extract semaphore ("queued");
|
||||
// audits serialize through the process-wide gate exactly as before
|
||||
// ("waiting"). The part rides the registry entry (feed suffix
|
||||
// "· i/N") only when the file actually chunked.
|
||||
const settled = await Promise.allSettled(
|
||||
chunks.map((chunk, i) => {
|
||||
const part = chunks.length > 1 ? { i: i + 1, n: chunks.length } : undefined;
|
||||
return runJob({ id, kind: "notes", label: input.filePath, part }, async (jobKey) => {
|
||||
setPhase(jobKey, "queued", "notes", input.filePath, part);
|
||||
const findings = await withExtractSlot(jobKey, async () => {
|
||||
setPhase(jobKey, "extracting", "notes", input.filePath, part);
|
||||
return notesExtract(jobKey, { ...input, text: chunk, part });
|
||||
});
|
||||
// SERIALIZED AUDIT (owner, 2026-09-28): extracts run in
|
||||
// parallel, but audit + merge one at a time, process-wide —
|
||||
// the auditor explores the graph as it stands AFTER every
|
||||
// earlier job's merge. Reacquired per chunk so runs
|
||||
// interleave fairly. The spine rides the same section
|
||||
// (owner order, 2026-09-30): extract → spine → audit → merge.
|
||||
return withAuditLock(jobKey, "notes", input.filePath, async () => {
|
||||
setPhase(jobKey, "assembling", "notes", input.filePath, part);
|
||||
const combined = await assemble(jobKey, id, "notes", findings);
|
||||
setPhase(jobKey, "auditing", "notes", input.filePath, part);
|
||||
const auditResult = await audit(jobKey, combined);
|
||||
setPhase(jobKey, "merging", "notes", input.filePath, part);
|
||||
const merged = mergeFindings(
|
||||
id,
|
||||
combined,
|
||||
auditResult,
|
||||
{ chatStore, graphStore },
|
||||
{ foldedAt: input.importedAt },
|
||||
);
|
||||
// The layout ledger evolves with each merge — the next
|
||||
// snapshot fetch reflects every finished chunk.
|
||||
settleGraphLayout(graphStore);
|
||||
console.info(`[extraction ${jobKey}] merged: ${merged.nodes} nodes, ${merged.chains ?? 0} chains`);
|
||||
return merged;
|
||||
});
|
||||
// PER-CHUNK JOBS (owner, 2026-09-28), SERIALIZED PER SOURCE (owner,
|
||||
// 2026-09-30 evening): same law as the chat door — chunks run
|
||||
// STRICTLY IN ORDER, each chunk's extractor continuing the
|
||||
// source's digest (everything earlier chunks merged). Cross-source
|
||||
// runs stay parallel; the semaphore caps concurrent extracts
|
||||
// process-wide; audits serialize through the gate as before. The
|
||||
// part rides the registry entry (feed suffix "· i/N") only when
|
||||
// the file actually chunked.
|
||||
const settled: PromiseSettledResult<EmissionStats>[] = [];
|
||||
for (const [chunkIndex, chunk] of chunks.entries()) {
|
||||
const part = chunks.length > 1 ? { i: chunkIndex + 1, n: chunks.length } : undefined;
|
||||
try {
|
||||
settled.push({
|
||||
status: "fulfilled",
|
||||
value: await runJob({ id, kind: "notes", label: input.filePath, part }, async (jobKey) => {
|
||||
setPhase(jobKey, "queued", "notes", input.filePath, part);
|
||||
const findings = await withExtractSlot(jobKey, async () => {
|
||||
setPhase(jobKey, "extracting", "notes", input.filePath, part);
|
||||
return notesExtract(jobKey, { ...input, text: chunk, part });
|
||||
});
|
||||
// SERIALIZED AUDIT (owner, 2026-09-28): audit + merge one
|
||||
// at a time, process-wide — the auditor explores the graph
|
||||
// as it stands AFTER every earlier job's merge. Reacquired
|
||||
// per chunk so cross-source runs interleave fairly.
|
||||
return withAuditLock(jobKey, "notes", input.filePath, async () => {
|
||||
setPhase(jobKey, "auditing", "notes", input.filePath, part);
|
||||
const auditResult = await audit(jobKey, findings);
|
||||
setPhase(jobKey, "merging", "notes", input.filePath, part);
|
||||
const merged = mergeFindings(
|
||||
id,
|
||||
findings,
|
||||
auditResult,
|
||||
{ chatStore, graphStore },
|
||||
{ foldedAt: input.importedAt },
|
||||
);
|
||||
// The layout ledger evolves with each merge — the next
|
||||
// snapshot fetch reflects every finished chunk.
|
||||
settleGraphLayout(graphStore);
|
||||
console.info(`[extraction ${jobKey}] merged: ${merged.nodes} nodes, ${merged.chains ?? 0} chains`);
|
||||
return merged;
|
||||
});
|
||||
}),
|
||||
});
|
||||
}),
|
||||
);
|
||||
} catch (err) {
|
||||
settled.push({ status: "rejected", reason: err });
|
||||
// Same source-wide-abort law as the chat door: stop the
|
||||
// run; isolated failures keep going.
|
||||
if (err instanceof Error && err.name === "AbortError") break;
|
||||
}
|
||||
}
|
||||
const stats = aggregateJobs(id, settled);
|
||||
console.info(
|
||||
`[extraction ${id}] merged: ${stats.nodes} nodes, ${stats.chains ?? 0} chains ` +
|
||||
|
||||
@@ -1,261 +0,0 @@
|
||||
/**
|
||||
* The spine stage (owner-approved design, 2026-09-30): the extraction
|
||||
* pipeline's ASSEMBLY pass. Extraction walks a source in stretches, each
|
||||
* stretch-blind — but the source's reasoning runs over the whole span,
|
||||
* and the data showed it: 0 of 133 chains crossed a stretch boundary.
|
||||
* The spine runs once per batch, INSIDE the serialized audit section,
|
||||
* between the extract and the audit (owner-specified order):
|
||||
*
|
||||
* extract → [lock: spine → audit → merge] → next batch
|
||||
*
|
||||
* It sees the same-source digest (everything merged so far) plus the
|
||||
* fresh stretch's findings, and mints ONLY what spans them: cross-stretch
|
||||
* chains and the connective propositions a chain needs that no stretch
|
||||
* stated on its own. The SPAN LAW is mechanical (validateSpine below):
|
||||
* every chain must touch ≥1 fresh endpoint AND ≥1 existing endpoint —
|
||||
* induction over the walk means no pair ever needs a second chance.
|
||||
*
|
||||
* Identity stays the AUDITOR's lane (owner ruling): the spine submits no
|
||||
* links. Its connective propositions ride into the audit as ordinary
|
||||
* findings — the auditor links them like any other — and its chains
|
||||
* resolve through the merger's job-local map (endpoints may be existing
|
||||
* graph ids verbatim; see mergeFindings' mixed-namespace resolution).
|
||||
*
|
||||
* Session and notes front doors share this stage; only the preamble
|
||||
* differs (the extractor's law, verbatim).
|
||||
*/
|
||||
|
||||
import type { Usage } from "@earendil-works/pi-ai";
|
||||
import { Type, type Static } from "typebox";
|
||||
import { runSubmitAgent, healStringifiedArrays } from "./submit-agent.ts";
|
||||
import type { PiEngine } from "../models.ts";
|
||||
import type { ExtractedChain, SessionFindings } from "../../core/contracts.ts";
|
||||
import type { DerivedGraph } from "../../core/graph.ts";
|
||||
|
||||
// ─── the prompts ─────────────────────────────────────────────────────────────
|
||||
|
||||
export const SESSION_SPINE_PREAMBLE =
|
||||
"You are the spine stage of a learning application's extraction pipeline. One source — a teaching " +
|
||||
"session, walked in stretches of the teacher's prose in reading order — has just advanced by one " +
|
||||
"stretch; you join what the new stretch established into the source's accumulated line of argument.";
|
||||
|
||||
export const NOTES_SPINE_PREAMBLE =
|
||||
"You are the spine stage of a learning application's extraction pipeline. One source — the author's " +
|
||||
"personal notes, walked in stretches in reading order — has just advanced by one stretch; you join " +
|
||||
"what the new stretch established into the source's accumulated line of argument.";
|
||||
|
||||
export const SPINE_CORE = `THE GOAL: carry the source's line of argument ACROSS stretches. Extraction walks the source in
|
||||
stretches; each stretch reports its own propositions and chains — but the source's reasoning runs over
|
||||
the whole span, and no stretch sees more than its own part. You see everything extracted from this
|
||||
source so far, plus what the latest stretch just established. Your deliverable: the chains that JOIN them.
|
||||
|
||||
Every chain you submit must SPAN: its premises and conclusion may not all come from the same stretch. A
|
||||
chain inside one stretch is that stretch's own work; a chain the graph already holds exists. Direction is
|
||||
whatever the reasoning runs — earlier into later, or later material completing an argument opened earlier.
|
||||
|
||||
The extraction doctrines bind you unchanged: premises JOINTLY entail the conclusion; entailment is the
|
||||
only link that earns a chain; every proposition stands alone for a reader who never saw the source;
|
||||
load-bearing connections only — when in doubt, omit.
|
||||
|
||||
Restatements: when the fresh stretch restates a proposition the graph already holds, build your chains on
|
||||
the EXISTING id, not the restatement.
|
||||
|
||||
SUBMIT with one submit_spine call: the connecting chains, and only the connective propositions a chain
|
||||
needs that no stretch stated on its own — a missing middle step, the general claim a stretch argued only
|
||||
locally. Pass empty arrays when the stretch stands alone — never omit the fields. The tool's field
|
||||
descriptions carry the details.`;
|
||||
|
||||
export const SPINE_INSTRUCTIONS = `${SESSION_SPINE_PREAMBLE}\n\n${SPINE_CORE}`;
|
||||
export const NOTES_SPINE_INSTRUCTIONS = `${NOTES_SPINE_PREAMBLE}\n\n${SPINE_CORE}`;
|
||||
|
||||
// ─── the submit schema ───────────────────────────────────────────────────────
|
||||
|
||||
const spinePropositionProps = {
|
||||
id: Type.String({ description: 'Local id: "s1", "s2", … — referenced by chains.' }),
|
||||
name: Type.String({
|
||||
description:
|
||||
"A very short label — established terminology, the term the discipline itself uses. " +
|
||||
"Not a sentence, needn't be unique or self-contained: this becomes the node's global label.",
|
||||
}),
|
||||
statement: Type.String({
|
||||
description:
|
||||
"One true, self-contained, world-facing reasoning step at full precision — the connective " +
|
||||
"tissue a spanning chain needs that no stretch stated on its own.",
|
||||
}),
|
||||
aliases: Type.Optional(
|
||||
Type.Array(Type.String(), {
|
||||
description: "Verbatim phrasings the source used for the same proposition, if any.",
|
||||
}),
|
||||
),
|
||||
};
|
||||
|
||||
const endpoint = (role: string) =>
|
||||
Type.String({
|
||||
description:
|
||||
`The proposition id this ${role} — EITHER an existing proposition id from the digest, ` +
|
||||
'verbatim ("n_…"), OR the local id of a proposition in this submission ("s1", …) or of the ' +
|
||||
'fresh stretch\'s findings ("p1", …).',
|
||||
});
|
||||
|
||||
const spineChainSchema = Type.Object({
|
||||
premises: Type.Array(endpoint("premise uses"), {
|
||||
minItems: 1,
|
||||
description:
|
||||
"Proposition ids used TOGETHER: they jointly entail the conclusion; independent reasons are separate chains.",
|
||||
}),
|
||||
conclusion: endpoint("conclusion is"),
|
||||
});
|
||||
|
||||
const spineSchema = Type.Object({
|
||||
propositions: Type.Array(Type.Object(spinePropositionProps), {
|
||||
description:
|
||||
"Connective propositions ONLY — steps a spanning chain needs that no stretch stated on " +
|
||||
"its own. Usually empty.",
|
||||
}),
|
||||
chains: Type.Array(spineChainSchema, {
|
||||
description:
|
||||
"The cross-stretch chains — each must join material that arrived at different times " +
|
||||
"(≥1 endpoint from the fresh stretch, ≥1 from before it). Pass [] when the stretch " +
|
||||
"stands alone — never omit the field.",
|
||||
}),
|
||||
});
|
||||
|
||||
/** What the spine SUBMITS. fillSpine normalizes it; validateSpine checks
|
||||
* the span laws; the pipeline combines it with the stretch's findings
|
||||
* before the audit (the auditor sees chunk + connective propositions,
|
||||
* propositions only — never chains, owner 2026-09-30). */
|
||||
export type SpineSubmit = Static<typeof spineSchema>;
|
||||
|
||||
export const SPINE_EMPTY: SpineSubmit = { propositions: [], chains: [] };
|
||||
|
||||
export function fillSpine(raw: SpineSubmit): SpineSubmit {
|
||||
return {
|
||||
propositions: raw.propositions.map((p) => ({ ...p, aliases: p.aliases ?? [] })),
|
||||
chains: raw.chains,
|
||||
};
|
||||
}
|
||||
|
||||
/** The span laws — the LOUD channel (error-back retry), mirroring
|
||||
* validateFindingsChains. `chunkIds` = the fresh stretch's finding ids
|
||||
* ("p1"…); `existingIds` = every node id already in the graph. A chain
|
||||
* must touch at least one FRESH endpoint (chunk finding or this
|
||||
* submission's connectives) and at least one EXISTING endpoint — the
|
||||
* mechanical form of the owner's all-or-nothing span rule. Identical
|
||||
* re-assertions are not errors: the merger dedupes them silently. */
|
||||
export function validateSpine(value: SpineSubmit, chunkIds: ReadonlySet<string>, existingIds: ReadonlySet<string>): string | null {
|
||||
const localIds = new Set(value.propositions.map((p) => p.id));
|
||||
for (const p of value.propositions) {
|
||||
if (chunkIds.has(p.id) || existingIds.has(p.id)) {
|
||||
return `propositions: "${p.id}" collides with an id this submission does not own — connective ids are fresh ("s1", "s2", …).`;
|
||||
}
|
||||
}
|
||||
const resolves = (id: string) => chunkIds.has(id) || localIds.has(id) || existingIds.has(id);
|
||||
const isFresh = (id: string) => chunkIds.has(id) || localIds.has(id);
|
||||
for (const chain of value.chains) {
|
||||
if (chain.premises.length === 0) {
|
||||
return "chains: premises must name at least one proposition.";
|
||||
}
|
||||
const endpoints = [...chain.premises, chain.conclusion];
|
||||
const unknown = endpoints.find((id) => !resolves(id));
|
||||
if (unknown !== undefined) {
|
||||
return `chains: "${unknown}" is not an existing digest id, a fresh-stretch finding id, or one of this call's proposition ids.`;
|
||||
}
|
||||
if (chain.premises.includes(chain.conclusion)) {
|
||||
return "chains: a conclusion cannot be its own premise.";
|
||||
}
|
||||
if (!endpoints.some(isFresh)) {
|
||||
return `chains: [${endpoints.join(", ")}] is entirely existing material — a chain the graph already had its chance to hold. Every chain must join the fresh stretch to what came before.`;
|
||||
}
|
||||
if (!endpoints.some((id) => existingIds.has(id))) {
|
||||
return `chains: [${endpoints.join(", ")}] lives entirely inside the fresh stretch — that is the stretch extractor's work, not yours. Every chain must join the fresh stretch to what came before.`;
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
// ─── the digest ──────────────────────────────────────────────────────────────
|
||||
|
||||
/** Render the same-source digest + fresh findings for the spine's prompt.
|
||||
* Pure: a string fold over the derived graph. The digest carries ids
|
||||
* verbatim ("n_…") — that is what the spine references in its chains. */
|
||||
export function buildSpinePrompt(graph: DerivedGraph, sourceId: string, findings: SessionFindings): string {
|
||||
const nameOf = new Map(graph.nodes.map((n) => [n.id, n.name]));
|
||||
const ours = graph.nodes.filter((n) => n.provenance.some((pt) => pt.chatId === sourceId));
|
||||
const ourChains = graph.chains.filter((c) => c.provenance.some((pt) => pt.chatId === sourceId));
|
||||
const lines: string[] = [];
|
||||
lines.push(
|
||||
ours.length > 0
|
||||
? `THE GRAPH SO FAR — this source's accumulated propositions (ids verbatim, referenced by your chains):`
|
||||
: `THE GRAPH SO FAR — this source has no accumulated propositions yet (nothing to join).`,
|
||||
);
|
||||
for (const n of ours) lines.push(` ${n.id} | ${n.name} | ${n.statement.replace(/\s+/g, " ")}`);
|
||||
if (ourChains.length > 0) {
|
||||
lines.push(`ARGUMENTS ALREADY RECORDED (do not re-submit these):`);
|
||||
for (const c of ourChains) {
|
||||
const names = c.premises.map((p) => nameOf.get(p) ?? p).join(" ∧ ");
|
||||
lines.push(` [${c.premises.join(" + ")} ⟹ ${c.conclusion}] ${names} ⟹ ${nameOf.get(c.conclusion) ?? c.conclusion}`);
|
||||
}
|
||||
}
|
||||
lines.push(``);
|
||||
lines.push(`THE FRESH STRETCH — what the extractor just reported (local ids "p1", …):`);
|
||||
for (const p of findings.propositions) lines.push(` ${p.id} | ${p.name} | ${p.statement.replace(/\s+/g, " ")}`);
|
||||
if (findings.chains.length > 0) {
|
||||
lines.push(`ITS OWN CHAINS (this stretch's local work):`);
|
||||
const freshName = new Map(findings.propositions.map((p) => [p.id, p.name]));
|
||||
for (const c of findings.chains) {
|
||||
const names = c.premises.map((p) => freshName.get(p) ?? p).join(" ∧ ");
|
||||
lines.push(` [${c.premises.join(" + ")} ⟹ ${c.conclusion}] ${names} ⟹ ${freshName.get(c.conclusion) ?? c.conclusion}`);
|
||||
}
|
||||
}
|
||||
lines.push(``);
|
||||
lines.push(`Join the source's line of argument across these two moments.`);
|
||||
return lines.join("\n");
|
||||
}
|
||||
|
||||
/** Whether the spine has anything to see at all — the first stretch of a
|
||||
* source has an empty digest, and the call is skipped (no spine run, no
|
||||
* spend) until a source owns at least one merged proposition. */
|
||||
export function spineHasDigest(graph: DerivedGraph, sourceId: string): boolean {
|
||||
return graph.nodes.some((n) => n.provenance.some((pt) => pt.chatId === sourceId));
|
||||
}
|
||||
|
||||
// ─── the runner ──────────────────────────────────────────────────────────────
|
||||
|
||||
/** The spine over one batch's findings — a single submit call, retry-only
|
||||
* (no exploration tools; everything it needs is in the prompt). */
|
||||
export function runSpine(
|
||||
engine: PiEngine | undefined,
|
||||
systemPrompt: string,
|
||||
prompt: string,
|
||||
chunkIds: ReadonlySet<string>,
|
||||
existingIds: ReadonlySet<string>,
|
||||
signal: AbortSignal | undefined,
|
||||
onStep: (usage: Usage, model: string, step: number, finishReason: string, tools: string) => void,
|
||||
): Promise<SpineSubmit> {
|
||||
return runSubmitAgent<SpineSubmit>({
|
||||
role: "spine",
|
||||
toolName: "submit_spine",
|
||||
toolDescription: "Submit the cross-stretch chains (and any connective propositions they need).",
|
||||
parameters: spineSchema,
|
||||
fill: fillSpine,
|
||||
validate: (value) => validateSpine(value, chunkIds, existingIds),
|
||||
systemPrompt,
|
||||
prompt,
|
||||
engine,
|
||||
signal,
|
||||
onStep,
|
||||
neverSubmitted: "the spine never submitted its chains",
|
||||
prepareArguments: healStringifiedArrays,
|
||||
});
|
||||
}
|
||||
|
||||
/** The combined findings the AUDIT + MERGE see: the stretch's propositions
|
||||
* plus the spine's connectives (ONE findings set — the auditor links
|
||||
* connectives like anything else), and both chain sets (endpoints may be
|
||||
* existing graph ids — the merger resolves the mixed namespace). */
|
||||
export function combineFindings(findings: SessionFindings, spine: SpineSubmit): SessionFindings {
|
||||
return {
|
||||
propositions: [...findings.propositions, ...spine.propositions.map((p) => ({ ...p, aliases: p.aliases ?? [] }))],
|
||||
chains: [...findings.chains, ...(spine.chains as ExtractedChain[])],
|
||||
};
|
||||
}
|
||||
@@ -16,8 +16,8 @@ import type { PiEngine } from "../models.ts";
|
||||
import { runSubagent } from "./pi-run.ts";
|
||||
|
||||
// ── the weak-model string-heal (owner, 2026-09-28; moved here from
|
||||
// extraction.ts with the spine stage, 2026-09-30 — the heal is wire-format
|
||||
// medicine shared by every submit agent) ──
|
||||
// extraction.ts with the digest-aware redesign, 2026-09-30 — the heal
|
||||
// is wire-format medicine shared by every submit agent) ──
|
||||
//
|
||||
// OpenRouter log evidence: weak models under a big submit payload often
|
||||
// emit a whole array as ONE JSON-STRING instead of a native array
|
||||
|
||||
@@ -92,11 +92,6 @@ export interface ExtractionRunUsage {
|
||||
stages: {
|
||||
extractor: ExtractionStageUsage;
|
||||
auditor: ExtractionStageUsage;
|
||||
/** The spine stage (2026-09-30): cross-stretch chain assembly.
|
||||
* OPTIONAL — records written before the stage existed (and runs
|
||||
* whose digest was empty, so the call was skipped) carry no
|
||||
* entry; readers default it to an empty step-set. */
|
||||
spine?: ExtractionStageUsage;
|
||||
};
|
||||
}
|
||||
|
||||
|
||||
@@ -63,6 +63,12 @@ export interface EmissionStats {
|
||||
nodes: number;
|
||||
/** New reasoning chains appended. */
|
||||
chains?: number;
|
||||
/** Chains the simplification pass removed in this merge (owner,
|
||||
* 2026-09-30 evening — ancestry-cover / prefer-fewest-premises).
|
||||
* Maintenance, not extraction: appends are counted in `chains`
|
||||
* regardless. Undefined when nothing dropped (pre-simplification
|
||||
* records read as zero). */
|
||||
droppedChains?: number;
|
||||
}
|
||||
|
||||
/** One provenance point — message-granular (DECIDED 2026-09-02): nodes and
|
||||
|
||||
@@ -61,7 +61,6 @@ export interface GraphCostSummary {
|
||||
* historical spend on either is simply not displayed.) */
|
||||
stages: {
|
||||
extraction: StageCost;
|
||||
spine: StageCost;
|
||||
audit: StageCost;
|
||||
};
|
||||
/** Newest first, capped at `recentCap` (default 10). */
|
||||
@@ -113,16 +112,11 @@ function mergeTokens(into: CallUsage, steps: readonly CallUsage[]): void {
|
||||
* same output — no clock, no fs, no store reads. */
|
||||
export function graphCostSummary(input: CostSummaryInput): GraphCostSummary {
|
||||
const extraction = emptyStage();
|
||||
const spine = emptyStage();
|
||||
const audit = emptyStage();
|
||||
const runs: RecentRun[] = [];
|
||||
|
||||
const foldExtractionRun = (run: ExtractionRunUsage, title: string) => {
|
||||
const stages = run.stages ?? { extractor: { model: "unknown", steps: [] }, auditor: { model: "unknown", steps: [] } };
|
||||
// The spine stage is OPTIONAL on the record (2026-09-30): records
|
||||
// written before it existed (and runs whose digest was empty, so
|
||||
// the call was skipped) carry no entry — default to empty steps.
|
||||
const spineStage = stages.spine ?? { model: "unknown", steps: [] };
|
||||
// Per-stage: tokens always count; dollars only when priced.
|
||||
extraction.runs++;
|
||||
mergeTokens(extraction.tokens, stages.extractor.steps);
|
||||
@@ -130,14 +124,6 @@ export function graphCostSummary(input: CostSummaryInput): GraphCostSummary {
|
||||
if (extDollars === null) extraction.unpricedRuns++;
|
||||
else extraction.dollars += extDollars;
|
||||
|
||||
if (spineStage.steps.length > 0) {
|
||||
spine.runs++;
|
||||
mergeTokens(spine.tokens, spineStage.steps);
|
||||
const spDollars = stageDollars(spineStage.model, spineStage.steps);
|
||||
if (spDollars === null) spine.unpricedRuns++;
|
||||
else spine.dollars += spDollars;
|
||||
}
|
||||
|
||||
audit.runs++;
|
||||
mergeTokens(audit.tokens, stages.auditor.steps);
|
||||
const audDollars = stageDollars(stages.auditor.model, stages.auditor.steps);
|
||||
@@ -149,7 +135,7 @@ export function graphCostSummary(input: CostSummaryInput): GraphCostSummary {
|
||||
const runDollars = allStages.every((d) => d === null)
|
||||
? null
|
||||
: allStages.reduce<number>((a, d) => a + (d ?? 0), 0);
|
||||
const allSteps = [...stages.extractor.steps, ...spineStage.steps, ...stages.auditor.steps];
|
||||
const allSteps = [...stages.extractor.steps, ...stages.auditor.steps];
|
||||
runs.push({
|
||||
startedAt: run.startedAt,
|
||||
at: run.at,
|
||||
@@ -177,7 +163,7 @@ export function graphCostSummary(input: CostSummaryInput): GraphCostSummary {
|
||||
return {
|
||||
totalDollars: extraction.dollars + audit.dollars,
|
||||
unpricedRuns,
|
||||
stages: { extraction, spine, audit },
|
||||
stages: { extraction, audit },
|
||||
runs: capped,
|
||||
};
|
||||
}
|
||||
|
||||
+133
-19
@@ -7,22 +7,22 @@
|
||||
*
|
||||
* extractor (LLM) → SessionFindings: the raw reasoning chains —
|
||||
* propositions (lonely relevant ones included) and
|
||||
* chains, from the stretch alone (one batch of the
|
||||
* source). Fully trusted: no stage re-checks it.
|
||||
* spine (LLM) → the ASSEMBLY stage (2026-09-30): cross-stretch
|
||||
* chains + connective propositions joining the
|
||||
* fresh stretch to the same-source digest (the span
|
||||
* law is mechanical — src/agent/subagents/spine.ts).
|
||||
* No links — identity stays the auditor's.
|
||||
* chains, from the stretch (one batch) PLUS the
|
||||
* same-source digest (digest-aware, 2026-09-30
|
||||
* evening: the extractor continues its own source,
|
||||
* references existing ids verbatim, assembles
|
||||
* cross-stretch itself). Fully trusted: no stage
|
||||
* re-checks it.
|
||||
* auditor (LLM) → AuditVerdicts: identity links (local id → an existing
|
||||
* node, or create) via graph-exploration tools —
|
||||
* links ONLY (owner, 2026-09-28 evening: the auditor
|
||||
* never mints edges; cross-source connection happens
|
||||
* solely through identified equivalences). Never sees
|
||||
* the source; never drops; never sees chains — its
|
||||
* input is the chunk propositions + the spine's
|
||||
* connectives, propositions only.
|
||||
* input is the fresh propositions only.
|
||||
* merger (HERE) → mergeFindings(): deterministic ledger appends
|
||||
* + the chain-simplification maintenance pass
|
||||
* (2026-09-30 evening — simplifyChains below).
|
||||
*
|
||||
* The merger is deliberately the ONLY stage that touches the ledgers, and it
|
||||
* is pure code: the LLM supplies exactly the judgment calls (what the
|
||||
@@ -200,11 +200,12 @@ export function mergeFindings(
|
||||
mapping.set(p.id, graphNodeId);
|
||||
}
|
||||
|
||||
// 2. Chains — the extractor's and the spine's (the two chain sources;
|
||||
// the stored truth, chains-primary, owner 2026-09-30). Endpoints
|
||||
// 2. Chains — the extractor's alone since the digest-aware redesign
|
||||
// (owner, 2026-09-30 evening: the extractor continues its own
|
||||
// source, so ITS chains may span stretches and digests). Endpoints
|
||||
// resolve through the job-local map OR pass through verbatim when
|
||||
// they name an existing node (the spine references digest ids
|
||||
// directly); chains carry the run's chat-level provenance point and
|
||||
// they name an existing node (digest references, ids verbatim);
|
||||
// chains carry the run's chat-level provenance point and
|
||||
// dedupe by (premises, conclusion) already asserted by THIS chat —
|
||||
// an assertion by another chat stays untouched (provenance unions
|
||||
// at read time; a known argument re-stated appends under the SAME
|
||||
@@ -218,11 +219,11 @@ export function mergeFindings(
|
||||
if (line.provenance.some((pt) => pt.chatId === chatId)) assertedByChat.add(key);
|
||||
}
|
||||
const emitChain = (chain: ExtractedChain): void => {
|
||||
// Mixed namespace (spine stage, 2026-09-30): an endpoint resolves
|
||||
// through the job-local map (this submit's propositions — chunk
|
||||
// findings and spine connectives alike) OR passes through verbatim
|
||||
// when it names an existing node (a spine chain referencing digest
|
||||
// ids directly). Anything else is unresolvable — skip.
|
||||
// Mixed namespace (digest-aware extraction, 2026-09-30): an
|
||||
// endpoint resolves through the job-local map (this submit's
|
||||
// propositions) OR passes through verbatim when it names an
|
||||
// existing node (THE GRAPH SO FAR references, ids verbatim).
|
||||
// Anything else is unresolvable — skip.
|
||||
const resolve = (endpoint: string): string | undefined =>
|
||||
mapping.get(endpoint) ?? (existingIds.has(endpoint) ? endpoint : undefined);
|
||||
const conclusion = resolve(chain.conclusion);
|
||||
@@ -241,7 +242,27 @@ export function mergeFindings(
|
||||
assertedByChat.add(key);
|
||||
stats.chains!++;
|
||||
};
|
||||
for (const chain of findings.chains) emitChain(chain); // the extractor's chains + the spine's spanning chains — the two chain sources
|
||||
for (const chain of findings.chains) emitChain(chain); // the extractor's chains — the one chain source (digest-aware, owner 2026-09-30)
|
||||
|
||||
// 2b. Simplification (owner, 2026-09-30 evening) — the deterministic
|
||||
// dedupe pass over the WHOLE chain ledger (this merge runs inside
|
||||
// the serialized audit section, so the rewrite is race-free). Two
|
||||
// rules, decided per conclusion group, greedy in (premiseCount,
|
||||
// createdAt, key) order — the leanest, earliest derivation stands:
|
||||
// • prefer-fewest-premises: a chain whose premise set strictly
|
||||
// contains a kept chain's premise set is redundant;
|
||||
// • ancestry-cover: a chain whose every premise is a premise of a
|
||||
// kept chain, or derivable from one (transitively, through the
|
||||
// FULL pre-simplification chain set) is redundant — [A,B]⟹X
|
||||
// plus [X,C]⟹Y makes [A,B,C]⟹Y redundant.
|
||||
// Dropped keys take every provenance line with them (the rewrite is
|
||||
// atomic); stats count APPENDS only — simplification is maintenance.
|
||||
const allChains = ledgers.readChains();
|
||||
const keptChains = simplifyChains(allChains);
|
||||
if (keptChains.length !== allChains.length) {
|
||||
ledgers.rewriteLedgers({ chains: keptChains });
|
||||
stats.droppedChains = allChains.length - keptChains.length;
|
||||
}
|
||||
|
||||
// 3. Checkpoint — THE COVERAGE DISCIPLINE (vision-extraction-coverage.md):
|
||||
// only now, after every append succeeded, is the batch covered. The
|
||||
@@ -257,6 +278,99 @@ export function mergeFindings(
|
||||
return stats;
|
||||
}
|
||||
|
||||
/** The deterministic chain-simplification pass (owner, 2026-09-30
|
||||
* evening): strips redundant derivations — same-conclusion chains whose
|
||||
* premises add nothing a leaner kept chain doesn't already derive. Pure
|
||||
* over the input list; the caller owns the ledger rewrite. Exported for
|
||||
* tests.
|
||||
*
|
||||
* Same (premises, conclusion) lines are ONE logical chain (read-time
|
||||
* merge semantics) — a key is kept or dropped as a whole, every
|
||||
* provenance line with it. */
|
||||
export function simplifyChains(lines: GraphChain[]): GraphChain[] {
|
||||
if (lines.length < 2) return lines;
|
||||
|
||||
// Group lines by chain key (logical chains).
|
||||
const groups = new Map<string, GraphChain[]>();
|
||||
for (const line of lines) {
|
||||
const key = chainKey(line.premises, line.conclusion);
|
||||
const bucket = groups.get(key);
|
||||
if (bucket === undefined) groups.set(key, [line]);
|
||||
else bucket.push(line);
|
||||
}
|
||||
|
||||
// Ancestry over the FULL chain set: every node id reachable as a
|
||||
// premise behind an id (A⟹X makes A an ancestor of X; [X,C]⟹Y makes
|
||||
// both A and C ancestors of Y). Memoized walk, cycle-guarded (a cycle
|
||||
// cannot be minted by the extractor — a premise equal to the
|
||||
// conclusion is refused — but the walk stays honest if one ever lands).
|
||||
const premisesOf = new Map<string, string[][]>();
|
||||
for (const line of lines) {
|
||||
const bucket = premisesOf.get(line.conclusion);
|
||||
if (bucket === undefined) premisesOf.set(line.conclusion, [line.premises]);
|
||||
else bucket.push(line.premises);
|
||||
}
|
||||
const ancestryMemo = new Map<string, Set<string>>();
|
||||
const ancestry = (id: string, visiting: Set<string>): Set<string> => {
|
||||
const memo = ancestryMemo.get(id);
|
||||
if (memo !== undefined) return memo;
|
||||
const out = new Set<string>();
|
||||
if (visiting.has(id)) return out;
|
||||
visiting.add(id);
|
||||
for (const premises of premisesOf.get(id) ?? []) {
|
||||
for (const premise of premises) {
|
||||
out.add(premise);
|
||||
for (const ancestor of ancestry(premise, visiting)) out.add(ancestor);
|
||||
}
|
||||
}
|
||||
visiting.delete(id);
|
||||
ancestryMemo.set(id, out);
|
||||
return out;
|
||||
};
|
||||
|
||||
// Per conclusion: greedy in (premiseCount, createdAt, key).
|
||||
const byConclusion = new Map<
|
||||
string,
|
||||
Array<{ key: string; premises: string[]; createdAt: string }>
|
||||
>();
|
||||
for (const [key, bucket] of groups) {
|
||||
const head = bucket[0]!; // the representative — its premises stand
|
||||
const entry = { key, premises: head.premises, createdAt: head.createdAt };
|
||||
const list = byConclusion.get(head.conclusion);
|
||||
if (list === undefined) byConclusion.set(head.conclusion, [entry]);
|
||||
else list.push(entry);
|
||||
}
|
||||
const droppedKeys = new Set<string>();
|
||||
for (const candidates of byConclusion.values()) {
|
||||
candidates.sort(
|
||||
(a, b) =>
|
||||
a.premises.length - b.premises.length ||
|
||||
a.createdAt.localeCompare(b.createdAt) ||
|
||||
a.key.localeCompare(b.key),
|
||||
);
|
||||
const kept: typeof candidates = [];
|
||||
for (const candidate of candidates) {
|
||||
const candidateSet = new Set(candidate.premises);
|
||||
const redundant = kept.some((k) => {
|
||||
// Rule 1 — prefer-fewest-premises: strict subset (equal keys are
|
||||
// grouped, so a kept ⊆ candidate with fewer premises is strict).
|
||||
if (k.premises.every((p) => candidateSet.has(p))) return true;
|
||||
// Rule 2 — ancestry-cover: every candidate premise is a kept
|
||||
// premise, or reachable behind one.
|
||||
const covered = new Set(k.premises);
|
||||
for (const p of k.premises) {
|
||||
for (const ancestor of ancestry(p, new Set())) covered.add(ancestor);
|
||||
}
|
||||
return candidate.premises.every((p) => covered.has(p));
|
||||
});
|
||||
if (redundant) droppedKeys.add(candidate.key);
|
||||
else kept.push(candidate);
|
||||
}
|
||||
}
|
||||
if (droppedKeys.size === 0) return lines;
|
||||
return lines.filter((line) => !droppedKeys.has(chainKey(line.premises, line.conclusion)));
|
||||
}
|
||||
|
||||
// ─── derivation (reads only — everything below is recomputable) ─────────────
|
||||
|
||||
/** The merged STORED truth: propositions + chains (chains-primary, owner
|
||||
|
||||
@@ -45,7 +45,6 @@ export const BACKGROUND_MODEL_ROLES = [
|
||||
"demoBuilder",
|
||||
"visualMaker",
|
||||
"extractor",
|
||||
"spine",
|
||||
"auditor",
|
||||
"partitioner",
|
||||
"problemGenerator",
|
||||
@@ -227,10 +226,6 @@ export const ROLE_SLOT: Readonly<Record<ModelRole, ModelSlotName>> = {
|
||||
researcher: "cheap",
|
||||
demoBuilder: "cheap",
|
||||
extractor: "cheap",
|
||||
// The spine (2026-09-30): cross-stretch assembly — judgment-dense
|
||||
// work over the whole same-source digest, strong tier like the
|
||||
// practice trio.
|
||||
spine: "strong",
|
||||
auditor: "cheap",
|
||||
visualMaker: "vision",
|
||||
pdfReader: "vision",
|
||||
|
||||
Reference in New Issue
Block a user