verdicts-only auditor: submit_review is {links, bridging edges} — the findings echo dies

Owner ruling 2026-09-28: the auditor has no control over the
extractor's output and never alters it, so re-emitting the full
findings per submit was pure payload cost — the retry collapse the
schema slim attacked, halved again. submit_review is now links
(local proposition id -> existing graph node; EQUIVALENCE BIAS
carried in the linkTo description) + bridging edges ONLY; the
merger joins extractor findings + verdicts.

mergeFindings(chatId, findings, verdicts, ...) — propositions come
from findings verbatim, links join by id, edges emit extractor's
then auditor's through one dedup-preserving emitter. The old echo
form hid an accidental drop channel: an auditor that omitted a
proposition from its echo silently dropped it from the merge.
Verdicts-only removes it structurally — extractor output always
fully merges (tests updated to full-merge expectations).

AuditResult/AuditedProposition -> IdentityLink/AuditVerdicts;
audit() logs links/bridges; batch view renders anchorless like
transcriptView; auditor prompt rewritten to the verdicts-only
deliverable; architecture.md division-of-trust updated.
This commit is contained in:
amosblomqvist
2026-09-28 19:22:58 +03:00
parent bd9694b0e4
commit f11b1664dc
6 changed files with 163 additions and 142 deletions
+47 -43
View File
@@ -15,7 +15,7 @@ import {
import type { PiEngine } from "../models.ts";
import { runSubagent } from "./pi-run.ts";
import type {
AuditResult,
AuditVerdicts,
CallUsage,
ChatRecord,
ExtractionRunUsage,
@@ -152,31 +152,36 @@ const findingsProps = {
const findingsSchema = Type.Object(findingsProps);
const auditProps = {
...findingsProps,
propositions: Type.Array(
Type.Object({
...propositionProps,
linkTo: Type.Optional(
Type.String({
// VERDICTS-ONLY (owner, 2026-09-28): the auditor never alters the
// extractor's findings, so it does not restate them — links + bridging
// edges are its entire deliverable. An order-of-magnitude smaller submit
// than the old full-findings echo.
const auditSchema = Type.Object({
links: Type.Optional(
Type.Array(
Type.Object({
id: Type.String({ description: 'The proposition id from the findings ("p1", …).' }),
linkTo: Type.String({
description:
"The id of the EXISTING graph node this proposition IS — your identity verdict. Absent = create a new node. EQUIVALENCE BIAS (2026-09-27): paraphrases, rewordings, and compressions of the same claim MERGE; only genuinely distinct propositions stay apart.",
"The id of the EXISTING graph node this proposition IS — your identity verdict. EQUIVALENCE BIAS (2026-09-27): paraphrases, rewordings, and compressions of the same claim MERGE; only genuinely distinct propositions stay apart.",
}),
),
}),
{
description: "Every distinct proposition the session touched — true statements, misconception-free.",
},
}),
{ description: "Identity verdicts — which findings propositions ARE existing graph nodes. A proposition absent from here = create a new node." },
),
),
};
const auditSchema = Type.Object(auditProps);
edges: Type.Optional(
Type.Array(edgeSchema, {
description:
"BRIDGING edges ONLY — new edges welding this batch into the existing graph, with an EXISTING graph node id as one endpoint (or both) and/or a findings proposition id. NEVER restate the extractor's own edges.",
}),
),
});
// The notes front door's shapes are IDENTICAL to the session shapes now
// (the only difference was quiz attribution, which no longer exists):
// one schema, one findings type, both front doors.
type NotesFindings = SessionFindings;
type NotesAuditResult = AuditResult;
type NotesAuditResult = AuditVerdicts;
// ── the weak-model string-heal (owner, 2026-09-28) ──
//
@@ -213,9 +218,9 @@ function fillFindings(raw: Static<typeof findingsSchema>): SessionFindings {
};
}
function fillAudit(raw: Static<typeof auditSchema>): AuditResult {
function fillAudit(raw: Static<typeof auditSchema>): AuditVerdicts {
return {
propositions: raw.propositions.map((p) => ({ ...p, aliases: p.aliases ?? [] })),
links: raw.links ?? [],
edges: raw.edges ?? [],
};
}
@@ -437,12 +442,12 @@ its work. Your job is IDENTITY: connecting those findings to the CURRENT knowled
You explore the graph through your tools: find (glob over names + statements + aliases), list_sessions, and
list_propositions (full statement text, per session). The graph may be large — search, don't assume.
IDENTITY — link-or-create, per proposition:
IDENTITY — links, per proposition:
- For each proposition, search the graph (its key words, its synonyms, its phrasings). Does an EXISTING node already
state this same proposition?
- If yes: set "linkTo" to that node's id. The session's phrasings join its aliases and its provenance; the node
keeps its existing name (labels stay stable — ignore your name for linked nodes).
- If no: omit "linkTo" and a new node is created with the extractor's name as its label.
- If yes: add { "id": <its proposition id>, "linkTo": <that node's id> } to "links". The session's phrasings join
its aliases and its provenance; the node keeps its existing name.
- If no: add nothing — a new node is created with the extractor's name as its label.
- EQUIVALENCE BIAS (owner ruling): paraphrases, rewordings, and compressions of the same claim MERGE. When two
statements are the same claim at different precision or from different angles, LINK them. Only genuinely distinct
propositions — different claims about the world — stay apart. You are NOT guarding against wrong merges; you are
@@ -453,13 +458,13 @@ BRIDGING EDGES — the one place you create edges:
existing node's real id as "from" or "to"). This is how the new material gets welded into the old graph: when the
existing graph's content makes a proposition follow, or grounds it, draw the edge.
- Keep edges typed like the extractor's: "follows" = derivable; "supports" = grounds without entailing.
- NEVER drop or rewrite the extractor's own propositions or edges — pass them through unchanged (adding linkTo is
the only edit). Never invent propositions.
- Never invent propositions. NEVER restate the extractor's own edges — they merge as-is.
Reproduce every proposition and edge you received verbatim. Your entire deliverable
is ONE submit_review call.
Your deliverable is ONLY the verdicts: "links" + "edges" (bridging ONLY). NEVER restate the propositions or the
extractor's edges — the merger already has them. Your entire deliverable is ONE submit_review call.
Include EVERY top-level field of the call even when empty — "edges": []. NEVER omit a field because it is empty.`;
Include EVERY top-level field of the call even when empty — "links": [] when nothing links, "edges": [] when there
are no bridges. NEVER omit a field because it is empty.`;
/** The auditor — one shape for both front doors now (the notes schema was
* identical once attribution died). An agentic loop: graph-exploration
@@ -470,11 +475,11 @@ function runAuditor(
signal: AbortSignal | undefined,
onStep: (usage: Usage, model: string, step: number, finishReason: string, tools: string) => void,
tools: Record<string, AgentTool<any>>,
): Promise<AuditResult> {
): Promise<AuditVerdicts> {
return runSubmitAgent({
role: "auditor",
toolName: "submit_review",
toolDescription: "Submit the findings with identity verdicts (linkTo) and any bridging edges — for the merge.",
toolDescription: "Submit the verdicts ONLY: identity links + bridging edges — never the propositions themselves.",
parameters: auditSchema,
fill: fillAudit,
systemPrompt: AUDITOR_INSTRUCTIONS,
@@ -763,7 +768,7 @@ export function createExtractionPipeline(deps: ExtractionPipelineDeps): Extracti
return findings;
}
async function audit(chatId: string, findings: SessionFindings): Promise<AuditResult> {
async function audit(chatId: string, findings: SessionFindings): Promise<AuditVerdicts> {
const signal = runningExtractions.get(chatId)?.controller.signal;
signal?.throwIfAborted(); // an abort during the extract→audit gap ends here
const graph = getGraph(graphStore);
@@ -788,11 +793,9 @@ export function createExtractionPipeline(deps: ExtractionPipelineDeps): Extracti
},
auditorGraphTools({ chatStore, graphStore }),
);
const bridged = auditResult.edges.filter((e) => !e.from.startsWith("p") || !e.to.startsWith("p")).length;
console.info(
`[extraction ${chatId}] audit: ${auditResult.propositions.length} propositions through, ` +
`${auditResult.propositions.filter((p) => p.linkTo !== undefined).length} linked to existing nodes, ` +
`${auditResult.edges.length} edges (${bridged} bridging)`,
`[extraction ${chatId}] audit: ${findings.propositions.length} propositions through, ` +
`${auditResult.links.length} linked to existing nodes, ${auditResult.edges.length} bridging edges`,
);
console.info(`[extraction ${chatId}] auditor done in ${Math.round(performance.now() - t0)}ms`);
return auditResult;
@@ -861,13 +864,14 @@ export function createExtractionPipeline(deps: ExtractionPipelineDeps): Extracti
);
for (const batch of batches) {
// The batch's source view: the ONE shared markdown engine
// (owner 2026-09-27) over the batch's messages — the same
// rendering the user-facing session export will use, id
// anchors preserved. The fingerprint's line format is
// separate and unchanged (no mass re-extraction).
// (owner 2026-09-27) over the batch's messages. Anchors OFF
// (2026-09-28): propositions report no message ids, so the
// [id] anchors are noise — same as transcriptView. The
// fingerprint's line format is separate and unchanged (no
// mass re-extraction).
const view = sessionMarkdown(
batch.map((slot) => slot.message),
{ title: record.title },
{ title: record.title, anchors: false },
);
setPhase(chatId, "extracting", "chat");
const findings = await extract(chatId, view);
@@ -896,7 +900,7 @@ export function createExtractionPipeline(deps: ExtractionPipelineDeps): Extracti
// merges in), the stored fingerprint won't match its new
// one and it re-opens: honest by construction.
const foldedAt = new Date().toISOString();
const batchStats = mergeFindings(chatId, auditResult, { chatStore, graphStore }, {
const batchStats = mergeFindings(chatId, findings, auditResult, { chatStore, graphStore }, {
foldedAt,
coverage: batch
.filter((slot) => messageViewLines(slot.message, slot.message.id).length > 0)
@@ -996,7 +1000,7 @@ export function createExtractionPipeline(deps: ExtractionPipelineDeps): Extracti
setPhase(id, "auditing", "notes", input.filePath);
const auditResult = await audit(id, findings);
setPhase(id, "merging", "notes", input.filePath);
return mergeFindings(id, auditResult, { chatStore, graphStore }, { foldedAt: input.importedAt });
return mergeFindings(id, findings, auditResult, { chatStore, graphStore }, { foldedAt: input.importedAt });
});
// Same post-merge settle as the chat path.
settleGraphLayout(graphStore);
+20 -16
View File
@@ -229,24 +229,28 @@ export interface SessionFindings {
edges: ExtractedEdge[];
}
/** The auditor's verdict on one proposition: the findings, plus `linkTo` —
* the identity judgment. Absent = create a new node. */
export interface AuditedProposition extends ExtractedProposition {
/** The existing GraphNode id this proposition IS — the auditor's
* identity verdict. On link, the session's phrasings (statement,
* aliases) join the node's aliases and the provenance unions.
* EQUIVALENCE BIAS (2026-09-27): paraphrases, rewordings, and
* compressions of the same claim MERGE — do not create a duplicate
* because the wording differs. Only genuinely distinct propositions
* stay apart. */
linkTo?: string;
/** One identity verdict: the extractor's local proposition id → the
* EXISTING graph node that proposition IS. On link, the session's
* phrasings (statement, aliases) join the node's aliases and the
* provenance unions. EQUIVALENCE BIAS (2026-09-27): paraphrases,
* rewordings, and compressions of the same claim MERGE — do not create
* a duplicate because the wording differs. Only genuinely distinct
* propositions stay apart. */
export interface IdentityLink {
id: string;
linkTo: string;
}
/** The auditor's output: the findings with identity verdicts, plus any
* bridging edges drawn to existing graph nodes. No drop channel — the
* auditor never rejects; the extractor is fully trusted (2026-09-27). */
export interface AuditResult {
propositions: AuditedProposition[];
/** The auditor's output — VERDICTS ONLY (owner, 2026-09-28): the auditor
* never alters the extractor's findings, so it does not restate them;
* the deterministic merger joins findings + verdicts. `links` = the
* identity verdicts (local id ⇄ existing node); `edges` = BRIDGING
* edges the auditor draws from existing graph nodes to the batch's
* propositions (endpoints: existing node ids and/or local proposition
* ids). No drop channel — the auditor never rejects; the extractor is
* fully trusted (2026-09-27). */
export interface AuditVerdicts {
links: IdentityLink[];
edges: ExtractedEdge[];
}
+34 -31
View File
@@ -7,10 +7,10 @@
*
* extractor (LLM) → SessionFindings: the raw reasoning chains —
* propositions (lonely relevant ones included) and
* follows/supports edges, each with verbatim evidence,
* follows/supports edges,
* from the source ALONE (whole session as markdown,
* or raw notes). Fully trusted: no stage re-checks it.
* auditor (LLM) → AuditResult: identity verdicts (linkTo an existing
* auditor (LLM) → AuditVerdicts: identity links (local id → an existing
* node, or create) via graph-exploration tools, plus
* any BRIDGING edges it draws between existing graph
* nodes and the new propositions. Never sees the
@@ -43,7 +43,9 @@
import { randomBytes } from "node:crypto";
import type { CoverageEntry } from "./contracts/graph.ts";
import type {
AuditResult,
AuditVerdicts,
ExtractedEdge,
SessionFindings,
Cluster,
EmissionStats,
GraphEdge,
@@ -114,7 +116,13 @@ const samePoint = (a: ProvenancePoint, b: ProvenancePoint): boolean =>
* idempotency, one graph. The `source` parameter carries the checkpoint
* stamp; the merger itself is chat-store-free.
*/
export function mergeFindings(chatId: string, audit: AuditResult, deps: GraphDeps, source: MergeSource): EmissionStats {
export function mergeFindings(
chatId: string,
findings: SessionFindings,
verdicts: AuditVerdicts,
deps: GraphDeps,
source: MergeSource,
): EmissionStats {
const { graphStore: ledgers } = deps;
const stats: EmissionStats = { nodes: 0, edges: 0 };
@@ -132,16 +140,20 @@ export function mergeFindings(chatId: string, audit: AuditResult, deps: GraphDep
seenMappingKeys.add(m.sourceId);
}
const now = new Date().toISOString();
const linkOf = new Map(verdicts.links.map((l) => [l.id, l.linkTo]));
for (const p of audit.propositions) {
// VERDICTS-ONLY (owner, 2026-09-28): the propositions come from the
// extractor's findings verbatim; the auditor's links join here by id.
for (const p of findings.propositions) {
if (mapping.has(p.id)) continue; // already resolved for this chat — idempotent
const points = pointOf(chatId);
const linkTo = linkOf.get(p.id);
let graphNodeId: string;
if (p.linkTo !== undefined && existingIds.has(p.linkTo)) {
if (linkTo !== undefined && existingIds.has(linkTo)) {
// Identity verdict: this proposition IS that node. Append a link
// line — the session's phrasings (statement + aliases) join the
// node's aliases, the provenance unions. Reads merge by id.
graphNodeId = p.linkTo;
graphNodeId = linkTo;
// Link-append: phrasings + provenance join at read time (the
// first line's name/statement/createdAt stand).
ledgers.appendNode({
@@ -173,42 +185,33 @@ export function mergeFindings(chatId: string, audit: AuditResult, deps: GraphDep
}
}
// 2. Edges. Resolve endpoints through the mapping — with BRIDGING
// (2026-09-27): an endpoint the extractor never proposed but that
// names an EXISTING graph node is the auditor drawing a connection
// between the graph it explored and this batch's propositions. Such
// edges carry chat-scoped provenance with NO message anchor (nothing
// in the transcript asserts them — the auditor is the asserter);
// transcript-articulated edges keep message-granular provenance.
// Dedupe by (from,to,type) already asserted by THIS chat (an existing
// assertion by another chat stays untouched — provenance unions at
// read time).
// 2. Edges — the extractor's own first, then the auditor's BRIDGING
// edges (endpoints may be local ids and/or EXISTING graph node ids —
// the auditor welding the batch into the graph it explored). Both
// resolve through the mapping, both carry the run's chat-level
// provenance point, and both dedupe by (from,to,type) already
// asserted by THIS chat: an echoed duplicate (a weak auditor
// restating an extractor edge) merges to one append; an assertion
// by another chat stays untouched (provenance unions at read time).
const assertedByChat = new Set<string>();
for (const line of ledgers.readEdges()) {
const key = `${line.from}\u0000${line.to}\u0000${line.type}`;
if (line.provenance.some((pt) => pt.chatId === chatId)) assertedByChat.add(key);
}
for (const edge of audit.edges) {
const emitEdge = (edge: ExtractedEdge): void => {
const resolve = (endpoint: string): string | undefined =>
mapping.get(endpoint) ?? (existingIds.has(endpoint) ? endpoint : undefined);
const from = resolve(edge.from);
const to = resolve(edge.to);
if (!from || !to || from === to) continue; // unresolvable endpoint — skip
const bridged = !mapping.has(edge.from) || !mapping.has(edge.to);
if (!from || !to || from === to) return; // unresolvable endpoint — skip
const key = `${from}\u0000${to}\u0000${edge.type}`;
if (assertedByChat.has(key)) continue;
ledgers.appendEdge({
from,
to,
type: edge.type,
// bridged vs transcript edges both carry the run's chat-level
// point now; the distinction lives in the endpoint resolution
// above (mapping vs existing node id), not in the provenance.
provenance: pointOf(chatId),
});
if (assertedByChat.has(key)) return;
ledgers.appendEdge({ from, to, type: edge.type, provenance: pointOf(chatId) });
assertedByChat.add(key);
stats.edges++;
}
};
for (const edge of findings.edges) emitEdge(edge);
for (const edge of verdicts.edges) emitEdge(edge);
// 3. Checkpoint — THE COVERAGE DISCIPLINE (vision-extraction-coverage.md):
// only now, after every append succeeded, is the batch covered. The