embedNewNodes: usage-on-error + pay-once (partial-failure accounting)

Teacher's Wave-6 review flagged a money-ledger hole: a mid-batch embed
throw discarded the spend of batches that had already landed (extraction
only sees success results). Owner ruling 2026-09-23 (option A, pay-once):
the landed vectors persist too — a retry embeds only what remains, so
each text is billed exactly once.

- embedNewNodes never throws on embed failures: per-batch try/catch,
  result gains `failed` (error message), `embedded` counts what actually
  landed, and usage carries the completed batches' steps either way
  (usage-on-error).
- Zero-landed failures write nothing — the old ledger (its own
  model/dim) survives untouched. Stale-wipe semantics unchanged.
- scripts/embed-graph.ts: honest failure output (lands count, not the
  in-memory total), exit 1 on failure.
- tests: 4 new cases — mid-run failure (32 land, 8 remain; retry
  resumes and re-bills only the rest), first-batch failure (no write, no
  spend, old vectors survive), success carries no failed key.

Live-verified on port 3004: bogus-model 400 from OpenRouter → exit 1,
ledger untouched; healthy re-run embeds 0 (no re-spend); reset scratch
setting to null. Gates: typecheck, lint, 899/899 tests, build.
This commit is contained in:
amosblomqvist
2026-09-23 12:21:36 +03:00
parent d755ae82b7
commit b1d4eadb93
3 changed files with 131 additions and 14 deletions
+13 -4
View File
@@ -22,14 +22,23 @@ async function main() {
}
const t0 = Date.now();
console.log(`[embed] model: ${resolved.port.model} (provider embeddings, batch 32)`);
const { embedded, total, usage } = await embedNewNodes(graphStore);
console.log(
`[embed] embedded ${embedded} nodes → ledger holds ${total} vectors (${((Date.now() - t0) / 1000).toFixed(1)}s)`,
);
const { embedded, total, usage, failed } = await embedNewNodes(graphStore);
const secs = ((Date.now() - t0) / 1000).toFixed(1);
if (usage) {
const tokens = usage.steps.reduce((sum, s) => sum + (s.inputTokens ?? 0), 0);
console.log(`[embed] embedding spend: ${tokens} prompt tokens over ${usage.steps.length} call(s)`);
}
// Usage-on-error + pay-once (owner ruling 2026-09-23): a mid-run provider
// failure still reports the landed batches' spend above; the landed
// vectors are persisted, so the honest count is what LANDED — the
// in-memory `total` would read as a ledger claim it can't make.
if (failed) {
console.error(
`[embed] failed after ${embedded} node(s) landed (${secs}s) — the ledger keeps exactly those (a retry embeds only the rest): ${failed}`,
);
process.exit(1);
}
console.log(`[embed] embedded ${embedded} nodes → ledger holds ${total} vectors (${secs}s)`);
}
main().catch((err) => {