embedNewNodes: usage-on-error + pay-once (partial-failure accounting)
Teacher's Wave-6 review flagged a money-ledger hole: a mid-batch embed throw discarded the spend of batches that had already landed (extraction only sees success results). Owner ruling 2026-09-23 (option A, pay-once): the landed vectors persist too — a retry embeds only what remains, so each text is billed exactly once. - embedNewNodes never throws on embed failures: per-batch try/catch, result gains `failed` (error message), `embedded` counts what actually landed, and usage carries the completed batches' steps either way (usage-on-error). - Zero-landed failures write nothing — the old ledger (its own model/dim) survives untouched. Stale-wipe semantics unchanged. - scripts/embed-graph.ts: honest failure output (lands count, not the in-memory total), exit 1 on failure. - tests: 4 new cases — mid-run failure (32 land, 8 remain; retry resumes and re-bills only the rest), first-batch failure (no write, no spend, old vectors survive), success carries no failed key. Live-verified on port 3004: bogus-model 400 from OpenRouter → exit 1, ledger untouched; healthy re-run embeds 0 (no re-spend); reset scratch setting to null. Gates: typecheck, lint, 899/899 tests, build.
This commit is contained in:
+13
-4
@@ -22,14 +22,23 @@ async function main() {
|
||||
}
|
||||
const t0 = Date.now();
|
||||
console.log(`[embed] model: ${resolved.port.model} (provider embeddings, batch 32)`);
|
||||
const { embedded, total, usage } = await embedNewNodes(graphStore);
|
||||
console.log(
|
||||
`[embed] embedded ${embedded} nodes → ledger holds ${total} vectors (${((Date.now() - t0) / 1000).toFixed(1)}s)`,
|
||||
);
|
||||
const { embedded, total, usage, failed } = await embedNewNodes(graphStore);
|
||||
const secs = ((Date.now() - t0) / 1000).toFixed(1);
|
||||
if (usage) {
|
||||
const tokens = usage.steps.reduce((sum, s) => sum + (s.inputTokens ?? 0), 0);
|
||||
console.log(`[embed] embedding spend: ${tokens} prompt tokens over ${usage.steps.length} call(s)`);
|
||||
}
|
||||
// Usage-on-error + pay-once (owner ruling 2026-09-23): a mid-run provider
|
||||
// failure still reports the landed batches' spend above; the landed
|
||||
// vectors are persisted, so the honest count is what LANDED — the
|
||||
// in-memory `total` would read as a ledger claim it can't make.
|
||||
if (failed) {
|
||||
console.error(
|
||||
`[embed] failed after ${embedded} node(s) landed (${secs}s) — the ledger keeps exactly those (a retry embeds only the rest): ${failed}`,
|
||||
);
|
||||
process.exit(1);
|
||||
}
|
||||
console.log(`[embed] embedded ${embedded} nodes → ledger holds ${total} vectors (${secs}s)`);
|
||||
}
|
||||
|
||||
main().catch((err) => {
|
||||
|
||||
Reference in New Issue
Block a user