amosblomqvist
b1d4eadb93
Teacher's Wave-6 review flagged a money-ledger hole: a mid-batch embed throw discarded the spend of batches that had already landed (extraction only sees success results). Owner ruling 2026-09-23 (option A, pay-once): the landed vectors persist too — a retry embeds only what remains, so each text is billed exactly once. - embedNewNodes never throws on embed failures: per-batch try/catch, result gains `failed` (error message), `embedded` counts what actually landed, and usage carries the completed batches' steps either way (usage-on-error). - Zero-landed failures write nothing — the old ledger (its own model/dim) survives untouched. Stale-wipe semantics unchanged. - scripts/embed-graph.ts: honest failure output (lands count, not the in-memory total), exit 1 on failure. - tests: 4 new cases — mid-run failure (32 land, 8 remain; retry resumes and re-bills only the rest), first-batch failure (no write, no spend, old vectors survive), success carries no failed key. Live-verified on port 3004: bogus-model 400 from OpenRouter → exit 1, ledger untouched; healthy re-run embeds 0 (no re-spend); reset scratch setting to null. Gates: typecheck, lint, 899/899 tests, build.
48 lines
2.0 KiB
TypeScript
48 lines
2.0 KiB
TypeScript
/**
|
|
* Embed the whole graph (backfill / cold start / model change): one-time
|
|
* pass that fills the embedding ledger for every node — the same
|
|
* embedNewNodes the post-merge hook runs, just with an empty (or
|
|
* stale-model) ledger so every node is "new".
|
|
*
|
|
* BYOK (wave 6, 2026-09-23): vectors come from the configured provider
|
|
* (graph.embeddingModel, keys from the auth store) — this script resolves
|
|
* the live state up front so an unconfigured machine says so before
|
|
* anything bills. A model CHANGE re-embeds everything (ledger staleness).
|
|
*
|
|
* npx tsx scripts/embed-graph.ts
|
|
*/
|
|
import { embedNewNodes, embeddingPort } from "../src/server/embeddings.ts";
|
|
import { graphStore } from "../src/store/graph.ts";
|
|
|
|
async function main() {
|
|
const resolved = embeddingPort();
|
|
if (!resolved.ok) {
|
|
console.error(`[embed] cannot run — ${resolved.reason}`);
|
|
process.exit(1);
|
|
}
|
|
const t0 = Date.now();
|
|
console.log(`[embed] model: ${resolved.port.model} (provider embeddings, batch 32)`);
|
|
const { embedded, total, usage, failed } = await embedNewNodes(graphStore);
|
|
const secs = ((Date.now() - t0) / 1000).toFixed(1);
|
|
if (usage) {
|
|
const tokens = usage.steps.reduce((sum, s) => sum + (s.inputTokens ?? 0), 0);
|
|
console.log(`[embed] embedding spend: ${tokens} prompt tokens over ${usage.steps.length} call(s)`);
|
|
}
|
|
// Usage-on-error + pay-once (owner ruling 2026-09-23): a mid-run provider
|
|
// failure still reports the landed batches' spend above; the landed
|
|
// vectors are persisted, so the honest count is what LANDED — the
|
|
// in-memory `total` would read as a ledger claim it can't make.
|
|
if (failed) {
|
|
console.error(
|
|
`[embed] failed after ${embedded} node(s) landed (${secs}s) — the ledger keeps exactly those (a retry embeds only the rest): ${failed}`,
|
|
);
|
|
process.exit(1);
|
|
}
|
|
console.log(`[embed] embedded ${embedded} nodes → ledger holds ${total} vectors (${secs}s)`);
|
|
}
|
|
|
|
main().catch((err) => {
|
|
console.error("[embed] failed:", err);
|
|
process.exit(1);
|
|
});
|