BYOK provider embeddings (wave 6 chunk B): kill local transformers
Replace the @huggingface/transformers local embedder (~316MB ONNX cache) with an EmbeddingPort seam: graph.embeddingModel (nullable "provider:id" slug) selects an OpenRouter/OpenAI/Gemini adapter, resolved fresh per call from settings + the auth store so key rotations land immediately. - src/server/embeddings.ts rewritten: slug grammar, injected-fetch adapters (OpenRouter input_type, Gemini taskType batching, OpenAI index-ordered rows), honest resolution (null = skip-with-info, set model with missing key = honest error), embedNewNodes with model-change re-embed wipe + ghost pruning + usage steps, federated catalog (OpenRouter /api/v1/embeddings/models — VERIFIED live, the inverse path 404s; OpenAI text-embedding- filter; Gemini embedContent filter). - GET /api/models/embeddings: federated catalog, empty when no keys. - Extraction tail records stages.embedding usage into the run's money channel; EMBEDDING_PRICES overlay in core/prices.ts (outside the sync-generated block), callCost falls back to it; cost-summary folds the fourth stage; graph-context injection resolves its embedder per run, fails honestly when unconfigured, records query tokens. - Settings: GraphSettings.embeddingModel (null factory), section parser validates slug shape (id may contain colons — OpenRouter :free suffixes, live catalog fact). - Removal: @huggingface/transformers uninstalled, PATHS.modelCache and onnx tracing entries deleted, @huggingface tracing-excludes guard added, stale sharp/libvips comments updated to historical. Live-verified on port 3004: catalog federates 36 rows (33 OpenRouter + 3 OpenAI), embedNewNodes embedded 10 nodes via the free liquid/lfm-2.5 model (dim 1024, 435 prompt tokens, 1 call, $0), settings PUT rejects bad slugs with 400. Gates: typecheck, lint, 895/895 tests, build.
This commit is contained in:
@@ -205,11 +205,13 @@ function svgViewport(svg: string, opts: RenderSvgOptions): { width: number; heig
|
||||
* are an arbitrary SPLIT of one zlib stream, so merging them is the more
|
||||
* standard shape, byte-identical stream, zero re-encoding.
|
||||
*
|
||||
* Pure Node on purpose (no sharp): the process already loads transformers'
|
||||
* nested sharp + libvips, and importing the root sharp (Next's optional
|
||||
* dependency) put a SECOND libvips into the process — macOS's ObjC runtime
|
||||
* warns (duplicate GNotificationCenterDelegate, "mysterious crashes").
|
||||
* One native image lib per process; app code carries none. */
|
||||
* Pure Node on purpose (no sharp): historically the process already carried
|
||||
* a native image lib via @huggingface/transformers' nested sharp + libvips,
|
||||
* and importing the root sharp (Next's optional dependency) put a SECOND
|
||||
* libvips into the process — macOS's ObjC runtime warns (duplicate
|
||||
* GNotificationCenterDelegate, "mysterious crashes"). The transformers
|
||||
* embedder died with wave 6 (2026-09-23) but the rule lives on: one native
|
||||
* image lib per process; app code carries none. */
|
||||
export function providerSafePng(png: Buffer): Buffer {
|
||||
const SIG_LEN = 8;
|
||||
if (png.length < SIG_LEN + 12 || png.subarray(0, SIG_LEN).toString("hex") !== "89504e470d0a1a0a") return png;
|
||||
|
||||
Reference in New Issue
Block a user