sorokin.oleg.edu/learnharness

Files

T

amosblomqvist 573e369db5 fix: dictation cloud path — bearer subprotocol, access_token grant field

Two live-verified Deepgram facts from the first real mic-test round:
- /v1/auth/grant returns { access_token }, not { token } — the route now
  parses the right field.
- Minted JWTs authenticate over the WebSocket via the "bearer"
  subprotocol; ["token", jwt] is rejected at the handshake (token is
  for raw API keys only, as in the pi dictate extension).

Also documents the env-precedence gotcha in docs/research/stt.md: a
DEEPGRAM_API_KEY exported in the shell (e.g. .zshrc) silently beats
.env.local in next dev and the desktop authEnv() seam.

2026-09-17 19:44:35 +03:00

7.5 KiB

Raw Blame History

STT research — verified findings (2026-09-17)

Two researcher passes over current (Sept 2026) sources, cross-checked against our codebase. Decisions at the bottom.

Cloud: Deepgram (the default)

Verified facts

Streaming is a WebSocket API (wss://api.deepgram.com/v1/listen); REST is batch-only. Query params carry config; binary audio frames up, JSON Results down; client messages KeepAlive / Finalize / CloseStream.
JS SDK: @deepgram/sdk v5 (v5.11.0, Sept 2026) — browser-capable WebSocket (live transcription works in the renderer directly; only REST needs a CORS proxy). listen.v1 socket = Nova; listen.v2 = Flux (conversational, turn-taking — overkill for dictation, skip).
Model: nova-3 (still the flagship general/dictation model as of Sept 2026; no Nova-4). language=multi exists for code-switching.
Auth: API key via Authorization: token header. For client-side use, Deepgram recommends short-lived access tokens (auth.v1.tokens.grant()) instead of embedding the key — fits our architecture exactly: key stays in auth.json → server env; /api/stt/token mints a token; renderer opens wss directly with it. No audio proxying through our server, no ws dependency on the server.
Pricing: streaming Nova-3 $0.0048/min promotional (regular $0.0077), pre-recorded $0.0043/min; per-second billing. $200 one-time free credit, no card. Diarization +$0.0020/min streaming — skip (single-user dictation).
http://127.0.0.1 is a secure context (potentially trustworthy origin per the Secure Contexts spec) — getUserMedia works without TLS. Our permission handlers (request + check, media-only, loopback-origin-only) are already the documented required pattern.
macOS: NSMicrophoneUsageDescription already in the packaged build. TCC bites fully only post-signing — test the packaged signed app later. Call app.focus() / win.show() before first capture so the TCC prompt isn't behind the window.

Capture: AudioWorklet → linear16, 16 kHz mono (not MediaRecorder)

getUserMedia({ audio: { sampleRate: 16000, channelCount: 1, echoCancellation: true } }), Float32→Int16 in an AudioWorklet (off-main-thread), stream linear16 frames with encoding=linear16&sample_rate=16000&channels=1. Deterministic format, no container surprises (webm/opus streaming has unresolved community reports of empty transcripts), no encoder lookahead, ~100–200ms chunks. This is what Deepgram's own browser layer does. The worklet module ships as a static file under public/ — rides the standalone bundle for free.
Zero new npm deps for the cloud path: getUserMedia + Worklet + WebSocket are platform APIs.

Gotchas that shape the design

10s idle timeout (server closes with NET-0001): send KeepAlive text frames every 3–5s during silence. Timestamps resume contiguously — long pauses in a dictation run are safe.
Interim results: interim_results=true → partials (is_final=false) to render live; finals (speech_final=true / is_final=true) commit. speech_final (endpointing-driven) and is_final (model-driven) are different flags — check both.
Endpointing + UtteranceEnd: endpointing uses Deepgram's server-side VAD (endpointing=500 for a real pause threshold; the documented "10 ms default" is a doc quirk — always set it explicitly). Add utterance_end_ms=1000 as the noise-robust backstop (word-timing-gap based, needs interim_results=true).
smart_format=true (punctuation, caps, numbers/dates) — but it can delay endpointing while formatting e.g. phone numbers; keep UtteranceEnd on.
No server-side buffering on disconnect: reconnect + resume is our job (SDK v5 auto-reconnect, cancellable via abortSignal).

Alternatives (why still Deepgram): OpenAI live-transcribe $0.017/min (~3.5×), no word timestamps/confidence, 60-min session cap. AssemblyAI Universal-Streaming $0.0025/min is cheaper but smaller ecosystem. Speechmatics best for non-English but ~2×. GCP is service-account-shaped — bad BYOK fit. Verdict: Deepgram stays the pragmatic default.

Cloud: Deepgram (the default) — landed

Live-verified facts (2026-09-17, cloud wave)

The grant response field is access_token, not token — POST /v1/auth/grant returns { "access_token": "<jwt>" }.
Minted JWTs authenticate over the WebSocket via the bearer subprotocol: new WebSocket(url, ["bearer", jwt]). The token subprotocol is for raw API keys only (the pi dictate extension's path) — ["token", jwt] is rejected at the handshake (TLS-level close, no 401).
The key needs ≥ Member permission (console → API Keys → Create Key → Advanced options → permissions dropdown → Member). A default key can transcribe but gets FORBIDDEN from /auth/grant.
Env precedence gotcha: a real DEEPGRAM_API_KEY exported in the shell (e.g. .zshrc) beats .env.local in both next dev and the desktop shell's authEnv() seam. Stale exports silently override the file — don't export the key in shell rc files; .env.local/auth.json are the canonical seams.

Local: whisper.cpp (the Advanced option)