amosblomqvist
e3bdbc2f2c
Two-band architecture from docs/research/stt.md: renderer captures mic audio in an AudioWorklet (linear16 16kHz mono, ~128ms frames, per-quantum RMS for the meter) and streams it directly to Deepgram over WebSocket — audio never loops through our server. The real key stays server-side; /api/stt/token mints a short-lived grant token (TTL 300s) per run. Patterns ported from the owner's pi dictate extension (tuned in real use): generation counter guarding every async handler, the latest-RMS-never-resets meter (-50..-10dB window, 60ms tick), mic-first stop + CloseStream + 3s safety net, endpointing=300 with the utterance_end_ms=1000 backstop. UI (owner placement): mic button in the stats bar's middle slot on all three surfaces (home graph-view, chat, draft); while recording the meter unfurls to fill the row's free width with the interim ghost at its flank. Esc ends dictation from the composer. Token mint comes before getUserMedia so a keyless app fails fast without a TCC prompt; one automatic reconnect on abnormal socket close, errors surface inline. Live-smoked the full error path in the dev shell (bar renders on all three pages, 403 grant failure shows inline, mic untouched). The full audio path needs a Deepgram key with Member permission to mint tokens.
56 lines
2.3 KiB
JavaScript
56 lines
2.3 KiB
JavaScript
/*
|
||
* pcm-capture — the dictation audio path (STT wave, 2026-09-17).
|
||
*
|
||
* Runs OFF the main thread (AudioWorklet): converts the mic's Float32
|
||
* samples to 16-bit little-endian PCM (linear16) and hands buffers to the
|
||
* main thread in two streams:
|
||
*
|
||
* { type: "level", rms } — every render quantum (128 samples ≈ 8ms at
|
||
* 16kHz): the quantum's RMS level, 0..1. This
|
||
* is the meter's heartbeat; posting per quantum
|
||
* is what makes the meter feel instant (the
|
||
* dominant latency in the owner's terminal
|
||
* version was a 256ms IO buffer — see
|
||
* ~/.pi/agent/extensions/dictate).
|
||
* { type: "audio", buffer } — Int16Array frames of 2048 samples (128ms),
|
||
* zero-copy (transferred). Deepgram recommends
|
||
* ~100–200ms chunks; silence is streamed too,
|
||
* so the socket never idles (no KeepAlive
|
||
* needed — the 10s NET-0001 timeout only bites
|
||
* a stream that stops SENDING).
|
||
*
|
||
* Ships as a static file under public/ so it rides the standalone bundle.
|
||
* Plain JS by necessity (worklets run outside the module graph).
|
||
*/
|
||
class PcmCaptureProcessor extends AudioWorkletProcessor {
|
||
constructor() {
|
||
super();
|
||
this.frame = new Int16Array(2048);
|
||
this.count = 0;
|
||
}
|
||
|
||
process(inputs) {
|
||
const channel = inputs[0] && inputs[0][0];
|
||
if (!channel) return true; // no input yet — stay alive
|
||
|
||
// RMS for the meter (float domain, before conversion).
|
||
let sumSquares = 0;
|
||
for (let i = 0; i < channel.length; i++) sumSquares += channel[i] * channel[i];
|
||
this.port.postMessage({ type: "level", rms: Math.sqrt(sumSquares / channel.length) });
|
||
|
||
// Float32 → Int16 (clamp first: a hot mic can exceed ±1.0).
|
||
for (let i = 0; i < channel.length; i++) {
|
||
const s = Math.max(-1, Math.min(1, channel[i]));
|
||
this.frame[this.count++] = s < 0 ? s * 0x8000 : s * 0x7fff;
|
||
if (this.count === this.frame.length) {
|
||
this.port.postMessage({ type: "audio", buffer: this.frame }, [this.frame.buffer]);
|
||
this.frame = new Int16Array(2048);
|
||
this.count = 0;
|
||
}
|
||
}
|
||
return true;
|
||
}
|
||
}
|
||
|
||
registerProcessor("pcm-capture", PcmCaptureProcessor);
|