feat: voice dictation (STT wave 1 — Deepgram cloud)

Two-band architecture from docs/research/stt.md: renderer captures mic
audio in an AudioWorklet (linear16 16kHz mono, ~128ms frames, per-quantum
RMS for the meter) and streams it directly to Deepgram over WebSocket —
audio never loops through our server. The real key stays server-side;
/api/stt/token mints a short-lived grant token (TTL 300s) per run.

Patterns ported from the owner's pi dictate extension (tuned in real
use): generation counter guarding every async handler, the
latest-RMS-never-resets meter (-50..-10dB window, 60ms tick), mic-first
stop + CloseStream + 3s safety net, endpointing=300 with the
utterance_end_ms=1000 backstop.

UI (owner placement): mic button in the stats bar's middle slot on all
three surfaces (home graph-view, chat, draft); while recording the meter
unfurls to fill the row's free width with the interim ghost at its
flank. Esc ends dictation from the composer. Token mint comes before
getUserMedia so a keyless app fails fast without a TCC prompt; one
automatic reconnect on abnormal socket close, errors surface inline.

Live-smoked the full error path in the dev shell (bar renders on all
three pages, 403 grant failure shows inline, mic untouched). The full
audio path needs a Deepgram key with Member permission to mint tokens.
This commit is contained in:
amosblomqvist
2026-09-17 19:31:25 +03:00
parent d46c3dae93
commit e3bdbc2f2c
8 changed files with 532 additions and 0 deletions
+14
View File
@@ -19,6 +19,7 @@ import { AnsweredCard } from "@/components/answered-card";
import { artifactRenderers } from "@/components/artifacts";
import { SubagentSteerCard } from "@/components/artifacts/steer-card";
import { Composer } from "@/components/composer";
import { DictationBar, useDictation } from "@/components/dictation";
import { JobsContext } from "@/components/jobs-context";
import { StatsBar } from "@/components/stats-bar";
import { GraphContextMarker } from "@/components/graph-context-marker";
@@ -246,6 +247,12 @@ export function Chat({
(messages.at(-1)?.parts.some((p) => isToolUIPart(p) && p.state === "input-streaming") ?? false);
const [text, setText] = useState("");
// Voice dictation (STT wave, 2026-09-17): finals append at the end of
// the current text (the page owns the separator, dictate-flush style);
// interims ghost in the stats bar's middle slot next to the meter.
const dictation = useDictation({
append: (t) => setText((v) => (v ? v.replace(/\s+$/, "") + " " + t : t)),
});
// Jobs never gate the composer (contracts.ts "THE ASYNC PROTOCOL" §4): job
// tools resolve with an ack mid-turn, so the learner can interject while
// sub-agents run — a feature, not a race: the wake pipeline queues steers
@@ -864,6 +871,7 @@ export function Chat({
phase={phases.at(-1) ?? "probe"}
turns={turns}
onReserve={setDockReserve}
dictationSlot={<DictationBar dictation={dictation} />}
composer={
<Composer
// Stop applies only to a LOCAL stream: a server-side turn has no
@@ -874,6 +882,7 @@ export function Chat({
onSend={onSend}
onStop={() => stop()}
canSend={canSend}
onEscape={dictation.active ? dictation.stop : undefined}
verdict={
planVerdict
? {
@@ -940,6 +949,7 @@ function DockOverlay({
onReserve,
plan,
onOpenPlan,
dictationSlot,
}: {
pending: ReturnType<typeof findPendingToolCall>;
/** Teach-phase pending question: arrives collapsed behind the reveal button (W5). */
@@ -962,6 +972,9 @@ function DockOverlay({
plan?: React.ReactNode;
/** Opens the full-DAG popup (the marginalia + chip share it). */
onOpenPlan: () => void;
/** The dictation strip (STT wave): rendered in the stats bar's middle
* slot — the mic button + live meter (components/dictation.tsx). */
dictationSlot?: React.ReactNode;
}) {
const { isAtBottom, scrollToBottom, scrollRef } = useStickToBottomContext();
// Teach-phase mount (session reload with a pending check): the dock starts
@@ -1226,6 +1239,7 @@ function DockOverlay({
chatId={chatId}
settings={settings}
onSettingsSaved={onSettingsSaved}
dictationSlot={dictationSlot}
/>
</div>
{/* The plan marginalia (owner, plan-placement round): the bare