@molecule/app-ai-voice-whisper

On-device Whisper voice provider using transformers.js.

Speech-to-text that runs entirely in the browser — no cloud speech service and no Web Speech API backend required. Works in Brave, ungoogled Chromium, and Firefox, where the native SpeechRecognition API is a non-functional stub.

Auto-generated, AI-first package reference for the molecule.dev ecosystem. It is written to be read by coding agents as much as by people, and is generated from this package's source — edit src/index.ts JSDoc, not this file.

Quick Start

import { setProvider } from '@molecule/app-ai-voice'
import { createProvider } from '@molecule/app-ai-voice-whisper'

setProvider(
  createProvider({
    onModelProgress: (e) => console.log(e.status, e.progress),
  }),
)

Type

provider

Installation

npm install @molecule/app-ai-voice-whisper @huggingface/transformers @molecule/app-ai-voice

API

Interfaces

`ModelProgressEvent`

Progress event emitted while the speech model downloads/initializes.

interface ModelProgressEvent {
  /** Lifecycle stage of the model load. */
  status: 'downloading' | 'loading' | 'ready' | 'error'
  /** Overall download progress from 0 to 100, when known. */
  progress?: number
  /** The file currently being fetched, when known. */
  file?: string
}

`WhisperVoiceConfig`

Configuration for the on-device Whisper voice provider.

interface WhisperVoiceConfig {
  /**
   * Hugging Face model id to run in the browser. Any transformers.js
   * automatic-speech-recognition model works — multilingual Whisper
   * (the default) or an English-only Moonshine model for lower latency.
   * Default: 'onnx-community/whisper-base'.
   */
  model?: string
  /**
   * Inference device. 'auto' (default) picks WebGPU when available and
   * falls back to WASM.
   */
  device?: 'auto' | 'webgpu' | 'wasm'
  /**
   * Model weight precision passed through to transformers.js. Leave unset
   * to use the library's per-device defaults.
   */
  dtype?: string | Record<string, string>
  /**
   * URL prefix (directory) the ONNX Runtime WASM/JS runtime files are served
   * from — e.g. '/transformers-ort/'. Without it, transformers.js falls back
   * to loading the runtime from the jsdelivr CDN, which any app with a
   * `script-src 'self'` CSP (correctly) blocks. Serve the
   * `ort-wasm-simd-threaded*` files from transformers.js's own
   * onnxruntime-web dependency (version must match).
   */
  wasmPaths?: string
  /**
   * Called with model download/initialization progress — wire this to a UI
   * indicator, since the first use downloads tens of MB (cached afterwards).
   */
  onModelProgress?: (event: ModelProgressEvent) => void
  /**
   * RMS amplitude above which a frame counts as speech. Default 0.01.
   */
  speechThreshold?: number
  /**
   * Milliseconds of silence after speech that closes a chunk and sends it
   * for transcription. Default 800.
   */
  silenceMs?: number
  /**
   * Hard cap on a single chunk's length in seconds — a chunk is flushed at
   * this size even without a pause. Default 12, max 30 (Whisper's window).
   */
  maxChunkSeconds?: number
}

Classes

`WhisperVoiceProvider`

On-device Whisper speech-to-text provider.

Captures microphone audio, segments it on pauses, and transcribes each segment locally with a transformers.js ASR model.

class WhisperVoiceProvider implements AIVoiceProvider {
  readonly name = 'whisper'

  private state: VoiceState = 'idle'
  private config: WhisperVoiceConfig
  private handlers: VoiceEventHandlers = {}
  private disposed = false

  private pipelinePromise: Promise<AsrPipeline> | null = null
  private modelReady = false

  private mediaStream: MediaStream | null = null
  private audioContext: AudioContext | null = null
  private processor: ScriptProcessorNode | null = null
  private sourceNode: MediaStreamAudioSourceNode | null = null

  private listening = false
  private recognitionLanguage: string | undefined

  // VAD/chunking state
  private speechFrames: Float32Array[] = []
  private inSpeech = false
  private silentSamples = 0
  private speechSamples = 0

  // Serialized transcription queue
  private transcribeQueue: Float32Array[] = []
  private transcribing = false

  /**
   * Creates a new WhisperVoiceProvider.
   * @param config - Provider configuration (model id, device, VAD tuning).
   */
  constructor(config: WhisperVoiceConfig = {}) {
    this.config = config
  }

  /**
   * Loads (or returns the cached) transformers.js ASR pipeline, reporting
   * download progress through `config.onModelProgress`.
   * @returns The ready-to-use ASR pipeline.
   */
  private loadPipeline(): Promise<AsrPipeline> {
    if (this.pipelinePromise) return this.pipelinePromise

    const progress = (event: ModelProgressEvent): void => {
      this.config.onModelProgress?.(event)
    }

    this.pipelinePromise = (async () => {
      progress({ status: 'loading' })
      const { pipeline, env } = await import('@huggingface/transformers')
      if (this.config.wasmPaths && env.backends.onnx.wasm) {
        env.backends.onnx.wasm.wasmPaths = this.config.wasmPaths
      }
      const device =
        this.config.device && this.config.device !== 'auto'
          ? this.config.device
          : (await hasUsableWebGpu())
            ? 'webgpu'
            : 'wasm'
      if (device === 'wasm' && env.backends.onnx.wasm) {
        // ORT proxy mode runs the WASM engine in its own worker. Without it,
        // model load + every transcription run ON THE MAIN THREAD and freeze
        // the whole page for seconds at a time (measured: a 54s UI stall on
        // moonshine-base fp32 load). Not set for WebGPU, where the JSEP EP
        // needs the main thread and does its heavy work on the GPU anyway.
        env.backends.onnx.wasm.proxy = true
      }

      const options: Record<string, unknown> = {
        device,
        progress_callback: (p: { status?: string; progress?: number; file?: string }) => {
          if (p.status === 'progress') {
            progress({ status: 'downloading', progress: p.progress, file: p.file })
          }
        },
      }
      if (this.config.dtype) options.dtype = this.config.dtype

      const model = this.config.model ?? DEFAULT_MODEL
      let asr: AsrPipeline
      try {
        asr = (await pipeline('automatic-speech-recognition', model, options)) as AsrPipeline
      } catch (error) {
        if (device === 'webgpu') {
          // WebGPU exists but initialization failed (driver/adapter issues are
          // common) — fall back to WASM before giving up.
          asr = (await pipeline('automatic-speech-recognition', model, {
            ...options,
            device: 'wasm',
          })) as AsrPipeline
        } else {
          progress({ status: 'error' })
          throw error
        }
      }
      this.modelReady = true
      progress({ status: 'ready' })
      return asr
    })()

    this.pipelinePromise.catch(() => {
      // Allow a retry on the next startListening call; the caller already
      // receives this rejection via startListening's error handling.
      this.pipelinePromise = null
    })

    return this.pipelinePromise
  }

  /**
   * Starts on-device speech recognition: opens the microphone, loads the
   * model if needed, and emits a final transcript after each pause.
   * @param options - Recognition options; `language` is honored by
   *   multilingual Whisper models.
   * @param handlers - Callbacks for transcripts, state changes, and errors.
   */
  startListening(options?: VoiceRecognitionOptions, handlers?: VoiceEventHandlers): void {
    if (this.disposed || this.listening) return

    this.handlers = handlers ?? {}
    this.recognitionLanguage = options?.language

    const getUserMedia = getGetUserMedia()
    if (!getUserMedia) {
      this.handlers.onError?.({
        code: 'not-supported',
        message: 'Microphone capture is not supported in this environment',
      })
      return
    }

    this.listening = true
    // 'processing' until the model is ready — consumers use the transition to
    // 'listening' to clear their "preparing dictation" indicator. Speech is
    // captured during the load and transcribed once the model arrives.
    this.setState('processing')

    void (async () => {
      try {
        // Kick off the model load and mic open in parallel — the model can
        // take a while on first use and the mic prompt needs to appear now.
        const pipelineReady = this.loadPipeline()
        const stream = await getUserMedia({
          audio: {
            channelCount: 1,
            echoCancellation: true,
            noiseSuppression: true,
            autoGainControl: true,
          },
        })
        if (!this.listening || this.disposed) {
          for (const track of stream.getTracks()) track.stop()
          return
        }
        this.mediaStream = stream

        const context = new AudioContext({ sampleRate: MODEL_SAMPLE_RATE })
        this.audioContext = context
        this.sourceNode = context.createMediaStreamSource(stream)
        // ScriptProcessorNode is deprecated but has no replacement that works
        // without shipping a separate AudioWorklet module file, which npm
        // consumers' bundlers would each need to handle. Revisit when
        // AudioWorklet.addModule accepts inline modules everywhere.
        this.processor = context.createScriptProcessor(FRAME_SIZE, 1, 1)
        this.processor.onaudioprocess = (event) => {
          this.handleFrame(event.inputBuffer.getChannelData(0), context.sampleRate)
        }
        this.sourceNode.connect(this.processor)
        this.processor.connect(context.destination)

        await pipelineReady
        if (this.listening && !this.disposed) {
          this.setState('listening')
        }
      } catch (error) {
        this.teardownCapture()
        this.listening = false
        this.setState('error')
        const isPermission = error instanceof DOMException && error.name === 'NotAllowedError'
        this.handlers.onError?.({
          code: isPermission ? 'not-allowed' : 'start-failed',
          message: error instanceof Error ? error.message : 'Failed to start speech recognition',
        })
      }
    })()
  }

  /**
   * Consumes one audio frame: tracks speech/silence via RMS energy and
   * flushes a chunk to the transcription queue when a pause ends it.
   * @param frame - Raw samples for this frame.
   * @param sampleRate - The capture context's actual sample rate.
   */
  private handleFrame(frame: Float32Array, sampleRate: number): void {
    if (!this.listening) return

    let sumSquares = 0
    for (let i = 0; i < frame.length; i++) sumSquares += frame[i] * frame[i]
    const rms = Math.sqrt(sumSquares / frame.length)

    const threshold = this.config.speechThreshold ?? 0.01
    const silenceMs = this.config.silenceMs ?? 800
    const maxChunkSeconds = Math.min(this.config.maxChunkSeconds ?? 12, 30)

    if (rms >= threshold) {
      this.inSpeech = true
      this.silentSamples = 0
    } else if (this.inSpeech) {
      this.silentSamples += frame.length
    }

    if (this.inSpeech) {
      // Copy — the buffer is reused by the audio pipeline.
      this.speechFrames.push(new Float32Array(frame))
      this.speechSamples += frame.length
// … truncated — the full declaration is in node_modules/@molecule/app-ai-voice-whisper/dist/provider.d.ts

Functions

`createProvider(config)`

Creates a WhisperVoiceProvider instance.

function createProvider(config?: WhisperVoiceConfig): WhisperVoiceProvider

config — Optional configuration (model id, device, VAD tuning).

Returns: A WhisperVoiceProvider running speech-to-text on-device.

Constants

`provider`

The provider implementation — the fleet-standard typed provider const.

Wire it once at startup: setProvider(provider) from @molecule/app-ai-voice. It is a lazy proxy: construction is deferred to the first property access, so importing this module never throws and needs no config up front. Use createProvider(config) instead when you need a custom model, device, or VAD tuning.

const provider: AIVoiceProvider

Core Interface

Implements @molecule/app-ai-voice interface.

Bond Wiring

Setup function to register this provider with the core interface:

import { setProvider } from '@molecule/app-ai-voice'
import { provider } from '@molecule/app-ai-voice-whisper'

export function setupAiVoiceWhisper(): void {
  setProvider(provider)
}

Injection Notes

Requirements

Peer dependencies:

@molecule/app-ai-voice ^1.0.1

Runtime Dependencies

@huggingface/transformers
@molecule/app-ai-voice

The first use downloads the model (tens to hundreds of MB, cached by the browser afterwards) — always wire onModelProgress to a visible indicator. Transcripts arrive as final chunks after each pause; there are no interim results. The default onnx-community/whisper-base model is multilingual; Moonshine models are faster but English-only. GOTCHA: the quantized moonshine decoder exports (q8/int8/q4/q4f16) all fail ONNX session creation under transformers 4.x's bundled dev ORT — use dtype: { encoder_model: 'fp32', decoder_model_merged: 'fp32' }. A failed session is cached by transformers.js for the page's lifetime, so a bad dtype cannot be retried without a reload — pick a working one up front.

E2E Tests

Integration checklist — drive the real UI (live preview, no mocks), adapt each item to this app's actual screens/flows, and check every box off one by one. A box you can't check is an integration bug to fix — not a skip:

Pressing the mic / press-to-talk control calls startListening() and speech appears as a live transcript in the UI — interim VoiceTranscriptEvent updates (isFinal: false) refresh the text as you speak, and the final one (isFinal: true) commits the recognized text via onTranscript.
Stopping (stopListening()) halts recognition cleanly: the transcript stops updating, the mic control returns to idle, and no stray final result fires afterward.
Denying mic permission (or unavailable hardware) fires onError with a VoiceErrorEvent (code: 'not-allowed') and shows a visible message — the mic control never sits as a silent dead button.
The app's text-to-speech action calls speak(text, ...) and you actually hear the given text; the chosen VoiceDescriptor / VoiceSynthesisOptions are honored (voice, language, and rate change the audible output), and stopSpeaking() cuts it off.
The recognition VoiceRecognitionOptions language is respected — setting it to a non-default locale (e.g. 'fr-FR') recognizes in that language rather than always defaulting to English.
A visible listening/speaking indicator tracks getState() / onStateChange — it reads 'listening' while the mic is open and 'speaking' during synthesis, and returns to 'idle' when each ends.
Voice UI is feature-gated on isRecognitionSupported() / isSynthesisSupported() (and getAvailableVoices() is awaited, not read synchronously) so an unsupported browser hides the control instead of throwing.
Microphone access is requested only from a user gesture, its denial is handled gracefully, and captured audio/transcripts stay within the session — nothing is logged or sent anywhere the app didn't intend.

`@molecule/app-ai-voice-whisper`

How it works

Reference