@molecule/app-ai-voice-whistle

On-device Whistle voice provider using the Cactus needle WASM engine.

Speech-to-text that runs entirely in the browser on CPU (WebAssembly — no WebGPU needed): Cactus Compute's 16.9 MB Whistle model transcribes seven languages (en, de, fr, es, it, nl, pl) with automatic language detection, optional keyword biasing and per-word timestamps. No audio ever leaves the device, and it works in Brave, ungoogled Chromium and Firefox, where the native SpeechRecognition API is a non-functional stub.

Quick Start

import { setProvider } from '@molecule/app-ai-voice'
import { createProvider, supportsRecognitionLanguage } from '@molecule/app-ai-voice-whistle'

if (supportsRecognitionLanguage(navigator.language)) {
  setProvider(
    createProvider({
      onModelProgress: (e) => console.log(e.status, e.progress),
    }),
  )
}

Type

provider

Installation

npm install @molecule/app-ai-voice-whistle @molecule/app-ai-voice

API

Interfaces

ModelProgressEvent

Progress event emitted while the Whistle engine and weights download/initialize.

interface ModelProgressEvent {
  /** Lifecycle stage of the model load. */
  status: 'downloading' | 'loading' | 'ready' | 'error'
  /** Overall download progress from 0 to 100, when known. */
  progress?: number
  /** The file currently being fetched, when known. */
  file?: string
}

WhistleEngine

The loaded engine, ready to transcribe.

interface WhistleEngine {
  /** Transcribes 16 kHz mono float PCM (≤ 30 s per call). */
  transcribe(
    pcm: Float32Array,
    options?: WhistleEngineCallOptions & { wordTimestamps?: boolean },
  ): WhistleTranscript
  /** Appends ~1 s of PCM to the live stream; commits agreed words. */
  streamProcess(pcm: Float32Array, options?: WhistleEngineCallOptions): WhistleStreamResult
  /** Ends the stream, committing the tail; the next process call starts fresh. */
  streamStop(): WhistleStreamResult
}

WhistleEngineCallOptions

Options the engine calls accept for language and keyword biasing.

interface WhistleEngineCallOptions {
  /** ISO 639-1 language to force, or null to detect. */
  language?: string | null
  /** Keywords as an array or newline-separated string, or null for none. */
  keywords?: string[] | string | null
}

WhistleEngineSource

Where the engine's three files come from.

interface WhistleEngineSource {
  /** URL of `needle.js`; `needle.wasm` is resolved from the same directory. */
  jsUrl: string
  /** URL of the `whistle.cact` weights. */
  weightsUrl: string
}

WhistleStreamResult

Result of one needle_stream_transcribe_process / _stop call.

interface WhistleStreamResult {
  /** Words this call committed (agreed on by two consecutive passes). */
  text: string
  /** The committed words with timings. */
  words: WhistleWord[]
  /** The unconfirmed tail (may still be revised by later passes). */
  pending: string
  /** Language used for the stream (`''` for silence). */
  language: string
  /** Seconds of audio the stream has received. */
  received: number
  /** Wall milliseconds this pass took. */
  pass_ms: number
}

WhistleTranscript

Result of one needle_transcribe call, as returned by the engine.

interface WhistleTranscript {
  /** The transcript. Silence/steady noise gives `''`. */
  text: string
  /** Detected or forced language (`''` for silence). */
  language: string
  /** Milliseconds to the first token. */
  ttft_ms: number
  /** Decoder tokens per second after the first token. */
  decode_tps: number
  /** Per-word timing, present when word timestamps were requested. */
  words?: WhistleWord[]
}

WhistleVoiceConfig

Configuration for the on-device Whistle voice provider.

interface WhistleVoiceConfig {
  /**
   * URL of the Cactus `needle.js` WASM glue file. `needle.wasm` is resolved
   * from the same directory. Default: the Hugging Face copy at
   * https://huggingface.co/Cactus-Compute/needle3/resolve/main/wasm/needle.js.
   * Self-host the three engine files same-origin (and pass this) when the app
   * has a Content-Security-Policy that blocks third-party scripts.
   */
  engineUrl?: string
  /**
   * URL of the `whistle.cact` weights file (16.9 MB). Default: the Hugging
   * Face copy at https://huggingface.co/Cactus-Compute/whistle/resolve/main/whistle.cact.
   */
  weightsUrl?: string
  /**
   * ISO 639-1 language to force for every transcription (one of `en de fr es
   * it nl pl`). Leave unset to auto-detect per chunk.
   */
  language?: WhistleLanguage
  /**
   * Words and phrases (or one newline-separated string) fed to Whistle's
   * keyword biasing — an Aho-Corasick automaton that steers transcription
   * toward names/product words (e.g. the project's identifiers). Explicit
   * opt-in only: biasing CHANGES transcripts, so never pass a large list "to
   * be helpful" — it can distort neutral words.
   */
  keywords?: string[] | string
  /**
   * Request per-word timestamps (start/end/probability). Off by default; when
   * on, transcript confidence is the average word probability instead of 1.
   */
  wordTimestamps?: boolean
  /**
   * Called with engine/weights download progress — wire this to a UI
   * indicator: the first use downloads ~18 MB (engine ~1 MB + weights
   * 16.9 MB; cached by the browser afterwards).
   */
  onModelProgress?: (event: ModelProgressEvent) => void
  /**
   * RMS amplitude above which a frame counts as speech. Default 0.01.
   */
  speechThreshold?: number
  /**
   * Milliseconds of silence after speech that closes a chunk and sends it
   * for transcription. Default 800.
   */
  silenceMs?: number
  /**
   * Hard cap on a single chunk's length in seconds — a chunk is flushed at
   * this size even without a pause. Default 15, clamped to 30 (Whistle's
   * per-pass window).
   */
  maxChunkSeconds?: number
}

WhistleWord

One transcribed word with timing and confidence.

interface WhistleWord {
  /** The word (includes its trailing punctuation). */
  word: string
  /** Start time in seconds from the start of the clip. */
  start: number
  /** End time in seconds from the start of the clip. */
  end: number
  /** Decoder confidence for this word, 0 to 1. */
  probability: number
}

Types

WhistleLanguage

The seven languages Whistle transcribes (auto-detected or forced).

type WhistleLanguage = 'en' | 'de' | 'fr' | 'es' | 'it' | 'nl' | 'pl'

Classes

WhistleEngineError

Error thrown when the Whistle engine cannot be loaded or a call fails. code names the stage: download-failed, load-failed, transcribe-failed, stream-failed.

WhistleVoiceProvider

On-device Whistle speech-to-text provider.

Captures microphone audio, segments it on pauses, and transcribes each segment locally with the 16.9 MB Whistle WASM engine.

Functions

createProvider(config)

Creates a WhistleVoiceProvider instance.

function createProvider(config?: WhistleVoiceConfig): WhistleVoiceProvider

• config — Optional configuration (URLs, language, keywords, VAD tuning).

Returns: A WhistleVoiceProvider running speech-to-text on-device.

loadWhistleEngine(source, onProgress)

Loads (or returns the cached) Whistle engine: one wasm instance per source combination, shared by every provider in the process. A failed load is evicted so the next call retries.

function loadWhistleEngine(
  source: WhistleEngineSource,
  onProgress?: (event: ModelProgressEvent) => void,
): Promise<WhistleEngine>

• source — Where the engine files come from.
• onProgress — Download/initialization progress sink.

Returns: The ready engine.

normalizeKeywords(keywords)

Normalizes keywords: an array becomes newline-separated (the engine's format); a string is passed through.

function normalizeKeywords(keywords: string | string[] | null | undefined): string | null

• keywords — Keywords as array or newline-separated string.

Returns: The newline-separated string, or null when absent/empty.

supportsRecognitionLanguage(language)

Checks whether Whistle can transcribe the given language. Whistle covers exactly seven languages; use this at wiring time to pick between this provider and a broader one (@molecule/app-ai-voice-whisper, ~99 languages).

function supportsRecognitionLanguage(language: string): boolean

• language — BCP-47 language tag (e.g. 'en-US', 'pl').

Returns: True when Whistle covers the language's base code.

Constants

DEFAULT_WHISTLE_ENGINE_URL

Default URL of the needle.js WASM glue (Hugging Face mirror).

const DEFAULT_WHISTLE_ENGINE_URL: 'https://huggingface.co/Cactus-Compute/needle3/resolve/main/wasm/needle.js'

DEFAULT_WHISTLE_WEIGHTS_URL

Default URL of the whistle.cact weights (16.9 MB, Hugging Face).

const DEFAULT_WHISTLE_WEIGHTS_URL: 'https://huggingface.co/Cactus-Compute/whistle/resolve/main/whistle.cact'

provider

The provider implementation — the fleet-standard typed provider const.

Wire it once at startup: setProvider(provider) from @molecule/app-ai-voice. It is a lazy proxy: construction is deferred to the first property access, so importing this module never throws and needs no config up front. Use createProvider(config) instead when you need custom URLs, keyword biasing or VAD tuning.

const provider: AIVoiceProvider

WHISTLE_LANGUAGES

Whistle's full language list, as ISO 639-1 codes.

const WHISTLE_LANGUAGES: readonly WhistleLanguage[]

WHISTLE_MAX_CLIP_SECONDS

Longest clip one needle_transcribe pass accepts.

const WHISTLE_MAX_CLIP_SECONDS: 30

WHISTLE_SAMPLE_RATE

Sample rate Whistle transcribes (16 kHz mono float PCM).

const WHISTLE_SAMPLE_RATE: 16000

Core Interface

Implements @molecule/app-ai-voice interface.

Bond Wiring

Setup function to register this provider with the core interface:

import { setProvider } from '@molecule/app-ai-voice'
import { provider } from '@molecule/app-ai-voice-whistle'

export function setupAiVoiceWhistle(): void {
  setProvider(provider)
}

Injection Notes

Requirements

Peer dependencies:

• @molecule/app-ai-voice ^1.0.1

Runtime Dependencies

• @molecule/app-ai-voice

The first use downloads the engine and weights (~18 MB total: ~1 MB WASM + 16.9 MB whistle.cact; cached by the browser afterwards) — wire onModelProgress to a visible indicator. Whistle transcribes EXACTLY seven languages (en de fr es it nl pl); for anything else wire @molecule/app-ai-voice-whisper (~99 languages, larger download) or @molecule/app-ai-voice-parakeet (25 languages, needs WebGPU for its fast path) — check with supportsRecognitionLanguage() before wiring. Transcripts arrive as final chunks after each pause (no interim results), and each chunk is at most 30 s (Whistle's per-pass window). Silence or steady noise transcribes to an EMPTY transcript — it is never reported as an error or a detected language. Keyword biasing (keywords) deliberately CHANGES transcripts toward the given words: pass identifiers you care about, not a big "helpful" list. The engine is process-global and not thread-safe — one wasm instance is shared by every provider in the page, and transcription blocks the main thread for the duration of a pass (tens to low hundreds of ms for dictation-length chunks). Under a Content-Security-Policy, self-host needle.js, needle.wasm and whistle.cact same-origin and pass engineUrl/weightsUrl pointing at them. Text-to-speech delegates to the browser's SpeechSynthesis API.

E2E Tests

Integration checklist — drive the real UI (live preview, no mocks), adapt each item to this app's actual screens/flows, and check every box off one by one. A box you can't check is an integration bug to fix — not a skip:

• Pressing the mic / press-to-talk control calls startListening() and speech appears as a live transcript in the UI — interim VoiceTranscriptEvent updates (isFinal: false) refresh the text as you speak, and the final one (isFinal: true) commits the recognized text via onTranscript.
• Stopping (stopListening()) halts recognition cleanly: the transcript stops updating, the mic control returns to idle, and no stray final result fires afterward.
• Denying mic permission (or unavailable hardware) fires onError with a VoiceErrorEvent (code: 'not-allowed') and shows a visible message — the mic control never sits as a silent dead button.
• The app's text-to-speech action calls speak(text, ...) and you actually hear the given text; the chosen VoiceDescriptor / VoiceSynthesisOptions are honored (voice, language, and rate change the audible output), and stopSpeaking() cuts it off.
• The recognition VoiceRecognitionOptions language is respected — setting it to a non-default locale (e.g. 'fr-FR') recognizes in that language rather than always defaulting to English.
• A visible listening/speaking indicator tracks getState() / onStateChange — it reads 'listening' while the mic is open and 'speaking' during synthesis, and returns to 'idle' when each ends.
• Voice UI is feature-gated on isRecognitionSupported() / isSynthesisSupported() (and getAvailableVoices() is awaited, not read synchronously) so an unsupported browser hides the control instead of throwing.
• Microphone access is requested only from a user gesture, its denial is handled gracefully, and captured audio/transcripts stay within the session — nothing is logged or sent anywhere the app didn't intend.