@molecule/app-ai-voice-parakeet

On-device NVIDIA Parakeet voice provider using parakeet.js.

Leaderboard-topping speech-to-text that runs entirely in the browser (WebGPU encoder + WASM decoder) — no cloud speech service and no Web Speech API backend required. Works in Brave, ungoogled Chromium, and Firefox, where the native SpeechRecognition API is a non-functional stub.

Auto-generated, AI-first package reference for the molecule.dev ecosystem. It is written to be read by coding agents as much as by people, and is generated from this package's source — edit src/index.ts JSDoc, not this file.

Quick Start

import { setProvider } from '@molecule/app-ai-voice'
import { createProvider, supportsRecognitionLanguage } from '@molecule/app-ai-voice-parakeet'

if (await supportsRecognitionLanguage(navigator.language)) {
  setProvider(
    createProvider({
      onModelProgress: (e) => console.log(e.status, e.progress),
    }),
  )
}

Type

provider

Installation

npm install @molecule/app-ai-voice-parakeet @molecule/app-ai-voice parakeet.js

API

Interfaces

`ModelProgressEvent`

Progress event emitted while the speech model downloads/initializes.

interface ModelProgressEvent {
  /** Lifecycle stage of the model load. */
  status: 'downloading' | 'loading' | 'ready' | 'error'
  /** Overall download progress from 0 to 100, when known. */
  progress?: number
  /** The file currently being fetched, when known. */
  file?: string
}

`ParakeetVoiceConfig`

Configuration for the on-device Parakeet voice provider.

interface ParakeetVoiceConfig {
  /**
   * parakeet.js model key or Hugging Face repo id. Default:
   * 'parakeet-tdt-0.6b-v3' (multilingual). 'parakeet-tdt-0.6b-v2' is
   * English-only but slightly smaller/faster.
   */
  model?: string
  /**
   * Inference backend. 'auto' (default) picks WebGPU when available and
   * falls back to WASM.
   */
  backend?: 'auto' | 'webgpu' | 'wasm'
  /**
   * URL prefix (directory) the ONNX Runtime WASM/JS runtime files are served
   * from — e.g. '/ort/'. Without it, parakeet.js loads the runtime from the
   * jsdelivr CDN, which any app with a `script-src 'self'` CSP (correctly)
   * blocks. Serve the `ort-wasm-simd-threaded*` files from parakeet.js's own
   * onnxruntime-web dependency (version must match). Applied through the
   * lib's global ort instance with one retry, because parakeet.js's own
   * wasmPaths option is ignored upstream.
   */
  wasmPaths?: string
  /** Encoder weight quantization. Default 'int8' (smallest download). */
  encoderQuant?: 'int8' | 'fp16' | 'fp32'
  /** Decoder weight quantization. Default 'int8'. */
  decoderQuant?: 'int8' | 'fp16' | 'fp32'
  /**
   * Called with model download/initialization progress — wire this to a UI
   * indicator, since the first use downloads the model (hundreds of MB for
   * the 0.6B models; cached by the browser afterwards).
   */
  onModelProgress?: (event: ModelProgressEvent) => void
  /**
   * RMS amplitude above which a frame counts as speech. Default 0.01.
   */
  speechThreshold?: number
  /**
   * Milliseconds of silence after speech that closes a chunk and sends it
   * for transcription. Default 800.
   */
  silenceMs?: number
  /**
   * Hard cap on a single chunk's length in seconds — a chunk is flushed at
   * this size even without a pause. Default 15.
   */
  maxChunkSeconds?: number
}

Classes

`ParakeetVoiceProvider`

On-device NVIDIA Parakeet speech-to-text provider.

Captures microphone audio, segments it on pauses, and transcribes each segment locally with a parakeet.js ONNX model.

class ParakeetVoiceProvider implements AIVoiceProvider {
  readonly name = 'parakeet'

  private state: VoiceState = 'idle'
  private config: ParakeetVoiceConfig
  private handlers: VoiceEventHandlers = {}
  private disposed = false

  private modelPromise: Promise<ParakeetModelLike> | null = null

  private mediaStream: MediaStream | null = null
  private audioContext: AudioContext | null = null
  private processor: ScriptProcessorNode | null = null
  private sourceNode: MediaStreamAudioSourceNode | null = null

  private listening = false

  // VAD/chunking state
  private speechFrames: Float32Array[] = []
  private inSpeech = false
  private silentSamples = 0
  private speechSamples = 0

  // Serialized transcription queue
  private transcribeQueue: Float32Array[] = []
  private transcribing = false

  /**
   * Creates a new ParakeetVoiceProvider.
   * @param config - Provider configuration (model, backend, VAD tuning).
   */
  constructor(config: ParakeetVoiceConfig = {}) {
    this.config = config
  }

  /**
   * Loads (or returns the cached) parakeet.js model, reporting download
   * progress through `config.onModelProgress`.
   * @returns The ready-to-use Parakeet model.
   */
  private loadModel(): Promise<ParakeetModelLike> {
    if (this.modelPromise) return this.modelPromise

    const progress = (event: ModelProgressEvent): void => {
      this.config.onModelProgress?.(event)
    }

    this.modelPromise = (async () => {
      progress({ status: 'loading' })
      const { fromHub } = await import('parakeet.js')
      const backend =
        this.config.backend && this.config.backend !== 'auto'
          ? this.config.backend
          : (await hasUsableWebGpu())
            ? 'webgpu'
            : 'wasm'

      const model = this.config.model ?? DEFAULT_MODEL
      const progressCallback = (p: { loaded: number; total: number; file: string }): void => {
        progress({
          status: 'downloading',
          progress: p.total > 0 ? Math.round((p.loaded / p.total) * 100) : undefined,
          file: p.file,
        })
      }
      // int8 encoders are rejected by the WebGPU execution provider —
      // parakeet.js silently force-upgrades them to fp32, a 2.4 GB download
      // (verified live). fp16 (1.24 GB) is the WebGPU sweet spot; int8
      // (652 MB) works on the WASM path.
      const encoderQuantFor = (b: 'webgpu' | 'wasm'): 'int8' | 'fp16' | 'fp32' =>
        this.config.encoderQuant ?? (b === 'webgpu' ? 'fp16' : 'int8')
      // WebGPU init failures (driver/adapter issues) fall back to WASM.
      const backends: Array<'webgpu' | 'wasm'> =
        backend === 'webgpu' ? ['webgpu', 'wasm'] : ['wasm']

      let parakeet: ParakeetModelLike | null = null
      let lastError: unknown = null
      for (const b of backends) {
        const options = {
          backend: b,
          encoderQuant: encoderQuantFor(b),
          decoderQuant: this.config.decoderQuant ?? ('int8' as const),
          ...(this.config.wasmPaths ? { wasmPaths: this.config.wasmPaths } : {}),
          progress: progressCallback,
        }
        // Up to two tries per backend: parakeet.js's initOrt IGNORES its
        // wasmPaths option (upstream bug) and defaults the ORT runtime to the
        // jsdelivr CDN, which any `script-src 'self'` CSP blocks. The failed
        // first try exposes the lib's ort instance on globalThis — point its
        // env at our same-origin files there and retry.
        for (let attempt = 0; attempt < 2 && !parakeet; attempt++) {
          try {
            parakeet = (await fromHub(model, options)) as unknown as ParakeetModelLike
          } catch (error) {
            lastError = error
            if (!this.applyGlobalWasmPaths()) break
          }
        }
        if (parakeet) break
      }
      if (!parakeet) {
        progress({ status: 'error' })
        throw lastError
      }
      progress({ status: 'ready' })
      return parakeet
    })()

    this.modelPromise.catch(() => {
      // Allow a retry on the next startListening call; the caller already
      // receives this rejection via startListening's error handling.
      this.modelPromise = null
    })

    return this.modelPromise
  }

  /**
   * Points parakeet.js's own ONNX Runtime instance (exposed on
   * `globalThis.ort` after its first init) at the configured same-origin
   * wasmPaths. Needed because parakeet.js's initOrt ignores the wasmPaths
   * option and we cannot import its NESTED onnxruntime-web copy directly.
   * @returns True when the path was applied (a retry is worthwhile).
   */
  private applyGlobalWasmPaths(): boolean {
    if (!this.config.wasmPaths) return false
    const ort = (globalThis as { ort?: { env?: { wasm?: { wasmPaths?: string } } } }).ort
    if (!ort?.env?.wasm) return false
    ort.env.wasm.wasmPaths = this.config.wasmPaths
    return true
  }

  /**
   * Starts on-device speech recognition: opens the microphone, loads the
   * model if needed, and emits a final transcript after each pause.
   * @param options - Recognition options. `language` is not passed to the
   *   model (Parakeet auto-detects) — check coverage up front with
   *   `supportsRecognitionLanguage()`.
   * @param handlers - Callbacks for transcripts, state changes, and errors.
   */
  startListening(options?: VoiceRecognitionOptions, handlers?: VoiceEventHandlers): void {
    if (this.disposed || this.listening) return
    void options

    this.handlers = handlers ?? {}

    const getUserMedia = getGetUserMedia()
    if (!getUserMedia) {
      this.handlers.onError?.({
        code: 'not-supported',
        message: 'Microphone capture is not supported in this environment',
      })
      return
    }

    this.listening = true
    // 'processing' until the model is ready — consumers use the transition to
    // 'listening' to clear their "preparing dictation" indicator. Speech is
    // captured during the load and transcribed once the model arrives.
    this.setState('processing')

    void (async () => {
      try {
        // Kick off the model load and mic open in parallel — the model can
        // take a while on first use and the mic prompt needs to appear now.
        const modelReady = this.loadModel()
        const stream = await getUserMedia({
          audio: {
            channelCount: 1,
            echoCancellation: true,
            noiseSuppression: true,
            autoGainControl: true,
          },
        })
        if (!this.listening || this.disposed) {
          for (const track of stream.getTracks()) track.stop()
          return
        }
        this.mediaStream = stream

        const context = new AudioContext({ sampleRate: MODEL_SAMPLE_RATE })
        this.audioContext = context
        this.sourceNode = context.createMediaStreamSource(stream)
        // ScriptProcessorNode is deprecated but has no replacement that works
        // without shipping a separate AudioWorklet module file, which npm
        // consumers' bundlers would each need to handle. Revisit when
        // AudioWorklet.addModule accepts inline modules everywhere.
        this.processor = context.createScriptProcessor(FRAME_SIZE, 1, 1)
        this.processor.onaudioprocess = (event) => {
          this.handleFrame(event.inputBuffer.getChannelData(0), context.sampleRate)
        }
        this.sourceNode.connect(this.processor)
        this.processor.connect(context.destination)

        await modelReady
        if (this.listening && !this.disposed) {
          this.setState('listening')
        }
      } catch (error) {
        this.teardownCapture()
        this.listening = false
        this.setState('error')
        const isPermission = error instanceof DOMException && error.name === 'NotAllowedError'
        this.handlers.onError?.({
          code: isPermission ? 'not-allowed' : 'start-failed',
          message: error instanceof Error ? error.message : 'Failed to start speech recognition',
        })
      }
    })()
  }

  /**
// … truncated — the full declaration is in node_modules/@molecule/app-ai-voice-parakeet/dist/provider.d.ts

Functions

`supportsRecognitionLanguage(language, model)`

Checks whether a Parakeet model can transcribe the given BCP-47 language. Use this at wiring time to decide between this provider and a multilingual Whisper provider.

function supportsRecognitionLanguage(language: string, model?: string): Promise<boolean>

language — BCP-47 language tag (e.g. 'en-US', 'ja').
model — parakeet.js model key or repo id. Defaults to the provider's default model.

Returns: A promise resolving to true when the model covers the language.

`createProvider(config)`

Creates a ParakeetVoiceProvider instance.

function createProvider(config?: ParakeetVoiceConfig): ParakeetVoiceProvider

config — Optional configuration (model, backend, VAD tuning).

Returns: A ParakeetVoiceProvider running speech-to-text on-device.

Constants

`provider`

The provider implementation — the fleet-standard typed provider const.

Wire it once at startup: setProvider(provider) from @molecule/app-ai-voice. It is a lazy proxy: construction is deferred to the first property access, so importing this module never throws and needs no config up front. Use createProvider(config) instead when you need a custom model, backend, or VAD tuning.

const provider: AIVoiceProvider

Core Interface

Implements @molecule/app-ai-voice interface.

Bond Wiring

Setup function to register this provider with the core interface:

import { setProvider } from '@molecule/app-ai-voice'
import { provider } from '@molecule/app-ai-voice-parakeet'

export function setupAiVoiceParakeet(): void {
  setProvider(provider)
}

Injection Notes

Requirements

Peer dependencies:

@molecule/app-ai-voice ^1.0.1

Runtime Dependencies

@molecule/app-ai-voice
parakeet.js

The first use downloads the model (hundreds of MB at int8, cached by the browser afterwards) — always wire onModelProgress to a visible indicator. The default parakeet-tdt-0.6b-v3 model covers en, fr, de, es, it, pt, nl, pl, ru, uk, ja, ko and zh with automatic language detection; for other languages wire @molecule/app-ai-voice-whisper instead (smaller download, broader language coverage, lower accuracy). Transcripts arrive as final chunks after each pause; there are no interim results.

E2E Tests

Integration checklist — drive the real UI (live preview, no mocks), adapt each item to this app's actual screens/flows, and check every box off one by one. A box you can't check is an integration bug to fix — not a skip:

Pressing the mic / press-to-talk control calls startListening() and speech appears as a live transcript in the UI — interim VoiceTranscriptEvent updates (isFinal: false) refresh the text as you speak, and the final one (isFinal: true) commits the recognized text via onTranscript.
Stopping (stopListening()) halts recognition cleanly: the transcript stops updating, the mic control returns to idle, and no stray final result fires afterward.
Denying mic permission (or unavailable hardware) fires onError with a VoiceErrorEvent (code: 'not-allowed') and shows a visible message — the mic control never sits as a silent dead button.
The app's text-to-speech action calls speak(text, ...) and you actually hear the given text; the chosen VoiceDescriptor / VoiceSynthesisOptions are honored (voice, language, and rate change the audible output), and stopSpeaking() cuts it off.
The recognition VoiceRecognitionOptions language is respected — setting it to a non-default locale (e.g. 'fr-FR') recognizes in that language rather than always defaulting to English.
A visible listening/speaking indicator tracks getState() / onStateChange — it reads 'listening' while the mic is open and 'speaking' during synthesis, and returns to 'idle' when each ends.
Voice UI is feature-gated on isRecognitionSupported() / isSynthesisSupported() (and getAvailableVoices() is awaited, not read synchronously) so an unsupported browser hides the control instead of throwing.
Microphone access is requested only from a user gesture, its denial is handled gracefully, and captured audio/transcripts stay within the session — nothing is logged or sent anywhere the app didn't intend.

`@molecule/app-ai-voice-parakeet`

How it works

Reference