@molecule/api-ai-speech-audio8

Audio8-ASR-Infinite provider for @molecule/api-ai-speech.

Low-latency streaming speech-to-text for Chinese and English on your own GPU (Edge0's 4B Audio8-ASR-Infinite, Apache-2.0 code and weights). A rolling KV cache lets one stream run for any length at constant memory and latency. Implements transcribeStream and a buffer transcribe built on it.

Auto-generated, AI-first package reference for the molecule.dev ecosystem. It is written to be read by coding agents as much as by people, and is generated from this package's source — edit src/index.ts JSDoc, not this file.

Quick Start

import { requireProvider, setProvider } from '@molecule/api-ai-speech'
import { provider } from '@molecule/api-ai-speech-audio8'

// Reads AUDIO8_REALTIME_URL (default ws://127.0.0.1:18191/v1/realtime) per call.
setProvider(provider)

const speech = requireProvider()
// micChunks: AsyncIterable<Uint8Array> of PCM16 mono audio forwarded by your API.
for await (const event of speech.transcribeStream!(micChunks, {
  sampleRate: 48000,
  language: 'zh',
})) {
  if (event.type === 'partial') process.stdout.write(event.text)
  if (event.type === 'final') console.log('\n' + event.text)
}

Type

provider

Installation

npm install @molecule/api-ai-speech-audio8 @molecule/api-ai-speech @molecule/api-secrets

API

Interfaces

DecodedWav

Decoded PCM16 audio.

interface DecodedWav {
  /** Sample rate of the decoded audio in Hz. */
  sampleRate: number
  /** Number of interleaved channels in the source file. */
  channels: number
  /** Mono PCM16 samples (multi-channel input is averaged down). */
  samples: Int16Array
}

RealtimeSocket

An open (or opening) realtime socket.

interface RealtimeSocket {
  /**
   * Send a text or binary frame.
   *
   * @param data - JSON text or raw bytes.
   */
  send(data: string | Uint8Array): void
  /**
   * Close the socket.
   *
   * @param code - Close code (default 1000).
   * @param reason - Close reason.
   */
  close(code?: number, reason?: string): void
}

RealtimeSocketHandlers

Callbacks a RealtimeSocketFactory reports socket activity to.

interface RealtimeSocketHandlers {
  /** The connection is open. */
  onOpen(): void
  /** A frame arrived (text frames as strings, binary frames as bytes). */
  onMessage(data: string | Uint8Array): void
  /** The connection closed. */
  onClose(code: number, reason: string): void
  /** The connection failed. */
  onError(message: string): void
}

Audio8SpeechConfig

Configuration for the Audio8-ASR-Infinite speech provider. Every field falls back to its env var, read on each call (never at import time).

interface Audio8SpeechConfig {
  /**
   * Realtime WebSocket URL. Defaults to the `AUDIO8_REALTIME_URL` env var, then
   * `ws://127.0.0.1:18191/v1/realtime` (the port the Docker Compose file publishes).
   */
  realtimeUrl?: string
  /**
   * Transcription delay in ms: 240–560 and a whole multiple of the server's
   * audio clock (80, 120 or 160 ms). Defaults to `AUDIO8_TARGET_DELAY_MS`, then 480.
   */
  targetDelayMs?: number
  /** Language used when a call passes none. Defaults to `'en'`. */
  defaultLanguage?: Audio8Language
  /** Model name sent in `session.update`. Defaults to `'audio8-asr-infinite'`. */
  model?: string
  /** Audio block size in ms sent per `input_audio_buffer.append`. Defaults to 100. */
  chunkMs?: number
  /**
   * How long a stream waits, after the audio ends, for `transcription.done`
   * before it gives up. Defaults to 30000 ms.
   */
  finalTimeoutMs?: number
  /** WebSocket factory. Defaults to the runtime's global `WebSocket`. */
  socketFactory?: RealtimeSocketFactory
}

Types

RealtimeSocketFactory

Opens a realtime socket.

type RealtimeSocketFactory = (
  url: string,
  headers: Record<string, string>,
  handlers: RealtimeSocketHandlers,
) => RealtimeSocket

• url — ws:// or wss:// URL.
• headers — Extra handshake headers (e.g. Authorization).
• handlers — Activity callbacks.

Returns: The socket.

Audio8Language

Languages Audio8-ASR-Infinite accepts.

type Audio8Language = 'zh' | 'en'

Classes

Pcm16Resampler

Streaming PCM16 little-endian resampler (linear interpolation). Keeps the state a chunked stream needs: an odd trailing byte and the fractional read position carry over to the next chunk, so chunk boundaries add no clicks.

class Pcm16Resampler {
  private readonly step: number
  private tail = new Int16Array(0)
  private position = 0
  private carry: number | null = null

  /**
   * Create a resampler.
   *
   * @param fromRate - Input sample rate in Hz.
   * @param toRate - Output sample rate in Hz.
   */
  constructor(
    private readonly fromRate: number,
    private readonly toRate: number,
  ) {
    if (!Number.isFinite(fromRate) || fromRate <= 0 || !Number.isFinite(toRate) || toRate <= 0) {
      throw new Error(`Invalid sample rate: ${fromRate} → ${toRate}`)
    }
    this.step = fromRate / toRate
  }

  /**
   * Resample one chunk of PCM16 LE bytes.
   *
   * @param chunk - Raw bytes (may split a sample; the odd byte is carried).
   * @returns Resampled PCM16 LE bytes (possibly empty).
   */
  push(chunk: Uint8Array): Uint8Array {
    return samplesToBytes(this.pushSamples(this.decode(chunk)))
  }

  /**
   * Emit whatever the last chunk left pending.
   *
   * @returns Remaining PCM16 LE bytes.
   */
  flush(): Uint8Array {
    return samplesToBytes(this.flushSamples())
  }

  /**
   * Resample decoded samples.
   *
   * @param samples - Mono samples.
   * @returns Resampled samples.
   */
  pushSamples(samples: Int16Array): Int16Array {
    if (this.fromRate === this.toRate) return samples
    const buffer = new Int16Array(this.tail.length + samples.length)
    buffer.set(this.tail)
    buffer.set(samples, this.tail.length)
    const out: number[] = []
    for (;;) {
      const index = Math.floor(this.position)
      if (index + 1 >= buffer.length) break
      const fraction = this.position - index
      out.push(Math.round(buffer[index] * (1 - fraction) + buffer[index + 1] * fraction))
      this.position += this.step
    }
    const drop = Math.min(Math.floor(this.position), Math.max(buffer.length - 1, 0))
    this.tail = buffer.slice(drop)
    this.position -= drop
    return Int16Array.from(out)
  }

  /**
   * Emit the final samples held back for interpolation.
   *
   * @returns Remaining samples.
   */
  flushSamples(): Int16Array {
    if (this.fromRate === this.toRate) return new Int16Array(0)
    const out: number[] = []
    while (this.tail.length > 0 && this.position <= this.tail.length - 1) {
      out.push(this.tail[Math.floor(this.position)])
      this.position += this.step
    }
    this.tail = new Int16Array(0)
    this.position = 0
    return Int16Array.from(out)
  }

  /**
   * Decode LE bytes into samples, carrying an odd trailing byte.
   *
   * @param chunk - Raw bytes.
   * @returns Decoded samples.
   */
  private decode(chunk: Uint8Array): Int16Array {
    let bytes = chunk
    if (this.carry !== null) {
      bytes = new Uint8Array(chunk.length + 1)
      bytes[0] = this.carry
      bytes.set(chunk, 1)
      this.carry = null
    }
    if (bytes.length % 2 === 1) {
      this.carry = bytes[bytes.length - 1]
      bytes = bytes.subarray(0, bytes.length - 1)
    }
    const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength)
    const samples = new Int16Array(bytes.length / 2)
    for (let i = 0; i < samples.length; i++) samples[i] = view.getInt16(i * 2, true)
    return samples
  }
}

Audio8SpeechError

Error thrown when an Audio8 stream fails during a buffer transcribe. status is the WebSocket close code (0 when the failure was not a close); code is the server's error code when it sent one. Deliberately NOT tagged with statusCode/errorKey, so API middleware answers its generic 500.

class Audio8SpeechError extends Error {
  /** WebSocket close code, or 0. */
  readonly status: number
  /** Server error code, when present. */
  readonly code: string | undefined

  /**
   * Create the error.
   *
   * @param message - Human-readable message.
   * @param status - WebSocket close code, or 0.
   * @param code - Server error code, when present.
   */
  constructor(message: string, status = 0, code?: string) {
    super(message)
    this.name = 'Audio8SpeechError'
    this.status = status
    this.code = code
  }
}

AsyncQueue

A single-consumer async queue: producers push, the consumer iterates, and end() finishes the iteration once everything pushed has been read.

class AsyncQueue<T> implements AsyncIterable<T> {
  private readonly items: T[] = []
  private waiting: ((result: IteratorResult<T>) => void) | null = null
  private ended = false

  /**
   * Add an item.
   *
   * @param item - The item (ignored after `end()`).
   */
  push(item: T): void {
    if (this.ended) return
    if (this.waiting) {
      const resolve = this.waiting
      this.waiting = null
      resolve({ value: item, done: false })
    } else {
      this.items.push(item)
    }
  }

  /** Finish the queue; the consumer drains what is left, then stops. */
  end(): void {
    if (this.ended) return
    this.ended = true
    if (this.waiting) {
      const resolve = this.waiting
      this.waiting = null
      resolve({ value: undefined, done: true })
    }
  }

  /**
   * Whether `end()` has been called.
   *
   * @returns True once ended.
   */
  get isEnded(): boolean {
    return this.ended
  }

  /**
   * Iterate the queue.
   *
   * @returns An async iterator over pushed items.
   */
  [Symbol.asyncIterator](): AsyncIterator<T> {
    return {
      next: (): Promise<IteratorResult<T>> => {
        if (this.items.length > 0) {
          return Promise.resolve({ value: this.items.shift() as T, done: false })
        }
        if (this.ended) return Promise.resolve({ value: undefined, done: true })
        return new Promise((resolve) => {
          this.waiting = resolve
        })
      },
    }
  }
}

Functions

isWav(bytes)

Checks whether the bytes start with a RIFF/WAVE header.

function isWav(bytes: Uint8Array<ArrayBufferLike>): boolean

• bytes — Audio bytes.

Returns: True for a WAV container.

decodeWav(bytes)

Decodes an uncompressed 16-bit PCM WAV file to mono samples.

function decodeWav(bytes: Uint8Array<ArrayBufferLike>): DecodedWav

• bytes — The WAV file bytes.

Returns: The sample rate, source channel count and mono samples.

resample(samples, fromRate, toRate)

Resamples mono PCM16 with linear interpolation.

function resample(
  samples: Int16Array<ArrayBufferLike>,
  fromRate: number,
  toRate: number,
): Int16Array<ArrayBufferLike>

• samples — Mono input samples.
• fromRate — Input sample rate in Hz.
• toRate — Output sample rate in Hz.

Returns: Resampled samples (the input itself when the rates match).

encodeWav(samples, sampleRate)

Encodes mono PCM16 samples as a WAV file.

function encodeWav(
  samples: Int16Array<ArrayBufferLike>,
  sampleRate: number,
): Uint8Array<ArrayBufferLike>

• samples — Mono samples.
• sampleRate — Sample rate in Hz.

Returns: The WAV file bytes.

toTargetWav(bytes)

Converts any 16-bit PCM WAV into a 16 kHz mono WAV.

function toTargetWav(bytes: Uint8Array<ArrayBufferLike>): Uint8Array<ArrayBufferLike>

• bytes — The source WAV bytes.

Returns: A 16 kHz mono 16-bit PCM WAV.

wavToTargetPcm(bytes)

Decodes a WAV file into 16 kHz mono PCM16 little-endian bytes.

function wavToTargetPcm(bytes: Uint8Array<ArrayBufferLike>): Uint8Array<ArrayBufferLike>

• bytes — The source WAV bytes.

Returns: Raw 16 kHz mono PCM16 LE bytes.

samplesToBytes(samples)

Serializes samples as PCM16 little-endian bytes.

function samplesToBytes(samples: Int16Array<ArrayBufferLike>): Uint8Array<ArrayBufferLike>

• samples — PCM16 samples.

Returns: Little-endian bytes.

toTargetBlocks(audio, sampleRate, blockBytes)

Converts a PCM16 LE stream at any sample rate to 16 kHz and re-chunks it into fixed-size blocks (the last block may be shorter).

function toTargetBlocks(
  audio: AsyncIterable<Uint8Array<ArrayBufferLike>>,
  sampleRate: number,
  blockBytes: number,
): AsyncGenerator<Uint8Array<ArrayBufferLike>, any, any>

• audio — Raw PCM16 LE mono chunks.
• sampleRate — Input sample rate in Hz.
• blockBytes — Output block size in bytes (even).

toBase64(bytes)

Encodes bytes as base64.

function toBase64(bytes: Uint8Array<ArrayBufferLike>): string

• bytes — Raw bytes.

Returns: Base64 text.

toAudio8Language(language, fallback)

Maps a language code to one Audio8 accepts.

function toAudio8Language(language: string | undefined, fallback: Audio8Language): Audio8Language

• language — An ISO 639-1 code or BCP-47 tag (zh, zh-CN, en-US).
• fallback — Language to use when none is given.

Returns: 'zh' or 'en'.

assertTargetDelay(delayMs)

Validates the transcription delay.

function assertTargetDelay(delayMs: number): number

• delayMs — Delay in ms.

Returns: The delay.

createProvider(config)

Creates an Audio8-ASR-Infinite speech provider instance.

function createProvider(config?: Audio8SpeechConfig): AISpeechProvider

• config — Optional overrides (URL, delay, language, socket factory).

Returns: An AISpeechProvider backed by a self-hosted Audio8 deployment.

openGlobalWebSocket(url, headers, handlers)

Default socket factory over the runtime's global WebSocket. Handshake headers use the Node (undici) WebSocketInit.headers extension.

function openGlobalWebSocket(
  url: string,
  headers: Record<string, string>,
  handlers: RealtimeSocketHandlers,
): {
  send: (data: string | Uint8Array<ArrayBufferLike>) => void
  close: (code: number | undefined, reason: string | undefined) => void
}

• url — ws:// or wss:// URL.
• headers — Extra handshake headers.
• handlers — Activity callbacks.

Returns: The socket.

toWebSocketUrl(baseUrl)

Turns an http(s):// base URL into its ws(s):// equivalent.

function toWebSocketUrl(baseUrl: string): string

• baseUrl — HTTP base URL.

Returns: The WebSocket base URL.

parseFrame(data)

Decodes a text frame as JSON.

function parseFrame(data: string | Uint8Array<ArrayBufferLike>): Record<string, unknown> | null

• data — The frame.

Returns: The parsed object, or null for binary/non-JSON frames.

Constants

TARGET_SAMPLE_RATE

The sample rate every self-hosted speech server in this family expects.

const TARGET_SAMPLE_RATE: 16000

DEFAULT_AUDIO8_REALTIME_URL

Default realtime URL — the port the Audio8 Docker Compose file publishes.

const DEFAULT_AUDIO8_REALTIME_URL: 'ws://127.0.0.1:18191/v1/realtime'

DEFAULT_TARGET_DELAY_MS

Default transcription delay in ms (the model card's benchmark setting).

const DEFAULT_TARGET_DELAY_MS: 480

DEFAULT_AUDIO8_MODEL

Model name the reference client sends in session.update.

const DEFAULT_AUDIO8_MODEL: 'audio8-asr-infinite'

DEFAULT_FINAL_TIMEOUT_MS

Default wait for transcription.done after the audio ends.

const DEFAULT_FINAL_TIMEOUT_MS: 30000

provider

The provider implementation.

const provider: AISpeechProvider

aiSpeechAudio8SecretDefinitions

Secret definitions used by the Audio8-ASR-Infinite speech bond.

const aiSpeechAudio8SecretDefinitions: SecretDefinition[]

Core Interface

Implements @molecule/api-ai-speech interface.

Bond Wiring

Setup function to register this provider with the core interface:

import { setProvider } from '@molecule/api-ai-speech'
import { provider } from '@molecule/api-ai-speech-audio8'

export function setupAiSpeechAudio8(): void {
  setProvider(provider)
}

Injection Notes

Requirements

Peer dependencies:

• @molecule/api-ai-speech >=1.1.0
• @molecule/api-secrets ^1.0.1

Environment Variables

• AUDIO8_REALTIME_URL (optional) — Audio8-ASR-Infinite realtime URL
  • Setup: The WebSocket URL of your own Audio8 Docker Compose deployment. Defaults to ws://127.0.0.1:18191/v1/realtime.
  • Get it here: https://github.com/Edge0-AI/Audio8-ASR-Infinite
  • Example: ws://127.0.0.1:18191/v1/realtime
• AUDIO8_TARGET_DELAY_MS (optional) — Audio8-ASR-Infinite transcription delay
  • Setup: Milliseconds of look-ahead, 240–560 and a whole multiple of the server audio clock (80/120/160 ms). Defaults to 480.
  • Get it here: https://github.com/Edge0-AI/Audio8-ASR-Infinite
  • Example: 480

Runtime Dependencies

• @molecule/api-ai-speech

• @molecule/api-secrets

• Chinese and English only. Any other language throws before connecting (zh-CN/en-US map to zh/en); omitted means 'en' unless createProvider({ defaultLanguage: 'zh' }). English accuracy trails dedicated English models — pick it for zh/en voice agents.

• You run the server, on a CUDA GPU. cd docker && AUDIO8_MODEL_DIR=/path/to/checkpoint docker compose up -d; the compose file publishes the WebSocket on 18191 (the service listens on 18190 inside the container, which is the reference client's default). Only a full merged weight directory loads — adapter-style or partially converted weights do not. It does not fit a molecule sandbox.

• No auth exists. The server documents none, so anyone who can reach the port uses your GPU: keep it on a private network or behind an authenticating reverse proxy, and never expose it publicly.

• Server-side only. A browser microphone streams to YOUR API, which calls transcribeStream; never connect the browser to the model host.

• Finality: partial increments, then ONE final. Deltas are provisional (streamingAppendOnly: false) and final carries the full text of the whole stream (transcription.done) after the audio ends — do not wait for a per-sentence final. There is no end-of-turn event: the semantic-VAD head is not surfaced by the server's protocol, so no turn-end is emitted. No word timestamps, no speakers (diarize is ignored).

• Audio: send PCM16 LE mono at sampleRate; the bond resamples to 16 kHz and sends 100 ms base64 blocks. transcribe accepts 16-bit PCM WAV only and throws for m4a/webm/mp3.

• AUDIO8_TARGET_DELAY_MS (default 480) must be 240–560 and a multiple of the server's audio clock (80, 120 or 160 ms) — others throw before connecting. Lower delay = faster text, lower accuracy.

• The bond needs a global WebSocket (Node 22+) or a socketFactory. A failed buffer transcribe throws Audio8SpeechError (status = close code or 0); streams report failures as an error event and end.

• Benchmarks on the model card are the vendor's own (greedy decoding, 80 ms clock, 480 ms delay).

E2E Tests

Integration checklist — drive the real UI (live preview, no mocks), adapt each item to this app's actual screens/flows, and check every box off one by one. A box you can't check is an integration bug to fix — not a skip. The wired provider implements a subset (TTS, STT, or both) — feature-detect and run only the direction(s) it actually exposes:

• TTS: each read-aloud / narration / voice-note flow the app defines returns real PLAYABLE audio — the response carries an audio Content-Type (e.g. audio/mpeg) and non-trivial bytes (not a 0-byte file, not a JSON-encoded blob), and the UI's <audio>/player actually plays it. The spoken audio reflects the input text you sent.
• TTS: synthesizing two different strings yields two audibly different clips (distinct byte payloads) — proves the text drives synthesis and you are not hearing a cached/placeholder file.
• TTS: any exposed voice/format/speed option takes effect — switching voice (voice/voiceId) or output format (responseFormat/ outputFormat) changes the returned audio (audible voice, Content-Type, or file extension); an unknown voice/format surfaces a visible error rather than silently falling back.
• STT: transcribing a KNOWN spoken clip returns text that MATCHES the words spoken (not empty, not garbage) and renders in the UI; translating a non-English clip returns English text. Any timestamps the UI shows line up with the audio.
• STT: a silent / empty / too-short clip is handled gracefully — an empty or "no speech detected" result shown in the UI, never a crash or a spinner that hangs forever.
• The audio artifact is served from the app's OWN origin (a stored upload or a route the app serves), NOT a raw expiring provider URL — copy the audio src, reload it later, and confirm it still plays.
• A provider error (bad key, quota, unsupported format/voice, or a capability the wired provider lacks) surfaces as a readable message in the UI, not an unhandled crash or a blank/broken player.
• Synthesis/transcription runs SERVER-SIDE and authorized: the provider key never reaches the browser (check the network tab / client bundle), and no unauthenticated or unbounded endpoint lets a caller fire arbitrary synthesize/transcribe calls — each is billed per character / per minute of audio, so an open endpoint is a cost-DoS vector.