@molecule/api-ai-speech-nemo-speech

NVIDIA NeMo-Speech.cpp provider for @molecule/api-ai-speech.

Batch and streaming speech-to-text plus speaker diarization ("who spoke when", Nemotron-3-Diarization) on your own nemo-speech serve instance — NVIDIA's native C++ runtime, which runs on CPU, CUDA, Metal and Vulkan. Implements transcribe (with diarize: true speaker labels), diarize and transcribeStream.

Auto-generated, AI-first package reference for the molecule.dev ecosystem. It is written to be read by coding agents as much as by people, and is generated from this package's source — edit src/index.ts JSDoc, not this file.

Quick Start

import { requireProvider, setProvider } from '@molecule/api-ai-speech'
import { provider } from '@molecule/api-ai-speech-nemo-speech'

// Reads NEMO_SPEECH_URL (default http://127.0.0.1:8080) on each call.
setProvider(provider)

const speech = requireProvider()
const result = await speech.transcribe!({ audio: wavBytes, diarize: true })
for (const segment of result.segments ?? []) {
  console.log(`${segment.speaker}: ${segment.text}`)
}

Type

provider

Installation

npm install @molecule/api-ai-speech-nemo-speech @molecule/api-ai-speech @molecule/api-secrets

API

Interfaces

DecodedWav

Decoded PCM16 audio.

interface DecodedWav {
  /** Sample rate of the decoded audio in Hz. */
  sampleRate: number
  /** Number of interleaved channels in the source file. */
  channels: number
  /** Mono PCM16 samples (multi-channel input is averaged down). */
  samples: Int16Array
}

RealtimeSocket

An open (or opening) realtime socket.

interface RealtimeSocket {
  /**
   * Send a text or binary frame.
   *
   * @param data - JSON text or raw bytes.
   */
  send(data: string | Uint8Array): void
  /**
   * Close the socket.
   *
   * @param code - Close code (default 1000).
   * @param reason - Close reason.
   */
  close(code?: number, reason?: string): void
}

RealtimeSocketHandlers

Callbacks a RealtimeSocketFactory reports socket activity to.

interface RealtimeSocketHandlers {
  /** The connection is open. */
  onOpen(): void
  /** A frame arrived (text frames as strings, binary frames as bytes). */
  onMessage(data: string | Uint8Array): void
  /** The connection closed. */
  onClose(code: number, reason: string): void
  /** The connection failed. */
  onError(message: string): void
}

NemoSpeechConfig

Configuration for the NeMo-Speech.cpp speech provider. Every field falls back to its env var, read on each call (never at import time).

interface NemoSpeechConfig {
  /**
   * Server base URL (`nemo-speech serve`), without `/v1`. Defaults to the
   * `NEMO_SPEECH_URL` env var, then `http://127.0.0.1:8080`.
   */
  baseUrl?: string
  /**
   * API key, sent as `Authorization: Bearer …` — only needed when the server
   * was started with `--api-key`. Defaults to `NEMO_SPEECH_API_KEY`.
   */
  apiKey?: string
  /** Model sent as the `model` form field when a call passes none. Omitted by default. */
  defaultModel?: string
  /** Per-request HTTP timeout in milliseconds. Defaults to 300000. */
  timeoutMs?: number
  /**
   * How long a stream waits, after the audio ends, for the server's final
   * `completed` event before it gives up. Defaults to 30000 ms.
   */
  finalTimeoutMs?: number
  /**
   * A raw `session.update` event body sent right after `session.created` on
   * streaming connections (e.g. to turn on streaming diarization). Its field
   * names are defined by the server's `docs/server.md`; the bond sends it
   * verbatim with `type: 'session.update'` added. Omitted by default.
   */
  sessionUpdate?: Record<string, unknown>
  /** WebSocket factory for streaming. Defaults to the runtime's global `WebSocket`. */
  socketFactory?: RealtimeSocketFactory
}

Types

RealtimeSocketFactory

Opens a realtime socket.

type RealtimeSocketFactory = (
  url: string,
  headers: Record<string, string>,
  handlers: RealtimeSocketHandlers,
) => RealtimeSocket

• url — ws:// or wss:// URL.
• headers — Extra handshake headers (e.g. Authorization).
• handlers — Activity callbacks.

Returns: The socket.

Classes

Pcm16Resampler

Streaming PCM16 little-endian resampler (linear interpolation). Keeps the state a chunked stream needs: an odd trailing byte and the fractional read position carry over to the next chunk, so chunk boundaries add no clicks.

class Pcm16Resampler {
  private readonly step: number
  private tail = new Int16Array(0)
  private position = 0
  private carry: number | null = null

  /**
   * Create a resampler.
   *
   * @param fromRate - Input sample rate in Hz.
   * @param toRate - Output sample rate in Hz.
   */
  constructor(
    private readonly fromRate: number,
    private readonly toRate: number,
  ) {
    if (!Number.isFinite(fromRate) || fromRate <= 0 || !Number.isFinite(toRate) || toRate <= 0) {
      throw new Error(`Invalid sample rate: ${fromRate} → ${toRate}`)
    }
    this.step = fromRate / toRate
  }

  /**
   * Resample one chunk of PCM16 LE bytes.
   *
   * @param chunk - Raw bytes (may split a sample; the odd byte is carried).
   * @returns Resampled PCM16 LE bytes (possibly empty).
   */
  push(chunk: Uint8Array): Uint8Array {
    return samplesToBytes(this.pushSamples(this.decode(chunk)))
  }

  /**
   * Emit whatever the last chunk left pending.
   *
   * @returns Remaining PCM16 LE bytes.
   */
  flush(): Uint8Array {
    return samplesToBytes(this.flushSamples())
  }

  /**
   * Resample decoded samples.
   *
   * @param samples - Mono samples.
   * @returns Resampled samples.
   */
  pushSamples(samples: Int16Array): Int16Array {
    if (this.fromRate === this.toRate) return samples
    const buffer = new Int16Array(this.tail.length + samples.length)
    buffer.set(this.tail)
    buffer.set(samples, this.tail.length)
    const out: number[] = []
    for (;;) {
      const index = Math.floor(this.position)
      if (index + 1 >= buffer.length) break
      const fraction = this.position - index
      out.push(Math.round(buffer[index] * (1 - fraction) + buffer[index + 1] * fraction))
      this.position += this.step
    }
    const drop = Math.min(Math.floor(this.position), Math.max(buffer.length - 1, 0))
    this.tail = buffer.slice(drop)
    this.position -= drop
    return Int16Array.from(out)
  }

  /**
   * Emit the final samples held back for interpolation.
   *
   * @returns Remaining samples.
   */
  flushSamples(): Int16Array {
    if (this.fromRate === this.toRate) return new Int16Array(0)
    const out: number[] = []
    while (this.tail.length > 0 && this.position <= this.tail.length - 1) {
      out.push(this.tail[Math.floor(this.position)])
      this.position += this.step
    }
    this.tail = new Int16Array(0)
    this.position = 0
    return Int16Array.from(out)
  }

  /**
   * Decode LE bytes into samples, carrying an odd trailing byte.
   *
   * @param chunk - Raw bytes.
   * @returns Decoded samples.
   */
  private decode(chunk: Uint8Array): Int16Array {
    let bytes = chunk
    if (this.carry !== null) {
      bytes = new Uint8Array(chunk.length + 1)
      bytes[0] = this.carry
      bytes.set(chunk, 1)
      this.carry = null
    }
    if (bytes.length % 2 === 1) {
      this.carry = bytes[bytes.length - 1]
      bytes = bytes.subarray(0, bytes.length - 1)
    }
    const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength)
    const samples = new Int16Array(bytes.length / 2)
    for (let i = 0; i < samples.length; i++) samples[i] = view.getInt16(i * 2, true)
    return samples
  }
}

NemoSpeechError

Error thrown when the NeMo-Speech.cpp server refuses or fails a request. Carries the HTTP status (0 when the server could not be reached or timed out) and the server's error type as code. Deliberately NOT tagged with statusCode/errorKey, so API middleware answers its generic 500 instead of echoing the upstream status to the caller.

class NemoSpeechError extends Error {
  /** HTTP status from the server; 0 for a network failure or timeout. */
  readonly status: number
  /** The server's error `type` (`invalid_request_error` / `server_error`), when sent. */
  readonly code: string | undefined

  /**
   * Create the error.
   *
   * @param message - Human-readable message.
   * @param status - HTTP status (0 for network failure/timeout).
   * @param code - Server error type, when it sent one.
   * @param cause - The underlying error, if any.
   */
  constructor(message: string, status: number, code?: string, cause?: unknown) {
    super(message, cause === undefined ? undefined : { cause })
    this.name = 'NemoSpeechError'
    this.status = status
    this.code = code
  }
}

AsyncQueue

A single-consumer async queue: producers push, the consumer iterates, and end() finishes the iteration once everything pushed has been read.

class AsyncQueue<T> implements AsyncIterable<T> {
  private readonly items: T[] = []
  private waiting: ((result: IteratorResult<T>) => void) | null = null
  private ended = false

  /**
   * Add an item.
   *
   * @param item - The item (ignored after `end()`).
   */
  push(item: T): void {
    if (this.ended) return
    if (this.waiting) {
      const resolve = this.waiting
      this.waiting = null
      resolve({ value: item, done: false })
    } else {
      this.items.push(item)
    }
  }

  /** Finish the queue; the consumer drains what is left, then stops. */
  end(): void {
    if (this.ended) return
    this.ended = true
    if (this.waiting) {
      const resolve = this.waiting
      this.waiting = null
      resolve({ value: undefined, done: true })
    }
  }

  /**
   * Whether `end()` has been called.
   *
   * @returns True once ended.
   */
  get isEnded(): boolean {
    return this.ended
  }

  /**
   * Iterate the queue.
   *
   * @returns An async iterator over pushed items.
   */
  [Symbol.asyncIterator](): AsyncIterator<T> {
    return {
      next: (): Promise<IteratorResult<T>> => {
        if (this.items.length > 0) {
          return Promise.resolve({ value: this.items.shift() as T, done: false })
        }
        if (this.ended) return Promise.resolve({ value: undefined, done: true })
        return new Promise((resolve) => {
          this.waiting = resolve
        })
      },
    }
  }
}

Functions

isWav(bytes)

Checks whether the bytes start with a RIFF/WAVE header.

function isWav(bytes: Uint8Array<ArrayBufferLike>): boolean

• bytes — Audio bytes.

Returns: True for a WAV container.

decodeWav(bytes)

Decodes an uncompressed 16-bit PCM WAV file to mono samples.

function decodeWav(bytes: Uint8Array<ArrayBufferLike>): DecodedWav

• bytes — The WAV file bytes.

Returns: The sample rate, source channel count and mono samples.

resample(samples, fromRate, toRate)

Resamples mono PCM16 with linear interpolation.

function resample(
  samples: Int16Array<ArrayBufferLike>,
  fromRate: number,
  toRate: number,
): Int16Array<ArrayBufferLike>

• samples — Mono input samples.
• fromRate — Input sample rate in Hz.
• toRate — Output sample rate in Hz.

Returns: Resampled samples (the input itself when the rates match).

encodeWav(samples, sampleRate)

Encodes mono PCM16 samples as a WAV file.

function encodeWav(
  samples: Int16Array<ArrayBufferLike>,
  sampleRate: number,
): Uint8Array<ArrayBufferLike>

• samples — Mono samples.
• sampleRate — Sample rate in Hz.

Returns: The WAV file bytes.

toTargetWav(bytes)

Converts any 16-bit PCM WAV into a 16 kHz mono WAV.

function toTargetWav(bytes: Uint8Array<ArrayBufferLike>): Uint8Array<ArrayBufferLike>

• bytes — The source WAV bytes.

Returns: A 16 kHz mono 16-bit PCM WAV.

wavToTargetPcm(bytes)

Decodes a WAV file into 16 kHz mono PCM16 little-endian bytes.

function wavToTargetPcm(bytes: Uint8Array<ArrayBufferLike>): Uint8Array<ArrayBufferLike>

• bytes — The source WAV bytes.

Returns: Raw 16 kHz mono PCM16 LE bytes.

samplesToBytes(samples)

Serializes samples as PCM16 little-endian bytes.

function samplesToBytes(samples: Int16Array<ArrayBufferLike>): Uint8Array<ArrayBufferLike>

• samples — PCM16 samples.

Returns: Little-endian bytes.

toTargetBlocks(audio, sampleRate, blockBytes)

Converts a PCM16 LE stream at any sample rate to 16 kHz and re-chunks it into fixed-size blocks (the last block may be shorter).

function toTargetBlocks(
  audio: AsyncIterable<Uint8Array<ArrayBufferLike>>,
  sampleRate: number,
  blockBytes: number,
): AsyncGenerator<Uint8Array<ArrayBufferLike>, any, any>

• audio — Raw PCM16 LE mono chunks.
• sampleRate — Input sample rate in Hz.
• blockBytes — Output block size in bytes (even).

toBase64(bytes)

Encodes bytes as base64.

function toBase64(bytes: Uint8Array<ArrayBufferLike>): string

• bytes — Raw bytes.

Returns: Base64 text.

createProvider(config)

Creates a NeMo-Speech.cpp speech provider instance.

function createProvider(config?: NemoSpeechConfig): AISpeechProvider

• config — Optional overrides (URL, API key, timeouts, socket factory).

Returns: An AISpeechProvider backed by a self-hosted nemo-speech serve.

openGlobalWebSocket(url, headers, handlers)

Default socket factory over the runtime's global WebSocket. Handshake headers use the Node (undici) WebSocketInit.headers extension.

function openGlobalWebSocket(
  url: string,
  headers: Record<string, string>,
  handlers: RealtimeSocketHandlers,
): {
  send: (data: string | Uint8Array<ArrayBufferLike>) => void
  close: (code: number | undefined, reason: string | undefined) => void
}

• url — ws:// or wss:// URL.
• headers — Extra handshake headers.
• handlers — Activity callbacks.

Returns: The socket.

toWebSocketUrl(baseUrl)

Turns an http(s):// base URL into its ws(s):// equivalent.

function toWebSocketUrl(baseUrl: string): string

• baseUrl — HTTP base URL.

Returns: The WebSocket base URL.

parseFrame(data)

Decodes a text frame as JSON.

function parseFrame(data: string | Uint8Array<ArrayBufferLike>): Record<string, unknown> | null

• data — The frame.

Returns: The parsed object, or null for binary/non-JSON frames.

speakerLabel(raw)

Normalizes a raw speaker value to a speaker_<n> label.

function speakerLabel(raw: unknown): string | undefined

• raw — An integer, a numeric string, or a speaker_<n>-style string.

Returns: The normalized label, or undefined when no speaker was given.

segmentsFromWords(words)

Groups consecutive same-speaker words into segments ("who said what").

function segmentsFromWords(words: TranscriptionWord[]): TranscriptionSegment[] | undefined

• words — Words carrying speaker labels.

Returns: One segment per speaker run, or undefined when no word has a speaker.

Constants

TARGET_SAMPLE_RATE

The sample rate every self-hosted speech server in this family expects.

const TARGET_SAMPLE_RATE: 16000

DEFAULT_NEMO_SPEECH_URL

Default server URL — nemo-speech serve's own default bind address.

const DEFAULT_NEMO_SPEECH_URL: 'http://127.0.0.1:8080'

DEFAULT_TIMEOUT_MS

Default per-request HTTP timeout.

const DEFAULT_TIMEOUT_MS: 300000

DEFAULT_FINAL_TIMEOUT_MS

Default wait for the final completed event after the audio ends.

const DEFAULT_FINAL_TIMEOUT_MS: 30000

provider

The provider implementation.

const provider: AISpeechProvider

aiSpeechNemoSpeechSecretDefinitions

Secret definitions used by the NeMo-Speech.cpp speech bond.

const aiSpeechNemoSpeechSecretDefinitions: SecretDefinition[]

MAX_DIARIZATION_SPEAKERS

Most speakers Nemotron-3-Diarization (the Sortformer diarizer) can tell apart.

const MAX_DIARIZATION_SPEAKERS: 8

Core Interface

Implements @molecule/api-ai-speech interface.

Bond Wiring

Setup function to register this provider with the core interface:

import { setProvider } from '@molecule/api-ai-speech'
import { provider } from '@molecule/api-ai-speech-nemo-speech'

export function setupAiSpeechNemoSpeech(): void {
  setProvider(provider)
}

Injection Notes

Requirements

Peer dependencies:

• @molecule/api-ai-speech >=1.1.0
• @molecule/api-secrets ^1.0.1

Environment Variables

• NEMO_SPEECH_URL (optional) — NeMo-Speech.cpp server URL
  • Setup: The base URL of your own nemo-speech serve instance, without the /v1 path. Defaults to http://127.0.0.1:8080.
  • Get it here: https://github.com/NVIDIA/NeMo-Speech.cpp
  • Example: http://127.0.0.1:8080
• NEMO_SPEECH_API_KEY (optional) — NeMo-Speech.cpp API key
  • Setup: Only if the server was started with --api-key; sent as a Bearer token. Leave unset for a server without auth.
  • Get it here: https://github.com/NVIDIA/NeMo-Speech.cpp
  • Example: my-server-key

Runtime Dependencies

• @molecule/api-ai-speech

• @molecule/api-secrets

• You run the server. nemo-speech serve --asr-model nemotron-3.5 --diar-model <model> [--api-key <key>] (default 127.0.0.1:8080). Speaker labels need a diarizer loaded (--diar-model); without one a diarize: true call fails on the server. sortformer is the documented example name — list the real ones with nemo-speech model list.

• Uploads must be 16-bit PCM WAV. The server documents WAV only, so the bond converts any 16-bit PCM WAV to 16 kHz mono itself and THROWS for m4a/webm/mp3 (a filename: 'memo.m4a' is never passed through). Decode browser recordings to WAV/PCM first (e.g. with ffmpeg).

• Speakers are per call, at most 8. Labels are speaker_0, speaker_1, … in the order people first speak IN THAT FILE — speaker_1 in two calls can be two different people; it is not speaker identification. maxSpeakers above 8 throws; below 8 it is validated but not sent (the server has no such field). diarize: true requires responseFormat: 'verbose_json' (the default) and throws otherwise. segments are built by the bond from consecutive same-speaker words, joined with spaces.

• Not forwarded: prompt, temperature, timestampGranularity (words always come back with verbose_json). No translate, no TTS.

• Streaming (transcribeStream): send PCM16 LE mono at sampleRate (resampled to 16 kHz here) over the /v1/audio/transcriptions/realtime WebSocket. Text arrives as partial events that the following final REPLACES — streamingAppendOnly is false, so never treat a partial as settled. Streaming diarization is not requested by default; to turn it on, pass the server's own session.update fields via createProvider({ sessionUpdate }) (see the server's docs/server.md). The bond needs a global WebSocket (Node 22+) or a socketFactory.

• Server-side only. A browser microphone streams to YOUR API, which calls transcribeStream; never expose the model host or the key to the browser. Keep the server on a private network, and set --api-key + NEMO_SPEECH_API_KEY (sent as Authorization: Bearer …) whenever it is reachable by anyone else.

• Errors are NemoSpeechError with status (0 = unreachable or timed out) and code = the server's error type (invalid_request_error / server_error). Nothing is retried; the HTTP timeout is 300 s.

• Nemotron-3-Diarization is OpenMDW-1.1 licensed (the card says "ready for commercial or non-commercial use"); confirm the licence terms before offering it as a hosted service to others.

E2E Tests

Integration checklist — drive the real UI (live preview, no mocks), adapt each item to this app's actual screens/flows, and check every box off one by one. A box you can't check is an integration bug to fix — not a skip. The wired provider implements a subset (TTS, STT, or both) — feature-detect and run only the direction(s) it actually exposes:

• TTS: each read-aloud / narration / voice-note flow the app defines returns real PLAYABLE audio — the response carries an audio Content-Type (e.g. audio/mpeg) and non-trivial bytes (not a 0-byte file, not a JSON-encoded blob), and the UI's <audio>/player actually plays it. The spoken audio reflects the input text you sent.
• TTS: synthesizing two different strings yields two audibly different clips (distinct byte payloads) — proves the text drives synthesis and you are not hearing a cached/placeholder file.
• TTS: any exposed voice/format/speed option takes effect — switching voice (voice/voiceId) or output format (responseFormat/ outputFormat) changes the returned audio (audible voice, Content-Type, or file extension); an unknown voice/format surfaces a visible error rather than silently falling back.
• STT: transcribing a KNOWN spoken clip returns text that MATCHES the words spoken (not empty, not garbage) and renders in the UI; translating a non-English clip returns English text. Any timestamps the UI shows line up with the audio.
• STT: a silent / empty / too-short clip is handled gracefully — an empty or "no speech detected" result shown in the UI, never a crash or a spinner that hangs forever.
• The audio artifact is served from the app's OWN origin (a stored upload or a route the app serves), NOT a raw expiring provider URL — copy the audio src, reload it later, and confirm it still plays.
• A provider error (bad key, quota, unsupported format/voice, or a capability the wired provider lacks) surfaces as a readable message in the UI, not an unhandled crash or a blank/broken player.
• Synthesis/transcription runs SERVER-SIDE and authorized: the provider key never reaches the browser (check the network tab / client bundle), and no unauthenticated or unbounded endpoint lets a caller fire arbitrary synthesize/transcribe calls — each is billed per character / per minute of audio, so an open endpoint is a cost-DoS vector.