@molecule/api-ai-speech-nemo-speech

NVIDIA NeMo-Speech.cpp provider for @molecule/api-ai-speech.

Batch and streaming speech-to-text plus speaker diarization ("who spoke when", Nemotron-3-Diarization) on your own nemo-speech serve instance — NVIDIA's native C++ runtime, which runs on CPU, CUDA, Metal and Vulkan. Implements transcribe (with diarize: true speaker labels), diarize and transcribeStream.

Quick Start

import { requireProvider, setProvider } from '@molecule/api-ai-speech'
import { provider } from '@molecule/api-ai-speech-nemo-speech'

// Reads NEMO_SPEECH_URL (default http://127.0.0.1:8080) on each call.
setProvider(provider)

const speech = requireProvider()
const result = await speech.transcribe!({ audio: wavBytes, diarize: true })
for (const segment of result.segments ?? []) {
  console.log(`${segment.speaker}: ${segment.text}`)
}

Type

provider

Installation

npm install @molecule/api-ai-speech-nemo-speech @molecule/api-ai-speech @molecule/api-secrets

API

Interfaces

DecodedWav

Decoded PCM16 audio.

interface DecodedWav {
  /** Sample rate of the decoded audio in Hz. */
  sampleRate: number
  /** Number of interleaved channels in the source file. */
  channels: number
  /** Mono PCM16 samples (multi-channel input is averaged down). */
  samples: Int16Array
}

NemoSpeechConfig

Configuration for the NeMo-Speech.cpp speech provider. Every field falls back to its env var, read on each call (never at import time).

interface NemoSpeechConfig {
  /**
   * Server base URL (`nemo-speech serve`), without `/v1`. Defaults to the
   * `NEMO_SPEECH_URL` env var, then `http://127.0.0.1:8080`.
   */
  baseUrl?: string
  /**
   * API key, sent as `Authorization: Bearer …` — only needed when the server
   * was started with `--api-key`. Defaults to `NEMO_SPEECH_API_KEY`.
   */
  apiKey?: string
  /** Model sent as the `model` form field when a call passes none. Omitted by default. */
  defaultModel?: string
  /** Per-request HTTP timeout in milliseconds. Defaults to 300000. */
  timeoutMs?: number
  /**
   * How long a stream waits, after the audio ends, for the server's final
   * `completed` event before it gives up. Defaults to 30000 ms.
   */
  finalTimeoutMs?: number
  /**
   * A raw `session.update` event body sent right after `session.created` on
   * streaming connections (e.g. to turn on streaming diarization). Its field
   * names are defined by the server's `docs/server.md`; the bond sends it
   * verbatim with `type: 'session.update'` added. Omitted by default.
   */
  sessionUpdate?: Record<string, unknown>
  /** WebSocket factory for streaming. Defaults to the runtime's global `WebSocket`. */
  socketFactory?: RealtimeSocketFactory
}

RealtimeSocket

An open (or opening) realtime socket.

interface RealtimeSocket {
  /**
   * Send a text or binary frame.
   *
   * @param data - JSON text or raw bytes.
   */
  send(data: string | Uint8Array): void
  /**
   * Close the socket.
   *
   * @param code - Close code (default 1000).
   * @param reason - Close reason.
   */
  close(code?: number, reason?: string): void
}

RealtimeSocketHandlers

Callbacks a RealtimeSocketFactory reports socket activity to.

interface RealtimeSocketHandlers {
  /** The connection is open. */
  onOpen(): void
  /** A frame arrived (text frames as strings, binary frames as bytes). */
  onMessage(data: string | Uint8Array): void
  /** The connection closed. */
  onClose(code: number, reason: string): void
  /** The connection failed. */
  onError(message: string): void
}

Types

RealtimeSocketFactory

Opens a realtime socket.

type RealtimeSocketFactory = (
  url: string,
  headers: Record<string, string>,
  handlers: RealtimeSocketHandlers,
) => RealtimeSocket

Classes

AsyncQueue

A single-consumer async queue: producers push, the consumer iterates, and end() finishes the iteration once everything pushed has been read.

NemoSpeechError

Error thrown when the NeMo-Speech.cpp server refuses or fails a request. Carries the HTTP status (0 when the server could not be reached or timed out) and the server's error type as code. Deliberately NOT tagged with statusCode/errorKey, so API middleware answers its generic 500 instead of echoing the upstream status to the caller.

Pcm16Resampler

Streaming PCM16 little-endian resampler (linear interpolation). Keeps the state a chunked stream needs: an odd trailing byte and the fractional read position carry over to the next chunk, so chunk boundaries add no clicks.

Functions

createProvider(config)

Creates a NeMo-Speech.cpp speech provider instance.

function createProvider(config?: NemoSpeechConfig): AISpeechProvider

• config — Optional overrides (URL, API key, timeouts, socket factory).

Returns: An AISpeechProvider backed by a self-hosted nemo-speech serve.

decodeWav(bytes)

Decodes an uncompressed 16-bit PCM WAV file to mono samples.

function decodeWav(bytes: Uint8Array<ArrayBufferLike>): DecodedWav

• bytes — The WAV file bytes.

Returns: The sample rate, source channel count and mono samples.

encodeWav(samples, sampleRate)

Encodes mono PCM16 samples as a WAV file.

function encodeWav(
  samples: Int16Array<ArrayBufferLike>,
  sampleRate: number,
): Uint8Array<ArrayBufferLike>

• samples — Mono samples.
• sampleRate — Sample rate in Hz.

Returns: The WAV file bytes.

isWav(bytes)

Checks whether the bytes start with a RIFF/WAVE header.

function isWav(bytes: Uint8Array<ArrayBufferLike>): boolean

• bytes — Audio bytes.

Returns: True for a WAV container.

openGlobalWebSocket(url, headers, handlers)

Default socket factory over the runtime's global WebSocket. Handshake headers use the Node (undici) WebSocketInit.headers extension.

function openGlobalWebSocket(
  url: string,
  headers: Record<string, string>,
  handlers: RealtimeSocketHandlers,
): {
  send: (data: string | Uint8Array<ArrayBufferLike>) => void
  close: (code: number | undefined, reason: string | undefined) => void
}

• url — ws:// or wss:// URL.
• headers — Extra handshake headers.
• handlers — Activity callbacks.

Returns: The socket.

parseFrame(data)

Decodes a text frame as JSON.

function parseFrame(data: string | Uint8Array<ArrayBufferLike>): Record<string, unknown> | null

• data — The frame.

Returns: The parsed object, or null for binary/non-JSON frames.

resample(samples, fromRate, toRate)

Resamples mono PCM16 with linear interpolation.

function resample(
  samples: Int16Array<ArrayBufferLike>,
  fromRate: number,
  toRate: number,
): Int16Array<ArrayBufferLike>

• samples — Mono input samples.
• fromRate — Input sample rate in Hz.
• toRate — Output sample rate in Hz.

Returns: Resampled samples (the input itself when the rates match).

samplesToBytes(samples)

Serializes samples as PCM16 little-endian bytes.

function samplesToBytes(samples: Int16Array<ArrayBufferLike>): Uint8Array<ArrayBufferLike>

• samples — PCM16 samples.

Returns: Little-endian bytes.

segmentsFromWords(words)

Groups consecutive same-speaker words into segments ("who said what").

function segmentsFromWords(words: TranscriptionWord[]): TranscriptionSegment[] | undefined

• words — Words carrying speaker labels.

Returns: One segment per speaker run, or undefined when no word has a speaker.

speakerLabel(raw)

Normalizes a raw speaker value to a speaker_<n> label.

function speakerLabel(raw: unknown): string | undefined

• raw — An integer, a numeric string, or a speaker_<n>-style string.

Returns: The normalized label, or undefined when no speaker was given.

toBase64(bytes)

Encodes bytes as base64.

function toBase64(bytes: Uint8Array<ArrayBufferLike>): string

• bytes — Raw bytes.

Returns: Base64 text.

toTargetBlocks(audio, sampleRate, blockBytes)

Converts a PCM16 LE stream at any sample rate to 16 kHz and re-chunks it into fixed-size blocks (the last block may be shorter).

function toTargetBlocks(
  audio: AsyncIterable<Uint8Array<ArrayBufferLike>>,
  sampleRate: number,
  blockBytes: number,
): AsyncGenerator<Uint8Array<ArrayBufferLike>, any, any>

• audio — Raw PCM16 LE mono chunks.
• sampleRate — Input sample rate in Hz.
• blockBytes — Output block size in bytes (even).

toTargetWav(bytes)

Converts any 16-bit PCM WAV into a 16 kHz mono WAV.

function toTargetWav(bytes: Uint8Array<ArrayBufferLike>): Uint8Array<ArrayBufferLike>

• bytes — The source WAV bytes.

Returns: A 16 kHz mono 16-bit PCM WAV.

toWebSocketUrl(baseUrl)

Turns an http(s):// base URL into its ws(s):// equivalent.

function toWebSocketUrl(baseUrl: string): string

• baseUrl — HTTP base URL.

Returns: The WebSocket base URL.

wavToTargetPcm(bytes)

Decodes a WAV file into 16 kHz mono PCM16 little-endian bytes.

function wavToTargetPcm(bytes: Uint8Array<ArrayBufferLike>): Uint8Array<ArrayBufferLike>

• bytes — The source WAV bytes.

Returns: Raw 16 kHz mono PCM16 LE bytes.

Constants

aiSpeechNemoSpeechSecretDefinitions

Secret definitions used by the NeMo-Speech.cpp speech bond.

const aiSpeechNemoSpeechSecretDefinitions: SecretDefinition[]

DEFAULT_FINAL_TIMEOUT_MS

Default wait for the final completed event after the audio ends.

const DEFAULT_FINAL_TIMEOUT_MS: 30000

DEFAULT_NEMO_SPEECH_URL

Default server URL — nemo-speech serve's own default bind address.

const DEFAULT_NEMO_SPEECH_URL: 'http://127.0.0.1:8080'

DEFAULT_TIMEOUT_MS

Default per-request HTTP timeout.

const DEFAULT_TIMEOUT_MS: 300000

MAX_DIARIZATION_SPEAKERS

Most speakers Nemotron-3-Diarization (the Sortformer diarizer) can tell apart.

const MAX_DIARIZATION_SPEAKERS: 8

provider

The provider implementation.

const provider: AISpeechProvider

TARGET_SAMPLE_RATE

The sample rate every self-hosted speech server in this family expects.

const TARGET_SAMPLE_RATE: 16000

Core Interface

Implements @molecule/api-ai-speech interface.

Bond Wiring

Setup function to register this provider with the core interface:

import { setProvider } from '@molecule/api-ai-speech'
import { provider } from '@molecule/api-ai-speech-nemo-speech'

export function setupAiSpeechNemoSpeech(): void {
  setProvider(provider)
}

Injection Notes

Requirements

Peer dependencies:

• @molecule/api-ai-speech >=1.1.0
• @molecule/api-secrets ^1.0.1

Environment Variables

• NEMO_SPEECH_URL (optional) — NeMo-Speech.cpp server URL
  • Setup: The base URL of your own nemo-speech serve instance, without the /v1 path. Defaults to http://127.0.0.1:8080.
  • Get it here: https://github.com/NVIDIA/NeMo-Speech.cpp
  • Example: http://127.0.0.1:8080
• NEMO_SPEECH_API_KEY (optional) — NeMo-Speech.cpp API key
  • Setup: Only if the server was started with --api-key; sent as a Bearer token. Leave unset for a server without auth.
  • Get it here: https://github.com/NVIDIA/NeMo-Speech.cpp
  • Example: my-server-key

Runtime Dependencies

• @molecule/api-ai-speech

• @molecule/api-secrets

• You run the server. nemo-speech serve --asr-model nemotron-3.5 --diar-model <model> [--api-key <key>] (default 127.0.0.1:8080). Speaker labels need a diarizer loaded (--diar-model); without one a diarize: true call fails on the server. sortformer is the documented example name — list the real ones with nemo-speech model list.

• Uploads must be 16-bit PCM WAV. The server documents WAV only, so the bond converts any 16-bit PCM WAV to 16 kHz mono itself and THROWS for m4a/webm/mp3 (a filename: 'memo.m4a' is never passed through). Decode browser recordings to WAV/PCM first (e.g. with ffmpeg).

• Speakers are per call, at most 8. Labels are speaker_0, speaker_1, … in the order people first speak IN THAT FILE — speaker_1 in two calls can be two different people; it is not speaker identification. maxSpeakers above 8 throws; below 8 it is validated but not sent (the server has no such field). diarize: true requires responseFormat: 'verbose_json' (the default) and throws otherwise. segments are built by the bond from consecutive same-speaker words, joined with spaces.

• Not forwarded: prompt, temperature, timestampGranularity (words always come back with verbose_json). No translate, no TTS.

• Streaming (transcribeStream): send PCM16 LE mono at sampleRate (resampled to 16 kHz here) over the /v1/audio/transcriptions/realtime WebSocket. Text arrives as partial events that the following final REPLACES — streamingAppendOnly is false, so never treat a partial as settled. Streaming diarization is not requested by default; to turn it on, pass the server's own session.update fields via createProvider({ sessionUpdate }) (see the server's docs/server.md). The bond needs a global WebSocket (Node 22+) or a socketFactory.

• Server-side only. A browser microphone streams to YOUR API, which calls transcribeStream; never expose the model host or the key to the browser. Keep the server on a private network, and set --api-key + NEMO_SPEECH_API_KEY (sent as Authorization: Bearer …) whenever it is reachable by anyone else.

• Errors are NemoSpeechError with status (0 = unreachable or timed out) and code = the server's error type (invalid_request_error / server_error). Nothing is retried; the HTTP timeout is 300 s.

• Nemotron-3-Diarization is OpenMDW-1.1 licensed (the card says "ready for commercial or non-commercial use"); confirm the licence terms before offering it as a hosted service to others.

E2E Tests

Integration checklist — drive the real UI (live preview, no mocks), adapt each item to this app's actual screens/flows, and check every box off one by one. A box you can't check is an integration bug to fix — not a skip. The wired provider implements a subset (TTS, STT, or both) — feature-detect and run only the direction(s) it actually exposes:

• TTS: each read-aloud / narration / voice-note flow the app defines returns real PLAYABLE audio — the response carries an audio Content-Type (e.g. audio/mpeg) and non-trivial bytes (not a 0-byte file, not a JSON-encoded blob), and the UI's <audio>/player actually plays it. The spoken audio reflects the input text you sent.
• TTS: synthesizing two different strings yields two audibly different clips (distinct byte payloads) — proves the text drives synthesis and you are not hearing a cached/placeholder file.
• TTS: any exposed voice/format/speed option takes effect — switching voice (voice/voiceId) or output format (responseFormat/ outputFormat) changes the returned audio (audible voice, Content-Type, or file extension); an unknown voice/format surfaces a visible error rather than silently falling back.
• STT: transcribing a KNOWN spoken clip returns text that MATCHES the words spoken (not empty, not garbage) and renders in the UI; translating a non-English clip returns English text. Any timestamps the UI shows line up with the audio.
• STT: a silent / empty / too-short clip is handled gracefully — an empty or "no speech detected" result shown in the UI, never a crash or a spinner that hangs forever.
• The audio artifact is served from the app's OWN origin (a stored upload or a route the app serves), NOT a raw expiring provider URL — copy the audio src, reload it later, and confirm it still plays.
• A provider error (bad key, quota, unsupported format/voice, or a capability the wired provider lacks) surfaces as a readable message in the UI, not an unhandled crash or a blank/broken player.
• Synthesis/transcription runs SERVER-SIDE and authorized: the provider key never reaches the browser (check the network tab / client bundle), and no unauthenticated or unbounded endpoint lets a caller fire arbitrary synthesize/transcribe calls — each is billed per character / per minute of audio, so an open endpoint is a cost-DoS vector.