@molecule/api-ai-speech-llm

Language-model speech-to-text provider for molecule.dev.

Implements transcribe from @molecule/api-ai-speech with any @molecule/api-ai chat model that accepts audio input, so an app with no speech service still gets batch transcription through the model it already pays for — hosted or self-hosted.

Auto-generated, AI-first package reference for the molecule.dev ecosystem. It is written to be read by coding agents as much as by people, and is generated from this package's source — edit src/index.ts JSDoc, not this file.

Quick Start

import { setProvider as setAi } from '@molecule/api-ai'
import { provider as google } from '@molecule/api-ai-google'
import { requireProvider, setProvider } from '@molecule/api-ai-speech'
import { createProvider } from '@molecule/api-ai-speech-llm'

setAi(google) // the app's AI provider — its model must accept audio
setProvider(createProvider()) // or createProvider({ ai, model }) for another

const { text } = await requireProvider().transcribe!({
  audio: new Uint8Array(recordingBytes),
  filename: 'memo.webm',
  language: 'en',
})

Type

provider

Installation

npm install @molecule/api-ai-speech-llm @molecule/api-ai @molecule/api-ai-speech

API

Interfaces

`LlmSpeechConfig`

Options for createProvider. ai falls back to the bonded ai provider; every other field has a prompt-level default.

interface LlmSpeechConfig {
  /**
   * The AI provider to transcribe with. Defaults to the provider bonded as
   * `ai` (`@molecule/api-ai` `requireProvider()`), whose model must accept
   * audio input.
   */
  ai?: AIProvider
  /** The model id to pass to the provider. Omit to use the provider's default. */
  model?: string
  /** Default language hint when a `transcribe` call passes none. */
  language?: string
  /** Extra instructions, e.g. "Medical dictation; keep drug names verbatim". Appended to the system prompt. */
  instructions?: string
  /** Output token ceiling. Defaults to a length derived from the audio size. */
  maxOutputTokens?: number
}

Functions

`systemPrompt(options)`

The system prompt for one transcription pass.

function systemPrompt(options: {
  diarize?: boolean
  maxSpeakers?: number
  instructions?: string
}): string

options — Diarization request and extra caller instructions.
options.diarize — Ask the model to label speakers line by line.
options.maxSpeakers — Upper bound on speakers to label.
options.instructions — Extra caller instructions, appended last.

Returns: The system prompt.

`userPrompt(language, context)`

The text part of the user message (the audio block is sent alongside it).

function userPrompt(language?: string, context?: string): string

language — Optional language hint.
context — Optional context / spelling hints.

Returns: The user-side instruction.

`audioMediaType(filename)`

Picks the audio MIME type from a filename.

function audioMediaType(filename: string | undefined): string

filename — Filename hint (defaults to WAV when absent or unknown).

Returns: The MIME type.

`parseSpeakerLines(transcript)`

Parses speaker_<n>: text lines into labelled segments.

function parseSpeakerLines(
  transcript: string,
): { segments: TranscriptionSegment[]; text: string } | null

transcript — The model's diarized answer.

Returns: The segments (start = end = 0 — the model gives no timings) and the transcript with labels removed, or null when no line carried a label.

`createProvider(config)`

Creates a language-model speech provider.

function createProvider(config?: LlmSpeechConfig): AISpeechProvider

config — Which AI provider/model to use and how to prompt it.

Returns: An AISpeechProvider backed by an audio-capable chat model.

Constants

`provider`

The provider implementation.

const provider: AISpeechProvider

Core Interface

Implements @molecule/api-ai-speech interface.

Bond Wiring

Setup function to register this provider with the core interface:

import { setProvider } from '@molecule/api-ai-speech'
import { provider } from '@molecule/api-ai-speech-llm'

export function setupAiSpeechLlm(): void {
  setProvider(provider)
}

Injection Notes

Requirements

Peer dependencies:

@molecule/api-ai >=1.3.0
@molecule/api-ai-speech >=1.1.0

Runtime Dependencies

@molecule/api-ai
@molecule/api-ai-speech
The model must accept audio input. Most chat models do not — a text-only model fails the request or invents a transcript. Do not bond this provider for a non-audio model; check the model catalog first and pass one with createProvider({ model }).
Batch only. No transcribeStream, no diarize(), no translate, no TTS. Feature-detect per the core. responseFormat: 'srt' | 'vtt' throws — a chat model returns no timings — and no result carries words or duration.
diarize: true is best-effort text labelling, NOT acoustic diarization — treat it as unreliable. The model is asked to prefix lines with speaker_<n>:; those become segments[].speaker, and every segment has start = end = 0 because there are no timings. Use @molecule/api-ai-speech-nemo-speech when "who said what" matters.
A model is a transcriber that may "improve" the speaker (fix grammar, drop fillers, translate) despite the prompt. Spot-check a new model before trusting it in bulk; markdown fences are stripped, other chatter lands in text verbatim.
The filename extension picks the audio MIME type (.webm → audio/webm, .m4a → audio/mp4, unknown → audio/wav); whether the provider accepts that container is up to the chat bond and model.
Uses the AI provider bonded as ai unless you pass one. Spend is that provider's token spend — audio bills as input tokens.

E2E Tests

Integration checklist — drive the real UI (live preview, no mocks), adapt each item to this app's actual screens/flows, and check every box off one by one. A box you can't check is an integration bug to fix — not a skip. The wired provider implements a subset (TTS, STT, or both) — feature-detect and run only the direction(s) it actually exposes:

TTS: each read-aloud / narration / voice-note flow the app defines returns real PLAYABLE audio — the response carries an audio Content-Type (e.g. audio/mpeg) and non-trivial bytes (not a 0-byte file, not a JSON-encoded blob), and the UI's <audio>/player actually plays it. The spoken audio reflects the input text you sent.
TTS: synthesizing two different strings yields two audibly different clips (distinct byte payloads) — proves the text drives synthesis and you are not hearing a cached/placeholder file.
TTS: any exposed voice/format/speed option takes effect — switching voice (voice/voiceId) or output format (responseFormat/ outputFormat) changes the returned audio (audible voice, Content-Type, or file extension); an unknown voice/format surfaces a visible error rather than silently falling back.
STT: transcribing a KNOWN spoken clip returns text that MATCHES the words spoken (not empty, not garbage) and renders in the UI; translating a non-English clip returns English text. Any timestamps the UI shows line up with the audio.
STT: a silent / empty / too-short clip is handled gracefully — an empty or "no speech detected" result shown in the UI, never a crash or a spinner that hangs forever.
The audio artifact is served from the app's OWN origin (a stored upload or a route the app serves), NOT a raw expiring provider URL — copy the audio src, reload it later, and confirm it still plays.
A provider error (bad key, quota, unsupported format/voice, or a capability the wired provider lacks) surfaces as a readable message in the UI, not an unhandled crash or a blank/broken player.
Synthesis/transcription runs SERVER-SIDE and authorized: the provider key never reaches the browser (check the network tab / client bundle), and no unauthenticated or unbounded endpoint lets a caller fire arbitrary synthesize/transcribe calls — each is billed per character / per minute of audio, so an open endpoint is a cost-DoS vector.

`@molecule/api-ai-speech-llm`

How it works

Reference