@molecule/api-ai-video-generation

AI video-generation core interface for molecule.dev.

Defines the AIVideoGenerationProvider contract — generate short video clips (with a synchronized audio track, when the model supports it) from text prompts and optional first-frame images, as submit→poll jobs — and the accessor (setProvider/getProvider/hasProvider/requireProvider). Interface-only: bond a provider package (e.g. @molecule/api-ai-video-generation-ltx, @molecule/api-ai-video-generation-kandinsky).

Auto-generated, AI-first package reference for the molecule.dev ecosystem. It is written to be read by coding agents as much as by people, and is generated from this package's source — edit src/index.ts JSDoc, not this file.

Quick Start

import { setProvider, requireProvider } from '@molecule/api-ai-video-generation'
import { provider } from '@molecule/api-ai-video-generation-kandinsky'

// Wire at startup. See the bond package for its config/env (e.g. KANDINSKY_BASE_URL).
setProvider(provider)

// Submit, then poll the job to a terminal state.
const job = await requireProvider().generate({
  prompt: 'A drone shot over a foggy pine forest at dawn',
  durationSeconds: 5,
})
let status = await requireProvider().getStatus(job.id)
while (status.status === 'pending' || status.status === 'processing') {
  await new Promise((resolve) => setTimeout(resolve, 5000))
  status = await requireProvider().getStatus(job.id)
}
// status.result?.url — provider URLs can expire; download and store it now.

Type

core

Installation

npm install @molecule/api-ai-video-generation @molecule/api-bond

API

Interfaces

VideoGenerateParams

Parameters for generating a short video clip (with audio, when the model supports it) from a text prompt and optionally a first-frame image.

interface VideoGenerateParams {
  /** Text description of the desired video. */
  prompt: string
  /** Negative prompt — things to exclude from the video. */
  negativePrompt?: string
  /**
   * First frame for image-to-video, as a Buffer of image bytes, a
   * base64-encoded string, or (provider permitting) a URI the provider can
   * fetch. Ignored by text-only paths.
   */
  image?: Buffer | string
  /** Model to use for generation (provider-specific). */
  model?: string
  /** Width of the generated video in pixels. */
  width?: number
  /** Height of the generated video in pixels. */
  height?: number
  /** Resolution as a "WxH" string (e.g. "1920x1080"). Alternative to width/height. */
  resolution?: string
  /** Frame rate in frames per second. */
  fps?: number
  /** Clip length in seconds. Omit for provider/automatic duration where supported. */
  durationSeconds?: number
  /** Whether to generate a synchronized audio track. */
  generateAudio?: boolean
  /** Random seed for reproducible generation. */
  seed?: number
  /** Number of inference steps. Higher = more detail, slower. */
  steps?: number
  /** Guidance scale / CFG scale — how closely to follow the prompt. */
  guidanceScale?: number
  /** Optional camera motion preset. Providers without camera control ignore it. */
  cameraMotion?: VideoCameraMotion
}

VideoJob

A submitted video generation job. Generation takes seconds to minutes — poll getStatus() with the id until it is terminal.

interface VideoJob {
  /** Job identifier — pass verbatim to `getStatus()`. Provider-opaque. */
  id: string
  /** The model the job was submitted to. */
  model: string
  /** When the job was created (ISO 8601), when the provider reports it. */
  createdAt?: string
}

VideoJobError

Error payload of a failed job.

interface VideoJobError {
  /** Machine-readable error type/code, when the provider sends one. */
  type?: string
  /** Human-readable description of the failure. */
  message: string
}

VideoJobResult

The generated video, once the job has completed.

interface VideoJobResult {
  /** URL of the generated video. Often short-lived — download promptly. */
  url?: string
  /** When the provider evicts `url` (ISO 8601), when it reports an expiry. */
  expiresAt?: string
  /**
   * The raw video bytes, when the provider delivers the video inline instead
   * of a fetchable URL (e.g. the download endpoint sits behind credentials
   * only the provider holds).
   */
  data?: Buffer
  /** MIME type of the video (e.g. 'video/mp4'). */
  mimeType?: string
  /** Clip length in seconds, when the provider reports it. */
  durationSeconds?: number
  /** The seed used, when the provider reports or echoes it. */
  seed?: number
}

VideoJobStatus

Status of a video generation job, from getStatus().

interface VideoJobStatus {
  /** Job identifier, as returned by `generate()`. */
  id: string
  /** Lifecycle state. `completed` carries `result`; `failed` carries `error`. */
  status: VideoJobState
  /** The model that serves the job, when the provider reports it. */
  model?: string
  /** When the job was created (ISO 8601), when the provider reports it. */
  createdAt?: string
  /** When the job reached a terminal state (ISO 8601), when the provider reports it. */
  completedAt?: string
  /** Why the job failed — present when `status` is `'failed'`. */
  error?: VideoJobError
  /** The generated video — present when `status` is `'completed'`. */
  result?: VideoJobResult
}

VideoUpscaleParams

Parameters for upscaling a generated video (Kandinsky VSR / LTX two-stage style super-resolution). Unsupported providers leave upscale undefined.

interface VideoUpscaleParams {
  /** The source video as a Buffer of MP4 bytes or a URL. */
  video: Buffer | string
  /** Upscale factor (e.g. 2, 2.25, 4). */
  factor?: number
  /** Prompt to guide the upscaling (if supported). */
  prompt?: string
}

AIVideoGenerationProvider

AIVideoGeneration provider interface.

Providers generate short video clips (usually with a synchronized audio track) from text prompts and optional first-frame images. Generation is asynchronous everywhere: generate() submits and returns a job handle; getStatus() polls it to a terminal state.

interface AIVideoGenerationProvider {
  /** Provider name identifier. */
  readonly name: string

  /**
   * Submit a text-to-video (or image-to-video, when `image` is set) job.
   *
   * @param params - Prompt plus model, resolution, duration and diffusion controls.
   * @returns The submitted job; poll `getStatus()` with its `id`.
   */
  generate(params: VideoGenerateParams): Promise<VideoJob>

  /**
   * Fetch the current status of a job.
   *
   * @param jobId - The `id` returned by `generate()`.
   * @returns The job's state, error (when failed) and result (when completed).
   */
  getStatus(jobId: string): Promise<VideoJobStatus>

  /**
   * Upscale a generated video to a higher resolution. Not all providers
   * support this operation.
   *
   * @param params - Source video and target factor.
   * @returns The upscale job; poll `getStatus()` with its `id`.
   */
  upscale?(params: VideoUpscaleParams): Promise<VideoJob>
}

AIVideoGenerationConfig

Base configuration for AI video generation providers.

interface AIVideoGenerationConfig {
  /** API key for the video generation service. */
  apiKey?: string
  /** Default model to use for generation. */
  defaultModel?: string
  /** Base URL override (for self-hosted endpoints or brokers). */
  baseUrl?: string
  /** Additional provider-specific options. */
  [key: string]: unknown
}

Types

VideoCameraMotion

Camera motion presets (LTX-style). Providers without camera control ignore the parameter.

type VideoCameraMotion =
  | 'dolly_in'
  | 'dolly_out'
  | 'dolly_left'
  | 'dolly_right'
  | 'jib_up'
  | 'jib_down'
  | 'static'
  | 'focus_shift'

VideoJobState

Lifecycle state of a video job.

type VideoJobState = 'pending' | 'processing' | 'completed' | 'failed'

Functions

setProvider(provider)

Registers the AI video generation provider singleton.

function setProvider(provider: AIVideoGenerationProvider): void

• provider — The AI video generation provider implementation to register.

getProvider()

Returns the bonded AI video generation provider, or null if none is registered.

function getProvider(): AIVideoGenerationProvider | null

Returns: The active provider, or null.

hasProvider()

Returns whether an AI video generation provider has been registered.

function hasProvider(): boolean

Returns: true if a provider is bonded.

requireProvider()

Returns the bonded AI video generation provider, throwing if none is configured.

function requireProvider(): AIVideoGenerationProvider

Returns: The active provider.

Available Providers

Provider	Package
Kandinsky	@molecule/api-ai-video-generation-kandinsky
LTX (hosted)	@molecule/api-ai-video-generation-ltx

Injection Notes

Requirements

Peer dependencies:

• @molecule/api-bond ^1.0.1

Runtime Dependencies

• @molecule/api-bond

• Wire it at startup with setProvider(...) — or the equivalent bond('ai-video-generation', provider). This core routes through the shared @molecule/api-bond registry, so either call registers the same provider and validateBonds() reports it as missing when unwired.

• Generation is asynchronous everywhere — this is a job API, not a one-call API. generate() submits and returns a VideoJob; poll getStatus(job.id) until status is 'completed' or 'failed' (leave seconds between polls — hosted providers document 5 s+; a diffusion clip takes tens of seconds to minutes). There is no synchronous variant, and there is no -llm degradation bond: a chat model cannot approximate video generation, so an unbonded app fails fast with the "not bonded" error above.

• Only prompt is fully portable. model, resolution/dimensions, durationSeconds, steps and guidanceScale map differently per provider (some snap dimensions, some require exact step/guidance pairings); cameraMotion is ignored by providers without camera control. Check the bonded provider's docs before relying on anything beyond prompt.

• Feature-detect upscale. Only generate() and getStatus() are required; upscale is optional — guard with if (provider.upscale) instead of calling unconditionally (an absent method is a runtime TypeError that type-checks).

• Handle every result shape, and persist what you must keep. A completed job's result carries url and/or raw data bytes — provider URLs are typically short-lived (LTX's expire on their own; a self-hosted server may evict them; some providers report the deadline as result.expiresAt), so download and store (e.g. via the uploads bond) anything the app needs to keep, promptly after completion.

• Server-side only, gated and budgeted. Keep the provider key on the API; require auth and rate-limit user-triggered generation — every clip is billed or GPU-expensive.

E2E Tests

Integration checklist — drive the real UI (live preview, no mocks), adapt each item to this app's actual screens/flows, and check every box off one by one. A box you can't check is an integration bug to fix — not a skip:

• Entering a prompt in the UI and submitting produces a REAL playable video in the live preview once the job completes — not a broken video element, a spinner that never ends, or an error toast. Confirm the video element actually loaded (its readyState/duration are set) and the clip visibly reflects the prompt (a "waves on a beach" prompt shows waves, not a placeholder).
• Two different prompts produce two visibly different clips — a fixed stub, or a cached first result that never changes, is a broken integration.
• The video is STORED AND SERVED FROM THE APP'S OWN ORIGIN: the played source/src (and any saved record) points at the app's uploads/storage, not the provider's temporary URL. Provider url results expire — reload the page (or revisit the record later) and the video must still play. Download the provider result server-side and persist it (uploads bond) once the job completes; never hotlink the provider URL. (data results must likewise be saved, not held only in the response.)
• Job progress is visible while polling: the UI shows the job is pending/processing (not frozen), and a failed job surfaces the provider's error message clearly so the user can recover and try another prompt.
• Any exposed generation options take effect: changing durationSeconds/resolution yields a different-length/-sized clip. If the UI exposes no such options, this box is n/a — say so.
• Generation is server-side and authorized: the provider API key never reaches the browser (check the network tab / built client bundle — no key, no direct provider call from the page), the generate endpoint requires auth, and a caller cannot run unbounded costly generations through an open or unrate-limited route. Every clip is billed or GPU-expensive.