@molecule/api-model-hosting-runpod

RunPod Serverless model hosting for molecule.dev — run a model server container on a load-balancing GPU endpoint (16 GB class and up) that scales to zero, called at https://<endpoint-id>.api.runpod.ai with your RunPod API key.

Load-balancing endpoints run YOUR HTTP server (no RunPod handler code), so laya-serve, jeff-serve or any /v1/systemone server works unchanged.

Auto-generated, AI-first package reference for the molecule.dev ecosystem. It is written to be read by coding agents as much as by people, and is generated from this package's source — edit src/index.ts JSDoc, not this file.

Quick Start

import { setProvider, requireProvider } from '@molecule/api-model-hosting'
import { provider } from '@molecule/api-model-hosting-runpod'

setProvider(provider) // reads RUNPOD_API_KEY on first use

const endpoint = await requireProvider().deploy({
  name: 'jeff',
  image: 'ghcr.io/acme/jeff-serve:0.8b',
  port: 8000,
  healthPath: '/ping',
  accelerator: { kind: 'gpu', minVramGb: 16 },
  region: 'us',
  scaling: { minInstances: 0, maxInstances: 2, idleTimeoutSeconds: 60 },
  access: 'private',
})
const headers = await requireProvider().authHeaders(endpoint.id) // { authorization: 'Bearer <RUNPOD_API_KEY>' }

Type

provider

Installation

npm install @molecule/api-model-hosting-runpod @molecule/api-bond @molecule/api-i18n @molecule/api-model-hosting @molecule/api-secrets

API

Interfaces

RunPodConfig

Configuration for the RunPod model hosting provider.

interface RunPodConfig {
  /** API key. Defaults to `RUNPOD_API_KEY`. */
  apiKey?: string
  /** GraphQL `locations` for `'us'`. Default `'US'`. */
  usLocations?: string
  /** GraphQL `locations` for `'eu'`. Default `'CZ,FR,GB,NO,RO'`. */
  euLocations?: string
  /** Container disk per worker, in GB. Default 20. */
  containerDiskGb?: number
  /** REST base URL (tests). Default `https://rest.runpod.io/v1`. */
  restBaseUrl?: string
  /** GraphQL URL (tests). Default `https://api.runpod.io/graphql`. */
  graphqlUrl?: string
  /** Endpoint URL template (tests); `{id}` is replaced. Default `https://{id}.api.runpod.ai`. */
  endpointUrlTemplate?: string
  /** How often `deploy` probes the health path while workers start, in ms. Default 5000. */
  pollIntervalMs?: number
}

RunPodTemplate

A RunPod template (only the fields read here).

interface RunPodTemplate {
  id: string
  name: string
}

RunPodEndpoint

A RunPod serverless endpoint (only the fields read here).

interface RunPodEndpoint {
  id: string
  name: string
  templateId?: string
  workersMin?: number
  workersMax?: number
  idleTimeout?: number
  gpuTypeIds?: string[]
  dataCenterIds?: string[]
  createdAt?: string
  workers?: unknown[]
}

Classes

RunPodApiError

A RunPod API error, carrying the HTTP status.

class RunPodApiError extends Error {
  /** HTTP status. */
  readonly status: number

  /**
   * Creates the error.
   *
   * @param message - What failed.
   * @param status - HTTP status.
   */
  constructor(message: string, status: number) {
    super(message)
    this.name = 'RunPodApiError'
    this.status = status
  }
}

RunPodClient

Thin RunPod client.

class RunPodClient {
  /**
   * Creates the client.
   *
   * @param apiKey - Returns the API key (read lazily).
   * @param restBaseUrl - REST base URL.
   * @param graphqlUrl - GraphQL URL.
   */
  constructor(
    private readonly apiKey: () => string,
    private readonly restBaseUrl: string,
    private readonly graphqlUrl: string,
  ) {}

  /**
   * Sends one REST request; `null` on 404.
   *
   * @param method - HTTP method.
   * @param path - Path under the REST base.
   * @param body - JSON body.
   * @returns The parsed body, or `null` on 404 / 204.
   */
  private async rest<T>(method: string, path: string, body?: unknown): Promise<T | null> {
    const res = await fetch(`${this.restBaseUrl}${path}`, {
      method,
      headers: { authorization: `Bearer ${this.apiKey()}`, 'content-type': 'application/json' },
      ...(body === undefined ? {} : { body: JSON.stringify(body) }),
    })
    if (res.status === 404 || res.status === 204) return null
    const text = await res.text()
    if (!res.ok) {
      throw new RunPodApiError(
        `RunPod ${method} ${path} failed (${res.status}): ${text.slice(0, 300)}`,
        res.status,
      )
    }
    return (text ? JSON.parse(text) : null) as T | null
  }

  /**
   * Sends one GraphQL operation.
   *
   * @param query - The GraphQL document.
   * @returns The `data` object.
   */
  private async graphql<T>(query: string): Promise<T> {
    const res = await fetch(this.graphqlUrl, {
      method: 'POST',
      headers: { authorization: `Bearer ${this.apiKey()}`, 'content-type': 'application/json' },
      body: JSON.stringify({ query }),
    })
    const json = (await res.json().catch(() => ({}))) as {
      data?: T
      errors?: Array<{ message?: string }>
    }
    if (!res.ok || json.errors?.length || !json.data) {
      throw new RunPodApiError(
        `RunPod GraphQL failed (${res.status}): ${json.errors?.map((e) => e.message).join('; ') ?? 'no data'}`,
        res.status,
      )
    }
    return json.data
  }

  /**
   * Lists templates.
   *
   * @returns The templates.
   */
  async listTemplates(): Promise<RunPodTemplate[]> {
    return (await this.rest<RunPodTemplate[]>('GET', '/templates')) ?? []
  }

  /**
   * Creates a template.
   *
   * @param body - `TemplateCreateInput`.
   * @returns The template.
   */
  async createTemplate(body: Record<string, unknown>): Promise<RunPodTemplate> {
    return (await this.rest<RunPodTemplate>('POST', '/templates', body))!
  }

  /**
   * Updates a template.
   *
   * @param id - Template id.
   * @param body - Fields to change.
   */
  async updateTemplate(id: string, body: Record<string, unknown>): Promise<void> {
    await this.rest('PATCH', `/templates/${id}`, body)
  }

  /**
   * Deletes a template.
   *
   * @param id - Template id.
   */
  async deleteTemplate(id: string): Promise<void> {
    await this.rest('DELETE', `/templates/${id}`)
  }

  /**
   * Lists endpoints.
   *
   * @returns The endpoints.
   */
  async listEndpoints(): Promise<RunPodEndpoint[]> {
    return (await this.rest<RunPodEndpoint[]>('GET', '/endpoints')) ?? []
  }

  /**
   * Reads one endpoint with its workers.
   *
   * @param id - Endpoint id.
   * @returns The endpoint, or `null`.
   */
  getEndpoint(id: string): Promise<RunPodEndpoint | null> {
    return this.rest<RunPodEndpoint>('GET', `/endpoints/${id}?includeWorkers=true`)
  }

  /**
   * Updates an endpoint.
   *
   * @param id - Endpoint id.
   * @param body - `EndpointUpdateInput` fields.
   * @returns The endpoint.
   */
  async updateEndpoint(id: string, body: Record<string, unknown>): Promise<RunPodEndpoint | null> {
    return this.rest<RunPodEndpoint>('PATCH', `/endpoints/${id}`, body)
  }

  /**
   * Deletes an endpoint.
   *
   * @param id - Endpoint id.
   */
  async deleteEndpoint(id: string): Promise<void> {
    await this.rest('DELETE', `/endpoints/${id}`)
  }

  /**
   * Creates a load-balancing endpoint (`saveEndpoint`, `type: "LB"`).
   *
   * @param input - `saveEndpoint` input fields.
   * @returns The new endpoint's id and name.
   */
  async createLoadBalancingEndpoint(
    input: Record<string, unknown>,
  ): Promise<{ id: string; name: string }> {
    const data = await this.graphql<{ saveEndpoint: { id: string; name: string } }>(
      `mutation { saveEndpoint(input: ${toGraphQLLiteral({ ...input, type: 'LB' })}) { id name } }`,
    )
    return data.saveEndpoint
  }
}

Functions

toGraphQLLiteral(value)

Serializes a plain object as a GraphQL input literal (unquoted keys, type values that are enums stay quoted strings — RunPod takes strings).

function toGraphQLLiteral(value: unknown): string

• value — The value.

Returns: The GraphQL literal.

gpuIdsFor(spec)

The GPU pools a spec may land on: the named pool, or every pool with enough memory.

function gpuIdsFor(spec: ModelEndpointSpec): string

• spec — The endpoint spec.

Returns: Comma-separated gpuIds.

toTemplate(spec, diskGb)

The serverless template body for a spec.

function toTemplate(spec: ModelEndpointSpec, diskGb: number): Record<string, unknown>

• spec — The endpoint spec.
• diskGb — Container disk.

Returns: TemplateCreateInput.

createProvider(config)

Creates a RunPod model hosting provider.

function createProvider(config?: RunPodConfig): ModelHostingProvider

• config — API key, locations, disk size.

Returns: A ModelHostingProvider backed by RunPod Serverless.

Constants

RUNPOD_REST

Default REST base URL.

const RUNPOD_REST: 'https://rest.runpod.io/v1'

RUNPOD_GRAPHQL

Default GraphQL URL.

const RUNPOD_GRAPHQL: 'https://api.runpod.io/graphql'

RUNPOD_GPU_POOLS

GPU pools by memory, smallest first (docs.runpod.io/sdks/graphql/manage-endpoints, 2026-09-30).

const RUNPOD_GPU_POOLS: readonly { id: string; vramGb: number }[]

RUNPOD_CAPABILITIES

What RunPod Serverless load-balancing endpoints offer here (GPU only).

const RUNPOD_CAPABILITIES: ProviderCapabilities

provider

The provider implementation — lazy, so env vars are read on first use.

const provider: ModelHostingProvider

modelHostingRunPodSecretDefinitions

Secret definitions used by the RunPod model hosting bond.

const modelHostingRunPodSecretDefinitions: SecretDefinition[]

Core Interface

Implements @molecule/api-model-hosting interface.

Bond Wiring

Setup function to register this provider with the core interface:

import { setProvider } from '@molecule/api-model-hosting'
import { provider } from '@molecule/api-model-hosting-runpod'

export function setupModelHostingRunpod(): void {
  setProvider(provider)
}

Injection Notes

Requirements

Peer dependencies:

• @molecule/api-bond ^1.0.1
• @molecule/api-i18n ^1.0.1
• @molecule/api-model-hosting ^1.0.0
• @molecule/api-secrets ^1.0.1

Environment Variables

• RUNPOD_API_KEY (required) — runpod api key
  • Setup: RunPod API key — Manages templates and serverless endpoints, and authorizes calls to them (Settings → API Keys). Workers stop when the account balance reaches $0.
  • Get it here: https://www.console.runpod.io/user/settings
  • Example: rpa_XXXXXXXXXXXXXXXXXXXXXXXX

Runtime Dependencies

• @molecule/api-bond

• @molecule/api-i18n

• @molecule/api-model-hosting

• @molecule/api-secrets

• Serve /ping on your port. RunPod's load balancer health-polls /ping (the HEALTH_CHECK_PATH override is documented but reported ignored — runpod/docs#853) and expects 200 healthy / 204 still loading. A server with only /health never becomes healthy; add /ping (or proxy it). The bond sets PORT, PORT_HEALTH and HEALTH_CHECK_PATH from the spec anyway, and deploy waits on YOUR healthPath through the public URL.

• Cold starts return "no workers available" until a worker is up; retry 3 times 5–10 s apart (RunPod's own guidance), or keep minInstances: 1.

• GPU only. minVramGb picks the smallest pool class that fits (AMPERE_16 → 16 GB; 24 GB → AMPERE_24,ADA_24; …); name one or a comma-separated set in accelerator.model. 'us' → US, 'eu' → CZ,FR,GB,NO,RO, or pass RunPod location codes.

• The endpoint is created with the GraphQL saveEndpoint mutation (type: "LB"), because the REST API cannot create load-balancing endpoints; updates, reads and deletes use REST. remove drains workers to 0 before deleting, then deletes the template.

• Anyone with your RunPod API key can call every endpoint — the key is both the admin credential and the endpoint credential. Keep it server-side.

• Workers stop when the account balance reaches $0 — a silent outage.

• list() returns every serverless endpoint on the account (RunPod does not say which are load-balancing).

• API errors throw RunPodApiError with the HTTP status.