@molecule/api-model-hosting

Model hosting for molecule.dev — run a container that serves a model on rented GPUs or CPUs, and get back an HTTPS URL plus the headers to call it.

One sentence: run this image on this hardware in this region, give me a URL and the headers to call it with, and let it scale between min and max. You pass the image, port, health path, hardware, region and scaling; the provider discovers nothing. Pair it with @molecule/api-ai-decisions to self-host Laya, Jeff, Intern-Decision or Kev — or with anything else that is a container behind an HTTP port.

This core defines the ModelHostingProvider contract and its accessor only. Bond one (or several, by name):

Bond	Where it runs	Scale to zero	Endpoint auth
@molecule/api-model-hosting-cloud-run	Google Cloud Run — CPU (request-billed, near free at low volume) or an NVIDIA L4	yes	IAM: a Google-signed ID token (authHeaders mints and refreshes it)
@molecule/api-model-hosting-modal	Modal — T4/L4/A10… or CPU, per-second billing, ~1 s container boot	yes	proxy auth (Modal-Key / Modal-Secret)
@molecule/api-model-hosting-runpod	RunPod Serverless load-balancing endpoints — 16 GB+ GPUs	yes	your RunPod API key as a bearer token
@molecule/api-model-hosting-koyeb	Koyeb — CPU or RTX 4000 / L4, EU or US	yes (sleep)	none — the server must check its own key
@molecule/api-model-hosting-docker	a Docker host you run (NVIDIA runtime for GPUs)	no	none — the server must check its own key

Quick Start

import { requireProvider, setProvider } from '@molecule/api-model-hosting'
import { provider as cloudRun } from '@molecule/api-model-hosting-cloud-run'
import { setProvider as setDecisions } from '@molecule/api-ai-decisions'
import { createProvider as createLaya } from '@molecule/api-ai-decisions-laya'

setProvider(cloudRun) // reads GOOGLE_CLOUD_PROJECT + GOOGLE_SERVICE_ACCOUNT_JSON on first use
const hosting = requireProvider()

const endpoint = await hosting.deploy({
  name: 'laya',
  image: 'us-docker.pkg.dev/acme/models/laya-serve:0.3.1',
  port: 8000,
  healthPath: '/health',
  accelerator: { kind: 'cpu' },
  cpu: 2,
  memoryMb: 4096,
  region: 'us',
  scaling: { minInstances: 0, maxInstances: 3, idleTimeoutSeconds: 900 },
  access: 'private',
})

setDecisions(createLaya({ baseUrl: endpoint.url, headers: () => hosting.authHeaders(endpoint.id) }))

Type

core

Installation

npm install @molecule/api-model-hosting @molecule/api-bond @molecule/api-i18n

API

Interfaces

CostEstimate

A monthly price with the inputs it came from.

interface CostEstimate {
  /** Always USD. */
  currency: 'USD'
  /** Estimated cost per 30-day month. */
  monthly: number
  /** Each price used: what, how much, per what, where it was read, and when. */
  basis: ReadonlyArray<{
    item: string
    unitPrice: number
    unit: string
    source: string
    verifiedAt: string
  }>
}

DeployOptions

Options for deploy.

interface DeployOptions {
  /** Progress lines (status changes, provider messages). */
  log?: (line: string) => void
  /** Give up waiting for the health path after this long (default is provider-specific). */
  timeoutMs?: number
}

EndpointScaling

How many instances may run, and when idle ones stop.

interface EndpointScaling {
  /** Instances kept running at all times. `0` = scale to zero when idle. */
  minInstances: number
  /** Upper bound on instances. */
  maxInstances: number
  /**
   * Seconds without traffic before an instance stops. Providers clamp this to
   * their own range (`capabilities.idleTimeoutRange`).
   */
  idleTimeoutSeconds: number
  /** Concurrent requests one instance takes before another starts (where the provider supports it). */
  concurrency?: number
}

ExpectedLoad

Traffic to price a spec at.

interface ExpectedLoad {
  /** Requests per day. */
  requestsPerDay: number
  /** Busy time one request takes on the chosen hardware, in ms. */
  busyMsPerRequest: number
}

ModelEndpoint

A deployed model server.

interface ModelEndpoint {
  /** Provider-scoped id; pass it to `get`, `scale`, `remove`, `authHeaders`. */
  id: string
  /** The spec's `name`. */
  name: string
  /** The provider's `name`. */
  provider: string
  /** Plain HTTPS (or, for self-hosted, HTTP) base URL of the server. */
  url: string
  /** Current status. */
  status: EndpointStatus
  /** Region it runs in. */
  region: string
  /** Hardware it runs on (as requested). */
  accelerator: Accelerator
  /** Scaling as last set. */
  scaling: EndpointScaling
  /** ISO timestamp, when the provider reports one. */
  createdAt?: string
  /** Last failure text from the provider, when `status` is `'failed'`. */
  error?: string
}

ModelEndpointSpec

Everything a provider needs to run one model server.

interface ModelEndpointSpec {
  /** Stable name. `deploy` with an existing name updates that endpoint in place. Lowercase letters, digits and `-`. */
  name: string
  /** OCI image reference, e.g. `'ghcr.io/acme/laya-serve:0.3.1'`. */
  image: string
  /** Port the server listens on inside the container. */
  port: number
  /** `GET` path that answers 2xx once the model is loaded (e.g. `'/health'`). */
  healthPath: string
  /**
   * Command that starts the server, when the image's own entrypoint does not.
   * Some providers require it (Modal starts the server from a Python function).
   */
  command?: string[]
  /** Plain environment variables. */
  env?: Readonly<Record<string, string>>
  /** Values the provider stores as secrets where it can, never logged. */
  secretEnv?: Readonly<Record<string, string>>
  /** Hardware. */
  accelerator: Accelerator
  /** vCPUs per instance (providers round to what they offer). */
  cpu?: number
  /** Memory per instance, in MB. */
  memoryMb?: number
  /** `'us'`, `'eu'`, or a provider region id (e.g. `'us-central1'`, `'fra'`). */
  region: string
  /** Scaling bounds. */
  scaling: EndpointScaling
  /**
   * `'private'`: only callers holding `authHeaders(id)` get through — on
   * providers WITHOUT their own endpoint auth (`capabilities.platformAuth`
   * false) you must also set `serverEnforcesAuth: true`, meaning the server
   * itself checks a key you passed in `secretEnv`. `'public'`: no auth at all
   * (tests only).
   */
  access: 'private' | 'public'
  /**
   * You confirm the model server checks its own API key (e.g. `LAYA_API_KEY`
   * passed in `secretEnv`). Required for `access: 'private'` on providers whose
   * `capabilities.platformAuth` is false; ignored elsewhere.
   */
  serverEnforcesAuth?: boolean
}

ModelHostingProvider

The model hosting provider every bond implements.

interface ModelHostingProvider {
  /** Provider identifier (`'cloud-run'`, `'modal'`, `'runpod'`, `'koyeb'`, `'docker'`). */
  readonly name: string
  /** What this provider supports. */
  readonly capabilities: ProviderCapabilities
  /**
   * Create the endpoint, or update it in place when one named `spec.name`
   * exists. Resolves once it serves traffic (or with `status: 'failed'`).
   */
  deploy(spec: ModelEndpointSpec, opts?: DeployOptions): Promise<ModelEndpoint>
  /** One endpoint by id, or `null` when it does not exist. */
  get(id: string): Promise<ModelEndpoint | null>
  /** Every endpoint this provider manages for you. */
  list(): Promise<ModelEndpoint[]>
  /** Change scaling without redeploying the image where the provider allows it. */
  scale(id: string, scaling: EndpointScaling): Promise<ModelEndpoint>
  /** Delete the endpoint and stop its billing. Removing a missing endpoint is not an error. */
  remove(id: string): Promise<void>
  /**
   * Headers to attach to every request to the endpoint. Async because some
   * are short-lived (Cloud Run ID tokens); providers cache and refresh them.
   * Empty for public endpoints and for providers without platform auth.
   */
  authHeaders(id: string): Promise<Record<string, string>>
  /**
   * Optional: price a spec at an expected load from the provider's dated list
   * prices (never guessed). Absent on providers that bill per account plan.
   */
  estimate?(spec: ModelEndpointSpec, load: ExpectedLoad): Promise<CostEstimate>
}

ProviderCapabilities

What a provider can do — check before deploying.

interface ProviderCapabilities {
  /** Accelerator ids it accepts, `'cpu'` first when CPU is offered. */
  accelerators: readonly string[]
  /** Region ids it accepts (plus the `'us'` / `'eu'` aliases it maps). */
  regions: readonly string[]
  /** Whether `minInstances: 0` actually stops billing for idle instances. */
  scaleToZero: boolean
  /** Inclusive range, in seconds, accepted for `idleTimeoutSeconds`. */
  idleTimeoutRange: readonly [number, number]
  /**
   * Whether the provider itself can refuse unauthenticated calls to an
   * endpoint (Cloud Run IAM, Modal proxy auth, RunPod API key). When false,
   * `access: 'private'` needs `serverEnforcesAuth`.
   */
  platformAuth: boolean
  /** The provider's own stated cold start, for display only. */
  coldStartHint?: string
}

Types

Accelerator

Hardware the caller asks for.

gpu picks the provider's cheapest GPU with at least minVramGb of memory, unless model names one exactly (a provider id such as 'nvidia-l4', 'nvidia-t4', 'AMPERE_16', 'gpu-nvidia-rtx-4000-sff-ada').

type Accelerator =
  | { kind: 'cpu' }
  | {
      kind: 'gpu'
      /** Smallest acceptable GPU memory, in GB. */
      minVramGb: number
      /** A provider-specific GPU id that overrides the `minVramGb` choice. */
      model?: string
    }

EndpointStatus

Lifecycle of an endpoint as the provider reports it.

type EndpointStatus = 'deploying' | 'ready' | 'scaled-to-zero' | 'failed' | 'removed'

Functions

assertDeployable(spec, capabilities, providerName)

Refuses a spec the provider cannot run as asked — before any API call, so a mistake never costs a half-created endpoint. Every bond calls this first.

function assertDeployable(
  spec: ModelEndpointSpec,
  capabilities: ProviderCapabilities,
  providerName: string,
): void

• spec — The endpoint spec.
• capabilities — The provider's capabilities.
• providerName — For the error message.

getAllProviders()

Every named provider, keyed by name.

function getAllProviders(): Map<string, ModelHostingProvider>

Returns: Map of provider name → provider.

getProvider()

The singleton provider, or null. Falls back to the only named provider when exactly one is bonded.

function getProvider(): ModelHostingProvider | null

Returns: The bonded provider, or null.

getProviderByName(name)

A named provider, or null if not bonded.

function getProviderByName(name: string): ModelHostingProvider | null

• name — The provider name.

Returns: The named provider, or null.

hasProvider(name)

Whether a provider is bonded.

function hasProvider(name?: string): boolean

• name — Optional provider name; omitted checks the singleton.

Returns: true when bonded.

requireProvider()

The bonded provider, throwing when none is bonded.

function requireProvider(): ModelHostingProvider

Returns: The bonded provider.

setProvider(provider)

Registers a model hosting provider in singleton mode.

function setProvider(provider: ModelHostingProvider): void

• provider — The default provider for this process.

Constants

ENDPOINT_NAME_RE

Names providers accept for an endpoint: lowercase, digits, -, starting with a letter.

const ENDPOINT_NAME_RE: RegExp

Available Providers

Provider	Package
Cloud Run model host	@molecule/api-model-hosting-cloud-run
Docker model host	@molecule/api-model-hosting-docker
Koyeb model host	@molecule/api-model-hosting-koyeb
Modal model host	@molecule/api-model-hosting-modal
RunPod model host	@molecule/api-model-hosting-runpod

Injection Notes

Requirements

Peer dependencies:

• @molecule/api-bond ^1.0.1
• @molecule/api-i18n ^1.0.1

Runtime Dependencies

• @molecule/api-bond

• @molecule/api-i18n

• Scale to zero is not free at steady traffic. A request every ~17 s (5,000 a day) keeps any instance with an idle window ≥ 60 s warm all day, so a GPU endpoint costs its always-on price during active hours. Scale to zero pays off for bursty or nightly work; batch GPU work to use it.

• Cold starts are real latency. The first request after idle waits for the instance AND the model to load (seconds on CPU, tens of seconds for a multi-GB GPU model). Keep minInstances: 1 for anything a user waits on, or retry — RunPod's own guidance is 3 retries 5–10 s apart.

• healthPath gates "ready". deploy resolves only when that path answers 2xx; a server whose health route answers before the model is loaded makes ready a lie.

• Small encoders do not need a GPU. Laya (421M) and GLiNER (340M) run on 2 vCPU / 4 GB; ask for { kind: 'cpu' } first and measure before renting a GPU. Qwen-based models (Jeff, Intern-Decision, Kev) need CUDA.

• Hosts that CANNOT do this (verified 2026-09-30): Fly.io has no GPUs any more; Cloudflare Workers AI only takes LoRA adapters on its own base models and Cloudflare Containers have no GPUs; SageMaker Serverless has no GPUs and needs SigV4 signing. There are no bonds for them on purpose.

• access: 'private' on a provider without endpoint auth (Koyeb, Docker — capabilities.platformAuth is false) is refused unless you set serverEnforcesAuth: true and pass the server's key in secretEnv (e.g. LAYA_API_KEY), then give the same key to the consumer (apiKey).

• authHeaders(id) is async and short-lived on Cloud Run (an ID token, about an hour). Call it per request — pass it as the consumer's headers hook — never store its result.

• Prices move. estimate() quotes dated list prices; re-check the vendor's page before committing to a spend.

• Every bond calls assertDeployable(spec, capabilities, name) before its first API call, so an impossible spec fails without creating anything.

• Use the core's setProvider, not bond('model-hosting', …) directly.