@molecule/api-model-hosting-docker

Self-hosted model hosting for molecule.dev — run a model server container on a Docker engine you run (your GPU box, a rented bare-metal server), with NVIDIA GPUs through the NVIDIA container runtime.

The self-hosted option beside the paid hosts (Cloud Run, Modal, RunPod, Koyeb): same contract, your hardware, no per-second bill. Also a local stand-in for tests and development.

Quick Start

import { setProvider, requireProvider } from '@molecule/api-model-hosting'
import { provider } from '@molecule/api-model-hosting-docker'

setProvider(provider) // uses /var/run/docker.sock and http://localhost:<port>

const endpoint = await requireProvider().deploy({
  name: 'laya',
  image: 'ghcr.io/acme/laya-serve:0.3.1',
  port: 8000,
  healthPath: '/health',
  accelerator: { kind: 'cpu' },
  region: 'local',
  scaling: { minInstances: 1, maxInstances: 1, idleTimeoutSeconds: 0 },
  access: 'public',
})
endpoint.url // http://localhost:49153

Type

provider

Installation

npm install @molecule/api-model-hosting-docker @molecule/api-bond @molecule/api-i18n @molecule/api-model-hosting @molecule/api-secrets

API

Interfaces

ContainerInspect

The fields of GET /containers/{id}/json read here.

interface ContainerInspect {
  Id: string
  Name?: string
  Created?: string
  State?: { Status?: string; Running?: boolean; Error?: string }
  Config?: { Labels?: Record<string, string>; Image?: string }
  HostConfig?: { DeviceRequests?: Array<{ Count?: number }> | null }
  NetworkSettings?: { Ports?: Record<string, Array<{ HostIp?: string; HostPort?: string }> | null> }
}

DockerModelHostingConfig

Configuration for the Docker model hosting provider.

interface DockerModelHostingConfig {
  /** Engine socket. Defaults to `DOCKER_SOCKET`, then `/var/run/docker.sock`. */
  socketPath?: string
  /** Engine API version prefix. Default `'v1.47'`. */
  apiVersion?: string
  /** Host name the endpoint URLs use. Defaults to `MODEL_HOST_PUBLIC_HOST`, then `'localhost'`. */
  publicHost?: string
  /** Address published ports bind to. Default `'127.0.0.1'` (this machine only). */
  bindAddress?: string
  /** GPUs per container when a GPU is asked for: a count, or `'all'`. Default `1`. */
  gpus?: number | 'all'
  /** Custom transport (tests, a TCP/TLS engine). */
  transport?: DockerTransport
  /** How often `deploy` probes the health path, in ms. Default 2000. */
  pollIntervalMs?: number
}

DockerResponse

One Engine API response.

interface DockerResponse {
  /** HTTP status. */
  status: number
  /** Raw body text. */
  body: string
}

Types

DockerTransport

Sends one Engine API request (swappable for tests or a TLS remote).

type DockerTransport = (method: string, path: string, body?: unknown) => Promise<DockerResponse>

Classes

DockerApiError

A Docker Engine API error, carrying the HTTP status.

Functions

createProvider(config)

Creates a Docker model hosting provider.

function createProvider(config?: DockerModelHostingConfig): ModelHostingProvider

• config — Engine socket, public host, bind address, GPUs.

Returns: A ModelHostingProvider backed by a Docker engine.

dockerCall(transport, method, path, body)

Sends one request and parses JSON; null on 404, throws on other errors.

function dockerCall(
  transport: DockerTransport,
  method: string,
  path: string,
  body?: unknown,
): Promise<T | null>

• transport — The transport.
• method — HTTP method.
• path — Path after the version prefix.
• body — JSON body.

Returns: The parsed body (or raw text when not JSON), or null on 404.

socketTransport(socketPath, apiVersion)

The default transport: HTTP over the engine's Unix socket.

function socketTransport(socketPath: string, apiVersion: string): DockerTransport

• socketPath — The socket.
• apiVersion — Version prefix, e.g. 'v1.47'.

Returns: A transport.

splitImage(image)

Splits repo[:tag] (keeping a registry port) into fromImage and tag.

function splitImage(image: string): { fromImage: string; tag: string }

• image — Image reference.

Returns: The two query values.

toCreateBody(spec, bindAddress, gpus)

The POST /containers/create body for a spec.

function toCreateBody(
  spec: ModelEndpointSpec,
  bindAddress: string,
  gpus: number,
): Record<string, unknown>

• spec — The endpoint spec.
• bindAddress — Address the published port binds to.
• gpus — GPUs per container (-1 = all).

Returns: The request body.

Constants

CONTAINER_PREFIX

Container names are molecule-model-<spec.name>; that name is the endpoint id.

const CONTAINER_PREFIX: 'molecule-model-'

DOCKER_CAPABILITIES

What a self-hosted Docker engine offers here.

const DOCKER_CAPABILITIES: ProviderCapabilities

MANAGED_LABEL

Label every container this bond creates carries.

const MANAGED_LABEL: 'molecule-model-hosting'

modelHostingDockerSecretDefinitions

Settings used by the Docker model hosting bond.

const modelHostingDockerSecretDefinitions: SecretDefinition[]

provider

The provider implementation — lazy, so env vars are read on first use.

const provider: ModelHostingProvider

Core Interface

Implements @molecule/api-model-hosting interface.

Bond Wiring

Setup function to register this provider with the core interface:

import { setProvider } from '@molecule/api-model-hosting'
import { provider } from '@molecule/api-model-hosting-docker'

export function setupModelHostingDocker(): void {
  setProvider(provider)
}

Injection Notes

Requirements

Peer dependencies:

• @molecule/api-bond ^1.0.1
• @molecule/api-i18n ^1.0.1
• @molecule/api-model-hosting ^1.0.0
• @molecule/api-secrets ^1.0.1

Environment Variables

• DOCKER_SOCKET (optional) — docker socket
  • Setup: Docker engine socket — Path to the engine socket the model containers run on. Defaults to /var/run/docker.sock.
  • Get it here: https://docs.docker.com/engine/security/protect-access/
  • Example: /var/run/docker.sock
• MODEL_HOST_PUBLIC_HOST (optional) — model host public host
  • Setup: Model host name — The host name endpoint URLs use (the machine running the containers). Defaults to localhost.
  • Get it here: https://docs.docker.com/engine/network/
  • Example: gpu-box.internal

Runtime Dependencies

• @molecule/api-bond

• @molecule/api-i18n

• @molecule/api-model-hosting

• @molecule/api-secrets

• One always-on container per endpoint named molecule-model-<name> (that name is the endpoint id), restarted unless stopped. There is no scale to zero: minInstances must be ≥ 1 on deploy; scale(id, { minInstances: 0, … }) stops it and minInstances ≥ 1 starts it again. maxInstances and idleTimeoutSeconds are ignored.

• GPUs need the NVIDIA Container Toolkit on the host (DeviceRequests with driver nvidia). One GPU per container by default; gpus: 'all' in the config for every GPU. minVramGb is not checked — the host has what it has.

• Ports bind to 127.0.0.1 by default, on a port the engine picks (endpoint.url reads it back). Set bindAddress: '0.0.0.0' and publicHost (or MODEL_HOST_PUBLIC_HOST) to reach it from another machine — and then put TLS in front: URLs here are plain http://.

• No endpoint auth of its own. For access: 'private' set serverEnforcesAuth: true and pass the server's key in secretEnv; authHeaders() returns {}. secretEnv becomes container env (visible to anyone who can docker inspect).

• Access to the Docker socket is root on that machine. Run this only in a process that is allowed to be.

• deploy replaces an existing container of the same name (force-remove, then create) and waits until healthPath answers 2xx. A failed image pull throws DockerApiError with the registry's message.