@molecule/api-ai-local | Molecule.dev

Local (OpenAI-compatible) ai-local provider for molecule.dev.

Streams chat completions from any local inference server that speaks the OpenAI chat/completions protocol (Ollama, LM Studio, llama.cpp, vLLM), keyless by default.

Auto-generated, AI-first package reference for the molecule.dev ecosystem. It is written to be read by coding agents as much as by people, and is generated from this package's source — edit src/index.ts JSDoc, not this file.

Quick Start

// npm install @molecule/api-ai-local --workspace=api
import { setProvider, requireProvider } from '@molecule/api-ai'
import { provider } from '@molecule/api-ai-local'

// Named registration — several AI providers can be bonded side by side
// (getProviderByName('local') targets this one); the FIRST one
// registered also answers requireProvider() (see @molecule/api-ai).
setProvider('local', provider) // keyless: LOCAL_AI_BASE_URL (Ollama http://localhost:11434/v1 by default), LOCAL_AI_MODEL (llama3.1)

let reply = ''
for await (const event of requireProvider().chat({
  messages: [{ role: 'user', content: 'Hello!' }],
})) {
  if (event.type === 'text') reply += event.content // or forward the chunk to the client (SSE)
}

Type

provider

Installation

npm install @molecule/api-ai-local @molecule/api-ai @molecule/api-bond @molecule/api-i18n @molecule/api-secrets

API

Interfaces

`LocalConfig`

Local (OpenAI-compatible) provider configuration.

interface LocalConfig {
  /** Called on each rate-limited/busy upstream response, before any retry sleep. */
  onRateLimit?: AiRateLimitCallback
  /**
   * Base URL of the OpenAI-compatible endpoint, INCLUDING the version segment
   * (e.g. `http://localhost:11434/v1`). Overrides `LOCAL_AI_BASE_URL` /
   * `OLLAMA_BASE_URL`. Defaults to Ollama's `http://localhost:11434/v1`.
   */
  baseUrl?: string
  /**
   * Optional API key. Most local servers ignore auth; when omitted (and no
   * `LOCAL_AI_API_KEY` env var is set) no `Authorization` header is sent.
   */
  apiKey?: string
  /**
   * Default model when a call doesn't specify one. Overrides `LOCAL_AI_MODEL`.
   * Defaults to `llama3.1`.
   */
  model?: string
}

Classes

`LocalAIProvider`

Local (OpenAI-compatible) chat provider implementing the AIProvider interface. Mirrors @molecule/api-ai-openai so the same handler code can dispatch to a local endpoint.

class LocalAIProvider implements AIProvider {
  readonly name = 'local'
  private apiKey: string
  private defaultModel: string
  private maxTokens: number
  private baseUrl: string
  private onRateLimit?: AiRateLimitCallback

  constructor(config: LocalConfig = {}) {
    // Keyless by default — many local servers ignore auth. Empty string means
    // "no key", so no Authorization header is sent. NEVER throws for a missing
    // key at construction.
    this.apiKey = config.apiKey ?? process.env.LOCAL_AI_API_KEY ?? ''
    this.defaultModel = config.model ?? process.env.LOCAL_AI_MODEL ?? DEFAULT_MODEL
    this.maxTokens = DEFAULT_MAX_TOKENS
    this.onRateLimit = config.onRateLimit
    this.baseUrl =
      config.baseUrl ??
      process.env.LOCAL_AI_BASE_URL ??
      process.env.OLLAMA_BASE_URL ??
      DEFAULT_BASE_URL
  }

  /**
   * Send a chat request and yield streamed `ChatEvent`s.
   *
   * @param params - Chat parameters.
   * @yields {ChatEvent}
   */
  async *chat(params: ChatParams): AsyncIterable<ChatEvent> {
    const model = params.model ?? this.defaultModel
    const maxTokens = params.maxTokens ?? this.maxTokens

    const messages = this.formatMessages(params.messages, params.system)

    const body: Record<string, unknown> = {
      // Abuse attribution (see ChatParams.endUserId): OpenAI-compatible APIs
      // scope enforcement by `user`, so an abusive tenant can be actioned
      // individually instead of the org key that serves everyone. The value is
      // the caller's opaque hash — never derived or logged here.
      ...(params.endUserId ? { user: params.endUserId } : {}),
      model,
      messages,
      max_tokens: maxTokens,
    }
    if (params.temperature !== undefined) body.temperature = params.temperature

    // Reasoning depth: the caller resolves the model's native reasoning_effort
    // value from the model catalog and passes it as `thinking.effort`. Local
    // servers that don't support it typically ignore an unknown field.
    if (params.thinking?.effort) {
      body.reasoning_effort = params.thinking.effort
    }

    const tools = params.tools?.length ? this.formatTools(params.tools) : null
    if (tools) {
      body.tools = tools
      if (params.toolChoice === 'required') {
        body.tool_choice = 'required'
      } else if (typeof params.toolChoice === 'object' && params.toolChoice?.type === 'tool') {
        body.tool_choice = { type: 'function', function: { name: params.toolChoice.name } }
      }
    }

    const useStream = params.stream !== false
    if (useStream) {
      body.stream = true
      body.stream_options = { include_usage: true }
    }

    // Per-call correlation (see ChatParams.requestId): a header rather than the
    // body's `user`, the stable per-account abuse-attribution value.
    const requestId = params.requestId ?? randomUUID()
    const call: LocalCallContext = {
      meta: { requestId, endpoint: this.baseUrl },
      endUserId: params.endUserId,
    }

    const headers: Record<string, string> = {
      'Content-Type': 'application/json',
      'X-Mol-Request-Id': requestId,
    }
    // Only authenticate when a key is configured — local servers are keyless.
    if (this.apiKey) headers.Authorization = `Bearer ${this.apiKey}`

    const DEFAULT_TIMEOUT_MS = 5 * 60 * 1000
    const signal = params.signal ?? AbortSignal.timeout(DEFAULT_TIMEOUT_MS)

    const url = `${this.baseUrl}/chat/completions`

    try {
      const MAX_RETRIES = 3
      let response: Response | null = null
      for (let attempt = 0; attempt <= MAX_RETRIES; attempt++) {
        response = await fetch(url, {
          method: 'POST',
          headers,
          body: JSON.stringify(body),
          signal,
        })

        if (response.status === 429 || response.status === 503) {
          const willRetry = attempt < MAX_RETRIES
          // Retry-After is equally-valid as delta-seconds or an HTTP-date;
          // parseInt on the date form yields NaN, degrading the backoff to
          // a ~0ms retry against an already-busy local endpoint. Guard it.
          const retryAfter = response.headers.get('retry-after')
          const parsedRetryAfterSeconds = retryAfter ? parseInt(retryAfter, 10) : NaN
          const hasRetryAfter =
            Number.isFinite(parsedRetryAfterSeconds) && parsedRetryAfterSeconds >= 0
          // Additive jitter (<500ms): concurrent callers rejected together must
          // not sleep identical delays and re-arrive as the same burst.
          const delayMs = willRetry
            ? (hasRetryAfter
                ? Math.min(parsedRetryAfterSeconds * 1000, 60_000)
                : Math.min(1000 * 2 ** attempt, 30_000)) + Math.floor(Math.random() * 500)
            : 0
          try {
            this.onRateLimit?.({
              provider: this.name,
              model,
              status: response.status,
              attempt: attempt + 1,
              willRetry,
              retryInMs: delayMs,
              ...(hasRetryAfter ? { retryAfterSeconds: parsedRetryAfterSeconds } : {}),
            })
          } catch (error) {
            // Host-app telemetry; its failure must never break the request.
            logger.warn('onRateLimit callback threw', { error })
          }
          if (willRetry) {
            logger.warn('Local AI endpoint busy, retrying', {
              status: response.status,
              attempt: attempt + 1,
              delayMs,
            })
            await new Promise<void>((resolve) => {
              const timer = setTimeout(resolve, delayMs)
              signal.addEventListener(
                'abort',
                () => {
                  clearTimeout(timer)
                  resolve()
                },
                { once: true },
              )
            })
            if (signal.aborted) break
            continue
          }
        }
        break
      }

      if (!response!.ok) {
        const errorBody = await response!.text()
        let detail = `HTTP ${response!.status}`
        try {
          const parsed = JSON.parse(errorBody) as { error?: { message?: string } }
          if (parsed.error?.message) detail = parsed.error.message
        } catch (_error) {
          // Best-effort: detail already has an HTTP-status fallback; malformed body just stays as-is.
          if (errorBody.length > 0 && errorBody.length < 200) detail = errorBody
        }
        logger.error('Local AI endpoint error', { status: response!.status, url, detail })
        const clientMessage =
          response!.status === 429
            ? 'AI rate limit exceeded. Please try again shortly.'
            : response!.status === 401 || response!.status === 403
              ? 'AI service configuration error.'
              : response!.status === 404
                ? 'Local AI endpoint or model not found. Is the server running and the model pulled?'
                : response!.status === 400 &&
                    /context.*length|maximum context|too many tokens|prompt is too long/i.test(
                      detail,
                    )
                  ? "Conversation too long for the model's context window."
                  : response!.status === 400
                    ? 'AI request was invalid — check the model and request parameters.'
                    : response!.status === 503 || response!.status === 502
                      ? 'AI service is temporarily overloaded. Please try again in a moment.'
                      : 'AI service error. Please try again.'
        // Non-OK is a soft failure surfaced as an in-band error EVENT then a
        // graceful return — matching the 8 sibling providers. molecule-dev's
        // chat-handler consumes this via its `case 'error':` branch, which is a
        // different path from its outer try/catch; throwing here would break
        // that consumer contract.
        yield { type: 'error', message: clientMessage, errorKey: 'ai.error.apiError' }
// … truncated — the full declaration is in node_modules/@molecule/api-ai-local/dist/provider.d.ts

Functions

`createProvider(config)`

Create a local (OpenAI-compatible) AI provider instance.

Constructs WITHOUT requiring any secret — local endpoints run keyless.

function createProvider(config?: LocalConfig): AIProvider

config — Local provider configuration.

Returns: An AIProvider backed by an OpenAI-compatible local endpoint.

Constants

`provider`

The provider implementation. Constructs keyless — no secret required.

const provider: AIProvider

`aiLocalSecretDefinitions`

Optional secret/config overrides recognised by the local AI bond.

const aiLocalSecretDefinitions: SecretDefinition[]

Core Interface

Implements @molecule/api-ai interface.

Bond Wiring

Setup function to register this provider with the bond system:

import { bond } from '@molecule/api-bond'
import { provider } from '@molecule/api-ai-local'

export function setupAiLocal(): void {
  bond('ai', 'local', provider)
}

Injection Notes

Requirements

Peer dependencies:

@molecule/api-ai ^1.0.1
@molecule/api-bond ^1.0.1
@molecule/api-i18n ^1.0.1
@molecule/api-secrets ^1.0.1

Runtime Dependencies

@molecule/api-ai
@molecule/api-bond
@molecule/api-i18n
@molecule/api-secrets

Error message disambiguation: a plain 400 that ISN'T a context-length error (bad param, malformed tool schema) gets its own non-retryable message distinct from the generic "AI service error. Please try again." used for retryable failures.

E2E Tests

Integration checklist — drive the real UI (live preview, no mocks), adapt each item to this app's actual chat/AI screens, and check every box off one by one. A box you can't check is an integration bug to fix — not a skip. The sandbox HAS an AI provider bonded, so the flow runs live end-to-end; AI output is NON-DETERMINISTIC, so assert on STRUCTURE/behavior, not exact text:

A message sent through the real chat UI comes back as a RELEVANT AI reply — not an echo of the prompt, a hardcoded stub, or an empty bubble. Ask something with a checkable answer (e.g. "What is 2 + 2?") and confirm the response actually contains it ("4"), proving a live model answered.
If the app streams, tokens render INCREMENTALLY — text grows word by word in the UI, not one final blob dumped after a long frozen spinner. (A streamed chat() yields text chunks then a final done; a single late blob means the reply was awaited whole and streaming is broken.)
Multi-turn CONTEXT is preserved: a follow-up that refers back to the previous turn (e.g. after "2 + 2", ask "now double that" -> understood as 8) works — proving the full messages history is sent, not just the last line.
A provider failure (bad/missing key, rate limit, timeout) surfaces as a graceful in-UI error message, NOT a crash, blank screen, a spinner that never resolves, or an unhandled 500. Force one and watch the UI recover.
The provider key + the provider call are SERVER-side only: the key never reaches the browser (check the network tab, the JS bundle, and page globals), and no route proxies arbitrary prompts to the model without auth
- a token cap — an open AI endpoint is an unbounded bill and abuse vector.

`@molecule/api-ai-local`

How it works

Reference