@molecule/api-content-moderation-openai

OpenAI content classifier for @molecule/api-content-moderation.

Scores text and images with OpenAI's moderation endpoint (omni-moderation-latest), which OpenAI does not charge for.

Quick Start

import { setClassifier, requireClassifier } from '@molecule/api-content-moderation'
import { classifier } from '@molecule/api-content-moderation-openai'

setClassifier(classifier) // reads OPENAI_API_KEY on first use

const verdict = await requireClassifier().check(post.body)
if (verdict.flagged) {
  // Reject, or hold for review, BEFORE saving.
}
const image = await requireClassifier().checkImage!(bytes, { mimeType: 'image/png' })

Type

provider

Installation

npm install @molecule/api-content-moderation-openai @molecule/api-content-moderation @molecule/api-secrets

API

Interfaces

OpenaiModerationConfig

Options for createClassifier. Every field falls back to an env var or a default, so the zero-argument classifier export works from .env alone.

interface OpenaiModerationConfig {
  /** OpenAI API key. Defaults to `OPENAI_API_KEY`. */
  apiKey?: string
  /** API origin. Defaults to `OPENAI_BASE_URL`, then `https://api.openai.com`. */
  baseUrl?: string
  /** Moderation model. Default `omni-moderation-latest` (text + images). */
  model?: string
  /** Per-request timeout in milliseconds. Default 30000. */
  timeoutMs?: number
}

ProcessEnv

Environment variables this classifier reads.

interface ProcessEnv {
  OPENAI_API_KEY?: string
  OPENAI_BASE_URL?: string
}

Classes

OpenaiModerationError

Error thrown when the moderation endpoint refuses or fails.

class OpenaiModerationError extends Error {
  /** HTTP status from OpenAI (0 when the request never completed). */
  readonly status: number

  /**
   * Create the error.
   *
   * @param message - Human-readable message.
   * @param status - HTTP status.
   */
  constructor(message: string, status: number) {
    super(message)
    this.name = 'OpenaiModerationError'
    this.status = status
  }
}

OpenaiContentClassifier

Classifier backed by OpenAI's moderation endpoint.

class OpenaiContentClassifier implements ContentClassifierProvider {
  readonly name = 'openai'

  private readonly apiKey: string
  private readonly baseUrl: string
  private readonly model: string
  private readonly timeoutMs: number

  /**
   * Create the classifier.
   *
   * @param config - Options; each falls back to its env var or default.
   */
  constructor(config: OpenaiModerationConfig = {}) {
    this.apiKey = config.apiKey ?? process.env.OPENAI_API_KEY ?? ''
    this.baseUrl = (
      config.baseUrl ??
      process.env.OPENAI_BASE_URL ??
      'https://api.openai.com'
    ).replace(/\/+$/, '')
    this.model = config.model ?? 'omni-moderation-latest'
    this.timeoutMs = config.timeoutMs ?? 30_000
  }

  /**
   * Classify text.
   *
   * @param content - The text.
   * @param options - Category filter and threshold.
   * @returns Per-category scores (highest first) and the overall decision.
   */
  async check(content: string, options?: ModerationOptions): Promise<ModerationResult> {
    return toModerationResult(await this.moderate([{ type: 'text', text: content }]), options)
  }

  /**
   * Classify an image (sent inline as a data URL).
   *
   * @param image - The image bytes.
   * @param options - Category filter, threshold and MIME type (default `image/jpeg`).
   * @returns Per-category scores (highest first) and the overall decision.
   */
  async checkImage(image: Uint8Array, options?: ImageModerationOptions): Promise<ModerationResult> {
    const mime = options?.mimeType ?? 'image/jpeg'
    const url = `data:${mime};base64,${Buffer.from(image).toString('base64')}`
    return toModerationResult(
      await this.moderate([{ type: 'image_url', image_url: { url } }]),
      options,
    )
  }

  /** One call to the moderation endpoint. */
  private async moderate(input: unknown[]): Promise<OpenaiModerationEntry> {
    if (!this.apiKey) {
      throw new OpenaiModerationError('OPENAI_API_KEY is not set.', 401)
    }
    const response = await fetch(`${this.baseUrl}/v1/moderations`, {
      method: 'POST',
      headers: { authorization: `Bearer ${this.apiKey}`, 'content-type': 'application/json' },
      body: JSON.stringify({ model: this.model, input }),
      signal: AbortSignal.timeout(this.timeoutMs),
    })
    if (!response.ok) {
      const detail = (await response.text()).slice(0, 300)
      throw new OpenaiModerationError(
        `OpenAI moderation failed (${response.status}): ${detail}`,
        response.status,
      )
    }
    const data = (await response.json()) as { results?: OpenaiModerationEntry[] }
    const entry = data.results?.[0]
    if (!entry || typeof entry.category_scores !== 'object') {
      throw new OpenaiModerationError('OpenAI moderation returned no result.', response.status)
    }
    return entry
  }
}

Functions

toModerationResult(entry, options)

Turn one moderation entry into the core's result, applying the caller's category filter and threshold.

• No threshold: a category is flagged exactly when OpenAI flagged it.
• With threshold: a category is flagged iff its score ≥ threshold.
• categories limits the returned categories AND the overall flagged.

function toModerationResult(
  entry: OpenaiModerationEntry,
  options?: ModerationOptions,
): ModerationResult

• entry — The endpoint's result for one input.
• options — Category filter and threshold.

Returns: The normalized result.

createClassifier(config)

Create an OpenAI content classifier.

function createClassifier(config?: OpenaiModerationConfig): ContentClassifierProvider

• config — Options; each falls back to its env var or default.

Returns: A ContentClassifierProvider.

Constants

classifier

The classifier implementation (wire with setClassifier).

const classifier: ContentClassifierProvider

contentModerationOpenaiSecretDefinitions

Secret definitions required by the OpenAI moderation bond.

const contentModerationOpenaiSecretDefinitions: SecretDefinition[]

Core Interface

Implements @molecule/api-content-moderation interface.

Injection Notes

Requirements

Peer dependencies:

• @molecule/api-content-moderation >=1.1.0
• @molecule/api-secrets ^1.0.1

Environment Variables

• OPENAI_API_KEY (required) — OpenAI API key
  • Setup: Create a secret key on the OpenAI platform (API keys page).
  • Get it here: https://platform.openai.com/api-keys
  • Example: sk-proj-...

Runtime Dependencies

• @molecule/api-content-moderation

• @molecule/api-secrets

• Wire with setClassifier — NOT setProvider: this is a classifier (scores only), not the full ContentModerationProvider with user reports.

• Config: OPENAI_API_KEY (SERVER-side only); optional OPENAI_BASE_URL.

• Categories are OpenAI's names: harassment, harassment/threatening, hate, hate/threatening, illicit, illicit/violent, self-harm, self-harm/intent, self-harm/instructions, sexual, sexual/minors, violence, violence/graphic. Results are sorted highest score first.

• Without threshold the verdict is OpenAI's own. With threshold, a category is flagged iff its score ≥ threshold — lower it to be stricter. Pass categories to decide on a subset (e.g. only sexual/minors).

• Errors are OpenaiModerationError with status; nothing is retried. Decide per surface whether a failed check blocks the content or holds it for review — never publish silently on an exception.

• No OpenAI account? @molecule/api-content-moderation-molecule exposes the same classifier through molecule.dev, billed to the molecule project.

E2E Tests

Integration checklist — drive the real UI (live preview, no mocks), adapt each item to this app's actual screens/flows, and check every box off one by one. A box you can't check is an integration bug to fix — not a skip:

• Every surface that accepts user-generated content (post, comment, image upload, bio — whatever this app has) runs it through check() / checkImage() SERVER-SIDE in the create/update handler, before persisting. Confirm the moderation call is on the write path, not client-side or skipped.
• Assert BOTH directions with real samples: clearly-violating content returns flagged: true and the UI rejects it with a visible reason; benign content returns flagged: false and publishes normally. A moderator that flags everything or nothing is broken.
• The decision gates persistence: a blocked item is NOT stored and is absent when you view the feed as a second user. Tighten ModerationOptions.threshold and a borderline item flips allowed → blocked.
• The failure mode is deliberate: when the moderation/AI call errors the request does NOT 500 — content is either published + logged (fail-open) or held for review (fail-closed) per the surface's intent.
• If the app has reporting, the round-trip works: a user's report() creates a 'pending' Report, it appears in the moderator queue via getReports(), and resolveReport() (approve/reject/dismiss) visibly changes its status and clears it from the pending queue.
• Moderator surfaces are privileged: a non-admin can't reach getReports() / resolveReport() (403) and can't see other users' flagged or pending content.