@molecule/api-uploads-s3

AWS S3 upload provider for molecule.dev.

Handles file uploads to AWS S3.

Auto-generated, AI-first package reference for the molecule.dev ecosystem. It is written to be read by coding agents as much as by people, and is generated from this package's source — edit src/index.ts JSDoc, not this file.

Quick Start

import { setProvider } from '@molecule/api-uploads'
import { provider } from '@molecule/api-uploads-s3'

// Default store: config is env-only, server-side — see the first remark for the full list.
setProvider(provider)

import { createProvider } from '@molecule/api-uploads-s3'

// A second, independent store (e.g. backups on Backblaze B2), alongside the default provider.
export const backups = createProvider({
  bucket: process.env.BACKUP_S3_BUCKET!,
  endpoint: process.env.BACKUP_S3_ENDPOINT,
  region: process.env.BACKUP_S3_REGION,
  credentials: {
    accessKeyId: process.env.BACKUP_S3_ACCESS_KEY_ID!,
    secretAccessKey: process.env.BACKUP_S3_SECRET_ACCESS_KEY!,
  },
  storageClass: 'STANDARD_IA',
  keyPrefix: 'nightly/',
})

const file = backups.upload('dump', dumpStream, info, onError) // file.id === 'nightly/<uuid>'
await file.uploadPromise
const head = await backups.headFile(file.id) // { bytes, etag, lastModified } | null

import { bucketProtectionMeets, createProvider } from '@molecule/api-uploads-s3'

// An immutable backup bucket: every object locked for 30 days, and the bucket itself checked.
const vault = createProvider({
  bucket: process.env.VAULT_S3_BUCKET!,
  objectLock: { mode: 'COMPLIANCE', retainDays: 30 },
  checksumAlgorithm: 'SHA256',
})

const check = bucketProtectionMeets(await vault.describeBucketProtection(), {
  mode: 'COMPLIANCE',
  minDays: 30,
})
if (!check.ok) throw new Error(`backup bucket is not protected: ${check.reasons.join('; ')}`)

Type

provider

Installation

npm install @molecule/api-uploads-s3 @aws-sdk/client-s3 @aws-sdk/lib-storage @molecule/api-analytics @molecule/api-bond @molecule/api-i18n @molecule/api-proxy-agent @molecule/api-secrets @molecule/api-uploads uuid

API

Interfaces

S3FileHead

Metadata of a stored object, as returned by headFile().

interface S3FileHead {
  /** Object size in bytes. */
  bytes: number
  /** The object's ETag, when the store returns one. */
  etag?: string
  /** When the object was last written, when the store returns it. */
  lastModified?: Date
}

BucketProtection

A bucket's protection against overwrite and deletion, as reported by describeBucketProtection().

interface BucketProtection {
  /** Bucket versioning state; `'off'` when versioning was never enabled. */
  versioning: 'Enabled' | 'Suspended' | 'off'
  /** The bucket's Object Lock configuration. */
  objectLock: {
    /** Whether Object Lock is enabled on the bucket. */
    enabled: boolean
    /** Mode of the bucket's default retention rule, when one is set. */
    mode?: ObjectLockRetentionMode
    /** Default retention in days, when the rule is expressed in days. */
    days?: number
    /** Default retention in years, when the rule is expressed in years. */
    years?: number
  }
  /** ISO-8601 time the bucket was checked. */
  checkedAt: string
}

S3UploadProvider

An S3 upload provider bound to one store. Implements UploadProvider and also exposes its client, its bucket and a metadata lookup.

interface S3UploadProvider extends UploadProvider {
  /** The store's S3 client (created on first use). */
  readonly client: S3Client
  /** The bucket this provider reads and writes. */
  readonly bucket: string
  /**
   * Streams a file into this store under a newly minted key (prefixed with `keyPrefix` when set).
   * @param fieldname - The form field name this file was submitted under.
   * @param stream - The readable stream of the file data.
   * @param info - File metadata (filename, encoding, mimeType).
   * @param onError - Called if the upload fails or the stream exceeds its size limit.
   * @returns The file record; its `id` is the full object key.
   */
  upload(
    fieldname: string,
    stream: NodeJS.ReadableStream,
    info: FileInfo,
    onError: (error: Error) => void,
  ): File
  /**
   * Aborts an in-progress upload; its `uploadPromise` rejects with `UploadAbortedError`.
   * @param file - The file record returned by `upload()`.
   */
  abortUpload(file: File): Promise<void>
  /**
   * Downloads a file.
   * @param id - The file id returned by `upload()`.
   * @returns A readable stream, or `null` if the file does not exist.
   */
  getFile(id: string): Promise<NodeJS.ReadableStream | null>
  /**
   * Looks up an object's size, ETag and modification time without downloading it.
   * @param id - The file id returned by `upload()` (includes any `keyPrefix`).
   * @returns The object's metadata, or `null` if it does not exist.
   */
  headFile(id: string): Promise<S3FileHead | null>
  /**
   * Reads the bucket's versioning state and Object Lock configuration (including
   * its default retention), so an app can assert at boot and periodically that an
   * immutable bucket is still immutable. A bucket with no Object Lock
   * configuration reports `objectLock.enabled: false`; any other failure (auth,
   * missing bucket, network) throws.
   * @returns The bucket's protection.
   */
  describeBucketProtection(): Promise<BucketProtection>
}

File

S3-uploaded file extending the core UploadedFile with S3-specific abort capabilities.

interface File extends UploadedFile {
  /**
   * Aborts the in-progress S3 upload.
   */
  abort?: () => Promise<void>
}

S3UploadsConfig

Configuration for one S3-compatible store, passed to createProvider().

Every field except bucket is optional; an omitted field falls back to the AWS SDK's own default (region, endpoint, credentials) or to this bond's default (timeouts, retries). Nothing here is read from the environment — use configFromEnv() for the env-driven configuration the default provider uses.

interface S3UploadsConfig {
  /** The bucket every object of this store is written to and read from. */
  bucket: string

  /**
   * The bucket's region.
   *
   * @default 'us-east-1'
   */
  region?: string

  /**
   * Service endpoint of an S3-compatible store (Cloudflare R2, MinIO,
   * DigitalOcean Spaces, Tigris, Backblaze B2, …). When unset the SDK resolves
   * the AWS regional endpoint. Outbound proxy agents are resolved against this
   * endpoint (or the regional one), so a `NO_PROXY`-listed internal host keeps
   * connecting directly.
   */
  endpoint?: string

  /** Use path-style addressing (`endpoint/bucket/key`), required by e.g. MinIO. */
  forcePathStyle?: boolean

  /**
   * Static credentials for this store. When set they take precedence over the
   * SDK's default chain (`AWS_ACCESS_KEY_ID`/`AWS_SECRET_ACCESS_KEY`, profiles,
   * instance roles); when unset that chain is used.
   */
  credentials?: {
    /** Access key id. */
    accessKeyId: string
    /** Secret access key. */
    secretAccessKey: string
    /** Session token, for temporary credentials. */
    sessionToken?: string
  }

  /**
   * Storage class sent as `StorageClass` on every upload (e.g. `STANDARD_IA`,
   * `GLACIER_IR`). When unset the bucket's default class applies.
   */
  storageClass?: string
  /**
   * Multipart part size in bytes (the SDK's default is 5 MiB, its minimum). A
   * multipart upload holds at most 10,000 parts, so the part size bounds the
   * largest object: ~48.8 GiB at the default. Set it from your size ceiling
   * (`ceil(maxBytes / 10000)`, at least 5 MiB) when streaming larger bodies.
   */
  partSizeBytes?: number

  /**
   * Prefix prepended to every object key this store mints (e.g. `backups/`).
   * The id returned by `upload()` includes it, so ids stay opaque: pass them
   * back to `getFile`/`deleteFile`/`headFile` unchanged.
   */
  keyPrefix?: string

  /**
   * Per-object S3 Object Lock retention, asserted on every upload this provider
   * makes (single PutObject and multipart alike): `ObjectLockMode` is `mode` and
   * `ObjectLockRetainUntilDate` is `retainDays` days after the moment that
   * upload starts. Use it for a backup/immutable bucket, so retention does not
   * depend only on the bucket's default rule (which a full-access credential can
   * shorten or remove).
   *
   * The bucket must have been created with Object Lock enabled; a PUT carrying
   * lock headers to a bucket without it is rejected. An Object Lock PUT also
   * needs a content checksum (Content-MD5 or an `x-amz-checksum-*`) — see
   * `checksumAlgorithm`. When `objectLock` is set and `checksumAlgorithm` is not,
   * `ChecksumAlgorithm: 'CRC32'` is sent explicitly, so an
   * `AWS_REQUEST_CHECKSUM_CALCULATION=WHEN_REQUIRED` environment cannot drop it.
   * `createProvider()` throws if `retainDays` is not a positive number or if
   * `checksumAlgorithm` is `'none'`.
   */
  objectLock?: {
    /** `COMPLIANCE`: no one can shorten or remove it. `GOVERNANCE`: holders of `s3:BypassGovernanceRetention` can. */
    mode: ObjectLockRetentionMode
    /** Retention length in days, counted from each upload's start. */
    retainDays: number
  }

  /**
   * Checksum algorithm sent as `ChecksumAlgorithm` on every upload. When unset the
   * AWS SDK default applies (`requestChecksumCalculation: 'WHEN_SUPPORTED'`: a
   * CRC32 checksum sent as a trailer). `'none'` turns the SDK's automatic
   * checksum off for this store (`requestChecksumCalculation: 'WHEN_REQUIRED'`),
   * for an S3-compatible store that rejects checksum trailers; it cannot be
   * combined with `objectLock`.
   *
   * S3-compatible stores differ in which checksums (and whether streamed
   * trailers) they accept — test an Object Lock upload against the real store
   * before relying on it (e.g. Hetzner Object Storage). `'SHA256'` is the most
   * portable explicit choice.
   */
  checksumAlgorithm?: 'CRC32' | 'CRC32C' | 'SHA1' | 'SHA256' | 'none'

  /**
   * Milliseconds to wait for a connection to open.
   *
   * @default 10000
   */
  connectionTimeoutMs?: number

  /**
   * Milliseconds of socket inactivity before a request is abandoned.
   *
   * @default 60000
   */
  socketTimeoutMs?: number

  /**
   * Total attempts per request, including the first.
   *
   * @default 3
   */
  maxAttempts?: number
}

UploadedFile

Properties describing an uploading/uploaded file.

interface UploadedFile {
  /**
   * The unique file identifier.
   */
  id: string
  /**
   * The file's fieldname from the form.
   * Used as the key for the file within `req.files` - e.g., `req.files[fieldname] = file`.
   */
  fieldname: string
  /**
   * The original filename - e.g., `some-image.jpg`.
   */
  filename: string
  /**
   * The file's encoding - e.g., `binary`.
   */
  encoding: string
  /**
   * The file's mimetype - e.g., `image/jpeg`.
   */
  mimetype: string
  /**
   * The file's size in bytes.
   */
  size: number
  /**
   * The source stream (available during upload).
   */
  stream?: NodeJS.ReadableStream
  /**
   * A promise that resolves when the upload completes.
   */
  uploadPromise?: Promise<void>
  /**
   * Whether the upload has completed.
   */
  uploaded: boolean
  /**
   * The URL/location of the uploaded file (if available).
   */
  location?: string
}

FileInfo

Information about a file being uploaded.

This interface is provider-agnostic. Implementations using busboy or other multipart parsers should adapt to this interface.

interface FileInfo {
  /**
   * The original filename - e.g., `some-image.jpg`.
   */
  filename: string
  /**
   * The file's encoding - e.g., `7bit`, `binary`.
   */
  encoding: string
  /**
   * The file's MIME type - e.g., `image/jpeg`.
   */
  mimeType: string
}

Types

ObjectLockRetentionMode

S3 Object Lock retention mode.

type ObjectLockRetentionMode = 'COMPLIANCE' | 'GOVERNANCE'

Functions

bucketProtectionMeets(protection, want)

Checks a bucket's reported protection against a required minimum: versioning Enabled, Object Lock enabled, a default retention mode at least as strict as wanted (COMPLIANCE satisfies a GOVERNANCE want, not the reverse) and a default retention of at least minDays (years count as 365 days).

function bucketProtectionMeets(
  protection: BucketProtection,
  want: { mode: ObjectLockRetentionMode; minDays: number },
): { ok: boolean; reasons: string[] }

• protection — The result of describeBucketProtection().
• want — The required mode and minimum default retention in days.
• want.mode — The weakest acceptable default retention mode.
• want.minDays — The shortest acceptable default retention, in days.

Returns: ok, and one plain-language reason per unmet requirement.

configFromEnv()

Builds the store configuration the default provider uses, from the environment.

Reads AWS_S3_BUCKET (or BUCKET_NAME), AWS_S3_ENDPOINT (or AWS_ENDPOINT_URL_S3), AWS_S3_REGION (or AWS_REGION, default us-east-1), AWS_S3_FORCE_PATH_STYLE, AWS_S3_CONNECTION_TIMEOUT_MS and AWS_S3_SOCKET_TIMEOUT_MS. Credentials are deliberately left unset so the AWS SDK's own chain (AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY, profiles, roles) resolves them.

function configFromEnv(): S3UploadsConfig

Returns: A snapshot of the env-driven configuration.

createProvider(config)

Creates an S3 upload provider bound to one store — its own client, endpoint, credentials, bucket, storage class and key prefix — independent of the default env-configured provider and of every other instance.

The client is created on first use. Upload, abort, delete and get behave exactly like the default provider's, against this store. With objectLock, every upload carries its own Object Lock retention.

function createProvider(config: S3UploadsConfig): S3UploadProvider

• config — The store configuration.

Returns: A provider for that store.

upload(fieldname, stream, info, onError)

Streams a file upload to S3 (default provider) using the @aws-sdk/lib-storage multipart Upload utility. Creates a UUID key in the configured S3 bucket and pipes the readable stream into it.

function upload(
  fieldname: string,
  stream: NodeJS.ReadableStream,
  info: FileInfo,
  onError: (error: Error) => void,
): File

• fieldname — The form field name this file was submitted under.
• stream — The readable stream of the uploaded file data.
• info — File metadata (filename, encoding, mimeType) from the multipart parser.
• onError — Callback invoked if the S3 upload fails or the stream exceeds its size limit.

Returns: A File object with the upload's ID, metadata, and a uploadPromise that resolves on completion.

abortUpload(file)

Aborts an in-progress S3 upload (default provider). Removes stream listeners and calls the S3 multipart abort. Rejects the file's uploadPromise with UploadAbortedError — never as a success, and never routed through the upload() call's onError (parity with the filesystem bond).

function abortUpload(file: File): Promise<void>

• file — The File object returned by upload().

deleteFile(id)

Deletes a file from S3 (default provider) by its key using DeleteObjectCommand.

function deleteFile(id: string): Promise<void>

• id — The UUID file identifier (S3 object key).

getFile(id)

Downloads a file from S3 (default provider) by its key using GetObjectCommand.

function getFile(id: string): Promise<NodeJS.ReadableStream | null>

• id — The UUID file identifier (S3 object key).

Returns: A readable stream of the file contents, or null if the file does not exist.

Constants

s3Client

Lazily-initialized S3 client proxy for the default provider. Property access is forwarded to the real client on first use.

const s3Client: S3Client

provider

The default S3 upload provider, configured from the environment (see configFromEnv()): stores files in the bucket named by AWS_S3_BUCKET (or BUCKET_NAME). Use createProvider() for additional stores.

const provider: S3UploadProvider

uploadsS3SecretDefinitions

Secret definitions required by the S3 uploads bond.

const uploadsS3SecretDefinitions: SecretDefinition[]

Core Interface

Implements @molecule/api-uploads interface.

Bond Wiring

Setup function to register this provider with the core interface:

import { setProvider } from '@molecule/api-uploads'
import { provider } from '@molecule/api-uploads-s3'

export function setupUploadsS3(): void {
  setProvider(provider)
}

Injection Notes

Requirements

Peer dependencies:

• @molecule/api-bond ^1.0.1
• @molecule/api-i18n ^1.0.1
• @molecule/api-secrets ^1.0.1
• @molecule/api-uploads ^1.0.1

Environment Variables

• AWS_ACCESS_KEY_ID (required) — AWS access key ID
  • Setup: Create an IAM user with the needed policy (SES/S3/SQS) and create an access key under Security credentials.
  • Get it here: https://console.aws.amazon.com/iam/
  • Example: AKIA...
• AWS_SECRET_ACCESS_KEY (required) — AWS secret access key
  • Setup: Shown once when creating the IAM access key — store it immediately.
  • Get it here: https://console.aws.amazon.com/iam/
• AWS_S3_BUCKET (required) — S3 bucket name
  • Setup: Name of the S3 bucket to store uploads in (create one in the S3 console).
  • Get it here: https://console.aws.amazon.com/s3/
  • Example: my-app-uploads
• AWS_S3_REGION (optional) — S3 bucket region — default: us-east-1
  • Setup: The AWS region of your uploads bucket.
  • Example: us-east-1
• AWS_S3_ENDPOINT (optional) — S3 endpoint override
  • Setup: Endpoint URL for S3-compatible stores (MinIO, Cloudflare R2, DigitalOcean Spaces) — molecule's managed storage sets this automatically; leave empty for real AWS S3.
  • Example: http://localhost:9000
• AWS_S3_FORCE_PATH_STYLE (optional) — S3 path-style addressing
  • Setup: Set to 'true' for MinIO-style path addressing (http://host/bucket instead of virtual-hosted buckets); set automatically by molecule's managed storage. Leave unset for real AWS S3.
  • Example: true

Runtime Dependencies

• @aws-sdk/client-s3
• @aws-sdk/lib-storage
• @molecule/api-analytics
• @molecule/api-bond
• @molecule/api-i18n
• @molecule/api-proxy-agent
• @molecule/api-secrets
• @molecule/api-uploads
• uuid

Bond this as the uploads provider (see @molecule/api-uploads for the handler pattern and the own-every-file / validate rules). Config is all ENV, server-side: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_S3_BUCKET, AWS_S3_REGION. For an S3-COMPATIBLE store (Cloudflare R2, MinIO, DigitalOcean Spaces) set AWS_S3_ENDPOINT (plus AWS_S3_FORCE_PATH_STYLE=true for MinIO) — no code change.

• The exported provider is the default, env-configured store, and stays that way. Existing apps need no change. To run MORE stores in the same app (user uploads on one, backups on another with its own endpoint, credentials, bucket and storage class), call createProvider({ bucket, endpoint?, region?, forcePathStyle?, credentials?, storageClass?, keyPrefix?, connectionTimeoutMs?, socketTimeoutMs?, maxAttempts? }) once per extra store. Each instance has its own lazily-created client and reads NOTHING from the environment (except that, with no credentials, the AWS SDK falls back to its default chain — pass credentials for a second account). configFromEnv() returns the config the default uses, if you want to derive one from it.

• keyPrefix is part of the id. An instance with keyPrefix: 'backups/' returns ids like backups/<uuid>; pass them back to getFile/deleteFile/headFile unchanged — never strip or re-add the prefix yourself.

• headFile(id) (on createProvider instances and the default provider) returns { bytes, etag?, lastModified? } without downloading, null when the object does not exist, and throws on any other failure (bad credentials, missing bucket, network). A missing bucket is a bodiless 404 just like a missing object, so it is told apart with a HEAD on the bucket and thrown as NoSuchBucket.

• Streams above ~48.8 GiB need partSizeBytes. A multipart upload holds at most 10,000 parts of the part size (5 MiB by default); set partSizeBytes to ceil(maxBytes / 10000) (at least 5 MiB) for a larger ceiling.

• Immutable (backup) buckets: objectLock + describeBucketProtection(). Operator setup: create the bucket WITH Object Lock enabled (it cannot be added to most existing buckets), which also turns versioning on, and set a default retention. Then:

  • createProvider({ ..., objectLock: { mode: 'COMPLIANCE' | 'GOVERNANCE', retainDays } }) sends ObjectLockMode + ObjectLockRetainUntilDate (now + retainDays, computed per upload) on every PutObject and multipart upload, so each object's retention is asserted by the app rather than left to a bucket default a full-access credential could shorten.
  • An Object Lock PUT needs a checksum or S3 rejects it. With objectLock and no checksumAlgorithm, ChecksumAlgorithm: 'CRC32' is sent (the SDK computes it as a trailer). checksumAlgorithm: 'CRC32' | 'CRC32C' | 'SHA1' | 'SHA256' picks another; 'none' disables the SDK's automatic checksum (for a store that rejects trailers) and cannot be combined with objectLock. S3-COMPATIBLE stores differ: test a locked upload against the real store (e.g. Hetzner) before relying on it; 'SHA256' is the portable choice.
  • describeBucketProtection() reads versioning + the Object Lock configuration and returns { versioning: 'Enabled' | 'Suspended' | 'off', objectLock: { enabled, mode?, days?, years? }, checkedAt } (no lock configuration → enabled: false; any other error throws). bucketProtectionMeets(protection, { mode, minDays }) returns { ok, reasons } (versioning Enabled, lock on, mode at least as strict — COMPLIANCE satisfies GOVERNANCE, not the reverse — and default retention ≥ minDays, years × 365). Call both at boot AND periodically, and refuse to write backups / alert when ok is false: a bucket created without lock, or a default retention later removed, otherwise goes unnoticed.

• Keep the bucket PRIVATE — block all public access. A public-read bucket/object leaks every user's files to anyone with the URL. Serve private files THROUGH your API (stream via getFile, scoped to the owner) or hand out a short-lived presigned URL; never make an object public just to "make it load".

• The AWS credentials are server-only (never in the browser) — the browser uploads to YOUR API, which streams to S3.

• Blocked MIME types: text/html, application/xhtml+xml, JavaScript types, image/svg+xml, and XML are REJECTED at upload() (reported via onError; same list as the filesystem bond) — SVG uploads must be rasterized/re-typed client-side. Content-Disposition: attachment (next bullet) is the second layer for everything that IS accepted.

• Every object is uploaded with Content-Disposition: attachment — a deliberate stored-XSS safety default (S3 has no server-side rendering, so this stops a browser from ever executing an uploaded HTML/SVG file inline). This means a browser hitting the object directly (a raw S3 URL, or an <img src>/<iframe> pointing at a presigned URL) always DOWNLOADS it instead of rendering it inline — including otherwise-safe images. If you need inline rendering, serve the file THROUGH your API's getFile route, which lets you set your own Content-Disposition/Content-Type after your own validation — do not rely on this bond's default for that. There is no override for this default in the current revision.

• Aborting an upload rejects uploadPromise with UploadAbortedError (from @molecule/api-uploads) — it never resolves as success and never calls the upload() call's onError. Identical behavior to the @molecule/api-uploads-filesystem bond; see that core package's AbortHandler remarks for the full cross-provider contract.

• Runs behind an outbound proxy when HTTPS_PROXY is set. The AWS SDK v3 builds its own agent and reads no proxy variable, so on a host whose only egress path is a proxy every upload used to fail with a bare connection error. The client now gets a CONNECT-capable agent through its own requestHandler hook (@molecule/api-proxy-agent, resolved against AWS_S3_ENDPOINT/AWS_ENDPOINT_URL_S3 when set and the regional endpoint otherwise). An internal S3-compatible endpoint listed in NO_PROXY keeps connecting directly, and with no proxy configured nothing is passed at all. Allowlist *.amazonaws.com (or your store's host) on the proxy.

E2E Tests

Integration checklist — drive the real UI (live preview, no mocks), adapt each item to this app's actual screens/flows, and check every box off one by one. A box you can't check is an integration bug to fix — not a skip:

• Uploading a valid file through the UI shows progress/confirmation and the file appears in the user's file list.
• The uploaded content is retrievable: opening/downloading it returns the same content (an uploaded image actually renders).
• A disallowed file type is rejected with a visible error and does NOT appear in the list.
• An over-the-cap file is rejected cleanly (visible error, no partial phantom entry).
• Ownership is enforced: a second signed-in user cannot retrieve the first user's file by its id/URL (404 — not the file).
• Deleting a file removes it from the list, and it stays gone (and unretrievable) after a full reload.