@molecule/api-uploads-mirror

Mirrored upload provider for molecule.dev — one UploadProvider that writes every file to several upload providers and reads it back from whichever has it.

Auto-generated, AI-first package reference for the molecule.dev ecosystem. It is written to be read by coding agents as much as by people, and is generated from this package's source — edit src/index.ts JSDoc, not this file.

Quick Start

import { setProvider } from '@molecule/api-uploads'
import { createProvider as createMirror } from '@molecule/api-uploads-mirror'
import { createProvider as createS3 } from '@molecule/api-uploads-s3'

const primary = createS3({
  bucket: process.env.PRIMARY_S3_BUCKET!,
  region: process.env.PRIMARY_S3_REGION,
})
const offsite = createS3({
  bucket: process.env.OFFSITE_S3_BUCKET!,
  endpoint: process.env.OFFSITE_S3_ENDPOINT,
  credentials: {
    accessKeyId: process.env.OFFSITE_S3_ACCESS_KEY_ID!,
    secretAccessKey: process.env.OFFSITE_S3_SECRET_ACCESS_KEY!,
  },
})

setProvider(
  createMirror({
    targets: [
      { name: 'primary', provider: primary },
      { name: 'offsite', provider: offsite, required: false },
    ],
    onTargetFailure: ({ target, operation, error }) =>
      console.warn(`uploads mirror: ${target} failed to ${operation}`, error),
  }),
)

Type

provider

Installation

npm install @molecule/api-uploads-mirror @molecule/api-analytics @molecule/api-uploads

API

Interfaces

MirrorTarget

One upload provider that receives a copy of every file.

interface MirrorTarget {
  /**
   * Stable name of the target, recorded inside every mirror id. Must match
   * `[A-Za-z0-9_-]{1,32}` and be unique within the config. Renaming a target
   * orphans the copies already stored under the old name.
   */
  name: string

  /**
   * The upload provider that stores this copy.
   */
  provider: UploadProvider

  /**
   * Whether an upload fails when this target fails. A failed OPTIONAL target is
   * reported and its copy is left out of the id; the upload still succeeds.
   *
   * @default true
   */
  required?: boolean
}

MirrorTargetFailure

A failure of one target, reported through onTargetFailure.

interface MirrorTargetFailure {
  /**
   * The target's name.
   */
  target: string

  /**
   * What the mirror was doing when the target failed.
   */
  operation: MirrorOperation

  /**
   * The thrown value or reported error. Never contains file contents.
   */
  error: unknown
}

MirrorUploadsConfig

Configuration for createProvider.

interface MirrorUploadsConfig {
  /**
   * The providers every file is written to, in read-preference order. At least
   * one; names unique.
   */
  targets: MirrorTarget[]

  /**
   * Called for every target failure that does not fail the whole operation
   * (optional-target upload/delete failures, a read that fell through to the
   * next copy) and for every target failure that does. Must not throw.
   */
  onTargetFailure?: (event: MirrorTargetFailure) => void
}

MirrorCopy

One stored copy of a mirrored file: which target holds it, under which id.

interface MirrorCopy {
  /**
   * The target's name.
   */
  target: string

  /**
   * The id that target's provider returned for its copy.
   */
  id: string
}

MirrorCopyLocation

A stored copy plus whether the target still has it.

interface MirrorCopyLocation extends MirrorCopy {
  /**
   * `true`/`false` from the target's `headFile`; `'unknown'` when the target has
   * no `headFile`, is no longer configured, or the lookup failed.
   */
  present: boolean | 'unknown'
}

MirrorUploadProvider

An UploadProvider that writes every file to several providers.

interface MirrorUploadProvider extends UploadProvider {
  /**
   * The configured targets, in read-preference order.
   */
  readonly targets: ReadonlyArray<Pick<MirrorTarget, 'name' | 'required'>>

  /**
   * Streams one file to every target at once (the source is read once).
   *
   * @param fieldname - The form field name.
   * @param stream - The file data.
   * @param info - Filename, encoding and MIME type.
   * @param onError - Called once if a required target fails.
   * @returns The file, whose `id` is a mirror id.
   */
  upload(
    fieldname: string,
    stream: NodeJS.ReadableStream,
    info: FileInfo,
    onError: (error: Error) => void,
  ): UploadedFile

  /**
   * Aborts an in-progress upload on every target.
   *
   * @param file - The file `upload()` returned.
   */
  abortUpload(file: UploadedFile): Promise<void>

  /**
   * Reads a file from the first target (in config order) that has it.
   *
   * A copy whose target is NOT in this mirror's config counts as an error for
   * that copy (not as missing): the read moves on to the next copy, and when
   * no copy served, the error is thrown rather than `null`.
   *
   * @param id - A mirror id, or a raw id stored before mirroring.
   * @returns A stream, or `null` when every target reported the file missing.
   */
  getFile(id: string): Promise<NodeJS.ReadableStream | null>

  /**
   * Deletes every copy of a file. THROWS when a REQUIRED target's delete fails
   * or when the id names a target that is not in this mirror's config; a
   * failed delete on an optional target is reported (`onTargetFailure` +
   * `trackBondFailure`) and the call still resolves — a caller that must
   * know a particular copy went deletes it through a mirror whose target is
   * marked required.
   *
   * @param id - A mirror id, or a raw id stored before mirroring.
   */
  deleteFile(id: string): Promise<void>

  /**
   * Decodes a mirror id into its copies.
   *
   * @param id - Any id.
   * @returns The copies in id order, or `null` when `id` is not a mirror id.
   */
  parseId(id: string): MirrorCopy[] | null

  /**
   * Whether `id` is a well-formed mirror id.
   *
   * @param id - Any id.
   * @returns `true` for a mirror id.
   */
  isMirrorId(id: string): boolean

  /**
   * Reports which targets still hold a copy, using each target's optional `headFile`.
   *
   * @param id - A mirror id, or a raw id (checked on every target).
   * @returns One entry per copy.
   */
  locate(id: string): Promise<MirrorCopyLocation[]>
}

UploadedFile

Properties describing an uploading/uploaded file.

interface UploadedFile {
  /**
   * The unique file identifier.
   */
  id: string
  /**
   * The file's fieldname from the form.
   * Used as the key for the file within `req.files` - e.g., `req.files[fieldname] = file`.
   */
  fieldname: string
  /**
   * The original filename - e.g., `some-image.jpg`.
   */
  filename: string
  /**
   * The file's encoding - e.g., `binary`.
   */
  encoding: string
  /**
   * The file's mimetype - e.g., `image/jpeg`.
   */
  mimetype: string
  /**
   * The file's size in bytes.
   */
  size: number
  /**
   * The source stream (available during upload).
   */
  stream?: NodeJS.ReadableStream
  /**
   * A promise that resolves when the upload completes.
   */
  uploadPromise?: Promise<void>
  /**
   * Whether the upload has completed.
   */
  uploaded: boolean
  /**
   * The URL/location of the uploaded file (if available).
   */
  location?: string
}

FileInfo

Information about a file being uploaded.

This interface is provider-agnostic. Implementations using busboy or other multipart parsers should adapt to this interface.

interface FileInfo {
  /**
   * The original filename - e.g., `some-image.jpg`.
   */
  filename: string
  /**
   * The file's encoding - e.g., `7bit`, `binary`.
   */
  encoding: string
  /**
   * The file's MIME type - e.g., `image/jpeg`.
   */
  mimeType: string
}

UploadProvider

Upload provider interface.

All upload providers must implement this interface.

interface UploadProvider {
  /**
   * Uploads a file from a stream.
   */
  upload: UploadHandler
  /**
   * Aborts an in-progress upload.
   */
  abortUpload: AbortHandler
  /**
   * Deletes a file.
   */
  deleteFile: DeleteHandler
  /**
   * Gets a file stream for reading.
   */
  getFile?: GetFileHandler
}

Types

MirrorOperation

The operation a target failed during.

type MirrorOperation = 'upload' | 'get' | 'delete' | 'abort'

HeadFileHandler

The optional metadata lookup some upload providers (e.g. @molecule/api-uploads-s3) expose. Returns null when the object does not exist.

type HeadFileHandler = (id: string) => Promise<unknown>

Functions

encodeMirrorId(copies)

Encodes copies into a mirror id.

function encodeMirrorId(copies: MirrorCopy[]): string

• copies — The stored copies, in target order.

Returns: The mirror id.

decodeMirrorId(id)

Decodes a mirror id.

function decodeMirrorId(id: string): MirrorCopy[] | null

• id — Any id.

Returns: The copies in id order, or null when id is not a well-formed mirror id.

createProvider(config)

Creates a mirrored upload provider.

function createProvider(config: MirrorUploadsConfig): MirrorUploadProvider

• config — The targets and an optional failure callback.

Returns: The provider.

Constants

MIRROR_ID_PREFIX

Prefix every mirror id starts with. The 1 is the format version.

const MIRROR_ID_PREFIX: 'mir1.'

MIRROR_TARGET_NAME_PATTERN

Pattern a target name must match.

const MIRROR_TARGET_NAME_PATTERN: RegExp

Core Interface

Implements @molecule/api-uploads interface.

Injection Notes

Requirements

Peer dependencies:

• @molecule/api-uploads ^1.0.1

Runtime Dependencies

• @molecule/api-analytics
• @molecule/api-uploads

Bond the result of createProvider({ targets }) as the uploads provider (see @molecule/api-uploads for the handler pattern). Each target is any other uploads bond — two S3-compatible stores in different regions or accounts, S3 plus the filesystem, and so on.

• Writes stream once to every target. The source is read a single time and teed to each target through its own PassThrough; the slowest target sets the pace (backpressure), and nothing buffers the whole file.
• Required vs optional targets. required defaults to true. If a REQUIRED target fails, onError is called with its error, the copies that did succeed elsewhere are deleted (best-effort), and uploadPromise rejects. If an OPTIONAL target fails, the upload still succeeds without that copy; the failure goes to onTargetFailure and to trackBondFailure (bond.failure uploads-mirror upload …). An upload where no target stored a copy always fails.
• The id is opaque. Store it and pass it back unchanged. It records which target holds which copy (format mir1.<base64url>, a single URL-safe token). Read file.id AFTER uploadPromise resolves: when an optional target failed, the final id leaves that copy out.
• Reads go in CONFIG order, not the order inside the id. A target that returns a stream wins; a target that reports the file missing (null) is skipped; a target that THROWS is reported and skipped. getFile returns null only when every copy was missing — if any target threw and none served, it throws the first error (an outage is not "missing"). A target with no getFile, or one the id names but the config no longer lists, counts as a failure, never as missing.
• Failover happens when a read OPENS, not mid-stream. A copy that stops answering after getFile returned its stream is not retried on the next copy; a reader that must survive that opens the next copy itself.
• A cut upload is a failed upload. A multipart parser's size limit (busboy's limit event) and a source that closes before it ends fail every copy (UploadTooLargeError / UploadSourceClosedError), rather than storing the truncated body as complete.
• Files stored before mirroring keep working. A raw (non-mirror) id is read from the first target that returns it, with the same error rules, and deleteFile tries it on every target. That is the migration path: keep the provider that stored them, under its existing configuration, as one of the targets, and existing ids resolve unchanged.
• deleteFile deletes every copy concurrently and throws when a required target failed (including a provider that throws for an object it does not have — the mirror cannot tell a provider's "missing" error from a real one). Optional-target failures are reported, not thrown.
• abortUpload aborts every target; uploadPromise rejects with UploadAbortedError and onError is not called, per the @molecule/api-uploads abort contract.
• locate(id) says which targets still hold a copy, using a target's headFile when it has one (as @molecule/api-uploads-s3 does) and 'unknown' otherwise.
• This bond does NOT encrypt. Every target stores the bytes it is given. For encryption at rest, wrap targets with @molecule/api-uploads-encrypted.

E2E Tests

Integration checklist — drive the real UI (live preview, no mocks), adapt each item to this app's actual screens/flows, and check every box off one by one. A box you can't check is an integration bug to fix — not a skip:

• Uploading a valid file through the UI shows progress/confirmation and the file appears in the user's file list.
• The uploaded content is retrievable: opening/downloading it returns the same content (an uploaded image actually renders).
• A disallowed file type is rejected with a visible error and does NOT appear in the list.
• An over-the-cap file is rejected cleanly (visible error, no partial phantom entry).
• Ownership is enforced: a second signed-in user cannot retrieve the first user's file by its id/URL (404 — not the file).
• Deleting a file removes it from the list, and it stays gone (and unretrievable) after a full reload.