Molecule.dev

@molecule/api-ai-embeddings-embeddinggemma

Provider bond · ai-embeddings · API (Node) · v1.0.1 · Apache-2.0

EmbeddingGemma 2 ai-embeddings provider for molecule.dev — Google's open (Apache-2.0) multimodal embedding model run in-process via Transformers.js (ONNX): text, images and audio in one 768-d space, task prefixes applied for you, Matryoshka 512/256/128

npm install @molecule/api-ai-embeddings-embeddinggemma

npm · Source on GitHub · Implements @molecule/api-ai-embeddings

How it works

@molecule/api-ai-embeddings-embeddinggemma is a provider bond on the API (Node) side: it implements the ai-embeddings core interface (@molecule/api-ai-embeddings) with a concrete vendor or library behind it.

Your code calls the core; you wire this provider once at startup. Swapping vendors later is one line in that wiring, not a rewrite.

import { readFile } from 'node:fs/promises'

import { requireProvider, setProvider } from '@molecule/api-ai-embeddings'
import { createProvider } from '@molecule/api-ai-embeddings-embeddinggemma'

// Text only by default; add the encoders you embed (each costs memory).
setProvider(createProvider({ modalities: ['text', 'image'] }))

const embeddings = requireProvider()
// Task prefixes are applied for you: queries and documents embed differently.
const queryVector = await embeddings.embedQuery('cats sleeping on a couch')
const docVectors = await embeddings.embedDocuments(['Mars is the Red Planet.'])

// Images share the text vector space — check support before calling.
if (embeddings.modalities?.includes('image') && embeddings.embedContent) {
  const {
    embeddings: [photoVector],
  } = await embeddings.embedContent({
    inputs: [{ image: { data: await readFile('photo.jpg'), mimeType: 'image/jpeg' } }],
  })
  console.log(queryVector.length, docVectors.length, photoVector?.length) // 768 1 768
}

Works with: @molecule/api-ai-embeddings

Reference

EmbeddingGemma 2 ai-embeddings provider for molecule.dev.

Runs Google's open (Apache-2.0) multimodal embedding model in-process via Transformers.js on onnx-community/embeddinggemma-2-ONNX — no API key, no per-call cost, no network after the first download. Text, images and audio map into ONE 768-dimensional, L2-normalized vector space (a dot product is the cosine similarity), with Matryoshka truncation to 512 / 256 / 128. Strong on code retrieval and 100+ languages.

Auto-generated, AI-first package reference for the molecule.dev ecosystem. It is written to be read by coding agents as much as by people, and is generated from this package's source — edit src/index.ts JSDoc, not this file.

Quick Start

import { readFile } from 'node:fs/promises'

import { requireProvider, setProvider } from '@molecule/api-ai-embeddings'
import { createProvider } from '@molecule/api-ai-embeddings-embeddinggemma'

// Text only by default; add the encoders you embed (each costs memory).
setProvider(createProvider({ modalities: ['text', 'image'] }))

const embeddings = requireProvider()
// Task prefixes are applied for you: queries and documents embed differently.
const queryVector = await embeddings.embedQuery('cats sleeping on a couch')
const docVectors = await embeddings.embedDocuments(['Mars is the Red Planet.'])

// Images share the text vector space — check support before calling.
if (embeddings.modalities?.includes('image') && embeddings.embedContent) {
  const {
    embeddings: [photoVector],
  } = await embeddings.embedContent({
    inputs: [{ image: { data: await readFile('photo.jpg'), mimeType: 'image/jpeg' } }],
  })
  console.log(queryVector.length, docVectors.length, photoVector?.length) // 768 1 768
}

Type

provider

Installation

npm install @molecule/api-ai-embeddings-embeddinggemma @huggingface/transformers @molecule/api-ai-embeddings

API

Interfaces

DecodedAudio

Decoded PCM audio.

interface DecodedAudio {
  /** Mono samples in [-1, 1]. */
  samples: Float32Array
  /** Samples per second. */
  sampleRate: number
}

EmbeddingGemmaConfig

Configuration for the EmbeddingGemma 2 provider. Every field is optional and has an env-var fallback (MOL_EMBEDDINGS_EMBEDDINGGEMMA_*), so the provider works with zero configuration.

interface EmbeddingGemmaConfig {
  /**
   * Model repo id. Defaults to `onnx-community/embeddinggemma-2-ONNX` or the
   * `MOL_EMBEDDINGS_EMBEDDINGGEMMA_MODEL` env var (e.g. a private mirror).
   */
  model?: string
  /** Weight precision. Defaults to `q8` or `MOL_EMBEDDINGS_EMBEDDINGGEMMA_DTYPE`. */
  dtype?: EmbeddingGemmaDtype
  /** Inference device. Defaults to `cpu` or `MOL_EMBEDDINGS_EMBEDDINGGEMMA_DEVICE`. */
  device?: EmbeddingGemmaDevice
  /**
   * Encoders to load. Defaults to `['text']` or the comma-separated
   * `MOL_EMBEDDINGS_EMBEDDINGGEMMA_MODALITIES` (e.g. `text,image,audio`). Each
   * extra encoder costs memory and download size (q8: vision 195 MB, audio 340 MB),
   * so load only what you embed. `text` is always loaded.
   */
  modalities?: EmbeddingGemmaModality[]
  /**
   * Default output size (Matryoshka truncation + re-normalization). Defaults to
   * 768 or `MOL_EMBEDDINGS_EMBEDDINGGEMMA_DIMENSIONS`.
   */
  dimensions?: EmbeddingGemmaDimensions
  /** Default task for prefixes. Defaults to `search` or `MOL_EMBEDDINGS_EMBEDDINGGEMMA_TASK`. */
  task?: EmbeddingTask
  /**
   * Prepend the model's task prefixes (`task: search result | query: …`,
   * `title: none | text: …`). Defaults to `true`; text that already starts with
   * `task: ` or `title: ` is never prefixed twice.
   */
  applyPrefixes?: boolean
  /** Directory weights are cached in (or `MOL_EMBEDDINGS_EMBEDDINGGEMMA_CACHE_DIR`). */
  cacheDir?: string
  /**
   * Directory holding pre-bundled weights for offline use (or
   * `MOL_EMBEDDINGS_EMBEDDINGGEMMA_MODEL_PATH`). Setting it disables remote fetch
   * unless `allowRemoteModels` is explicitly `true`.
   */
  localModelPath?: string
  /** Allow downloading weights on first use. Defaults to `true`, or `false` with `localModelPath`. */
  allowRemoteModels?: boolean
  /**
   * Texts per forward pass (or `MOL_EMBEDDINGS_EMBEDDINGGEMMA_BATCH_SIZE`). Defaults
   * to 16. A MEMORY bound: activations for the whole batch are resident at once and
   * attention grows with sequence length (up to 8,192 tokens here). Media inputs
   * always run one per pass. Values below 1 fall back to the default.
   */
  batchSize?: number
}

Types

EmbeddingGemmaDtype

ONNX weight precision. q8 matches fp32 to ~0.9997 cosine; q4 to ~0.975 (text ~0.988).

type EmbeddingGemmaDtype = 'fp32' | 'fp16' | 'q8' | 'q4' | 'q4f16'

EmbeddingGemmaDevice

Where inference runs. Node uses cpu; webgpu/wasm are browser backends.

type EmbeddingGemmaDevice = 'cpu' | 'webgpu' | 'wasm'

EmbeddingGemmaDimensions

The Matryoshka sizes the model was trained for. Any other size is rejected.

type EmbeddingGemmaDimensions = 768 | 512 | 256 | 128

EmbeddingGemmaModality

Modalities this bond can load in Node. Video needs a browser (see the package remarks).

type EmbeddingGemmaModality = Exclude<EmbeddingModality, 'video'>

Functions

truncateEmbedding(vector, dimensions)

Truncate a vector to its first dimensions values and re-L2-normalize it (Matryoshka). A full-size vector is returned unchanged.

function truncateEmbedding(vector: number[], dimensions: number): number[]
  • vector — A unit-length embedding.
  • dimensions — Target size.

Returns: The truncated, unit-length vector.

unwrapMediaSource(source)

Split a media source into its data and (optional) declared MIME type.

function unwrapMediaSource(source: EmbedMediaSource): {
  data: Uint8Array | string
  mimeType?: string
}
  • source — The media source.

Returns: The bytes or URL/path, plus the MIME type when the caller gave one.

readMediaBytes(data)

Read a media source into bytes: bytes as-is, http(s):// via fetch, anything else as a local file path.

function readMediaBytes(
  data: string | Uint8Array<ArrayBufferLike>,
): Promise<Uint8Array<ArrayBufferLike>>
  • data — Bytes, URL, or file path.

Returns: The bytes.

decodeWav(bytes)

Decode a RIFF/WAVE file (integer PCM 8/16/24/32-bit or 32/64-bit float, any channel count) into mono samples.

function decodeWav(bytes: Uint8Array<ArrayBufferLike>): DecodedAudio
  • bytes — The WAV file bytes.

Returns: Mono samples and their sample rate.

resample(audio, targetRate)

Resample mono audio by linear interpolation.

function resample(audio: DecodedAudio, targetRate: number): Float32Array<ArrayBufferLike>
  • audio — The decoded audio.
  • targetRate — Output sample rate.

Returns: Samples at targetRate.

loadAudio(source)

Load an audio source as 16 kHz mono samples, ready for the audio encoder.

function loadAudio(source: EmbedMediaSource): Promise<Float32Array<ArrayBufferLike>>
  • source — WAV bytes, a WAV URL / file path, or the object form.

Returns: 16 kHz mono samples.

applyTaskPrefix(text, task, inputType)

Prefix one text for the given task and side.

Asymmetric tasks: a query becomes task: <task> | query: <text> and a document title: none | text: <text>. Symmetric tasks (classification, clustering, similarity) use the query form for every input.

function applyTaskPrefix(text: string, task: EmbeddingTask, inputType: EmbeddingInputType): string
  • text — The raw text.
  • task — What the vectors are for.
  • inputType — Query or document side.

Returns: The prefixed text (unchanged if it already carries a prefix).

isEmbeddingTask(value)

Whether a string is a supported task name (for env-var parsing).

function isEmbeddingTask(value: string): boolean
  • value — Candidate task name.

Returns: true if it is an EmbeddingTask.

createProvider(config)

Create an EmbeddingGemma 2 provider. Config falls back to env vars, so createProvider() with no arguments works (text only, q8, 768-d, search).

function createProvider(config?: EmbeddingGemmaConfig): AIEmbeddingsProvider
  • config — Optional configuration.

Returns: An AIEmbeddingsProvider with embedContent for the loaded modalities.

Constants

MATRYOSHKA_DIMENSIONS

Output sizes the model was trained for.

const MATRYOSHKA_DIMENSIONS: readonly number[]

AUDIO_SAMPLE_RATE

Sample rate the EmbeddingGemma 2 audio encoder was trained on.

const AUDIO_SAMPLE_RATE: 16000

EMBEDDINGGEMMA_DEFAULT_MODEL

Default model — the ONNX export of google/embeddinggemma-2.

const EMBEDDINGGEMMA_DEFAULT_MODEL: 'onnx-community/embeddinggemma-2-ONNX'

provider

The provider implementation (text only by default). Bond it with the ai-embeddings core's setProvider. The model loads on the first embed call.

const provider: AIEmbeddingsProvider

Core Interface

Implements @molecule/api-ai-embeddings interface.

Bond Wiring

Setup function to register this provider with the core interface:

import { setProvider } from '@molecule/api-ai-embeddings'
import { provider } from '@molecule/api-ai-embeddings-embeddinggemma'

export function setupAiEmbeddingsEmbeddinggemma(): void {
  setProvider(provider)
}

Injection Notes

Requirements

Peer dependencies:

  • @molecule/api-ai-embeddings >=1.1.0

Runtime Dependencies

  • @huggingface/transformers

  • @molecule/api-ai-embeddings

  • Use embedQuery for what the user types and embedDocuments for what you index. The model needs task prefixes (task: search result | query: … vs title: none | text: …); embedding both sides with the same call — or with embed(), which treats input as DOCUMENTS unless you pass inputType: 'query' — silently lowers search quality. Set task ('code-retrieval', 'question-answering', 'classification', …) in the config or per embed call. Text that already starts with task: or title: is left alone; applyPrefixes: false turns prefixing off.

  • Never mix vector spaces. These vectors are not comparable with EmbeddingGemma 1, gemini-embedding-2, bge (-local) or OpenAI vectors, nor across dimensions settings. Switching to this bond means re-embedding the whole index; store the model id + dimensions with each index.

  • dimensions must be 768, 512, 256 or 128 — the only trained sizes; anything else throws. Truncated vectors are re-normalized.

  • Load only the encoders you need. modalities defaults to ['text'] (314 MB of q8 weights, ~850 MB RSS measured with Node). image adds the vision encoder (195 MB), audio the audio encoder (340 MB). Calling embedContent with a medium whose encoder is not loaded throws.

  • Video is NOT supported in Node (Transformers.js decodes video only in a browser): modalities: ['video'] and video inputs throw. Embed sampled frames as images, or use @molecule/api-ai-embeddings-gemini.

  • Audio must be WAV in Node (Transformers.js cannot decode audio there). The bond decodes integer-PCM / float WAV, mixes to mono and resamples to the 16 kHz the model expects; convert MP3/OGG to WAV first.

  • One embedContent input = one vector, even when it combines text with an image or audio clip (write <|image|> / <|audio|> in the text where the media belongs; a missing marker is appended). Media inputs run one forward pass each.

  • The 8,192-token context is shared by everything in one input: an image costs ~280 tokens, audio 25 tokens per second (~5 minutes max).

  • dtype defaults to q8 (cosine ≥ 0.9999 vs the Python reference). q4 is ~45% smaller but drifts to ~0.988 on text; fp16/q4f16 quality is unreported. Changing dtype changes the vectors — re-embed.

  • First use downloads the weights from Hugging Face and caches them (set cacheDir to a persistent path). For offline deployments bundle them and set localModelPath. Loading takes seconds; the model then stays resident.

  • usage is always zero — local inference is not billed.

  • Configure with createProvider({...}) or the MOL_EMBEDDINGS_EMBEDDINGGEMMA_* env vars (MODEL, DTYPE, DEVICE, MODALITIES, DIMENSIONS, TASK, CACHE_DIR, MODEL_PATH, BATCH_SIZE). batchSize (default 16) bounds peak memory, not throughput.

  • Pulls @huggingface/transformers + onnxruntime-node — a real third-party dependency. To serve the model from a GPU box instead, run vLLM and use @molecule/api-ai-embeddings-openai (you then add the prefixes yourself).

E2E Tests

Integration checklist — drive the real flow (no mocks), adapt each item to this app's actual data and features, and check every box off one by one. A box you can't check is an integration bug to fix — not a skip. Embeddings are infrastructure, so PROVE them through the feature they power (semantic search / "related items" / dedup) AND with a direct property check on the vectors:

  • embedQuery(text) returns a non-empty numeric number[] of the model's fixed dimension, and every vector from embed/embedDocuments has that SAME length — no empty arrays, no NaN/null entries, and the length is identical across calls (a query and a document must be comparable).
  • The semantic property holds — there is NO built-in similarity helper, so compute cosine similarity yourself (dot product over the two magnitudes): two related texts ("dog"/"puppy", or a query and a matching doc) score HIGH, two unrelated texts ("dog"/"quarterly taxes") score clearly LOWER. If every pair scores alike the vectors are dead — one positive check alone doesn't prove it.
  • Ranking works: embed a query plus a handful of documents and sort by cosine similarity — the semantically closest document ranks ABOVE the unrelated ones. This ordering is the whole point; a query that ranks an off-topic doc first is broken.
  • The feature built on it works end-to-end from the real UI: semantic search / "related items" / dedup returns results ranked by MEANING, not keyword — a search for a synonym or paraphrase finds the right item even when it shares NO words with the query (the case a plain text/keyword search would miss).
  • Embedding is stable: the same text embedded twice yields (near-)identical vectors, so a stored index stays valid — re-embedding an item must not silently drift it out of its own neighborhood.
  • Batching is order-preserving: embedDocuments([a, b, c]) (or embed({ input })) returns exactly one vector per input in the SAME order — embeddings[i] is the vector for input[i], never shuffled, merged, or dropped.
  • Embedding runs SERVER-SIDE: the call goes through the app's API and the provider key never reaches the browser (no provider request or key in the Network tab). Any endpoint that embeds caller-supplied text is authenticated and rate-limited — an open embed endpoint is an unbounded per-token cost/abuse vector.