@localmode/mediapipe

Run Google's MediaPipe Tasks directly in the browser. @localmode/mediapipe wraps @mediapipe/tasks-vision, @mediapipe/tasks-audio, and @mediapipe/tasks-text as a single unified LocalMode provider — hand, pose, and face landmarks, gesture recognition, image and audio classification, image embeddings, language detection, and text embeddings, plus real-time streaming trackers for video.

Features

• 13 model catalog — landmarks, gestures, classification, detection, segmentation, embeddings, and language detection, all curated and verified against Google's CDN
• Real-time streaming — createHandTracker / createPoseTracker / createFaceTracker / createGestureTracker run over a <video> element at 30–60fps
• Universal browser support — pure WebAssembly + WebGL; no WebGPU required
• Tiny models — most are under 10MB; face detection and selfie segmentation are ~250KB
• Unified LocalMode interface — landmark tasks call detectHands(), detectPose(), etc.; classification/detection/embedding tasks reuse the existing classifyImage(), detectObjects(), embed() functions
• AbortSignal cancellation — every single-frame function supports cancellation
• GPU or CPU delegate — choose the WebGL GPU delegate (default) or the CPU delegate per provider or per model

Installation

The package depends on @mediapipe/tasks-vision, @mediapipe/tasks-audio, and @mediapipe/tasks-text — all installed automatically. The WASM runtime loads from the jsDelivr CDN by default; see WASM Runtime to self-host it.

Quick Start

Single-frame detection

Detect hand landmarks in a still image:

import { detectHands } from '@localmode/core';
import { mediapipe } from '@localmode/mediapipe';

const { hands } = await detectHands({
  model: mediapipe.handLandmarker(),
  image: imageBlob,
  numHands: 2,
});

for (const hand of hands) {
  console.log(`${hand.handedness} hand — ${hand.landmarks.length} landmarks`);
}

Real-time streaming

Track hands live from a webcam <video> element:

import { mediapipe } from '@localmode/mediapipe';

const tracker = mediapipe.createHandTracker({
  video: videoElement,
  numHands: 2,
  onResults: (hands, timestampMs) => {
    // Called once per processed frame (up to ~60fps)
    drawHands(hands);
  },
});

await tracker.start();

// later
tracker.stop();
await tracker.close();

Tasks

The provider exposes a factory method per task. Landmark and gesture tasks return MediaPipe-specific model interfaces used with new core functions; the remaining tasks return standard LocalMode interfaces and reuse existing core functions.

Landmarks & Gestures

Vision (standard core interfaces)

Audio & Text

Streaming trackers

Model Catalog

MEDIAPIPE_MODELS ships 13 curated models. Every model is verified against Google's public CDN (storage.googleapis.com). Vision models use the .task bundle format; audio and text models are raw .tflite files.

import { MEDIAPIPE_MODELS } from '@localmode/mediapipe';

Each factory method uses its catalog default. Pass a catalog ID or a direct model URL to override:

// Catalog default
const lite = mediapipe.poseLandmarker();

// Catalog ID — higher-accuracy pose model
const full = mediapipe.poseLandmarker('pose_landmarker_full');

// Direct URL to any compatible MediaPipe model file
const custom = mediapipe.handLandmarker('https://your-cdn.com/hand_landmarker.task');

You can also set an explicit modelPath in the per-model settings, which always takes precedence:

const model = mediapipe.handLandmarker('hand_landmarker', {
  modelPath: 'https://your-cdn.com/hand_landmarker.task',
});

Provider Configuration

Custom provider

Use createMediaPipe() for a provider instance with custom settings, or import the default mediapipe singleton.

import { createMediaPipe } from '@localmode/mediapipe';

const myMediaPipe = createMediaPipe({
  delegate: 'CPU',
  wasmBasePath: '/wasm/mediapipe',
});

const model = myMediaPipe.handLandmarker();

Per-model settings

Every factory method accepts a second settings argument that overrides the provider defaults for that model only:

const model = mediapipe.handLandmarker('hand_landmarker', {
  modelPath: 'https://your-cdn.com/hand_landmarker.task',
  delegate: 'CPU',
  wasmBasePath: '/wasm/mediapipe/vision',
});

WASM Runtime

MediaPipe Tasks ships a WebAssembly runtime — one binary set per domain (vision, audio, text). By default, @localmode/mediapipe loads it from the jsDelivr CDN, so no extra setup is needed.

To run fully offline, self-host the runtime. Copy the wasm directory from each @mediapipe/tasks-* package into your public assets and point wasmBasePath at it:

const myMediaPipe = createMediaPipe({
  wasmBasePath: {
    vision: '/wasm/tasks-vision',
    audio: '/wasm/tasks-audio',
    text: '/wasm/tasks-text',
  },
});

Browser Compatibility

MediaPipe Tasks runs on pure WebAssembly with a WebGL GPU delegate — it does not require WebGPU.

If the WebGL GPU delegate is unavailable, set delegate: 'CPU' to fall back to CPU inference.

Error Handling

import { detectHands, ModelLoadError, VisionError, ValidationError } from '@localmode/core';

try {
  const { hands } = await detectHands({
    model: mediapipe.handLandmarker(),
    image: imageBlob,
  });
} catch (error) {
  if (error instanceof ModelLoadError) {
    console.error('Failed to load model:', error.hint);
  } else if (error instanceof VisionError) {
    console.error('Detection failed:', error.hint);
  } else if (error instanceof ValidationError) {
    console.error('Invalid input:', error.hint);
  }
}

Single-frame functions retry transient failures up to maxRetries (default 2) before throwing. Streaming trackers report per-frame errors through the onError callback instead of throwing.

Next Steps

Showcase App