Photo Library
React hook for a shared CLIP-family photo library — multimodal embeddings and zero-shot categorization over one injected model.
Photo Library Hook
See it in action
Try the Photo Search block for a working demo — its Gallery / Search / Duplicates / Categories tabs run over one shared photo library and one loaded CLIP model powering both embeddings and zero-shot categorization.
usePhotoLibrary
A shared in-memory photo library where one injected CLIP-family model powers both multimodal embeddings (text + image, one vector space → search + duplicates) and zero-shot categorization. It owns the photo entries and their embeddings, the model-load lifecycle, the active editable label set, the adaptive batch profile, and every mutation — progressive, adaptively-batched, cancellable ingest with per-batch categorization (with an other/0 fallback), text and image search over the shared vector space, per-photo and bulk delete, a confirmed model-switch re-index, and label re-categorization.
Provider-agnostic by injection: the embedding model, the zero-shot classifier, and the optional cache probe are supplied as factories, so @localmode/react gains no @localmode/transformers dependency. Zero model bytes move until an explicit action calls loadModel(), ingest, or search.
import { usePhotoLibrary } from '@localmode/react';
import { transformers, isModelCached } from '@localmode/transformers';
const lib = usePhotoLibrary({
modelId: 'Xenova/clip-vit-base-patch32',
createEmbeddingModel: (id, onProgress) => transformers.multimodalEmbedding(id, { onProgress }),
createZeroShotClassifier: (id) => transformers.zeroShotImageClassifier(id),
isModelCached: (id) => isModelCached(id),
labelPresets: { photo: { labels: ['nature', 'people', 'other'] } },
getModelDimensions: () => 512,
});Options
Prop
Type
Return Value
The PhotoLibrary surface groups into model lifecycle, library, search, labels / categorization, model-switch re-index, and shared status.
Model lifecycle
Prop
Type
Library and search
Prop
Type
Labels, categorization, re-index, and status
Prop
Type
Pure library algorithms
The ranking and grouping algorithms are exported standalone — deterministic, model-free, and unit-testable. They run over the entries' cached embeddings via core's cosineSimilarity.
| Function | Signature | Purpose |
|---|---|---|
rankByEmbedding | (query, entries, topK, minSimilarity) => RankedHit[] | Rank the library against a query embedding, filtered by a min-similarity floor and capped at top-K. Powers both text→image and image→image search (the query vector lives in the same shared space). |
computeSimilarCounts | (entries, threshold?) => Map<string, number> | Each photo's count of OTHER embedded photos above the similarity threshold (default 0.7). |
groupDuplicates | (entries, threshold) => DuplicateGroup[] | Union-find grouping over pairwise cosine similarity; groups keep 2+ members, carry the average pairwise similarity, and sort descending. Cheap to re-run on a threshold change (no re-embedding). |
For the CLIP cross-modal embedding functions (embedImage, streamEmbedManyImages) and the shared vector space, see the Core Multimodal Embeddings guide.
Blocks
| App | Description | Links |
|---|---|---|
| Photo Search | Gallery / Search / Duplicates / Categories over one shared library and one CLIP model via usePhotoLibrary | Live block · Source |
Provider Fallback
React hook for per-capability Chrome Built-in AI ⇄ Transformers.js provider resolution with truthful badge provenance.
Knowledge Base
React hook for knowledge-base session orchestration over the KnowledgeBaseEngine contract — document store, embedding-model lifecycle, and switch-driven re-ingest.