Streaming LLM chat with persistence, token usage, regenerate, and reply variants.

useChat

Streaming LLM chat with message history, IndexedDB persistence, system prompts, cancellation, per-turn token usage, a lifecycle status, and regenerate-into-variants for branch-style UIs.

See it in action

Try the Chat block and Knowledge Base block for working demos of these hooks.

Basic Usage

import { useChat } from '@localmode/react';
import { webllm } from '@localmode/webllm';

const model = webllm.languageModel('Llama-3.2-1B-Instruct-q4f16_1-MLC');

function Chat() {
  const { messages, isStreaming, send, cancel, clearMessages } = useChat({
    model,
    systemPrompt: 'You are a helpful assistant.',
  });

  return (
    <div>
      {messages.map((m) => (
        <div key={m.id}><b>{m.role}:</b> {m.content}</div>
      ))}
      <input onKeyDown={(e) => {
        if (e.key === 'Enter') send(e.currentTarget.value);
      }} />
      {isStreaming && <button onClick={cancel}>Stop</button>}
      <button onClick={clearMessages}>Clear</button>
    </div>
  );
}

status tracks the request lifecycle: 'ready' → 'submitted' (send/regenerate called, first chunk not yet arrived) → 'streaming' (chunks are flowing) → back to 'ready' (or 'error' if the request failed).

isStreaming is derived from status — it is true while status is 'submitted' or 'streaming'. Use status when you need to distinguish the pre-first-chunk phase (e.g. a "thinking" indicator) from active streaming:

const { status } = useChat({ model });

{status === 'submitted' && <Spinner label="Thinking..." />}
{status === 'streaming' && <TypingIndicator />}
{status === 'error' && <ErrorBanner />}

Regenerate & Variants

regenerate() re-runs the last user turn with the same prior context and appends the result as a new variant of the last assistant reply. The first regeneration freezes the original reply as variants[0]; setVariantIndex() swaps which variant is shown as the last assistant message (the choice persists). Sending a new message resets the variants, and cancelling mid-regeneration restores the previously active variant.

import { useChat } from '@localmode/react';

function BranchBar() {
  const { status, regenerate, variants, variantIndex, setVariantIndex } = useChat({ model });

  if (variants.length === 0) return null;

  return (
    <div className="branch-bar">
      <button
        onClick={() => setVariantIndex(variantIndex - 1)}
        disabled={variantIndex === 0}
      >
        ‹
      </button>
      <span>{variantIndex + 1} / {variants.length}</span>
      <button
        onClick={() => setVariantIndex(variantIndex + 1)}
        disabled={variantIndex === variants.length - 1}
      >
        ›
      </button>
      <button onClick={regenerate} disabled={status !== 'ready'}>
        Regenerate
      </button>
    </div>
  );
}

Token Usage

usage holds the last completed turn's GenerationUsage (inputTokens, outputTokens, totalTokens, durationMs); totalUsage accumulates across all completed turns this session. Feed totalUsage into a context meter to warn before hitting the model's context window:

const { usage, totalUsage } = useChat({ model });
const CONTEXT_WINDOW = 4096;

const used = totalUsage.totalTokens;
const percent = Math.min(100, (used / CONTEXT_WINDOW) * 100);

return (
  <div>
    <progress value={used} max={CONTEXT_WINDOW} />
    <span>{used} / {CONTEXT_WINDOW} tokens ({percent.toFixed(0)}%)</span>
    {usage && <span>Last turn: {usage.totalTokens} tokens in {usage.durationMs}ms</span>}
  </div>
);

Persistence

Messages are persisted to IndexedDB by default. This means chat history survives page refreshes.

// Disable persistence
const chat = useChat({ model, persist: false });

// Custom storage key (useful for multiple chat instances)
const chat = useChat({ model, persistKey: 'my-project-chat' });

Persisted messages take precedence over initialMessages. If IndexedDB has saved messages, initialMessages is ignored.

Cancellation

Calling cancel() during streaming preserves the partial assistant message — it won't be cleared.

const { isStreaming, cancel } = useChat({ model });

// The partial response stays in messages after cancel

Vision (Image Input)

For vision-capable models, pass images alongside text using the send() options:

import { useChat } from '@localmode/react';
import { webllm } from '@localmode/webllm';

const model = webllm.languageModel('Phi-3.5-vision-instruct-q4f16_1-MLC');

function VisionChat() {
  const { messages, send } = useChat({ model });

  const handleImageUpload = async (file: File) => {
    const reader = new FileReader();
    reader.onload = async () => {
      const dataUrl = reader.result as string;
      const base64 = dataUrl.split(',')[1];
      await send('Describe this image', {
        images: [{ data: base64, mimeType: file.type, name: file.name }],
      });
    };
    reader.readAsDataURL(file);
  };

  // ... render messages and file input
}

Check model.supportsVision to conditionally show image upload UI. Messages with images have content as ContentPart[] instead of string. Use the getTextContent() and normalizeContent() helpers (re-exported from @localmode/react) when rendering mixed content:

import { getTextContent } from '@localmode/react';

{messages.map((m) => (
  <div key={m.id}><b>{m.role}:</b> {getTextContent(m.content)}</div>
))}

For the full multimodal API reference, see the Core Generation — Vision guide.

For full API reference on streamText(), see the Core Generation guide. For model setup, see WebLLM or Transformers.

Blocks

App	Description	Links
Chat (LLM Chat)	Full chat interface with streaming, model selection, and vision image input	Live block · Source
Knowledge Base (PDF Search)	Document Q&A chat powered by RAG	Live block · Source
Device & Model Lab (GGUF Explorer)	Chat with locally-loaded GGUF models	Live block · Source

Chat

useChat

Basic Usage

Options

Return Value

Status Lifecycle