LocalMode
React

Audio

Hooks for speech-to-text, text-to-speech, streaming speech, live transcription, and voice turn-taking.

Audio Hooks

See it in action

Try the Audio Studio block for a working demo of these hooks.

useTranscribe

Transcribe audio to text (speech-to-text). language, task, and returnTimestamps can be set once at the hook level and overridden per call via execute(audio, { language, task, returnTimestamps }).

import { useTranscribe } from '@localmode/react';
import { transformers } from '@localmode/transformers';

const model = transformers.speechToText('onnx-community/whisper-tiny.en');

function Demo() {
  const { data, isLoading, execute } = useTranscribe({ model, returnTimestamps: true });

  const handleFile = (file: File) => execute(file);

  return (
    <div>
      <input type="file" accept="audio/*" onChange={(e) => {
        if (e.target.files?.[0]) handleFile(e.target.files[0]);
      }} />
      {isLoading && <p>Transcribing...</p>}
      {data && <p>{data.text}</p>}
      {data?.segments?.map((seg, i) => (
        <p key={i}>[{seg.start}s] {seg.text}</p>
      ))}
    </div>
  );
}

Options

OptionTypeDescription
modelSpeechToTextModelThe speech-to-text model to use (required)
languagestringLanguage code (ISO 639-1) applied to every execute() call
task'transcribe' | 'translate'Task type applied to every execute() call
returnTimestampsboolean | 'word'Return timestamps — segments arrive on data.segments

Per-call overrides win over hook-level options. language/task hints require a multilingual model — English-only checkpoints like whisper-tiny.en reject them:

const whisper = transformers.speechToText('Xenova/whisper-tiny'); // multilingual
const { execute } = useTranscribe({ model: whisper });

await execute(audioBlob, { language: 'fr', task: 'translate' });

useSynthesizeSpeech

Generate speech audio from text (text-to-speech). voice, speed, and pitch can be set once at the hook level and overridden per call via execute(text, { voice, speed, pitch }).

import { useSynthesizeSpeech } from '@localmode/react';

const { data, isLoading, execute } = useSynthesizeSpeech({ model, voice: 'af_heart' });

await execute('Hello world');                                 // uses af_heart
await execute('Bonjour', { voice: 'bf_emma', speed: 1.2 });   // per-call override
// data.audio = Float32Array, data.sampleRate = 16000

Options

OptionTypeDescription
modelTextToSpeechModelThe text-to-speech model to use (required)
voicestringVoice ID applied to every execute() call
speednumberSpeech rate (0.5–2.0, default: 1.0) applied to every execute() call
pitchnumberPitch adjustment applied to every execute() call

To record microphone input for transcription, pair with useVoiceRecorder (device selection via deviceId, live stream, getVolume() metering), and render recorded blobs with useObjectUrl.

useStreamSpeech

Streaming text-to-speech: composes streamSynthesizeSpeech() (clause-by-clause synthesis) with playStreamedSpeech() (gap-free Web Audio playback) into a single speak/pause/resume/stop primitive.

import { useStreamSpeech } from '@localmode/react';

function Speaker({ reply }: { reply: string }) {
  const { speak, pause, resume, stop, reset, isSynthesizing, isPlaying, currentClause, clauses, error } =
    useStreamSpeech({ model, voice: 'af_heart' });

  return (
    <div>
      <button onClick={() => speak(reply)}>Speak</button>
      <button onClick={isPlaying ? pause : resume}>{isPlaying ? 'Pause' : 'Resume'}</button>
      <button onClick={stop}>Stop</button>
      {currentClause && <p>Now playing: {currentClause.text}</p>}
    </div>
  );
}

Return Value

PropertyTypeDescription
speak(text: string) => Promise<void>Start synthesizing and playing; resolves when playback ends
pause / resume() => voidSuspend / resume the underlying AudioContext
stop() => voidStop playback and halt upstream synthesis
reset() => voidClear clauses, currentClause, and error from the previous speak() call. No-op while synthesis or playback is in progress — call stop() first
isSynthesizingbooleanTrue while a speak() operation is producing audio
isPlayingbooleanTrue while scheduled clauses are actively playing
currentClauseSynthesizedClause | nullThe clause currently being played
clausesSynthesizedClause[]All clauses observed so far during the active speak() call
errorError | nullThe last error thrown by speak()

Safari and mobile browsers require the AudioContext to be created or resumed inside a user-gesture handler. Pass your own audioContext option for full control; otherwise the hook lazily creates one on the first speak() call. See Streaming Speech for the underlying core API.

useLiveTranscribe

Streaming microphone-driven speech-to-text with VAD. Wraps createLiveTranscriber() — lazily constructed on the first start() call so the getUserMedia permission prompt happens during a user gesture, and auto-disposed on unmount. Options mirror LiveTranscriberOptions (minus abortSignal, which the hook owns) plus an onBargeIn callback.

import { useLiveTranscribe } from '@localmode/react';
import { transformers } from '@localmode/transformers';

function PushToTalk() {
  const { state, currentUtterance, utterances, lastBargeIn, start, stop, clearUtterances } =
    useLiveTranscribe({
      model: transformers.speechToText('onnx-community/moonshine-tiny-ONNX'),
      mode: 'push-to-talk',
      onBargeIn: () => stopPlayback(), // user spoke over external playback
    });

  return (
    <>
      <button onMouseDown={start} onMouseUp={stop}>Hold to talk</button>
      {state === 'listening' && <p>{currentUtterance}</p>}
      <ul>{utterances.map((u) => <li key={u.utteranceId}>{u.text}</li>)}</ul>
      <button onClick={clearUtterances}>Clear transcript</button>
    </>
  );
}

Return Value

PropertyTypeDescription
stateLiveTranscriberStateCurrent state of the underlying controller
isListeningbooleanTrue when state === 'listening'
currentChunksLiveChunk[]Chunks emitted for the current utterance (cleared at each utterance start)
currentUtterancestringThe in-progress utterance text (last partial chunk's text)
lastUtteranceLiveUtterance | nullThe most recent completed utterance
utterancesLiveUtterance[]All completed utterances accumulated this session (oldest first)
lastBargeInBargeInEvent | nullThe most recent barge-in event (only fires when bargeInWhilePlaying is configured; also delivered to onBargeIn)
errorError | nullLatest error
start / stop() => Promise<void>Begin / stop listening (the controller stays alive and re-startable after stop)
dispose() => Promise<void>Dispose the controller and release all resources
clearUtterances() => voidClear the accumulated utterances list (does not touch lastUtterance)

useTurnTaker

Full voice-loop orchestration (listen → plan → speak) wrapping createTurnTaker(). Accumulates both sides of the conversation in turns for transcript rendering. Options mirror TurnTakerOptions (minus abortSignal) plus an onBargeIn callback.

import { useTurnTaker } from '@localmode/react';

function VoiceAssistant() {
  const turn = useTurnTaker({ transcriber, planner, voice, systemPrompt: 'Be concise.' });

  return (
    <>
      <button onClick={turn.start}>
        {turn.isListening ? 'Listening…' : turn.isSpeaking ? 'Speaking…' : 'Start'}
      </button>
      <button onClick={turn.interrupt}>Interrupt</button>
      <ul>{turn.turns.map((t, i) => <li key={i}>{t.role}: {t.text}</li>)}</ul>
    </>
  );
}

Return Value

PropertyTypeDescription
stateTurnTakerStateCurrent state of the orchestrator
isListening / isPlanning / isSpeakingbooleanConvenience flags for the corresponding states
turnsTurnEntry[]Accumulated conversation turns — { role: 'user' | 'agent', text, timestamp }, oldest first
lastUserUtterancestring | nullThe most recent user utterance text
lastAgentResponsestring | nullThe most recent agent response text
lastBargeInDate | nullWhen the most recent barge-in fired (voice or programmatic; also delivered to onBargeIn)
errorError | nullLatest error
start / stop() => Promise<void>Begin / stop the voice loop
interrupt() => voidProgrammatic barge-in
dispose() => Promise<void>Dispose and release all resources
clearTurns() => voidClear the accumulated turns list

For model recommendations, see the Transformers Audio guide. For the underlying core APIs, see Live Transcription and Streaming Speech.

Blocks

AppDescriptionLinks
Audio Studio (Voice Notes)Record and transcribe with useTranscribeLive block · Source
Audio Studio (Meeting Assistant)Transcribe and summarize with useTranscribe + useSummarizeLive block · Source
Audio Studio (Audiobook Creator)Text-to-speech with useSynthesizeSpeechLive block · Source

On this page