Speech-to-text
ASR · Automatic speech recognition · Transcription
Automatic speech recognition turns recorded audio into text, ideally with timestamps for each segment or word, so every passage can point back to a moment in the recording.
In DocSlurpIllustrated explainer
On this page
The problem: the answer is in minute three
investor-call.mp4 is a 90-minute recording of the Q3 investor call. At 3 minutes 12 seconds, someone says: “EBITDA guidance rises to ten point six million this year.” That sentence answers the question every analyst on the call wanted answered. Nobody will scrub through 90 minutes to find it, and search cannot index a sound wave.
Speech-to-text, or automatic speech recognition (ASR), turns the audio into words. For search and citations it has to do one more thing: remember when each word was said, so a result can say “03:12” instead of “somewhere in this video”.
How a recording becomes a transcript
Modern ASR models such as OpenAI’s Whisper follow the same broad recipe.
- Resample. Convert the audio to a single channel at 16,000 samples per second.
- Make a spectrogram. Slice the audio into short overlapping frames (Whisper uses 25 ms windows every 10 ms) and measure the energy in each frequency band. The result is an image of sound: time across, pitch up, loudness as brightness. Whisper uses an 80-band log-Mel spectrogram.
- Encode and decode. A transformer encoder reads 30 seconds of spectrogram at a time. A decoder writes text tokens one by one, and can also emit special timestamp tokens that mark where segments start and end.
- Align words. Segment timestamps cover a phrase or sentence. Word-level timestamps need an extra alignment step that matches each word to the stretch of audio it came from.
The figure shows the result for a few seconds around 03:12: the waveform, each word with its start time, and a transcript line that carries its time range, [03:10–03:14] Speaker 2: EBITDA guidance rises to ten point six million this year. The speaker label comes from speaker diarization, a separate step.
Long recordings and the boundary problem
Models read a bounded window, so a long recording is cut into pieces. Cut in the wrong place and a sentence is split across two pieces, half-heard in each.
The usual fix is overlap. Give each window a slice of the timeline it owns, send a few extra seconds on each side, then keep only the segments that begin inside the owned slice. DocSlurp uses 300-second slices with 4 seconds of overlap. For a 600-second recording:
| Window | Owns | Audio sent |
|---|---|---|
| 1 | 0–300 s | 0–304 s |
| 2 | 300–600 s | 296–600 s |
A sentence spoken from 298 s to 303 s is heard whole in window 1, which owns its start, so window 1 keeps it. Window 2 hears its second half too, sees that window 1 already covered that stretch, and drops it. Nothing is cut, duplicated or lost. The same idea appears in text as chunk overlap.
Timestamps turn a transcript into evidence
Once each segment keeps its start and end seconds, everything downstream can use them. A chunk built from the segment carries its time range. A search hit can show “investor-call.mp4 · 03:10” and a player can seek there. A chat answer can cite the moment the sentence was spoken, so a reader can listen and check. That is provenance for audio, and it is what makes a citation to a recording useful.
Where it breaks
Accuracy is usually reported as word error rate: substitutions, deletions and insertions divided by the number of words in a reference transcript. Work one by hand:
reference: ebitda guidance rises to ten point six million this year
hypothesis: ebit da guidance rises two ten six million this year
“ebitda” became “ebit” (a substitution) plus an extra “da” (an insertion), “to” became “two” (a substitution), and “point” was dropped (a deletion), turning $10.6M into “ten six million”. That is 2 + 1 + 1 = 4 errors over 10 reference words: a word error rate of 0.4. Notice which words broke. “EBITDA” and the guidance figure are exactly what someone would type into a search box.
- Jargon and names. “NRR” becomes “and our”; product and person names come out spelled by sound.
- Numbers. “Ten point six million” and “$10.6M” are the same fact and different strings. Search must handle both.
- Silence and music. Whisper-style models can produce fluent sentences during long silences or music. A study of Whisper transcripts found invented phrases that nobody said.
- Crosstalk and noise. Overlapping speakers, speakerphones and accents raise error rates sharply.
- Timing precision. Segment times are approximate, and word times from alignment can be off by a fraction of a second. Good enough to seek a player, not to prove who interrupted whom.
When to use it
Transcribe any recording you want to search: meetings, interviews, support calls, lectures, the audio track of a video. Keep the timestamps, always. Add diarization when “who said it” matters, and keyframe extraction when the screen carries information the speech does not, such as slides or dashboards.
In DocSlurp
In DocSlurp
Audio and video use a hosted Whisper model (default openai/whisper-large-v3-turbo on DeepInfra), only when the operator sets MEDIA_REMOTE_ENABLED and the privacy mode allows it (core/worker/src/media/config.ts, core/worker/src/media/providers.ts). Audio is cut into 300-second windows with 4 seconds of overlap, and each block keeps start and end seconds (core/worker/src/media/segments.ts, core/worker/src/media/canonical.ts).