Skip to content

LearnIngest & extract

Speech-to-text

ASR · Automatic speech recognition · Transcription

Automatic speech recognition turns recorded audio into text, ideally with timestamps for each segment or word, so every passage can point back to a moment in the recording.

In DocSlurpIllustrated explainer

On this page
Speech-to-text keeps a timestamp for every word, so a citation can seek to 03:12Five seconds of audio from investor-call.mp4, from 03:09.5 to 03:14.5. Speech bursts line up with ten words: EBITDA at 3:10.05, guidance 3:10.60, rises 3:11.15, to 3:11.60, ten 3:12.05, point 3:12.40, six 3:12.75, million 3:13.05, this 3:13.50 and year 3:13.75, ending at 3:13.95. The transcript line reads "[03:10–03:14] Speaker 2: EBITDA guidance rises to ten point six million this year." A citation chip, investor-call.mp4 at 03:12, points to the playhead at the word "ten".investor-call.mp4audio track, 16 kHz monoEBITDA3:10.05guidance3:10.60rises3:11.15to3:11.60ten3:12.05point3:12.40six3:12.75million3:13.05this3:13.50year3:13.7503:1003:1103:1203:1303:1403:12[03:10–03:14] Speaker 2: EBITDA guidance rises to ten point six million this year.▶ investor-call.mp4 · 03:12
A transcript becomes evidence when every word keeps its time. Each word carries a start time, the transcript line carries its range and speaker, and a citation can send the reader to 03:12 instead of "somewhere in a 90-minute call". Word timings here are illustrative.

The problem: the answer is in minute three

investor-call.mp4 is a 90-minute recording of the Q3 investor call. At 3 minutes 12 seconds, someone says: “EBITDA guidance rises to ten point six million this year.” That sentence answers the question every analyst on the call wanted answered. Nobody will scrub through 90 minutes to find it, and search cannot index a sound wave.

Speech-to-text, or automatic speech recognition (ASR), turns the audio into words. For search and citations it has to do one more thing: remember when each word was said, so a result can say “03:12” instead of “somewhere in this video”.

How a recording becomes a transcript

Modern ASR models such as OpenAI’s Whisper follow the same broad recipe.

  1. Resample. Convert the audio to a single channel at 16,000 samples per second.
  2. Make a spectrogram. Slice the audio into short overlapping frames (Whisper uses 25 ms windows every 10 ms) and measure the energy in each frequency band. The result is an image of sound: time across, pitch up, loudness as brightness. Whisper uses an 80-band log-Mel spectrogram.
  3. Encode and decode. A transformer encoder reads 30 seconds of spectrogram at a time. A decoder writes text tokens one by one, and can also emit special timestamp tokens that mark where segments start and end.
  4. Align words. Segment timestamps cover a phrase or sentence. Word-level timestamps need an extra alignment step that matches each word to the stretch of audio it came from.

The figure shows the result for a few seconds around 03:12: the waveform, each word with its start time, and a transcript line that carries its time range, [03:10–03:14] Speaker 2: EBITDA guidance rises to ten point six million this year. The speaker label comes from speaker diarization, a separate step.

Long recordings and the boundary problem

Models read a bounded window, so a long recording is cut into pieces. Cut in the wrong place and a sentence is split across two pieces, half-heard in each.

The usual fix is overlap. Give each window a slice of the timeline it owns, send a few extra seconds on each side, then keep only the segments that begin inside the owned slice. DocSlurp uses 300-second slices with 4 seconds of overlap. For a 600-second recording:

Window Owns Audio sent
1 0–300 s 0–304 s
2 300–600 s 296–600 s

A sentence spoken from 298 s to 303 s is heard whole in window 1, which owns its start, so window 1 keeps it. Window 2 hears its second half too, sees that window 1 already covered that stretch, and drops it. Nothing is cut, duplicated or lost. The same idea appears in text as chunk overlap.

Timestamps turn a transcript into evidence

Once each segment keeps its start and end seconds, everything downstream can use them. A chunk built from the segment carries its time range. A search hit can show “investor-call.mp4 · 03:10” and a player can seek there. A chat answer can cite the moment the sentence was spoken, so a reader can listen and check. That is provenance for audio, and it is what makes a citation to a recording useful.

Where it breaks

Accuracy is usually reported as word error rate: substitutions, deletions and insertions divided by the number of words in a reference transcript. Work one by hand:

reference:  ebitda guidance rises to  ten point six million this year
hypothesis: ebit da guidance rises two ten       six million this year

“ebitda” became “ebit” (a substitution) plus an extra “da” (an insertion), “to” became “two” (a substitution), and “point” was dropped (a deletion), turning $10.6M into “ten six million”. That is 2 + 1 + 1 = 4 errors over 10 reference words: a word error rate of 0.4. Notice which words broke. “EBITDA” and the guidance figure are exactly what someone would type into a search box.

  • Jargon and names. “NRR” becomes “and our”; product and person names come out spelled by sound.
  • Numbers. “Ten point six million” and “$10.6M” are the same fact and different strings. Search must handle both.
  • Silence and music. Whisper-style models can produce fluent sentences during long silences or music. A study of Whisper transcripts found invented phrases that nobody said.
  • Crosstalk and noise. Overlapping speakers, speakerphones and accents raise error rates sharply.
  • Timing precision. Segment times are approximate, and word times from alignment can be off by a fraction of a second. Good enough to seek a player, not to prove who interrupted whom.

When to use it

Transcribe any recording you want to search: meetings, interviews, support calls, lectures, the audio track of a video. Keep the timestamps, always. Add diarization when “who said it” matters, and keyframe extraction when the screen carries information the speech does not, such as slides or dashboards.

In DocSlurp

In DocSlurp

Audio and video use a hosted Whisper model (default openai/whisper-large-v3-turbo on DeepInfra), only when the operator sets MEDIA_REMOTE_ENABLED and the privacy mode allows it (core/worker/src/media/config.ts, core/worker/src/media/providers.ts). Audio is cut into 300-second windows with 4 seconds of overlap, and each block keeps start and end seconds (core/worker/src/media/segments.ts, core/worker/src/media/canonical.ts).

Sources

Jump to