Skip to content

LearnIngest & extract

Keyframe extraction

Frame sampling

Choosing a small set of representative frames from a video, so a vision model can describe what was on screen without reading every frame.

In DocSlurpGlossary entry

On this page

Keyframe extraction picks a small set of frames from a video to stand in for the whole thing. A 30-minute screen recording at 30 frames per second has 54,000 frames; a vision model can afford to look at a few hundred at most, and most frames are near-duplicates of their neighbors anyway.

Two basic strategies, each with a blind spot:

  • Uniform sampling: take one frame every N seconds. Predictable coverage, but a slide that was on screen for four seconds between samples is missed entirely.
  • Change detection: compare each frame with the previous one, and keep frames where many pixels changed (a scene cut, a new slide). Catches the four-second slide, but a shaky handheld camera or a scrolling page looks like constant change and eats the budget.

Practical systems combine them: spread some anchor frames evenly across the timeline, then spend the rest of the budget on the frames with the biggest changes, with a minimum gap so one busy moment doesn’t take everything. Comparisons are done on tiny grayscale thumbnails, which is cheap and still sees a slide change.

The selected frames, each with its timestamp, go to a vision-language model, which describes what is visible: “03:40 On screen: dashboard showing error rate rising in us-east”. Those lines are indexed alongside the speech-to-text transcript, so a search can find what was shown as well as what was said.

The gotcha: a keyframe is a sample, not proof of what happened between samples. A description of two frames ten seconds apart says nothing about the eight seconds in between, and the model should not be allowed to guess.

In DocSlurp

In DocSlurp

Only when the operator enables video vision (MEDIA_VIDEO_ENABLED). ffmpeg scans the video at 96×54 grayscale, scores pixel change, and the selector keeps evenly spaced anchors plus the most-changed frames, at most one per 5 seconds and 1,440 per video (core/worker/scripts/video_scan.py, core/worker/src/media/video-policy.ts). A vision model turns batches into timestamped "On screen" lines.

Sources

Jump to