The video keyframe tools provide access into the video through keyframes along the timeline.

Our approach for selecting keyframes from a video starts with extracting video frames that are of high quality with respect to sharpness and sufficient brightness. Those extracted frames form the starting point for our other algorithm, such as the one that produces a Manga.

We use hierarchical agglomerative clustering to select keyframes from the extracted video frames. Our algorithm divides the cluster tree such that there are as many cluster as keyframes are needed for presentation. One keyframe is selected from each cluster. For temporal presentations such as timelines, keyframes are selected from clusters such that the temporal distribution of the selected keyframes is somewhat uniform.

Here we present two user interfaces that make use of the selected keyframes. One interface lets the the user browse through a visual-temporal cluster tree to quickly locate a video clip of interest. Another uses keyframes to aid the navigation in a video player.

Video Keyframe Browser

We created a novel interface for browsing through a video keyframe hierarchy to find frames or clips. We developed algorithms for selecting quality keyframes and for clustering keyframes hierarchically. At each level of the hierarchy, a single representative keyframe from each cluster is shown. Users can drill down into the most promising cluster and view representative keyframes for the sub-clusters. Our clustering algorithms optimize for short navigation paths to the desired keyframe. This publication provides additional details.

The blue part of the timeline is linear and indicates the part of of the video that is represented by the keyframes in the center, initially the whole video. The rest of the timeline is non-linear with darker shades of gray indicating denser time. While the mouse is over a keyframe, the corresponding video segment is indicated in yellow in the timeline. Thin blue lines connect the keyframes to the corresponding times in the timeline.

Move the mouse across images to enlarge them. Scroll forward with the mouse scroll-wheel (one tick) while being over a keyframe to zoom in on that video segment. Scroll backwards to zoom out. Mouse clicks are intended for video playback but are not functional in this demonstration. On a touch-screen, drag across to enlarge images, drag up to see additional keyframes, and drag down to zoom out.

Video Keyframe Player

We created an interactive video keyframe player based on our keyframe selection algorithm. Keyframes are attached to the timeline and appear on mouse-over or touch. In the video player shown below, only the keyframes near the playback position or mouse position on the timeline are shown. That approach supports navigation near the current position. It can be easily switched to keyframes with increasingly larger gaps that cover the whole video.

The video player below is fully functional and can be controlled both with a mouse and on a touch device such as a mobile phone or a tablet.

Video from the NudgeCam project

Moving the mouse over the timeline or touching it makes the keyframes appear. Clicking on a keyframe moves the playback position to that time. Dragging the timeline thumb also changes the playback position. Both operations can be performed either when the video is paused or when it is playing. The larger dots in the timeline indicate the positions of the currently visible keyframes and the smaller dots indicate the positions of the other keyframes. The keyframe under the mouse is indicated in yellow.