Video Manga


What is Manga?

Manga is a pictorial summary of video named after a Japanese word for “comic book”. A video is automatically analyzed and represented with different-sized keyframes packed in a visually pleasing form reminiscent of a comic book. Video Manga allows users to get a quick overview of a video’s contents at a glance without watching the video from beginning to end. The visual summaries are suitable for printing, and they can also be used to help users browse through videos.

To generate a Manga, we start with high-quality video keyframes. A video is segmented based on the color features of each keyframe. The segments are also clustered according to their similarities. We have introduced an importance score to rank the segments. A segment is considered to be important if it is long and rare. Keyframes are selected from highly ranked segments and sized according to their scores so that more important keyframes are presented as bigger frames.

Our frame-packing algorithm puts the different-sized keyframes in a compact “comic book” format such that images are placed in rows in temporal order from top-left to bottom-right. Within a row, there is some freedom for placing images but the overall order is maintained. The packing algorithm has to change the size of some of the images to avoid gaps and to fill the available space.

Manga image sizes
Sequence of images in the unit sizes determined from the importance score. The sequence is broken up into rows for easier matching to the layout on the right.
Manga layout
Resulting layout with changed image sizes indicated by “-1” and “+1”.

Manga on the Web

A previous version of Video Manga was implemented as a Java applet that was included in the Fuji Xerox product MediaDEPO. Recently, we implemented an interactive version of the pictorial summary using HTML5 technologies. This version will be included in a future version of MediaDEPO.

Moving the mouse over the displayed frames highlights the frame and the corresponding segment in the timeline. This allows users to explore the temporal properties of a video. Clicking on a keyframe starts video playback from the beginning of that segment. We have also implemented a way to explore segments that are not presented at the top level of the summary.

Video from the NudgeCam project

The Manga video summary above is fully interactive. Keyframes under the mouse expand. Moving the mouse along the timeline also expands the corresponding keyframes. Single-clicking on a keyframe starts video playback at that position. Double-clicking on a keyframe displays a more detailed view of that part of the video. On touch devices, touch-dragging takes the place of the mouse hover. Single and double taps have the same effects as mouse clicks.

Consistent keyframe selection

Manga video summaries of different sizes can use very different keyframes. That is visually confusing when a user changes the size of a Manga video summary. For video content owners who present their content as Manga summaries, this generates uncertainty of what users viewing those summaries on different display sizes will see. This uncertainty prevents the attachment of additional information to certain keyframes because those may not be shown in some summaries.

The solution is to use all keyframes from a smaller summary in a larger one and to select additional keyframes for the larger summary. Importance scores from the smaller summary are propagated to the larger one to keep image sizes similar.

This memorandum provides more details about the approach.

Technical Contact: Andreas Girgensohn

Related Publications

Copyright ©1999-2014 FX Palo Alto Laboratory | Send feedback to the webmaster