Video Manga

What is Manga?

Manga is a pictorial summary of video named after a Japanese word for "comic book". A video is automatically analyzed and represented with different-sized keyframes packed in a visually pleasing form reminiscent of a comic book. Video Manga allows users to get a quick overview of a video's contents at a glance without watching the video from beginning to end. The visual summaries are suitable for printing, and they can also be used to help users browse through videos.

To generate a Manga, a video is segmented based on the color features of each frame. The segments are also clustered according to their similarities. We have introduced an importance score to rank the segments. A segment is considered to be important if it is long and rare. Keyframes are extracted from highly ranked segments and sized according to their scores so that more important keyframes are presented as bigger frames. Our frame-packing algorithm puts the different-sized keyframes in a compact "comic book" format.

The visual summary can be enhanced by adding captions to its frames. Text data for captions may be retrieved from transcripts of the video or embedded closed-caption data. Manga can also handle caption data manually typed in by users.

Manga on the Web

We have implemented an interactive version of the pictorial summary as an abstract-level browsing tool for videos in our MBase system. Moving the mouse over the displayed frames highlights the frame and the corresponding segment in the timeline. This allows users to explore the temporal properties of a video. Clicking on a keyframe starts video playback from the beginning of that segment. We have also implemented a way to explore segments that are not presented at the top level of the summary.

Technical Contact: Andreas Girgensohn

Related Publications