- Publication Details
-
- IEEE International Conference on Multimedia and Expo, v. II, pp. 77-80
- Jul July 7, 2003
Manga
Interactive video summaries in a comic book style
Manga is an interactive video summary that provides video playback from the keyframes comprising the summary.
What is Manga?
Manga is a pictorial summary of video named after a Japanese word for “comic book”. A video is automatically analyzed and represented with different-sized keyframes packed in a visually pleasing form reminiscent of a comic book. Video Manga allows users to get a quick overview of a video’s contents at a glance without watching the video from beginning to end. The visual summaries are suitable for printing, and they can also be used to help users browse through videos.
To generate a Manga, we start with high-quality video keyframes. A video is segmented based on the color features of each keyframe. The segments are also clustered according to their similarities. We have introduced an importance score to rank the segments. A segment is considered to be important if it is long and rare. Keyframes are selected from highly ranked segments and sized according to their scores so that more important keyframes are presented as bigger frames.
Our frame-packing algorithm puts the different-sized keyframes in a compact “comic book” format such that images are placed in rows in temporal order from top-left to bottom-right. Within a row, there is some freedom for placing images but the overall order is maintained. The packing algorithm has to change the size of some of the images to avoid gaps and to fill the available space.
Manga on the Web
A previous version of Video Manga was implemented as a Java applet that was included in the Fuji Xerox product MediaDEPO. Recently, we implemented an interactive version of the pictorial summary using HTML5 technologies. This version will be included in a future version of MediaDEPO.
Moving the mouse over the displayed frames highlights the frame and the corresponding segment in the timeline. This allows users to explore the temporal properties of a video. Clicking on a keyframe starts video playback from the beginning of that segment. We have also implemented a way to explore segments that are not presented at the top level of the summary.
Video from the NudgeCam project
The Manga video summary above is fully interactive. Keyframes under the mouse expand. Moving the mouse along the timeline also expands the corresponding keyframes. Single-clicking on a keyframe starts video playback at that position. Double-clicking on a keyframe displays a more detailed view of that part of the video. On touch devices, touch-dragging takes the place of the mouse hover. Single and double taps have the same effects as mouse clicks.
Consistent keyframe selection
Manga video summaries of different sizes can use very different keyframes. That is visually confusing when a user changes the size of a Manga video summary. For video content owners who present their content as Manga summaries, this generates uncertainty of what users viewing those summaries on different display sizes will see. This uncertainty prevents the attachment of additional information to certain keyframes because those may not be shown in some summaries.
The solution is to use all keyframes from a smaller summary in a larger one and to select additional keyframes for the larger summary. Importance scores from the smaller summary are propagated to the larger one to keep image sizes similar.
This memorandum provides more details about the approach.
Technical Contact
Related Publications
- Publication Details
-
- Proceedings IEEE International Conference on Multimedia and Expo, Lausanne, Switzerland, August 2002
- Aug August 26, 2002
Abstract
Close- Publication Details
-
- IEEE Computer, 34(9), pp. 61-67
- Sep September 1, 2001
Abstract
CloseTo meet the diverse needs of business, education, and personal video users, the authors developed three visual interfaces that help identify potentially useful or relevant video segments. In such interfaces, keyframes-still images automatically extracted from video footage-can distinguish videos, summarize them, and provide access points. Well-chosen keyframes enhance a listing's visual appeal and help users select videos. Keyframe selection can vary depending on the application's requirements: A visual summary of a video-captured meeting may require only a few highlight keyframes, a video editing system might need a keyframe for every clip, while a browsing interface requires an even distribution of keyframes over the video's full length. The authors conducted user studies for each of their three interfaces, gathering input for subsequent interface improvements. The studies revealed that finding a similarity measure for collecting video clips into groups that more closely match human perception poses a challenge. Another challenge is to further improve the video-segmentation algorithm used for selecting keyframes. A new version will provide users with more information and control without sacrificing the interface's ease of use.
- Publication Details
-
- In CHI 2000 Conference Proceedings, ACM Press, pp. 185-192, 2000.
- Mar March 31, 2000
Abstract
Close- Publication Details
-
- In Proceedings ACM Multimedia, (Orlando, FL) ACM Press, pp. 383-392, 1999.
- Oct October 30, 1999
Abstract
Close- Publication Details
-
- In Human-Computer Interaction INTERACT '99, IOS Press, pp. 205-212, 1999.
- Aug August 30, 1999
Abstract
Close- Publication Details
-
- In Proceedings of the International Conference on Acoustics, Speech, and Signal Processing (Phoenix, AZ), vol. 6, pp. 3041-3044, 1999.
- Mar March 14, 1999