TalkMiner: Searching Lecture Videos Across the Web
Lecture webcasts are readily available on the Internet. These include class lectures, research seminars and product demonstrations. The webcasts often combine presentation slides with either a synchronized audio stream (i.e., podcast) or an audio/video stream.
Conventional web search engines will retrieve this content if you include “webcast” or “lecture” among your search terms, or perform a search on a website that specifically organizes lecture content. But users, particularly students, want to find the points when an instructor covers a specific topic in a lecture. Answering these queries requires a search engine that can search within the webcast to identify important keywords.
The system processes RSS feeds from various sites to collect lecture videos. TalkMiner does not maintain a copy of the original videos. Rather, processing generates metadata about the talk including the video frames containing slides and their time codes, and the text recovered from those frames by OCR. When a user plays a lecture, the video is played from the original website on which the lecture webcast is hosted. As a result, storage requirements for the system are modest.
TalkMiner analyzes web videos to identify unique slide images and builds a search index from words on those presentation slides. Additionally, each talk may be browsed by its captured slide images for efficient non-linear playback. Visual cues indicate slides containing search terms within relevant talks.
A slide identification algorithm was developed to handle common video production techniques and correctly identify the slides.
Such production techniques include shooting a slide screen from the back of the room, picture in picture compositing, and multiple camera videos that intersperse shots of slides with shots of the speaker. Additionally, we developed specialized processing for slides with built up content, e.g. bulletted lists that are revealed gradually. Because detected slide images underlie both our search index and browsing interface, their automatic detection is a critical component in the system design.
TalkMiner builds its index and interface from commonly recorded video rather than using dedicated lecture-capture systems, or requiring careful post-capture authoring, or even imposing onerous constraints on the style of the video capture. Thus, the system can scale to include a greater volume and variety of content at a much lower cost than would otherwise be possible.
TalkMiner is offered online as a public demo at talkminer.com.
The system is detailed in our 2010 ACM Multimedia paper.
Related Blog Posts