Talkminer: a Lecture Video Search Engine
Total Page：16
File Type：pdf, Size：1020Kb
技術論文 TalkMiner: A Lecture Video Search Engine 要 旨 Web配信されるレクチャー・ビデオを対象とした検索 エンジンの設計と実装について述べる。検索可能なテキ スト・インデックスを作成することで、YouTubeをはじ めとするさまざまなWebサイトにあるレクチャー・ビデ オの中で使われている資料を見つけることができる。こ のインデックスは、ビデオ中に表示されるプレゼンテー ション・スライドのテキストから、また利用可能なタイ トルや要約などの関連するメタデータから構築される。 ビデオの連続した流れの中でスライドを自動的に識別 するためにはさまざまな困難が伴う。例えば、話者とス ライドを同時表示するpicture-in-picture、カメラの切替 り、スライド・ビルドなど、スライド画像を抽出する基 本アルゴリズムだけでは対応が難しい。本稿ではこれら の課題に対応し、スライド識別の性能を改善する拡張ア ルゴリズムについて述べる。 アルゴリズムの評価と検索エンジンの有用性を試験す るためにwww.talkminer.comを公開している。これまで にさまざまなサイトで公開されている17,000を超えるレ クチャー・ビデオのインデックス作成を完了している。 Abstract The design and implementation of a search engine for lecture webcasts is described. A searchable text index is created allowing users to locate material within lecture videos found on a variety of websites such as YouTube and Berkeley webcasts. The searchable index is built from the text of presentation slides appearing in the video along with other associated metadata such as the title and abstract when available. The automatic identification of distinct slides within the video stream presents several challenges. For example, picture-in-picture compositing of a speaker and a presentation slide, switching cameras, and slide builds Authors confuse basic algorithms for extracting keyframe slide John Adcock*1 images. Enhanced algorithms are described that Matthew Cooper*1 improve slide identification. Laurent Denoue*1 Hamed Pirsiavash*2 A public system was deployed to test the algorithms Lawrence A. Rowe*1 and the utility of the search engine at www.talkminer.com. *1 FX Palo Alto Laboratory, Inc To date, over 17,000 lecture videos have been indexed *2 University of California, Irvine from a variety of public sources. 118 富士ゼロックス テクニカルレポート No.21 2012 技術論文 TalkMiner: A Lecture Video Search Engine lecture video despite the wide variety of video 1. Introduction capture practices and production techniques in Lecture videos or webcasts are increasingly use. available on the Internet. These webcasts may TalkMiner is not a video hosting system; it be class lectures (e.g., Berkeley webcasts, Kyoto does not retain or serve a copy of the original OCW), research seminars (e.g., Google Tech video files. The videos are processed to identify Talks, PARC Forum, etc.), or product their distinct slides and corresponding time demonstrations or training. Many of these video codes, and a search index is constructed from lectures are organized around typical slide-based text recovered from the slides. When a user presentation materials (e.g.: PowerPoint). reviews a lecture, the video is embedded from Conventional web search engines can find a the original hosting website. As a result, storage webpage with a lecture on it if the searcher requirements for our system are minimal. The includes “webcast" or “lecture" among the search current system has indexed over 17,000 lecture terms, or searches on a website oriented videos but only consumes approximately 20 GB towards video lecture content (e.g: of storage. AcademicEarth). This type of search can be This report describes the design and successful if the hosting website includes a implementation of the TalkMiner system. It is detailed textual description of the contents of the organized as follows. Section 2 describes the video, indexable with standard text-based search basic operation of the interactive search system. technology. But simply locating a topical video Section 3 reviews related work in this area. may not be satisfactory. Users, particularly Section 4 details the implementation of the students, need to find the precise points inside a system including the back-end slide detection video when an instructor covers a specific topic algorithms and the front-end web-based search within a lecture (e.g., course requirements or system. We also review some experiments discussion of a specific concept). Answering validating our slide detection scheme. Section 5 these needs requires a search engine that can concludes the paper with a summary. analyze the content inside the webcast, expose it to search, and allow a searcher to seek to the 2. User Experience interesting portion of the video. TalkMiner proposes to address this need. It The TalkMiner search interface resembles processes lecture videos to identify slide typical web search interfaces. The user enters images, to extract text using optical character one or more search terms and a list of talks that recognition (OCR), and to build a text search match those terms in the title, abstract, or index from the information it recovers. presentation slides are listed as in Figure 1. This video processing task can be Notice that search term matches are highlighted. challenging. Image quality of web videos is The information displayed for each talk on the often poor, lighting is inconsistent, and the search results page(s) includes a representative relative positions of the camera and projection keyframe, the date and title of the lecture, and screen are unpredictable. Also, many lecture the channel (or source) of the talk. Other videos employ production elements including metadata shown includes the duration of the cuts between multiple cameras and talk, the number of slides, the publication date, picture-in-picture compositing. and the date of indexing by TalkMiner. A link is TalkMiner uses automatic video analysis provided to the original page on which it was methods developed to robustly identify slides in published. 富士ゼロックス テクニカルレポート No.21 2012 119 技術論文 TalkMiner: A Lecture Video Search Engine 3. Related Work Production of rich media lecture webcasts can be challenging. Collections of academic lecture material can be found on a few websites [22, 19, 2, 24] but the experience is rarely more advanced than a video with a basic seek and playback interface. When an enhanced presentation of the lecture is made available, it is generally the product of manual authoring Figure 1. Search results page. that may require the original presentation materials (e.g.: PowerPoint). Research groups By default, talks are ordered by relevance to and commercial companies have created the query terms. Other sortable attributes applications to produce high quality rich media include publication date, number of slides, and lecture webcasts, but most solutions impose duration. The search results page also includes constraints to simplify content production [21]. controls to filter the search results according to Given a source lecture video, the first task is specific criteria (e.g., the year of publication, the the detection of presentation slides. Slide time channel, etc.). codes can be acquired by manual annotation, From the search results page the user can or for an automatic solution, time codes can be click through to the detailed view for a specific recorded by instrumenting the presentation talk, as shown in Figure 2. The right side of the system [26]. These solutions fail when interface contains the embedded video player non-preconfigured PCs, such as a guest above the lecture title and abstract. The left of lecturer's personal laptop, or unsupported the page displays detected slide thumbnails presentation software is used. Another solution extracted from the video. Slide thumbnails are is to capture slide images by mounting a still highlighted with a yellow border to indicate camera in front of the projection screen or by matches with at least one of the search installing an RGB capture device between the keywords. Selecting a keyframe cues the video presentation PC and the projector [7, 23]. to that specific time point in the embedded Solutions that use dedicated video cameras player on the right. and room instrumentation can produce rich lecture records, but are notoriously expensive to install, operate, and maintain [1, 18, 5, 17]. Systems that process lecture videos or projector image streams typically use simple frame differencing techniques and assume that the location of the slides within the video frame is known and fixed, and that slides dominate the video frame [5, 18, 7, 23]. Haubold and Kender [9, 10] focused on multi-speaker presentation videos and developed an enhanced multimodal segmentation using the audio stream to detect Figure 2. Viewing slides and using them to control the embedded player. speaker changes. The use of audio for segmentation has also been studied in [13] for 120 富士ゼロックス テクニカルレポート No.21 2012 技術論文 TalkMiner: A Lecture Video Search Engine lectures and in more general purpose video interactive browsing system. retrieval systems [12, 20]. Our work also employs basic visual frame differencing as a 4. System Description first step, which we extend with both spatial filtering and speaker appearance modeling to This section details the design and reduce the number of non-slide images in the implementation of TalkMiner including the final set of keyframes. system architecture, video indexing, and After capture, the next step is to build a text retrieval interface. An overview of the index for retrieval. Systems that rely on architecture appears in Figure 3. The system instrumenting the presentation software or has two main components: the back-end video assume the availability of original presentation processing and the front-end web server. The files can extract the original slide text [26, 14, back-end searches the web for lecture 15] for indexing. Other work uses automatic webcasts and indexes them. It executes speech recognition (ASR) to build a algorithms to identify slide images and applies time-stamped transcript of the lecture for OCR to create the keyword search index. retrieval