Retrieval System
Total Page:16
File Type:pdf, Size:1020Kb
Multimedia in Digital Libraries− Some Research Issues in CADAL Project Yueting Zhuang Zhejiang University, P. R. China Email: [email protected] May 21,2004, CMU 1 Contents Introduction Personal Active Library Content-based multimedia analysis and retrieval Video structuring and summarization Rare cultural relic digital restoration Reasoning based on multimedia resources Conclusion 2 Introduction Multimedia, like video/audio/image/graphics etc, have been increasing in DLI. Tools used in the document-centered digital libraries should find their way in multimedia. Relevance feedback: text Æ image Keyword-based search Æ content-based retrieval In CADAL project, in addition to the book scanning, researches have been done on multimedia. 3 GraphicsDB DocumentDB ImageDB VideoDB AudioDB Digital Library Applications Users 4 Personal Active Library (PAL) Basic Idea of PAL Digital Libraries Bill’s Jack’s York’s Library Library Library 6 Design Goal of PAL Create a virtual digital library for every user by providing intelligent and personalized information service. Assign every user a personal space, allowing him to have specific resources. Design new method to simplify the navigation of complex information. Automatically discover new resource of user’s interest according to the metadata, people’s evaluation and user profile. Provide personalized report of latest resource. Design friendly user interface, enable user to navigate the library in a virtual 3D environment. Design a convenient setup procedure, in order that the system can migrate to other libraries. 7 Framework of PAL 8 System Architecture 9 Implementation Personalized Book Recommendation System User Behavior-based Learning System Client Browser with 3D Navigation and Message Exchanging 10 Personalized Book Recommendation System User profile is in the server database User profile contains user’s personal description Book Recommendation Scheme Static RecommendationCreate personal library for every new user by selecting resources of user’s interest Dynamic RecommendationReport of new books, recommendation among similar users). Bookmark Message Exchanging 11 User Behavior-based Learning System Track user’s access in a log file, analyze the log periodically, and provide the following service: ¾ Adjust user’s interest. If access to a certain class of books is particularly frequent, increase the degree of interest to this class. ¾ Assign a weight to each book and adjust it dynamically. Put books with higher weight in a notable place. 12 Client Browser with 3D Navigation and Message Exchanging 3D navigation Create different models of the library components, including book shelf, book, aisle, desk, etc., allowing user to customize the style of the library. Link the 3D model of the books with the digital books in the library. Design a scheme which enables the user to move, open or insert a book mark into a book on the shelf. Design the message exchanging scheme 13 Media Analysis and Summarization Video Structure Video Stream Video Table of Contents Concept Clustering Scene Scene Construction Group Grouping Shot Shot Boundary Detection Temporal Features Frame Spatial Features Key Frame Extraction 15 Video Metadata Generation and Annotation Get video metadata/annotation, support video retrieval and semantic analysis 16 Video Database Retrieval and Structure Browsing Video Retrieval Video Structure Browsing 17 Content Based Multimedia retrieval Contents Content Based multimedia retrieval: Including image, video, audio(music) and graphic retrieval. Multimedia integration and retrieval: Integrating the information of different media to provide integrated semantic analysis and retrieval of multimedia resources. Cross index and retrieval system for multimedia:Combining the semantic information and low level features of different media type to support multi-channel index and retrieval for multimedia resources. Integrating various media retrieval systems: Provide functions of image retrieval, audio retrieval, graphic retrieval, video retrieval and also support multi-channel retrieval. 19 TaskContent Based multimedia retrieval CBMR Image Video Audio Graphics retrieval Retrieval Retrieval Retrieval Image Video Audio Graphics Analysis Analysis Analysis Analysis 20 TaskMulti-channel index and retrieval system Text search User profile Text db engine adjusting Image search Image db engine Result integrate Audio search Audio db engine interface SVM clustering Video search Video db engine Cross index Graphic search Graphic db engine multi-channel query And cross index Search engine Cross index Preproces Models db s User feedback Semantic learning feedback Multimedia to adjust And results Documents factor of Refining module db importance 21 pcm2001-Yang Jun.pdf 22 ProgressImage retrieval system ImageOctopus System Features: Web based Client/Server architecture Integrating semantic and visual features of images Relevance feedback Retrieval effect: To get reasonable result using relevance feedback 23 ImageOctopus: The interface After two rounds of relevance feedback, we get acceptable results according to the semantic features. 24 ProgressVideo retrieval features Automatic segmentation of video flow. Detecting sudden and gradual changes of video shots. Extracting key frames of video shots Clustering similar video shots Video shots retrieval 25 ProgressAudio retrieval Features: Web based Client/Server architecture Extracting low level features directly from information of compressive domain of raw audio flow Blur clustering according to time-space constraints Efficient match of similar audio clips Adjusting retrieval results using relevance feedback 26 Audio clips retrieval system using relevance feedback technique 27 Publications: A Hierarchical Approach: Query Large Music Database by Acoustic Input(SIGIR2002) Popular Song Retrieval Based on Singing Matching 28 ProgressFlash retrieval system Features: Web based Client/Server architecture Get the expression of Flash animation, extract the features of embedded objects of animation Create index of Flash animation Content based retrieval of Flash animation Functions: Keywords based retrieval Advanced retrieval, shape based, motion based and interactive retrieval 29 ProgressFlash retrieval system 30 Publications: TM Search for Flash Movie on the Web 31 Progress3D models retrieval system Features: Depth weighted normal map: A new method for describing features of 3D models of any shape Efficient retrieval of 3D models 32 3D models retrieval results 33 Next Step Integrate the search engines for different types of media Develop the cross index and multi-channel multimedia retrieval system Construct the web-based search engine for very large scale of multimedia resources 34 Rare cultural relic digital restoration contents Multimedia information acquirement of cultural relics Data modeling of the cultural relics information Scene modeling of digital museum Integrated and interactive display of multimedia cultural relics information 36 Progress Accomplished Requirement analysis Research the related projects and technologies In progress Data modeling of the cultural relics information Integrated and interactive display of multimedia cultural relics information New method to digitalize 3-dimention cultural relics ranges from 10cm×10cm×10cm to 30cm×30cm×40cm. 37 The contrast between spoiled murals and restored murals 38 solutions According to the specific requirement of digitalizing cultural relics in precision and safety, considering current technologies and devices: Design an economic and effective Digitization method that suits the needs of cultural relics; Based on the device configuration and the research of key technologies, design and develop a information acquiring software system. 39 solutions With achievements of the research in data modeling of the cultural relics information and scene modeling of digital museum, a web based integrated and interactive display of multimedia cultural relics information can be developed. (Considering the tradeoffs among reality, interactive and real-time issues). 40 Reasoning of multimedia resources Construction Contents Retrieval System Calligraphy learning Calligraphy regeneration Meta data retrieval Professor comment Regeneration of new style Calligraphy character Facsimile lost Calligraphy Works retrieval stroke Visualization disk cache Shape Calligrap Web feature Feature hy base interface extraction Raw Networks Platform data 42 Retrieval system metadata-based retrieval : according to the author’s name,article’s name, dynasty’s name , terms etc. Document Image based retrieval: extract features from calligraphic characters which can’t be retrieved through OCR approach, and develop straight-forward retrieval system. 43 Calligraphy Learning System Facsimile the lost strokes and learning on- line To a calligraphy character with lost stroke,the students try to resume the lost stroke ,so as to get the experience of calligraphy imagination and help the calligraphy teaching. Professor Comment Famous professors’ comment to the classic calligraphy works,including the experience and learning feelings. 44 Regeneration of Calligraphy Character Generation of new calligraphy works styles By learning the current given calligraphy character style,the system will generate new styles. Visualization of Calligraphy works production process By the information analysis and feature extraction for a two- dimensional calligraphy works, the system