Multimedia in Digital Libraries− Some Research Issues in CADAL Project
Yueting Zhuang Zhejiang University, P. R. China Email: [email protected] May 21,2004, CMU
1 Contents
Introduction Personal Active Library Content-based multimedia analysis and retrieval Video structuring and summarization Rare cultural relic digital restoration Reasoning based on multimedia resources Conclusion
2 Introduction
Multimedia, like video/audio/image/graphics etc, have been increasing in DLI. Tools used in the document-centered digital libraries should find their way in multimedia. Relevance feedback: text Æ image Keyword-based search Æ content-based retrieval In CADAL project, in addition to the book scanning, researches have been done on multimedia.
3 GraphicsDB DocumentDB ImageDB VideoDB AudioDB
Digital Library
Applications
Users
4 Personal Active Library (PAL) Basic Idea of PAL Digital Libraries
Bill’s Jack’s York’s Library Library Library
6 Design Goal of PAL
Create a virtual digital library for every user by providing intelligent and personalized information service.
Assign every user a personal space, allowing him to have specific resources. Design new method to simplify the navigation of complex information. Automatically discover new resource of user’s interest according to the metadata, people’s evaluation and user profile. Provide personalized report of latest resource. Design friendly user interface, enable user to navigate the library in a virtual 3D environment. Design a convenient setup procedure, in order that the system can migrate to other libraries.
7 Framework of PAL
8 System Architecture
9 Implementation
Personalized Book Recommendation System User Behavior-based Learning System Client Browser with 3D Navigation and Message Exchanging
10 Personalized Book Recommendation System
User profile is in the server database User profile contains user’s personal description Book Recommendation Scheme Static Recommendation Create personal library for every new user by selecting resources of user’s interest Dynamic Recommendation Report of new books, recommendation among similar users). Bookmark Message Exchanging
11 User Behavior-based Learning System
Track user’s access in a log file, analyze the log periodically, and provide the following service: ¾ Adjust user’s interest. If access to a certain class of books is particularly frequent, increase the degree of interest to this class. ¾ Assign a weight to each book and adjust it dynamically. Put books with higher weight in a notable place.
12 Client Browser with 3D Navigation and Message Exchanging
3D navigation Create different models of the library components, including book shelf, book, aisle, desk, etc., allowing user to customize the style of the library. Link the 3D model of the books with the digital books in the library. Design a scheme which enables the user to move, open or insert a book mark into a book on the shelf. Design the message exchanging scheme
13 Media Analysis and Summarization Video Structure
Video Stream Video Table of Contents Concept Clustering Scene Scene Construction
Group Grouping
Shot Shot Boundary Detection Temporal Features Frame Spatial Features
Key Frame Extraction
15 Video Metadata Generation and Annotation
Get video metadata/annotation, support video retrieval and semantic analysis
16 Video Database Retrieval and Structure Browsing
Video Retrieval Video Structure Browsing
17 Content Based Multimedia retrieval Contents
Content Based multimedia retrieval: Including image, video, audio(music) and graphic retrieval. Multimedia integration and retrieval: Integrating the information of different media to provide integrated semantic analysis and retrieval of multimedia resources. Cross index and retrieval system for multimedia:Combining the semantic information and low level features of different media type to support multi-channel index and retrieval for multimedia resources. Integrating various media retrieval systems: Provide functions of image retrieval, audio retrieval, graphic retrieval, video retrieval and also support multi-channel retrieval.
19 Task Content Based multimedia retrieval
CBMR
Image Video Audio Graphics retrieval Retrieval Retrieval Retrieval
Image Video Audio Graphics Analysis Analysis Analysis Analysis
20 Task Multi-channel index and retrieval system
Text search User profile Text db engine adjusting
Image search Image db engine Result integrate
Audio search Audio db
engine interface SVM clustering Video search Video db engine Cross index
Graphic search Graphic db engine multi-channel query And cross index Search engine Cross index Preproces Models db s
User feedback Semantic learning feedback Multimedia to adjust And results Documents factor of Refining module db importance 21 pcm2001-Yang Jun.pdf
22 Progress Image retrieval system ImageOctopus
System Features: Web based Client/Server architecture Integrating semantic and visual features of images Relevance feedback Retrieval effect: To get reasonable result using relevance feedback
23 ImageOctopus: The interface
After two rounds of relevance feedback, we get acceptable results according to the semantic features. 24 Progress Video retrieval
features Automatic segmentation of video flow. Detecting sudden and gradual changes of video shots. Extracting key frames of video shots Clustering similar video shots Video shots retrieval
25 Progress Audio retrieval
Features: Web based Client/Server architecture Extracting low level features directly from information of compressive domain of raw audio flow Blur clustering according to time-space constraints Efficient match of similar audio clips Adjusting retrieval results using relevance feedback
26 Audio clips retrieval system using relevance feedback technique
27 Publications:
A Hierarchical Approach: Query Large Music Database by Acoustic Input(SIGIR2002) Popular Song Retrieval Based on Singing Matching
28 Progress Flash retrieval system
Features: Web based Client/Server architecture Get the expression of Flash animation, extract the features of embedded objects of animation Create index of Flash animation Content based retrieval of Flash animation Functions: Keywords based retrieval Advanced retrieval, shape based, motion based and interactive retrieval
29 Progress Flash retrieval system
30 Publications:
TM Search for Flash Movie on the Web
31 Progress 3D models retrieval system
Features: Depth weighted normal map: A new method for describing features of 3D models of any shape Efficient retrieval of 3D models
32 3D models retrieval results
33 Next Step
Integrate the search engines for different types of media Develop the cross index and multi-channel multimedia retrieval system Construct the web-based search engine for very large scale of multimedia resources
34 Rare cultural relic digital restoration contents
Multimedia information acquirement of cultural relics Data modeling of the cultural relics information Scene modeling of digital museum Integrated and interactive display of multimedia cultural relics information
36 Progress
Accomplished Requirement analysis Research the related projects and technologies
In progress Data modeling of the cultural relics information Integrated and interactive display of multimedia cultural relics information New method to digitalize 3-dimention cultural relics ranges from 10cm×10cm×10cm to 30cm×30cm×40cm.
37 The contrast between spoiled murals and restored murals
38 solutions
According to the specific requirement of digitalizing cultural relics in precision and safety, considering current technologies and devices: Design an economic and effective Digitization method that suits the needs of cultural relics; Based on the device configuration and the research of key technologies, design and develop a information acquiring software system.
39 solutions
With achievements of the research in data modeling of the cultural relics information and scene modeling of digital museum, a web based integrated and interactive display of multimedia cultural relics information can be developed. (Considering the tradeoffs among reality, interactive and real-time issues).
40 Reasoning of multimedia resources Construction Contents
Retrieval System Calligraphy learning Calligraphy regeneration
Meta data retrieval Professor comment Regeneration of new style
Calligraphy character Facsimile lost Calligraphy Works retrieval stroke Visualization
disk cache
Shape Calligrap Web feature Feature hy base interface extraction Raw Networks Platform data
42 Retrieval system
metadata-based retrieval : according to the author’s name,article’s name, dynasty’s name , terms etc. Document Image based retrieval: extract features from calligraphic characters which can’t be retrieved through OCR approach, and develop straight-forward retrieval system.
43 Calligraphy Learning System
Facsimile the lost strokes and learning on- line To a calligraphy character with lost stroke,the students try to resume the lost stroke ,so as to get the experience of calligraphy imagination and help the calligraphy teaching. Professor Comment Famous professors’ comment to the classic calligraphy works,including the experience and learning feelings.
44 Regeneration of Calligraphy Character
Generation of new calligraphy works styles By learning the current given calligraphy character style,the system will generate new styles. Visualization of Calligraphy works production process By the information analysis and feature extraction for a two- dimensional calligraphy works, the system will simulate the process of the calligraphy art’s writing.
45 Retrieval based on the meta data
Retrieval based on meta data
46 Retrieval based on Document Image
Calligraphic characters have many variations and deformations, even the same person will write different shapes of same character. Traditional OCR technique can’t work with such characters. The retrieval of the same character in different styles actually is the similarity matching base on characters’ shape similarity. Key problem:how to extract features from calligraphic character, and how to design matching algorithms.
47 Implementing approach
Extract shape context features of the calligraphic character Making similarity matching by X2 algorithm. Implemented Calligraphic characters retrieval based on Document Image or a scratched skeleton character.
48 Retrieved by shape similarity(can’t be done by OCR)
49 The query example is scratched by the user on the spot. 50 Next Task :The Visualization of the calligraphy works production
Stroke order database
Adjust Adjust visual display Skeleton Analysis of angel rate stroke order Character generation information Structural depiction Boundary Of boundary Paper parameter visualization of model production process Brush parameter model
51 Next Task Generation of new calligraphy styles
Boundary of Calligraphy character Information of calligraphy character borderline Skeleton of Calligraphy character Information of strokes in the Calligraphy character and over-all topology structure Pixels distribution of Calligraphy Character statistics Features of Pixels distribution of Calligraphy
By analogy learning of the above information , the system will generate new styles Automatic Artistic Calligraphy Generation
The accomplished Automatic Rule-based Artistic
Calligraphy Generation 53 Conclusion
The following supporting systems are being built: Multimedia Information Retrieval. Features: • Image retrieval • Video retrieval • Audio retrieval • ……. A Personal Active Library Chinese Calligraphic database and retrieval
54 ThankThank youyou !!!!!!
55