Journal of Virtual Reality and Broadcasting, Volume 3(2006), no. 11 Video Search: New Challenges in the Pervasive Digital Video Era Katerina Pastra and Stelios Piperidis Institute for Language and Speech Processing, Artemidos 6 and Epidavrou, 151-25, Maroussi, Greece, email kpastra,[email protected] Abstract REVEAL THIS The explosion of multimedia digital content and the development of technologies that go beyond tradi- 1 Introduction tional broadcast and TV have rendered access to such The proliferation of digital multimedia content and the content important for all end-users of these technolo- subsequent growth of the number of digital video li- gies. While originally developed for providing access braries have boosted research on the development of to multimedia digital libraries, video search technolo- video retrieval systems. Video search technology has gies assume now a more demanding role. In this paper, traditionally been conceived as a way to enable effi- we attempt to shed light onto this new role of video cient access to large video data collections, with au- search technologies, looking at the rapid developments tomatic indexing and cataloguing of this data being in the related market, the lessons learned from state an essential derivative of its development. The quest of art video search prototypes developed mainly in for efficient video retrieval mechanisms is still ongo- the digital libraries context and the new technologi- cal challenges that have risen. We focus on one of ing, with a number of different unimodal and mul- the latter, i.e., the development of cross-media deci- timodal techniques being implemented and evaluated sion mechanisms, drawing examples from REVEAL (cf. TRECVID competitions, [HC04]). THIS, an FP6 project on the retrieval of video and lan- However, the emergence of technologies crossing guage for the home user. We argue, that efficient video the boundaries between traditional broadcast and the search holds a key to the usability of the new “per- Internet, and between traditional television and com- vasive digital video” technologies and that it should puters broadens the scope of developing video search involve cross-media decision mechanisms. functionalities. Does this, new, reinforced role of video search render it indispensable for end-users of Keywords: Interactive TV, Video Search, Video the new technologies? Within the rapidly changing Retrieval, Pervasive Digital Video, Digital Libraries, multimedia-processing context, does video search be- come an even more challenging task in terms of the Digital Peer Publishing Licence retrieval performance required for achieving high us- Any party may pass on this Work by electronic ability of the new technologies it is embedded in? means and make it available for download under In this paper, we look into the role of video search in the terms and conditions of the current version the light of the new “convergent” technologies and the of the Digital Peer Publishing Licence (DPPL). technological challenges that are subsequently posed The text of the licence may be accessed and on its development. In order to do so, we present the retrieved via Internet at market status and trends in video search for the new http://www.dipp.nrw.de/. technologies, as well as the search mechanisms used within commercial and research prototypes. We dis- First presented at the 4th European Interactive cuss the effects of the new role subsumed by video TV Conference EuroITV 2006, extended and revised search and focus mainly on the use of cross-media de- for JVRB cision mechanisms for dealing with such effects. Last, urn:nbn:de:urn:nbn:de:0009-6-10736, ISSN 1860-2037 Journal of Virtual Reality and Broadcasting, Volume 3(2006), no. 11 we present REVEAL THIS, a research project which 2.1 The market players attempts to implement cross-media mechanisms for increasing the usability of both its pull and push video Electronics manufacturers, software companies, access scenarios. telecommunication giants, cable companies, broad- casters, content owners all have an interest in the new market that is being created and which extends 2 Market Status and Trends in Video beyond professional users to everyday laymen, to Search home users. As expected, the convergence of internet and TV has not only resulted in the convergence The formation of regulations for the new technolo- of the corresponding business sectors, but has also gies that enable traditional broadcast and the internet created new ones, which are interested in providing to converge (e.g. Internet Protocol TV (IPTV), Peer- end-to-end services, i.e., aggregation of video (and to-Peer (P2P) networks, Mobile TV) proves that these other) digital content and distribution of this content technologies -with the ever growing popularity- have to interested users through a push (data routing become something more than a trend or optimistic according to a user profile) or pull (data search and prospect ([Car05, Bro05]). They already are a new re- retrieval) model. ality, in which: The business sectors which are being actively in- volved in providing an enhanced TV experience (i.e. ² Traditional TV sets can be extended with in- IPTV, i-TV, mobile TV etc.) could be classified in the telligent digital video recorders (DVRs), set-top following categories according to their main business boxes with PC-like functionalities, or can even activities1: communicate with personal computers for dis- playing streamed digital media through gaming consoles enabling an enhanced, interactive TV a Content owners: production companies (broad- experience [SBH+05]. casters who also produce their own content e.g. BBC are included in this category too) ² TV viewing goes beyond traditional TV sets, in mobiles and portable digital media play- b TV service providers: satellite and cable compa- ers (i-pods) enabling on-the-move TV watching nies (e.g. Comcast, BeTV etc.) and TV broad- [Cha05]. casters (e.g. CNN, RTBF etc.) ² File-swapping networks and headline syndica- c P2P service networks: networks that allow for tion technology facilitates the exchange of not file-swapping e.g. BitTorrent, e-donkey etc. only professional/copyrighted TV programmes but also of consumer-generated ones (e.g. pod- d Electronics manufacturers: manufacturers of casts, video blogs, etc.) [Bor05, Jaf05]. DVRs and set-top boxes (e.g. TiVo, Akimbo, Sci- In opening new markets, suggesting new business entific Atlanta etc.), portable digital media play- models, and indicating new content distribution chan- ers (e.g. Apple), mobiles (e.g. Siemens). nels, all these technologies boost the availability of digital video content and motivate digitisation. In this e Computer networking companies (e.g. Cisco), light, the question of easy and efficient access to video Internet Service Providers (e.g. AOL) and Phone content that was once posed in relation to digital video companies (e.g. BellSouth, Verizon etc.) collections raises again more demanding than ever. In this section, we will look into the role of video f Internet Protocol TV software developers (e.g. search within this new context. This will shed light Microsoft, Myrio, Virage) on the implications this new role has on the develop- ment of video search mechanisms and will point to- g Content service providers: content monitoring wards possible aspects that need to be taken into con- companies which provide push and/or pull ser- sideration for developing highly usable video access vices (e.g. TVEyes, BlinkX) services. We present the interested parties in the mar- 1The categories are not mutually exclusive; on the contrary, ket of the new TV/video technologies and their stance there are organisations with a wide range of activities, which span towards video search functionalities. more than one category. urn:nbn:de:urn:nbn:de:0009-6-10736, ISSN 1860-2037 Journal of Virtual Reality and Broadcasting, Volume 3(2006), no. 11 h Web content aggregators: companies that aggre- the features of its DVRs, allowing the user to down- gate digital media (text, audio, video) or links to load movies from the internet, buy products, search these media and present them online to a user local movie theatre listings etc. [Shi06]. Computer upon request/search e.g. Google and Yahoo networking companies such as Cisco are also extend- ing their reach to the consumer networking market and i Content re-packaging companies: companies IPTV for set-top boxes, in particular, through a num- that acquire content e.g. sports videos/TV pro- ber of company acquisitions [Rea05b]. In most cases, grammes and re-package it for meeting various in-house IPTV software comes part-and-parcel with user needs e.g. interactive viewing of a car the products of the electronics manufacturers (i.e. set- race, where the user can choose his/her preferable top boxes, DVRs etc.). When not, alliances or acquisi- viewing angle(s) (e.g. Nascar) tions of businesses take place, with IPTV software de- velopers playing a key-role; for example, Siemens has All these market players have their own interest acquired Myrio in its attempt to enter the IPTV market in the new technologies and therefore attempt to ex- [Rea05a]. Deals between more than one business sec- tend the services they provide by either enriching in- tors are also growing; for example, Microsoft, BT and house developments or by making alliances with each Virgin are collaborating for providing mobile TV ser- other. Figure 1 illustrates the relative position of each vices, Microsoft contributing all software needed for business sector in the chain from content owners to packaging and viewing TV on mobiles, BT contribut- end-users of enhanced-TV. The chain reveals the de- ing a new network for mobile TV and Virgin mak- pendencies between the different market sectors for ing use of the network and providing the service in achieving an end-to-end service; these dependencies a new mobile phone [Rea06].