Synote: Weaving Media Fragments and Linked Data Yunjia Li,Mike Wald,Tope Omitola,Nigel Shadbolt and Gary Wills School of Electronics and Computer Science University of Southampton UK {yl2,mw,tobo,nrs,gbw}@ecs.soton.ac.uk ABSTRACT how these relationships can be processed automatically. How- While end users could easily share and tag the multime- ever, most applications are yet to expose sufficient semantic dia resources online, the searching and reusing of the in- annotations for media fragments, which leads to the diffi- side content of multimedia, such as a certain area within an culty of processing complex searching and providing accu- image or a ten minutes segment within a one-hour video, rate searching results. To solve this problem, some research, is still difficult. Linked data is a promising way to inter- such as Core Ontology of Multimedia (COMM) [3], has en- link media fragments with other resources. Many applica- abled semantic annotations for media fragments. But this tions in Web 2.0 have generated large amount of external approach has not yet proved to be scalable on the Web annotations linked to media fragments. In this paper, we [13]. Linked data [6] is more light-weight compared with use Synote as the target application to discuss how media traditional semantic Web technologies and is highly scalable fragments could be published together with external anno- on the Web. There are some research interests in apply- ing linked data principles to media fragments [13, 21]. The tations following linked data principles. Our design solves 1 the dereferencing, describing and interlinking methods prob- recent announced schema.org also defined vocabularies to lems in interlinking multimedia. We also implement a model improve the online presence of image, audio and video ob- to let Google index media fragments which improves media jects, but the media fragment is undefined in schema.org. fragments' online presence. The evaluation shows that our Most research about media fragments focuses on exposing design can successfully publish media fragments and anno- the closed data, such as tracks and video segments, within tations for both semantic Web agents and traditional search the multimedia file or the host server using semantic Web engines. Publishing media fragments using the design we and linked data technologies. But these efforts are inade- describe in this paper will lead to better indexing of multi- quate for the fully interlinking and indexing of media frag- media resources and their consequent findability. ments, because the large amount user-generated annotations in existing Web 2.0 applications are rarely published, such as YouTube interactive transcript2 and photo tagging func- Categories and Subject Descriptors tions in Facebook3. When a person is tagged in a photo, an I.7.2 [Document Preparation]: Hypertext/hypermedia, annotation is created and connected to the media fragment, Multi/mixed media, Standards; H.5.m [Information Sys- but the annotation is saved in an external source. More at- tems]: Miscellaneous tention should be given to how these external annotations from outside of multimedia host servers could benefit the media fragment interlinking and search. General Terms In this paper, we will present the design and evaluation of Theory, Design an example of publishing media fragments and annotations following linked data principles. We also discuss how the media fragments could be made available for both semantic Keywords indexing services, such as Sindice [40], and Google Search. annotation,linked data,media fragment,schema.org,search The external annotation data is from Synote [22], which is a Web 2.0 application allowing users to embed audio-visual resources from other domains and make synchronised anno- 1. INTRODUCTION tations. The discussion in this paper focuses on the tempo- The term \media fragment" refers to the inside content of ral dimension of audio-visual resources, which could also be multimedia objects, such as a certain area within an image, extended to images and spatial dimension. or a ten minutes segment within a one-hour video. With In the reminder of this paper, Section 2 reviews related the rapid development of multimedia applications, such as vocabularies that could be used for media fragment publish- YouTube and Flickr, end users could easily upload, share ing, and the state of art of the existing applications about and tag the multimedia resources online. Compared with multimedia and linked data. Section 3 briefly introduces traditional annotations, semantic annotations can tell a com- the Synote system and discusses the requirements of me- puter how media fragments are related to other data and 1 http://schema.org 2 http://goo.gl/t1nMj Copyright is held by the author/owner(s). 3 LDOW2012, April 16, 2012, Lyon, France. http://www.facebook.com/help/photos/tag-photos dia fragment publishing. Section 4 discusses the design to framework to describe and annotate complex multimedia solve the problems in interlinking multimedia (iM) princi- resources. It fills the gap between the structured metadata ples [13]. Then we present the implementation of the design models, such as SMIL and EXIF [35], and semantic anno- and a model to allow Google indexing media fragments. Sec- tations. M3O can be integrated with various standards to tion 5 presents the evaluation results and Section 6 provides provide semantic annotations. The key concept of M3O is conclusions and recommendations for future work. separation of information objects and information realiza- tions. Based on this idea, M3O comes up with a core set of 2. RELATED WORK ontology design patterns, including annotation, decomposi- tion, collection and provenance patterns. Linked data describes a series of methods of publishing International Press Telecommunication Council (IPTC) structured data using semantic Web technologies, such as designed an ontology for IPTC News Architecture [36] in Resource Description Framework (RDF) [24] and SPARQL order to integrate news and multimedia metadata from the query language [27]. Linked data enables machines to au- industrial area into the existing knowledge on the semantic tomatically discover more data from the data they already Web. In [36], a local view of a video resource is introduced know. Generally, there are four rules which must be followed so that arbitrary sequences of a video, i.e. media fragments, when publishing linked data on the Web [6]:1) Use URIs as can be played in the semantic browser. As the further de- names for things 2) Use HTTP URIs so that people can look velopment of IPTC News Architecture, rNews vocabulary up those names 3) When someone looks up a URI, provide has be adopted by schema.org, which provides shared vo- useful information 4) Include links to other URIs, so that cabularies in the form of HTML Microdata [15]. A great they can discover more things. advantage of using schema.org is that the structured data Heath and Bizer [14] summarised that there are some com- including the audio and video objects could be recognised mon steps which developers need to take from choosing URIs and highlighted by major search engines, such as Google, to the final testing and the publishing of linked data fall Bing and Yahoo!. The vocabularies in schema.org can also into several patterns. They also pointed out that developers be used in the form of RDFa4 [1]. should not totally abandon \existing data management sys- tem and business applications", but add an \extra technical layer of glue to connect these into the Web of Data". 2.2 Multimedia Applications for Linked Data Many applications have already published multimedia and 2.1 Multimedia Annotation Vocabularies annotation as linked data, which offers experience for us If we want to publish media fragments in Synote, we need on multimedia resource publishing. Yovisto open academic to choose how different dimensions are encoded in URIs. We video search platform publishes its database containing video also need to consider which vocabularies should be used to and annotations as linked data using MPEG-7, COMM to describe multimedia resources and annotations. Thus this describe multimedia data [42]. Annomation [20] is a tool to section reviews related standards and vocabularies. handle the input annotations from users developed by the The standardisation of media fragments is identified as an Open University in the UK. The annotations are saved in urgent need [37]. Many standards use non-URI based mech- RDF quad store with users' own privacy and provenance anisms to identify media fragments, such as MPEG-7 [25] data. SemWebVid [33] can automatically generate RDF de- and Synchronized Multimedia Integration Language (SMIL) scription by analysing plain-text resources using multiple [17]. In these standards, the descriptions of temporal and Natural Language Processing (NLP) Web services. The en- spatial dimensions are divided into several attributes, thus tities extracted from the NLP services are used to generate the media fragment is not represented by a single URI. In the the RDF description of the video resources. URI-based approaches, temporal, spatial and other dimen- LEMO multimedia annotation framework [12] publishes sions of media fragments are encoded in query (\?"), hash media fragments using MPEG-21. LEMO converts existing (\#") or after slash (\/"). The temporalURI [26] defines me- video files to MPEG compatible versions and streams them dia fragments after URI query. But Query parameter has a from LEMO server. LEMO also derived a core annotation weakness that the resulting URI loses the direct link to the schema from Annotea Annotation Schema [19] in order to parent resource and it does not follow the \Cool URIs" prin- link annotations to media fragments identifications. Nin- ciple [30], so it is difficult to be applied in linked data. The Suna (Metadata-driven media adaptation and delivery) [9] proposed URIs for Linked Stream Data [31] applied slash has a server-side implementation of MFURI 1.0.