Proceedings of the Thirtieth International Florida Artificial Intelligence Research Society Conference

ViewpointS: When Social Ranking Meets the

Philippe Lemoisson,1 Guillaume Surroca,2 Clément Jonquet,2 Stefano A. Cerri2 1UMR Territories, Environment, Remote Sensing and Spatial Information, CIRAD, Montpellier, France 2Laboratory of Informatics Robotics Micro-Electronics of Montpellier, France [email protected];{surroca, jonquet, cerri}@lirmm.fr

Abstract and Apostolou 2006). The frontier between them has nev- Reconciling the ecosystem of semantic Web data with the ertheless become porous: on one hand, the Web 2.0 con- ecosystem of social Web participation has been a major is- tributors attempt to explore more and more Web data be- sue for the Web Science community. To answer this need, fore posting comments and ranking contents; on the other we propose an innovative approach called ViewpointS hand the data analysis algorithms start to integrate social where the knowledge is topologically, rather than logically, micro-contributions to improve their results. Bridging the explored and assessed. Both social contributions and linked gap between social ranking and the semantic Web is a big data are represented by triples agent-resource-resource called “viewpoints”. A “viewpoint” is the subjective decla- issue: it might be the key of a collective knowledge system ration by an agent (human or artificial) of some semantic able to bring in new levels of understanding in the sense proximity between two resources. Knowledge resources and coined by Gruber in (Gruber 2008). viewpoints form a bipartite graph called “knowledge The current paper is a proof of concept for an approach graph”. Information retrieval is processed on demand by stepping forward in this direction called ViewpointS. After choosing a user’s “perspective” i.e., rules for quantifying and aggregating “viewpoints” which yield a “knowledge presenting our major sources of inspiration within the state map”. This map is equipped with a topology: the more of the art, we propose a formalism supporting the storage viewpoints between two given resources, the shorter the dis- and exploitation of knowledge, and detail a topological tance; moreover, the distances between resources evolve assessment of information. Then we experiment all the along time according to new viewpoints, in the metaphor of novel aspects of our approach with the MovieLens dataset. synapses’ strengths. Our hypothesis is that these dynamics actualize an adaptive, actionable collective knowledge. Finally, we discuss the formalism and measurements and We test our hypothesis with the MovieLens dataset by briefly expose directions for future work. showing the ability of our formalism to unify the semantics issued from linked data e.g., movies’ genres and the social Web e.g., users’ ratings. Moreover, our results prove the Related work and inspirations relevance of the topological approach for assessing and comparing along the time the respective powers of ‘genres’ Facing the issue of bridging the gap between social ranking and ‘ratings’ for recommendation. and the semantic Web, (Breslin, Passant, and Vrandečić 2011) distinguish two main options in the literature con- sisting respectively in “post-formalizing” and “pre-

Introduction formalizing” the informal content of the social Web. Since the last decade, reconciling the ecosystem of seman- The ViewpointS approach takes a third option: to repre- tic Web data with the ecosystem of social Web participa- sent both social contributions and linked data by triples tion has been a major issue for the Web Science communi- agent-resource-resource. The major consequence of this ty (Hendler et al. 2008). Each ecosystem has its own dy- choice of a memory built upon connections made by agents namics and challenges: the countless subjective contribu- is to let the collected knowledge evolve continuously along tions of the social Web yield , but without the interactions of the contributors. Another consequence is centrally controlled coherence (Mika 2007; Mikroyannidis to ensure trustworthiness, well in line with the recent shift 2007) whereas semantically linking datasets whose models in the Web paradigm bringing the contributor back in the evolve independently is a never ending task (Karapiperis loop (O’Reilly 2015). In ViewpointS, we started from a tripartite model agent- resource-tag generalizing social networks such as Flickr Copyright © 2017, Association for the Advancement of Artificial Intelli- and proposed as building block the more abstract triple gence (www.aaai.org). All rights reserved. agent-resource-resource, in which agents may themselves

329 be resources, expressing: “the belief of a resource/agent that two resources are close”. This yields a topological, /07-'&)'2'33052%' rather than logical, exploitation of the “wisdom of the crowd” in a manner similar to (Mika 2007; Markines et al. 2009; Specia and Motta 2007). Taking inspiration from the contexts of annotation, disambiguation, concept alignment 5.#/ )'/4 5.'2+% 02')#- '230/ 0%5.'/4 and information retrieval (Harispe et al. 2014; Lee et al. 2008), we then grouped the triples in order to yield ‘seman- '3%2+1402 tic similarity’ or ‘semantic proximity’. We firstly aggregate 24+(+%+#- *83+%#- all the triples connecting two given knowledge resources )'/4 0%5.'/4 into a higher level binary link called a synapse. In a second step, we made the hypothesis that any user's context could Figure 1: Knowledge resources be translated into a set of quantification rules for the view- • Human Agent or Legal person: entity “performing acts points called a perspective; this is in line with (Kim and and undertaking obligations” [32] such as humans or or- Scerri 2008) which advice to evaluate ontologies on de- ganizations, by emitting connections between resources. mand with respect to a specific context. Once a perspective • Artificial Agent: numeric entity emitting connections is adopted, the initial heterogeneous semantics carried by between resources. the viewpoints are transformed into a knowledge map • Physical document: document of the real world such as a equipped with a distance. As a consequence, the agents can book. use the proximities resulting from the existing viewpoints • Numeric document: numeric entity such as a Web page. when browsing knowledge maps and reversely update the • Descriptor: meaningful linguistic expression, playing the knowledge through new viewpoints expressing their feed- role of tag. back e.g., “I believe this agent matches/does-not-match the Knowledge resources participate to connections called topic of my query” or “I deny concept-a/ instance-1 and viewpoints: each viewpoint is a subjective connection es- concept-b/ instance-2 are connected through relation-r”. tablished by an agent (Human or Artificial) between two Along these exploitation/feedback cycles, the shared knowledge resources; each viewpoint implicitly brings in knowledge is continuously elicited against the beliefs of the specific semantics of its emitter. However, these se- the agents in a selection process supported by the evolving mantics do not need to be shared by the agents that will strength of synapses. Here comes the metaphor of the further exploit the knowledge, as it will be explained when brain. According to the Theory of Neuronal Group Selec- describing the “knowledge maps”. The formalization of tion (Edelman and Tononi 2000), which recently re- these heterogeneous semantics is illustrated in Figure 2: appeared in the front scene under the name of connectome (Seung 2012), knowledge results from a process of contin- /07-'&)' ual re-categorization. In our approach, the dynamics of 2'33052%' synapses aim at yielding an evolving topology where knowledge is constantly reorganized1. /07-'&)' )'/4 T W The next section firstly presents the formalism support-  2'33052%' ing the ViewpointS “Knowledge graph”, then goes through 6+'710+/4 some aspects exploring its topology in order to assess col- lective knowledge on top of heterogeneous semantics. Figure 2: A “viewpoint”. The straight arrow gives the prove- nance; ‘T’ gives the semantics, ‘τ’ gives the time stamp

A topological, unified view on heterogeneous The viewpoint (a1, {r2, r3}, T, τ) stands for: the agent a1 semantics believes at time τ that r2 and r3 are related according to the semantics carried by T. For instance: In the ViewpointS approach, all the resources (identified The viewpoint (AA-MovieLens, {movieX, genreY}, by a URI) contributing to knowledge are grouped in a mo:genre, τ1) stands for: the artificial agent ‘AAMov- single class: Knowledge resource. Figure 1 illustrates five ieLens’ declares at τ1 that the physical document ‘movieX’ mutually exclusive subclasses of knowledge resources matches the descriptor ‘genreY’; in this example, covering most of the practical cases: T=mo:genre is a standard unambiguous semantic Web property issued the MovieLens linked data. The viewpoint (userX, {movieY, ***}, vp:rating, τ2) stands for: the legal person ‘userX’ declares at τ2 that the 1 This point requires a real-life scenario and cannot be demonstrated in physical document ‘movieY’ matches the descriptor ‘*** = this paper; setting such a scenario is our next objective.

330 of average interest’; in that example, T=vp:rating is a tran- ogy of which can be exploited with standard graph algo- scription of the MovieLens linked data. rithms in order to exhibit knowledge. In practice,  is The viewpoints’481'3 are grouped into 7 .'4#481'3 never built exhaustively! Instead, the synapses are comput- {author, like, similar, onto-match, social-match, algo- ed on demand along a Dijkstra-inspired exploration bound- match, preview}. ed by a parameter ‘m’. We denote . < /'+)*$05232the neighborhood of a target ‘r’ The knowledge graph resulting from an exploration bounded by ‘m’ in the per- Knowledge resources and viewpoints together form a bi- spective . This can be compared to the semantic similarity partite Knowledge graph2 denoted , where the of (Zhu and Iglesias 2017); however using the perspective knowledge is persistently stored. By contrast, retrieving only requires knowledge about .'4#481'3i.e., the information happens in transient maps computed on de- user does not need any preliminary knowledge about the mand. ontologies source of the viewpoints of .'4#481' “onto- match”. The knowledge maps Two things should be noted: i) the bigger the synapse the shorter the distance and ii) as long as is small com- A preliminary step for retrieving knowledge consists in ‘m’ pared to the size of, the exploration of new branches reifying the querying context by use of a perspective  quickly stops; therefore the worst-case complexity3 is nev- i.e., a set of quantification rules applied to the viewpoints. er reached in practical cases. The practical complexity does It may be default rules adopted by a group of users in a not depend on the size of but on its local density. recurrent context, or specific rules filtering according to preferences such as: ignoring the viewpoints anterior to a given date, privileging the viewpoints emitted by some The topological assessment of the knowledge agents or privileging viewpoints of a given .'4#481'. (Adomavicius and Tuzhilin 2005; Yamaba, Tanoue, and Given a , computing the knowledge map associated to Takatsuka 2013) write that powerful recommender systems a context involves three steps: i) grouping all the view- will exploit more and more the underlying topologies. points connecting any given pair of knowledge resources Taking inspiration from this, our objective is to provide a into higher level link called synapses and then ii) choosing means to characterize the underlying topology in  in the rules for valuating the synapses i.e., setting the per- terms of information. For example, if when browsing spective as a map-reduce process. This is illustrated in through a given movie dataset the uncertainty about genres Figure 3. is reduced by its topology, we may say according to (Klir 2005) that the dataset embeds information about ‘genres’. 7 Let us consider in  a collection ‘O’ of 0$,'%43 2 2 (e.g., ‘movies’) and a collection ‘D’ of &'3%2+14023 7  (e.g., ‘genres’), we measure whether close elements of ‘O’  2  2  have similar elements of ‘D’ in their respective neighbor- 7 hoods. We call “local homogeneity” (of O with respect to 2 D) the probability to find similar elements of ‘D’ in the 2     neighborhoods of close elements of ‘O’. Let |D| be the cardinal of D, in order to compute the “local homogeneity” we use a |D|-dimensional vector space.     Let be a perspective, let ‘m’ be the parameter for com-      puting neighborhoods, let O={0+}+:: and D ={&,},:: 7   be two collections of knowledge resources, we denote 7       , D0  the vector of dimension :: {& 0 }  such  ,  + , :: 7  ,  . that & 0+number of occurrences of ‘d,’ in < U Figure 3: Building a knowledge map KM /'+)*$05230+. .-0%#-0.0)'/'+48Ois the average value of We call knowledge map and denote the undirected  cosine similarity (D0 , D0 ) computed upon all the labelled graph interpretingthrough the perspective . pairs0 0 verifying <0 0 d.. Depending on the perspective adopted, one single may This will be denoted: .-O). therefore be interpreted into several distinct , the topol-

2 Although our approach has not yet addressed large datasets such as DBpedia, it has been thought to do so. 3 The worst case complexity of the algorithm is O(|W|²|R|²).

331 The value of local homogeneity varies between ‘ ’ and x the viewpoint 06+''/39.06+'!)'/ ‘1’. A local homogeneity of ‘ ’ corresponds to the 2'"; .0)'/2'W stands for: the #24+(+ absence of information expressed by D with respect to O. %+#-#)'/4 ‘AAMovieLens’ connects the 0$ A local homogeneity of ‘1’ corresponds to the maximal ,'%4 ‘movieX’ to the descriptor ‘genreY’ information expressed by D with respect to O. Since the We then consider that each 53'2& emits self- computation of “local homogeneity” only comes into play describing viewpoints with .'4#481' “onto-match”: for remotely assessing the evolution of collective x the viewpoint 53'2& 953'2& 0%%51# knowledge, we do not worry about the complexity4 result- W ing from the size of |D|. 4+0/!;%6,0$ 81'  We finaly consider that each 53'2& rate movies through viewpoints with .'4#481' “social-match”: The MovieLens Experiment x the viewpoint 53'2! 9.06+'" ; To prove the concept of ViewpointS, we take a Web da- 612#4+/)W stands for: the -')#- 1'230/ taset where explicit knowledge expressed by linked data is ‘user&’ considers the 0$,'%4 ‘movieY’ as ‘of mixed with implicit knowledge issued from social contri- average interest’ butions: MovieLens5. The complete dataset consists of 100.000 movies and 1.000.000 ratings which have been Protocol and measurements collected by the GroupLens Research Project at the Uni- We initialize the knowledge graph by transcribing the versity of Minnesota. Their data have played the role of movies descriptions and users profiles as explained upper. experimental matter for many authors (Peralta 2007; Each movie is linked to one or more of the 18 ‘genres’ by Harpale and Yang 2008; Jung 2012). In the following, we viewpoints of type .0)'/2' and to one ‘year’ by a take an extract of the original dataset: 1682 movies rated viewpoint of type .02'-#3'#4'. Each user is linked by 943 commenters providing 5.000 ratings. We aim at to one ‘class of age’ by a viewpoint of type illustrating: (0#(.'.$'2, one ‘gender’ by a viewpoint of type a) how a Web dataset involving linked data from the (0#()'/&'2, and one ‘occupation’ by a viewpoint of semantic Web together with individual opinions is- type %6,0$ 81'. All these viewpoints are time-stamped sued from social ranking can be transcribed into a W ; this is called %8%-' . We then arbitrary split the rat- unique knowledge graph; ings into 5 subsets of 1000 ratings each (called %8%-' to b) how the implicit knowledge associated to ‘ratings’ %8%-'), using viewpoints of type 612#4+/) time- and the explicit knowledge issued from linked data stamped W+, where‘i’ is the cycle number. can be both (and yet distinctly) topologically as- We consider 3 perspectives: sessed.  reflects priority given to explicit knowledge, i.e., viewpoints of .'4#481' “social-match” are weighted  The ViewpointS model for MovieLens whereas viewpoints of .'4#481' “onto-match” are To transcribe the explicit information embedded in Mov- weighted . In this perspective, the distance between a ieLens into ViewpointS, we firstly create knowledge re- ‘movie’ and its ‘genre’ is  , so that consequencetwo sources: movies of same ‘genre’ are at distance <=  ; we there- x 06+'&stands for movies asPhysical documents fore expect ‘m=1’ to be relevant for measuring the local x 3'2&stands for users asLegal persons homogeneity in ‘genres’ through this perspective. x 06+'-'/3is the Artificial agent corresponding  reflects balanced importance of explicit versus im- to the MovieLens Ontology plicit knowledge, i.e., all viewpoints are weighted . x )', )'/2'&, )'/&'2, %%51#4+0/, "'#2  reflects priority given to implicit knowledge, i.e., are considered as Descriptors viewpoints of .'4#481' “social-match” are weighted  We then consider that 06+''/3 is the emitter whereas viewpoints of .'4#481' “onto-match” are 6 of viewpoints with .'4#481' “onto-match ”: weighted . In the perspectives and , viewpoints of 481' 4 It might happen that we later need to undergo dimensional reduction in a .0)'/2' are valuated ‘1’, so that two movies con- way similar to the “AffectiveSpace” of (Cambria et al. 2015). nectedto the same ‘genre are at distance’ <= ; we there- 5 http://datahub.io/dataset/movielens fore expect ‘m=2’ to be relevant for measuring the local 6 We use the semantic Web properties in the transcription process as often as possible, e.g., mo:genre, where the prefix ‘mo:’ refers to the Movie homogeneity in ‘genres’ through these two perspectives. Ontology (Bouza 2010) which unambiguously defines its resources and Besides, themultiplicity of ratings by different users in properties. Doing so, we enable ViewpointS to embrace the semantic Web the dataset will yield strong synapses and in consequence standards and import/export RDF representations. short distances. As a consequence, both ‘m=1’and ‘m=2’

332 are expected suitable for measuring the local homogeneity Moreover, the intersection of the curves corresponding in ‘ratings’, whatever the perspective. to these two dimensions occurs after cycle3 i.e., after The experiment consists in going through %8%-' to viewpoints ‘rating’, to be compared with  viewpoints %8%-' in 3 successive runs and measuring “local homo- ‘genre’. We interpret it as: “ratings and genres are equally geneities” at the end of each cycle of each run. powerful informational dimensions”.

-  1,1 corresponds to the perspective with . Finally, the curves intersect at a point quite independent -  2,2 corresponds to the perspective with . from the perspective. This reinforces the hypothesis that -  3,2 corresponds to the perspective with . “local homogeneity” proves information within the hidden topology of , whatever the map. A close look shows that Denotations the highest local homogeneity at intersection point appears - M is the population of 1682 movies in the perspective :  for    ,   for    , - Genres is the collection of 18 genres  for    . We interpret it as “the balanced perspec- - Ratings is the collection {, , , , } tive is the best suited for observing simultaneously - +.  -Mis the local homogeneity of movies genres and ratings”. with respect to genres - +.#4+/)3- (M) is the local homogeneity of movies Discussion with respect to ratings The MovieLens dataset has fulfilled our demonstration objectives: Results and interpretation a) we have transcribed all the information stored in the The 3 runs corresponding to the three perspectives associ- MovieLens dataset (social ranking and explicit se- mantics) into a unique knowledge graph. ated with the suitable ‘m’ parameters (namely  1,1, b) we have topologically assessed the progressive dis-  2,2 and  3,2) are presented in Figure 4. semination of the implicit knowledge associated to ‘ratings’, in a knowledge graph initially embedding only the explicit knowledge of linked data. Let us discuss these two points more in detail. Our for- malism seems suitable for capturing both explicit knowledge delivered by linked data and implicit

RUN1,1 RUN2,2 RUN3,2 knowledge available in the Social Web. We cannot pretend however to capture all the richness of the semantic Web: for instance, the viewpoints fail in expressing the conceptu- al ‘parent-child’ relationship or ‘composition-component’      relationships, usually known as ‘is a’ and ‘is part of’ re- Figure 4: Assessment of explicit knowledge (genres) and spectively. We capture only flat descriptions, not the con- implicit knowledge (ratings) along the interaction cycles ceptual verticality of ontologies. This built-in limitation rules out the possibility of logical assessment, but allows We firstly observe that +.   M always takes -  the topological assessment of proximity in a context of significant values (between and ), whatever   heterogeneous semantics. the cycle and the perspective. We interpret it as “genres Moreover our unified characterization of the collective provide information about movies, despite the noise due to knowledge seems well suited for observing the Web dy- other informational dimensions”. namics: we have provided topological arguments for as- We then observe that +.  M always in- -  sessing that “genres and ratings are complementary dimen- creases along the cycles (between and ), what-    sions competing for providing information about movies ever the cycle and the perspective. We interpret it as “rat- recommendation, with comparable power”. ings provide information about movies, roughly in propor- Another important aspect of the approach is the discon- tion with their number, despite the noise due to other in- nection between the storage of the knowledge events (the formational dimensions”. viewpoints), and their delayed interpretations through The major output of the experiment is that the value of knowledge maps responding to distinct perspectives. The +.   M decreases while +.#4+/)3 (M) -   - viewpoints are purely qualitative; interpreting them in increases along the cycles, whatever the run. This illus- terms of quantities entirely depends on the perspective trates concurrence and complementarity between the topo- taken by the final user. Perspectives can be seen as a kind logical assessments respectively associated to ‘genres’ and of global social ranking occurring “at exploration time”. to ‘ratings’. We interpret it as “genres and ratings are inde- This might be a response to the usual biases in social net- pendent informational dimensions”.

333 works resulting from self-promotion or dishonest recom- Weitzner, D. 2008. Web Science: An Interdisciplinary Approach mendation, since untrusted viewpoints emitters can easily to Understanding the World Wide Web. Communications of the ACM 51 (7): 60–69. be discarded when tuning the perspective. Jung, J. J. 2012. Attribute Selection-Based Recommendation Framework for Short-Head User Group: An Empirical Study by MovieLens and IMDB. Expert Systems with Applications 39 (4): Future work 4049–4054. We are currently developing an API offering intuitive Karapiperis, S; and Apostolou, D. 2006. Consensus Building in input, easy browsing of the knowledge and one-click feed- Collaborative Ontology Engineering Processes. Journal of back. The next step in our agenda is to prove the concept in Universal Knowledge Management, 199–216. real life scenarios, i.e., to invite users to elicit knowledge Kim, H.L.; and Scerri, S. 2008. The State of the Art in Tag Ontologies: A Semantic Model for Tagging and Folksonomies. collectively by using the ViewpointS approach through the API mentioned above. Two use cases have been planned, Klir, G. J. 2005. Uncertainty and Information: Foundations of Generalized Information Theory. 2005. Wiley-IEEE Press both oriented towards cross-disciplinary discoveries: one in Lee, W-N.; Shah N., Sundlass, K.; and Musen, M. 2008. the biomedical domain, in the context of the SIFR project, Comparison of Ontology-Based Semantic-Similarity Measures. the other in the agronomic domain will be hosted by Cirad. AMIA, Annual Symposium 2008, January, 384–388. Markines, B.; Cattuto, C.; Menczer, F.; Benz, D.; Hotho, A.; and Stumme, G. 2009. Evaluating Similarity Measures for Emergent Acknowledgements Semantics of Social Tagging. In 18th International Conference on World Wide Web - WWW ’09, 641–50. Madrid, Espagne: This work was supported in part by the French National ACM Press. Research Agency under JCJC program, grant ANR-12- Mika, P. 2007. Ontologies Are Us: A Unified Model of Social JS02-01001, as well as by University of Montpellier, Networks and Semantics. Web Semantics: Science, Services and CNRS and the CIRAD. Agents on the World Wide Web 5 (1): 5–15. Mikroyannidis, A. 2007. Toward a Social Semantic Web. Computer. 40(11): 113-115. References O’Reilly, T. 2015. What Is Web 2.0: Design Patterns and Adomavicius, G.; and Tuzhilin, A. 2005. Toward the next Business Models for the Next Generation of Software. Generation of Recommender Systems: A Survey of the State-of- http://papers.ssrn.com/abstract=1008839 (accessed on november the-Art and Possible Extensions. IEEE Transactions on 2016). Knowledge and Data Engineering 17 (6). IEEE: 734–749. Peralta, V. 2007. Extraction and Integration of Movielens and Bouza, A. 2010. MO - the Movie Ontology | Leverage Your Imdb Data. Laboratoire Prisme, Université de Versailles, Movie Information, http://www.movieontology.org/ (accessed on Versailles, France. february 20th 2017). Seung, S. 2012. Connectome: How the Brain’s Wiring Makes Us Breslin, J. G.; Passant A.; and D Vrandečić D. 2011. Social Who We Are. Houghton Mifflin Harcourt. Semantic Web. In Handbook of Semantic Web Technologies, Specia, L.; and Motta, E. 2007. Integrating Folksonomies with the 467–506. Springer Berlin Heidelberg. Semantic Web. In ESWC '07 Proceedings of the 4th European Cambria, E.; Fu J., Bisio, F.; and Poria, P. 2015. AffectiveSpace conference on The Semantic Web: Research and Applications. 2: Enabling Affective Intuition for Concept-Level Sentiment Pages 624-639. Innsbruck, Austria. Springer-Verlag. Analysis. In Proceedings of the Twenty-Ninth AAAI Conference Yamaba, H.; Tanoue, T.; and Takatsuka, K. 2013. On a on Artificial Intelligence (AAAI'15). AAAI, 508–514. Serendipity-Oriented Recommender System Based on Edelman, G. M., and Tononi, G. A Universe of Consciousness: . Artificial Life and Robotics 18(1-2): 89–94. How Matter Becomes Imagination. 2000. Basic Books - 274 Zhu, G.; and Iglesias, C. A. 2017. Computing Semantic Similarity pages of Concepts in Knowledge Graphs. IEEE Transactions on Gruber, T. 2008. Collective Knowledge Systems: Where the Knowledge and Data Engineering 29 (1): 72–85. Social Web Meets the Semantic Web. Web Semantics: Science, Services and Agents on the World Wide Web 6 (1): 4–13. doi:10.1016/j.websem.2007.11.011. Harispe, S.; Ranwez, S.; Janaqi, S.; and Montmain, J. 2014. The Semantic Measures Library and Toolkit: Fast Computation of Semantic Similarity and Relatedness Using Biomedical Ontologies. Bioinformatics (Oxford, England) 30 (5): 740–742. Harpale, A. S.; and Yiming, Y. 2008. Personalized Active Learning for Collaborative Filtering. In Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval - SIGIR ’08, 91. New York, USA: ACM Press. Hendler, J.; Shadbolt, N.; Hall, W.; Berners-Lee, T.; and

334