Exploring the Value of Folksonomies for Creating Semantic Metadata Hend S

Int’l Journal on Semantic Web & Information Systems, 3(1), 13-39, January-March 2007 13

Exploring The Value Of Folksonomies For Creating Semantic Metadata Hend S. Al-Khalifa, University of Southampton, UK

Hugh C. Davis, University of Southampton, UK

Abstract

Finding good keywords to describe resources is an on-going problem. Typically, we select such words manually from a thesaurus of terms, or they are created using automatic keyword extraction techniques. Folksonomies are an increasingly well-populated source of unstructured tags describing Web resources. This article explores the value of the folksonomy tags as a potential source of keyword metadata by examining the relationship between folksonomies, community produced annotations, and keywords extracted by machines. The experiment has been carried out in two ways: subjectively, by asking two human indexers to evaluate the quality of the generated keywords from both systems; and automatically, by measuring the percentage of overlap between the folksonomy set and machine generated keywords set. The results of this experiment show that the folksonomy tags agree more closely with the human generated keywords than those automatically generated. The results also showed that the trained indexers preferred the semantics of folksonomy tags compared to keywords extracted automatically. These results can be considered as evidence for the strong relationship of folksonomies to the human indexer’s mindset, demonstrating that folksonomies used in the del.icio. us bookmarking service are a potential source for generating semantic metadata to annotate Web resources.

Keywords: metadata; Web technologies; folksonomy; keyword extraction; tags; social bookmarking services; collaborative tagging

INTRODUCTION us2, and Furl3 rely extensively on folk- Nowadays, contemporary Web sonomies. Folksonomies, as a widely applications such as Flickr1, del.icio. accepted neologism and one of Web

Copyright © 2007, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited. 14 Int’l Journal on Semantic Web & Information Systems, 3(1), 13-39, January-March 2007

2.0 signatures, can be thought of as social bookmarking services, followed keywords that describe what a docu- by a review of related work concerning ment is about. folksonomies and keyword extraction Since people started using the del. techniques. We then discuss the ex- icio.us service in late 2003, many re- perimental setup and the data selection, sources have been bookmarked and along with the four experiments we tagged collaboratively. Using the ser- have carried out to examine the degree vice, people usually tag a resource with of the relationship. Finally, the results words they feel best describe what it is of these experiments, as well as a case about; these words or tags are popularly study, conclusion, and future work are known as folksonomies and the process discussed. as collaborative tagging. We believe that most folksonomy Folksonomy and Social words are more related to a profes- Bookmarking Services sional indexer’s mindset than keywords The growing popularity of folkson- extracted using generic or proprietary omies and social bookmarking services automatic keyword extraction tech- has changed how people interact with niques. the Web. Many people have used social The main questions this experiment bookmarking services to bookmark Web tries to answer are (1) Do folksonomies resources they feel most interesting to only represent a set of keywords that them; and folksonomies were used in describe what a document is about, or these services to represent knowledge do they go beyond the functionality about the bookmarked resource. of index keywords? (2) What is the relationship between folksonomy tags Folksonomies and keywords assigned by an expert The word folksonomy is a blend of indexer? (3) Where are folksonomies the two words “folks” and “taxonomy.” positioned in the spectrum from profes- It was first coined by the information sionally assigned keywords to context- architect Thomas Vander Wal in August based machine extracted keywords? of 2004. Folksonomy, as Vander Wal In order to find out if folksonomies (2006) defines, is: can improve on automatically extracted keywords, it is significant to examine … the result of personal free tagging of the relationship between them, and information and objects (anything with between them and professional human a URL) for one’s own retrieval. The indexer keywords. tagging is done in a social environment To study these relationships, our (shared and open to others). The act of article is organized as follows: we begin tagging is done by the person consuming with an overview of folksonomies and the information.

Figure 1. Excerpt from the del.icio.us service showing the tags (Blogs, Internet, ... ,cool) for the URL of the article by Jonathan J. Harris, the last bookmarker (pacoc, 3mins ago) and the number of people who bookmarked this URL (1494 other people)

From a categorization perspec- Social Bookmarking Service tive, folksonomy and taxonomy can Social bookmarking services are be placed at the two opposite ends of server-side Web applications where categorization spectrum. The major people can use these services to save difference between folksonomies and their favorite links for later retrieval. taxonomies is discussed thoroughly in Each bookmarked URL is accompanied Shirky (2005) and Quintarelli (2005). by a line of text describing it and a set Taxonomy is a top-down approach. of tags (aka folksonomies) assigned by It is a simple kind of ontology that pro- people who bookmarked the resource vides hierarchical and domain specific (as shown in Figure 1). vocabulary, which describes the ele- A plethora of bookmarking services ments of a domain and their hierarchal such as Furl4, Spurl5, and del.icio.us ex- relationship. Moreover, they are created ists; however, del.icio.us is considered by domain experts and librarians, and one of the largest social bookmarking require an authoritative source. services on the Web. Since its introduc- In contrast, folksonomy is a bottom- tion in December 2003, it has gained up approach. It does not hold a specific popularity over time and there have vocabulary nor does it have an explicit been more than 90,000 registered users hierarchy. It is the result of peoples’ using the service and over a million own vocabulary, thus, it has no limit (it unique tagged bookmarks (Menchen, is open ended), and tags are not stable 2005; Sieck, 2005). Visitors and users nor comprehensive. Most importantly, of the del.icio.us service can browse folksonomies are generated by people the bookmarked URLs by user, by who have spent their time exploring keywords (i.e., tags or folksonomies), and interacting with the tagged resource or by a combination of both techniques. (Wikipedia, 2006). By browsing others’ bookmarks, people can learn how other people tag their

Copyright © 2007, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited. 16 Int’l Journal on Semantic Web & Information Systems, 3(1), 13-39, January-March 2007 resources thus increasing their aware- icio.us community). By the same token, ness of the different usage of the tags. Guber is working on a system called In addition, any user can create an “TagOntology” to build ontologies inbox for other users’ bookmarks by out of folksonomies, and in his article subscribing to another user’s del.icio. entitled “Ontology of Folksonomy: us pages. Also, users can subscribe to A Mash-up of Apples and Oranges” RSS feeds for a particular tag, group of (Gruber, 2005) he casts some light on tags or other users. design considerations needed to be taken into account when constructing ontolo- Related Work gies from tags. In addition, Ohmukai, In this section, we review related Hamasaki, and Takeda (2005) proposed work discussing state-of-the-art folk- a social bookmark system called “so- sonomy research and various techniques cialware” using several representations in the keyword extraction domain. of personal network and metadata to construct a community-based ontology. State-of-the-Art Folksonomy The personal network was constructed Research using Friend-Of-A-Friend (FOAF), There is alot of recent research deal- Rich Site Summary (RSS), and simple ing with folksonomies, among these are Resource Description Framework overviews of social bookmarking tools Schema (RDFS), while folksonomies with special emphasis on folksonomies were used as the metadata. Their system as provided by Hammond, Hannay, allows users to browse friends’ book- Lund, and Scott (2005) and other re- marks on their personal networks and search papers that discuss the strengths map their own tag onto more than one and weaknesses of folksonomies (Guy tag from different friends, so that they & Tonkin, 2006; Kroski, 2006; Mathes, are linked by the user. This technique 2004; Quintarelli, 2005). will allow for efficient recommenda- Another genre of research has ex- tion for tags because it is derived from perimented with folksonomy systems. personal interest and trust. They also For instance, Mika (2005) has carried used their social bookmark system out a study to construct a community- “socialware” to design an RDF-based based ontology using del.icio.us as a metadata framework to support open data source. He created two lightweight and distributed models. ontologies out of folksonomies; one is Golder and Huberman (2006) have the actor-concept (user-concept) ontol- analyzed the structure of collaborative ogy and the other is the concept-instance tagging folksonomies) to discover the ontology. The goal of his experiment regularities in user activity, tag frequen- was to show that ontologies can be built cies, the kind of tags used, and bursts of using the context of the community in popularity in bookmarked URLs in the which they are created (i.e., the del. del.icio.us system. They also developed

Copyright © 2007, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited. Int’l Journal on Semantic Web & Information Systems, 3(1), 13-39, January-March 2007 17 a dynamic model that predicts the stable (around 250k bookmarked resources) patterns in collaborative tagging and and showed that their algorithm yielded relates them to shared knowledge. Their a set of related users and resources for results show that a significant amount of a given tag. Therefore, “FolkRank” can tagging is done for personal use rather be used to generate recommendations than public benefit. However, even if within a folksonomy system. the information is tagged for personal Versa (2006a) has presented a study use other users can benefit from it. They in which the linguistic properties of also state that del.icio.us, for most users, folksonomies demonstrated that us- functions as a recommendation system ers engaged in resource tagging are even without explicitly providing rec- performing classification according to ommendation. principles similar to formal taxonomies. In MIT labs, an experiment was car- To prove his findings, Versa analyzed the ried out by Liu, Maes, and Davenport, kinds of classification observed in user 2006) to generate a taste fabric of social tags using the non-taxonomic categories networks. Folksonomies were used in proposed by the linguist Anna Wierz- the experiment to weave the taste fabric. bicka. He then compared users’ patterns Their idea was based on philosophical to those observed for two well-known and sociological theories of taste and sources of classification schemes on identity to weave a semantic fabric the Internet: the open directory project of taste. They mined 100,000 social (DMOZ) and the Yahoo directory. His network profiles, segmented them into findings showed that there is a clear interest categories and then normalized difference between folksonomy tags the folksonomies in the segments and and the two classification schemes. mapped them into a formal ontology of Tags are drawn from most categories identity and interest descriptors. Their while DMOZ and YAHOO were biased work has inspired us with the idea of only towards one category (namely using folksonomies in the process of functional category). In another article semantic annotation. by the same author entitled “Concept Hotho, Jäschke, Schmitz, and Stum- Modeling by the Messes: Folksonomy me (2006) have presented a new search Structure and Interoperability,” (Versa, algorithm for folksonomies called 2006b) has used folksonomies to model “FolkRank,” which exploits the struc- concepts in a domain. He used a method, ture of the folksonomy. Their proposed based on the linguistic properties of the algorithm is used to support the retrieval tags, to extract structural properties of of resources in the del.icio.us social free form user tags to construct ontol- bookmarking services by ranking the ogy. The resultant ontology is a simple popularity of tags. They demonstrated conceptual domain model built from their findings on a large-scale dataset automatically mediated collaboration;

Copyright © 2007, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited. 18 Int’l Journal on Semantic Web & Information Systems, 3(1), 13-39, January-March 2007 this ontology has been used to facilitate only if user tagging provides a similar interoperability between applications or better search context. dependent tag sets. Finally, Kipp (2006) has examined Apparently, the method that Kipp the differences and similarities between used did not compare folksonomies to user keywords (folksonomies), the keywords extracted automatically us- author, and the intermediary (such as ing context-based extraction methods. librarians) assigned keywords. She used This extra evaluation method will be a sample of journal articles tagged in significant in measuring the relationship the social bookmarking sites citeulike6 between automatic machine indexing and connotea7, which are specialized mechanisms led by a major search en- for academic articles. Her selection of gine such as Yahoo compared to human articles was restricted to a set of jour- indexing mechanisms. nals known to include author assigned From the previous discussion, the keywords and to journals indexed in reader can observe that most research on the information service for physics, folksonomies is either user-centric (e.g., electronics, and computing (INSPEC Mika (2005) and Ohmukai et al. (2005) ) database so that each article selected or tag-centric (e.g., Gruber (2005), Versa would have three sets of keywords as- (2006a,b), Liu et al. (2006), and Hotho signed by three different classes of meta- et al. (2006). Little research has been data creators. Her methods of analyses conducted on examining the relation- were based on concept clustering via ship between folksonomies and other the INSPEC thesaurus and descriptive indexing systems. statistics. She used these two methods to examine differences in context and KEYWORD EXTRACTION—A term usage between the three classes BRIEF LITERATURE REVIEW of metadata creators. Kipp’s findings Keywords extraction--as a field showed that many users’ terms were of information retrieval (IR)—is an found to be related to the author and approach to formally study document intermediary terms, but were not part text to obtain “cognitive content hidden of the formal thesauri used by the in- behind the surface” (Hunyadi, 2001). termediaries; this was due to the use of Keyword extraction tools vary in com- broad terms, which were not included plexity and techniques. Simple term in the thesaurus or to the use of newer extraction is based on term frequency terminology. Kipp then concluded her (tf) while complex ones use statistical article by saying that: techniques (e.g., Matsuo and Ishizuka (2004) or linguistic techniques “natu- User tagging, with its lower apparent ral language processing (NLP)” (e.g., cost of production, could provide the ad- Sado, Fontaine, and Fontaine (2004) ditional access points with less cost, but supported by domain specific ontolo-

Copyright © 2007, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited. Int’l Journal on Semantic Web & Information Systems, 3(1), 13-39, January-March 2007 19 gies (e.g., Hulth, Karlgren, Jonsson, tribution tendency. Statistical methods Boström, and Asker (2001). There are lack precision and they fail in extracting a wide variety of applications that use the semantic indexes to represent the automatic keyword extraction; among main concepts of a document (Kang & these are document summarization and Lee, 2005). This problem might be par- news finding (e.g., Martínez-Fernández, tially solved by using manually assigned García-Serrano, Martínez, and Villena keywords or tags (i.e., folksonomies) in (2004). Keyword analyzer services8 bookmarking systems like del.icio.us. used by most search engine optimiza- tion (SEO) companies are another type Experiment Setup and of keyword extraction application using Test Data term frequency. Most complex keyword There are plenty of keyword ex- extraction techniques require corpus traction techniques in the IR literature, training in a specific domain for example most of which are either experimental Kea9—a keyphrase extraction algorithm or proprietary, so they do not have a cor- (Witten, Paynter, Frank, Gutwin, and responding freely available product that Nevill-Manning, 1999). can be used. Therefore we were limited On the other hand, search engines to what exists in this field such as SEO use one kind of keyword extraction keyword analyzer tools, Kea, an open called indexing, where the full search is source tool released under the GNU constructed by extracting all the words General Public License, and Yahoo API of a document except stop words. After term extractor10. Of these, the Yahoo all the keywords have been extracted, API was the preferred choice. the document needs to be filtered; since Kea requires an extensive train- not all words can be adequate for indexing in a specific domain of interest to ing. The filtering can be done using the come out with reasonable results; SEO vector space model or more specifically tools on the other hand, were biased by latent semantic analysis (Landauer, (i.e., they look for the appearance of Foltz, & Laham, 1998; Martinez-Fer- popular search terms in a Web page nandez et al., 2004). when extracting keywords), besides the From our previous discussion, we IR techniques they are using are very find that most indexing methods are basic (e.g., word frequency/count). The based on term frequency, which ignores decision to use Yahoo API was made the semantics of the document content. for the following reasons: This is because the term frequency technique is based on the occurrences • The technique used by Yahoo’s API of terms in a document assigning a to extract terms is context-based weight to indicate its importance. Most as described in Kraft, Maghoul, indexing techniques rely on statistical Chang, and Kumar (2005), which methods or on the documents term dis- means it can generate results based

Copyright © 2007, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited. 20 Int’l Journal on Semantic Web & Information Systems, 3(1), 13-39, January-March 2007

on the context of a document; this along with the accomplished results. will lift the burden of training the system to extract the appropriate The Comparison System keywords. Framework • Also, Yahoo’s recent policy of We constructed a system to auto- providing Web developers with a matically compare the overlap between variety of API’s encouraged us to the folksonomy, Yahoo TE, and human test the quality of their term extrac- indexer keywords and generate the tion service. desired statistics. The system consisted of three distinct components: the term The experiment was conducted extractor, the folksonomy extractor, in four phases: in the first phase, we and the comparison tool as shown in exposed a sample of both folksonomy Figure 2. The term extractor consists and Yahoo keywords sets to two trained- of two main components: JTidy11, an human indexers who, given a generic open source Java-based tool to clean classification, evaluated which set held up HTML documents and Yahoo Term greater semantic value than the other. Extractor (TE)12, a Web service that In the second phase, we used another provides “a list of significant words modified instrument from Kipp (2006) or phrases extracted from a larger to further explore the semantic value of content.” After removing HTML tags folksonomy tags and the Yahoo key- from a Web site, the result is passed to words. In the third phase, we measured, Yahoo TE to generate the appropriate for a corpus of Web literature stored in keywords. the del.icio.us bookmarking service, the The folksonomy extractor that we overlap between the folksonomy set and developed is designed to fetch the key- Yahoo extracted keyword set. In the final words (tags) list for a particular Web phase, one of the human indexers was site from del.icio.us and then clean-up asked to generate a set of keywords for the list by pruning and grouping tags. a sample of Websites from our corpus Finally, the comparison tool role is to and compare the generated set to the compare the folksonomy list to Yahoo’s folksonomy set and the Yahoo TE set keywords by counting the number of to measure the degree of overlap. Thus, overlapped keywords between the two the analysis of the experiment can be sets. The tool then calculates the per- thought of as being in two forms: term centage of overlap between the two sets comparison (phase 1 and 2) and descrip- using the following equation (1): tive statistics (phase 3 and 4). The rest of this article will talk about N P = ×100 the comparison system framework used (Fs + Ks) - N (1) for evaluating phase 3 and 4, the data set, and the different phases of the experiment

Figure 2. The comparison system framework

Del.icoi.us DB Folksonomy Folksonomy List Link Extractor Web

Comparison Document

Term Result Terms Extractor html List text REST XML

Yahoo API

The previous equation can be also traction service only extracts terms expressed using set theory as (2): from textual information. By the same token, whole Blog sites were avoided because they usually hold a F ∩K diversity of topics; we tried to look P = s s ×100 (2) for Web pages with a single theme F s∪K s (e.g., a specific post in a Blog). Data Selection • We only choose bookmarked sites The test data used in this experiment with 100 participating taggers; this was randomly collected from the del. was necessary to ensure there were icio.us social bookmarking service. One enough tags describing the Web hundred bookmarked Web sites span- site. ning various topics from the popular tags Web page were selected13, as shown Other General Heuristics in Table 1. Some other heuristics were used The selected Web resources were during the experiment lifecycle to chosen based on the following heu- improve the quality of the extraction ristics: results, which are listed as follows: • Bookmarked sites that are of a mul- 1. Most Web sites that use Google timedia nature such as audio, video, Adsense (an advertisement tool by flash, Word/PDF documents, etc., Google) affected the results of the were avoided, as the Yahoo term ex- terms returned by Yahoo extractor.

Copyright © 2007, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited. 22 Int’l Journal on Semantic Web & Information Systems, 3(1), 13-39, January-March 2007

Table 1. Topics covered in the experiment data set

Topic Number of Web Sites Software 11 Open source 14 Education 6 Programming 18 Sciences 8 Linux 10 References 13 Development 20 Total 100

Therefore, in some cases we were Thus, given the sets of keywords forced to manually enter (i.e., copy from Yahoo TE and del.icio.us, the and paste) the text of a Web site and two indexers were asked to blindly14 place it in a Web form that invokes evaluate each keyword from both sets. the Yahoo TE service. The indexers were provided by a five- 2. Yahoo TE is limited to produce only category table to classify the keywords twenty terms, which may consist from both sets. The table has the follow- of one or more words to represent ing values: “strongly relevant,” encoded the best candidate for a Web site 5, “relevant,” encoded 4, “undecided,” (as mentioned on the service Web encoded 3, “irrelevant,” encoded 2, and site). These terms were split out “strongly irrelevant,” encoded 1. into single words so that they might After evaluating 10 Web sites from match del.icio.us style single word our data set, an inter-rater reliability tags. test was conducted for each evaluated Web resource to measure the evaluation Results agreement between the two indexers. This step is essential to measure the Phase 1 consistency among the two indexers. The role of phase one is to deter- The inter-rater agreement reliabil- mine whether or not folksonomies carry ity test that we used to measure the more semantic value than keywords consistency of classifying keywords extracted using Yahoo TE. In this phase, into categories without any ordering the phrase “semantic value” means that (i.e., nominal data), was the Kappa the tag or keyword used to describe a (k) coefficient, the widely accepted Web resource is relevant to its gist (i.e., measurement developed by Cohen the tag or keyword contributes to the (1960). The value of the resulting description of the resource meaning). Kappa coefficient indicates the degree

Table 2. Average inter-rater agreement for the 10 evaluated Web resources in phase 1 Average Inter-Rater Agreement [Kappa- coefficient value] Folksonomy 0.2005 Yahoo TE 0.2162

of agreement between the two raters. evaluated Web site from both indexers. For interpreting the meaning of the For all values except for site 2, 5, and 8, resulting Kappa value we used Landis the results for the folksonomy set was and Koch’s (1977) interpretation, where higher or equal to Yahoo TE values. By 0 ≤ k < 0.2 means slight agreement, 0.2 further inspecting the three cases (2, ≤ k < 0.4 means fair agreement, 0.4 ≤ k 5, and 8), the authors have found that < 0.6 means moderate agreement, 0.6 what affected the average mode value ≤ k < 0.8 means substantial agreement, in these three cases in the folksonomy and 0.8 ≤ k < 1.0 means almost perfect set was the amount of general tags agreement. used to describe these Web resources Table 2 shows the overall average compared to the same Yahoo TE set degree of agreement between the two for these resources, which extracted indexers for the 10 evaluated Web more specific keywords (i.e., same or resources. The obtained Kappa value narrower term). for both sets falls in the fair level of The results also show that the agreement, which is considered satis- folksonomy and Yahoo TE sets scored factory for the purpose of this experi- an equal mode value (4 = relevant) for ment. However, the results show that all sites. The values for the Yahoo TE agreement between the indexers about varied considerably compared to the the folksonomy set is slightly lower folksonomy values but the most fre- (0.2005) than their agreement about the quent value in Yahoo TE was still 4, Yahoo TE set (0.2162); the difference is which appeared 3 times compared to 7 statistically significant at p< 0.001. The times in the folksonomy set. lower kappa value for the folksonomy Moreover, the results show that set was due to a slight disagreement the folksonomy set has a higher mean in evaluating one of the Web sites in and lower standard deviation (i.e., that set, which affected the results ac- 4.15(0.24)), this indicates a low vari- cordingly. ance in the views of the two indexers The values summarized in Table 3 toward classifying folksonomy tags, show the average mode value for each compared to the values for Yahoo TE

Copyright © 2007, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited. 24 Int’l Journal on Semantic Web & Information Systems, 3(1), 13-39, January-March 2007

Table 3. The average mode values for each Web site in both folksonomy (F) and Yahoo TE (K) set along with the mean, mode, and standard deviation for all 10 evaluated Web sites Site F K 1 4.5 4 2 4 4.5 3 4 3 4 4 2.5 5 4 4.5 6 4.5 3 7 4 1.5 8 4 4.5 9 4 4 10 4.5 4 Mean 4.15 3.55 SD. 0.24 1.01 Mode 4 4

(i.e., 3.55(1.01)), which indicates a of the folksonomy set and the Yahoo high variance in the views of the two keywords set compared to the Web re- indexers. source hierarchical listing in the dmoz. These results indicate that the org directory and to its title keywords folksonomy tags are more relevant to (afterwards, these will be called de- the human indexer’s conception than scriptors). Thus, the two indexers were Yahoo TE keywords. Furthermore, the provided with another categorization. difference between the two means was The new categorization values were statistically significant at p< 0.001. adopted from Kipp (2006). Kipp built The results of this phase give us the her scale instrument based on the dif- big picture of the semantic relationships ferent relationships in a thesaurus as an held in the folksonomy and Yahoo TE indication of closeness of match, into keywords compared to the two indexers the following categories: views. To better understand the semantics of each classified keyword in the folk- • same: The descriptors and tags or sonomy and Yahoo TE sets, an in depth keywords are the same or almost the analysis is carried out in phase 2. same (e.g., plurals, spelling varia- tions, and acronyms); encoded 7, Phase 2 • synonym: The descriptors and tags The role of phase 2 was to inspect or keywords are synonyms; encoded in more detail the semantic categories 6,

Table 4. Average inter-rater agreement for the ten evaluated Web resources in phase 2

Average Inter-Rater Agreement [Kappa- coefficient value] Folksonomy 0.2257 Yahoo TE 0.2241

• broader Term (BT): The keywords Table 4 shows the degree of agree- or tags are broader terms of the ment between the two indexers. The descriptors; encoded 5, agreement between the two indexers • Narrower Term (NT): The key- gave us a fair level of agreement with words or tags are narrower terms almost equal scores for the folksonomy of the descriptors; encoded 4, set (0.2257) and the Yahoo TE set • related Term: The keywords or (0.2241). The difference between the tags are related terms of the descrip- two means was statistically significant tors; encoded 3, at p< 0.001. • related: There is a relationship The values summarized in Table 5 (conceptual, etc.) but it is not obvi- show the average mode value for each ous to which category it belongs to; evaluated Web site from both indexers. encoded 2, Notice this time for all values, except • Not Related: The keywords and for site 3, the results for the folksonomy tags have no apparent relationship set was higher than Yahoo TE values. to the descriptors, also used if the By further inspecting site 3, the authors descriptors are not represented at have found that what caused the drop all in the keyword and tag lists; down of the average mode value in this encoded 1. site was the number of tags assigned to this Web site (i.e., 18 tags compared to The two indexers applied the modi- 28 keywords from Yahoo TE) and also fied categorization scale to a sample the class of the tags used to describe of 10 bookmarked Web sites that were the Web site, which fall more in the chosen from the experiment corpus. related category. After evaluating the 10 bookmarked The results also show that the folk- Web sites, an inter-rater reliability test sonomy set scored a higher mode value was conducted to evaluate the agree- (5) compared to Yahoo TE (2). However, ment between the two indexers for their the results show that the folksonomy evaluation of each Web resource. set has a higher mean and higher stan-

Copyright © 2007, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited. 26 Int’l Journal on Semantic Web & Information Systems, 3(1), 13-39, January-March 2007

Table 5. The average mode values for each Web site in both Folksonomy (F) and Yahoo TE (K) set along with the mean, mode, and standard deviation for all 10 evaluated Web sites

Site F K 1 5 1.5 2 5 1 3 1.5 2 4 5 2.5 5 5 2 6 3.5 2 7 5 3 8 6 2 9 3.5 3 10 5 1 Mean 4.45 2 SD. 1.28 0.71 Mode 5 2

dard deviation (i.e., 4.45(1.28)), which purple bars denote the folksonomy tags indicates a high variance in the views frequency. of the two indexers towards classifying Figure 3 shows the accumulated folksonomy tags, compared to the val- bar graph obtained by juxtaposing each ues for Yahoo TE (i.e., 2(0.71)), which individual bar graph of the 10 evaluated indicates a lower variance in the views Web resources, for both indexers, in a of the two indexers--the difference be- layered fashion so that a general conclu- tween the two means was statistically sion can be drawn easily. significant at p< 0.001. Comparing the two figures shows The resultant statistical analysis us that there is almost a good agreement of this phase stressed the finding of between indexer (a) and indexer (b) in the previous phase and gave us more the assignment of Yahoo TE keywords insight in how folksonomies are con- in the “not-related” category. However, sidered semantically richer than Yahoo this agreement starts to fluctuate, in TE keywords. order of magnitudes between the two Furthermore, to visualize the results indexers, in the similarity categories of this phase, a two-column bar graph (i.e., same, synonym, BT, NT, related was generated for each evaluated Web term, and related). resource to reflect the result of each For instance, in Figure 3(a), the category (i.e., the blue bars denote the folksonomy tags are accumulating Yahoo keywords frequency and the more around the “broader term” and

Figure 3. A visualization of the categorization results for the 10 Web resources layered on top of each other resulting in a ghost effect—(a) corresponds to the results of the first indexer; (b) corresponds to the results of the second indexer. Frequency

Same Synonym BT NT Related Related Not Term Related

Yahoo TE (a)

Folksonomy

Not Same Synonym BT NT Related Related Term Related

(b)

“related” category, while in Figure(b), pared to a small portion distributed in the folksonomy tags are accumulating the similarity categories. Also, the figure more around the “‘broader term” and shows that in all similarity categories “related term” category. the folksonomy set outperforms the The figure also shows that most of Yahoo keyword set. the folksonomy tags fall in the similarity Finally, we believe that the variance categories compared to a small portion, between the two indexers categorization which falls in the “not related” category. was due to either the different interpreta- In contrast, most of the Yahoo keywords tion of the meaning of the categories or fall in the “not related” category com-

Copyright © 2007, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited. 28 Int’l Journal on Semantic Web & Information Systems, 3(1), 13-39, January-March 2007 the use of single category with different what the users think of the bookmarked frequencies, as in the case of indexer resource. These tags suggest that the (b), thus a further marginal homogeneity bookmarked Web resource might be analysis using the Stuart and Maxwell useful. test to identify the sources of variability Self-reference tags include any will be considered for future work. tags that have to do with the user’s own interest. Examples are dates (e.g., More In-Depth Analysis of Phase 2 “January,” “monthly,” and “night”) In this section, a detailed analysis names (e.g., “tojack” and/or own ref- of both the Yahoo keywords set and erence (e.g., “mylink,” “mysite,” and the folksonomy set falling in the “not “myblog.”) These tags usually appear related” and “related” categories is once or twice among all the tags in a discussed. given bookmarked Web resource. On the other hand, Yahoo keywords A. Unrelated Tags falling in the “not related” category To explore in greater depth the na- do not follow a recognized pattern as ture of tags falling in the “not related” folksonomy tags do. Most keywords category, a further inspection was car- seem to be words that have occurred ried out to analyze the type of tags and frequently in the text or in the URL of a keywords found in this category. Web resource; alternatively the position Folksonomy tags falling in the of the word and its style (e.g., heading “not related” category tend to be either or sub-title) might be the reason for time management tags (e.g., “todo,” extracting it. The algorithms that Yahoo “toread,” “toblog,” etc.,) or expression TE uses to extract keywords from Web tags (e.g., “cool” self-reference tags sites are obscure which affects further and sometimes unknown/uncommon analyses of the extracted keywords. abbreviations). Time management tags, as Kipp B. Related Tags said, suggest that the users want to be This category represents relation- reminded of the bookmarked resource, ships that are ambiguous or difficult but have not yet decided what to do to place into the previous similarity with it. These kinds of tags do not ap- categories. These tags often occur when pear in any controlled vocabulary or there is a relationship between the tag or thesaurus; they are made up for the keyword and its field of study, or/and a user’s own needs and do not have any relationship between two fields of study value to anyone except the individual (Kipp, 2006) who created them. An example of the first mentioned Another common type of unrelated relationship would be of a Web resource tag is the use of expression tags (e.g., talking about open source software, “cool,” “awesome,” etc.) These reflect which has tags such as “code” or “down-

Copyright © 2007, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited. Int’l Journal on Semantic Web & Information Systems, 3(1), 13-39, January-March 2007 29 load.” These two tags do not appear spectrum from professionally assigned explicitly in the dmoz.org directory list- keywords to context-based machine ing nor in the title of the Web resource; extracted keywords, and to measure the however, they describe the field of “open scope of this overlap. source” software where someone can The overlap measurement used in download and play with the code. Fur- our comparison framework was inter- thermore, in a Web resource that gives preted using set theory (Stoll, 1979). examples about FreeBSD, a particular We considered the folksonomy set of version of the UNIX operating system, tags as set F, keywords set from Yahoo del.icio.us users have tagged the Web TE as set K and keywords set from the resource with related tags such as tuto- indexer as set I, hence: rial, tips, and how-to, these tags were not explicitly mentioned in the Web F = {the set of all tags generated by resource; however, they contributed to people for a given URL in del.icio. the description of the Web resource by us} giving it a new contextual dimension. K = {the set of all automatically extract- Another example of a relationship ed keywords for a given URL} between two fields of study is a Web I = {the set of all keywords provided resource about an open source office ap- by the indexer} plication called “NeoOffice” for the Mac operating system. This Web resource is Using set theory the degree of over- tagged with tags such as “Microsoft” lap was described using the following and “OpenOffice” to denote the rela- categories: tionship between the “Mac OS” and “Microsoft,” and between “NeoOffice” 1. No overlap (e.g., F≠K or F∩K=∅ and “OpenOffice” applications. (i.e., empty set)). 2. Partial overlap (this is know as the Phase 3 intersection) (e.g., F∩K). As mentioned in the experiment 3. Complete overlap (also know as setup, the role of phases 3 and 4 was to containment or inclusion). This find the percentage of overlap between can be satisfied if the number of the folksonomy set and the keywords overlapped keywords equals to the generated by Yahoo TE. In this phase folksonomy set (i.e., F⊂K) or if the and the next one, folksonomy tags, number of overlapped keywords Yahoo TE keywords, and the indexer equals to the Yahoo keyword set keywords are treated as abstract entities, (i.e., K⊂F) or if the number of which do not hold any semantic value. overlapped keywords equals both This assumption will help us see where folksonomy and keyword set (i.e., folksonomies are positioned in the F=K).

Copyright © 2007, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited. 30 Int’l Journal on Semantic Web & Information Systems, 3(1), 13-39, January-March 2007

Figure 4. Histogram of the percentage of overlap (PoL) for 100 Web sites

8 Frequency 6

2 Mean =9.5107� Std. Dev. =4.47369� 0 N =100 2.50 5.00 7.50 10.00 12.50 15.00 17.50 20.00 22.50 PoL

The collected data set (described most frequent percentage of overlap in a previous subsection on data selec- (i.e., mode) was 12.5%. tion) was dispatched to our comparison Figure 4 shows a histogram of the framework to measure the percentage frequency of the results, which graphi- of overlap between folksonomy tags cally summarizes and displays the distri- and Yahoo TE keywords. bution of the percentage of the overlaps After observing the results of 100 using short intervals (2.5 percentages Web sites, we can detect that there is wide). Notice that most of the overlap a partial overlap (F∩K) between folk- values (14 values) fall in the interval sonomies and keywords extracted using between 7.5 and 8.75, while the least Yahoo TE. The results show that the of the overlap values fall at the ends of mean of the overlap was 9.51% with the histogram. The shape of the histo- a standard deviation of 4.47%, which gram forms the beginning of a normal indicates a moderate deviation from curve, thus, we believe that with more the sample mean. Also the results show evaluated Web sites the histogram will both the maximum and the minimum ends up being an approximate normal possible overlap with values equal to curve, which can be used as a tool to 21.82% and 1.96% respectively. This estimate proportion of overlaps with indicates that there is neither complete appropriate margins of errors. overlap nor no overlap at all, and the Finally, the results of this phase

Figure 5. A Venn diagram that shows folksonomy (F), Yahoo TE (K), and the human indexer (I) sets as three distinct circles and highlights the percentage of the overlap between the three sets

9.51%

F k

19.48% I 11.69%

showed us that folksonomy tags cannot the human assignment. This step is be replaced by automatically extracted necessary to see which technique is keywords, even if there was a marginal most closely related to a cataloguing overlap between the two sets. However, (indexation) output. to inspect in more depth the position Therefore, tools from library and of folksonomies in the spectrum from information science were used to index professionally assigned keywords to a sample of 20 Web sites taken from context-based machine extracted key- our data set and to check them against words, phase 4 is carried out to envision folksonomy and Yahoo TE sets. The as- the place of folksonomy tags in this signment of keywords was done using spectrum. the following guidelines:

Phase 4 1. The use of controlled vocabularies The role of phase four is to check of terms for describing the subject the correlation between folksonomy of a Web site such as DMOZ15 (the and human keyword assignment, and open directory project) and Yahoo also between Yahoo TE keywords and directory.

Copyright © 2007, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited. 32 Int’l Journal on Semantic Web & Information Systems, 3(1), 13-39, January-March 2007

2. The source code of each Web site sionally assigned keywords to context- was checked to see if it contains any based machine extracted keywords. keywords provided by the Web site creator. DISCUSSION 3. The position (i.e., in titles) and After completing the four phases of emphasis (such as bold) of words this experiment, a number of observa- in a Web site were considered. tions were made. As a first impression, 4. The indexer also was asked to read phase 1 was used to evaluate the rele- the content of the Web site and vance of the folksonomy tags and Yahoo generate as many keywords as pos- TE keywords to the human conception. sible. Thus, the results of this phase indicate a significant tendency of the folksonomy After the end of this process, the set tags towards depicting what a human of produced keywords for each Web site indexer might think of when describing was compared using our comparison what a Web resource is about compared framework, once with the keywords to Yahoo TE keywords. from the Yahoo TE set and another with Another interesting observation the folksonomy set. This step is essential was found in phase 2, where some to see whether folksonomies produced folksonomy tags fall in the “narrower the same results as if a human indexer term” and “synonym” categories. These was doing the process. categories were less common than the The results show (see Figure 5) that “broader term,” “same,” and “related there is partial overlap between the two term” categories, which implies from sets and the indexer set, but this time our point of view, that this might be due with higher scores. The folksonomy set to the low number of specialized people was more correlated to the indexer set who uses the del.icio.us bookmarking with a mean of 19.48% and a standard service, or it might be due to the varied deviation of 5.64%, while Yahoo TE set backgrounds of the del.icio.us users. scored a mean of 11.69% with a stan- In phase 3 and 4, the folksonomy dard deviation of 7.06%. Furthermore, tags showed a greater tendency to the experiment showed one case where overlap with the professional indexer there is a complete overlap (inclusion) produced keywords than with the Yahoo between the folksonomy set and the TE keywords. Thus, in phase 3, the aver- indexer set. age overlap between the folksonomy set The results of this phase showed us and Yahoo keywords was 9.51%, which that folksonomy tags are more oriented implies that there was only a minor toward the professional indexer key- intersection between the two sets, and words. Therefore, this finding positioned that folksonomy tags cannot be replaced the folksonomy tags nearer to the indexer completely with keywords generated by keywords in the spectrum from profes- machine (in this case Yahoo TE). This

Figure 6. System architecture of the “FolksAnnotation” tool

finding also opens the door for other po- the correlation between professional tential research directions, for instance indexing and context-based machine in the field of language technology and extracted keywords. semantics, which is out of the scope of Finally, it is worth mentioning that this experiment. the results from this experiment have In phase 4, the results showed that not been evaluated against a large cor- the folksonomy set was more corre- pus, especially where this concerns the lated to the indexer set with a mean of sample size used by the indexers. This 19.48%, while Yahoo TE set scored was due to the high effort needed for a mean of 11.69%. This finding also manual indexing. However, to get a fair emphasis our claim about the better judgment we have attempted to choose correlation between folksonomies and varied Web sites topics spanning mul- professional indexing compared to tiple domains as shown in Table 1.

Copyright © 2007, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited. 34 Int’l Journal on Semantic Web & Information Systems, 3(1), 13-39, January-March 2007

The FolksAnnotation to singular) using the Porter stemmer16 Tool—A Case Study then similar tags are grouped (e.g., in- To emphasize the usefulness of the clusion of substrings). Finally, general results obtained from this experiment concept tags in our domain of interest (i.e., the rich semantics of folkson- are eliminated. The process of normal- omies), a working example illustrating ization is done automatically and it is the value of the findings is demon- potentially useful to clean up the noise strated using a prototypical tool called in people’s tags. FolksAnnotation (Al-Khalifa & Davis, The semantic annotation process 2006). This tool uses folksonomy tags is the backbone process that generates to semantically annotate Web resources semantic metadata using pre-defined on- with educational semantics from the del. tologies. The process attempts to match icio.us bookmarking service, guided by folksonomy terms (after normalizing appropriate domain ontologies. them) from the bookmarked resource Figure 6 shows the system architec- against terms in the ontology (which ture of the implemented FolksAnnota- it uses as a controlled vocabulary) and tion tool; the detail of the implementa- only selects those terms that appear in tion of the tool has been previously the ontology. reported in Al-Khalifa et al. (2006); After assigning semantic descrip- however, a brief description of the tool tors to the Web resource, the inference is presented here. engine is responsible for associating The tool consists of two processes: pedagogical semantics (i.e., difficulty (1) tags extraction/normalization level and instructional level) to the an- pipeline and (2) semantic annotation notated Web resource. These two values pipeline. are generated from a set of reasoning The normalization process is re- rules when enough information is avail- sponsible for cleaning and pruning tags. able in the basic semantic descriptors. The process starts by fetching all tags One of the evaluation procedures assigned to a Web resource bookmarked we have carried out on this tool was in the del.icio.us bookmarking service to compare the number of folksonomy then passes these tags to the normaliza- tags attached to our ontologies concepts tion pipeline, which does the following. against the Yahoo TE keywords attach- First, tags are converted to lower case ment for the same Web resource. For the so that string manipulation (e.g., com- purpose of this evaluation, a set of 30 parison) can be applied to them easily. Web resources were randomly selected Secondly, non-English characters are from the del.icio.us bookmarking ser- dropped. This step is to ensure that only vice, and for each Web resource a two English tags are present when doing the sets of keywords (namely, folksonomy semantic annotation process. Thirdly, tags and Yahoo TE keywords) were tags are stemmed (e.g., convert plural prepared to be passed through the se-

Copyright © 2007, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited. Int’l Journal on Semantic Web & Information Systems, 3(1), 13-39, January-March 2007 35 mantic annotation pipeline. The results results of phase four showed that the of this experiment showed that the folksonomy set was more correlated number of attached keywords from the to the human indexer set with a mean folksonomy set is much higher than the of 19.48%, while Yahoo TE set scored Yahoo TE set with mean and standard a mean of 11.69%. deviation of 14.17(8.25) and 4.24(2.47), It is clear from the results of this respectively. The difference between experiment that the folksonomy tags the means was statistically significant agree more closely with the human at p< 0.001. The results demonstrate generated keywords than those auto- that folksonomy tags are more useful matically generated. The results also in generating semantic metadata than showed that the trained indexers pre- context-based keywords. ferred the semantics of folksonomy tags compared to keywords extracted CONCLUSION and future by Yahoo TE. These results were very work encouraging, and illustrated the power In this article, we have described of folksonomies. We have demonstrated four experiments that explore the value that folksonomies have an added new of folksonomies in creating semantic contextual dimension that is not pres- metadata. The first and second experi- ent in automatic keywords extracted ments evaluated the relevance of the by machines. folksonomy tags and Yahoo TE gener- This experiment was a first step ated keywords to the human conception. toward future evaluation techniques The evaluation was performed by two on which we are planning to embark. trained indexers using an evaluation These techniques will measure the scale based on the different relationships semantic value of folksonomies based in a thesaurus as an indication of the on knowledge engineering principles closeness of match. The third and fourth and methods such as formal concept experiments were conducted to find analysis (FCA) and frame-based sys- the percentage of overlap between the tems (Stuckenschmidt & Harmelen, folksonomy tags, keywords generated 2004). In such techniques, concept by Yahoo TE, and the human indexer hierarchies (or “concepts lattices”) are keywords. used to define a given term. By using The results of phases one and two this approach, the intended meaning of show that the two human indexers have a term is addressed instead of finding both agreed on the richer semantics the exact syntactic match. of the folksonomy tags compared to So to conclude, folksonomies Yahoo TE, with p< 0.001. The results are very popular and a potential rich of phase three showed that the average source for metadata. The rational of overlap between the folksonomy set and this work was based on the motivation Yahoo keywords was 9.51%, and the of investigating whether folksonomies

Copyright © 2007, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited. 36 Int’l Journal on Semantic Web & Information Systems, 3(1), 13-39, January-March 2007 could be used to automatically annotate and ranking. In Proceedings of Web resources. The findings of this the 3rd European Semantic Web experiment was used to justify the use Conference (ESWC2006). Budva, of folksonomies in the process of gener- Montenegro: LNCS, Springer. ating semantic metadata for annotating Hulth, A., Karlgren, J., Jonsson, A., Bo- learning resources; see (Al-Khalifa et ström, H., & Asker, L. (2001). Au- al., 2006). tomatic keyword extraction using domain knowledge. In Proceedings REFERENCES of the 2nd International Conference Al-Khalifa, H. S., & Davis, H. C. (2006). on Computational Linguistics and FolksAnnotation: A semantic meta- Intelligent Text Processing (LNCS, data tool for annotating learning Vol. 2004, pp. 472-482). Springer. resources using folksonomies and Hunyadi, L. (2001). Keyword extrac- domain ontologies. In Proceedings tion: Aims and ways today and of the 2nd International Conference tomorrow. Proceedings of the Key- on Innovations in Information Tech- word Project: Unlocking Content nology. IEEE Computer Society, through Computational Linguistics. Dubai, UAE. London: Oxford University Press. Cohen, J. (1960). A coefficient of agree- Kang, B. Y., & Lee, S. J. (2005). Docu- ment of nominal data. Educational ment indexing: A concept-based and Psychological Measurements, approach to term weight estimation. 20(1), 37-46. Information Processing and Man- Golder S., & Huberman B. A. (2006). agement: An International Journal, Usage patterns of collaborative tag- 41(5), 1065-1080. ging systems. Journal of Informa- Kipp, M. E. (2006). Exploring the tion Science, 32(2), 198-208. context of user, creator, and inter- Gruber, T. (2005). Ontology of folk- mediate tagging. IA Summit 2006. sonomy: A mash-up of apples and Vancouver, Canada. oranges. AIS SIGSEMIS Bulletin, Kraft, R., Maghoul, F., Chang, C .C., & 2(3&4). Kumar, K. (2005). Y!Q: Contextual Guy, M., & Tonkin, E. (2006). Folk- search at the point of inspiration. sonomies: Tidying up tags? D-Lib The ACM Conference on Informa- Magazine, 12(1). tion and Knowledge Management Hammond, T., Hannay, T., Lund, B., & (CIKM’05), Bremen, Germany. Scott, J. (2005) Social bookmarking Kroski, E. (2006). The hive mind: tools (I): A general review. D-Lib Folksonomies and user-based tag- Magazine, 11(4). ging. Retrieved January 14, 2006, Hotho, A., Jäschke, R., Schmitz, C., & from http://infotangle.blogsome. Stumme, G. (2006). Information com/category/folksonomies retrieval in folksonomies: Search Landauer, T. K., Foltz, P. W., & Laham,

D. (1998). Introduction to latent and semantics. In Proceedings of the semantic analysis. Discourse Pro- Fourth International Semantic Web cesses, 25, 259-284. Conference (ISWC 2005) (LNCS). Landis, J. R., & Koch, G. G. (1977). The Galway, Ireland measurement of observer agreement Ohmukai, I., Hamasaki, M., & Takeda, for categorical data. Biometrics, H. (2005). A proposal of com- 33(1), 159-174. munity-based folksonomy with Liu, H., Maes, P., & Davenport, G. RDF metadata. The 4th Interna- (2006). Unraveling the taste fabric tional Semantic Web Conference of social networks. International (ISWC2005). Galway, Ireland. Journal on Semantic Web and In- Quintarelli, E. (2005). Folksonomies: formation Systems, 2(1), 42-71. Power to the people. ISKO Italy- Martínez-Fernández, J. L., García-Ser- UniMIB meeting. 2005. Milan, rano, A., Martínez, P., & Villena, Italy. Retrieved February 3, 2007, J. (2004). Automatic keyword ex- from http://www.infospaces.it/ traction for news finder. Adaptive docs/Folksonomies%20-%20Pow Multimedia Retrieval, (LNCS, Vol. er%20to%20the%20People.doc 3094, pp. 99-119), Springer. Sado, W. N., Fontaine, D., & Fontaine, Matsuo, Y., & Ishizuka, M. (2004). Key- P. (2004). A linguistic and statistical word extraction from a single docu- approach for extracting knowledge ment using word co-occurrence sta- from documents. In Proceedings tistical information. International of the 15th International Workshop Journal on Artificial Intelligence on Database and Expert Systems Tools, 13(1), 157-169. Applications (DEXA’04), IEEE Mathes, A. (2004). Folksonomies—Co- Computer Society. operative classification and commu- Shirky, C. (2005). Ontology is over- nication through shared metadata. rated: Categories, links, and tags. Retrieved February 28, 2006, from Retrieved March 27, 2006, from http://www.adammathes.com/aca- http://shirky.com/writings/ontol- demic/computer-mediated-com- ogy_overrated.html munication/folksonomies.html Sieck, S. (2005). Connotea and citeu- Menchen, E. (2005). Feedback, moti- like: Folksonomies emerge within vation, and collectivity in a social scholarly communities. In E. In- bookmarking system. Kairosnews sight (Ed.). Retrieved February 3, Computers and Writing Online 2007 from http://www.connotea. Conference. Retrieved February org/EPS-Connotea.pdf. 3, 2007, from http://kairosnews. Stoll, R. R. (1979). Set theory and logic. org/node/4338 Mineola, NY: Dover Publications. Mika, P. (2005). Ontologies are us: A Stuckenschmidt, H., & Harmelen, F. V. unified model of social networks (2004). Information sharing on the

Copyright © 2007, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited. 38 Int’l Journal on Semantic Web & Information Systems, 3(1), 13-39, January-March 2007

semantic Web. Berlin: Springer. ENDNOTES Vander Wal, T. (2006). Folksonomy 1 http://www.flickr.com definition and Wikipedia. Retrieved 2 http://del.icio.us April 29, 2006, from http://www. 3 http://www.furl.net vanderwal.net/random/category. 4 http://www.furl.net/ php?cat=153 5 http://www.spurl.net/ Versa, C. (2006a). The language of folk- 6 http://www.citeulike.org sonomies: What tags reveal about 7 http://www.connotea.org user classification.NLDB 2006 (pp. 8 Example: http://www.searchen- 58-69). LNCS 3999. gineworld.com/cgi-bin/kwda.cgi Versa, C. (2006b). Concept modeling by 9 http://www.nzdl.org/Kea/ the messes: Folksonomy structure 10 Yahoo API term extractor service and interoperability. ER 2006 (pp. was launched on May 2005 325-338). (LNCS 4215). 11 http://sourceforge.net/projects/ Witten, I., Paynter, G., Frank, E., Gu- jtidy twin, C., & Nevill-Manning, C. 12 http://developer.yahoo.net/search/ (1999). KEA: Practical automatic content/V1/termExtraction.html keyphrase extraction. Proceedings 13 http://del.icio.us/tag/, Data was of ACM DL’99. collected between 24/2 and 27/2 Wikipedia. (2006). Folksonomy. 2006 Retrieved March 26, 2006, from 14 By blindly, we mean that both http://en.wikipedia.org/wiki/Folk- indexers do not know which key- sonomy word list belongs to which set (i.e., folksonomy or Yahoo TE). 15 http://dmoz.org/ 16 http://www.tartarus.org/~martin/ PorterStemmer/index-old.html

Hend S. Al-Khalifa is a PhD candidate in her final year of study in the learning societies lab research group within the School of Electronics and Computer Science (ECS) at the University of Southampton, UK. She received her MSc degree in information systems (2001) from King Saud University, Riyadh, KSA. Her publications are in the fields of web technologies, computers for people with visual impairments and applications of e-learning.

Hugh C. Davis is the head of the learning societies lab within the School of Electron- ics and Computer Science (ECS) at the University of Southampton, UK. He is also the University Director of Education with responsibility for eLearning strategy. He has been involved in hypertext research since the late 1980’s and has interests in the applications of hypertext for learning, open hypertext systems and architectures for adaptation and personalization. He has extensive publications in these fields, and experience of start- ing a spin-off company with a hypertext product. His recent research interests revolve around social hypertext, web and grid service frameworks for e-learning and he has a particular focus on the e-assessment domain. He has led many projects focusing on both the technology and application of e-learning.