Requirement Evaluation for Expert-Based Ranking of Web Interface Environments: the ZEEF.Com Case Study

Requirement evaluation for expert-based ranking of web interface environments: the ZEEF.com case study

Submitted in partial fulfillment of the degree of Master of Science

Yana Ledeneva 10837418

Master Information Sciences Business Information Systems

Faculty of Science University of Amsterdam

June 16, 2015

Thesis Supervisor: Second Reader: Dr. Frank Nack Dr. Peter Weijland Table of Contents Abstract ...... 2 1. Introduction ...... 2 2. Related work and definitions ...... 3 2.1 Overview of content curation ...... 3 2.2 Ranking of web-pages ...... 3 2.2.1 Definition of ranking ...... 3 2.2.2 Search engines ranking ...... 4 2.2.3 Human-based ranking ...... 4 2.2.4 Factors for evaluating web pages ...... 4 2.3 Concepts of trust and agreement ...... 6 2.4 Conclusion and research gap ...... 7 3. Conceptual Framework and research question ...... 7 4. The ZEEF use case ...... 9 4.1 Why ZEEF ...... 10 4.2 Survey for the curators ...... 10 4.2.1 Design ...... 10 4.2.2 Participants ...... 11 4.3 Survey for the users ...... 11 4.3.1 Design ...... 11 4.3.2 Participants ...... 13 4.4. Results of the surveys ...... 14 4.4.1 The process of creating a curator page ...... 14 4.4.2 Commission and fraud ...... 15 4.4.3 Ranking factors ...... 16 4.4.4 Visitors’ general perception ...... 16 4.4.5 Conclusions ...... 20 5. Interface experiment ...... 20 5.1 Design ...... 20 5.2 Results ...... 21 5.3 Discussion ...... 22 6. Overall discussion ...... 22 6.1. General discussion and limitations ...... 22 6.2. Guidelines for the curators ...... 24 7. Conclusions and future work ...... 26 Appendix 1. Preliminary list of questions for surveys ...... 29 Appendix 2. Table with ranking factors ...... 34

Requirement evaluation for expert-based ranking of web interface environments: the ZEEF.com case study

Yana Ledeneva 10837418

Abstract This research is aimed to analyze the process of creation of the expert-based rankings and their perception by the users of a human-based search environment. The first part of the research is focused on the investigation of the main criteria that influence the ranking of websites in human-based search engines, as well as the main factors that influence trust, agreement and general perception of these rankings by the visitors. The second part of the research covers the analysis of the main use case of this study - a human-based search engine ZEEF.com, in which the curators create their pages and establish the rankings of web sites for one or several topics. The empirical part of this study was conducted with the help of two surveys. The first survey addressed how curators of ZEEF pages generate their pages to check the reasoning behind, the effort invested, and the time needed to establish a ranking. The second survey addressed existing and potential visitors of ZEEF pages to check how, in their opinion, the rankings were established, and whether the results are satisfactory and trustworthy. Based on this evaluation, the list of the main ranking factors that were taken into account by both parties was developed. Finally, an interface test was performed to check which aspects of web pages influence the perception of the quality of the page by the visitors. As a result, recommendations for the curators of ZEEF pages regarding the process of creation of ZEEF pages were designed.

1. Introduction In the era of information overload and significant use of search engines, the problem of filtering the world’s information is getting more and more important. However, the algorithmic search that is implemented in the main search engines, social networks and other websites sometimes does not provide a 100% retrieval quality. Taking into account that people usually tend to trust other people more than machines, especially if other people are experts in their field, it is possible to come up with a better alternative to algorithmic search: a human-based search environment. Nowadays such an environment is available in form of a content curation or human-based search engine. One example of such a search engine is ZEEF.com. However, the main algorithm of ZEEF - a human-based ranking of the links to different websites - has its risks. For instance, the curator may be too subjective or may use ranking factors that do not align with those applied by visitors of the page, or the visitor may distrust the curator, his page and/or the platform in general. Therefore, in order to analyze the process of ranking websites in a human-based search environment in general, and to reduce the business risks of ZEEF company in particular, the main factors and parameters that are taken into account by the curators for making the lists of links, and their methods of making rankings need to be analyzed.

The aim of this research is to understand what influences the perception of the quality of a human- based search environment by the visitors, and to check the difference between the ranking criteria curators take into account, and the criteria visitors think should be taken. Significant difference between these criteria may lead to possible misunderstandings, which in turn may lead to decrease of quality of the content from visitors’ point of view.

2. Related work and definitions

2.1 Overview of content curation Due to information overload these days it becomes much more challenging to find the information one is interested in (Stephen Dale, 200). One of the solutions to this problem is content curation, which becomes more popular as a way to filter a massive torrent of information (Nancy K. Herther, 2012a). Content curation is “the process of sifting through information on the web, organizing, filtering and making sense of it and sharing the best and most relevant content on a specific issue within your network”. (Nancy K. Herther, 2012b, p. 41)

A content curator is a person who continually finds, groups, organizes, and shares the best and most relevant content on a specific issue online (Nancy K. Herther, 30). It is worth mentioning that the key component of this definition is the word “continually”, because data in the internet updates incessantly, and it is important to be up-to-date in order to provide the most relevant, right and useful information.

Automatic algorithm-based approaches of searching for information (such as Google1 or Bing2 search) lack some of very important features of content curation: for example, the web-sites can be easily optimized to appear on top of Google’s list, which means that the searcher will find not necessarily the best content on a topic (instead, they will find the web sites who invested the largest amount of resources into Search Engine Optimization). On the other hand, a content curator organizes and categorizes information for easier search and manipulation, which is not done by automatic search engines yet.

However, most content curation tools today, such as scoop.it3, pinterest.com4 or curata.com5 lack clear internal organization, meaning that users are still left to browse through long lists of content within a theme (Nancy K. Herther, 2012a). That means that it is more convenient and useful to combine content aggregation (which means that the content is discovered from different sources, classified into a pre-established order, and low-quality sources are filtered out) and content curation (which focuses on human involvement in managing content, adding editorial point of view and insights on why content is important) (Heidi Cohen, 2014). As a result, the curators not only collect the most relevant and high-quality information, but also provide their personal insight and comments on why this particular piece of information is important. One of the examples of such insight is ranking of links to websites.

2.2 Ranking of web-pages

2.2.1 Definition of ranking Ranking is reorganizing search results so that the most relevant information appears higher in the list. Put it simply, ranking can be defined as “reordering of a result set (i.e. a search that has already been carried out) based on certain criteria” (Alain Materne, p. 5). The criteria (ranking factors) taken into account for ranking are critical: if a person makes two queries where each features different criteria (e.g. price of products

1 google.com 2 bing.com 3 www.scoop.it 4 www.pinterest.com 5 www.curata.com

3 in the shop and distance of the shop from the customer), he or she will receive two completely different rankings.

2.2.2 Search engines ranking Most of search engines work with the query-dependent ranking factors, such as frequency of occurrences of the search items, position of the query terms in the document, relevance between a document and a query, etc. However, there also are the query-independent criteria, such as PageRank – “a measure of popularity and importance of web page by counting number of links to this page” (Altman, p. 2), number of clicks on the link, size of the website, etc. (Lewandowski, 2005). These criteria guarantee accurate, relevant, and useful results for a user searching for information on a particular query, which lead to high level of satisfaction among search engine users: in 2014, customer satisfaction with portals and search engines was 80%, 5.3% higher than in 2013 (ASCI, p. 3). However, even the most current advanced algorithms have their disadvantages. The main drawback is that search engines cannot define link manipulation and intrusive or spam-like promotion of the web site (Lewandowski, 2005). For instance, if the user looks for Java tutorials, as the first link he or she may find the web site, which has spent some money on deliberate manipulation of search engine indexes and affecting the visibility of the web site in a search engine’s results. However, this tutorial may be worse by the quality of the lessons or the content. The best tutorials may be much lower in the ranking because they pay attention on the quality of the content, not Search Engine Optimization (SEO). One of the solutions to this problem is human-based (or expert-based) ranking.

2.2.3 Human-based ranking “Human-based models do not usually work as an alternative for all other models, but rather as a stopgap to fulfill a need while the other models are not available to present a high-quality solution” (Engler, 17). Particularly, the human-based method of ranking websites is used as additional method to filter irrelevant search results and facilitate a deeper analysis of website content. However, with this method, it is impossible to analyze a big amount of web sites (Vaughan, 2004), and experts, who work on the ranking, may be too subjective and rely solely on personal opinion or personal benefits, which means that expert-based ranking may have biases. Finally, it is difficult to identify factors used by an expert for their individual ranking process: if the expert takes into account ranking factors that do not match with the user’s information need parameters then the ranking may become irrelevant or, in the worst case erroneous.

2.2.4 Factors for evaluating web pages Human-based ranking of websites involves the use of particular factors (criteria) that are taken into account by the curators for evaluation of web pages. An analysis of the literature (Bailey, Pearson, 1983); (Fogg, 2002), (Dholakia, 1998); (Hsuan Yu Hsu, Hung-Tai Tsou, 2011); (Evaluating Web Pages: Techniques to Apply & Questions to Ask, 2012); (Natalie Clewley Sherry, Chen Xiaohui Liu, 2009); (More guidance on building high-quality sites, 2011); (Gagan Mehra, 2012); (Hsiu-Fen Lin, 2007) resulted in the following list of factors that influence motivation and enjoyment of web experience among the visitors (see table 1). These factors are proven to be the most relevant for establishing the validity (the quality of being logically or factually sound; soundness or cogency), authorship, timeliness, integrity (the state of being whole and undivided) and presentation of the information found on the Internet.

Table 1. Factors used for evaluation of web pages

№ Factor Definition Establish:

1 Accuracy of the content The correctness of the output information Validity

2 Completeness of the content The comprehensiveness of the output information content. Integrity

3 Information credibility Perceived quality of the information by consumers Validity

4 Information about the author Information about the author, his contacts and background, plus any Authorship reference needed for future correspondence

5 Links to other sources Little footnote numbers or links that might refer to documentation, Authorship (references) take the time to explore them.

6 Timeliness of the content The availability of the information at a time suitable for its use. Timeliness

7 Relevancy of the content The degree of congruence between what the user wants or requires and Validity what is provided by the information products and services.

8 Confidence in system/ The user's feelings of assurance or certainty about the website. Validity personal trust in web site

9 Understanding of a system The degree of comprehension of a user Validity

10 Size of the web site Amount of separate pages/articles/content in general Integrity

11 Position in ranking of the Including Pagerank and other query-related quantitative factors Validity main search engines

12 Visual layout The material design of the layout and display of the output contents. Presentation

13 Performance of the web site Availability, response time, processing speed, memory consumption Presentation and utilization of computing resources, throughput, etc.

14 Money received as Amount of money received as commission from affiliate links, or Validity commission directly from the site-owners

15 Recommendations by Web site may be recommended to the user by other trusted web sites, Validity external parties people, friends and relatives.

16 Adequate URL The unique address of any Web page or a document, including a Presentation domain extention

17 Amount of promotional Amount of advertisements and pop-ups, the tone of writing (should be Presentation material clear, direct, sincere and free of “sales-speak”).

18 Personal experience Previous positive experience using this web site Validity

19 Prices of the products offered (for online shops only). Validity

20 Uniqueness of the products (for online shops only). Whether the product is unique (original) or Validity offered manufactured by the company.

21 Quality of the products (for online shops only). Validity offered

2.3 Concepts of trust and agreement One of the main reasons why people go online and visit particular websites is search for information (Thomas Young, 2009), and entertainment (Why People Go Online, 2009). In other words, the main interest of the visitors going to a particular website is to find the needed information about a particular topic or to spend time entertaining themselves (Yong-Soon Kang, Yong Jin Kim, 2006), (Rik Crutzen et al, 2014). Therefore, the main goal of search engines, including human-based search environments, is to satisfy the visitors’ need of gaining the most relevant, useful and high-quality information. Informativeness of web pages and quality of the content are the most important aspects that have a positive effect on the attitude of users towards the web site and positively influences traffic. In order to increase traffic and make people start using the web site, the content of the page (in case of search engines - the rankings of the retrieved links) should be relevant and useful for the visitors.

The next step is to make people continue using this web site in the future. Engagement process of the visitors may be initiated by interest or information need (O’Brien, Toms, 2008); however, it is also important to take into account such aspects as trust of the users and their agreement with the ranking of the curator.

According to (Mirjam Seckler, p. 40), trust is a multidimensional construct, consisting of three different facets: - benevolence (user’s belief that a website is concerned with the present and future interests, desires and needs of its users and gives useful advice and recommendations); - honesty (there are no false statements and the information on the site is sincere and honest); - competence (the website has the resources [whether technical, financial, or human] and capabilities needed for the successful completion of the transaction and the continuance of the relationship).

Trust of the visitors of web pages may be influenced by multiple factors, such as design of the page, personal information about the creator of the page and his/her domain knowledge, visitors’ background and expertise, statistics of the page (including the number of views and/or clicks), feedback from other people, quality of the content and so on (Ye Diana Wang, Henry H. Emurian, 2005). In human-based curation platforms, however, these factors were not tested yet, because in these environments, the pages are personalized, and the curators play more significant role in establishing trust. On these platforms, if the visitors do not trust a particular curator, they can leave the page of this curator and switch to the page of the same topic but of another curator. However, if the page is curated by only one person (due to limited amount of the curators), the visitor has no alternative page about this topic to visit. In this case, the visitor stops using the platform at all.

Besides that, the visitors may lose trust in content curation platform in general if they see the next curators to be trustless as well. They may distrust the platform itself, because they may think that the platform does not prove the expertise and/or experience of the curators in the field they create their pages about. Moreover, this may happen if the platform does not provide the visitors with the guarantee that only experts (professionals) can create the page on the platform. Therefore, the design of the website and personal information about the curator (including profile picture, description and links to the profiles in social networks) play important roles in establishing trust of the visitor. Finally, the fact that content curator is usually a stranger to a visitor negatively influences trust level in curation platforms in general, because people tend to distrust strangers more than their friends and relatives (Mirjam Seckler, 2014). In case the visitors distrust the content curation platform, they will leave the website and never return back, thus decreasing traffic of the site and its revenue.

Trust represents one side of the engagement process of the visitors, while agreement with the content represents the other side. The concept of trust is correlated with the information about the curator, design of the page and overall perception of the curation platform, while the concept of agreement relates to the content itself and level of satisfaction in terms of finding the most relevant and useful information in short time. According to Oxford dictionary, agreement is “the state of sharing the same opinion or feeling” (Agreement, Oxford Dictionary of English). For the curators of the pages it is extremely important to keep the level of disagreement as low as possible, and in order to reach this it is important to see how the content is made. Talking about human-based search environments - it is necessary to understand, which factors are taken into account by the creators while making up the rankings of websites, and whether visitors of the page perceive these rankings in a similar way that creators of the pages do.

The failure of one or both sides of the engagement process in human-based search environments leads to decrease in traffic and, consequently, revenue stream of the web site. This assumption is taken because the affiliate marketing program, which is implemented in search environments, implies that revenue of the company depends on the amount of people clicking on the links and purchasing the products via these links.

2.4 Conclusion and research gap The problem addressed in this research covers two main areas: human-based methods for creating rankings and evaluation of web sites. In this research, these topics will be combined. The literature review revealed that there are very few articles and researches about human-based ranking of web sites yet, as well as about factors and methods used for such rankings. Moreover, this paper will have practical value, because according to the results of my research I will be able to come up with some practical recommendations for the curators of human-based search environments, with the help of which they will increase the quality of their rankings, lower level of disagreement among the visitors and increase traffic (and, consequently, their revenue). Though this investigation only one human-based search environment, namely ZEFF, will be tested, however I assume that this research can be useful for other content curation platforms as well.

3. Conceptual Framework and research question In this research, the issue of different perceptions of human-based rankings by the curators and the visitors will be addressed. The main aspect which will be analyzed is the factors that are taken into account both by the users of the ranking (the visitors) and by the creators of these rankings (curators). The main and only case on which my research is based is the environment provided by ZEEF. In picture 1, the conceptual framework of the research is described. Starting from the right side: the revenue of the company and of the curators who create the pages in ZEEF depends on the traffic, which is directly related to the visitors’ perception of the quality of the content represented on ZEEF pages and their intention to use these pages in the future. The perception of the quality is dependent on many different aspects, but in this research, only two will be analyzed: trust (which mostly concerns the design of human-based search environments and the curators themselves) and agreement with the ranking. Agreement of the visitor with the ranking may depend on difference in the factors that establish the basis for this ranking - from the point of view of both the curator and of the visitor. The arrows in the scheme represent the connections of the topic, red and green arrows - the hypotheses that were checked in this paper.

Picture 1. Conceptual framework of the research

With this conceptual framework, I intend to answer the following questions:

Main research question: “What are the trust and agreement factors that influence both curators and users perception of human-curated recommender environments?”

In order to answer this question, the following sub-questions should be answered (see table 2):

Table 2. Subquestions of the research Curators Visitors

C1. What factors do they consider when making V1. What factors they believe were taken? rankings?

C2. Why do they create the lists? V2. Do they agree with the ranking? C3. How do they create the lists? V3. Do they trust the curator? C4. Is there a risk of the fraud?

C5. Do they see themselves as experts? V4. Do they see the curators as experts?

C6. Commission: do they want to receive it and how? V5. How do they think the curators receive the commission? How do they feel about it?

C7. Do they care about the ranking of the links? V6. Do they understand the ranking of the links?

V7. Will they use the lists in the future?

Based on the literature the following hypotheses will be tested in this research: H1. There is a positive correlation between the level of disagreement of the visitor and level of discrepancy between curator’s and visitor’s ranking factors (C1-V1 vs V2). H2. The visitor, who disagrees with the ranking, will not use it for the future search. (V2 vs V7). H3. The visitor, who does not trust the curator, will not use his page for the future search (V3 vs V7). H4. Rankings of the curators who are prone to get bribed or do not make enough effort in making rankings, have higher level of disagreement among the visitors (C2, C3, C4, C5 vs V2) H5. Rankings of the curators who are prone to get bribed or do not make enough effort in making rankings, have lower level of trust among the visitors (C2, C3, C4, C5 vs V3)

Taking into account the assumption that background, experience and knowledge of the visitors of human-based curated environments could potentially influence their perception of the page (see section 2.2.4). Therefore, it was decided to test additional hypotheses that were not covered by the sub-questions and framework: H6. Experienced and inexperienced users take into account different ranking factors H7. There is a significant difference between the perception of the page by experienced and inexperienced users

4. The ZEEF use case The main use case of this research is ZEEF - a human search engine based on a directory with results that are selected and ranked by the curators – experts in a particular topic, on which they create their pages. This is a curated search and advertising directory with an advertising model that is monetized via affiliate marketing.

The methodology of the research includes qualitative and quantitative information analysis. Qualitative method includes literature review (which was mostly used for theoretical background and for making up the list of the factors that are generally used for evaluation of websites) and interviews with employees of ZEEF for a deeper understanding of the business processes of the company.

Quantitative method includes two surveys. The first one was addressed to the curators of active ZEEF pages, and it helped to investigate the process of making pages and developing rankings. The second survey was addressed to the potential and current visitors of ZEEF pages. The respondents saw the real ZEEF pages (the ones of the curators who took the first survey) and were asked to analyze how the rankings on these pages, in their opinion, have been made. The respondents could click on the links to check out the content of web pages. For this survey, the same Likert scale for the same factors was used as the one in the experts’ survey. In addition, the visitors were asked general questions about their perception of a particular ZEEF page (see Appendix 1).

For making the research as representative and reliable as possible, the amount of the respondents was crucial. That is why it was decided to make surveys and to disseminate them through the Internet. This method provided an easy access to the people who were willing to participate in the research. In addition, survey results are much easier to turn into quantitative data and to analyze using statistical analytics software such as Excel, Spss and Stata.

4.1 Why ZEEF Aside from the opportunity to have an internship in this company, the reason for choosing ZEEF as the main use case of this research is that it perfectly combines curated content and human-based ranking of the web sites.

ZEEF tries to solve the problem of information overload and filtering: the company adds the human filter to existing search algorithms by crowdsourcing the expertise of trusted curators and builds a new, better performing online advertising model on the back of this. Each ZEEF page has a clear, well filtered, SEO and spam free ranking of links to web sites, which have the best content on a particular topic curated by real people.

ZEEF is made for people, and if they do not use it – it becomes abandoned and unprofitable. Therefore, the main risks of the company are related to the people. These are low amount of the curators who generate high-quality content and guarantee topic diversity, and low amount of visitors who use the pages and generate the traffic and revenue stream for the company. In order to reduce these risks it is critically important to analyze how the curators make their lists, whether visitors are interested in curated content and expert-based rankings, and whether they agree with the rankings made by the curators. In case the visitors do not trust or do not agree with the ranking, they will more likely stop using the page, because it brings no value for them. In addition, it is important to increase the quality of the content the curators make (particularly, the quality of the rankings), because without good and valuable content ZEEF may struggle to get more visitors. Therefore, these risks should be analyzed from two sides: from the point of view both of the curators and of the visitors.

4.2 Survey for the curators

4.2.1 Design The first survey consists of 13-16 questions (some questions appeared depending on the answers people chose). First, the curators were asked to type down the URL of their page (for the survey they had to consider only one page - in case they have multiple pages on ZEEF platform) and to choose the topic of their page. After that, they were asked to answer other questions, which covered the following topics (see table 3).

Table 3. The structure of the survey for the curators Groups of Examples of questions Type of question Why it is included in the questions survey (research subquestions)

Creation of What influenced you to make the page? Mult. choice To understand the process the page of creating the page by the curators in order to see How did you choose the topic of your page? Mult. choice possible flaws or mistakes that may influence the visitors’ perception of the Do you consider yourself a professional in this topic and Open question quality of the page (C2, why? C3, C5)

Where do you take the links for the ranking lists you Mult. choice make?

How much time does it take you to come up with the very Mult. choice best link?

How often do you check and/or revise the rankings or Mult. choice update the lists after finishing them?

Ranking Is the the first link of each list (at the top of the list) the Mult. choice To understand whether the best among all the links in the list? lists on the pages are ranked or not (C7)

Ranking How important are the following factors for evaluation of Likert scale To see which factors the factors the websites and of the links you add and rank on your curators take into account page? when creating the ranked lists (C1)

Bribery Would you (theoretically) put the link on the 1st (2nd, 3rd) Mult. choice To evaluate the possibility place for money secretly received from a third party? of the curators to get bribed and to change their If yes - How much money (minimum) would you take for Mult. choice lists for money (C4) changing your ranking?

Official Would you like to be paid by ZEEF (officially) for Mult. choice To check the curators’ commission maintaining your page? attitude towards official from ZEEF commission from ZEEF and how it may influence If yes - What would you like to receive commission for? Mult. choice the quality of the pages (C6)

The survey consists of mostly closed questions, so it is easier for the respondents to take it. However, some questions that need a deeper analysis of the respondents’ opinion, are open (for example, in the question “Do you consider yourself a professional in this topic and why?” the respondents should have provided the reasons why they thought positively or negatively).

4.2.2 Participants The first survey was disseminated via e-mail to all the curators who have their own ZEEF pages published. The overall sample consisted of 1017 active curators who were registered in ZEEF and created the pages there. All the curators received an e-mail with the link to the survey, and in 1 week, there were reminders for those who did not take the survey yet. Thirty e-mails were not delivered due to technical reasons. All in all, 78 ZEEF curators took part in the survey, which means that response rate for such e-mails was 7.6%. Such a low response rate can be explained by the fact that surveys in general have low response rate (Robson K, 2011), and the e-mails from the company to the curators can be perceived as SPAM, so majority of the respondents could simply have this message in a spam folder.

From the whole population of possible respondents one group was taken - this group (strata) contains people who have created and published pages on the ZEEF platform. The e-mail with the link to the survey was sent to all the curators; however, it was impossible to predict who exactly would take the survey. Thus, the sampling technique used for the first survey is stratified random sampling.

4.3 Survey for the users

4.3.1 Design The second survey was designed for both current and potential visitors of ZEEF pages. This survey helped to evaluate how the quality of the pages and curated content in general perceived by the visitors.

In the beginning of the survey, the respondents were asked to choose the topic they are interested in the most, and were asked to visit the page and click on the links to check out the websites they lead to. Six pages from the first survey were chosen for the survey for the visitors based on the overall quality and completeness of the page (representing six most popular topics from the 1st part of the research) (see table 4).

Table 4. The characteristics of the pages chosen for the second survey Time spent on Commissi Page Ranked? finding the best Updates Bribery on link Business: https://energy- No Several days Every month Yes, 10 Yes startups.zeef.com eur Travelling: Yes Less than 1 minute Never No No https://amsterdam.zeef.com/willem.ij ssel.de.schepper Technology: https://office- No Several days Every month No Yes 365.zeef.com Food: https://omega3.zeef.com No Several hours Never Yes - Yes 100 eur Education: https://teacher- No Several days Every week No No playground.zeef.com Shopping: Yes Several days Every month No Yes https://shopping.zeef.com/robert.w.h all

In order to compare different aspects of the pages and their curators and their influence on the perception of the page by the visitors, it was decided to choose the pages that differ from each other by such characteristics as: 1. Type of ranking used (whether the links are ranked based on their quality or time on which they were added), 2. Time spent on making up the ranking 3. Frequency or reviews/updates, 4. Attitude towards the official commission from ZEEF and bribery.

After viewing the page, the respondents had to answer general questions about quality of the page, ranking and ranking criteria, curators of the pages and intention to use such curated pages in the future. The structure of the survey is shown in the table 5.

Table 5. The structure of the survey for the visitors Groups of Type of Why it is included in the Examples of questions questions question survey Evaluation How much do you agree with the rankings on Likert scale To understand the visitors’ of the page this page? perception of the quality of How often do you think the curator Mult. choice the page and their level of revises/updates the list of the links after finishing agreement with the content the page? (V2) Ranking In your opinion, how relevant are the following Likert scale To see which factors the factors factors for ranking the links to the web sites on visitors think should be taken the page? into account when creating the ranked lists to compare them with the curators’

opinion. (V1)

Ranking Do you think the order of the links depends on Mult. choice To understand whether the the date when they were added or on the quality visitors see the lists on the of web sites? pages ranked or not (V5) Trust Do you think the curator is an expert in this topic Open question To check the level of the and why? visitors’ trust in the curator Do you trust the curator’s choice? Mult. choice and the content of the page (V2) Bribery Continue the sentence: If I find out that the Mult. choice To check the visitors’ reaction curator was secretly paid (bribed) by the third to the curators being bribed party for changing the lists of the links... (V2) Official Imagine this curator receives commission from Mult. choice To check the curators’ commissio ZEEF. How do you think the curator may be attitude towards official n from paid? commission from ZEEF and ZEEF How do you feel about the fact that some of the Mult. choice how it may influence the curators may receive commission from ZEEF? quality of the pages (V4) Intention Would you (theoretically) use similar ZEEF lists Likert scale To check the possibility that to use in for your search in the future? the visitor will use ZEEF in the future the future (V6)

The majority of the questions were of multiple-choice type; however, some of them required open answer for a deeper insight and getting more feedback from the respondent. In addition, in some of the questions Likert scale was used for a better evaluation of the respondent’s attitude towards a problem. For example, in questions “How much do you agree with the rankings on this page?” and “Would you (theoretically) use similar ZEEF lists for your search in the future?” 10-point Likert scale was used for as accurate as possible estimation of the level of agreement of the visitors, and possibility that they will use curated content pages in the future.

4.3.2 Participants The target audience of ZEEF is everyone who uses the Internet for information search. As the respondents of the second survey are already existing customers as well as potential users of this platform, it was decided to accept all incoming replies to the survey from all the respondents who could potentially use ZEEF for information search. However, during the analysis of the data, the respondents were clustered into basic demographic groups based on the age, gender and education. The main group were friends, relatives, colleagues of the author, as well as students and other people which were easy to reach via social networks (e.g. facebook.com or vk.com) and other related web sites and blogs (e.g. zeef.org). The sampling of the respondents was made by simple random sampling technique. In addition, the respondents were asked to promote the survey to their friends, which is related to the snowball technique (Leo A. Goodman, 1961).

Finally, in order to achieve a representative sample, it was decided to divide the population into specific groups based on age, country of residence and education level. In addition, the goal was to reach the same amount of respondents for each of six topics of ZEEF pages. In order to reach this the quota technique (C. A. Moser, 1952) was used. All in all, 174 people started taking the survey, however only 80 fully answered all the questions. The average age of the respondents is 26 years, while median is 23, and mode is 22. Minimum age is 18, maximum - 48. 46.3% of the respondents are male, 53.7% - female.

4.4. Results of the surveys The most popular topics of the pages in the first survey were “Technology, gadgets, games and internet” (24 respondents), “Business, management and companies” (19 respondents), “Travelling and culture” (14), “Food and drinks” (4), “Education” (4) and “Shopping” (3). These topics were subsequently chosen for the second survey for the visitors in order to check the difference in ranking factors and other aspects among the topics. In the second survey the most popular topic among the respondents was Travelling (20 responses), followed by Technology (14) and Food (13). Business, Education and Shopping got 11 responses each.

4.4.1 The process of creating a curator page The most frequent reasons for creating the page by the curators on ZEEF are “the opportunity to help people find information” (was chosen 27 times), “to build online authority and to show expertise in the topic” (19 choices) and “to bookmark and organize personal favorite links” (18 choices). The particular topic of the page was chosen because the curators believe that they have much expertise and experience in it, and because they are passionate about this topic. These results support the main idea and concept of ZEEF.com:

“The content curators on ZEEF are passionate people with knowledge on a certain topic. They search the Internet and find the best content on their topic. Then the curators organize, filter and rank the links into a convenient list. Simple as that.”6

The majority of the curators take links for their lists from their personal bookmarks (59.5%), or just know them by heart (45.9%). In addition, 39.2% of the curators use search systems like Google or Bing for searching for links for rankings, which means that the links are filtered both by algorithmic search engines and by the curators. This may result in a better quality of websites, because the curators choose the best links from Google, Bing, or other search engines, and these links are the best based on quantitative factors such as PageRank, frequency of keywords (queries) in the content and their position in the document, number of clicks on the link, size of the website, etc. In addition, the curators add the human factor to their webpage ranking. They evaluate websites from a qualitative point of view: starting from the design of the web site and resulting user experience, and finishing with the completeness and correctness of the content and credibility of the source - the factors that are not checked by algorithmic methods.

It takes less than one day to establish the ranking and to come up with the best link in the list for 80% of the curators, and 31.1% of the respondents know beforehand which link will be the best (need less than 1 minute to realize which link will be the first in the ranking).

As for revisions - the sample was split into three parts: 30% of the curators never revise their rankings, 40% updates them every month, and 29% updates them every week. The visitors, on the other hand, assume that the lists are updated every week (40% of the respondents) or every month (35%). Analysis of this variable for different topics revealed two interesting observations: first, 73% of the visitors who were analyzing the Shopping page stated that, in their opinion, the rankings on the page are updated every day. However, the curator of this page stated that he updates the ranking once per month. The same happened to Travelling page: 60% of the visitors thought that the page is updated at least once per week; however, the curator stated that he never updates it.

Finally, 54% of the curators do not rank the links in their lists (so the order of the links does not depend on the quality of websites they lead to). Instead, they put the links in the list based on the date (time)

6 Retrieved from: https://zeef.com/about, 15.06.2015

14 they were added, alphabetical order or popularity (amount of clicks of the visitors). On the other hand, 61% of the visitors believe that the links are ranked according to the quality of websites they lead to.

In a sample of the pages that were shown to the visitors, only 2 out of 6 have ranked lists of the links - these are Travelling and Shopping pages. However, the visitors stated that the links are ranked not only for Travelling and Shopping pages, but for others as well. Particularly, the only page the visitors saw as unranked was Technology page (57% of the respondents stated that the lists are not ranked). Thus, the majority of ZEEF visitors see almost any list to be ranked based on the quality of the website, company or a product the link in this list leads to. The curators, on the other hand, tend to create the lists without comparing the links with each other.

4.4.2 Commission and fraud The question about the risk of fraud among the curators was stated as “Would you (theoretically) put the link on the 1st (2nd, 3rd) place for money secretly received from a third party?” 31% of the curators answered this question positively, and after that were asked for a minimum sum of money they would agree to take for changing their ranking - this was a multiple-choice question with the answers 1, 10, 100 and 1000 EUR. Only two respondents chose the minimum amount of 1 euro, and only two chose maximum 1000, while both mode and median for this variable are 100 euros. The average sum is 140.5 euros.

The question about possibility of fraud has a correlating question in the survey for visitors, who were asked whether they would change their attitude towards ZEEF if they knew that the curator was bribed. 54% of the respondents would be disappointed that the ranking on the page is biased, but they would still use it for their search. 17% of the visitors would not pay attention on it, and 29% would never use the page of a bribed curator again. Thus, there is 9% possibility for ZEEF to lose its users forever (among all the curators 31% may be bribed, and because of them 29% of the visitors would never use the page anymore. If we multiply these figures, we will find the possibility of losing the visitors because of the bribery of the curators: 31%*29% = 9%).

Besides the secret, “illegal” payments, we have asked whether the curators would like to be paid officially by ZEEF for maintaining their pages and getting visitors. 53.8% of the respondents answered “yes” to this question. Among those who answered his question positively, 70.7% would agree to have a special sign on their page showing that they are paid by ZEEF. The most popular reasons of being paid were amount of clicks, amount of views and Click-through Rate (CTR) - clicks/views ratio. The same variants were chosen by the visitors; however, the visitors also paid attention on the amount of purchases made via the links provided by the curators. It is interesting that the visitor’s perception is much closer to the reality, than the curators’, because in real life paid publishers in ZEEF get commission according to the purchases made by the visitors through the affiliate links.

From the visitors’ point of view, official commission encourages curators to make better rankings (57.5% of the respondents chose this answer), and only 5% think that it negatively affects the quality of the ranking.

Finally, the visitors were asked if the information about how the curator is paid would influence their behavior (e.g. whether they would click on more links or share the page in social networks). The majority (70%) of the respondents stated that this information would not influence their behavior - they would click only the most interesting links and would not share the page if they do not like it. However, eight respondents

15 stated that they would try to help the curator earn money by clicking additional links, “liking” or sharing the page only if they have positive personal attitude to the curator and/or really love the content on the page.

4.4.3 Ranking factors The most important outcome of this survey is the list of the factors the curators take into account while making the ranking lists, and which factors the visitors think have been taken. The respondents of both surveys were asked to evaluate each factor using Likert scale from 1 (not important at all) to 5 (very important). The list of the ranked factors is presented in the table 6 in the Appendix 2.

The factors presented in the Table 6 (see appendix 2) are general for all the topics. From the table it is clear that there are some differences between the visitors’ and the curators’ perceptions. The last column represents the difference between rankings of the curators’ and the visitor’s factors. The average absolute difference between the rankings of the factors is 4.63, standard deviation is 3.89, however we can see 4 factors, where the difference in rankings is more than the sum of the average and standard deviation (4.63+3.89=8.5). These factors have the largest difference, which means the biggest discrepancy between both parties’ opinions. For example, for the visitors it is very important how much advertising the website has, but for the curators this factor is the second non-important. The curators, on the other hand, take into account more quality of the products, their uniqueness and personal experience/knowledge that contribute to the ranking of the website, whereas for the visitors these factors are the least important.

The ranking of the curators’ factors does not considerably differ among the topics of the pages. The average correlation coefficient is 0.86, and the correlation between business, technology and travel pages and general data it is from 0.96 to 0.97. However, there are some features in Education and Food pages, but they can be explained by the fact that there are few respondents with the pages on these topics (4 respondents for each topic). Particularly, for Food pages the recommendations from the third parties and performance of the web site are more important than in general (the grade is 4.25 - third place in the ranking, whereas for all topics these factors are only 11th and 12th respectively). For Education pages ease of understanding (usability), quality of the products and their uniqueness are vital (the average, mean and median for these factors are 5 out of 5), while in ranking for all the topics these factors are 10th, 8th and 9th respectively.

The same analysis for the visitors’ estimations revealed that the correlation coefficient among the topics is 0.8, can be considered as high. However, there is one exception: for visitors of Education page such factors as amount of advertisements on a website and up-to-datedness are the most important (1th and 2th respectively, while in general ranking they are 5th and 4th).

Standard deviation of the means of the rankings of each factor among the curators is almost as twice as high as for visitors. This shows that the visitors, in general, are more consistent with each other, while the curators are more critical in evaluation of each factor. Interesting that both for the curators and for the visitors standard deviation is lower for the factors that are ranked higher. In other words, the respondents have a higher level of agreement on which factors are the most important.

4.4.4 Visitors’ general perception In general, the average level of agreement of the visitors is 7.14 out of 10 (minimum value is 3, maximum = 10, median - 7.5, mode - 8, standard deviation - 2.04). The highest level of agreement was with the Education page (7.82), the lowest - with the Shopping page (6.27). These results correlate with the level of trust in these pages: the Education page has the highest level of trust (90% of the visitors stated that they trust the page), whereas the Shopping page has the lowest percentage (63%). In average, 75% of all the respondents

16 stated that they trust the curators and their choice of the links. The levels of trust and agreement do not correlate with such variables as tendency of the curator to get bribed and receive commission from ZEEF, reasons of creating the page and choosing a particular topic of the page, time spent on making the page and frequency of updates: the Spearman correlation coefficients for all these variables are less than 0.2 (Spearman correlation coefficient was used here because some variables were turned into ranked data: the variables were given a particular rank depending on the answer that was chosen by the respondent. For example, for question “How often do you update/revise your ranking?” the answers were coded as following: 1 = “Once per day”, 2 = “Once per week”, 3 = “Once per month, 4 = “Never”). Therefore, the hypotheses H4 and H5 are refuted. However, due to the fact that there were just 6 pages analyzed by the visitors, this result may be biased, because the sample is too small to be representative.

The level of trust relates to the fact whether the visitor sees the curator as an expert. The Phi coefficient7 between these two variables is 0.65, and it is the largest correlation coefficient among all the variables. The fact that the correlation is moderate (more than 0.5, but less than 0.8) means that even though the visitors see the curators as experts, they do not necessarily trust them. The most popular reasons why the visitors do not trust the curators, according to the survey, were: - “I don’t know this man personally/i don't know who this man is” (22% of the respondents). - “His appearance does not have credibility/non-professional photo” (19% of the respondents). - “Too many lists, like he himself doesn’t know which ones are the best” (13% of the respondents). - “He doesn't have any reliable/scientific sources” (9% of the respondents).

68.8% of the visitors have never used expert-based rankings and curated content before. With the help of Phi coefficient, this variable was compared to other variables, but the biggest values of correlation coefficients were -0.29 (whether the person used curated content before, and whether they trust the curators), - 0.22 (whether the person used curated content before, and whether they think that the curator is expert) and - 0.18 (whether the person used curated content before, and whether he sees the list of links ranked). This slight negative correlation means that: - Non-experienced visitors tend to trust the curators more (84% of non-experienced visitors trust the curators, comparing to 56% of experienced visitors). - Non-experienced visitors believe that the curators are experts more (67% against 44% as for experienced visitors). - Non-experienced visitors see the lists of the links as ranked based on the quality of websites they lead to (67% against 48% as for experienced visitors).

Talking about ranking factors, there is no significant difference in the importance of each factor from the point of view of both types of visitors. The average difference in the rankings of the factors is 2.31, and the highest difference was connected to such factor as “Size of the website”: for non-experienced users this factor has rank of 15, however for experienced users the rank of this factor is 5. The average ranking of this factor for both types of the visitors is 3.66 for experienced users and 3.24 for non-experienced. The t-test for equal means (however, for this test not means of the factors, but their rankings were tested) resulted in a P-value = 0.5, which is much higher than the level of significance (be is either 0.05 or 0.1). This means that the null hypothesis (that the difference in rankings of the factors equals 0) should be confirmed. Thus, the sixth hypothesis or the research (H6. Experienced and inexperienced users take into account different ranking factors) is rejected.

7 Pearson correlation coefficient between two binary variables.

To check the perception of the pages by two types of visitors, the same t-test was used for checking the null hypothesis that the means of each variable (connected to the perception of the page by the visitor) for two samples is equal 0. The results of the tests are shown in the table 7.

Table 7. The results of unpaired t-test for means of different variables Variable p-value

Level of trust 0.009

Curator is expert 0.029

Links are ranked 0.058

The P-values for all three tests are less than significance level (for these tests the significance level is taken as 0.06), which means that the null hypotheses is rejected. Therefore, the seventh hypothesis of the research (H7. There is a significant difference between the perception of the page by experienced and inexperienced users) is confirmed.

It is worth mentioning, that 70% of all the curators consider themselves experts in the topic they create their page about, while only 60% of the visitors believe that the curators have a considerable background. However, visitors of ZEEF do not have enough information to check the expertise of the curator. They only see the photo of the curator and a couple of sentences about the curator’s background. Thus, if the photo of the curator is of low quality or non-professional, the visitors will be influenced by this fact and might build negative image regardless of the real expertise and experience of the curator. On the other hand, if the curator does not have enough knowledge about the topic, he or she may write just one sentence in a description field that will convince the visitor (e.g. “I have been living in this country for all my life”, or “I am experienced in teaching”).

Finally, the visitors were asked whether they would use the same pages with curated content and expert-based rankings for future investigations. This question had a Likert scale ranging from 1 to 10. The average grade on this question was 5.8, median - 6, mode - 5, standard deviation - 2.34. Taking the theoretical intention to use similar pages in the future as dependable variable, and level of agreement and trust - as independent, the regression model was built (see pic. 2). As a result of 2-tailed p-values used for testing the null hypothesis that the coefficient (parameter) equals 0, the probability of both variables was less than 0.05. Therefore, the null hypothesis can be rejected, and we can state that these variables predict intention to use curated pages in the future. Adjusted R-squared of this regression is 0.28, which means that the level of agreement with the ranking explains only 28% of the changes of future use of curated pages by the user. In other words, the strength of association of two variables is moderate. However, P-value for the whole regression is zero, which is less than level of significance taken as 0.05. Therefore, the regression is statistically significant.

Picture 2. Regression analysis of level of agreement, level of trust and future use (STATA)

In order to check the first hypothesis (“there is a positive correlation between the level of disagreement of the visitor and level of discrepancy between curators’ and visitors’ ranking factors”) the correlation coefficient between the level of disagreement of the visitors, and difference between the perceptions of both sides on importance of the ranking factors was calculated. First, I calculated the absolute average difference between the rankings of all the factors of each visitor and the factors of the curator, whose page was shown to the visitor. After that, the correlation coefficient was found for two samples (differences in rankings, and level of agreement). For general sample the coefficient is 0.035, which means that there is no significant correlation between these two variables. The P-value for a regression model built for these two variable is 0.75, which means that the regression is not significant. Thus, the first hypothesis about significant correlation between these two variables is rejected.

However, the same analysis for each topic showed the following results (see table 7). For the food page, the correlation coefficient is also very close to zero, however for the business and shopping pages, the correlation is negative. This means that the more the difference between the rankings of the factors, the lower the level of agreement is. For the education, technology and travelling pages, however, this correlation is positive. Moreover, for education page this correlation is quite strong (coefficient is 0.56). In other words, for these pages the level of agreement is higher if there is more discrepancy between the rankings of the factors of the curators and of the visitors, which contradicts the logic.

Table 7. Correlation coefficients between the level of agreement and difference in factors for each topic Topic Correlation coefficient

Business -0.24151

Education 0.561468

Food 0.028381

Shopping -0.36125

Technology 0.251988

Travel 0.144784

TOTAL: 0.035026

The regression model for the level of agreement and difference between the rankings of the factors for the Education page has a P-value that equals to 0.09, so this model is not significant if the level of significance is 95%. However, it can be accepted if the level of significance is 90%. The adjusted coefficient of determination is 0.3, which means that only 30% of the changes in level of agreement are explained by the changes in difference between the rankings of the ranking factors.

Taking into account that the amount of the respondents of each page (only 10 people chose to evaluate the education page, and for other pages this number is less than 30) and a lot of other factors that may influence the level of agreement (e.g. pictures in the headers of each block, the type of the content of each link, the background of the visitors, etc.), it was decided to count such a high correlation as a coincidence, and the regression as not significant.

4.4.5 Conclusions All in all, the process of creating the page on ZEEF by the curators correlates with the main idea and concept of ZEEF.com: the analysis of the results of the surveys did not reveal anything unusual. However, there are some differences and correlations between points of view of the visitors and the curators of ZEEF pages. Particularly, the curators, in general, update their pages less often than the visitors think they do. What is more, the majority of the curators do not rank the lists; however, the majority of the visitors think that they are ranked. Thus, it is important to make both parties perceive the lists in a similar way. For example, ZEEF may teach the curators that the lists should be ranked based on the quality of the website they lead to.

Talking about level of disagreement of the visitors - it is not related to how much the curators are prone to get bribed or how much effort they make in making rankings (there is no significant correlation between level of disagreement of visitors, and such variables as time needed for creating the page, or, for example, reasons for making the page). Thus, the 4th and 5th hypotheses (“rankings of the curators who are prone to get bribed or do not make enough effort in making rankings, have higher level of disagreement [or lower level of trust] among the visitors”) are refuted. However, the second and the third hypotheses (H2. “The visitor, who disagrees with the ranking, will not use it for the future search” and H3. “The visitor, who doesn’t trust the curator, will not use his page for the future search”) were confirmed, as the regression model built for intention for future use as dependent variable and the level of agreement with the ranking and trust in the curator as independent variables is statistically significant. Finally, the first hypothesis is also rejected, so the level of agreement with the rankings does not depend on the factors that are taken into account by the curator and by the visitors. Instead, this variable may be influenced by other factors, such as the background of the visitors and their knowledge, the topic of the page, the type of the websites in the lists, and even the names of the links in the blocks. These factors were not covered by this research, so they provide great opportunities for future work.

5. Interface experiment

5.1 Design In addition to the main questions, the second survey for the visitors contains an experimental part - an A/B test of design (interface) of ZEEF pages. The respondents should have compared two screenshots of the pages with minor differences in interface, such as presence or non-presence of the personal information of the curator; pictures on headers of each list of the links; paid publisher sign, etc. Based on these screenshots, the visitors were asked to choose which page they would rather use. An example of the question is presented on a picture 2.

The third option “No difference” could be chosen by those who did not see the difference between two screenshots. Such test helped to understand whether particular design features of content curation pages influence the perception of the quality of these pages by visitors. If the respondents choose “No difference” option, it means that the interface aspect that is being tested does not influence the perception of the quality of the page and possible future use of this page.

Picture 2. An example of a question about the interface

5.2 Results The majority of the respondents would rather use the page with a curator’s personal info (62.5%), with pictures in headers of the lists (92.5%), and recently updated (76.3%). They see no difference between pages with or without sign “paid publisher” (60% chose “No difference”), and would rather use pages with popularity ranking, however this preference is not very strong (43% chose popularity ranking, 36% - curator’s ranking and 21% saw no difference). Finally, the statistics of the page (amount of views, clicks, likes, shares, etc.) also does not play too much a role in the perception: 46% did not see the difference between to the same pages with different statistics; however, 39% chose the page with big figures (opposed to 15% who chose the page with small statistics).

5.3 Discussion The results of design experiment support the results received in the main part of the second survey. Particularly, the visitors confirmed the importance of frequency of updates of the page. This means that they are interested in up-to-date information. This correlates with the fact that the ranking factor “Up-to-datedness of the information on a web site” is fourth most important one.

In addition, the curator’s personal information and contacts also play important role in building the perception of the quality of the page by the visitor. The majority of the visitors would rather use personalized pages than pages without any information about the curator. In addition, both the visitors and the curators stated that credibility of the source and contacts of the author is the third most important factor influencing the quality of any website. Finally, this aspect influences the level of trust of the visitor, because without the information about the person who curates the content on a page it is hard to estimate the validity and authorship of the content.

The sign “Paid publisher” on the page correlates with the visitors’ attitude to official commission from ZEEF to the curators. Particularly, 33% of the visitors stated that they do not care whether the curator receives the commission or not. Moreover, 40% of them believe that such a commission positively affects the quality of the pages in general. Still, the majority of the visitors do not see the difference between two pages with and without the “paid publisher” sign. This can be explained by the fact that this sign is quite unobtrusive, so people simply did not pay attention on it. However, each screenshot in the test had headers (comments) which described the difference between the variants, in order to understand the general attitude of the visitors towards the idea standing behind a design feature. Therefore, money received by the curator as an official commission does not influence the perception of the page by the visitor.

Finally, the fact that the visitors do not see the difference between the type of the ranking and the statistics of the page means that they tend to not to rely on other people’s opinions in building their opinion about the page. In other words, they do not care how many people use this page, share it and follow the updates: they build their perception on the basis of their own opinion, on the popularity of the page and opinions of other users.

6. Overall discussion

6.1. General discussion and limitations With the help of this research, it was possible to check which factors influence the perception of the quality of the page with curated content (human-based ranking of the websites) in it. Particularly, it was proved that visitor’s trust in the curator and his page, as well as agreement with expert-based ranking of the websites influence the visitors’ attitude to use the page in the future.

Ranking factors play a crucial role in establishing the ranking. The results of our investigation showed that there is no correlation between the level of agreement of the visitor, and level of discrepancy between the ranking factors. This can be explained by the fact that the visitors do not know anything about the topic in which they are searching for information, and cannot say whether there are websites with the content of a better quality. In order to evaluate the ranking that they need some background in this topic. However, taking into account the main purpose of human-based search engines - to help people to find information - the assumption is that the visitors use the website to fill the gap in their background. Thus, they cannot say for sure whether they agree with the ranking or not, because they do not have enough expertise in the topic to judge.

Therefore, for expert-based ranking of the websites, ranking factors do not play that crucial role in establishing the level of agreement of the visitors due to their (visitors’) low level of knowledge or background. However, there are certain limitations of this research that may have led to this result. Particularly, the sample of the visitors of each topic was not representative (there were less than 30 people evaluating each of the six pages). In addition, some of the respondents may not have checked out the websites in the ranking (not click on the links in the list) and evaluated them only on the basis of their names (however, some respondents were personally asked to click on the links and estimate the quality of the websites. In addition, it was highlighted in the survey).

It was proven that experienced users tend to perceive the pages differently. Particularly, they are more critical: they see the curators less as experts, trust them less, and understand the ranking of the lists better (e.g. they understand that non-ranked lists are not ranked, and ranked ones are ranked). In addition, the level of agreement and tendency to use ZEEF pages in the future for experienced users is lower (however, the difference is relatively small: the level of agreement for non-experienced and experienced users are 7.21 and 6.96 respectively, an intention for future use is 5.92 and 5.64 respectively). This means that the first impression of the users is more positive; however, they need some time to understand the real purpose and a working principle of human-based search engines and curated platforms. After getting some experience, the users realize that the curators may be a usual person like the visitor himself, with personal opinion, which may differ from the truth. Therefore, in order to retain new users, the platform should be able to stand up to a more critical scrutiny.

Another aspect of the quality of the page is level of trust. In order to understand, what exactly should be done to keep the level of trust high, ZEEF and other human-based curation systems should keep in mind three facets of trust: benevolence, honesty and competence.

First of all, the platform has to build an image of a benevolent party, whose main goal is to satisfy present and future interests, desires and needs of its users. This is successfully done by ZEEF: the platform highlights its vision to help people find the content that was ranked, categorized and organized by passionate people. ZEEF helps people to get out of the “filter bubble”8 and get trustworthy results.

Secondly, the information on a platform should be sincere and honest. The content of ZEEF pages is mostly represented by the links on other sources: there is a limited opportunity to create unique content and type it into the page, therefore, create the content that is not verified. In addition, in curated platforms such as ZEEF the quality of the content depends solely on curators, their choice of the links and adequacy of the rankings. However, the page of the curator represents his/her personal opinion regarding the topic, so even though the first link is not necessarily the best - it is only the opinion of the curator. There is always an opportunity to suggest other links, or create a page on the same topic, which will provide the visitors with several points of view. Therefore, in order to achieve higher level of trust, the curated platforms should always give the visitors opportunity to share their opinion and collaborate in creating and filtering the information. In addition, in order to avoid the subjectiveness of the curator, the platform should show that other, “secondary” curators participated in creation of the content. In ZEEF, this is made by inserting a small picture of the curator next to the proposed link. However, the picture is only added if the person who proposed the link is registered

8 A filter bubble represents customized results from search engines that are geared to the individual based on that person's past search preferences. It means that two people searching for the same thing receive a different sequence of results. Source: Eli Pariser (2012) The Filter Bubble: How the New Personalized Web Is Changing What We Read and How We Think. ISBN-13: 978-0143121237

23 on the platform. Further research is needed for understanding whether the visitors will trust “secondary” curators, and how many proposed links, related to the initial links of the curator, should be on the page (the hypothesis may be that the more links of other curators there are on the page, the less people trust the main curator due to his small participation in creation of the content).

From the literature review it was clear that design of the page plays a significant role in building up trust level of the visitors. The results of the A/B test proved that some changes in the interface of the page influence the perception of the visitors. Particularly, for the respondents it was critical that the page has personal information of the curator, date when the page was last updated, and images on the page, so it looks more catching and lively. The limitation of A/B test in this research is that the users had to choose which page they would rather choose to use. In order to get more precise results, it would be better to use Likert scale for each screenshot, in order to estimate the difference in the estimates of each page. Also, it would be useful to evaluate which types of header images are more appealing and useful for the visitors, and what are the keywords that should be included into a personal description.

One of the most popular feedback from the potential visitors of ZEEF was the fact that some pages were too big, and it made them feel that the curator him(her)self did not know which link was the best. Therefore, the size of the curated content is also important (in other words, it is important to not only curate and aggregate content, but also filter out sources of a lower quality). However, the exact limit in the amount of the links has to be investigated in the future work.

Other hypotheses about the influence of time and effort the curator has made for creating the page, means of finding and filtering information, and reasons for creating the page on both trust and agreement of the visitor, were refuted. Therefore, it does not matter how and why a curator makes up the lists - in any case the visitors tend to see what they want to see: a high-quality, recently updated page with ranked lists of links which is made by a professional for the sake of helping people to find the best and most relevant sources of information. This supports the significance of the design of the page in establishing its quality.

6.2. Guidelines for the curators On the basis of the results of this research the following guidelines for the curators of ZEEF and other expert-based curated platforms were developed. These guidelines include information not on why to create a page and which topic to choose, but on how to do it, and what to pay attention on. With the help of these guidelines the curators of the pages may achieve the higher quality of their content (particularly, expert-based rankings of the websites), lower level of disagreement and higher level of trust of the visitors.

Resources. The content on the page should be curated and aggregated - thus, the links on the page should lead to many different sources. In order to achieve the variety in the sources and to get rid of the effect of “information bubble” it is highly recommended to use not only personal bookmarks and main search engines, but also other curated websites and search engines that do not use information of the user for establishing the ranking (thus creating the information bubble). For example, if the curators creates a page about the company, not only the links on the main website of the company should be included, but also links to different sources with articles where this company was mentioned, or the links to the articles about the business models or economic aspects this company is connected with. In addition, it is also recommended to ask for a feedback from the visitors and take into account their recommendations on improvement of the page. Although the way the content is made does not influence the perception of the quality of the page by the

24 visitors, it is still important to provide the visitors with the best service (in case of curated platforms the main service is to provide high-quality information on a subject).

Personal information. The curators should upload their professional photo in order to make the visitors see them as trustful expert. Other psychological researches already investigated the importance of profile pictures in building trustworthiness of the website user or, in our case, a curator (Ulrike Steinbrück et al, 2002), (Yen-Chun Jim Wu, 2014), however this aspect needs to be tested in the context of human-based search engines. In general, non-professional or low-quality photo considerably decreases the trust level of the visitors. In addition, the curators should write concise and relevant description of their expertise. It is recommended to critically review personal information: would you trust the person on this photo, and is there enough information about your background and expertise? The level of trust of the visitor is partly influenced by the curators themselves, and it is the curator's’ responsibility to build a professional and trustful image.

The structure of the page. The page should be concise and not very big - otherwise the visitor may easily get lost. If there are too many sources, which are of the same quality - it is important to categorize them into blocks for ease of use. In order to make the page more pleasant and nice it is better to use relevant pictures as headers (however, the pictures should not be distractive). The names of the links should be readable and informative, and the first links should be of higher quality than the next ones (even though the curators may not rank them, they remember that the majority of visitors think that they are ranked). Finally, the page should be updated regularly: if the topic requires regular updates, (these topics may be travelling, events or shopping - due to regular changes in sales, dates of events, seasons, etc.) it is better to update the page every week. For topics that change slowly, such updates may happen every month. If no changes in the topic happen, then at least an indicator should be given that the curator checked the accuracy of the old links. Finally, the page should have the added value, bring benefits for the visitor and solve the problem the visitors may have. That is why it is highly recommended to put links to particular tools, programs and solutions for the users, not to the articles, reviews and news.

Ranking factors. Although the difference in ranking factors of the curators and of the visitors does not influence the visitor’s perception of the quality of the page, it is still necessary for the visitor to know which factors are more important for the visitor. Therefore, for evaluation of the webpage the curator should first of all think of the following criteria: relevance of the content to the topic, correctness of the information, credibility of the source, and amount of advertisements on the page (the page should not have too many banners or pop-up windows which disturb the visitor, and the content should be clear, direct and free of “sales- speak”).

On the other hand, the curator should not rely too much on their personal experience or knowledge of web site. In order to be as objective as possible, the curators should remember that the websites they have used before are not necessary the best. Therefore, they should be open to new sources and be ready that there are better websites than the ones they got used to. For example, talking about online shops: the price of the product is more important than quality and uniqueness for majority of the visitors, however this factor depends on the type of the product and target audience of the shop (for example, the price of the product is not that important for the products of premium class or custom-made ones). Therefore, it cannot be applied to all the products, but the curators should always keep in mind that the most popular factor for ranking the links, from the point of view of the visitors, is the price.

7. Conclusions and future work In this research, the process of creation of the expert-based rankings of the websites was evaluated from the point of view of two parties: the curators (creators of the rankings) and the visitors (the users of the rankings). The list of the most relevant ranking factors for website evaluation was created, and their importance for the both parties was analyzed. In addition, both quantitative and qualitative data was gathered regarding the process of creating expert-based rankings of the websites (including decisions for establishing a ranked list), and the perception of these rankings by the visitors. With the help of two surveys and an A/B test, the hypotheses about the relations between the intention to use the expert-based ranking of the websites in the future and the level of trust and agreement of the users were tested. As a result, it was revealed that the users would agree to use expert-based rankings of the websites in the future in case they have high level of trust and agreement with these rankings. Therefore, in order to get high traffic for a website it is important to build a trustful image of both the curators and the platform in general. Moreover, the ranking itself should not be too subjective in order to decrease the level of disagreement between the creator and its user. This problem of potential disagreement can be addressed by establishing a new adaptation algorithm, which takes into account not only the opinion of the creator, but also the opinion of the visitors. For example, the visitors of a page may specify on which place they want to see a particular link (“vote” for it). After that, the algorithm counts the average rank of this link and builds up a new ranking. Therefore, if the curator puts some link as the last one, but the visitor shows that he or she wants to see this link being the first, in the end this link will be somewhere in the middle (will take the average of the creator’s and the visitor’s opinions). However, the implementation of this algorithm and its analysis may be established in the future work.

In addition, the influence of different aspects of curated directories on user’s perception of the quality of the page was analyzed. It was revealed that the intention to use the curated page in the future is influenced by the level of trust of the users, and their agreement with the content. However, in case of expert-based ranking of the websites (which is the main content in such curated platforms as ZEEF) the level of agreement of the visitor does not depend on the factors that are taken into account by the curators and the visitors for establishing the ranking of the websites. In addition, the process of creation of the ranking by the curator (such questions as where the curators find the content, how often they update the ranking, why they create the page with the ranking, etc.) does not influence the level of trust of the visitor. Instead, particular features of the design of the interface of the page with curated content play crucial role in establishing the quality of the page and influence the perception of the page by the visitors. These features are personal photo of the curator, personal information about the curator, links to his/her social profiles, frequency of the updates of the ranking on the page, etc. However, further research is needed for a deeper analysis of particular features of the interface and their influence on the level of trust of the visitor.

This and previous research showed that trust is one of the main factors that influence the perception of the website by the visitor, however it is not yet clear, which exact features influence the level of trust. As a future work, first of all, it is necessary to research what other factors influence the level of trust and agreement of the visitor of curated pages with expert-based ranking of the websites on them. Among these factors may be the size of the page, trustworthiness of the experts creating curated content, their personal profile pictures and personal information, structure of the page, its design, presence of “secondary” curators who contribute to the creation of content, and so on. Next, it is important to analyze what other factors, aside from trust and agreement, influence the perceived quality of human-curated recommender environments, and the intention of the visitors to use these environments in the future. Finally, it is possible to dive into the sphere of human- based search engines and evaluate what other factors influence the level of agreement of the users of expert- based rankings of the websites, aside from ranking factors.

8. List of references 1. Agreement. Oxford Dictionary of English (3 ed.) Oxford University Press. Published online: 2010. Available at: http://www.oxforddictionaries.com/definition/learner/agreement 2. Alain Materne, Gershom Sleightholme (2014) Methods of ranking search results for searches based on multiple search concepts carried out in multiple databases. World Patent Information, Volume 36, March 2014, P. 4-15 3. Altman, Alon; Moshe Tennenholtz (2005). "Ranking Systems: The PageRank Axioms". Proceedings of the 6th ACM conference on Electronic commerce (EC-05). Vancouver, BC. 8p. 4. ASCI (2014) ACSI E-Business Report 2014. ACSI LLC. July 22, 2014, p8. 5. C. A. Moser (1952) Quota Sampling. Journal of the Royal Statistical Society. Series A (General) Vol. 115, No. 3 (1952), pp. 411-423 6. Dirk Lewandowski (2005) Web searching, search engines and Information Retrieval. Information Services & Use 25 (2005) 137–147 7. Evaluating Web Pages: Techniques to Apply & Questions to Ask (2012), UC Berkeley - Teaching Library Internet Workshops. Retrieved from: http://www.lib.berkeley.edu/TeachingLib/Guides/Internet/Evaluate.html 8. Fogg, B.J. (May 2002). "Stanford Guidelines for Web Credibility." A Research Summary from the Stanford Persuasive Technology Lab. Stanford University. Retrieved from: https://credibility.stanford.edu/guidelines/ 9. Gagan Mehra, (2012) 10 Ways to Evaluate Online Marketplaces. Practical e-commerce, May 25, 2012. Retrieved from: http://www.practicalecommerce.com/articles/3558-10-Ways-to-Evaluate- Online-Marketplaces 10. Geoffrey Yeo (2013) Trust and context in cyberspace, Archives and Records: The Journal of the Archives and Records Association, 34:2, 214-234 11. Heidi Cohen (2014) Content Curation Versus Content Aggregation. Retreived February 22, 2015 from: http://heidicohen.com/content-curation-versus-content-aggregation/ 12. Herther, N. K. (2012a). Content Curation. A look on some of the best. Searcher, Oct; 20 (8): 26-35. 13. Herther, N. K. (2012b). Content Curation. Searcher, 20(7), 30-41. 14. Honigman B. (2013) 3 differences between content curation & content aggregation. Retreived February 22, 2015 from: http://blog.newscred.com/article/3-differences-between-content-curation- content-aggregation/7507e57cafeba0ae5910fa9ee7151bf9 15. Hsiu-Fen Lin (2007) The Impact of Website Quality Dimensions on Customer Satisfaction in the B2C E-commerce Context, Total Quality Management & Business Excellence, 18:4, 363-378 16. Hsuan Yu Hsu, Hung-Tai Tsou (2011) Understanding customer experiences in online blog environments. International Journal of Information Management. Volume 31, Issue 6, December 2011, Pages 510–523 17. James E. Bailey and Sammy W. Pearson (1983) Development of a Tool for Measuring and Analyzing Computer User Satisfaction. Management Science Vol. 29, No. 5 (May, 1983), pp. 530-545 18. Junghyun Nam (2014) Understanding the motivation to use web portals. Computers in Human Behavior. Volume 36, July 2014, Pages 263–273 19. Leo A. Goodman (1961), Snowball Sampling. The Annals of Mathematical Statistics, Vol. 32, No. 1 (Mar., 1961), pp. 148-170 20. Liwen Vaughan (2004) New measurements for search engine evaluation proposed and tested. Information Processing and Management 40, Pages 677–691 21. Melody M. Madrid (2013) A study of digital curator competences: A survey of experts. The International Information & Library Review. Volume 45, Issues 3–4, December 2013, Pages 149–156

22. Mirjam Seckler, (2014) Trust and distrust on the web: User experiences and website characteristics. Computers in Human Behavior, Volume 45, April 2015, Pages 39–50 23. More guidance on building high-quality sites, (2011) Google webmaster central blog. May 06, 2011. Retreived from: http://googlewebmastercentral.blogspot.nl/2011/05/more-guidance-on-building-high- quality.html 24. Natalie Clewley Sherry Y. Chen Xiaohui Liu, (2009),"Evaluation of the credibility of internet shopping in the UK", Online Information Review, Vol. 33 Iss 4, Pages 805 - 826 25. O'Brien, H. L. and Toms, E. G. (2008) What is user engagement? A conceptual framework for defining user engagement with technology. J. Am. Soc. Inf. Sci., 59: 938–955. 26. Rik Crutzen et al (2014) Can interest and enjoyment help to increase use of Internet-delivered interventions?, Psychology & Health, 29:11, 1227-1244 27. Robson C., Real World Research. A resource for users of Social Research Methods in Applied Settings, John Wiley & Sons; 3rd Edition edition (4 Mar. 2011), P. 608 28. Rohit Barghava’s 5 Models of Content Curation: http://www. rohitbhargava.com/2011/03/the-5- models-of-content-curation.html 29. Shrinky, C. (2010) Talk about curation. Retreived February 21st, 2015, from http://curationnationvideo.magnify.net/video/Clay-Shirky-6#c=1RMHLF18HLLBQ6WK&t=Talk about Curation 30. Thomas Young (2009) Why People Visit Websites. April, 2009. Retreived from: http://intuitivewebsites.com/articles/why-people-visit-websites/ 31. Ulrike Steinbrück et al. (2002) A Picture Says More Than A Thousand Words - Photographs As Trust Builders in E-Commerce Websites. Short Talk: User Studies-Lessons for HCI, pp.748-749 32. Utpal M. Dholakia, Lopo L. Rego, (1998) "What makes commercial Web pages popular?: An empirical investigation of Web page effectiveness", European Journal of Marketing, Vol. 32 Iss: 7/8, pp.724 – 736 33. Venkatesh Shankar et al (2002) Online trust: a stakeholder perspective, concepts, implications, and future directions The Journal of Strategic Information Systems, Volume 11, Issues 3–4, December 2002, Pages 325–344 34. Why People Go Online. July, 17, 2009. Retreived from: http://www.emarketer.com/Article/Why- People-Go-Online/1007184 35. William O. Engler (2013) A Methodology for Creating Expert-based Quantitative Models for Early Phase Design. A Thesis Presented to The Academic Faculty in Partial Fulfillment of the Requirements for the Degree Doctor of Philosophy in the School of Aerospace Engineering, p. 282 36. Ye Diana Wang, Henry H. Emurian (2005) An overview of online trust: Concepts, elements, and implications. Computers in Human Behavior, Volume 21, Issue 1, January 2005, Pages 105–125 37. Yen-Chun Jim Wu, et al. (2014) Do Facebook profile pictures reflect user’s personality? Computers in Human Behavior, p. 10 38. Yong-Soon Kang, Yong Jin Kim, (2006) Do visitors' interest level and perceived quantity of web page content matter in shaping the attitude toward a web site? Decision Support Systems Volume 42, Issue 2, November 2006, Pages 1187–1202 39. Zeinab Abbassi, Nidhi Hegde, and Laurent Massoulie. (2014). Distributed content curation on the web. ACM Transactions on Internet Technology, Vol. 14, No. 2-3, Article 9 (October 2014), 15 pages.

Appendix 1. Preliminary list of questions for surveys

Survey 1. The curators The aim of this survey is to understand the process of making the ranking of the lists of links to web sites by the curators. If you have more than one ZEEF page, please choose ONE to consider while answering the questions. The survey consists of 15 questions and will take no more than 7 minutes.

1. Please, write down the URL of your page. (Open question) 2. What general topic does your page cover? a. Technology, gadgets, games and Internet b. Travelling, cities, events and culture c. Sport and health d. Food and drinks e. Shopping and brands f. Hobby, games, movies g. Business, management and companies h. Science and nature 3. What influenced you to make the page? a. Build online authority and show my expertise in a topic b. Bookmark and organize my favorite links c. Help people find information d. Earn money via affiliate links e. Friends asked to create a page 4. How did you choose the topic of your page? (multiple answers possible) a. I wanted to compete with another curator at ZEEF b. I have much expertise and experience in this topic c. I am passionate about this topic d. I wanted to learn more about this topic e. I did not find anything about this topic at ZEEF 5. Do you consider yourself a professional in this topic and why? (open question) 6. Is the first link of each list (at the top of the list) the best among all the links in the list? a. Yes b. No 7. Where do you take the links for the ranking lists you make? (multiple answers possible) a. Take from my personal bookmarks b. I know them by heart c. Curate panel (to the right side of the screen) d. Suggestions from visitors e. Other rankings in similar (curated) pages f. Google/Bing search 8. How important are the following factors for evaluation of the websites and of the links you add and rank on your page? (For each question there is a Likert 5-scale from “Not important at all” to “Very important”. For each respondent the factors were put in random order) - Relevancy of the content to the subject

- Correctness and accuracy of the content - Credibility of the source - Personal trust in web site - Personal experience/knowledge of the web site - Completeness of the content - Money received as commission from affiliate links, or directly from the site-owners - Up-to-dateness (how new the content is) - Ease of understanding of the web site (usability) - Performance of the web site - Recommendations from other parties - Visual layout (design of the site) - URL and domain - Position in Google ranking - Advertisements on a web site (banners) - Size of the web site - Uniqueness (originality) of the products - Price of the products - Quality of the products 9. How much time does it take you to come up with the very best link? a. Less than 1 minute b. Less than 1 day c. Less than 1 week d. More than 1 week 10. How often do you check and/or revise the rankings or update the lists after finishing them? a. Every day b. Every week c. Every month d. Never 11. Would the amount of products sold via your links influence your ranking? (Imagine you have a list of links to different brands of sneakers. If you knew the amount of sneakers bought through your links, would you change your ranking on the basis of this information?) a. Yes b. No c. I don’t know 12. Would you (theoretically) put the link on the 1st (2nd, 3rd) place for money secretly received from a third party? (The third party may be, for example, a site-owner, who wants to promote their page via your ranking list) a. Yes b. No 13. (If the answer to previous question was “Yes”) How much money (minimum) would you take for changing your ranking? a. 1 eur b. 10 eur c. 100 eur d. 1000 eur 14. Would you like to get paid by ZEEF (officially) for maintaining your page?

(The payment may be an official commission from ZEEF on the basis on particular characteristics of your page). a. Yes b. No 15. (If the answer to previous question was “Yes”) What would you like to receive commission for? (multiple questions possible) a. Amount of clicks b. Amount of views c. Click-through Rate (CTR) - clicks/views ratio d. Amount of purchases on a web site e. Amount of shares f. Amount of "thanks" g. Amount of followers 16. (If the answer to question #14 was “Yes”) Would you agree to have a special sign on your page showing that you receive commission from ZEEF? a. Yes b. No

Survey 2. The visitors ZEEF is a human search engine based on a directory with results that are selected and ranked by the real people (curators). The aim of this survey is to analyze how the visitors perceive and understand the rankings of the lists of links to web-sites on ZEEF pages. You will be asked to choose the topic and open a ZEEF page covering this topic. Please, open the proposed page and analyze the rankings on that page carefully, as all the questions of this survey are based on these rankings. Don't forget to check out the web-sites the links in the rankings lead to. The survey consists of 15 questions and will take no more than 10 minutes.

1. Please, choose the topic you are interested in the most a. Shopping b. Travelling, cities c. Food and drinks d. Education e. Business f. Technology Please, check out any list on this page (the link is given to a particular page). You may click on the links to see the content of web sites listed in the ranking. The next questions of the survey will be connected to your attitude to this page in general and its content.

2. How much do you agree with the rankings on this page? – Likert scale from 1 to 10 3. If you don't agree with the rankings - why? What would you change in any of these rankings? (open question) 4. In your opinion, how relevant are the following factors for ranking the links to the web sites on the page? (for each factor there was a Likert chart ranging from 1 – not relevant at all to 5 – very relevant) - Relevancy of the content to the subject - Correctness and accuracy of the content - Credibility of the source - Personal trust in web site

- Personal experience/knowledge of the web site - Completeness of the content - Money received as commission from affiliate links, or directly from the site-owners - Up-to-dateness (how new the content is) - Ease of understanding of the web site (usability) - Performance of the web site - Recommendations from other parties - Visual layout (design of the site) - URL and domain - Position in Google ranking - Advertisements on a web site (banners) - Size of the web site - Uniqueness (originality) of the products - Price of the products - Quality of the products

5. Do you think the curator is an expert in this topic and why? (open question) 6. Do you trust the curator’s choice? a. yes b. no 7. How often do you think the curator revises/updates the list of the links after finishing the page? a. Every day b. Every week c. Every month d. Never 8. Do you think the order of the links depends on the date when they were added or on the quality of websites? a. Date added (it's just a normal list) b. Quality of the web-sites (it's a ranked list) 9. Would you use similar ZEEF lists for your search in the future? - Likert scale from 0 (not at all likely) to 10 (extremely likely) 10. Imagine this curator officially receives commission from ZEEF. How do you think the curator may be paid? a. for amount of clicks b. for amount of views c. for CTR (clicks/views) d. for amount of purchases made on the web sites via his links e. for amount of shares f. for amount of “thanks” g. for amount of followers 11. If you had an information about what the curator is paid for, would this information influence your behavior on ZEEF pages and why? (e.g. would you click on the links more?) (open question) 12. How do you feel about the fact that some of the curators could receive official commission from ZEEF? a. I think that it encourages curators to make better rankings. b. I don’t know / don’t care. c. I think that it negatively affects the quality of the rankings.

13. Continue the sentence: If I realise that the curator was paid by the third party for changing the lists of the links… a. ...I would not care about it b. ...I would be disappointed that the ranking is biased, but would still use his page (in Google, for instance, it’s the same anyway) c. ...I would never use his page anymore

14. Have you ever used expert-based rankings (ranked lists)? a. Yes b. No

The most difficult part of the survey is over! Now I kindly ask you to choose which pages you would rather choose to use. Some of them have minor differences, so if you don't see them - just click "No difference". It is just 6 easy choices - and you will finally finish this survey!

15. Which page would you rather choose to use? (for each question two screenshots of the same page with minor differences + option “No difference” were shown) a. With and without curator’s personal info b. With and without sign “Paid publisher” c. With and without header pictures of each block d. With small statistics and with big statistics e. Ranking made by curator and Ranking based on amount of clicks f. Recently updated and non-updated

16. Select your gender a. Male b. Female

17. Your age (open question) 18. What is the highest level of education you have completed? a. Less than High School b. High School / GED c. 2-year College Degree d. Bachelor Degree e. Masters Degree f. Doctoral Degree g. Professional Degree (JD, MD)

Appendix 2. Table with ranking factors

Table 6. Lists of the factors and their rankings by the curators and the visitors Curators’ estimation Visitors’ estimation

St. deviation of means 0,81 0,46 Factors Mean St. dev Median Mode Rank Mean St. dev Median Mode Rank difference in ranks Relevancy of the content 4,67 0,55 5 5 1 3,49 1,04 4 4 6 -5 to the subject Correctness and accuracy 4,41 0,73 5 5 2 4,36 0,86 5 5 1 1 of the content Credibility of the source (contacts of the author, 4,15 0,97 4 5 3 4,04 0,86 4 4 3 0 references) Personal trust in web site 4,05 1,08 4 5 4 3,64 1,09 4 4 7 -3 Completeness of the 4,03 0,82 4 4 5 4,14 0,85 4 4 2 3 content Personal experience/knowledge 4,00 0,94 4 4 6 3,49 1,03 3,5 3 16 -10 of the web site Up-to-dateness (how 3,97 0,99 4 4 7 3,86 0,92 4 4 4 3 new the content is) Quality of the products 3,94 1,15 4 5 8 3,95 0,79 4 4 18 -10 Uniqueness (originality) of the products 3,76 1,04 4 4 9 3,52 0,98 4 4 19 -10 Ease of understanding of 3,72 1,12 4 4 10 3,09 1,14 3 3 8 2 the web site (usability) Performance of the web site (it should be fast and 3,38 1,2 4 4 11 2,7 1,06 3 3 12 -1 error-free) Recommendations from 3,26 1,16 3 4 12 3,09 1,10 3 3 14 -2 other parties Visual layout (design of 3,24 1,14 3 4 13 3,93 0,94 4 4 11 2 the site) Price of the products 2,85 1,22 3 4 14 3,41 0,98 4 4 17 -3 URL and domain 2,58 1,27 2,5 3 15 2,95 1,02 3 3 15 0 Position in Google 2,55 1,38 2 1 16 3,55 0,99 4 4 10 6 ranking Size of the web site 2,46 1,14 2 2 17 2,89 1,09 3 3 9 8 Advertisements on a 2,22 1,1 2 1 18 4,03 1,06 4 5 5 13 web site (banners) Money received as commission from 1,81 1,15 1 1 19 3,44 0,95 4 4 13 6 affiliate links, or directly from the site-owners