Design and Implementation of Search Awareness Cues in Explicit Collaborative Information Seeking

A thesis submitted to the University of Manchester for the degree of

Doctor of Philosophy in the Faculty of Humanities

2017

Husain AlArayedh

Manchester Business School

Table of Contents

List of Figures ...... 6 List of Tables ...... 8 Abstract ...... 10 Declaration ...... 11 Copyright Statement ...... 11 Dedication ...... 12 Acknowledgments ...... 12 List of Abbreviations ...... 13 Word Count ...... 13 Chapter 1. Introduction ...... 14 1.1 Research Domain ...... 14 1.2 Significance and Opportunity ...... 20 1.3 Research Overview ...... 24 1.3.1 Research Question ...... 25

1.3.2 Research Objectives ...... 26

1.3.3 Research Design Process ...... 27

1.4 Thesis Structure ...... 28 Chapter 2. Literature Review ...... 30 2.1 Information Seeking and Retrieval (IS&R) ...... 30 2.1.1 Situating IS&R and Related Definitions ...... 31

2.1.2 Information Seeking Models and Forms ...... 33

2.1.3 Search User Interfaces (SUI) ...... 37

2.2 Collaboration ...... 40 2.2.1 Processes of Collaboration ...... 41

2.2.2 Computer-Supported Cooperative Work (CSCW) ...... 46

2.3 Collaborative Information Seeking (CIS) ...... 48 2.3.1 Situating CIS ...... 48

2.3.2 CIS Models and Taxonomies ...... 52

2.4 CIS Aspects ...... 54 2.4.1 Awareness ...... 57

2.4.2 Communication ...... 60

2.4.3 User Interface (UI) ...... 61

2.4.4 Division of Labour (DoL) ...... 63

2.4.5 Roles ...... 64

2.4.6 Collaborative Grounding ...... 65

2.4.7 Algorithmic Mediation ...... 66

2.5 Implementation of Awareness Cues and Mechanisms in CIS ...... 67 2.5.1 Search Activities ...... 68

2.5.2 Timelines ...... 73

2.5.3 Ratings and Annotations ...... 75

Chapter 3. Methodology ...... 78 3.1 Iterative and Incremental Approach ...... 78 3.2 Design Science Research (DSR) ...... 79 3.2.1 DSR Process Model ...... 80

3.2.2 DSR Philosophical Stance ...... 83

3.3 Methodological Approach ...... 84 Chapter 4. Design and Implementation of SearchAware v1 ...... 85 4.1 SearchAware v1 Overview ...... 85 4.1.1 Design Objectives ...... 86

4.1.2 Design Guidelines ...... 88

4.2 Interface ...... 90 4.2.1 Input Features ...... 93

4.2.2 Informational Features ...... 94

4.2.3 Control Features ...... 94

4.2.4 Further Collaboration and Awareness Cues ...... 94

4.3 Mashup Implementation ...... 95 4.3.1 Social Network: Twitter ...... 97

4.3.2 Mendeley ...... 98

4.3.3 Microsoft Academic Search ...... 98

4.3.4 Data Exchange Format: JSON ...... 99

4.4 Architecture and Backend ...... 99

Chapter 5. User Study ...... 103 5.1 Objective ...... 103 5.2 Study Method ...... 104 5.3 Usage Results ...... 105 5.4 Reflections...... 107 5.5 Discussion ...... 109 Chapter 6. Design and Implementation of SearchAware v2 ...... 112 6.1 Overview ...... 112 6.1.1 Design Objectives ...... 112

6.1.2 Further Improvements ...... 113

6.2 Interface ...... 114 6.2.1 Input Features ...... 115

6.2.2 Informational Features ...... 115

6.3 Architecture and Backend ...... 120 Chapter 7. Experimental Study ...... 123 7.1 Background ...... 123 7.2 Hypothesis and Measurements ...... 125 7.2.1 Productivity and Performance Measurements ...... 127

7.2.2 Interactivity Measurements ...... 129

7.2.3 Usability Measurements ...... 131

7.2.4 Hypothesis Summary ...... 133

7.3 Experiment Design ...... 134 7.3.1 Experiment Tasks ...... 134

7.3.2 Experiment Protocol and Procedures ...... 136

7.4 Results ...... 142 7.4.1 Overview ...... 142

7.4.2 Productivity and Performance Measurements Results ...... 143

7.4.3 Interactivity Measurements Results ...... 146

7.4.4 Usability Measurements Results ...... 149

7.5 Discussion ...... 151 7.5.1 Hypothesis Testing Results ...... 151

7.5.2 Communication ...... 154

7.6 Summary ...... 156 Chapter 8. Conclusion ...... 157 8.1 Summary ...... 157 8.2 Contribution ...... 160 8.2.1 Implications for the Design and Implementation of Awareness Cues ...... 160

8.2.2 SearchAware as a Research Software Artefact ...... 162

8.3 Limitations and Future Work ...... 163 8.3.1 Extended Use of Interface Control and Input Features ...... 163

8.3.2 Further Search Contexts ...... 164

8.3.3 Use of Modern Web Technologies...... 164

8.3.4 Limitations of System-focused Evaluation ...... 164

8.3.5 Laboratory Experiments’ Constraints ...... 165

References ...... 167 Appendix 1. User Study Participants’ Reflections ...... 186 Appendix 2. Participant Information Sheet and Consent Form ...... 189 Appendix 2.1 Participant Information Sheet ...... 190 Appendix 2.2 Consent Form ...... 193 Appendix 3. Experimental Tasks ...... 194 Appendix 3.1 Task 1: Social Media Marketing (BUS) ...... 195 Appendix 3.2 Task 2: Major Sports Event Hosting (MGT) ...... 196 Appendix 3.3 Task 3: Mobile Retail Payment (ICT) ...... 197 Appendix 4. Questionnaires ...... 198 Appendix 4.1 Demographics and Background (Pre-Experiment) ...... 199 Appendix 4.2.1 Engagement in SearchAware (Post-Trial) ...... 201 Appendix 4.2.2 Awareness of Collaborators in SearchAware (Post-Trial) ...... 202 Appendix 4.2.3 Task Load Index (Post-Trial) ...... 204 Appendix 4.2.4 Usability (Post-Trial) ...... 205

List of Figures

Figure 2-1: Research continuum for conceptualising IIR research (Kelly, 2009, p. 10) ...... 32 Figure 2-2: A nested model of the information seeking and information searching research areas (Wilson, 1999, p. 263) ...... 33 Figure 2-3: Modes of Information Seeking (Bates, 2002, p. 4) ...... 33 Figure 2-4: Disciplines affecting the design of Search User Interfaces (Wilson, 2011, p. 10) ...... 38 Figure 2-5: Two collaboration models: 5-C (Taylor-Powell, Rossing and Geran, 1998, p. 4) right, 3-C collaboration model (Fuks et al., 2008, p. 638) ...... 42 Figure 2-6: The 5-C collaboration model (Shah, 2013b, p. 1124) ...... 42 Figure 2-7: The Space / Time Groupware Matrix - A variation of the CSCW Matrix – User: Pascal (Momo54) © Wikipedia Commons (https://commons.wikimedia.org/wiki/File:Cscwmatrix.jpg) ...... 47 Figure 2-8: Taxonomy of social search approaches (Burghardt, Heckner and Wolff, 2012, fig. 1) ...... 49 Figure 2-9: CIR conforming disciplines (Fernández-Luna et al., 2010, fig. 1) ...... 51 Figure 2-10: CIS as an interdisciplinary field, (Shah, 2013a, fig. 1)...... 51 Figure 2-11: Graphical representation of (Golovchinsky, Qvarfordt and Pickens, 2009) CIS model .... 53 Figure 2-12: CoSense - Keyword History (Paul and Morris, 2011, p. 86) ...... 69 Figure 2-13: Querium v2 – SUI / SERP (Golovchinsky, Dunnigan and Diriye, 2012, p. 1804) ...... 69 Figure 2-14: Coagmento v3 – Workspace (Kelly and Payne, 2014, p. 809 - Copyright © Chirag Shah) 70 Figure 2-15: CollabSearch SERP Interface (Yue, Han and He, 2014, p. 822) ...... 71 Figure 2-16: ResultsSpace SUI/SERP (Capra et al., 2013, p. 3) ...... 71 Figure 2-17: Instant Messaging in CoFox (Perez, Leelanupab and Jose, 2012, p. 265) ...... 72 Figure 2-18: CoSense - Workspace (Paul and Morris, 2011, p. 88) ...... 72 Figure 2-19: CoSense - Search Strategies Summary (Paul and Morris, 2011, p. 85)...... 73 Figure 2-20: CaseLine timeline (Bohøj et al., 2010, p. 528) ...... 74 Figure 2-21: CoSense - Timeline’s implementation (Paul and Morris, 2011, p. 87); ...... 75 Figure 2-22: SearchTogether Rating - (Morris and Horvitz, 2007, p. 3) ...... 76 Figure 2-23: CollabSearch - Rating and Commenting Interface (Yue et al., 2014, p. 47) ...... 76 Figure 2-24: Google SearchWiki (from http://justinhileman.info/article/searchwiki-googles- customized-social-search-is-back/) ...... 77 Figure 2-25: ezDL SERP (Böhm, Klas and Hemmje, 2014b, p. 36) ...... 77 Figure 3-1: Design science research process model (DSR cycle) (Vaishnavi and Kuechler, 2015, p. 15) ...... 80 Figure 4-1: Guidelines for building a successful CIS environment (Shah, 2012, p. 22) ...... 88 Figure 4-2: General Search User Interface Guidelines (Hearst, 2009, sec. 1.12) ...... 90

Figure 4-3: SearchAware Version 1 - Main SUI and SERP ...... 92 Figure 4-4: SearchAware v1 - Interact Page ...... 93 Figure 4-5: SearchAware v1 architecture ...... 99 Figure 5-1: Participants activities summary chart ...... 106 Figure 6-1 SearchAware Version 2: Main SUI and SERP ...... 117 Figure 6-2: SearchAware v2 - Detailed view and description of the SERP ...... 118 Figure 6-3: SearchAware v2 - Abstract Window ...... 119 Figure 6-4: SearchAware v2 - Comment Entry Window ...... 119 Figure 6-5: SearchAware v2 - Keyword Summary Page ...... 119 Figure 6-6: SearchAware v2 - Activities Summary Page ...... 120 Figure 6-7: SearchAware v2 – Architecture ...... 121 Figure 7-1: Simulated Work Tasks recommendations (Ingwersen and Järvelin, 2005, p. 253) ...... 135 Figure 7-2: Experiment protocol flow ...... 141 Figure 7-3: SearchAware v2 – Experiment Participants ...... 142 Figure 7-4: Partial IM log for Group 15; Trial 3; Condition 2; Task: MGT ...... 155 Figure 7-5: Partial IM log for Group 1; Trial 1; Condition 1; Task: BUS ...... 155

List of Tables

Table 2-1: Key terms in IS&R; based on (Ingwersen and Järvelin, 2005) with notes from (Baeza-Yates and Ribeiro-Neto, 1999; Manning, Raghavan and Schütze, 2008; Kelly, 2009; Ruthven, 2009; Blandford and Attfield, 2010)...... 32 Table 2-2: Summary of a selected set of major Information Seeking models ...... 37 Table 2-3: Explicit People-powered search categories. Summarised from (Burghardt, Heckner and Wolff, 2012) ...... 50 Table 2-4: Summary of main CIS systems in the literature in terms of Platform, Domain, and Design Focus, Research Methods and the main Awareness Cues...... 56 Table 2-5: Summary of Activity awareness facets definitions and protocol type (Carroll et al., 2009)...... 59 Table 2-6: Common Awareness Types (Liechti and Sumi, 2002) ...... 60 Table 4-1: CIS design guidelines applied to SearchAware v1; Guidelines based on (Shah, 2012, p. 90) ...... 89 Table 5-1: Descriptive Statistics for the Participants Activities ...... 106 Table 5-2: Group timelines followers for SearchAware v1...... 106 Table 5-3: Coded references descriptive statistics ...... 108 Table 5-4: Sample of the excerpts from reflective summaries by the participants, and the coded words and their categorization. The full list is in [Appendix 1]...... 108 Table 7-1: Engagement Measurement ...... 130 Table 7-2: Cognitive Load (Task Load) Measurement ...... 130 Table 7-3: Activity Awareness Measurement ...... 131 Table 7-4: Usability Measurement (PUEU-12) (Davis, 1989) ...... 132 Table 7-5: Hypothesis-Measurement Mapping ...... 133 Table 7-6: Experiment Summary ...... 137 Table 7-7: Experimental Trials Arrangement ...... 140 Table 7-8: Descriptive Statistics for unique queries across groups ...... 144 Table 7-9: Total Viewed Results - descriptive results ...... 144 Table 7-10: Abstract Viewed Results - descriptive results ...... 145 Table 7-11: Page Viewed Results - descriptive results ...... 145 Table 7-12: Rated Results - descriptive results ...... 145 Table 7-13: Redundant queried keyword - descriptive results ...... 146 Table 7-14: Engagement index descriptive results ...... 147 Table 7-15: Engagement index reliability check ...... 147

Table 7-16: Cognitive load measurement descriptive results ...... 148 Table 7-17: Cognitive load index reliability check...... 148 Table 7-18: Activity awareness index descriptive results ...... 149 Table 7-19: Activity awareness index reliability check ...... 149 Table 7-20: Usability measurement descriptive results ...... 150 Table 7-21: Usability Measurements ...... 150 Table 7-22: Summary of Hypothesis testing results ...... 151 Table 7-23: T-Test between Abstract Views and Page Views for all conditions ...... 152

Abstract

The intentional and explicit activity of searching for information using digital platforms between two or more persons is recognised as a major form of information seeking activity. This common collaborative activity had been overlooked in the design of most Search User Interfaces (SUIs). Collaborators have been pushed to adapt to a variety of workarounds to share search activities and results, and yet, it is a growing practice between groups due to the abundance of digital platforms and the availability of a wide range of online collaborative services.

Therefore, interdisciplinary research in the fields of Human-Computer Interaction (HCI), Computer- Supported Cooperative Work (CSCW) and Interactive Information Retrieval (IIR) have realised the importance of this shared activity across various professional and personal contexts. Current research in the specialised area of Collaborative Information Seeking (CIS), and the design and implementation of CIS systems particularly, highlights several factors that facilitate a seamless and effective collaborative search process and establishes new approaches to enhance this process. Awareness, a multidimensional concept widely used in this context, is coined as an essential factor in collaborative systems, and it is crucial for a successful collaborative information seeking activity.

This research, situated within the interdisciplinary field of CIS, investigated how the introduction of ‘awareness cues’ can be designed and implemented visually and contextually to best aid explicit collaborative information seeking. The cues are defined as a visual, non-disruptive, form of related activities awareness mechanism. The research demonstrates two novel search awareness approaches by displaying cues of collaborators activities during the search. These cues are aimed to provide visually and contextually appropriate and adequate awareness notification, thereby assisting the collaborators’ awareness of each other’s shared search activities with minimal overhead distraction.

These cues are designed and tested within a broader, functional porotype web app, named here as SearchAware. This mashup collaborative information seeking system aims to aid collaborators to search together for scholarly literature. To facilitate a naturalistic approach, SearchAware v1 utilised the search engine of the digital libraries of Mendeley’s crowdsourced database and Microsoft Academic Search. It also connected to Twitter as a platform to provide a timeline of the activities cues. In the second version, SearchAware v2, the awareness cues visual were designed and implemented to facilitate synchronous awareness of the collaborators’ activities within the search engine results page (SERP).

The research illustrated how these cues affected the collaborative search the productivity, interaction and interface usability of the collaborators. Each version was evaluated through a curated study that included a set of evaluative measurements. In v1, a formative user study in which students were asked to search for relevant literature for their group projects. For v2, a controlled experiment of information seeking simulated work tasks for postgraduate students and researchers was performed. A mixture of user and system based evaluations were applied to comprehend the awareness cues effect on the collaborative search experience.

Based on the results of these studies, the research concluded with implications for the design and implementation of an adequate and appropriate awareness cues in CIS systems.

Declaration

No portion of the work referred to in this thesis has been submitted in support of an application for another degree or qualification of this or any other university or other institute of learning.

i. The author of this thesis (including any appendices and/or schedules to this thesis) owns certain copyright or related rights in it (the “Copyright”) and s/he has given The University of Manchester certain rights to use such Copyright, including for administrative purposes. ii. Copies of this thesis, either in full or in extracts and whether in hard or electronic copy, may be made only in accordance with the Copyright, Designs and Patents Act 1988 (as amended) and regulations issued under it or, where appropriate, in accordance with licensing agreements which the University has from time to time. This page must form part of any such copies made. iii. The ownership of certain copyright, patents, designs, trade marks and other intellectual property (the “Intellectual Property”) and any reproductions of copyright works in the thesis, for example graphs and tables (“Reproductions”), which may be described in this thesis, may not be owned by the author and may be owned by third parties. Such Intellectual Property and Reproductions cannot and must not be made available for use without the prior written permission of the owner(s) of the relevant Intellectual Property and/or Reproductions. iv. Further information on the conditions under which disclosure, publication and commercialisation of this thesis, the Copyright and any Intellectual Property and/or Reproductions described in it may take place is available in the University IP Policy (see http://documents.manchester.ac.uk/DocuInfo.aspx?DocID=487), in any relevant Thesis restriction declarations deposited in the University Library, The University Library’s regulations (see http://www.manchester.ac.uk/library/aboutus/regulations) and in The University’s policy on Presentation of Theses

Dedication

To my parents, Mohsen and Najma, for their endless love and support and for standing by my side in this very lengthy journey

Acknowledgments

A lot of dear people supported by me through this lengthy journey. This is an appreciation to all their help.

First and foremost, I’d like to thank my main supervisors: Dr Oscar de Bruijn and Prof Andrew Howes for guiding me from my previous research topic through discussions to this research from its inception till its final implementation step. I couldn’t have achieved this work without their continuous support throughout. They have been, and continue to be, most supportive. I’d also thank my previous supervisor, Dr Víctor González for shaping my earlier ideas and supporting me all the way in my first year.

Special thanks to my examiners Dr Nadia Papamichail and Dr George Buchanan for their valuable discussion and useful comments in the viva and revisions to shape my thesis. I am extremely grateful for their valuable patience and support.

Extended thanks to my dear friends and colleagues that started MBS journey with me: Drs Guilherme, Liwei and Sahar, they’ve made this journey a memorable one. This thanks extends to Drs Abdelaziz, Amir, Khalil, Mahmood, Mohammed, and Yusuf for their constant help and encouragement through this lengthy journey. Thanks to my friends in Bahrain and my supportive colleagues in the Department of Information Systems at the University of Bahrain for their support.

Thanks to all the participants in my user studies and experiment, their contribution to this research made it possible and is highly valued.

Finally, thanks to my parents for their endless love and support and having faith in me all the way through this very lengthy journey.

Thank you all!

List of Abbreviations

• API Application Programming Interfaces • CIB Collaborative Information Behaviour • CIR Collaborative Information Retrieval • CIS Collaborative Information Seeking (or Searching) • CS Computer Science • CMC Computer-Mediated Communication • CSCL Computer-Supported Collaborative Learning • CSCW Computer-Supported Cooperative Work • DoL Division of Labour • DSR Design Science Research • HCI Computer-Supported Cooperative Work • IIR Interactive Information Retrieval • IM Instant Messaging • IR Information Retrieval • IS Information Seeking • IS&R Information Seeking and Retrieval • LIS Library and Information Science • SERP Search Engine Results Page • SNS Social Network Service (Site) • SPA Single Page Application • SUI Search User Interface • TREC Text REtrieval Conference • UI User Interface • URL Uniform Resource Locator • UX User Experience • WWW World Wide Web Word Count

51500 (Approx.)

Chapter 1. Introduction

This chapter introduces the research topic and is structured as follows: Section [1.1] presents the general research domain and narrows down to the specific research topic. Section [1.2] describes the significance of this research and identifies the literature gap where the research can contribute to the domain knowledge. Following that, section [1.3] presents the main research question, articulates the research objective, and lays out the accompanying research process. The chapter ends with a final section [1.4] that summarises the rest of this thesis chapters’ content.

1.1 Research Domain

The human activity of searching for information on digital platforms has been a primary focus of information technology related studies since the digital revolution in the late 1950s and continues to be at the forefront of the human-computer interaction with information. According to a 2012 Pew Research report on Search Engine Use, information search represents a primary usage on the World Wide Web (WWW) since its inception in the early 1990s, just below email usage (Purcell, Brenner and Rainie, 2012). In 2013, general web search was ranked as the fourth most time-spending online activity in the US, behind social networking, email, and online video (Richter, 2013). Prominent global search engines like Google, Yahoo and Bing along with their international portals and local search engines, like China’s Baidu and Russia’s Yandex, are amongst the top twenty most visited sites on the Internet globally according to Alexa web traffic data and analytics firm (Alexa, 2016). Therefore, the search process is such a primary and basic task for any web user, and continues to be so.

Search is almost never an end in itself, nor is it an activity that people often plan. It tends to be an invisible form of interaction with information that is constantly shaping how people deal with information in general (Blandford and Attfield, 2010). On the web, the information seeker typically wants the find some information or investigate a topic in the form of textual or multimedia content to satisfy a need or curiosity, and starts interacting with their knowledge to discover new information or recall forgotten information.

This primary need to search has been recognised by major web browser vendors, and it is illustrated in their design and implantation of the search feature in their core products. It is deemed so crucial to the extent that all modern browsers include the search feature in the forefront of their current iterations. It is typically implemented as a simple text entry box in the browser’s main toolbar, which utilises a search engine that starts working as it is being populated with characters. Moreover, web search is now built in the URL address bar of most modern web browsers. This was demonstrated in

Chapter 1 Introduction

Google Chrome search box input field, technically known as Omnibox, in which the text entry search box in the browser serves both as a keyword-based search text box and a standard URL address text box. The design proved to be successful, as common web browsers such as Mozilla Firefox and Microsoft Edge also replicated the same design and implementation. This trend also extends to modern websites and webapps, where the search service is considered a permanent fixture in most pf these apps as well as other desktop and mobile interfaces. This service allows the user to search, not only the device’s settings and content, but also extends to particular websites, such as references and specialised databases and the web. This is the case with prominent operating systems such Apple iOS and Mac OS spotlight search and Google Now launcher. More recently, web search is combined with intelligent digital voice assistants, such as Apple’s Siri, Microsoft’s Cortana, and Amazon Echo’s Alexa, which is available is several form-factor devices ranging from desktop computers and mobile to external speakers and car entertainment systems.

However, the process of finding the appropriate resources across the web’s exponentially growing content can be a tedious task due to the “deluge of data” as The Economist coined it (Garcia-Molina, Koutrika and Parameswaran, 2011). Paradoxically, this information overload is due to the abundance of various search engines and mechanisms, as well as the way search results are displayed or ordered on the device interface, can additionally lead to confusion amongst searchers, despite the apparent simplicity of search process itself. Moville and Callender state:

Unfortunately, [search] is also the source of endless frustration. Search is the worst usability problem on the Web. It’s held that title for many years. We find too many results or too few, and most regular folks don’t know where to search, or how. (Morville and Callender, 2010, p. 1)

The above statement is not justified by the lack of the search resources or slow retrieval speed, but fundamentally from the process information seekers comprehend the search results. In particular, this problem is not a result of hard to find information, but, as Endsley and Jones affirm, because:

there is a huge gap between the tons of data being produced and disseminated, and our ability to find the bits that are needed and process them together with the other bits to arrive at the actual needed information. That is, it seems to be even harder to find out what we really want or need to know. (Endsley and Jones, 2011, p. 5)

Chapter 1 Introduction

To overcome this issue of information deluge, is the of information seekers searching collaboratively for information together as a group (Attfield, Blandford and Makri, 2010), which is identified the general domain of this research. It occurs when a group of information seekers typically have one or more shared information goals with common examples of this activity being observed in small groups of people of two to five members between family members, study and work colleagues, and friends in an variety of personal and professional contexts (Sonnenwald et al., 1999; Foster, 2007). This activity may contain one or more several sub activities such as asking for help and guidance on search and dividing up search tasks, sharing searched keywords, and highlighted search result links, in addition to the search process itself (Morris and Teevan, 2009). These activities are common in different place setups and times, e.g. searching together at common times across multiple locations or at different locations at the same times (Golovchinsky, Pickens and Back, 2009). Within the academic context, these activities are discussed and studied under the different terms, but commonly the umbrella term Collaborative Information Seeking (CIS), which focuses on the approaches, methods and tools used in collaboration in information search.

Even though collaborative information search might seem counterintuitive due to the abundance of information seeking interfaces and tools that cater for solitary search (Wilson and Schraefel, 2010), it is observed and supported by empirical research in a variety of life and work contexts (Foster, 2007). Researchers assert that while the individual or solitary form of search might seem like the norm, search has always been a group activity as common as it is an individual activity (Karamuftuoglu, 1998) (Talja and Hansen, 2006; Morris, 2013). Therefore, researchers promote that the search for knowledge is a collaborative activity, as it, eventually, extends to knowledge production. As Karamuftuoglu states:

“the fundamental intellectual problems of information retrieval are the production and consumption of knowledge. Knowledge production is fundamentally a collaborative labor, which is deeply embedded in the practices of a community of participants constituting a domain.” (Karamuftuoglu, 1998, p. 1070)

The knowledge production aspect stressed in the quote above is in terms of discovery and filtering systems that rely on other people who formulate, locate and evaluate relevant resources and are willing to share the discovered information. In the modern era, this collaborative activity continues to be reshaped by the widespread availability of digital devices and increased connectivity in the workplace, in educational and academic environments, at the homes, and increasingly on the go. The growth of the ‘searching together’ phenomena is researched and examined via different methods and

Chapter 1 Introduction contexts. The main methods of research are detailed next, with examples of CIS research in various contexts.

From the perspective of this research, two general categories of modern interactive CIS research can be identified: the first category of research typically includes a developed software artefact being designed and implemented specifically for aiding CIS. The research in this category is typically preceded by an investigatory study and followed by an evaluation. The second category includes studies of collaborations in information seeking using available communication channels including traditional communication channels and through Computer-Mediated Communication (CMC) channels such as email, instant messaging and audio/video chat. These may also include other specific software artefacts such as websites or dedicated applications. Moreover, studies in this category include other forms of systems and digital services that are not necessarily dedicated for communication nor information seeking, but adopted to be used in CIS activities. Examples of such include social bookmarking sites and web annotations. As these categories highlight the important role collaboration plays in the search process on digital platforms, they are discussed further below.

The first category focused on studies that concentrate on the development of an artefact to support CIS started in the early days using digital libraries and databases but now includes online database and the WWW. Early web-based artefacts included additional browsing tools and extensions to support collaborative web browsing. One of the earliest browser extensions that supported the sharing of selected web pages with other collaborators is W4 which supported annotations on shared web pages and pinning of these pages to a whiteboard or brainstorm workspace (Gianoutsos and Grundy, 1996). Other similar early tools include ARIADNE (Twidale, Nichols and Paice, 1997; Twidale and Nichols, 1998) and Web4Groups (Alton-Scheidl, Bockle and Schmutzer, 1997). It is important to highlight that these software artefacts are focused on the explicit form of collaborative information seeking, i.e. that the search collaborators are aware that their search is in collaboration with a set of identified group members. Moreover, these tools are typically interactive in nature as their focus is on the interface level of the search and can also include backend algorithmic search improvements (Smyth, Balfe, Boydell, et al., 2005).

This trend to support collaborative information seeking on the general web and other shared resources or continues with sophisticated and newer prototypes. These include the browser add-on SearchTogether (Morris and Horvitz, 2007), and its improved standalone version, CoSense (Paul and Morris, 2009, 2011). These interface-based interactive search systems facilitate remote collaboration and communication through multiple features that enhance the awareness of what other group members are searching and bookmarking, amongst other features. Another prominent example is the

Chapter 1 Introduction browser toolbar Coagmento which evolved into the full web application (web app) Coagmento v3 (Shah and Marchionini, 2010; Shah, 2013b). Like SearchTogether, Coagmento offers various features to support CIS, mainly by having a shared workspace of collected results. Other examples include dedicated web apps such ResultsSpace (Capra, Chen, Hawthorne and Arguello, 2012), Querium (Golovchinsky, Dunnigan and Diriye, 2012), and CollabSearch (Yue et al., 2014). These developed artefacts, along with several other, represent the core literature for this research and are critically examined in the next chapter [Chapter 2].

Moreover, there is a growing number of focused studies that expand the functionality systems listed above and explores further aspects and wider contexts of CIS. This specific subtype of systems studies includes an evaluative study of the interface and interactions of SearchTogether (Wilson and schraefel, 2010), the use of Coagmento for daily task routines (Kelly and Payne, 2014), and a study about happiness in collaborative information seeking (González-Ibáñez, Shah and Córdova-Rubio, 2011).

Other artefacts, focusing on collaborative search, range in application and specialisation. These typically are dedicated systems for specialised digital libraries and for specific contexts. Such prototypes include MUST, a system to search the PubMed search engine medical publications library (Reddy, Jansen and Krishnappa, 2009), an Electronic Health Records system (EHR) for patient medical records (Zheng, Mei and Hanauer, 2011), video libraries (Villa and Jose, 2012), scholarly digital libraries (Böhm, Klas and Hemmje, 2014a), explicit collaborative question and answer (Filho, Olson and de Geus, 2010), and technical support material (Albar, 2015). Furthermore, a subfield of CIS studies focusing web search on larger tabletop computers have contributed to this research category, particularly in understanding co-located real time search. These include WeSearch (Morris, Lombardo and Wigdor, 2010), Físchlár-DT (Smeaton, Lee, Foley and McGivney, 2006) and TwisterSearch (Rädle, Jetter and Reiterer, 2013).

The second category of CIS research include surveys, case and diary studies, ethnographic studies, in addition to literature reviews. Prominent CIS surveys such as a 2007 survey of tech-savvy knowledge workers found that 53.4% of them have cooperated to search the web for information about other people for a variety of tasks (Morris, 2008). The survey was re-administered in 2011 to a wider range of respondents that 65.3% of those surveyed have searched as a group together, albeit from a wider spectrum of professions, engaged in a collaborative search with colleagues and friends at least once on a weekly basis (Morris, 2013).

Another survey investigated the social aspects of collaborative search through a questionnaire distributed to 150 users from Amazon Mechanical Trunk, a crowdsourcing system. In that context,

Chapter 1 Introduction social interactions with others, whether they are directly related to the information need or not, were found to play a fundamental role throughout the search process. For over a third of the respondents, the interactions occurred both before and after their most recent web searching behaviour and have occurred during the search process for the other two-thirds (Evans and Chi, 2008).

CIS Survey studies range in context. For example, focusing on the work-related task, as Karunakaran and Reddy survey of CIS practices in organisations uncovered four types of barriers for CIS: individual, team-level barriers, organisational and technological barriers. (Karunakaran and Reddy, 2012). Meanwhile, in a study focusing on leisure and recreational activities, the survey used found that 87% of the respondents search in groups of 2 to 3 members for going out to places of interests such as restaurants and hotels (Aldosari et al., 2016).

Moreover, searching for information as a group was studied by other forms of research such as case study research, e.g. in post-graduate research assignment (Hyldegård, 2009), and by using observations and diary studies in Swedish patent offices (Hansen and Järvelin, 2005). Another study used interviews and ethnographic studies such as shadowing workers in a US hospital (Reddy and Dourish, 2002), which in subsequent study “uncovered the central role that collaboration plays in information seeking and retrieval activities” (Reddy, Jansen and Spence, 2010, p. 80). Additionally, this category of research includes studies that are used to uncover potential collaborative aspects in specific domains. An example is the study of collaborative web search for the visually impaired, which emphasised on supporting awareness of search key terms and results (Al-Thani, Stockman and Tombros, 2015).

Other studies include commercial systems which were not necessarily devised for CIS activities but are examined to be used as such. For example, social bookmarking and tagging sites such as del.icio.us1, which are found to “helps users filter information at the most aggregate level” (Lee, 2006, p. 194), yet they did not particularly offer search nor cater for explicit form of collaborative search.

While there are attempts to create commercial CIS web apps and platforms, most have not yet achieved mainstream use (Morris, 2013). Microsoft So.Cl2 (Social) attracted some attention early in 2012 but had since been shifted into an aggregated image sharing network. Google’s SideWiki and SearchWiki were two initiatives by the search giant to address collaboration in its search engine interface. The former allowed users to read and contribute helpful information next to any webpage

1 Delicious (http://del.icio.us) 2 So.Cl (http://www.so.cl)

Chapter 1 Introduction through the SideWiki that appears as a browser sidebar (Paul, 2009). The latter added a way to customise search by re-ranking, deleting, adding, and commenting on search results which a dedicated search group can view (Dupont, 2008). However, Google has since abandoned both projects. Another example is Diigo3, an enhanced social bookmarking with the ability to annotate web page and create group workspaces of bookmarked pages. This web app was investigated in a study to understand how CIS tasks as such travel planning and house shopping are carried out using existing solutions. Kelly and Payne's study found Diigo adequate for such tasks with participants appropriating its features to serve their collaboration. They identified the key strategies employed by the participants and suggested features that could enhance the overall collaborative search experience, such as results reordering and resolving the overwhelming history logs of the group members (Kelly and Payne, 2014).

In summary, the two categories of studies, developmental studies and other forms of studies, imply a surging interest in interactive CIS systems. However, none of the dedicated CIS systems had achieved mainstream use nor had they improved in their features beyond academic research scope, with the noted exception of Coagmento being released as a free product4 with an accompanying mobile app in early 2015. Moreover, the interest amongst major search engines seems to have faded out. This gap in the limited availability of CIS system leaves the field for CIS research and studies with an opportunity to conduct further investigation. The importance of further focus in CIS research and opportunity to contribute in this field is presented next.

1.2 Significance and Opportunity

This section identifies the importance of the research and the literature gap, which is followed by the identification of the research opportunity and the contribution to the knowledge domain. Therefore, it will articulate the novelty of this research and its anticipated importance and outcome.

While the information gap, stated in the previous section, can be addressed by adding collaboration to assist information seekers to expand and process the search results, the opposite is also true, i.e. collaboration had also been found to lead a shared information seeking process (Shah, 2012, sec. 1.2.2). This is justified by reasons such as shared interest in a topic or a group work task or social interests such a planning family trip or a deciding on a house or a car purchase (Morris and Teevan, 2009, chap. 6). The interwoven of collaboration and information seeking activities on digital platforms is, as stated above, the domain of this research. The statistics and surveys presented above highlight

3 Diigo (https://www.diigo.com) 4 Coagmento (http://www.coagmento.org)

Chapter 1 Introduction the growing use of collaborative information seeking activities in various contexts, and the research interest, of various domains, lays a good foundation for this field. These, coupled with a surge of technological advances in interfaces and the availability of high-speed connectivity, present an ample opportunity to explore and contribute further to the field of CIS.

There are obvious advantages for modern day interconnected computers and mobile devices provide over manual information sources such as books and older devices. Moreover, these computers are equipped with the capability to provide large and clear screens, and can handle complex interface controls and widgets that are increasingly responsive to touch and even voice control. These serve the information needs of their users in a various means. However, catering to the various needs of information interaction of different users working together as a team is quite complex. Even though collaboration in search, ‘makes [collaborators] smarter’ (Evans, Kairam and Pirolli, 2010), it requires the understanding of the search and collaboration processes and strategies of people. Furthermore, it needs the proper interface and backend to adequately aid the search process for collaboration to provide the best fit and experience for the group of information seekers searching together.

Both the interface and backend challenges have been addressed in previous research, as will be discussed later. The research in these challenges facilitates one of the important research methods that have the potential to contribute to the understanding and aiding of collaborative search. This research type emanates from work on the design and implementation of software artefacts.

Foster’s comprehensive review of CIS highlights that:

Research evaluating the impact of such designs on collaborative information seeking, as a whole or on specific processes within it (such as, e.g., relevance judgments), remains rare. (Foster, 2007, p. 338)

Even though several leading studies have been published since then in the field of CIS, the explicit form of collaborative search, observed in real-life contexts such as educational and information workspaces environments, continues to be underexplored in terms of the various forms of search interaction and collaborative interaction in a group search. There are tremendous challenges in the coordination and communication to overcome in a collaborative work, yet it also presents great opportunities to study a multitude of people interacting with each other and with information, as Hearst states:

There is some friction inherent in switching from a standard search tool to another tool for collaborative search.

Chapter 1 Introduction

To date, people just do not seem willing to move from a standard search tool to another tool for collaborative search. The reason for this may be that there must be enough additional value as yet in the tools offered, and/or they may not yet be easy enough to use, to justify using a specialized tool. (Hearst, 2014, p. 59)

Therefore, supporting explicit collaborative information seeking requires further understanding of this phenomenon. These include, as mentioned, issues such as the search interface, and an understanding of the search and collaboration process itself. While advances in some of these aspects are being achieved at a rapid pace, particularly in the backend algorithmic aspect, there is a notable lack of commercial systems to support the explicit form of CIS, and particularly for the interface aspect, as Shah confirms:

A deeper understanding is needed to know the relative merits of different components of CIS interfaces. (Shah, 2015, p. 15).

In fact, attempting to cater for interactions and activities that occur between people and information is often quite challenging from a design and implementation perspective, and the research into aiding the explicit and intentional collaborative search interface is still considered limited (Marchionini, 2008; Blandford and Attfield, 2010; Morris, 2013; Hearst, 2014).

The literature suggests that one of the key aspects to facilitate explicit CIS is through offering ample and consistent awareness during the collaboration (Morris and Horvitz, 2007; Shah and Marchionini, 2010). Awareness is an important concept in collaborative studies, and is defined as “The quality or state of being aware, consciousness; the condition of being aware (of something or that something is)”, and also as the “knowledge or perception of a situation or fact” (‘awareness’, 2015) in a group context. In the context of CIS, awareness refers to the information seeker being aware of the issues surrounding

“the task and its context, past and present actions, and various attributes of the information objects and the system” (Shah, 2013b, p. 1124).

Several types of awareness are identified in CIS and related studies. For example, basic awareness can be achieved through the use of traditional CMC channels. These techniques range from basic email or instant messaging applications whereby collaborators exchange their search terms, results or strategy (Morris, 2008) all the way to a live telecast of collaborators browser screens, e.g. CoFox (Perez, Leelanupab and Jose, 2012).

Chapter 1 Introduction

Although awareness mechanisms are facilitated in most sophisticated systems, not only by communication channels, and as equally important as this research will argue, by using some other forms of explicit visual mechanisms. These mechanisms provide the information about collaborators’ search activities like searched keywords, viewed results, bookmarks or favourites, tags, and ranking of search to the other group members. The visual awareness mechanisms are displayed using a mixture of static and interactive approaches, such as logs and timelines, collaborative workspaces, notifications, and rank rating. These were determined to be key components to help achieve a successful CIS task.

Therefore, awareness is singled out as an essential aspect of CIS. Its importance, particularly within specific contexts, is iterated in more recent literature, as Shah states:

we need to not only create a support system that connects the collaborators and makes it easy for them to communicate, but also provide appropriate and adequate awareness. Such requirements and specifications may vary from domain to domain. (Shah, 2015, p. 14).

The above quote highlights, the need to specify it with a domain for the collaborative search in addition to the stressing the importance of awareness. This presents another opportunity for the use of specific fields, rather than focusing on a general search domain. This need for a specific context focus is also remarked in Kelly and Payne’s concluding study on the use of CIS systems for non-work related tasks, which asserts:

More specialised systems could be developed to provide targeted support for specific tasks. (Kelly and Payne, 2014, p. 818)

Moreover, this is elaborated in Hearst’s 2014 article “What's Missing from Collaborative Search?”, published in IEEE’s Computer magazine, where she provides an example of a PhD group searching about a topic thoroughly using current CIS system, which needs to be specific for this domain of interest:

there are several fundamental functionalities that are missing from current collaborative search tools. For one, the tool should be aware both of what has already been stashed away in the bibliography and what has been viewed by anyone in the group of searchers. (Hearst, 2014, p. 59)

Chapter 1 Introduction

She also highlights some examples of providing the above features and functionalities, also stressing the need for naturalistic studies of CIS systems:

but assessments of these tools are done in laboratories in which users explicitly set up their tasks at fixed starting times, over fixed document collections. (Hearst, 2014, p. 60)

This also presents an opportunity to situate the research within a more naturalistic context, whereby collaborators feel more natural and perform normally compared to simulated contexts.

Based on the identified gaps above, this research is motivated by for further investigation to aid explicit collaborative information seeking from an interaction perspective. This will be through the design and implementation of awareness mechanism in dedicated IT artefact with a focus on awareness as a key aspect.

By studying the use of novel suggested approaches of visual, non-distributive, automated awareness ‘cues’ in the proposed artefact using proven evaluation methods in naturalistic context, the research aims to understand the mechanism to which awareness can be best designed and implemented in an appropriate and adequate manner.

1.3 Research Overview

The research communities in the field of Information Science (IS), and particularly within the Information Seeking and Retrieval (IS&R) field in IS, provide a strong foundation to understand information seeking behaviours with a plentiful set of theories and models to understand the search process, and design the system to cater for the information seeker. Moreover, a typical search system includes a front-end interactive interface, in which the field of Human Computer Interaction (HCI) lays out the theoretical and practical foundations to design proper interfaces. This intersection of HCI and IS&R is also referred to as Interactive Information Retrieval (IIR), in which users are typically studied along with their interactions with systems and information (Kelly, 2009).

Moreover, the growth of communication and collaboration on digital platforms produces further complexities in the context of interactions in IS&R systems. The Computer-Supported Cooperative Work (CSCW) research community provides an ample set of studies of the communication and collaboration between people in this digital era in various contexts personally and professionally (Stahl, 2011). It is focused on groups of people and how technologies can people work, particularly

Chapter 1 Introduction with the design and implementation of artefacts, and the assessment of how these or other technologies can assist people to work together in synergy.

The Collaborative Information Seeking (CIS) field can be situated at the intersection of the IS&R and CSCW fields (Fernández-Luna et al., 2010). Research in this field is concerned with catering for collaborative information seeking, and contains multiple aspects and perspectives, due to the multitude of the many common uses of this term and similar terms (Shah, 2012, p. 25). For the explicit form of CIS, one important aspect is the design of the user interface for such a process. Here, the complexity of designing interfaces for information seeking escalates as social interactions between more than single person interacting with information have to be catered for in the collaborative search process (Crumlish and Malone, 2009).

Considering all the aforementioned fields, the research area of this thesis is the study of people interaction with software artefacts to search, as a group, for information in an explicit and declarative manner of relevant information to achieve a common understanding on issues sought after. In specific, this research focuses on the design and implementation of a software artefact that caters for collaborators searching for information together. Of interest to this research is the aspect of design and implementation of the awareness mechanism to aid search interface to aid the information seekers in a collaborative information seeking process.

In the next subsection, the main research question is presented, followed by the research objective and the accompanying research design process.

1.3.1 Research Question Based on the stated gap in the literature and the research overview. The research question is formed as:

How can awareness cues be designed and implemented visually and contextually to aid explicit collaborative information seeking; and how can these cues affect the search interface usability?

There are two aspects to the research question, the first is exploratory in nature and aims to understand, from a practical perspective, how the awareness cues can be best designed to cater for the explicit form of collaborative information seeking in an appropriate and adequate manner. The second aspect of the research question is the evaluation process to measure the effect extent of the introduced cues on the search interface usability. Usability can be defined as “a quality attribute that assesses how easy user interfaces are to use and refers to methods for improving ease-of-use during

Chapter 1 Introduction the design process.” (Nielsen, 2012). Usability, which is an essential concept in HCI, is measured with an evaluation testing of how a piece of software is easy to use using multiple aspects such as effectiveness and efficiency and satisfaction. Therefore, as the cues are meant to improve the search process for the collaborators within a prototype, the research will measure the impact of the cues in the aspects of productivity and performance, and interactivity and usability.

The use of the word ‘cues’ in the question above is meant to signify an awareness mechanism of minimal disruption to the collaborators. It is to also used to stress that cues, unlike the main form of visual awareness alerts of popup notifications, are intended to ensure a seamless flow of the search process.

1.3.2 Research Objectives To address the research question, a set of research objectives are defined below:

1. Development of the research artefact and the introduction of proper and adequate awareness cues The research instrument will be a software artefact solution in the form of a functional web-based application, i.e. a web app, that is designed and implemented to integrate the SUI specifically for collaborative information seeking process. The web app should include the ability to integrate various curated sets of awareness cues. These awareness cues will be automatically generated in the form of a visual and non-distributive alerts of the collaborators’ search and other activities during the search process. These cues will be tested through the web app; therefore, the web app should allow experimental and evaluation measurements to be deployed.

2. User evaluative studies to measure the effectiveness and usability of the awareness cues As stated in the research question, the studies are evaluative in nature to understand the effect of the awareness cues of the collaborative search. These will be set in a naturalistic context to ensure that the study participants feel more natural and perform as normal than in simulated contexts. This means that the artefact needs to be connected to an appropriate search engine service to match the context of the study. Moreover, this software novelty depends on available digital libraries search engine and resources, rather than being tied to an artificial database. This extensibility feature in terms of being able to connect with most of the available search engines constitutes a unique aspect of this research.

Chapter 1 Introduction

3. Reproducibility of the artefact As the research methodology, presented in [ Chapter 3] will emphasise, the generation of knowledge is achieved through the iterative cycles of design and development of the artefact. This research will detail and justify the design process, the selection of implementation approach and technologies, the interface, middleware and backend architecture. These will allow the reproducibility of the key parts of the developed web app for further research. The conclusion will include implications for design recommendations based on the design approach and implementation process and results of the evaluation.

1.3.3 Research Design Process The design and implementation of experimental software to study and understand a phenomenon has always been a pillar of various technological fields to gain insight on it and contribute to the knowledge of that field. Furthermore, previous work had revealed several triggers and effects for CIS and tested various technologies in collaborative research. It further highlighted an extensive use of various forms of artefacts and digital communication (Hansen and Järvelin, 2005; Paul and Reddy, 2010). Nevertheless, further calls have been voiced for more technological research into collaborative search, particularly in interdisciplinary collaborative work (Foster, 2007).

The selected methodological approach for the research will be Design Science Research (DSR), which is well established in the field of Information Systems and includes several established renowned resources that detail its processes and methods (Hevner and Chatterjee, 2010; Gregor and Hevner, 2013). This approach stresses the rigour methodological approach for the design and evaluation of IT artefacts to essentially provide some contributions to knowledge. DSR is well established in the field of Information System and has been used in HCI studies. For CIS studies, current research does not explicitly mention a methodological approach; however, the use of an iterative and incremental cycle of design and implementation is almost universal for these studies. In that sense, DSR encapsulates an iterative approach in the design, development and evaluation steps that matches the studies methods.

Regarding the evaluation, study methods in CIS have been considered within the traditional context of IR (Information Retrieval), and more recently within IIR (Interactive Information Retrieval) fields (Kelly, 2009). The study objectives fit within the broader fields of HCI (Human-Computer Interaction) and CSCW (Computer-Supported Cooperative Work), as these fields take into account the collaborative and interactive nature from the user’s perspectives. Leading research on CIS using novel technologies in textual, multimedia and web search (Morris and Teevan, 2008; Fernández-Luna et al., 2010; Paul and Reddy, 2010) and activity awareness within academic communities (Collins et al., 2005;

Chapter 1 Introduction

Farooq et al., 2008) explored several methods for conducting studies from the perspectives of the above fields.

In specific, two studies are conducted: A user study and a laboratory experiment. Both studies have different, yet complementary, objectives. In addition, both forms of the study will entail quantitative and qualitative data collection with balanced emphasis to meet the two objectives set. By having two forms of studies, the research will cover both synchronous and asynchronous search activities (Golovchinsky, Qvarfordt and Pickens, 2009).

These studies have been critically examined to provide a set of study procedure for the investigated topic. These will include selected mixture of quantitative and qualitative HCI (Cairns and Cox, 2008), collaborative workspace studies and selected set of IIR (Kelly, 2009) research methods and measures. The proposed studies are aimed at exploring the direct effect of an automated sharing of three search- related activities between collaborators: searched keywords, viewed result hits and highlighted result hits. This flow of information between the collaborators will be represented via lightweight notifications in the form of short feeds initially and then in the form of graphical cues in the search results page. In the context of the research, awareness in CIS is considered a form of activity awareness, which has been identified as an important facilitator for sharing mutual information amongst collaborators (Carroll et al., 2003; Morris, 2008; Villa, Gildea and Jose, 2008; Paul and Reddy, 2010).

1.4 Thesis Structure

The thesis is organised as follows:

• In the next chapter, [Chapter 2], reviews the literature on information seeking and collaboration as the main two background fields in collaborative information seeking, followed by an analytical review of main CIS concepts and studies. • In [Chapter 3] the methodological approach for the research will be presented and based on that the research study approach and concepts will be presented. • In [3.3], the actual research detailed design and implementation of SearchAware v1, which is the first iteration of the research artefact. The chapter covers the main interface and functionality design considerations and justifications as well as the internal components. • In [Chapter 5] a description of the first user study of SearchAware v1 which is a naturalistic user evaluation and usability study in an academic module context is presented along with its results and discussion.

Chapter 1 Introduction

• In [Chapter 6] presents the second iteration of research artefact, SearchAware v2. Based on the result of user study, • In [Chapter 7] covers the controlled experiment for the improved version, which is systematically tested by with various manipulating of awareness and communication features. • In the final chapter, [Chapter 8], a conclusion of this research is presented. This includes the research summary, the contribution to knowledge, its limitations, and future work.

Chapter 2. Literature Review

Following the Research Domain overview presented in section [1.1] and the Significance and Opportunity section [1.2], this chapter presents a background synthesis of the literature on Collaborative Information Seeking (CIS), which is the main research domain. Two important domains define CIS, information seeking and collaboration, and both these topics are discussed in the first two sections.

Section [2.1] presents a review of the activity of human-centred information seeking and retrieval, and presents its definitions and models by focusing on the interface and interaction aspects of information seeking on digital platforms. Section [2.2] considers the collaboration aspect using the information and communication technologies through the focus on the field of CSCW (Computer- Supported Cooperative Work). Section [2.3] presents the main research domain of this thesis: Collaborative Information Seeking, critically examining its main definitions, taxonomies and models. Section [2.4] lists and details the identified aspects of CIS from empirical studies that are deemed necessary to achieve a successful collaborative search experience. The final section [2.5] covers the key studies in the design and implementation of CIS systems by focusing on the approaches used in the discussed research throughout this chapter in order to design and implement the awareness mechanisms.

2.1 Information Seeking and Retrieval (IS&R)

Human interaction with information in this digital era continues to be a fascinating research theme with a vast and interconnected set of issues to understand and tackle. It includes a plethora of definitions and research areas that shape and contribute to this interdisciplinary field. The first section starts by situating IS&R. In subsection [2.1.1] it critically examines the main definitions in the literature, by focusing on a subset of IS&R is Interactive Information Seeking. It will also emphasise the difference between seeking and searching in this context. Next, in subsection [2.1.2], a review of the Information Seeking models which provide the theoretical foundation for the IS&R is presented. The final subsection [2.1.3] focuses on a relevant aspect of this research, which is the design and implementation of Search User Interfaces (SUIs).

Chapter 2 Literature Review

2.1.1 Situating IS&R and Related Definitions The Information Science (IS) field provides an abundant wealth of research about dealing with information and usually includes closely related concepts, which are Information Seeking (IS) and Information Retrieval (IR), occasionally denoted together as IS&R.

Before these are defined in this context, a quick summary of the concept of information is needed. Ingwersen and Järvelin provide a dual perceptive on information as “the result of a transformation of a generator’s cognitive structures” and “being something which, when perceived, affects and transforms the recipient’s state of knowledge” (Ingwersen and Järvelin, 2005, p. 20). From a human cognitive perspective, the latter is relevant to this research. It is essential to understand these terms are usually coined within the holistic term Information Behaviour which describes the overall human activities of identifying information needs, searching and using or transferring the information across all forms of communication media and channels (Wilson, 2000).

Moreover, Ingwersen and Järvelin continue to define important processes related to the information behaviour and interaction from a cognitive point of view, which are summarised in the table below [Table 2-1]. These are reordered from the general term of Information Behaviour to the specific term Interactive Information Retrieval (IIR) with added notes from the. The terms below provide the required definitions and context for this research.

Term Definition Information Behavior The human behaviour dealing with generation, communication, use and other activities concerned with information, such as information seeking behaviour and interactive information retrieval. Information Retrieval The processes involved in representation, storage, searching, finding, filtering and presentation of potential information perceived relevant to a requirement of information desired by a human user in context.

Information Seeking The human information behaviour dealing with searching or seeking Behaviour information using information sources and (interactive) information retrieval systems.

Information Interaction The exchange between two or more cognitive actors in contexts of IS&R.

Interactive Information The interactive communication processes that occur during retrieval of Retrieval (IIR) information by involving all major participants in IS&R. In this field, users are typically studied along with their interactions with systems and information. IIR focuses on users’ behaviours and experiences and

Chapter 2 Literature Review

the interactions that occur between users, search systems and information.

As mentioned above in [Table 2-1], researchers use the term Interactive Information Retrieval (IIR) to describe the field of evaluating human performance aspects in information seeking systems. IIR can be thought of as in alignment with the traditional information retrieval system performance aspects of search engines e.g. (Borlund, 2003; Kelly, 2009; Ruthven, 2009). Kelly provides a useful conceptualization to situate IIR field within the system-focused end and the human-focused end, shown here in [Figure 2-1]. The latter focuses on experimental information behaviour, and information behaviour within IR system and context (Kelly, 2009, p. 10).

Figure 2-1: Research continuum for conceptualising IIR research (Kelly, 2009, p. 10)

One important distinction to be drawn here is that between Information Seeking and Information Searching. Information Seeking represents a wider scope of search covering activities beyond entering a keyword. Seeking includes the entire process: from the identification of an information need to resolving it. This is articulated by Gary Marchionini in his seminal text “Information Seeking in Electronic Environments” where he, also from a human-focus perspective, labels seeking as “the process where people purposely engage in order to change their state of knowledge”. Moreover, he states that search is the “behavioural manifestation of humans engaged in information seeking” (Marchionini, 1995, p. 5).

This distinction affected the term usage in other fields’ literature, where search is viewed as focused subset or stage of information seeking. For example, Wilson review of information seeking models situates information search as the most focused area in the general information behaviour, and

Chapter 2 Literature Review between both the general information behaviour and the detailed search behaviour [Figure 2-2]. He further elaborates that while seeking is the consequence of an information need, searching is focused primarily on the interactions between the user and the retrieval system (Wilson, 1999, p. 263). In a similar hierarchy, Bates (2002) puts searching as the most focused form of information seeking in her model, Modes of Information Seeking, shown in [Figure 2-3]. Bates describe search as an active mode, referring to an individual actively seeking information, and directed mode referring to an individual actively acquiring information, seeking activity which entails a more intended and focused form of seeking, with being aware as the most unintended form of information seeking.

Other authors do not subscribe to the same structure. For example, Fidel states the terms seeking, searching, surfing and browsing can be used interchangeably. It is the purpose of information obtained that distinguishes the process of acquiring information, not the methods of acquiring information nor the technology used (Fidel, 2012, p. 22).

Figure 2-2: A nested model of the information seeking Figure 2-3: Modes of Information Seeking and information searching research areas (Bates, 2002, p. 4) (Wilson, 1999, p. 263)

2.1.2 Information Seeking Models and Forms Models of information seeking, such as the ones described above in the figures [Figure 2-2] and [Figure 2-3], attempt to present a theoretical framework of the information seeking processes and activities. There are various ways to summarise the wealth of models in information seeking. In fact, the scope of these models varies from the different contexts, and to some extent the different fields they were established in. Models, by nature, do not all attempt to describe the same set of phenomena or activities as they can be considered complementary, not competing (Wilson, 1999). Additionally,

Chapter 2 Literature Review models do not directly imply a cause direct hypotheses testing mechanism, but rather simply provide an overview of the covered topic, from which it might suggest issues (Ingwersen and Järvelin, 2005).

Naturally, these several focal points and perspectives had produced numerous models that cover human-centred information seeking. Perhaps the most dominate point, related to this research, is its focus on the individual information seeking perspective. Two main reasons were quoted for the intrinsically individual focus of these models: firstly “the focus on the conventional pattern of interaction between a single user and technology” and secondly “[an] emphasis on individual, not on collaborative work” (Reddy and Jansen, 2008, p. 257).

There is an abundance of recent reviews to approach these models from a human cognition perspective, some of which are explored below following the thesis focus on the collaborative aspect of information seeking and retrieval. Nevertheless, as the bulk of these models focused solely on solitary or individual information seeking, several had been further developed for the collaborative context. This section will focus on the models that serve as a basis for some CIS models, or were extended to include CIS activities. These additions are reflected in the extended models for Collaborative Information Seeking detailed in [2.3.2].

The main reviews here come from four main literature reviews. Ingwersen and Järvelin provide a focused review of five main models (Ingwersen and Järvelin, 2005, sec. 3.1.2) which are used to provide their model as well. Additionally, Hearst groups several main models into a comprehensive set of six categories attempting to capture the “most commonly discussed” models of information seeking (Hearst, 2009, chap. 3). Russell-Rose and Tate regrouped several of the same models highlighted by Hearst into five major groups (Russell-Rose and Tate, 2012, chap. 2), and finally Karunakaran et al. summarises seven dominated models of collaborative information seeking (Karunakaran, Reddy and Spence, 2013).

These models are considered under the general category of cognitive models of information seeking. According to Hearst (2009, chap. 3), Norman’s duality model of the Gulf of Execution and the Gulf of Evaluation can be thought of as a way to describe the cognitive basis for these models. In this design- oriented perspective, when people use something, they try to understand how it operates, i.e. gulf of execution, and then they try to understand what have happened, i.e. the gulf of evaluation (Norman, 2013, p. 38). In this sense, the information seeker, willing to achieve a goal of getting relevant information, formulates an intention to seek information, then attempts to seek information by executing the actions necessary to achieve that goal, comprising the gulf of execution. The outcome of the execution is then evaluated, by analysing the results found at to verify that the goals have been

Chapter 2 Literature Review achieved, comprising the gulf of evaluation. The identification and formulation of information needs represent the goal and results to check if the information needs are satisfied is akin to the gulf of evaluation. A key aspect of this model is that it underpins the usefulness of the domain knowledge of the information seekers.

Nevertheless, it has been noted that it is hard to cover these models, even if the focus is on cognitive models, this is due to their attempt to cover many aspects of the information seeking and retrieval (Ingwersen and Järvelin, 2005, chap. 5). Therefore, the models summarised below in [Table 2-2] which follows the general categorization description of Hearst and Russell-Rose & Tate taxonomy. The summary will focus on prominent models that considered and extended as collaborative information seeking models, which are detailed later in section [2.3.2].

Model Summary (main references) (with added commentary) Information Search ISP is one of the most prominent models in information seeking. The Process (ISP) model incorporates cognitive and affective (emotional) aspect of the (Kuhlthau, 1991; information seeking process. It consists of 6 stages: work-task initiation, Kuhlthau and Tama, selection of topic, exploration, formulation, collection and presentation. 2001; Kuhlthau, Originally induced from students studying behaviours in libraries, it has Heinström and Todd, been found applicable in the digital environment, (Vakkari, Pennanen 2008) and Serola, 2003; Hyldegård, 2006). While some studies show partial stages of ISP do apply in collaborative information seeking processes (Blake and Pratt, 2006b). Others, however, found that it “does not fully comply with group members’ problem solving process and the involved information (seeking) behaviour members” (Hyldegård, 2009, p. 157).

The Integrated IS&R In this holistic cognitive framework, the information seekers are cognitive Research Framework actors acting in a social, organisational and cultural context. Their search process and interactions are viewed as cognitive transformation and (Ingwersen and influences through interfaces to information objects and other artefacts. Järvelin, 2005) This exchange occurs between the different actors (i.e. search collaborators) through direct interaction with the Interface, and indirectly with the Information Objects (knowledge representation, e.g. databases or web archives) and IT structures (e.g. search engines, algorithms).

The Berrypicking This four-part model consists of awareness, monitoring, browsing, and Model searching and is shown in [Figure 2 3]. The central concept in this model is that the search results, even from a poorly defined search, allows the (Bates, 1989, 2002) searcher to identify opportunities “berries” to gain new insight into the

Chapter 2 Literature Review

searched theme, thus allowing the search to be refined and redefined if needed. Bates argues that this model accommodates for variations in the search queries formulation as the search process progresses. That was shown to be applicable, also partially, in an empirical study of CIS activities of scientists in the medical context (Blake and Pratt, 2006a, 2006b) where the collaborators were found to exhibit the model four parts.

Sensemaking Information seeking model in this holistic methodology focuses on the information actors and their need for information in various contexts (Dervin, 1998) stemming from particular situations pinned with a specific space and time. To bridge the situations gaps (questions), information seekers will make sense of the information to reach outcomes (either it helps or it hindrances).

Sensemaking has been studied in the context of CIS through observational studies, for example in a hospital (Paul and Reddy, 2010) and through user interfaces and experimental CIS systems, such as CoSense (Paul and Morris, 2011).

Information Foraging Similar to the food foraging mechanisms in animals and living creatures, information seeking is considered as a strategic process. The theory (Pirolli and Card, presumes the information seekers attempt to increase their relevant 1999; Pirolli and Fu, information ‘intake’. The mains constructs in this model are: scent, which 2003; Chi and Pirolli, refers to how judgments are made about which information source to 2006) pursue and consume; diet, which is the selected intake of information; while patches refer to the fact that valuable and sought-after information are located sporadically (Harr, Wiberg and Whittaker, 2011). Pirolli explored the notion of social information foraging mathematical model in Wikipedia edits and social tagging and other studies and concluded “that so long as the diversity of agents increases with group size, then the size of a group increases the overall power of cooperative discovery” (Pirolli, 2009, p. 613).

Exploratory Search This form of search goes beyond the basic ‘lookup’ of information form to include a more holistic ‘learn’ and ‘investigate’ form of search, which is (Marchionini, 2006; about supporting the information seeker acquire the knowledge and be White et al., 2006; able to comprehend, compare and integrate the search. This, in turn, White and Roth, enables the seeker to synthesise and evaluate the learned knowledge. 2009) The web enables a more diverse and investigatory search with an unclear set of results, but in this model, the seeker learns more of about the searched topic and generate an information space (White et al., 2006). Thus, exploratory search “blends querying and browsing strategies” (Marchionini, 2006, p. 42). Exploratory Search is tied with interactive information retrieval in its focus on the interface aspects and is best

Chapter 2 Literature Review

demonstrated in artefacts designed for collaborative search systems, such as Cerchiamo (Golovchinsky et al., 2008) and Coagmento (Shah, 2010a).

Table 2-2: Summary of a selected set of major Information Seeking models

The models provide a plethora of perspectives on information seeking from a cognitive perspective and provide a foundation to understand the interactions of the information seeker from the search formulation to the results comprehension. The next section focuses on the enabler for this interaction, which is the Search User Interface (SUI).

2.1.3 Search User Interfaces (SUI) Typically, the search functionality is offered to the user through a front-end Search User Interface (SUI) which caters to the information search interaction between the user and the backend search engine that performs the actual search operation. The simplest form of search interaction with an SUI involves entering one or few keywords in a search text box (also known as keyword textbox), and after being processed by a search engine, a list of results - or hits - appear below - or a new page - that match the entered keyword. This form of interaction continues to be the basis of stranded SUIs till date.

The growth in search, and particularly web search, accompanied by the mature understanding of the human-centered information seeking, has generated a particular interest on SUIs as an essential part of any system interface. Since this interaction with information occurs mostly through interfaces, great emphasis on the design and implementation of the search interface should be considered. As the growth in information seeking advances, interest in SUI design and development has gained tremendous attention from various fields to attempt to be catered for. This is coupled with the developments in IIR studies and evaluation methods focusing on the searchers’ interaction with information, e.g. (Borlund, 2003; Kelly, 2009; Ruthven, 2009). Wilson identified six main disciplines that are essential to understand the complexities of the design and implementation of SUIs in systems, as displayed in [Figure 2-4]. The disciplines include three more fields in addition to the HCI, Information Seeking and Information Retrieval topics examined in this research, which ate the User Experience (UX), Graphic Design, and Library and Information Sciences (Wilson, 2011).

Chapter 2 Literature Review

Figure 2-4: Disciplines affecting the design of Search User Interfaces (Wilson, 2011, p. 10)

The complexity of SUIs intertwined fields is reflected in its design complexity beyond the keyword textbox. To systematically understand the design of SUI, Hearst proposed a detailed list of guidelines that is beneficial for the practice of the SUI design (Hearst, 2009, sec. 1.12). That list will be used in the design and implementation of the research instrument used here in form of a software artefact, as described later in [4.1.2]. Additionally, Wilson provides a 4-type framework to approach the design of SUIs by focusing on the visual and functional features (Wilson, 2011, chap. 4). The main features of the framework, Input, Control, Informational and Personalisable, provide a good starting point for approaching SUI design and are discussed below.

2.1.3.1 Input Input refers to the method the searcher would use to indicate the desired information to be located. These include the classical textbox to enter the keywords or search terms and any enhancements to this textbox. In most SUIs, the basic textual input of keywords is the de facto method, but other techniques such as Query-by-Example (QBE) are offered by some database systems such as Microsoft Access. QBE has been extended to multimedia forms (Flickner et al., 1995) and is used for content- based image retrieval (CBIR) such as Google Images, which searches using a sample image. QBE also is available for audio-based search that searches for files based on a sample audio, which are popularly used voice-recognition in music matching apps such as Shazam.

For textboxes, techniques in populating the search input box with keywords can be of great help to users during the search process, such as autocomplete and search operators (White and Marchionini,

Chapter 2 Literature Review

2007). Automated Query Expansion and Interactive Query Expansion and Real Time Query expansions to populate the search box at the start of typing and through a list or during typing, respectively. Currently, most users expect input boxes on search interfaces to provide auto-suggestion of keywords to match most major web search engines (Ruotsalo et al., 2014).

2.1.3.2 Control Control features help searchers to modify, sort, refine, restrict, or expand their results. These include features such as the pagination controls to specify the results to appear per page or filter controls to display only a subset of the results based on a set of provided criteria, and many other combinations of controls.

For example, In Google’s search page, control features include limiting the results to include related categories such as Images, Videos, Maps and the likes which are available after the search is performed using the landing search page. Further refinements are typically available in the advanced search menus.

2.1.3.3 Informational These features focus on how results are organised and displayed, and usually appear on the Search Engine Results Page (SERP). The default form for displaying results is a vertical list of the results. Moreover, the order of the results is a key factor of the results. Mainstream web search engines order these by the calculated relevance of each result to the keyword(s) and any the use of any explicit control operators indicated. Assessing relevance is typically based on the algorithms of the backend search engine and is determined by a multitude of factors and perspectives, mainly those between cognitive and system focused (Hjørland, 2009).

Each single result within a results listing is called a search hit and any summary information related to the particular result is referred to as the surrogate (Marchionini and White, 2007; Hearst, 2009, sec. 5.1). Typical information can include the number of results and the number of pages that include the results. Moreover, these features can include summaries and previews of the results which were found to increase the ease of search and relevancy (Haas et al., 2011).

Variations within these Information features were found to impact the search experience. For example, an eye tracking study of the hit surrogate length found that relatively lengthier summaries significantly improved performance for information tasks, e.g. a precise fact, however it degraded performance for navigational tasks such as locating a specific site or a page (Cutrell and Guan, 2007). Therefore, from an interface and interactive perspective, Informational features should balance between the amount of the provided features and the customization of the available features.

Chapter 2 Literature Review

2.1.3.4 Personalisable To personalise the search experience to the user, parts of the SUI / SERP are shaped by the searcher’s previous actions. These may include recommendations, recent history and auto-complete based on previously typed keyword. Personalisation can be based on implicit relevance, such as previous search history, or explicit preferences as part of the control settings, as mentioned above in [2.1.3.2], which include a range of specifications such as a timestamp or document size (Hearst, 2009, chap. 9).

Personalisation in SUI is typically indicated by showing searchers their previous viewing history in a website. Beyond the different colour of a previously visited link that is offered by the browser, sites incorporate several sections e.g. such as ‘Recently Searched’ or ‘Recently Viewed’ which aggregates the user search and user browse history.

2.2 Collaboration

Collaboration, as a process, includes several meanings and can have several forms of people working together to different degrees, and continues to be “essential for solving messes” in today’s complex challenges (Denning and Yaholkovsky, 2008). It involves a concentred set of actions and activities by people in different roles, the practices they have applied, and the artefacts they have used attempting to achieve a shared set of goals (Schmidt, 2011, chap. 1).

Moreover, collaboration has been studied in major fields such as social sciences, psychology, organisational studies and information technology (Grudin and Poltrock, 2012). With regards to the latter field, the increasing availability of connected digital devices meant that collaboration between people through technological channels occurs in a plethora of work and life contexts, and is established in all aspects of modern day workplaces (London, 1995; Olson and Olson, 2013). Before delving into the design and implementation of collaborative systems, it is crucial to think about the social aspect of collaboration, as Grudin and Poltrock state:

Spatial and temporal distinctions retain technical and behavioral implications, but as conventions for handling digital capabilities come into place, those distinctions are less central. The social behaviors that have governed groups and organizations for thousands of years again rise to prominence. (Grudin and Poltrock, 2012, p. 29)

Chapter 2 Literature Review

Therefore, the behaviours of people are crucial to understand how to design and implement collaborative systems, and as Schmidt and Bannon state:

“understand cooperative work practices with a view to developing adequate computational technologies to facilitate cooperative work, mediate communication, and support the regulation of coordinative practices” (Schmidt and Bannon, 2013).

These behaviours associated with collaboration are apparent in the previous quote as communicate, cooperative, and coordinative. Collaboration usually comes with little, or even no distinction, from terms such as coordination and cooperation, which tends to be used interchangeably on occasion. All of which converge towards an understanding that collaboration entails people working together and utilising digital devices and platforms towards achieving common goals. The distinction between these terms and how the converge towards achieving collaboration is explained in the next subsection.

2.2.1 Processes of Collaboration Considering collaboration as a process to be achieved by collaborators, Gary describes it as “a process through which parties who see different aspects of a problem can constructively explore their differences and search for solutions that go beyond their own limited vision of what is possible” (Gray, 1989, p. 5). Taylor-Powell et al. expand this definition by stretching the collaboration processes, named as the 5-C model, to five different processes to achieve a collaborative state, as shown in [Figure 2-5 (right)] (Taylor-Powell, Rossing and Geran, 1998). An alternative model, the 3-C model, depicts a cyclic view of collaboration as shown in [Figure 2-5 (left)]. That model was developed from the authors’ experience in the development of AulaNet project, a learning management system being used in some Brazilian universities which includes synchronous and asynchronous discussion boards and instant messaging features (Fuks et al., 2008). The 3-C model, devised from their work, focuses on awareness as a key component for a successful collaboration in which all the three processes of communication, cooperation, and coordination should foster. Moreover, Fuks et al. assert that addressing these three process will facilitate the design and implementation of groupware. As highlighted in the introductory chapter, Awareness is an essential concept to the research which is discussed in details in [2.4.1] and a focused section on the awareness cues used by CIS systems is presented in [2.5].

Chapter 2 Literature Review

Figure 2-5: Two collaboration models: 5-C (Taylor-Powell, Rossing and Geran, 1998, p. 4) right, 3-C collaboration model (Fuks et al., 2008, p. 638)

Another model, inspired by and named as the Taylor-Powell model, is the 5-C model of Collaboration (Shah, 2013b) with an emphasis that each inner set is essential to support its outer set. In this model, shown in [Figure 2-6] below, the stages are important to consider in the design and implementation of CSCW, as the achievement of each process is crucial to the next process.

Figure 2-6: The 5-C collaboration model (Shah, 2013b, p. 1124)

Denning and Yaholkovsky lists four main categories to achieve a synergic flow of work. These levels of joint actions are: Sharing Information, Coordination, Cooperation and Collaboration These levels are ordered ascendingly according to the strength of the integration of activities involved. As with the other models, Collaboration is characterised as the ultimate form of working together. However, having a successful information sharing or coordination process is not a determinate of a successful collaboration as they state, instead it is achieved with the collaborators’ experience of solidarity. In

Chapter 2 Literature Review that sense, technology facilitates, but cannot ensure, a successful collaboration (Denning and Yaholkovsky, 2008).

This discussion above reveals that catering for collaboration involves more than the sum of each contribution. Yet all these processes listed above should be catered for from a design and implementation perspective, from the communication process until collaboration process. In the next subsection, each of these four preceding processes in the 5-C model to collaboration will be examined and discussed in the sense of enabling collaboration.

2.2.1.1 Communication Process Communication is the most basic process and describes the exchange of information and meaning between the collaborators. The field of Computer-Mediated Communication (CMC) provides a wealth of theories and study aspects for the ways communication enables collaboration. Olson and Olson list a variety of technologies used in modern day collaborations from email and texting to the use of virtual worlds and these were featured in various CIS research studies, some of which are highlighted next (Olson and Olson, 2013, sec. 9.1.1).

In tightly coupled work, collaborators must frequently communicate about different aspects of the shared project. Sonnenwald identified 13 communication roles to achieve collaboration in her study published in 1996 of four design teams across north America and Europe (Sonnenwald, 1996). These included various roles such as inter- and intra- group and organisational level communicator as well as agent, sponsor or mentor of information communicator. These roles are defined by the knowledge exchange in the interactions between the collaborators as well as their assigned or handled tasks. She concludes that these communication roles did not appear to be related to the organisational hierarchy, but raised interest in the nature of the task and personalities of the collaborators (Sonnenwald and Lievrouw, 1997).

Moreover, two categories of communication forms were identified by Pinelle and his colleagues in their experimentation study to devise a Collaboration Usability Analysis model (Pinelle, Gutwin and Greenberg, 2003). The first category is explicit communication, which includes: spoken, written, gestural. The second category is information gathering, which includes: basic awareness, feedthrough, and overhearing. The model includes three more stages to achieve collaboration, but the first two are concerned with communication and both of which are discussed next.

Examples of activities in the explicit communication process include the use of email and instant messaging, which was used by employees in as an alternative form of asking for help, solving problems and exchanging information (Quan-Haase, Cothrel and Wellman, 2005) and even social networks, such

Chapter 2 Literature Review as Twitter, which was studied for a duration of time with a random selections of tweets and coded (Honeycutt and Herring, 2009). The communication process nowadays is well supported, with advanced network and sophisticated communication technologies, which eases the communication process greatly. For example, communication via audio chat lowers the cognitive load for a shared search experiment and reduces the negative emotions (González-Ibáñez, Haseki and Shah, 2013).

Information gathering, which is the second category, detail subtle forms of communications, or “decoupled communication” as Pinelle et al. describe it (Pinelle, Gutwin and Greenberg, 2003). In this form, the producer of the information does not necessarily intend to communicate, and the movement of information is initiated by the receiver. These are facilitated mainly by awareness, which is examined in subsection [2.5].

2.2.1.2 Contribution Process In this process, the collaborators help each other by providing a mutual exchange of information to support their efforts to achieve a common target and help in building mutual obligations and trust. Examples of where contribution process is apparent in technologies are public forums and online social groups, where strangers can contribute to help others with their knowledge and experience. This was examined in a large-scale survey and content analysis of almost 600 respondents of a professional network where the researchers uncovered complex social processes on how individuals contribute in a collaborative task. Mainly, their contributions occur when it enhances their professional reputations, when they have the experience to share, and when they are in the network of coworkers from within and outside their organisation (Wasko and Faraj, 2005).

As with previous processes, the contribution process is important to proceed the collaboration process, and need to be catered for in terms of design and implementation. These include for example the form of identification of the contribution, the form of responses, and the impact the contribution might affect (Kock, 2008).

2.2.1.3 Coordination Process Malone and Crowston use the dictionary definition of coordination: “harmonious action needed to connect collaborators together” as a starting point to define coordination. They elaborate that it is “the additional information processing performed when multiple, connected actors pursue goals that a single actor pursuing the same goals would not perform” (Malone and Crowston, 1990, p. 366). These definitions, in the context of collaboration, capture the essential difference of collaboration and coordination. During this process, the collaborators may share resources, responsibilities, and goals which require further efforts to achieve (Shah, 2013b).

Chapter 2 Literature Review

Coordination of activities between collaborators enables successful collaboration, as demonstrated in key studies in the field. In Heath and Luff’s classic study of London underground controllers, they noted that they did not explicitly communicate directly, yet they observed each other work and actions and coordinated their tasks accordingly (Heath and Luff, 1991). Another study focused on the use of ShrEdit co-authoring workspace allowed designers to coordinate their work through the shared feedback of each other’s work by being aware of their actions (Dourish and Bellotti, 1992; Olson et al., 1992). Moreover, coordination entails the transfer of tools or resources between collaborators, this subprocess of sharing resources includes action to obtain, reserve, and protect resources needed (Pinelle, Gutwin and Greenberg, 2003).

In recent studies and more specific to information seeking, Evans and Chi's survey revealed that people might coordinate their search activities before, during, and after search sessions with various levels, as devised in the Social Search model they developed in their survey of Amazon Mechanical Trunk workers (Evans and Chi, 2010). In contrast, Shah and González-Ibáñez experiment coded the chat messages for coordination using a slightly different time indicator of past, present and future search coordination during a search session. Their experiment manipulated three conditions across the custom CIS: baseline, personal and group awareness and found that with ‘right kind’ of awareness in the latter condition, the messages exchanged were better for collaborative search tasks. In this study ‘better coordination’ was measured using three components: First as the less number of past coordination messages, which is deemed costly in collaboration and more future-planning coordination and, secondly, with less number of inaccuracies in the exchanged messages, and finally with the amount of time. So ultimately the right kind, according to Shah, is the adequate and appropriate kind of awareness based on the context and application of the CIS system (Shah and González-Ibáñez, 2011).

The advances in technology since these studies has far-reaching implications for enhancing the coordination process of collaboration. Coordination, therefore, must be supported in terms of communication channels as well as awareness, yet is a subtle manner.

2.2.1.4 Cooperation Process In this process, different collaborators, with shared similar interests, partake in arranging activities and assigning roles to fulfil shared goals but may focus on different aspect of them. They engage in a process through which they can go beyond their own individual expertise by exploring their abilities and searching for common solutions. Shah noted that the difference between cooperation and coordination is not usually mentioned explicitly in the literature (Shah, 2008), and both coordination

Chapter 2 Literature Review and cooperation should include all collaborators, but stresses that the fact that the latter entails further levels of interaction, involvement and trust (Shah, 2012).

2.2.1.5 Collaboration Process As discussed in the introductory of this section, collaborators aim towards achieving a common complex goal beyond that they can achieve individually, even if it is not the “first choice” towards achieving it (Denning and Yaholkovsky, 2008). Collaboration work is aimed towards a solution that is “beyond the total of each collaborator contribution” (Shah, 2015). Moreover, taking collaboration in an abstract manner, London discusses that the process of collaboration is typically a complex and multifaceted and it usually moves through several distinct phases (London, 1995).

2.2.2 Computer-Supported Cooperative Work (CSCW) The explicit collaboration between two or more people using digital platforms had been explored through multiple focus points. One field is Groupware (Gutwin and Greenberg, 2002) which tilts towards the design and implementation of technological artefacts. Another term is Virtual teams which is used to label the study of people and work interaction using communication and information technologies (Wilson et al., 2008; Olson and Olson, 2013). However, the most common and encompassing term used to describe collaboration through digital artefacts is Computer-Supported Cooperative Work (CSCW). While CSCW is central to this research, it will also consider Collaboration as a holistic term because it encompasses broader aspects and takes into consideration that all these terms are in the context digital platforms technologies due to its pervasiveness.

It is also noted that there is some confusion about what whether CSCW is the social extension of HCI or a separate field. Dix et al. provide a simple analogy to that “both HCI and CSCW draw on knowledge from a wide range of disciplines, but whereas the principal axis in HCI is psychology–computing, the equivalent axis in CSCW is sociology–computing” (Dix et al., 2003, p. 664). Moreover, CSCW shares and overlaps with several parts with the field of Computer-Mediated Communication as described in [2.2.1.1].

CSCW has emerged as the umbrella term to study and describe the interaction of people working together using digital platforms. Due to the interdisciplinary nature of this field, its meaning resulted in several loose, interchangeable and sometimes overlapping definitions. This is due to considering a diverse array of perspectives into a coherent framework of collaborative work. The term inherited a lot of discussions and debates amongst researchers (Schmidt and Bannon, 1992; Simone and Schmidt, 2000) and there is a sense in which CSCW is continually being formed (Schmidt and Bannon, 2013) as new technologies with collaborative potentials are emerging and adopted and adapted by users. This

Chapter 2 Literature Review leads to discussing CSCW not in terms of its definitions only, but in terms of its taxonomies and the fields it is being discussed in. Therefore, the focus in this thesis will from the technology design and implementation point of view of CSCW systems.

One of the most considered aspects of CSCW is that of the Space / Time dichotomy of the work interaction facilitated by technology, which is modelled as the CSCW matrix (Johansen et al., 1991). Also known as the concurrency and dimension axis, this model resembles the two widely used aspects of CSCW, Space and Time, also known as the CSCW Matrix (Johansen, 1988) and the time space taxonomy (Ellis, Gibbs and Rein, 1991). The figure below [Figure 2-7] describes the typical 2 × 2 matrix with a sample list of communication forms and computer-mediated channels (CMCs) for each quadrant, as posted by a Wikipedia contributor (Computer-supported cooperative work, 2014).

In this matrix, the Time horizontal axis indicates whether the collaborators are working together, i.e. synchronous or working at different times, i.e. asynchronous. On the other hand, the Place vertical axis which demonstrates the geographical situation of the collaborators, either working together at the same place, i.e. co-located or working afar, i.e. remotely distributed.

Figure 2-7: The Space / Time Groupware Matrix - A variation of the CSCW Matrix – User: Pascal (Momo54) © Wikipedia Commons (https://commons.wikimedia.org/wiki/File:Cscwmatrix.jpg)

The model has been refined and modified by various researchers, e.g. in a survey of CSCW taxonomies (Rodden, 1991), and in a refined version which was illustrated with a 3 × 3 columns by dividing the different time and different place to two as designating the columns as different predictable and different unpredictable, based on a study groupware developers and designers (Grudin, 1994a).

Chapter 2 Literature Review

2.3 Collaborative Information Seeking (CIS)

The term Collaborative Information Search (CIS) implies that Collaboration and Information Seeking are the main dimensions to define and understand this interdisciplinary field, both of which are discussed above. However, as interest in collaborative aspects of information seeking on the web and digital resources soared, more aspects and considerations are being examined in this domain. This chapter examines the various aspects of groups of people interaction with information search to narrow down the exact focus aspect for this research.

2.3.1 Situating CIS To have a clear definition for CIS, this subsection will first provide a discussion of the different terms used to describe the activity of people searching together. Then it will approach this topic from the research field perspective and consider the modern contributions done in CIS. Finally, this subsection concludes with a selected set of definition of CIS based on the discussion.

To understand the confusion terms can cause, an example will be presented next. The term Social Search is currently being used more frequently to describe various forms of searching that encompass a multitude of meanings and context. This is due primarily to the surge in the use of social networking sites. This is apparent in how Chi argues that search is not a solitary fact-finding activity, promoting that “Information Seeking can be Social” and that social search is exploratory in nature, ill-defined and can be done collaboratively (Chi, 2009). In that article, Chi discusses two forms of social search: social answering systems, like forums and Q&A websites, and social feedback system like social bookmarking sites. However, Kang et al. consider tagging websites (Kang, Fu and Kannampallil, 2010), which also contrasts with how Mao et al. describe it in the experimental system Baijia sharing keyword based on other users search logs (Mao, Shen and Sun, 2010). Therefore, a clear categorisation system is essential to comprehend the literature domain.

Burghardt et al. taxonomy, illustrated in [Figure 2-8], represents a useful approach to cluster the various examples of social search terminology and provide some distinction between these terms (Burghardt, Heckner and Wolff, 2012). They define the term people-powered search as an umbrella term for any form of collaborative search on the Internet to designate socially searching system, as oppose to automated collaborative search systems which accounts for the action of bots like web crawler or intelligent information search agents (Chakrabarti, 2003).

Chapter 2 Literature Review

Research Focus

Figure 2-8: Taxonomy of social search approaches (Burghardt, Heckner and Wolff, 2012, fig. 1)

Furthermore, Burghardt et al. classify people-powered search into two categories: implicit and explicit. The former signifies the common blueprint method for modern search engines like Google’s original search algorithm PageRank which utilises the click popularity and user visitation statistics, in addition to its main criteria links to the web resources (Brin and Page, 1998). As for the explicit category, it takes into consideration the visible social interaction of a hit which is performed directly and deliberately, with various degrees and contexts, during the search for information process. The research highlights five distinctive forms of explicit social search interaction that occur on the Internet, summarised in [Table 2-3].

Out of these subcategories, Collaborative Search, which is highlighted in [Figure 2-8], is noted here as the most explicit form of social search and represents the focus point of this research. Characteristics of this form of search tend to indicate a direct cooperation between users. The collaborators often know each other and have a specific goal they want to achieve together.

Chapter 2 Literature Review

Explicit People-powered Description Search Categories Explicit manual indexing and tagging of web resources. Helps in Social Tagging resolving vocabulary problems in search. Users formulate their own question in natural language and the Social Question Answering community member can reply. Searchers share an information need and actively work together Collaborative Search to fulfil that need through. Explicit recommendations by community members that matched Collaborative Filtering a specific set queries. Systems that consider the behaviour of other users of the system Personalised Social Search Engines when generating search results and recommendations.

Table 2-3: Explicit People-powered search categories. Summarised from (Burghardt, Heckner and Wolff, 2012)

Moreover, in addition to term Collaborative Search other terms include Collaborative Information Seeking (CIS) and Collaborative Information Seeking & Retrieval (CIS&R) or Collaborative Information Retrieval (CIR), albeit designating the same concept, with minor variation. This is partially a consequence of CIS being an interdisciplinary field and reflects the growing interest from several disciplines in the collaborative manner of dealing with, and searching for, information digitally. Overall, there is a tendency to use these terms interchangeably (Karunakaran, Reddy and Spence, 2013), with the current consensus being the use of the term Collaborative Information Seeking, as it is used primarily by studies focusing on the design and implementation of software artefacts. It will therefore be used in this research and across the thesis.

As with many interdisciplinary fields, CIS has been researched with a variety of focus lenses in a similar fashion to information seeking. Very early research in CIS did not explicitly consider people searching together as a collaborative activity, but rather as an extension of the solitary perspective of information seeking (Wilson and Schraefel, 2010; Hearst, 2014). Moreover, it has been researched under the umbrella of the more prominent interdisciplinary fields of HCI, IR and CSCW as shown in [Figure 2-9], where the authors used CIR to indicate collaborative information retrieval (Fernández- Luna et al., 2010). Because of these multiple lens perspectives, the term CIS itself continues to be used interchangeably with the aforementioned terms.

Chapter 2 Literature Review

Figure 2-9: CIR conforming disciplines (Fernández-Luna et al., 2010, fig. 1)

Furthermore, Shah suggested an expanded perspective of CIS, incorporating two more fields to the models by Fernández-Luna et al.,(Fernández-Luna et al., 2010) the first is social media and networking to emphasise the surge of social networking sites (SNSs) this view by adding this growing influential medium; and secondly the addition of Collaboration (Shah, 2013a) as distinct field from CSCW. This is illustrated in [Figure 2-10].

Figure 2-10: CIS as an interdisciplinary field, (Shah, 2013a, fig. 1).

Chapter 2 Literature Review

Naturally, the various research disciplines influencing CIS is reflected in the definitions researchers use to define it, a couple of highly cited definitions are selected here. Foster asserts that the very nature of this mixed perspective of CIS definitions in his basic, yet inclusive, definition of Collaborative Information Seeking:

“the study of the systems and practices that enable individuals to collaborate during the seeking, searching, and retrieval of information” (Foster 2007, p.330).

Furthermore, Hansen and Järvelin seminal ethnographic research in an information-intensive Swedish patent registration office, define Collaborative Information Seeking and Retrieval as:

“an information access activity related to a specific problem solving activity that, implicitly or explicitly, involves human beings interacting with other human(s) directly and/or through texts (e.g., documents, notes, figures) as information sources in an work task related information seeking and retrieval process either in a specific workplace setting or in a more open community or environment.” (Hansen & Järvelin 2005, p.1102).

The difference between the former definition of CIS as field and the latter with CIS as a process reflects the subtlety of the researchers’ considerations and perspective of CIS. Some literature prefers to define CIS in context of the domain, rather than be limited by a single definition but “by reference to the ‘medium’ of collaboration, the ways CIS is conducted, the tools used and physical setting of CIS, and the ‘context’ of CIS, the purposes for which an instance of CIS occurs in that discipline” (Newman et al., 2015, p. 37). In this thesis, the consideration is on these definition, and on the use of CIS term as the field of study, and as the activity itself. Both of these terms highlight the non-solitary nature of information seeking.

2.3.2 CIS Models and Taxonomies Based on the summary of information seeking models listed in [2.1.2], several studies attempted to extend existing models to CIS and, ultimately, some were found explicitly not be fitting. For example, Hyldegård concluded that ISP model does not fully comply with group members’ problem solving process and the involved information seeking behaviour (Hyldegård, 2009).

Chapter 2 Literature Review

The holistic Social Search model, proposed and elaborated in (Evans and Chi, 2008, 2010), is described as:

“an umbrella term used to describe search acts that make use of social interactions with others. These interactions may be explicit or implicit, co-located or remote, synchronous or asynchronous.” (Evans and Chi, 2010, p. 657)

In particular, they have identified insightful information on an adopted taxonomy of query needs during a search episode, which Evans and Chi placed between a ‘Before search’ and ‘After search’ activities. The insightful Identified originally by (Broder, 2002), these distinct needs are:

• Informational needs: These include information “assumed to be available on the web” • Transaction needs: These include web pages that contain further resources • Navigational needs: These include the access to a particular website or web page

Furthermore, Evans and Chi uphold that these three search phases (before, during and after search) are social-based throughout each session. While this model of social search details the understating of the various search activities in a social context, it lacks some clarification on how the design and implementation can be addressed for CIS. To precisely identify this research focus, Golovchinsky et al. propose a taxonomy of CIS forms which is based on four dimensions: intent, depth of mediation, concurrency and location (Golovchinsky, Qvarfordt and Pickens, 2009). A graphical representation is found in [Figure 2-11].

Depth of Intent Mediation Explcicit vs. Algorithmic vs. Implicit Interfacial

Concurrency Location Synchronous Co-located vs. vs. Remotely Asynchronous Located

Figure 2-11: Graphical representation of (Golovchinsky, Qvarfordt and Pickens, 2009) CIS model

The use of this four-lens perspective allows narrowing down the literature to explore the relevant studies for this thesis. The first dimension, Intent reflects the implicit or explicit nature of the

Chapter 2 Literature Review collaborative search; i.e. whether the search activity is intended to be of collaborative nature between collaborators. The second dimension, depth of mediation, refers to the level where the collaboration occurs, either at the search interface or at the backend search algorithm, or both. The third dimension, concurrency, refers to the synchronicity or asynchronicity of the search, i.e. describes whether the search activity is done at the same time or at separate times. The final dimension, describes whether the collaborators are in co-located in the same place or conducting the search in distributed locations. The last two, concurrency and location, resemble the CSCW matrix discussed earlier in [2.2.2]. While the models cover the essential aspects of CIS, there are several others that discuss addition aspects and these are covered below in details.

2.4 CIS Aspects

These CIS aspects provide a solid foundation to cover a wide range of concepts and issues in CIS that stem from the fields of IS&R and CSCW and explained above. The main literature in CIS tend to approach CIS by explaining it in terms of situational aspects such as ‘Who’, ‘What’, ‘Where’, ‘When’ or ‘Why’ question about CIS, which covers a variety of issues in CIS (Morris and Teevan, 2009) and, in additions, by ‘How’ and ‘So What’ which deals with best approaches to cater for CIS (Shah, 2015).

Hearst presents a set of three scenarios which cover recent challenges in CIS: firstly, the shared group travel booking search for hotels, tickets and car rentals which represents the case of CIS to aid in the search of select one option from a number of different but similar alternatives; Secondly, the example the PhD research group searching for relevant literature for writing a paper, which represents the case of CIS to fully cover a topic space, or thirdly, the discovery of known information in multifaceted and complex problems (Hearst, 2014; White, 2016).

Research on explicit collaboration in information seeking targets behaviours and collaborative practices that can be supported with technology. This can be grouped into the following two categories based on the nature of the mediation that forms part of the collaboration: Algorithmically mediated and user interface mediated. In the former, the system acts as an active agent and provides mediation among the collaborators to enhance their productivity and experience. Example systems include Cerchiamo (Golovchinsky et al., 2008), Searchius (Papagelis and Zaroliagis, 2007) and Collaborative Web Search (CWS) (McNally et al., 2011). A summary of these approaches is discussed in below in [2.4.7]. In the latter, the control lays with the collaborators where the system serves as a passive component. Searchers drive the collaboration, and the system primarily provides various functions on the interface level.

Chapter 2 Literature Review

This research is focused on the latter aspect, i.e. that of user interface mediation of collaborative search. A selected set of these studies were referenced throughout the introduction and in the leading literature review section till here. These are essential to this research and represent the main canonical studies, and naturally include a porotype CIS. The table below [Table 2-5] summarises these studies in the domain of system-focused CIS studies and identifies the following aspects: The platform of the prototype developed; the Search engine domain; The specific design focus; the type of the usability study and the main method used to deliver the cues. All of these studies listed are focused on user-interface mediation, except for Querium, which represents an interesting take combining both these aspects (Diriye and Golovchinsky, 2012) but that did not materialise5.

Therefore, to understand the literature on CIS, a selected set of devised aspects is synthesised from these studies and publications. The selected set of aspects starts with Awareness as the key factor of a successful CIS process, followed by Communication in CIS, which is identified earlier in [2.2.1.1] as the key aspect of collaboration. This is followed by the User Interface, Division of Labour, Roles, Collaborative Grounding. Algorithmic Mediation is also added to ensure that its role in the CIS is recognised, but mainly for the implicit domain of CIS.

5 Sadly, the lead author and the principle designer behind for Querium, Dr Gene Golovchinsky, has sadly passed away in 2013, may he rest in peace.

Chapter 2 Literature Review

System Search Usability Specific Design Main Awareness (main Platform Engine Study / Study Focus Cues publications) Domain Methods SearchTogether/ Browser General • Awareness • Experimental • Sidebar Search CoSense plug-in web search • Persistence Activities (Morris and • DoL • Instant Messaging Horvitz, 2007; Paul • Sensemaking • Summaries and Morris, 2009, (CoSense) • Basic Rating 2011) MUSE / MUST Web Medical • Communication • Observational • Results Sharing (Reddy and application publications • Case Study • Instant Messaging Jansen, 2008; Reddy, Jansen and Krishnappa, 2009) Coagmento (v1 - Web General • Awareness • Experimental • Sidebar Search v3) application web search Activities (Shah and • Instant Messaging Marchionini, 2010; • Summaries Shah and González-Ibáñez, 2011; Shah, 2012) CoFox Standalone General • Awareness • Experimental • Screenshot (Perez, Whiting application web search • Communication telecast and Jose, 2011; • Synchronous • Instant Messaging Perez, Leelanupab only and Jose, 2012) Querium Web General • Session Handoff None5 • Sidebar Histories (Diriye and application web search • Design and and Summaries Golovchinsky, Development 2012; • Implementation Golovchinsky, Dunnigan and Diriye, 2012) Results Space Web TREC • Awareness • Experimental • Sidebar Histories (Capra, Arguello, application Database • Persistence and Summaries et al., 2012; Capra, • Ranking • Accumulative Chen, Hawthorne, Rating Arguello, et al., • Results filtering 2012) ezDL Standalone Academic • Design and • Usage Survey • Integrate external (Böhm, Klas and application Libraries implementation Instant Messaging Hemmje, 2013, / and • Work task • Basic Rating 2014b) Framework Databases granularity • Extensibility CollabSearch Web General • Query • Experimental • Sidebar Search (Yue, Han and He, application web search Formulation Activities 2012, 2014) • Search • Extended Rating Behaviour (Star)

Table 2-4: Summary of main CIS systems in the literature in terms of Platform, Domain, and Design Focus, Research Methods and the main Awareness Cues

Chapter 2 Literature Review

2.4.1 Awareness In their special issue on differentiated awareness support in CSCW, Kolfschoten et al. state that “Awareness [is] a topic of research especially in new (virtual) interaction environments and in various challenging domains in which collaboration is becoming an answer to the increasing complexity of interconnected systems and globalization” (Kolfschoten, Herrmann and Lukosch, 2013, p. 107) which reflects its importance and impact in the research domain. It is considered a key aspect of coordination (Heath and Luff, 1991; Olson and Olson, 2013).

Therefore, awareness is a major aspect of CSCW studies and increasingly in HCI, CSCW and in IIR studies amongst other fields and like interdisciplinary concepts, awareness has had its share of interchangeable definitions and taxonomies mainly within these fields (Schmidt, 2002; Gross and Prinz, 2003; Carroll et al., 2006; Rittenbruch and McEwan, 2009) due, in part, to the lack of decisive and overlapping definitions in scope and usage. Furthermore it is one of those elastic terms that are sometimes implied, rather than explicitly stated (Gutwin and Greenberg, 2002). Nevertheless, the notion of awareness with all its definitions and scope did achieve a lot of mainstream research and realisations in artefacts and is still relevant beyond the apparent pitfalls that might rise of this (Carroll et al., 2009).

In their seminal paper, Dourish and Bellotti define awareness as “an understanding of the activities of others, which provides a context for your own activity” (1992, p. 107). This definition, nevertheless, has been redefined in several contexts and extended to meet different requirements. In particular, focus on collaborative systems that are “intended to help people construct and maintain awareness of each other’s activities, context or status, even when the participants are not co-located” (Markopoulos, De Ruyter and Mackay, 2009, p. v) is apparent in terms of abundant technological and research solutions within empirical research. A third definition emphasises the non-distributive aspect of awareness as “the ability of users to understand and interpret the activity of others engaged in a cooperative effort without causing interruptions through explicit communication, such as asking questions” (Villa, Gildea and Jose, 2008, p. 221).

In this review four main taxonomies will be discussed, related primarily to the design and development of collaborative systems: situation awareness, workspace awareness, mutual awareness and activity awareness.

2.4.1.1 Situation Awareness An early type of awareness is situation awareness (Endsley, 1995) stemming from psychology and human factors fields and has been associated with critical tasks, such as aviation controls; and has

Chapter 2 Literature Review been also studied in the context of team collaboration such as a qualitative study of a military command and control dense communication between various members of different roles (Sonnenwald and Pierce, 2000). It was also tested through an experiment of devised framework in a collaboratory context which compared between face-to-face and a remote video report writing task (Sonnenwald, Maglaughlin and Mary C. Whitton, 2004). Formally extending it to teams, Endsley and Jones define a team situation awareness as “the degree to which every team member possesses the SA required for his or her responsibilities” (2011, p. 195). Furthermore, they suggest a ‘shared situation awareness’ approach to cater for team awareness is by specifying a set of requirements, devices, mechanism and processes along with a list of design principles. However, they affirm that to achieve a team situation awareness “first requires that we design systems to support the [situation awareness] of each individual team member” (2011, p. 218).

2.4.1.2 Workspace Awareness Closely related is the concept of workspace awareness, which is defined as “the up-to-the-moment understanding of another person’s interaction with the shared workspace” (Gutwin and Greenberg, 2002, p. 417). Their extensive framework, demonstrates what information to provide, what perceptual mechanisms to use to convey the information, and when and where in the interface to provide the information. The framework includes a practical interface design application of workspace awareness, using their GroupKit development tool, and identifies the placement and presentation techniques for widgets and controls. Ultimately, to support workspace awareness and “sustain effective team cognition when working over a shared visual workspace, groupware systems must give team members a sense of workspace awareness.” (Gutwin and Greenberg, 2004, p. 30). This type of awareness is crucial in CIS studies and systems, and commonly referred to in the literature simply as awareness.

2.4.1.3 Mutual Awareness Schmidt and Simone focus on mutual awareness, which in itself stems from Strauss’s articulation work (Strauss, 1988) in organisational and sociology studies meaning the preparatory work to accomplish actual work. To elicit their four subtypes of awareness, listed here in the words of Cheverst et al. (2009): Perception of the field of work; Inferences from that to enable indirect perception of activities of others; Direct perception of “bodily conduct” of others which includes heir focus of attention and also overheard conversations, etc.; and overhearing of other participants’ explicit acts to coordinate their awareness with each other.

Chapter 2 Literature Review

2.4.1.4 Activity Awareness Finally, a comprehensive activity awareness taxonomy is proposed by John Carroll and his colleagues (2009) stemming from activity theory and includes four main facets of awareness drawn from various fields. These facets include Common Ground, which is a key term in these fields in refers to the signalling and enhancing shared knowledge and belief (Clark and Brennan, 1991; Clark, 1996); the other three are also Communities of Practice (CoP), Social capital and Human Development. These are defined below along with protocol type used for each facet. It is noted here how similar these facet protocols are to the processes of collaboration detailed in [2.2.1], albeit without the second process of contribution and the final process is named as Group development as shown in the summarised table of the awareness activity model below [Table 2-5].

They further elaborated these facets to include various dimensions such as issues of distance and social dynamics and the group lifecycle amongst collaborators.

Activity Definition Protocol Type Awareness Facet Common Ground Signalling and enhancing shared knowledge and belief Communication

Communities of Developing and applying community-specific practices Coordination Practice (CoP) through enactment

Resource exchanges that engender and sustain generalised Social Capital Cooperation reciprocity and trust

Encouraging innovative decisions and approaches in open- Human Group system problem solving to evolve group capacities and Development Regulation performance

Table 2-5: Summary of Activity awareness facets definitions and protocol type (Carroll et al., 2009).

2.4.1.5 Awareness for CIS In the CIS context, Awareness is considered the key aspect to be achieved, and is explicitly highlighted in the key literature on the design and Implementation of CIS systems. While most research tends to use one of the aforementioned above definitions, two prominent perspectives of awareness are discussed here.

CiteSeer is considered one of the early digital libraries on the WWW that consolidated several early scholarly databases in one search engine as it automatically crawls and indexes scientific documents in the field of computer and information science (Li et al., 2006). To aid its users, Farooq et al.

Chapter 2 Literature Review attempted to include an RSS6 feed-based notification to communities within CiteSeer to alert the users with new publications in their community. To test this mechanism after a couple of design and implementation iterations, Farooq et al. surveyed users for the best awareness mechanism out of three choices of feeds and then tested the functional prototypes with participants (Farooq et al., 2008).

The second form of awareness it refers to the information seeker being aware of the issues surrounding searching and sense-making processes, including the task and its context, past and present actions, and various attributes of the information objects and the system (Shah, 2013b, p. 1124). Shah often referenced awareness taxonomy in his studies using Liechti and Sumi’s research which states four common types of awareness:

Awareness Type Definition The ability that peers may have to stay in touch and to keep track of each Group other’s activities.

Emphasises the fact that awareness generally emerges when people Workspace share a space, or at least when they share artefacts.

The human ability to process information at the periphery of the Peripheral attention, with a very low overhead.

The ability of a system to adapt its behaviour to the current situation that Contextual of the system itself, of its environment, and of its users.

Table 2-6: Common Awareness Types (Liechti and Sumi, 2002)

This diversity of awareness discussions provides an ample understanding of the expected nature of the awareness in CIS. These range from generalised aspects of the environment and human collaborations, to being focused on the awareness mechanisms and cues and collaborative workspace.

2.4.2 Communication As described earlier in possesses of collaboration [2.2.1.1], Communication is an essential and precondition for any collaborative activity to be achieved. Communication between collaborators in CIS is mainly achieved through Instant Messaging (IM), as it allows the most natural flow of information exchange. While being the simplest form of mediation to achieve coordination, it can add some burden on the collaborators to explicitly define and articulate their search activities.

6 RSS: Rich Site Summary / Really Simple Syndication: A textual machine-readable format for automatic web feeds

Chapter 2 Literature Review

Furthermore, studies have shown that the disruptiveness IM can inflect on productivity tasks can be minimized when the messages exchanged are somewhat relevant to achieving the intended goal (Czerwinski, Cutrell and Horvitz, 2000).

All the discussed CIS systems included some support for instant messaging, they differ in the form of inclusion. While several studies integrated chat functionalities in their research tools, the use of external chat clients has been utilised in earlier CIS studies. For example, the experimental prototype Multi-User Search and Talk (MUST) used Yahoo messenger as the chat medium (Reddy, Jansen and Krishnappa, 2009). While other systems integrated support for third-party communication services such as Skype (Böhm, Klas and Hemmje, 2014b). When explicitly mentioned in their finding, the chat implementation of research approaches mainly depended on dedicated built-in chat functionality or on freely available external chat clients.

For the former method, SearchTogether (Morris and Horvitz, 2007), CoFox (Perez, Whiting and Jose, 2011) and (Joho, Hannah and Jose, 2008) all include built-in Instant messaging interfaces. A different approach was used with CoSense, which includes a dual chat interface, embedded in the timeline and in a separate chat-centric window. Paul and Morris compare this dual setup with the older SearchTogether, in IM which was found to be essential for the synchronous search tasks, but a bit confusing and out of context at later stages of the chat. Nevertheless, in the asynchronous search trials, they note how chat helped participants in the handoff aspect from one searcher to another using the offline chat messages (Paul and Morris, 2011).

2.4.3 User Interface (UI) Supporting CIS at the interface level has been achieved in several forms, these include a combination of window and screen arrangements, awareness mechanisms such as notifications and other forms of controls cues and integrated computer-mediated communication (CMC) channels (Rädle, Jetter and Reiterer, 2013). UI mediation is a crucial aspect in explicit collaborative search systems to distinguish it from traditional solo-based search interfaces. This review will look at various screen setup used in CIS system. An in-depth inspection of the particular awareness mechanisms is in section [2.4.7].

Split search screens displaying other collaborators’ activities is a common approach in CIS (Morris and Horvitz, 2007; Amershi and Morris, 2008). A basic split screen setup is found in CoFox, which splits the display between the collaborators’ active windows browser horizontally along with chat windows. CoFox functionality includes a recorded time-lapse of the other collaborators’ screen activity, giving them a similar feature of a time shifting on a personal video recorder (PVR) to view the each other

Chapter 2 Literature Review activities. However, CoFox main focus is on the synchronous search and works exclusively with pairs (Perez, Leelanupab and Jose, 2012).

A more informative approach to is SearchTogether, where the screen is split between the browser search screen on one side and the group search activity timeline and communication screen on the other side vertically. However, the split screen approach is found not to be heavily used amongst the study participants, which is attributed to the study context (Morris and Horvitz, 2007). This mechanism of displaying spilt-screen awareness cues has researched in several other studies based on the SearchTogether platform, e.g. CoSense (Paul and Morris, 2009) which included further categorisation of the tabs.

Other experimental tools opted for full-screen UI mediation and to bring DoL in the centre of the collaborative search activity workspace. Querium, a prototype CIS web app, arranges search sessions into specific tasks where collaborators can join a task, which is essentially a SERP interface with ranking (Diriye and Golovchinsky, 2012) and rank hits and view each other activities in summaries. In a similar manner, ResultsSpace focuses the group activity in the centre of the web page, along with sidebars of relevant awareness controls (Capra, Arguello, et al., 2012). Coagmento brings the snippets also in the forefront of the collaborative workspace (Shah, 2012, sec. 6.4.2.2; Kelly and Payne, 2014, p. 809). Multimedia collaborative search utilises cards and thumbnails to support CIS, in addition to the use of split-screens technologies (Villa and Jose, 2012).

UI mediation is has been studied in tablets and tabletops collaborative application where the touch interface promotes innovative ways to tackle CIS (Smyth, Balfe, Freyne, et al., 2005; Smeaton, Lee, Foley and McGivney, 2006; Morris, Lombardo and Wigdor, 2010; Rädle, Jetter and Reiterer, 2013). Tabletops7 provided several interesting insights into synchronous co-located search scenarios large surface screens that enabled novel approaches in collaborative activities. Earlier in this review, the Físchlár-DiamondTouch was mentioned as an example of CIS to support is a dedicated video CIS system on a tabletop (Smeaton, Lee, Foley and McGivney, 2006; Smeaton, Lee, Foley, McGivney, et al., 2006). In this experiment, pairs were asked to collaboratively search for video clips from an archived library. An alternating control was a different design that created a trade-off between awareness and individual efficiency of interaction. Despite having mixed results, the analysis indicated that the awareness condition edged the individual efficiency in terms of coordination interaction and search effectiveness. This was echoed in WeSearch, a specifically designed CIS application for a 6 × 4

7 Tabletop: A similar setup to surface computing that typically relies on a roof projector and table-level hand and arm movement sensors.

Chapter 2 Literature Review feet tabletop, enabled successful facilitation of CIS activities for a relatively new form of interaction amongst the usability evaluation participants (Morris, Lombardo and Wigdor, 2010). DoL in that study has been realised at a different search stage during the evaluation process than originally envisioned by the researchers. The participants divided the tasks in the sensemaking phase rather than assigning subtasks amongst them.

Tabletops provide a unique experience for co-located collaborative information search with an emphasis on UI mediation. Morris et al. review of four CIS applications on tabletops and surface computers confirm that such interfaces provide several design aspects that remain to be explored. However they acknowledge the challenges presented with this form of interaction for search activities such as the orientation of search results and the implied limitation of tabletops such as text entry (Morris, Fisher and Wigdor, 2010).

2.4.4 Division of Labour (DoL) Borrowed from social sciences, Division of Labour (DoL) is one of the key conceptual coordination techniques in analysing collaborative activities (Strauss, 1985; Schmidt and Bannon, 1992). Derived from cooperative and collaboration studies in CSCW, e.g. Malon’s coordination theory (Malone and Crowston, 1990), DoL is observed in CIS studies in coordination between collaborators (Poltrock et al., 2003; Chi and Pirolli, 2006; Smeaton, Lee, Foley and McGivney, 2006). Additionally, Morris’s survey of over 200 employees’ information seeking habits found that they “described ad-hoc methods to avoid duplication of effort during a search task”, which in turn must be catered for in CIS by “Supporting mechanisms for dividing up and sharing work among participants is important to the success of a UI for multi-user search” (Morris, 2007, p. 6).

The definition of Division of labour refers to the “process of dividing up the group task across collaborators in order to share the workload across the group” (Foley and Smeaton, 2010, p. 764). DoL is being promoted as the key feature to have across various types of researched CIS systems (Halvey et al., 2010; Perez, Whiting and Jose, 2011). Foley describes it as essential coordination techniques for collective information searching (Foley and Smeaton, 2009, 2010). Additionally, Morris identifies DoL as the one of a trio of main features to support in CIS, along with awareness and persistence (Morris and Horvitz, 2007).

Kelly and Payne categorised the fragmented notion of DoL in using a subset of Golovchinsky et al. taxonomy model of CIS detailed in [2.3.2]. Their taxonomy for DoL in CIS has four approaches: Communicative, Algorithmic, User Interface and Role-Based (Kelly and Payne, 2013). These resemble, however essential aspects of CIS in general, whereby DoL is catered for.

Chapter 2 Literature Review

The key issue is how to translate division of labour into a design feature in a proposed CIS is of utmost importance. This can be realised in the through the other factors essential to CIS, most notably the User interface as with SearchTogether, and role-based aspects of CIS, as described in the section [2.4.5].

2.4.5 Roles In CIS, having designated roles set upfront for a search task has been tested in few studies and argued to be conditionally beneficial. Roles impacting CIS have been observed in work situations with explicit organisational and hierarchical structures. These roles can be symmetric, with similar roles presumed by collaborators or asymmetric roles with distinct roles amongst collaborators. For example, a command and control qualitative study in a military context identified information roles like information gatherer, information referrer and information verifier (Prekop, 2002). As oppose to symmetric, ad-hoc role assignments, or brute force assignment as noted in (Morris, 2007). Typically, this asymmetric search between collaborators arise from the different expertise and background of the searchers or the different needs of the collaborators’ explicit divide and conquer strategies can have advantages and disadvantages.

Studies that included explicit roles of assigned to collaborators are limited in the literature and most are algorithmically mediated as the searches of prior collaborators in an asynchronous search is seeded into a search algorithm in an explicit manner known to the collaborators (Smyth, Balfe, Boydell, et al., 2005; Joho, Hannah and Jose, 2008).

Earlier studies used the searches by collaborators to algorithmically seed subsequence searches by collaborators. Pickens et al. investigated assigned the roles of Prospector and Miner (Pickens et al., 2008). This was followed by Shah et al. (2010) which examined explicit assigning of roles in an asynchronous collaborative search system that. Using the assigned roles of surveyor and gatherer, the study incorporated an algorithmic search system that caters for each searcher role. These roles are defined as follows: Gatherer, who focuses on quickly finding as much relevant information as possible, and the Surveyor, who will delve into a more diverse set of documents (Shah, Pickens and Golovchinsky, 2010, p. 776). With the primary role of investigating the algorithmic mediation to distribute the search results amongst the collaborators, the study concludes that “while in some cases collaboration with identical roles is appropriate, asymmetric roles can be effective in modeling certain kinds of collaboration. Both team members in the same role may get the benefit of division of labor in a collaborative setup, but it is the ability to combine diverse actions and inputs that gives collaboration its real strength” (Shah, Pickens and Golovchinsky, 2010, p. 780).

Chapter 2 Literature Review

This explicit division of labour was tested in an experimental study of CIS for a fictitious session to arrange a travel trip. Imazu et al concluded that assigning “Explicit division of roles can have significant effects on a group’s knowledge building during the collaborative search task” (Imazu, Nakayama and Joho, 2011, p. 205). In their experiment, the collaborators were assigned explicit roles in a synchronous co-located setting. The roles of the collaborators were those of a searcher and writer, where the former conducts the searches for a fictitious travel trip on a designated laptop and the latter notes down the details on a paper sheet. Compared to the non-assigned roles groups, the assigned roles’ groups travel plans did not differ significantly on the executability of the devised travel plans, i.e. their ability to be executed in reality, neither with provided details. However, on other measured factors, the explicitly role-assigned groups produced fewer search activities, yet more advanced chat utterance that reflected in conversations focused more on the travel plans than on the communication, implying more focused search behaviour. The study observed that the collaborators with explicit role-assignment were ‘more aware’ of the details of the task, yet were hindered from further ‘opportunities’ of travel options found by groups with no explicit role mediation (Imazu, Nakayama and Joho, 2011).

In summary, the use of roles in system-based literature is limited in scope and most studies presume the searchers will have equivalent roles in explicit collaborative search sessions. However, in ethnographic and observational studies where roles are crucial. Collaborators take on different roles in collaborative information seeking tasks. These can be implied based on the context, e.g. or defined explicitly (Imazu, Nakayama and Joho, 2011). It may be the collaborators’ actual role in the real context such as organisational hierarchy, such as in the military (Sonnenwald and Pierce, 2000) or command and control (Bohøj et al., 2010).

2.4.6 Collaborative Grounding Researchers in HCI and CSCW studies utilised the term of Grounding in Communication from communication studies and in particular Clark’s contribution theory (Clark and Brennan, 1991) which highlights the importance of grounding for communication and puts it at the same level as the communication medium. Common Ground, which is an essential concept in CSCW, refers to the “knowledge that the participants have in common, and they are aware that they have it in common” (Olson and Olson, 2000, p. 157). Common Ground goes beyond this simplistic definition for a much elaborate concept of grounding in communication, which must be established through an intrinsic process. Proponents of common ground promote that its successfulness is a key indicator of a successful collaboration (Gray, 1989).

Chapter 2 Literature Review

Asserting its importance in CIS, Hertzum attempts to establish common ground as the main pillar for collaborative information seeking (Hertzum, 2008, 2010). He argues that CIS is achieved through two main activities: Information Seeking and Collaborative Grounding. As he defines CIS as “the information-seeking activities performed by actors to inform their collaborative work combined with the collaborative-grounding activities involved in making this information part of the actors’ shared understanding of their work” (Hertzum, 2008, p. 958). This is based on the assumption that while CIS is a group activity, it is being studied and discussed in contrast with what Hertzum refers to as individual reductionism and group reductionism, yet CIS is much more of assisting common ground.

To support that theory, extensive studies of electronic medication records were conducted, involving surveys and follow-up interviews as well as an observational study. In particular, the studies focused on incidents in the medication process caused by erroneous collaboration between the users of the electronic medical records system, which the study labels as a breakdown in collaborative grounding between users of the system. The author stressed that these were breakdowns in collaborative grounding rather than breakdowns in CIS. The study argues that improvements in the grounding process will enhance the collaborative search activity(Hertzum, 2010).

2.4.7 Algorithmic Mediation Search algorithms and optimisations are the essence of modern day search engines (Ingwersen, 1992; Baeza-Yates and Ribeiro-Neto, 1999). However, applying explicit division of labour collaborative activities to optimise a search experience the through has not been thoroughly researched in CIS prototypes and systems (Sperber and Wilson, 1995). These algorithms and techniques fall beyond the scope of this review; however, the closest algorithmic technique using Collaborative Filtering will be briefly explained to contrast between how explicit algorithmic mediation is employed in CIS. Moreover, the studies that have considered the explicit form of collaboration utilised their results to aid CIS also from an implicit and backend perspective, like algorithmic solutions considering ‘communities’ by e.g. and algorithmic mediation for collaborative search (Pickens et al., 2008) and collaborative filtering e.g. (Konstan et al., 1997) have paid little attention explicit the dedicated aspect to support. While most of these and other solutions were not dedicated towards explicit CIS process intentionally, their contribution to that particular activity had been important in defining the field as well as the practices.

Several approaches that mimic individually centred search engines that and optimise collaborative explicit search like a simple list of hits that have already been seen by another user or that are contained in other user’s current ranked list will count towards that approach. These improvements to the search engine algorithms learning from other users’ previous search queries to produce

Chapter 2 Literature Review relevant results as part of their functionality. Ultimately, these resulted in the improved relevancy for search results by from an implicit manner, whereby the users’ queries and search histories are fed to the engine algorithm.

In their pursuit to achieve a personalised search experience form of individual and community perspective, Smyth et al. demonstrate that the use of the collaborative nature of search will achieve better results than simple reliance on individual search history. Motivated by the target to reduce vague queries; Collaborative Web Search (CWS) premises is that an intersection between social networks and collaborative search may provide a more accurate set of results (Smyth, Balfe, Freyne, et al., 2005). The basis of this is that the search patterns of an individual are “repetitive and regular”; that is the repetition of and the regularity of selecting result hits queries between members of a community. Smyth followed up on the notion search communities, describing examples of employees of smaller companies or pupils in class as search communities “Individuals search for similar information in similar ways” (Smyth, 2007). uses the repeated queries of members of a team to deliver a relevant set of results to collaborators using a community profile is sole use of algorithmic mediation (Freyne and Smyth, 2006; Smyth, 2007; Coyle and Smyth, 2008). Other examples of Algorithmic mediated prototype systems include Cerchiamo where the information retrieval algorithm explicitly catered for synchronous collaborative search (Golovchinsky et al., 2008)

2.5 Implementation of Awareness Cues and Mechanisms in CIS

Following the discussion on Awareness and its different definitions and types in [2.4.1], it is apparent that the success of CIS systems depends on providing supporting awareness to the collaborators, their actions, and the creation of a shared outcomes. This section will focus on the design and implementation of the visual and contextual awareness controls mechanism in several key studies.

The main challenge and concern for awareness design are the nature, amount and frequency of awareness cues which has been reflected in several awareness studies. Kolfschoten et al. highlight:

“it is a well-known problem with awareness mechanisms that the amount of information being presented to the user may appear inappropriate - either too little or too much information is displayed. We suggest that this problem can be solved by contextualizing awareness support with respect to the concrete task domain which is addressed by CSCW software” (Kolfschoten, Herrmann and Lukosch, 2013, p. 108).

Chapter 2 Literature Review

Moreover, it awareness mechanism and cues are context sensitive and have to be considered carefully,

This indicates the importance of providing an interface that matches the complexity and nature of the task. (Shah and Marchionini, 2010, p. 1984)

Therefore, careful attention to the visual and contextual awareness cues is identified as a necessary process in the design and implementation of CIS systems, and the literature suggested that most of these elements are found in the main collaborative workspace.

In the context of CIS, the collaborative workspace, typically refers to the interface that includes SUI (Search User Interface) and or SERP (Search Engine Results Page). In addition, several additional collaborative and awareness cues are added in other windows, toolbars and sidebars, and other visual container elements. To understand these various approaches to design and implementation of awareness cues, a review of the current key articles in the explicit CIS systems are summarised below in [2.5.1]. These systems focus on the interface to aid the collaboration in search and highlight the visual component of awareness aspects. The last column of the table highlights the cues and will be discussed accordingly in the upcoming sections.

Typically, the main SUI elements identified by (Wilson, 2011, chap. 3) and detailed in [2.1.3] are identified for CIS. For awareness cues, three main categories of visual and contextual awareness mechanisms and elements are identified from the recent studies: Search Activities, Timelines, Ratings and Annotations and detailed next.

2.5.1 Search Activities Perhaps the most important awareness cues are the search activities of the collaborators. These are the keywords searched, i.e. the queries or query awareness, and the results they viewed, i.e. the documents or pages viewed (Morris and Horvitz, 2007; Shah, Capra and Hansen, 2014). It also can include other document specific activities such as Bookmarks, Favourites or Highlighted results (Nichols and Twidale, 2011). For the queried keyword(s), most system designers opt to have them appear as a list of these in a sidebar timeline, usually with metadata. Coagmento v3 [Figure 2-14] expands the activities sidebar by including a tabbed container for shared.

ResultsSpace [Figure 2-16] limits that to the last 10 queries by the user, and last 10 queries by the rest of the group collaborators, implying an aim to reduce the number of clustered keywords with no usernames. CoSense [Figure 2-12] uses a tabular form with the collaborators on the horizontal axis and a chronological time stamp of the vertical axis.

Chapter 2 Literature Review

Figure 2-12: CoSense - Keyword History (Paul and Morris, 2011, p. 86)

Apart from the sidebars, CIS systems include some search activity awareness cues in the results themselves. Querium v2 [Figure 2-13] includes a histogram near each result where the bar height indicates the degree of match to each query.

Figure 2-13: Querium v2 – SUI / SERP (Golovchinsky, Dunnigan and Diriye, 2012, p. 1804)

An extended list of search activities is usually referred to a set of search histories or summaries. It has been referred to as fusion and defined as “The document history, also called “fusion” and “fusion query”) combines ranked lists of all queries in a session to produce a single list of documents retrieved in a session.” (Golovchinsky, Dunnigan and Diriye, 2012, p. 1802). This is included in Querium v2 through tabs representing the various activities and can be filtered by different criteria such as user or data and time (Golovchinsky, Dunnigan and Diriye, 2012, fig. 2). In Coagmento v3, their approach

Chapter 2 Literature Review for visited history are snippets, or small thumbnails, that presents the main workspace for collaborators as shown in [Figure 2-14].

Meanwhile CoSense, which aims specifically to aid sensemaking and search strategies in CIS, includes several search summaries under its “Search Strategies” window, shown in [Figure 2-19]. This includes a variety of visual cues to aid collaborators through visualising the query and browsing activity of individual group members and of the group as a whole. Examples include a tag cloud of the visited URLs and searched keywords that are typically used with social tagging (folksonomies). was found to be useful in browsing or non-specific information discovery (Sinclair and Cardew-Hall, 2007) and in CoSense was found to provide an initial starting point for resuming or joining an ongoing search session. Other visual cues includes charts with basic statistics such as the number of searches and the number of visited URLs.

Chapter 2 Literature Review

Figure 2-15: CollabSearch SERP Interface (Yue, Han and He, 2014, p. 822)

Figure 2-16: ResultsSpace SUI/SERP (Capra et al., 2013, p. 3)

Chapter 2 Literature Review

Figure 2-17: Instant Messaging in CoFox (Perez, Leelanupab and Jose, 2012, p. 265)

Figure 2-18: CoSense - Workspace (Paul and Morris, 2011, p. 88)

Chapter 2 Literature Review

Figure 2-19: CoSense - Search Strategies Summary (Paul and Morris, 2011, p. 85)

2.5.2 Timelines The use of visual timelines in modern major social networks such as Facebook, Twitter and Instagram is featured prominently in their main user interfaces. These timelines display posts or updates, usually in chronological order along a horizontal or vertical timeline and include several types of activities. Facebook, in particular, has a dedicated section in user page titled Timeline to indicate the user’s personal activities like posts, likes, and friendship additions. The use of timeline in CSCW and CIS systems has been also studied in several contexts to extend basic textual history logs to more interactive and visual controls. Four examples will be demonstrated next.

To facilitate awareness in their Classroom BRIDGE software, Ganoe et al. implemented a horizontal timeline of activities for school student projects. By letting this timeline act as the main repository for students to access their project documents on their desktop, the authors noted very little cost to users in terms of synchronous and co-located collaboration after interviewing them. Moreover, the entire class groups’ timelines were shown on a larger screen with a public time gave a sense of awareness across the groups (Ganoe et al., 2003). A more recent example of the use of visual timelines was explored by Batrinca et al. to coordinate a critical process of collaboration on manoeuvring a space shuttle, which includes several subtasks, between two different remote locations. Simulating the

Chapter 2 Literature Review process, they experimented with participants using a visual timeline versus the default method of text- based communication. Despite the timeline including all the needed features, the statistical analysis showed no conclusive evidence of better performance in terms of shorter time periods to perform the tasks. Nevertheless, the comments collected from the participants mentioned liking the timeline, yet unfamiliar, in addition to some implementation lagging in the visual timeline (Batrinca et al., 2013).

In a different context, Bohøj et al. used their CaseLine system to manage the maternal and post- maternal leaves for parents in accordance with the Danish laws. The browser plugin, shown below in [Figure 2-20] allowed three collaborator types: parents, municipal workers, and employers, to keep track of the pregnancy leave dates along with the required documents. The timeline provided interesting qualitative insights of collaborating parties with different interests but with a common goal of coordinating and served as a plan and also as a record for asynchronous and remotely located collaborators. (Bohøj et al., 2010).

Figure 2-20: CaseLine timeline (Bohøj et al., 2010, p. 528)

Focusing on CIS, in CoSense, the activities are displayed chronologically in an expandable tree. The activities are identified by icons, which are include searched keywords, visited URLs, comments and includes the chat threads with a small icon of the user profile as shown in [Figure 2-21]. In their observation study in mixed remotely / co-located and synchronous and asynchronous settings, Paul and Morris found that the timeline enhanced the sensemaking (summarised in [Table 2-2]) and the search handoff process from one collaborator to another is an asynchronous search process. It was also accessed by 83% of participants as the second most viewed section in CoSense and found to enhance and evolve their search queries. Moreover, in the post-study questionnaire, they found it better to view the timeline to discussed visited websites than the chat feature alone (Paul and Morris, 2009, 2011).

Chapter 2 Literature Review

Figure 2-21: CoSense - Timeline’s implementation (Paul and Morris, 2011, p. 87);

These studies of timelines showed increasing interest and benefit from the collaborators when timelines were included and compared, coupled with preference to utilise the timeline. However, with usage analysis usually not showing significant improvements in various measures such as the aforementioned case with space station timeline planning (Batrinca et al., 2013).

2.5.3 Ratings and Annotations Relevance has its roots in information science and in particular information retrieval algorithms, but is also important in cognitive based IIR (Ingwersen and Järvelin, 2005, sec. 4.3.3). In the context of CIS, relevance ranking or feedback does not necessarily need to be used with any algorithmic mediation as discussed in [2.4.7], but can be used for highlighting and rating results of the collaborators within a particular search group. Current CIS systems employ these concepts to various degrees, and some of the visual awareness examples are discussed here.

The Querium evolutionary design process documents this focus in their changes in the several iterations of their CIS system, whereby the relevance judgments were replaced with a streamlined thumb up and thumbs down icons, as shown in [Figure 2-16]. That method resembles SearchTogether approach, shown in the summary pane [Figure 2-22]. A further step is ResultsSpace, as it shows the accumulative number of collaborators rating by indicating the total number of rating per search result inside a three colour-coded indication represented as: High, Maybe or Low relevance to the search topic, as shown in [Figure 2-16] (Capra, Arguello, et al., 2012).

Chapter 2 Literature Review

Figure 2-22: SearchTogether Rating - (Morris and Horvitz, 2007, p. 3)

The use of star ranking is also used in the context of CIS. Expanding that, CollabSearch uses a five-star rating to include the relevancy of results by collaborators, as shown below, but does not indicate any explicit utilisation of these rating, nor does it show these on the results page, but only on the bookmarked page. CollabSearch also allowed the annotations to be associated with the bookmarked results, which is being unused with the other systems, as they only use the results’ own summary or description.

Figure 2-23: CollabSearch - Rating and Commenting Interface (Yue et al., 2014, p. 47)

Commercially, an experimental feature from Google in 2008, known as SearchWiki, briefly allowed users to vote and annotate search result within a community (selected group members), as shown in [Figure 2-24]. The annotations by other members appear below the search results as speech balloons. A better implementation is used in the annotations and comments for search results is used with ezDL as well, which links the notes on the side of each result in the SERP (Search Engine Results Page), this enabling the collaborators to have a click on the corresponding icon to view the annotations and comment [Figure 2-25].

Chapter 2 Literature Review

Figure 2-24: Google SearchWiki (from http://justinhileman.info/article/searchwiki-googles-customized-social-search-is-back/)

Figure 2-25: ezDL SERP (Böhm, Klas and Hemmje, 2014b, p. 36)

Chapter 3. Methodology

This chapter focuses on the methodological approach used in the research. Grounding this research in an established methodological standpoint ensures the outcomes in both its design and implementation phases and evaluation phases contribute to the knowledge of the domain. Moreover, it aligns the research question and objective to the methods used and the results analysis (Saunders, Lewis and Thornhill, 2009).

This chapter covers two main sections. In section [3.1], the iterative and incremental approach used by the key studies identified in the literature review is explained and examined in terms in of approach and use. Next, section [3.2] presents an overview of Design Science Research (DSR) which is the research methodology used for this research. Section [3.3] summarises the methodological approach adopted.

3.1 Iterative and Incremental Approach

Since the main scope of the research is in the CIS context, the main studies discussed in the last two subsections of the previous chapter are relevant here. In particular, the main studies using the design and implementation of CIS systems and evaluations summarised in [Table 2-5] demonstrates the application of an iterative and incremental development and evaluation approach designated for this research.

This approach is demonstrated in the development of SearchTogether versions. It started with a browser plugin application that focuses on the aspects, awareness, division of labour, and persistence. This porotype was evaluated through a qualitative user study using semi-structured interviews and also through a log analysis of the features used (Morris and Horvitz, 2007). This is followed by the improved CoSense which featured refined summary pages, enhanced awareness such as a graphical timeline and chat-centric view, and collaborative functionality such as a collective workspace between collaborators in two further studies (Paul and Morris, 2009, 2011). Another ‘fork’ of SearchTogether is CoSearch, which is designated to work specifically in co-located search sessions where, for example, a collaborator works on a desktop and another works on a mobile together (Amershi and Morris, 2008). These three sets of studies exemplify using an iterative research methodology where improvements and enhancements are deployed in the design and implementation.

A similar approach is also used in Coagmento versions and the introduction of improved features and redesigned awareness mechanism. It also evolved from using browser toolbars (v1) (Shah and

Chapter 3 Methodology

Marchionini, 2010; Shah and González-Ibáñez, 2011) to a full-fledged web-based application (v3) used in other CIS studies (Shah, 2012, sec. 6.4.2.2; Kelly and Payne, 2014).

Other studies used a variation of this incremental method, where the various software iterations are discussed and improved based on user studies in feedback within a single publication. Querium design and implementation method illustrates this evolutionary approach of a series of design and implementations cycles, detailing the improved features along the difficulties faced and design modifications per iteration (Golovchinsky, Diriye and Pickens, 2011; Golovchinsky, Dunnigan and Diriye, 2012). This approach is also apparent in the development of MUST, a prototype CIS application for sharing the search results of the health and medical library PubMed for a large hospital staff. In that study, an older prototype, MUST, was initially used but was limited by technical and functional issues when tested. Therefore, Reddy et al. identified and addressed that in the improved MUST software (Reddy, Jansen and Krishnappa, 2009).

This research follows a closely similar approach, albeit using the formal Design Science Research (DSR) approach, along with a proper review and justification for applying it. Therefore, the selected methodological approach which is described and justified first in section [3.2]. In that section, the main selected process aspects are presented [3.2.1] and the philosophical aspects of DSR in [3.2.2].

3.2 Design Science Research (DSR)

Design, put simply, “is achieving goals within constraints, […] and choosing which goals and constraints can be relaxed so that other can be met” (Dix et al., 2003, p. 193), in which the goals focus on the what, who and why of the product designed. In particular, for software design and implementation, a suitable and holistic approach to understand how research can be conducted and knowledge can be produced is known as Design Science Research (DSR), which is well established in the field of Information Systems and includes several established renowned resources that detail its processes and methods (Hevner and Chatterjee, 2010; Gregor and Hevner, 2013).

In more related context, design is noted as an essential component of HCI research, in particular, as they presented by the research community iterative design efforts (Zimmerman, Forlizzi and Evenson, 2007) and can also be extended to the design of artefacts in CSCW, IIR. Moreover, DSR which has been used in a limited manner in the HCI field in such as the study of an agile software development and usability testing (Adikari, McDonald and Campbell, 2009) although it is still considered a new trend as the book title that study is published in suggest. Furthermore, Vaishnavi and Kuechler promote artefact construction as highly valued precisely for its contribution to theory (2015, p. 23), and as discussed below, it constitutes a crucial process in DSR.

Chapter 3 Methodology

3.2.1 DSR Process Model The design science research paradigm can be best explained in terms of the one of the popular DSR process cycle which is refined in (Vaishnavi and Kuechler, 2015), and shown in Figure 3-1 below along with the seven list of guidelines detailed and revised in (Hevner et al., 2004; Gregor and Hevner, 2013). This approach provides a comprehensive philosophical and methodological base for the research and is linked to the CIS literature discussed in the previous chapter.

Figure 3-1: Design science research process model (DSR cycle) (Vaishnavi and Kuechler, 2015, p. 15)

Focusing on the process to design an IT artefact, DRS process stems from the research questions that are driven by field problems and being aware of its existence, which usually is realised by the production of a research proposal as an output.

This is followed by suggested approach which entails a design proposal of an IT artefact. Next, this is followed by the artefact development. IT artefacts are defined as “those bundles of cultural properties packaged in some socially recognisable form such as hardware and software” (Orlikowski and Iacono, 2001, p. 121), which was also refined to include “the constructs, models, and methods applied in the development and use of information systems” (Hevner et al., 2004, p. 82). The development process in DSR is when the artefact devised in the design phase is being realised, depending on the nature of the artefact. The actual implementation might be using standard approaches in the implementation as the novelty is primarily in the design, not the construction of the artefact (Vaishnavi and Kuechler,

Chapter 3 Methodology

2015, p. 16). However, both the design and the implementation needs sufficient details for reproducibility, as the IT artefacts themselves are also crucial in terms of action and effect in knowledge due to its various conceptualizations as Orlikowski and Iacono stress (2001). Thus, the focus on the artefact will be a key aspect of this research.

A given rigorous artefact design draws from various methods, such as analytics, case studies, experiments, or simulations. It is worth to note that development and evaluation of the IT artefact are iterative steps; this allows the produced artefact and its evaluation to contribute to the knowledge pool referred to the DSR process cycle. Thus, the circumscription noted in the cycle is the resulting new knowledge when things do not work according to theory and where the iterative approach of design science build upon existing knowledge and produce new knowledge (Vaishnavi and Kuechler, 2015, p. 15). Once constructed, the artefact is evaluated according to the research objectives made earlier in the proposal. That should contain, in specific detail, what hypotheses are made about the expected impact and behaviour of the artefact usage in relation to the problem domain.

As mentioned, an iterative design process is selected for the experimental software used in the research. Unlike the traditional waterfall model, which typically starts with a general and clear set of requirements, through design, coding, integration and finally testing, an iterative approach overcomes the inherent problems of an incomplete or vague set of requirements. Iterative design usually ends up with a porotype IT artefact, which is likely to receive incremental additions and typically evolutionary in nature (Dix et al., 2003, sec. 6.4).

From an interface design perspective, the use of an iterative approach had been established early with case studies research of online banking and cash tills (Nielsen, 1993). In addition, the design of early collaborative and groupware systems such as Ishii et al. shared workplaces environment with the integration of real-time video and audio, which was challenging at the time. They stress that “through the iterative design of these collaboration media, we believe it is the most important to respect that skills that people use in everyday life” (Ishii, Kobayashi and Arita, 1994, p. 96). These publications lead to the use of this approach through several interactive and collaborative systems.

The conclusion is the communication of the end result of the DRS processes. Vaishnavi and Kuechler sum up the types of knowledge contribution as either firm or loose results. The former is signified by the smaller arrow from the contribution to knowledge, and the latter is which can serve as the trigger for awareness of future research along with external knowledge, as signified by the arrow on top of the awareness of problem process.

Chapter 3 Methodology

Apart from the process detailed above, seven DSR guidelines are listed below based on the provided summary by Pigneur (2013) from the full descriptions available in the core DSR literature and modified here with related refinements (Hevner et al., 2004; Gregor and Hevner, 2013). These guidelines provide an essential checklist for the research and are adhered to in the design of the software artefacts.

1. The design-science research produces IT artefacts and they are rarely full-grown information systems that are used in practice. Instead, they define the “ideas, practices, technical capabilities, and products through which the analysis, design, implementation, and use of information systems can be effectively and efficiently accomplished”, which implies the experimental and prototype nature of the produced artefact (Hevner et al., 2004, p. 83).

2. “The artefact must be relevant to a given environment” (Pigneur, 2013) in the sense that it address the problems and challenges faced by the intended field practitioners and stakeholders.

3. The artefact, moreover, must be useful as a utility and then must be evaluated because the design is inherently an iterative and incremental activity. “The evaluation phase provides essential feedback to the construction phase as to the quality of the design process and the design product under development” (Hevner et al., 2004, p. 85).

4. Novelty is similarly crucial and design research must provide a novel contribution. It must detail its implementation and ensure reproducibility and accurately represent the business and technology environments used in the research.

5. Design science research must balance rigour and relevance both in the construction and evaluation of the design artefact.

6. Design science is iterative by nature and its core research process, whereby a problem domain is constructed and a process to find an effective solution, is performed in an iterative and also incremental manner.

7. The design research results must be communicated to the appropriate audiences. The outputs described by including two main target audiences: technology-oriented and management- oriented (Hevner et al., 2004, p. 90).

Chapter 3 Methodology

3.2.2 DSR Philosophical Stance From a philosophical perspective, design science research is often compared between mainstream positivist and interpretive schools. DSR stress the developmental and measuring aspects of the designed artefact. The complete beliefs of DSR, which includes in addition to the methodology, the ontology (the nature of the reality), epistemology (nature of knowledge) and axiology (type of values) are summarised below. The following philosophical assumptions are extracted from the full comparison in (Vaishnavi and Kuechler, 2015, p. 31) with added commentary.

• Ontology refers to the nature of the reality and how the world is viewed. In DSR it is specified as: Ontology: Multiple, contextually situated alternative world-states. Socio-technologically enabled (ibid).

• Axiology refers to judgment about values. In DSR, like pragmatism, values play a large role in interpreting results, adopting objective and subjective points of views (Saunders, Lewis and Thornhill, 2009). It is defined Axiology as encompassing the following activities with a design process: Control; creation; problem-solving; progress (i.e., improvement); [and] understanding (Vaishnavi and Kuechler, 2015, p. 31) .

• The epistemological position is concerned with the nature of knowledge and how it can be acquired, is considered as one of the core factors that impact the selection of the research. In DSR, the making of artefacts allows for empirical knowledge to be obtained objectively, but with embedded in the context of its reality. The knowledge production is also linked to the iterative methodology in the design and development of the artefact. The DSR epistemology is stated as: Epistemology: Knowing through making: objectively constrained construction within a context. Iterative circumscription reveals meaning (ibid).

• Methodologically, DSR emphasises that knowledge production is incremental process made through the phases of the creation and refinements in the design, and subsequently, the implementation of an artefact is produced. The process generates understanding that could only be gained from the specific act of an IT artefact construction. This is illustrated in the iterative and incremental development process used in the interface design, followed by the integration of the middleware and backend subsystems (Nielsen, 1993) explained in the next section. In summary, the methodological approach is summarised as: Methodology: Developmental Measure artefactual impacts on the composite system.

Chapter 3 Methodology

3.3 Methodological Approach

Based on the discussion of the relevant research methodological approaches’ in previous studies, it is demonstrated that the use of an iterative and incremental set of design and implementation phases which are followed by evaluative studies to aid the process of devising an artefact to aid an identified gap, is a viable and proven approach. This research will adopt a similar methodological approach which is underpinned using the discussed DSR process model to generate knowledge through these iterative cycles. This methodology shall be apparent in the upcoming 4 chapters which covers 2 cycles of both a design and implementation case including the relevant explanation and justification, followed by an evaluative set of studies, each in a dedicate chapter.

Therefore, this research thesis content is mapped to the process model shown in [Figure 3-1] as explained next. Firstly, the Problem Awareness statement is identified and defined within the relevant literature in the research domain section [1.1], the significance and gap section [1.2], and detailed in [Chapter 2]. These are accompanied by the Proposal which includes Tentative mechanisms to address this problem which is highlighted in section [1.3].

The Development first cycle of the design and implementation of the first version of the devised Artefact is explained in the context of the selected domain is detailed in [Chapter 4], and followed by an Evaluative set of studies of the suggested artefact in [Chapter 5]. Throughout this cycle, the Knowledge contribution is accumulated through the selection of tools and techniques in the former chapter and the evaluation results in the latter chapter. To address the results of this cycle, a second cycle is presented in [Chapter 6] and [Chapter 7]. In both iterations, Design science knowledge is generated in the design and implementation chapters. Additionally, each evaluative studies chapter includes a detailed description and justification of the Performance measurements set used in each cycle, along with the results and discussion.

Finally, the Results knowledge accumulated from both studies is concluded with a set of proposed implications for design recommendations and the artefact itself as a contribution to the Design science knowledgebase, as detailed in the thesis concluding chapter [Chapter 8].

Chapter 4 Design and Implementation of SearchAware v1

Chapter 4. Design and Implementation of SearchAware v1

The chapter details how SearchAware is designed and implemented based on the identified gap and research objectives, as well as the discussed literature review and the selected iterative approach. As explained in the previous chapter, SearchAware is developed and tested using an iterative approach for development and is evaluated through appropriate user studies. Therefore, two versions of SearchAware were developed and used in this research. Version 1 (v1) is explained here.

The chapter starts in section [ 4.1] with an overview of SearchAware and the main design considerations, including the main objectives. This is followed by a detailed section [4.2] that explains the interface features and the collaboration aspects. Next, the mashup implementation approach is detailed in section [4.3], including the selection choices for the devised artefact middleware. In the final section [4.4] the system architecture of SearchAware different subsystems along with the external providers is explained. In addition, the required database to achieve its design objectives are explained.

The study results and the study participants feedback which lead to minor refinements and improvements were applied throughout this study, and is discussed in [Chapter 5]. Based on these results, a revised version of SearchAware v2, with the enhanced functionality and tweaked awareness features is developed in [Chapter 6] and tested with participants as a controlled experiment in [Chapter 7].

4.1 SearchAware v1 Overview

SearchAware v1 is an experimental prototype web-based application (web app) for collaborative information seeking that allows co-located (i.e. on different devices) or remotely-located collaborators to search synchronously or asynchrony using a selected set of one or more search engines. It features a dedicated SUI (Search User Interface) and facilitates collaboration by supporting awareness cues and mechanisms. The design for SearchAware is meant to resemble a typical SUI yet influenced by the interfacial design of key studies summarised in [2.4] with regards to the collaborative information seeking.

From an architectural perspective, SearchAware is a mashup of multiple search engines’ results in the SUI to allow users to select a search engine and retrieve results from that particular engine based on the entered keywords. The result hits from these engines appears in the same interface, just below

Chapter 4 Design and Implementation of SearchAware v1 the search keyword box and turning the page into a Search Engine Results Page (SERP). The main awareness cues will be in the form timeline that can be controlled to post group members, i.e. collaborators’ search activities that can be configured to cater for the context of its usage.

As a CIS system, SearchAware natively caters for groups of users. The intended audience for SearchAware is typically small search groups of collaborators such as dyads or triads to an expected of 6 members although, by design, the search groups can handle a maximum of 8 members per group.

SearchAware contains a controlled access authorization layer to allow only registered users to access it. Membership can be activated via the registration signup page. This feature can be overridden to disallow registrations or allow a predefined list of members by the web app administrator (admin). As soon as they sign up, members must join a search group from a list of predefined groups, which are can also be set by the admin. Furthermore, the admin can enable guest visitors without the need for registration nor specific group membership.

For this research, SearchAware v1 is intended for university-level students and higher as well as researchers. Therefore, it will be linked to the search engine of two digital libraries using their available API (Application Programmable Interface): these are Mendeley and Microsoft Academic Search8, which are clearly marked and selectable by the user on the main interface; with each search results appearing below the search entry. The selection of the academic domain is for practical reasons, and be able to recruit participants for the evaluation process. However, SearchAware is designed to be able to be quickly configured and integrate with any generic digital library as detailed in the mashup implementation approach section [4.3].

SearchAware web app (the current version, v2) is accessible through any modern desktop browser9. The next subsection will detail the design objectives.

4.1.1 Design Objectives As the research question states:

8 Google Scholar was used in SeacrhAware v1 preliminary study but was removed in the experiment due to the lack of a dedicated search API from Google which prevented the results from appearing in native manner in SearchAware and inability to track the user once the search is performed.

9 SearchAware v1 is currently deactivated due to changes in the Twitter API since the study was conducted.

Chapter 4 Design and Implementation of SearchAware v1

How can awareness cues be designed and implemented visually and contextually to aid explicit collaborative information seeking; and how can these cues affect the search interface usability?

As an explicit CIS system, SearchAware caters for group members, i.e. collaborators through visual and contextual awareness cues in an adequate and appropriate manner. The literature on awareness in CIS suggests, the pivot question is ‘awareness of what?’, what in this context is explained as the exact information about the search process that will be shared with collaborators in a CIS system and the mechanism it can be shared with.

To answer ‘what to share’, the literature focuses on the context of the search domain itself, which plays a high role in detailing the awareness cues content. As highlighted above, in this research the domain will be digital academic libraries of published documents. This selection allows for the identification of clear information about the results (e.g. the publication title; author(s); date, etc.) and the intended audience (students, academics and researchers).

The design objective for SearchAware v1 is to present these awareness cues in an automated, real- time form through the inclusion of a social network timeline within SearchAware that will automatically post user search activity cues on the timeline. This form of search activity timeline is unique for each group and populated automatically with certain search and rating activities. The timeline is embedded in a different page in the web app and, the control itself independent from SearchAware.

Utilising an independent social network timeline that can be followed independently by the group collaborators using the social network site or dedicated app is another design aspect of SearchAware. This allows the collaborators to interact with any activity through a reply, which is visible to the group members as well provided they follow the same timeline. Moreover, it allows the collaborators to follow the group timeline on using the mobile apps.

Furthermore, SearchAware contains a basic ranking mechanism. As discussed in section [2.5.3], relevance ranking is an important feature in CIS algorithms and interfaces, however as SearchAware is focused on the explicit aspect of CIS and utilises external databases, the rating will be limited to the awareness aspect of collaborators’ ratings and will be used to calculate the average rating.

The type and content of the awareness cues are discussed in detail in [4.2.4]; and the social network integration in [4.3.1] along with the justification for the selection of Twitter timeline in particular.

Chapter 4 Design and Implementation of SearchAware v1

4.1.2 Design Guidelines Two sets of guidelines are used here, both are adopted from (Shah, 2012). The first is a suggested general list of guidelines building a successful CIS environment, which are shown in [Figure 4-1]. The second is a specific list of the designing CIS interfaces, which is detailed and analysed.

1. Provides effective ways for the participants to communicate with each other. 2. Allows (and encourage) each participant to make individual contributions to the collaboration. 3. A CIS system should coordinate participant actions, information requests, and responses to have an active and interactive collaboration. 4. Supports such a discussion and negotiation process among the participants. Participants need to agree to and follow a set of rules to carry out a productive collaboration. For instance, if they have a disagreement on the relevancy of an information object, they should discuss and negotiate; they should arrive at a mutually agreeable solution rather than continuing to dispute it. 5. Provide a mechanism to let the participants not only explore their individual differences, but also negotiate roles and responsibilities

Figure 4-1: Guidelines for building a successful CIS environment (Shah, 2012, p. 22)

Moreover, Shah discusses the inherent implications of CIS systems which are mainly the cost of learning, adaption and adoptions, and cognitive load. To reduce these, he suggests five design guidelines that can be useful for CIS system designers (Shah, 2012, p. 90) which are addresses in the challenges of groupware design identified in (Grudin, 1994b). These are detailed along with SearchAware implementation in [Table 4-1] below.

Chapter 4 Design and Implementation of SearchAware v1

SearchAware v1 Suggestion based on (Shah, 2012, p. 90) Implementation (1) Understand real needs The designers need to understand various SearchAware v1 experimental awareness cues aspects of the target domain, educate the users will comprise of two main controls: the first is and managers, and design a system that can an activity timeline, based on a popular social provide a good balance of costs and benefits to network timeline, Twitter; the second is each user. relevance ranking with share controls, which is a common 5-star rating control. (2) Keep it simple The design of the interface needs to be very The interface for SearchAware v1 has basic intuitive and easy to use, as there are costs structure and layout of commercial search associated with learning a new system and engines with basic entry box and relevant adopting it. Users may feel more comfortable if criteria box. The search engine selector will the system appears very user-friendly and will appear in clearly marked tabs. The rating will help lower the costs for learning and continual be linked to each result. usage of the system. (3) Make it accessible While several of the components of a CIS system As stated above, linking the relevance rating to may be new to a typical user, we should try to each result will ease the learning curve. minimise imposing a rigidly structured system Moreover, using the automated share on a new user. The components should be mechanism to post awareness cues simplifies familiar to users and allow them to explore the collaboration aspects, with fine-grain other innovative tools provided. control of ratings and messages. (4) Provide the right tools Support for control, communication, and The purpose of SearchAware is to experiment a awareness are very crucial to a good CIS system. new approach for awareness, which the use of Tools implemented should provide these a social network timeline. Therefore, it will be features in an unobtrusive manner. the main evaluation measure. (5) Allow private working One of the requirements of a successful The main sharing mechanisms for SearchAware collaboration is independence. Users should be can be controlled by the users and the web app able to work without the pressure of being admin. The participant’s status is clearly “watched”. Though it is expected that the indicated on the main interface. The use of a collaborators will share their findings and have shared timeline for a group of collaborators will interactions that can lead to better solutions. be tested in this research. This can help in reducing the cost of cognitive load induced by the system as well as the collaboration, and bring in the benefit of individual contributions.

Table 4-1: CIS design guidelines applied to SearchAware v1; Guidelines based on (Shah, 2012, p. 90)

Chapter 4 Design and Implementation of SearchAware v1

4.2 Interface

The main interface pages of SearchAware v1 are the ‘Home’ page, which is referred here as the ‘Search User Interface and the Search Engine Results Page’ (SUI / SERP), and the ‘Interact’ page, which hosts the timeline. These are detailed next.

Building upon literature discussions on Search User Interfaces in [2.1.3], Awareness and CIS, and taking into consideration the various guidelines mentioned above, the main interface design considerations for SearchAware, the activity awareness cues on the interface during the search process. Therefore, the interface element is devised appropriately to lower the burden on the collaborators and aid their search performance.

However, before detailing the interface aspects of SearchAware, attention to general interface guidelines is listed. Hearst (Hearst, 2009, sec. 1.12) emphasise several guidelines specific to search user interfaces stemming from Shneiderman’s eight golden rules of interface design (Shneiderman and Plaisant, 2005) and other cited guidelines in HCI literature. These guidelines are detailed below in [Figure 4-2]. • Offer efficient and informative feedback, • Balance user control with automated actions, • Reduce short-term memory load, • Provide shortcuts, • Reduce errors, • Recognize the importance of small details, and • Recognize the importance of aesthetics.

Figure 4-2: General Search User Interface Guidelines (Hearst, 2009, sec. 1.12)

Following these guidelines will along the listed set of two guidelines in the first section will ensure that the devised CIS’s SUI and SERP. In SearchAware, the focus will be on the aiding the explicit collaboration between group members, so in perspective, the personalization aspect will not be explicitly catered for, but can be added at a later stage. These are also referred to in the literature as personal peripheral awareness, which was tested in Coagmento CIS system (Shah, 2013b).

Designed to resemble a typical search engine interface, which is after one successful search also the SERP, albeit with further details and functionality, the main interface elements of SearchAware v1 consists of several components and controls. A screenshot of the SUI and SERP of SearchAware v1,

Chapter 4 Design and Implementation of SearchAware v1 shown in [Figure 4-3], resembles a typical interface populated with results after a search process was performed.

This interface is proceeded by a Sign-In screen with a username and password with the ability to register for new accounts. Once authenticated, a familiar simple search box will appear, with three groups of controls: the digital library search engines selectors, Search Type selectors and Sharing control selectors. If the user inputs few search keywords, a list of the matching results, as determined by the selected search engine will appear.

These features are explained and categorized according to (Wilson, 2011) as detailed in [2.1.3] with the lettered controls refers to the [Figure 4-3] with the exception of the personalisable features, which are included in the UserName ‘Profile’ page of SearchAware and in control [Figure 4-3 G]. Furthermore, an additional Collaboration and Awareness Cues Features subsection is added to this section to elaborate on these features, and is found in [4.2.4].

Chapter 4 Design and Implementation of SearchAware v1

User name and G login control

Keyword and Digital library H viewed results A search engine sharing controls selector D Navigation menu

Comments sharing I B Search criteria control selector

C Keyword entry Relevance Rating Number of search text box J E hits Control

Sharing button Search results K column F (clickable title)

Figure 4-3: SearchAware Version 1 - Main SUI and SERP

Chapter 4 Design and Implementation of SearchAware v1

Comment entry A text box

Interaction B timeline

C Activities

Figure 4-4: SearchAware v1 - Interact Page

4.2.1 Input Features The design will utilise the simple search text box to allow the user to input keywords and passes the results to the selected search engine. The second input will be the search engine / database selector in case multiple sources are available.

A) Digital Library Search Engine selector: This tab control switches the digital library search engine form the available list of search engines. For SearchAware v1 two scholarly search engines were used: Mendeley and Microsoft Academic Search APIs, which is discussed later in [4.3.2] and in [4.3.3].

B) Search Criteria Selector: Based on the selected engine, the corresponding criteria options for the searched keywords are applied. For Mendeley, it allows search by All documents, Title and Author; for Microsoft Academic Search, it provides search criteria the same parameters with the addition of a “From Year” date selector, indicating the publication date.

C) Keyword entry text box: the primary input control. This text box does not support the autocomplete feature but supports search operators that are supported by the search engines.

J) Sharing button column: Each publication hit row will have a ‘Share’ button to allow the user to share that particular publication title and link on activity timeline. Once clicked, the button will change colour to blue from the default grey and the button label will be changed to ‘Shared’.

Chapter 4 Design and Implementation of SearchAware v1

K) Relevance Rating button column: Users can rank the publication relevance using a simple star rating, where 5 is most relevant and 1 is low relevance. The user can only rank each result once.

4.2.2 Informational Features

E) Number of Search Hits: Display the total number of search hits as supplied by the selected search engine based on the entered keywords under the label ‘Results found’.

F) Search Results: This represents the main part of the SERP, containing the search results hits ordered by the search engine. For each publication, the title and the authors information are displayed. The APIs provided limited any additional sorting, so these are displayed in the order received from the engine. Clicking on the title will highlight that particular result and will open a new window on the landing page of the search engine for that particular result.

4.2.3 Control Features H) Keyword and viewed results sharing controls: These controls will allow the user to trigger the switch to share or not share the search activities and/or the viewed results in the activities timelines.

I) Comments sharing Control: This switch will trigger the comment text box appearing in the Interact page shown in [Figure 4-4 A]. Both the controls above can be manipulated by the web app administrator to be enabled or disabled for the users.

There are also pagination controls with First, Back, Next, and Last page buttons along with page number and a ‘total number of pages’ label that appears below the results table.

4.2.4 Further Collaboration and Awareness Cues SearchAware will post four types of activities in the social network timeline, shown here as an embedded timeline within SearchAware v1 in [Figure 4-4 B] as listed below. Each activity is displayed as single post/tweet [Figure 4-4 C]. a. Searched Keywords: These are the terms used per search entry (i.e. with every click), indicating the search engine used. b. Viewed Publication: Pressing on any of the viewed results shown in [Figure 4-3 F] will post the result’s Title, with a clickable link to the publication entry page (depending on the search engine). c. Relevance Rating: Pressing on any result This will share the title of the result, with a clickable link to the publication entry / page (depending on the search engine). d. Comments: Messages on the timeline can be inserted directly using the comments section as shown in [Figure 4-4], which is the only activity type on a different page from the SUI, to separate

Chapter 4 Design and Implementation of SearchAware v1

between the automated cues being tested in SearchAware and the typical method used as implied from the literature.

Each of the above activities is posted in an automated update as tweets (i.e. twitter status updates). These activities are worded as “John searched for ‘Human Computer Interaction’ in Mendeley via SearchAware” or “Jane viewed the ‘Human-computer interaction (Dix et al., 2003)’”, for example. Each tweet the username, the action, and the result title (in case of activities b. and c.) and by default Twitter shows the date in a relative format as follows: posts in the same day are shown.

The automated posting of activities types: a. Searched Keywords., b. Viewed Publication and c. can be manipulated by each member from the SUI directly as seen in [Figure 4-3 C]. In the case of d. Comments, it disables the comment entry text box [Figure 4-4 A]. Furthermore, the posting controls can be triggered on or off by the SearchAware administrator per member, per group or by the whole site.

This timeline is accessible also on the social network site independently SearchAware v1. For this version, the timeline will be linked with the set of provided accounts, which are also linked to the predefined groups in SearchAware v1.

4.3 Mashup Implementation

A web mashup implementation approach is to be used. The term mashup started trending in the second half of the 2000s with the growing use of Web 2.0 and Social networks. Web mashups are “Web applications generated by combining content, presentation, or application functionality from disparate Web sources. They aim to combine these sources to create useful new applications or services.” (Yu et al., 2008, p. 44). Mashups are an example of reusable components with similarity to web services, yet they are focused on the interface layer of services. Usually, mashups work by utilising APIs offered by other websites and services, simple examples such as Weather and Maps services typically require some sort of registration and can be integrated into another website. More complex mashup integration such as those of displaying and using social network services and web databases provide further features and interaction between the mashed interface and the provider. These typically require additional layers of coding and more sophisticated set of data exchange which entails authentication and authorization layers and formatting.

Technically, mashups are typically offered in Service-Oriented Architecture (SOA) environment with a technology that follows web services approach. SOA is defined as “an application architecture in which all functions, or services, are defined using a description language and have invokable interfaces that

Chapter 4 Design and Implementation of SearchAware v1 are called to perform business processes” (Beal, no date). Moreover, most current implementations use a specific implementation known as RESTful services. RESTful stands for Representational State Transfer (REST) is a protocol-independent design and implementation for services offered on the web. RESTful services, as they are known, allow programs to mainly consume data offered over the Internet de facto protocol, HTTP (Hypertext Transfer Protocol).

SearchAware interface uses the mashup development approach due to the versatility available from different services to be integrated, and it allows the design and implementation of the interface to be flexible with a diverse set of offered third-party services and web databases. The intended use of the selected microblogging service and scholarly search database, therefore, can be replaced with other types of social networks and another set of database, if needed, with greater simplicity than traditional interface development. Therefore, SearchAware is extensible and meant to allow integration with current and future services to accommodate the growth of APIs and the use of mashup development. However, while mashups offer the capability to integrate services from multiple sources, that represents a challenge due to the limited control over what can be offered by such services, which hinders the development of mashup interfaces. For example, some services require certain contexts for their content to be displayed e.g. only for non-profit uses and other may require payment for usage.

APIs for digital libraries typically enables consuming services through a search engine connected to its database of documents. These search engines can be utilised via keywords, tags and bibliographical fields, just like their native services, although enforced restrictions are common. Academic publishing firms and organisations such as Elsevier, Thomson Reuters and Wiley offer several forms of restricted APIs typically through via of EDI10 (Electronic Data Interchange) or Web Services for higher education libraries, research institutes and other parties. Furthermore, these companies offer APIs to access their search features typically through their dedicated cross-publisher search or like Elsevier’s Scopus and ScienceDirect, and Thomson Reuter’s Web of Knowledge. The contents of these digital libraries are often indexed and searchable by several scholarly search engines available on the Web. For example, technology giants Google and Microsoft both offer a dedicated, cross-publisher, scholarly

10 EDI Short for Electronic Data Interchange, the transfer of data between different companies using networks, such as VANs or the Internet. (Webopedia, 2014)

Chapter 4 Design and Implementation of SearchAware v1 search in addition to other dedicated search engines of specialised fields such as the PubMed in Biomedical fields, IEEE for Electrical and Electronics Engineering and SSRN in Social Sciences.

For SearchAware v1, three APIs are selected, one social network, Twitter, and the search engines of two digital libraries: Mendeley and Microsoft Academic Search. An overview of each and the rationale for the selection and the process of integration for these services is explained below.

4.3.1 Social Network: Twitter Twitter was launched in the summer of 2006 as a microblogging social network where the fundamental form of interaction between its members is through posting short blog posts or status updates called tweets that are limited to 140 characters per post. Users subscribe, follow, other accounts see their tweets ordered chronologically on an interactive timeline which allows them to reply, favourite or retweet (i.e. re-share) any tweet. Twitter became one of the most popular global social networks and attracts over 300 million active users in 2015 (Welch and Popper, 2015).

Twitter gained the attention of researchers in CSCW and collaboration early on due to its quick popularity. One of the earliest studies about the use of Twitter in collaboration activities was by Honeycutt and Herring in 2009, where they developed a script to collect Tweets for a duration of 12 hours and coded random samples of tweets for a coding analysis. In one of their finding, the responses found users replying to a user questions moreover, “In the context of this study, it suggests that if Twitter is used for collaboration, communication in dyads or small groups would be more effective than large, open discussions.” (Honeycutt and Herring, 2009, p. 9).

In SearchAware, Twitter is selected for three main reasons: firstly, its simple and short form of tweets is suitable for the subtle, lightweight awareness cues specified in SearchAware objectives. Links can be easily embedded in the tweet, and the maximum number of characters makes it idle for short status updates. Moreover, the fact the tweets are represented using a chronological timeline. Twitter API allows posting of updates and embedding of the timeline from within other websites. Secondly, the availability of Twitter dedicated websites and cross-platform apps allows collaborators to connect to the group dedicated timeline from multiple. Finally, it is less burden for participants to connect accounts of Twitter, especially at time when the largest social network, Facebook, uses the ‘Friendship’ metaphor to connect to accounts, while Twitter uses the ‘Follow’ to connect the account. Moreover, it was found that Twitter is used for “informational purposes … and for its utilitarian value and cognitive stimulation” more than Facebook (Hughes et al., 2012, p. 567), which matches the objectives for the SearchAware.

Chapter 4 Design and Implementation of SearchAware v1

4.3.2 Mendeley The first choice of the backend digital library is the scholarly search engines is Mendeley. Conceived in 2008 as reference management system with both a web interface and desktop standalone application, Mendeley integrated the users’ publication database metadata with privileged access to scientific journals and conferences publishers’ catalogues. Growing in popularity, Mendeley is an established social network, literature search engine and cloud-based reference management system with sharing capabilities. In April 2013, Mendeley was acquired by Reed Elsevier, one of the largest media conglomerates but continues to operate independently and had over 2.5 million users11 that September.

Mendeley offers an open API (Application Programmable Interface) to manage user collections. Moreover, it also offers access to its large crowed-based database of publication metadata which is utilised by SearchAware.

The API offering by Mendeley was very limited at launch with crippled with frequent faults and occasional downtimes. These issues caused SearchAware, one of the earliest users of its API, to be slow and buggy. However, with the increase in capabilities and further stability of the APIs, Mendeley integration became more stable in SearchAware, allowing its users to perform the requested operations.

4.3.3 Microsoft Academic Search This search engine is a product from a team at Microsoft Research in 2010 and covers academic journals and conferences through keyword and categorical. By 2012, they had indexed 38 million publications and 18 million authors12. A unique aspect of Microsoft Academic Search is its extensive API features, which are available without any charges.

For SearchAware, Microsoft Academic Search represents an excellent open access search engine with large academic providers. Its API provides a comprehensive list of search functionalities that facilities the features for a customised search system such as unique id, structured set of authors and linkable references and citations.

11 Mendeley Has 2.5 Million Users! Available at http://blog.mendeley.com/start-up-life/mendeley-has-2-5- million-users/ [Accessed on 24/03/2014]. 12 Microsoft Academic Search is now shut down and replaced with Microsoft Academic and does not include the search functionality.

Chapter 4 Design and Implementation of SearchAware v1

4.3.4 Data Exchange Format: JSON As mentioned above, all three middleware mashup data exchange is performed using. The format of the data available in the API is in JavaScript Object Notion (JSON), which is a lightweight, text-based, language-independent data interchange format. It is considered an alternative to XML but currently is preferable due to its efficient encoding system and is faster and uses fewer resources than its XML counterpart. To be able to utilise JSON-encoded data exchange in SearchAware, a dedicated customised wrapper needs to be implemented for each service API. This is then linked with the SearchAware main data handlers.

4.4 Architecture and Backend

SearchAware web app architecture resembles a typical mashup web application, which is detailed below in [Figure 4-5]. The mashup services appear in the cloud diagrams, while the main internal components appear inside the dashed shape.

Results (JSON Responses)

Queries (JSON Requests) Group JSON DB Converter

Search Results (SQL)

Membership Logger DB DB User Profiles User Activities (Authentication) (SQL) SearchAware (

Follow groups(s) account timeline User

Figure 4-5: SearchAware v1 architecture

Chapter 4 Design and Implementation of SearchAware v1

The web app front-end interface elements including the grids and visual controls are abstracted from the backend system to ensure compatibility with any external digital library or Social Network.

The narrow arrows represent requests to these services, often with parameters. The bold arrows represent the responses to SearchAware. All these data exchanges need to be converted to JSON when interacting with SearchAware using the JSON .NET converter.

The main technologies and development tools used to design and implement the interface, integrate the middleware API and program the backend systems connectivity to the database management system, are listed below.

1. Microsoft Visual Studio 2010 (VS2010)

The integrated Development Environment (IDE) is VS2010 provides a full set editor to develop interactive web applications for the following main web page editing components: structure, layout and scripting using the following standard technologies: HTML 4.01 (Hypertext Markup Language), CSS (Cascading Style Sheets 2) and JavaScript, respectively. Moreover, Visual Studio provides a full set of tools to download several community-developed add-ons and extensions that will be used in both the interface and development. The rational to select Visual Studio can be summarised as:

• Powerful IDE for the design and implementation web apps. • Membership and Profile management systems, which is discussed later in this section. • The ability to use third-party components by independent and commercial developers available through its package manager, NuGet, which included several components to simplify the integration of the mashup services.

2. Microsoft ASP.NET 4.0 (with Visual Basic .NET 2010 (VS2010)

This server-side scripting language, which can be written in either Visual Basic or C# is the default choice for VS2010. It provides a flexible set of frontend visual web controls and a fully object-oriented programming environment at the backend with built-in libraries to connect to various DBMSs (Database Management Systems).

3. jQuery and jQuery UI library jQuery is prebuilt JavaScript library that deals with the browser DOM (Document Object Model) in simpler code than JavaScript. jQuery UI is extension library that uses jQuery to add dynamic effects to

100

Chapter 4 Design and Implementation of SearchAware v1

Examples of the controls available in this library are the relevance rating control [Figure 4-3 J] used to allow users to rate the result and the Share button control [Figure 4-3 G] used to commit the rating.

4. Telerik UI for ASP.NET Ajax

A propriety set of visual controls that can be integrated with a web application based on ASP.NET. These visual controls extend the functionality of a web application interface and can be bound to various data sources such as databases, web services and social networks. The digital library search engine selector tab [Figure 4-3 A], the search result table [Figure 4-3 F], and the star rating control [Figure 4 3 J].

5. Additional Libraries

As explained in the in the first point 1, using Visual Studio with NuGET package manager allowed the inclusion of third-party packages and add-ons to SearchAware. Three main external packages are used in SearchAware, all of which use GNU General Public License (GPL) or one of its variations.

• Hammock: is an HTTP API library for .NET framework that simplifies consuming and wrapping RESTful services. • JSON.NET: a JSON conversion library for .NET that allows serialising and deserializing JSON encoded data into textual or XML data. • TweetSharp: is a Twitter API library which includes the mapping of the main functionality offered by Twitter for .NET framework access. This includes basic authentication, posting and reading tweets.

The backend for SearchAware is the subsystem that manages all the data generated and saved for the usage and analysis, and mainly comprises of the databases adapters to connect to the Database Management System (DBMS) which includes three distinct databases. These are seen using the cylindrically shaped diagrams in [Figure 4-5]. The other services include JSON converter to convert the activities and interactions of users to the external mashup services, which are in this case: Mendeley, Microsoft Academic Search and Twitter. These converters use the additional libraries listed in the previous section.

The main database used in SearchAware is SQL Server 2008 R2 Web DBMS, with SQL Server Management Studio. The database is used to store the membership and profile systems data, the logger data, both of which are described next. The membership and profile management system Managed by ASP.NET Membership system through Internet Information Services (IIS) Server. This allows the creation of users, managing profiles and user related activities. A dedicated second

101

Chapter 4 Design and Implementation of SearchAware v1 database is used to store the activities for the groups of collaborators. SearchAware allows members to join several groups, each with their own timeline.

Furthermore, SearchAware includes a logger that logs the all the user interactions including all the group interactions mentioned above, are stored along with the individual actions of the user. These are used to perform the evaluation analysis for this experimental web app system.

One more aspect regarding the backend is communication technologies. In the design of SearchAware, several communication mediums were considered, and the integration of a built-in instant messaging subsystem was ruled out because of the implementation overhead to develop a real-time interaction with the selected development environment (Microsoft .NET framework, as explained in the next section) and the design objective of having short and limited instant messaging between the group members, which the comments. These can be integrated or added at a later stage as commercial instant messages (IM) systems can be used, as with previous experiments highlighted in the communication aspect of CIS review in subsection [2.4.2].

102

Chapter 5. User Study

Overall, two studies using SearchAware as a collaborative information seeking tool are conducted in this research. The first study is the formative exploratory usability evaluation study using SearchAware v1, which is covered in this chapter.

This study chapter starts with an explanation of the study objectives [5.1], which is followed by the method section [5.2] explaining the context and procedures of the study, the participants using SearchAware v1 and the type of tasks assigned. Section [5.3] explores the main study results of SearchAware v1 usage and performance. In Section [5.4], an analysis of the collected students written reflections from the participants is presented. The final section [5.5] provides an analytical discussion of the study results and provides several recommendations based on the analysis results for the next iteration of SearchAware.

5.1 Objective

As highlighted in chapter 3, the DSR methodological approach stresses the use of an evaluation step to ensure that the design and implementation of the IT artefact find a solution to the stated problem. The artefact is tested through the use of SearchAware v1 CIS system designed specifically to include a social network timeline that is automatically updated with collaborators search activities, using the timeline as a shared and persistent awareness mechanism of the collaborative search activities.

The second part of the research question:

and how can these cues affect the search interface usability?

This is achieved by testing the effectiveness of SearchAware in terms of providing appropriate functionalities for people working together while seeking information through a user study. The study is a formative exploratory usability evaluation study. Formative in the sense that its intention is to make recommendations for improvement, and exploratory in the sense it aims to show that the introduced artefact works in practice in the context of a typical user (Purchase, 2012, sec. 1.1). Therefore, the second part of the research question will be answered through the evaluation of participants’ feedback of using SearchAware v1 in the context of an academic module, where it can be integrated with a collaborative task such as a group report writing.

Thus, the objective of the user study is to examine and test the functionality of SearchAware v1 through and obtain preliminary system-usage method and user feedback of SearchAware to

103

Chapter 5 User Study understand the information seeking collaboration aspects that can occur a naturalistic academic context using this novel approach. Furthermore, insight from the students’ usage and performance is gathered through verbal and written qualitative feedback.

5.2 Study Method

The course used as context for the first pilot study is titled Virtual Teamwork which is compulsory course for undergraduate students enrolled in the Information Technology and Management B.Sc. program and optional for all undergraduate students of the Business school. The course contained four practical laboratory sessions in the computer labs. In these labs, the entire class 29 students were divided into 8 groups of with each group having 3 to 4 students.

Following the completion of all lab sessions, an assessed coursework was required from the students where each group present their activities and experience about a specific lab session. Additionally, the students had to underpin their presentations with class lectures and notes and perform further research about the topic.

As part of the learning experience of the course, the group members were asked to prepare the presentations collaboratively without meeting in a face-to-face setting. Instead, the students are asked to utilise ICT tools to communicate and write the presentation. Although, the choice of communication channels for all the tasks was left the students, SearchAware was presented as preferred choice for using remote collaboration to search for publication and references about their presentation topic. Henceforth, the students will be referred to as the study participants.

In order for the participants to understand and familiarise themselves with this new tool, SearchAware was presented in the dedicated lab session. The presentation explained SearchAware main features and capabilities along with a functional demonstration of its usage. Two main benefits of using SearchAware were stressed in the demonstration. The first is that SearchAware was built with two main scholarly databases which included Mendeley and Microsoft Academic Research. Secondly, SearchAware timeline dedicated feed for the research group and will not include unrelated activity, oppose to other communication mediums such as email or social networks. The participants were encouraged to contact the system designer for any queries or issued faced. Additionally, an instructions document was also handed to the students and uploaded to the course website.

In order to use SearchAware, the participants had to create accounts in SearchAware which included a user membership system management system, and they were asked to sign up using their university email address and use a specific form for the username (firstname.lastname) in order to be identifiable

104

Chapter 5 User Study to their group members. Once registered, the user account management page allowed each participant to register in the designated group. These groups were predefined in the system and corresponded to the actual student groups in the class. Therefore, a total 8 groups were defined, with letters A to H, with each group having 3 to 4 members. Once a participant is registered in a group, he/she can use SearchAware. Groups had between six to two weeks to use present their work and afterwards had to submit their reflective report towards the end of the semester after their completing their presentation. For this particular study, the sharing options were switched on by default, and the students did not have the ability to switch them off.

Furthermore, after the groups completed their presentations, each participant had to individually write a reflective feedback report which is also a part of the module coursework. The reflective report description asked, amongst its requirements, about the usage of the communication and collaboration tools used in the presentation preparation, and relate that to the module content. While using SearchAware was not a requirement nor an assessed part of the coursework, both the presentations and reflective reports were compulsory.

5.3 Usage Results

As detailed in the Study Method subsection [5.2], two forms of the data were collected in this study: The exchanged interactions through SearchAware and the reflective report. This section focuses on the exchanged interaction included all the following activities in SearchAware, which are also posted on the corresponding Twitter timeline: Searched Keywords, Viewed Results, Ranked Results and Comments. The logger data were not analysed in this study as its main aim is not to fully track the user detailed statistics.

In total 28 out of the 29 students registered for SearchAware, with 26 students using SearchAware at least once, and are considered as participants. The data had to be consolidated and cleaned because few students had to register twice, and few registered in the wrong groups, which was a result of the last two groups actually joining the two groups, therefore groups A and B had, in addition to their own members, the members of groups G and H, which made their results inflated. Therefore, the group distribution analysis will be discarded, and the focus will be on individual participant performance. After performing data screening and cleaning. A summary of the results is presented below in [Figure 5-1].

105

Chapter 5 User Study

Activity Summary Activites per particpant (Grouped) 20 Group A Group B Group C Group D Group E Group F 40, 15 28% 10

102, 5 72%

0 A1 A2 A3 A4 A5 A6 B1 B2 B3 B4 B5 B6 C1 C2 C3 C4 D1 D2 D3 D4 E1 E2 E3 E4 F1 F2 F3 F4 Search View Search View

Figure 5-1: Participants activities summary chart

A total of 142 interactions were recorded, and only the Search and View activities were used by the students. Out of which 72% were search activities, and 28% were view activities, as seen in [Table 5-1] below. The average of activities is 5.1 per collaborator, with 3.6 for the search activity and 1.8 for the view activity. No recorded ratings nor comments were recorded. These numbers can be broken as follows:

Type No. of Activities Average Std. Deviation Search 102 3.6 2.2 View 40 1.4 1.8 Total 142 5.1 3.2

Table 5-1: Descriptive Statistics for the Participants Activities

As for the social network aspect, only 10 participants in total followed the 6 social network account on Twitter. Here again, some participants did not follow the proper groups initially, and when asked to follow their own group, some did not adhere to that, while others felt it is not needed for the research project. These were distributed as shown in [Table 5-2] below. As for the external interactions on Twitter, there were only 3 external interactions in the first group from a single member commenting on his own findings.

Group Timeline 1 2 3 4 5 6 Ʃ Followers 10 3 (50%) 3 (50%) 2 (50%) 1 (25%) 1 (25%) 0 (0%) (% of the group) (38.5%*) * The overall percentage of the participants that used SearchAware is 38.5%

Table 5-2: Group timelines followers for SearchAware v1.

106

Chapter 5 User Study

5.4 Reflections

As mentioned, the students were asked to write a reflective feedback on their group collaborative research and presentation. 29 reflections were collected from the participating students. Out of which 20 (68%) explicitly mentioned their interaction with SearchAware. Paragraphs which mentioned SearchAware were analysed to infer the feedback on which aspects the students mentioned. These excerpts, although analysed individually, were considered in the context of the full reflective report.

Open coding was selected to analyse these paragraphs. A code in qualitative inquiry is defined as “a word or short phrase that symbolically assigns a summative, salient, essence-capturing, and/or evocative attribute for a portion of language-based or visual data” (Saldaña, 2009, p. 3). The coding scheme adopted is resembles the preferable, non-preferable comments and additionally any other comments. Thus, the categories are Positive, Negative and Commentary. Commentary included any other mentions that did not meet the main categories. For example, mentions of desired features or particular usage of SearchAware fit into the latter category. Furthermore, each of these codes were assigned an aspect item(s) based on the particular focus of that code, either on the concept/system- wide, a particular system feature, or an aspect of collaboration.

Two rounds of coding were used to analyse the reflections. The first round was to read all reflections and identify the overall feedback from each student and locate any related SearchAware specific comment or feedback, either implicitly or explicitly. A preliminary coding assessment also was conducted at this stage. The second round focused only on the paragraphs related to SearchAware and then were recoded. Any mentions of SearchAware that only indicated using it as a part of the group work without any feedback or opinion was discarded in this phase since all students were asked to use it. Therefore 17 student feedbacks were considered for coding analysis.

In those 17 participants’ feedbacks, 45 referenced paragraphs or sentences is highlighted and expressed some opinionated feedback on SearchAware as part of the ICT (Information and Communication Technologies) tools used by the group members to communicate, research and collaborate. These references were further filtered for specific mention of SearchAware that did not include other tools as it not possible to infer the exact tool the particular feedback indicated. In total, 37 separate references are analysed. Descriptive statistics of the results of this round is shown in [Table 5-3] below.

107

Chapter 5 User Study

Topics Positive Negative Total Error 0 7 7 Productivity 4 0 4 Use of Search Engines 5 0 5 Social Network Integration 1 0 1 Usability 1 7 8 Timeline Awareness 9 0 9 Commentary* (e.g. Suggestion) N/A N/A 3 (8.1%) Total 20 (54.1%) 14 (37.8%) 37

Table 5-3: Coded references descriptive statistics

Examples of these sentences and the matching codes are presented below, with the full coding table in [Appendix 1].

Sentence Feature / ID P# Category (Highlighted coded words) Aspect 13 B4 However, having time to reflect individually on our work Positive Timeline and be able to see very fast what everyone else was Awareness doing, which is not similar or non-virtual teams, definitely improved our performance as individuals. 23 D1 in SearchAware, another group’s searches were coming Negative Usability up. This made the task very confusing of seeing who in our group had actually viewed what. 33 F3 I liked the fact that SearchAware allows me to see what Positive Timeline other members were researching in real-time so I could Awareness look at it if I needed to; this made group research more efficient and effective. It was also an indication to the team about how much research effort everyone did contribute. 34 F3 However, I had a bad user experience using SearchAware. Negative Error It often goes into an error page when I wished to browse the search results that were returned in later pages; unfortunately, I could only see the first page

Table 5-4: Sample of the excerpts from reflective summaries by the participants, and the coded words and their categorization. The full list is in [Appendix 1].

The most positive category is the timeline awareness cues, with 9 (24%) of the overall reflective statements. Participants noted the advantage such cues in speeding up their research and being able to monitor their colleagues’ performance, as participant E3 said “[SearchAware] greatly benefited the process, as members were not always available at the same time but could however view all searched documents” and participant F4 said “it reduced our duplication of efforts by showing each other’s research articles and keywords simultaneously”.

108

Chapter 5 User Study

The second aspect was productivity, which is highlighted by participants as being efficient and effective to support searching together, such as participant C4 comment: “[SearchAware] which influenced my performance thoroughly”.

Usability is highlighted as the most negative aspect by the participants, with 7 of the total 8 references in this category identified as negative references. This is apparent in several aspects: such as the confusion in selecting the proper group numbers, even though that was pre-set for all groups. Participant D1 mentioned “but provided no means to see which search results were of high or low pertinence” and “This was due to a lack of communication as to which group number we were” as participant D4 said. This suggests that the group members did not make use of the rating mechanism due to the not needing to use it to share with others if they did not feel connected.

Several participants had a buggy interaction with SearchAware that included with failed search and posting problems. This issue was identified after deployment of the web app and being used from remote hosts outside the university network, and as most participants used SearchAware for a very limited period, it had high impacted on the reported errors, which included 7 total reflections. The issue was resolved. This issue also affected the usage of SearchAware and users felt they can achieve a better experience with other CMC technologies

Finally, several participants found the inclusion of an aggregated search functionality which is offered in SearchAware very beneficial for their research. The search engine selector offering both Mendeley and Microsoft Academic Search represented two new choices for most participants.

Despite SearchAware being not exactly an aggregator in the sense that the results were combined into one set of results, the participants highlighted that aspect, as B6 mentions “I could find various literatures quickly through this technology without visiting each site”.

5.5 Discussion

The overall outcomes of the study are mixed. Although this study is considered as a pilot study to test SearchAware v1, the engagement by the participants can be considered low. The average number of search activities is 5.1 per collaborator and only includes search and view activities, with no ranking activities nor usage of commenting activities. The reflections show dependency on current used social networks, primarily Facebook, rather which participants noted due to encountered errors with SearchAware. Nevertheless, few found the integration of Twitter a useful aspect, and more comments regarding further integration with Facebook timeline, for example, are suggested.

109

Chapter 5 User Study

The focus on awareness was appreciated by the participants, and it gained the highest number of feedbacks, particularly the ability to know what others are reading and having an estimate of their progress both in synchronous “allows me to see what other members were researching in real-time so I could look at it if I needed to” (participant F3) and asynchronous manner “[SearchAware] greatly benefited the process, as members were not always available at the same time but could however view all searched documents”. This suggests that balancing between synchronous and asynchronous interaction achieved via the timeline was helpful for collaboration and raised the awareness level of the collaborator activities.

However, usability issues of SearchAware v1 affected the participants’ experience as the reflections indicate. Most of the issues happened because some participants joined the wrong group due to confusion about the exact group number. Moreover, SearchAware v1 did not have any feature to change the group once registered. By the time the participants realised that they joined the incorrect group, most simply have continued with the same search group they have joined, causing some overlapping in the group timelines. This system drawback can be attributed to two aspects: the limitation of the SearchAware prototype, and limitation of being aware which group the member should register for.

In addition, this was coupled with excessive errors in the first few days of using SearchAware, preventing participants from accessing the website and incomplete redirections to the search engines of the digital libraries. Encountering implementation problems that affect the user experience is risk identified in the design of artefacts, which is encountered in previous CIS studies, e.g. Coagmento bugs (Kelly and Payne, 2014, p. 808). Moreover, as SearchAware was used mostly for a limited time, the identification of the exact system error and rectification process was belated. Thus, this iterative approach is aligned with the use of the DSR methodological approach, and other similar studies discussed in [Chapter 3].

The study context impacted the results in terms of the reduced use of SearchAware. Although the inclusion of an experimental artefact in a naturalistic study was promoted in the literature, as detailed in the identified research opportunity section [1.2], it does carry certain limitations, specifically “Naturalistic research, on the other hand, is always carried out in a certain context, and therefore addresses the contextual nature of evaluating information” (Fidel, 2012, p. 31).This is apparent as the students essentially relied on the module references rather than doing further research using SearchAware, as demonstrated in their presentations as part of the coursework in which most participants highlight their experience with SearchAware verbally.

110

Chapter 5 User Study

Moreover, the duration to prepare for these presentations was limited for some groups, particularly the last ones as is seen from the lower performance of groups E and F, and the mix-up the last groups G and H encountered by joining the groups A and B.

However, insight from the participants’ usage and reflections provided several aspects and issues to be considered in the next incremental iteration of the design and implementation of SearchAware. A closer inspection based on this feedback and the collaborators’ comments uncovers that external timeline, despite including the proper contextual awareness information for CIS, proved to be harder to follow by the collaborators. Hence, having an external timeline did not have a strong appeal to the collaborators, and thus suggest that a CIS can be embedded within a social network, e.g. as an app or via a different awareness mechanism. Or, as will be tested in v2, have all the awareness cues visually and contextual in the same location with the results in the SERP.

The outcome of this study provides essential information and aspects for the design of a revised version of SearchAware. Moreover, the evaluation approach must be reconsidered and will utilise an experimental approach to ensure that the new and refined features are to be tested.

111

Chapter 6. Design and Implementation of SearchAware v2

This chapter presents the second iteration of SearchAware. In section [6.1] an overview of the direction of the design is presented, which includes stating the main new design objectives and further improvements from the previous version. Section [6.2] details the main interface elements and features. The final section [6.3] list the main changes in the architecture and backend for this version.

6.1 Overview

Following the DSR methodology iterative approach presented in [Chapter 3] and based on the identified aspects of awareness cues uncovered in the previous study, a revised design, implementation and evaluation techniques for SearchAware is performed. Like v1, SearchAware v2 is a collaborative information seeking web app is accessible through any modern desktop browser13.

6.1.1 Design Objectives The main design objective for SearchAware v2 is to shift most of the awareness cues visually and contextually from the previous social network timeline to be accessible from within the SUI/SERP itself, and particularly to associated with results themselves. This should allow for a seamless form of interaction to simplify awareness and access to cues for collaborators. The aim of this to minimise the shifting between windows or tabs during the search session, both in a synchronous or asynchronous form and to distribute the awareness cues visually and contextually in SERP maintaining minimal and non-distributive impact on the core search process.

One design pattern identified with the previous CIS systems is the emphasis on the keywords already used, i.e. past queries. As the previous version of SearchAware included all search activities in a single timeline, this will be added to the front page of the SUI/SERP by using a simpler history timeline.

Therefore, this version will focus on the other search activities. Three sets of visual awareness cues are intended for this design. The first is a new view activity and comment on a new result which will be easily noticeable. The second is the actual details of this view or comment which will be available and easily accessible. The third are the cues to identify the addition of new activities to a search result already viewed and commented by. To cater for the aforementioned cue changes, the novelty of

13 SearchAware v2 is available at: http://searchaware.alarrayed.info (Username: user1; password: mbsresearch)

112

Chapter 6 Design and Implementation of SearchAware v2

SearchAware 2 approach is to have the relevant awareness cues associated with each hit. These cues will appear within the results of the search on the same page, rather than having a separate tab or a sidebar.

The first type the cues will appear on the side of each result (hit) using a designated set of icons. These will inform the collaborators if a result was viewed previously, commented, or both viewed and commented. It will, however, not show who or when these activities occurred, which appears in the second set.

The second set of cues is the detailed list of search activities for any particular hit. Their design is inspired by the use visual form of expandable email threads which helps to increase the familiarity of the tool to the collaborators. In this case, these cues are only available for those results that have one or more interaction by any number of the collaborators at any given search session. That is, it will appear only if one of icons from the first set is also there. This list, hence, appears as a collapsible list and will only be expanded when need, therefore this list is visible through an ‘expand’ visual icon near a hit. When that icon is pressed, it reveals a chronological list of the all the activities including personal and group history right below the search result.

The third and final set of cues include the additional search activities that will change the result’s title into bold. This design is also similar to email threads, which typically highlights incoming new messages in bold. Therefore, by using this to indicate a new search activity or set of activities to each result will allow the collaborators to be aware of the search process. This is to address identified issues with previous systems when new search activities are not being identified to the collaborators adequately, e.g. using a notification sidebar or a daily email notification (Kelly and Payne, 2014).

All these are the main search awareness design and as highlighted aimed at allowing the collaborators to perform the search process and being aware of their collaborators’ activities in the same window without any interruptions or the need to switch windows or tabs. Moreover, the cues are aimed to be visually distinct in nature between the personal and group activities, yet still be in the context of a collaborative process.

6.1.2 Further Improvements Several improvements are also added to SearchAware v2 beyond the main deign objectives, and aim to improve the collaborative, division of labour, and communications aspects. In addition to some issues addressed from the previous study. These are summarised below.

113

Chapter 6 Design and Implementation of SearchAware v2

An Abstract button is added with each result hit. This eases the access to the search result and allows collaborators to swiftly view the Abstract (or any form of summary) if needed. A summary below the result, like most SERPs, is not used here because it might suggest a cluttered interface with especially with awareness cues. Moreover, this abstract can be used to display any form of summary information of the actual result, depending on the context.

The relevance rating in SearchAware v2 is also improved by showing the average of ratings near each result [Figure 6-2 H]. Including this type of awareness cues is aimed at reducing the overhead to check the exact number of ratings and provide the collaborators with an estimate of the relevance of their groups’ feedback. Like v1, this relevance assessment is not used in any algorithmic ranking.

Moreover, two summary pages are added to facilitate the sensemaking aspect the in collaborative search process. The first is focused on the Keywords queried, and the other includes the other search activities, i.e. views, comments and ratings. Both of which included tables of these activities that can be sorted and filtered, as these particular features were identified in previous studies (Morris and Horvitz, 2007).

The digital library scholarly search engine is limited to Mendeley exclusively because having further engines did not essentially add to the value of the research in this experiment and shifted the participants to focus on the collaboration aspect rather than the exploratory search, this was apparent in the first study as explained in the discussion section [5.5] and also have been identified in SearchTogether (Morris and Horvitz, 2007). Nevertheless, the engines of these digital libraries can be added later. Moreover, Mendeley retrieved the unique document referencing mechanism combined to the document DOI14, which is becoming the de facto identification reference for published articles. This potentially allows SearchAware to be easily converted into an aggregated SUI that spans multiple search engines at once and be able to include awareness cues for multiple engines across a unified results list.

6.2 Interface

The main interface, as with v1, contains the main SUI and once a search is performed, the SUI becomes also the SERP interface. However, using the same aesthetics and design as SearchAware v1, the v2

14 Document Object Identifier: Short for Digital Object Identifier, a standard for online content identification and linking based on universal resource identifier. (Webopedia, 2015)

114

Chapter 6 Design and Implementation of SearchAware v2

SERP part is completely overhauled to match the stated objectives. [Figure 6-1] shows the SERP after a typical search.

In this SERP, the results list contains all the awareness cues per result hit per row. Several visual cues are available here, and these are clarified in the detailed view of the SERP in [Figure 6-2]. These include a zoomed set of images of the search results list the rating/commenting controls as well as the detailed awareness cues. The Results table includes a typical results list with a variety of awareness cues, shown in [Figure 6-1 B]. In addition, the keywords history timeline is available [Figure 6-1 D], which is aimed to be basic and by design includes the last 10 searched keywords.

As with v1, the new SUI/SERP features will be categorised and explained according to Wilson’s SUI framework (Wilson, 2011, chap. 3) in separate subsections. Note that similar features as v1 are only listed, and the reader is referred to [4.2] for their discussion.

6.2.1 Input Features

Comment Button [Figure 6-2 G]: Displays the Comment entry window to enter comments associated with the result hit as shown in [Figure 6-4].

Comment Entry Window [Figure 6-4]: A modal window that disabled the whole web app to enable the comment text entry. This is done to ensure that the web app page is refreshed once a comment is posted by pressing the Comment button.

Relevance Rating Control Button [Figure 6-2 I]: Same functionality as SearchAware v1.

Relevance Rating Save Button [Figure 6-2 J]: Same functionality as SearchAware v1.

6.2.2 Informational Features Group Awareness Mode Indicator [Figure 6-1 A]: This indicator can be configured and customised by the web admin, based on the settings. Three basic configuration modes are available: None; Personal and Group. The ‘None’ mode will disable all types of awareness and only keeps the search functionality, ‘Personal’ will maintain the user’s own search activities and rankings. The ‘Group’ mode will enable the collaborators’ search activities feed as well. In the former two, the ‘Avg’ (Average) rating column [Figure 6-1 B] in the results list is disabled. Moreover, the web app admin can customise the mode name appearing in the indicator. These configurations are only applicable on the SUI/SERP/ Regardless of the mode, the Summary pages [Figure 6-5] and [Figure 6-6] will always show the full group activities.

115

Chapter 6 Design and Implementation of SearchAware v2

Keyword History List [Figure 6-1 D]: This list will be populated in chronological order and automatically refreshed with the entered keywords by the collaborators.

View/Comment Icon [Figure 6-2 A]: The eye icon indicates that the result’s abstract or full paper has been viewed by a collaborator or the user themselves. Having the green plus sign near the icon indicates a note is associated with this results. In both cases, the details about this activity can be seen by clicking on the Expand / Collapse activities button as described below.

Abstract Button [Figure 6-2 B]: Displays the Abstract modal window as shown in [Figure 6-3].

Title Link [Figure 6-2 C]: clicking on the link will open a new browser tab pointed at the publication’s landing page in Mendeley. Bold links indicate an unseen awareness activity (view, rated, or comment) by another collaborator in the search group. The activities can be viewed by clicking the Expand / Collapse Activities button as described below.

Expand / Collapse Activities button [Figure 6-2 D]: These arrow icons appears only in result hits that are associated with any activities. Pressing these can expand (or collapse) the activities grid [Figure 6-2 F] associated with the particular result.

Activities Grid [Figure 6-2 F]: The activity expandable grid is available for any result that includes one or more search activities such as views, ratings, comments. The list is ordered chronologically with the latest activity appearing first. A Bold item in the grid indicates that this collaborator activity has not been seen the user.

Average Column [Figure 6-2 H]: The average column ‘Avg’, displays the average rating value based on collaborators relevance rating for each particular result. The complete list of ratings for each result can be accessed via the Activities Grid.

Keyword Log Summary [Figure 6-5 A]: The table displays a full timeline of all the searched keywords including the full metadata for that search activity and can be filtered and ordered.

Keyword Cloud Summary [Figure 6-5 B]: This tag cloud represents a list of keywords that have been queried by the collaborators with the font size proportional to the number of queries. Hovering the mouse courser on each keyword displays the number of time it has been used.

Activities Summary Page [Figure 6-6 C]: This table displays a full timeline of the selected activity type from either the rated, viewed or commented or all depending including the full metadata for that search activity and can be filtered and ordered. These can be exported to other formats such as Excel spreadsheet or PDF via the Export Selector [Figure 6-6 B].

116

Chapter 6 Design and Implementation of SearchAware v2

C Navigation menu

Group Awareness A Mode Indicator

Keyword History D List

Results Table B * See figure [Figure 6-2]

Figure 6-1 SearchAware Version 2: Main SUI and SERP

117

A B C

D E

Zoomed View of the search results

A. View icon : indicates that the result’s abstract / full paper has been viewed by the user. View+ icon : indicates that the result has a comment. B. Abstract button : clicking on the will preview a small window displaying the publication abstract. C. Title link: clicking on the link will open a new browser tab of G H I J with the landing page for the landing page in Mendeley. (Bold links) indicates an unseen activity (view, rank, or save) by another collaborator in the search group. D. Expand / Collapse Activities button: appears only in Hits that are associated with any activities. E. Highlighted Result: Selected hit is highlighted in dark grey. F. Activities Grid: Personal and collaborators group activities appear in an ascending chronological order. Bold text signifies a new interaction by other group members Zoomed view of the Rating, G. Comment button : Displays the comment entry window. Comment and Saving H. Group relevance ranking average: The average of all the group members is calculated for a ranked result. I. Relevance rating : The user can rate a result for relevance using a start rating from 1 to 5. J. Save button : Commit the relevance ranting, once saved it will be disabled and turn grey.

Activities and awareness cues legend

Figure 6-2: SearchAware v2 - Detailed view and description of the SERP

118

Chapter 6 Design and Implementation of SearchAware v2

Figure 6-3: SearchAware v2 - Abstract Window

Figure 6-4: SearchAware v2 - Comment Entry Window

Keyword Cloud B Keyword Log Summary A Summary

Figure 6-5: SearchAware v2 - Keyword Summary Page

119

Chapter 6 Design and Implementation of SearchAware v2

A Activities Type Filter

B Export Selector Activities Table C Summary

Figure 6-6: SearchAware v2 - Activities Summary Page

6.3 Architecture and Backend

In comparison to SearchAware v1, the main implementation changes in SearchAware are in the SUI/SERP interfaces. These include the visual awareness cues on the left results and the list of search activities in the embedded collapsible hierarchical grid for these results. Both these changes entailed comprehensive changes in the architecture of the front-end components and backend structure as well. In addition, the revamped Summary pages also required modifications to the backend functionality. Nevertheless, the overall architecture of SearchAware v2, shown in [Figure 6-7], is almost identical to v1.

Furthermore, the development and implementation technologies used in SearchAware v2 are in essence the same as v1. However, the page structure including with new cues will have to cater for several changes. These require adjustments so that the pages handle the rapid changes in their content, essentially rendering the SUI/SERP as the main application page. Therefore, technologies related to Single-Page Applications (SPAs) will have to be used. SPAs are defined “Web apps that load a single HTML page and dynamically update that page as the user interacts with the app.” (Wasson, 2013). SPAs are known to ensure a better user experience and improve the web app usability, especially for demanding tasks. This makes it an essential feature to peruse in the implementation of SearchAware v2.

120

Chapter 6 Design and Implementation of SearchAware v2

SPAs typically require a sophisticated set of technologies to function properly, as the typical web app can refresh content between pages, these SPAs should be able to refresh their content dynamically without a reload. This represented a technical challenge as SearchAware is also a mashup application that utilises external data and services for its operation. Therefore, a solution is needed to allow the page to be refreshed, with both with cues and with external sources and without the need for a page reload. This is achieved by a workaround to force a partial reload with every user activity that is not on the web app page. The details for this is explained in the last paragraph of this section. By this, SearchAware v2 is considered as an SPA.

As mentioned, the backend contained several main changes, three significant changes. Firstly, the backend ‘Group’ system is revised to accommodate changes in the collaboration and group history management. The Group database now includes, in addition to search activities entries found in SearchAware v1, a “viewed-by” set of relational tables to accommodate each collaborator viewership of other collaborators’ search activities in their group. The also entailed several changes to the external connectivity to Mendeley to accommodate the collaborator's search frequency and viewership.

Queries Results (JSON Requests) (JSON Responses)

Group JSON DB Converter Search Results (SQL)

Membership Logger DB DB User Profiles User Activities (Authentication) (SQL) SearchAware

User

Figure 6-7: SearchAware v2 – Architecture

121

Chapter 6 Design and Implementation of SearchAware v2

Secondly, the changes in the awareness cues and viewership described above required significant changes to SUI/SERP to accommodate the awareness cues. Mainly, the use of custom visual controls to display the awareness cues required additional reprogramming of how the keyword list and general hierarchal grids are integrated in web application to allow partial refresh of the several components within the page to accommodate the real-time changes the to USI/SERP page content, particularly in the synchronous collaborative search.

Thirdly, achieving real-time or near-realtime (NRT) web application in the WWW is comparatively a complex implementation issue. Despite the availability of numerous technologies that are available to achieve real-time web application from simple stock prices and instant messaging to complex collaborative document editors, deploying such technologies for small to medium scale web apps is challenging due to the high technical requirements. Nevertheless, to facilitate a near-real time sense between collaborators, which is essential in synchronous search, a couple of compromises had to be done. This includes a background partial refresh when the modal windows appear, i.e. when a comment is written [Figure 6-4], or when an abstract is viewed [Figure 6-3]; and secondly when opening an external link in Mendeley, which allows SearchAware to also achieve a partial refresh of the results lists and accompanying cues. Moreover, it is achieved with the browsing of search results.

122

Chapter 7. Experimental Study

Chapter 7 presents the second study detailing the experimental study of SearchAware v2 that was covered in the previous chapter. The flow of the chapter is as follows. It starts with section [7.1] by looking at three key experimental studies with similar approaches. This is to provide a suitable experimental setup to evaluate the key parts in SearchAware 2 with relation to the research objectives. Section [7.2] presents the experiment hypothesis in relation to the research question, the research artefact (SearchAware v2) and accordingly identifies the key sub-hypothesis. The hypothesis is coupled with a detailed explanation of needed measurements and the justification for their selection. Section [7.3] details the experimental design used to evaluate SearchAware v2 and includes the design and rationale for each step, along with the full experiment protocol. In section [7.4], an overview of the experiment conduct method is presented, followed by the complete results of the experiment. Section [ 7.5] examines the results in depth and takes into consideration the communication aspect. The final section, [7.6] summarises the experiment hypothesis and results and relates them back to the research objectives.

7.1 Background

Laboratory studies are the primary means of evaluating interactive search systems, and they have been used extensively for this purpose (White, 2016, sec. 11.2.4). For CIS systems, evaluative studies usually require a much broader understanding of their usage context than HCI and IIR, as they include the additional collaborative aspect which requires consideration of the CSCW field too. These can be studies, therefore, tend to be system-focused, user-focused, or collaborative-focused, but Shah suggests that they should be not only multi-diminished but also balanced in their focus, depending on the system and context being investigated (Shah, 2014b). In this perspective, the CIS literature that is focused on prototypes and their impact on the collaborative aspects suggest that these studies are usually experimental in nature, as highlighted by the discussed CIS aspects and their studies in section [2.4]. Three main canonical experimental examples are explained next, from which the objective, hypothesis and experimental protocol for this research experiment is inspired. The studies will be examined in terms of context, software artefact, experimental procedure, manipulation of factors, and measurements used. One aspect to note before discussing these is that several experimental studies with CIS prototypes tend to systems that are aimed primarily at groups by using an individual user as a baseline level and the collaborative use in another level. To increase the validity of the studies, other factors are manipulated as well.

123

Chapter 7 Experimental Study

For the main Coagmento v1 experiment, Shah and Marchionini aimed to investigate and measure their web app awareness cues effectiveness in terms of collaborators’ search activities and user feedback. Using the WWW as an open search domain, participants, in pairs, used Coagmento to compile a report based collected web page snippets on a given simulated task. Shah and Marchionini study manipulated the awareness mechanism support randomly between participants where each pair was assigned one of three different conditions: 1) a baseline individual search session, with none of the group search activities available, 2) a personal awareness search session, where the history of each participant activities visible only to themselves, and 3) a group search session with all the collaborators’ awareness cues are available. All these conditions were tested in synchronous, remotely located rooms with instant messaging available to collaborators in all conditions. For the experimental stimuli, simulated work tasks, detailed here in [7.3.1], were devised and handed to the participants. A selection of various system usage measures and user-focused that questionnaires were used to evaluate the results (Shah and Marchionini, 2010).

Aiming to understand the strategies people employ in collaborative search, Joho et al. experiment focused on recall-oriented search in synchronous and co-located search. In this system-focused experiment, the participants, in pairs, were asked to search and collect as many relevant documents as possible using a dedicated search interface with collaborative search support. Three conditions were tested for each pair: 1) Independent search, where the same topic was given to each collaborator to be searched independently with the availability of personal history awareness. 2) Group search, in which all the search activities (keywords queried, documents visited, and documents saved) were sent to each collaborator through an automated instant messaging (IM) feed. Both this and the former method did not allow any communication between the collaborators. 3) Same as condition 2, but with the ability to communicate using either verbally or through the same instant messaging system. As for the measurements, traditional IR measures such as Precision, Recall and F-Measure (Kelly, 2009, p. 109) were analysed as the study utilised a Text Retrieval (TREC) dataset. In addition, other measurements such as observational data and semi-structured interviews were used (Joho, Hannah and Jose, 2008). In a follow-up study, with a focus on relevant feedback, Joho and his colleagues analysed the new study with resorting to tradition IR study methods and, consequently, opted to focus on the algorithmic and relevancy feedback results (Joho, Hannah and Jose, 2009).

The third example is a user-focus study comparing between individual and collaborative exploratory web searcher transitions and actions in a CIS prototype. Yue et al. used their CollabSearch system to compare between explicit (instant messaging IM) and implicit (shared search activities) collaborative communication (Yue, Han and He, 2012). Their experimental setup also included three conditions: 1)

124

Chapter 7 Experimental Study collaborative search with all awareness cues and communication; 2) collaborative search with awareness cues only, and 3) individual search. Each of which included different participants dyad. Their measures focused on transitions from various actions such and as such their focus was neither traditional IR measures nor usability and interaction measures. Like Coagmento, CollabSearch used Google’s search engine and results (Yue, Han and He, 2012).

These studies all used various search awareness approaches to aid collaborators in collaborative information seeking environments with manipulating the awareness and communication availability. In addition, they used pairs of participants and also used an individual search, either with or without typical instant messaging as a baseline for their experiment.

7.2 Hypothesis and Measurements

SearchAware v2 presents an extensive and novel form of awareness cues mechanisms that require investigation and evaluation in order to validate the various aspects of it as a new CIS system. Based on the discussion of the results of the formative exploratory user study presented in [Chapter 5] and the improvement in SearchAware v2 discussed in [Chapter 6], this study evaluation will proceed with an experimental approach. This is to ensure that the new and refined features are tested systematically in a comparative manner and to expose the participants to this new form of awareness cues in a controlled environment.

In SearchAware v2, the search awareness mechanism is to support collaborative search through the visual and contextual awareness cues that are linked to the search results via a collapsing hierarchal grid associated with each result. The experiment compares between the effect of the availability of these cues on the collaborator’s search process. Therefore, the availability of these cues will be the main independent variable in the experiment. However, to fully understand the effect and impact of the awareness cues, the communication aspect of collaboration is also used as part of the manipulated independent variable. Communication, as discussed in [2.4.2], is identified as the pillar for any collaboration, and its availability is crucial for a successful collaboration. (Sonnenwald, 1996; Reddy, Jansen and Krishnappa, 2009). Therefore, manipulating this factor provides a clear indication of the effect this novel method of delivering cues and enforces the power of this experiment.

Following the discussion in the previous section, and considering the second part of the research question stated in [1.3.1]:

[How] can these cues affect the search interface usability?

125

Chapter 7 Experimental Study

It is expected that the introduction of these cues in the search process will help the user in terms of the search activities during explicit collaborative information seeking session and the improve the collaborator’s experience of the search process. Moreover, these cues in SearchAware v2 will display the various actions and communications that collaborators are known to use during a CIS process.

This form of awareness cues is expected to impact search interface usability positivity in terms of the search process collaborators productivity and the collaborators’ interactivity with the system and the collaborators. For this variable, three conditions for this are applied to all the participants, 1) SearchAware v2 with awareness cues, without any communication channel, 2) with IM communication exclusively and, 3) with both awareness cues and IM communication. Therefore, the main experiment hypothesis can be formalised as:

H1: Search awareness cues will positively influence the search productivity of the participants.

H2: Search awareness cues will positively influence the search process interactivity of the participants.

Additionally, and as part of the search experience, the research hypothesises that the collaborators will find search awareness very usable and easy to used when enabled in collaborative information seeking activities, this is an essential the unique usability aspect of SearchAware. Therefore, it can be added as the third part of the effect search usability on the collaborator experience.

H3: Search awareness cues will positively influence the usability of SearchAware to the participants.

Based on the formalised hypothesis, the effect of the introduced awareness cues on the search interface usability can be measured by several methods. Most user-focused measurements methods in IIR stem from HCI studies, many of which are also applicable to CSCW studies as well (Kelly, 2009; Shah, 2014a; White, 2016, chap. 10). Therefore, based on the explicit CIS studies discussed in the previous section, this experimental study formally uses mainly quantitative measures from the system usage and as evaluative feedback from the participants to test the hypothesis.

There are several approaches to categorise the measures used broadly in HCI and IIR. Kelly divided these into four general ones: (1) Contextual: which includes in addition to the demographical data, the domain in which the study is administered, and the type of tasks handed. (2) Interactional: used to characterise the interaction between the user and the system and the user’s search behaviours. (3) Performance: which includes measures related to the outcome of the interaction, such as the number of relevant documents saved, mean average precision and others; and finally, (4) Usability: includes those based on evaluative feedback elicited from subjects. Common dimensions of usability are

126

Chapter 7 Experimental Study effectiveness, efficiency, and satisfaction. These typically are self-reporting measures which participants fill (Kelly, 2009, chap. 10).

In the field of HCI, Purchase suggests The Five Ps model which describes several forms of data collected in these studies. It comprises of Performance, Preference, Perception, Process, and Product (Purchase, 2012, sec. 4.3). However, Kelly notes that performance measures have typically been separated from usability measures, even though effectiveness and efficiency are standard dimensions of usability and are often measured in HCI and measures such as recall, task completion, error rate and time. Thus, in HCI, performance is most often subsumed under usability. Meanwhile, in IIR, performance is treated as a separate entity from usability and usually use the term usability as a synonym for self-report measures (Kelly, 2009, p. 101).

White divides IIR measure broadly into two groups: The first is the Outcome-Oriented measures that target the outcomes attained because of the process and based on the aspects such as relevance, accuracy, and benefit of the gained information as a result of the information seeking process. The second is the Process-Oriented measures that assess the search process in which the searcher was engaged and can be only based on the behavioural activities or information can be captured by the data collected during the search process (White, 2016, chap. 10).

Based on the above discussion, and using the experimental studies in explicit CIS, the main measures used to evaluate SearchAware v2 search interface usability are divided into three main categories as related to each hypothesis. These are Productivity and Performance measures; Interactivity Measurements and Usability Measurements. All of which are detailed next and linked in the

7.2.1 Productivity and Performance Measurements Traditionally, system outcome measurements are applicable specifically when using the traditional IR studies based on predefined set of documents known as text corpus, traditionally associated with the TREC (Text REtrieval Conference) tracks (Baeza-Yates and Ribeiro-Neto, 1999, chap. 3). However in IIR studies, these measures are not fully appropriate as they are dominated by these classical IR measures e.g. Precision, Recall, F-Measure and several relevance ranking measures and cannot measure the interactions in search user interfaces for example (Kelly, 2009, p. 109). Moreover, Shah noted that these measures are not applicable to studies on the WWW; he further suggests an alternative definition for these measures when the search domain is open, such as the WWW. He proposed that union of all the web pages visited by all the participants involved be the universe of all relevant documents (Shah, 2012, 2014b). However, these were not applied in user-focused CIS system studies.

127

Chapter 7 Experimental Study e.g. (Shah and Marchionini, 2010; Paul and Morris, 2011; Capra, Chen, Hawthorne, Arguello, et al., 2012) and thusly are not be adopted here.

In the field of HCI, the productivity measures are the data collected of the artefacts usage by participants as required by the experimental task, and these may be judged as correct and incorrect (Purchase, 2012, p. 109). Shah and Marchionini analysed productivity in several terms such as total queries (keywords), total unique queries, total web pages visited and total web pages saved (Shah and Marchionini, 2010).

For SearchAware, these measurements include the unique queries, the number of viewed (visited) results, and the number of rated result. This latter measure is based on the assumption that SearchAware relevance rating is used to obtain an average rating between the collaborators. Therefore it is assumed that it will be higher with all awareness cues and communication active. Recalling that the first hypothesis was declared earlier as:

H1: Search awareness cues will positively influence the search productivity in SearchAware.

Hence, the following sub-hypothesises can be defined as:

H1a: Search awareness cues will increase the number of unique keyword queries. H1b: Search awareness cues will increase the number of viewed results terms of abstracts and full articles viewed. H1c: Search awareness cues will increase the number of rated results.

Furthermore, for SearchAware, part of the improved search experience is the assumption that awareness cues will reduce the number of redundant searches due to the division of labour, as detailed in [2.4.4]. This also aligns with Joho et al. CIS experimental system (Joho, Hannah and Jose, 2008), Coagmento v2 (González-Ibáñez, Haseki and Shah, 2013) and ResultsSpace (Capra et al., 2013) evaluation studies measured. Therefore, the number of redundantly used keywords will be calculated by the redundant used of keyword after the initial usage by one of the participants during a collaborative search. This can be formalised as the fourth sub-hypothesis:

H1d: Search awareness cues will decrease the number of redundant queried keywords.

In this measurement, the compounded keywords are considered unique as it might change or alter the results. For example: ‘Mobile networks’ might yield different results or order from ‘Network mobile’.

128

Chapter 7 Experimental Study

7.2.2 Interactivity Measurements User evaluative measures collect data from participants about their opinions on the system being examined or experimented with and reflect evaluative feedback from the participants. IIR studies emphasise that the use of self-reported measures is crucial in such studies and provide important feedback on the experimental systems (Kelly, 2009, p. 110). Furthermore, Kelly highlights several user preference measures which are applicable in most IIR studies, including efficiency, effectiveness, usability, and engagement amongst others. Shah (2012) matched these measures to actual questionnaires and scales which were adopted in several related studies (González-Ibáñez, Shah and Córdova-Rubio, 2011).

In SearchAware, these measures play an essential role in studying the extent of achieving the design objective of aiding the collaborative search process from a user perspective. Therefore, the general hypothesis was formulated earlier as:

H2: Search awareness cues will positively influence the search process of the participants.

Two selected user interactivity measurements are used in this experiment: Engagement and Cognitive Load. A third measurement, activity awareness, is related to the interactivity aspects and awareness is also added to ensure that the search experience takes into consideration the collaborators’ awareness of each other. These measures are explained below.

7.2.2.1 Engagement Engagement has been defined as “a quality of user experiences with technology that is characterised by challenge, aesthetic and sensory appeal, feedback, novelty, interactivity, perceived control and time, awareness, motivation, and interest and affect” (O’Brien et al., 2009). The first interactivity sub- hypothesis is that awareness cues will correlate positively with the engagement, and can be stated as:

H2a: Search awareness cues will positively influence the engagement.

In this study, the participants are presented with the engagement questionnaire from the study conducted by Ghani, Supnick, and Rooney (1991) on the flow and engagement in CMC environments and had been suggested as a suitable measure for IIR and CIS studies (Kelly, 2009, p. 123). The original 5-points Likert scale contained 12 questions, but Shah readopted these into eight questions (Shah and Marchionini, 2010). The scale is shown below in [Table 7-1].

129

Chapter 7 Experimental Study

Item # Question (Likert 7-point scale) 1 Uninteresting ... Interesting 2 Not enjoyable ... Enjoyable 3 Dull ... Exciting 4 Not fun ... Fun 5 Not absorbed intensely ... Absorbed intensely 6 Attention was not focused ... Attention was focused 7 Did not concentrate fully ... Concentrated fully 8 Not deeply engrossed ... Deeply engrossed

Table 7-1: Engagement Measurement

7.2.2.2 Cognitive Load Measurement SearchAware awareness cues are expected to reduce the effort of the collaborators as an indicator of better experience, particularly during the synchronous search process. Therefore, the level of task workload is a suitable metric to compare between the experiment conditions. Based on that, the second interactivity sub-hypothesis is:

H2b: Search awareness cues will negatively influence the cognitive load.

IIR studies typically use NASA’s Task Load Index scale (NASA TLX) (Hart and Staveland, 1988; Hart, 2006) to measure the cognitive load measure. The workload is defined as “the cost incurred by a human operator to achieve a particular level of performance” (Hart and Staveland, 1988). This scale is used extensively in the field of HCI and Human Factors to measure mental workloads for complex interfaces such as aeroplanes cockpits (Kelly, 2009, p. 122). Naturally, NASA TLX is also used in IIR and SUI evaluations (Wilson, 2011) as well. Additionally, it has been adopted in CIS studies, e.g. Coagmento (Shah and Marchionini, 2010). The original scale contained six items, but one item is removed because it stresses physical demands. The table below shows the remaining five items:

Item # Questions (Very Low – Very High, Likert 10-point scale) 1 How mentally demanding was this task? 2 How hurried or rushed was the pace of the task? 3 How successful were you in accomplishing what you were asked to do? (Good – Bad) 4 How hard did you have to work to accomplish your level of performance? 5 How insecure, discouraged, irritated, stressed, and annoyed were you?

Table 7-2: Cognitive Load (Task Load) Measurement

7.2.2.3 Activity Awareness As the name implies, awareness is at the core of SearchAware, yet to measure awareness in the collaborative context requires complex understanding of the various factors that play a role in

130

Chapter 7 Experimental Study facilitating awareness with the higher aim to achieve a satisfactory level of collaboration. The Search awareness cues will, presumably, increase the activity awareness of the participants. Based on that, the third and final interactivity sub-hypothesis is:

H2c: Search awareness cues will positively influence the activity awareness.

Carroll et al. activity awareness taxonomy, explained in [2.4.1.4], highlights that collaborators share activities in complex circumstances, and not only abstract concepts of exchange. Nevertheless, they stress that “evaluating awareness levels in computer-supported collaborative activities is difficult.” (Carroll et al., 2006, p. 39). Therefore, the selected measurement is based on an empirically-backed method to study activity awareness in natural and simulated CSCW studies (Convertino et al., 2004). This questionnaire, in particular, is used to measure activity awareness through a collaborative tool.

Item # Questions (Likert 7-point scale) 1 I found it difficult to tell what work my partners had done using SearchAware 2 It was easy to find what my partners had worked on in SearchAware I could tell what my partners were doing while we were collaborating online in 3 SearchAware 4 I always knew what my partner was going to work in SearchAware 5 It was always clear what my partner was going to do in SearchAware 6 I became more aware of my partners' plans over the course of the task 7 My partner and I planned adequately 8 My partner and I communicated well with each other 9 My partner collaborated with me to complete the task 10 My partner contributed equally to this task 11 I enjoyed collaborating with a partner online I would enjoy interacting with others in the community (outside of the workplace with 12 interest or knowledge) on my task I would prefer to work on group projects over other types of workplace research 13 activities

Table 7-3: Activity Awareness Measurement

7.2.3 Usability Measurements Evaluating usability is a fundamental practice in HCI and typically involves measuring ease of learning, ease of use, and user satisfaction. Therefore, the hypothesis for these measures was formulated earlier as:

H3: Search awareness cues will positively influence the usability of SearchAware to the participants.

131

Chapter 7 Experimental Study

To acquire feedback on the usability of SearchAware search awareness mechanism, a final questionnaire is presented to the participants at the end of each experimental session. The questionnaire adopted is the Perceived Usefulness (PU) and Perceived Ease of Use (PEoU) evaluation as part of the Technology Acceptance Model (TAM). TAM is used to explain the behavioural intention to use a technological innovation (Davis, 1989). Despite the debate on that model, it is used extensively in a large number of technology related studies (Legris, Ingham and Collerette, 2003; Bagozzi, 2007) as there remains a wide variation in the predicted effects in various studies with different types of users and systems. Nevertheless, TAM and its subsequent refinements are currently one of the widely used models in information technology, in part because of its impact and discussion amongst scholars as well as its simplicity for the participants (Venkatesh and Davis, 1996, 2000). TAM is also flexible in terms of being adapted partially, as the PU and PEoU constructs can be are used without external variables constructs for evaluating interfaces in experimental contexts in (Kelly, 2009, sec. 10.4.1; Shah and Marchionini, 2010)

H3a: Search awareness cues will positively influence the perceived usability (PU). H3b: Search awareness cues will positively influence the Ease of Use (PEoU).

PU and PEoU measurements question items are listed below in [Table 7-4].

Item # Question (Likert 7-point scale) Part 1 Perceived Usefulness (PU) 1 Using SearchAware in my search would enable me to accomplish tasks more quickly 2 Using SearchAware would improve my information seeking performance 3 Using SearchAware in my search task would increase my productivity 4 Using SearchAware would enhance my effectiveness on the search task 5 Using the SearchAware would make it easier to do my search task Part 2 Perceived Ease of Use (PEoU) 6 I would find SearchAware useful in my information seeking tasks 7 Learning to operate the SearchAware would be easy for me 8 I would find it easy to get SearchAware to do what I want it to do 9 My interaction with SearchAware would be clear and understandable 10 I would find SearchAware to be flexible to interact with 11 It would be easy for me to become skilful at using SearchAware 12 I would find SearchAware easy to use

Table 7-4: Usability Measurement (PUEU-12) (Davis, 1989)

132

Chapter 7 Experimental Study

7.2.4 Hypothesis Summary Based on the discussion and hypotheses listed above, summary of the full hypothesis is listed below:

H1: Search Awareness cues will positively influence the search productivity of the participants. H1a: Search awareness cues will increase the number of unique keyword queries. H1b: Search awareness cues will the number of viewed results terms of abstracts and full articles viewed. H1c: Search awareness cues will increase the number of rated results. H1d: Search awareness cues will decrease the number of redundant queried keywords.

H2: Search awareness cues will positively influence the search process interactivity of the participants. H2a: Search awareness cues will positively influence the engagement. H2b: Search awareness cues will negatively influence the cognitive load H2c: Search awareness cues will positively influence the activity awareness.

H3: Search awareness cues will positively influence the usability of SearchAware to the participants. H3a: Search awareness cues will positively influence the perceived usability (PU). H3b: Search awareness cues will positively influence the Ease of Use (PEoU).

Moreover, based on the discussed measurements literature above, this experiment will use the following measures taxonomy in relation to the hypothesis items listed in the previous sections.

Measurement Hypothesis Data Collection Type Item Method H1a Unique Keyword Queries System Log Productivity and H1b Viewed Results System Log Performance H1c Dual Rated Results System Log Measurements H1d Redundant Queries System Log H2a Engagement Questionnaire Interactivity H2b Cognitive Load Questionnaire Measurements H2c Activity Awareness Questionnaire Usability H3a Perceived Usability (PU) Questionnaire Measurements H3b Perceived Ease of Use (PEoU) Questionnaire

Table 7-5: Hypothesis-Measurement Mapping

133

Chapter 7 Experimental Study

7.3 Experiment Design

This section explains the detailed experimental design, which starts off with a description of the experimental tasks that are to be handed to the participants, including the method to prepare these tasks and ensure their suitability for this experiment, all of which is accompany with detailed explanation and justification. This section concludes with the full protocol flowchart diagram that is to be followed in the experiment.

7.3.1 Experiment Tasks As the main focus is to investigate the effect of awareness cues on the participants during the collaborative search process, the search task needs to be defined to ensure that they explore the main features of SearchAware, and in the process, be measured for any impact on the user productivity and performance.

The concept of work tasks is embedded in several research fields such HCI, IR and CSCW and ultimately in the IIR core research methodology. Therefore, most studies in CIS use search tasks as means to experiment with prototype systems. From a cognitive approach, one way of solving an IR lab experiment problem would be to provide for more realism by allowing participants to understate of the search context and, at the same time, to achieve the experimental objective (Ingwersen and Järvelin, 2005, sec. 5.9). These simulated work tasks are designed to give the participants a scenario and provide a set of starting search terms to initiate the search process (Wilson, 2011).

However, the form and nature of tasks differ between studies according to the research design. In general, exploratory search tasks characteristics are: “uncertainty, ambiguity, discovery, be an unfamiliar domain for the searcher, provide a low-level of specificity about how to find the information, and be a situation that provides enough imaginative context for the participant to relate and apply the situation” (Kules et al., 2009, p. 315). One of the most established and formally defined forms of tasks in IIR are Simulated Work Tasks which have been also applied extensively in relevant research (Borlund and Ingwersen, 1997; Borlund, 2000; Borlund and Schneider, 2010). Borlund defines Simulated Work Task as “a short cover story which describes an IR (information retrieval) requiring situation” (Borlund, 2000, p. 73). Moreover, they are created “in order to provide broad, ambiguous, open ended tasks for the users” (Halvey et al., 2009, p. 93).

Besides being an established concept that has been integrated into a variety of studies, selecting this particular approach has a two-fold advantage. The first is that Simulated Work Tasks balance between experimental control and realism. Experimental control here implies the use of standardised task across. Realism means that any potential participants of a study employing Simulated Work Tasks can

134

Chapter 7 Experimental Study develop individual and subject information needs (Borlund, 2000, p. 76). These needs in the context of the cover story stimulate an interactive with the information. The second is the ability for the selected approach to be evaluated in an open-ended environment in particular as oppose to the confined pool of articles or databases used in classical IR studies. This is crucial in the case of the research tool SearchAware which utilises the dynamic and constantly increasing the database of Mendeley.

Two main assumptions underlie the use of simulated work task are mentioned by Borlund: a simulated work task situation and the use of dynamic and multidimensional relevance judgment. These translate into the elements for the task handed to the study participants. The simulated work task situation includes the context “cover story” and the can include an optional “indicative request” (Borlund, 2000; Borlund and Schneider, 2010).

Developing the experimental tasks is also critical and challenging. It is critical because it needs to ensure that the validity and reliability of the measurements are maintained, which helps to reduce the variance created by participants and their individual background. This achieved by using the following recommendations by Ingwersen and Järvelin:

1) To employ both the simulated situation/simulated work task situation and real information needs within the same test for comparative reasons. 2) To tailor the simulated work task situations towards the test persons with reference to: a. a situation of the type which the test persons can relate to easily and with which they can identify themselves; b. a situation that the test persons find topically interesting; and c. a situation that provides enough imaginative context in order for the test persons to be able to apply the situation; 3) To permute the order of search jobs between the test persons in order to avoid possible bias of the relevance assessments owing to human behaviour when comparing across system features and test persons 4) To pilot test prior to actual testing.

Figure 7-1: Simulated Work Tasks recommendations (Ingwersen and Järvelin, 2005, p. 253)

Moreover, it is challenging because the choice and nature of the task will impact how the participants will approach the task (Capra, Chen, Hawthorne, Arguello, et al., 2012). To address the challenge, Capra et al. suggested using a task in the domain relevant to the participant population (Capra, Chen, Hawthorne, Arguello, et al., 2012). Therefore, the justification for the type of tasks that should be applied in CIS studies is limited by the technology used and the searched databases (e.g. document corpus or the WWW).

135

Chapter 7 Experimental Study

Consequently, to balance between realism and the simulated work tasks guidelines above, and taking into consideration the suggested tasks from elaborate CIS studies with extensive use of work tasks, such as (Kules et al., 2009; Shah and Marchionini, 2010; Capra, Chen, Hawthorne and Arguello, 2012), a set of three simulated work tasks are formulated. The experimental work tasks used for the three conditions are created as typical job environments and are identified by three dimensions, using a layout approach layout also suggested by (Ingwersen and Järvelin, 2005, p. 253). The dimensions are:

• Context: Describes the domain of workplace environment. • Setting: Describes the type and nature of the task. • Process: Represents the exact request of the fictitious report.

In this experiment, most of the anticipated participants are postgraduate and research students in a large business school. Therefore, three contexts were selected to fit the task scenarios to be used and reflected the expected participants’ fields of study for this experiment which include fields of Business, Management, and Management Information Technology. Accordingly, three settings were set: Marketing through social network sites, economics and project management of sports events and mobile payments in the retail industry context. The exact nature of the task, while being fairly defined, will include a large scope with several aspects to be searched for as well. The tasks will also leave an opportunity for the collaborators to expand the search in an exploratory nature.

The full tasks are found in [Appendix 3], and they were tested for having appropriate and relevant literature in Mendeley, as well as participants in two preliminary experiments, based on which the tasks were modified accordingly. Moreover, the tasks encouraged the rating of the relevant articles using the provided star scale to ensure accumulating for the productivity measurement purposes, as discussed in [7.2.1].

7.3.2 Experiment Protocol and Procedures SearchAware v2 experiment is set as a synchronous, remotely located (simulated) setup. In this experiment, three categories of Search Awareness are tested and manipulated. The first are communication achieved through instant messages client and the second, which is being investigated in this research, are the search awareness cues presented above. In addition, a third condition having both IM and search awareness cues is also tested. The experiment is a within-group repeated measures which means all participants will be performing with all three conditions. The chat client for the collaborators is the web-based version of Google Chat (currently Hangouts) with predefined accounts.

136

Chapter 7 Experimental Study

The table details a summary of this and other information [Table 7-6]. Following that is some explanation about the participants’ allocation and experiment aspects.

Experiment Aspect Details Independent Variable Only one independent variable (IV) is used, which is Search awareness (IV) cues

The IV will have 3 conditions • C1: IM Only: Only access to the IM client be available between Independent Variable participants during the search session Conditions / Levels • C2: Search awareness cues Only: The cues be available to the participants • C3: Search awareness IM and Chat: Both the cues and the IM client will be available to the participants

• Productivity and Performance (4 measurements) Dependent Variables • Interactivity (3 measurements) • Usability (2 measurements)

Participant Allocation The experiment is conducted using a within-participant design. The order of the levels is rotated and counterbalanced amongst the total number of experiments as shown in [Table 7-7]

Number of Participants In total 36 participants are recruited, these are paired in dyads.

The experiments are held in the premise of University in a dedicated Location experimentation room with two PCs. Despite being the same room; participants will be refrained from talking during the search sessions. In addition, a spectator was placed between their workstation.

• Each pair of participants performs one experiment; therefore 18 experiment be performed. Number of Cases, Trials • Each experiment group is tested with all conditions. Thus, three trials and Experiments will be performed by each group. • Each experiment group is tested with the three conditions. In total 54 trials are conducted.

Table 7-6: Experiment Summary

Furthermore, the participants are informed that thy will be working are as a dyad formation. The justification of the use of two collaborators per group can be found in the literature, which mostly used dyad groups for evaluation studies, e.g.(Shah and Marchionini, 2010; Capra, Chen, Hawthorne, Arguello, et al., 2012; Yue, Han and He, 2012). Moreover, a one-week diary study of employees in large cooperation has shown that collaborative searches occur in pairs over 70% of the time at homes

137

Chapter 7 Experimental Study and 85% of the time at workplaces (Amershi and Morris, 2009). This is also reflected in the explicit number of participants was deliberately limited to two,

Early studies showed that as the number of collaborators working together increases, the complexity of possible interactions increases exponentially, thus increasing the likelihood of misinterpretation and misunderstanding. Minimizing group size allowed us to have better control of each experimental condition with respect to the variables of interest, which in this case relate to communication practices and performance. (González-Ibáñez, Haseki and Shah, 2013, p. 1168)

Prior to each task, the participants are encouraged to discuss their search approach. Collaborative information seeking experiments had different approaches for allowing participants to discuss their search approach and strategy before the actual search process. Some studies explicitly mention that they provide some time for participants to converse before they start the search session as this relates to the division of labour amongst collaborators, e.g. (Foley and Smeaton, 2010).

As the participant’s roles, these are implicitly set in the task description as being equivalent, other experiments showed that the participants tend to divide and conquer the search in such cases. In this experiment, the participants are not assigned to any explicit roles. However, the dyad is given few minutes before each search session (i.e. each search object) but are not instructed with any roles or division of labour strategies to adopt. The discussion about division of labour is in [2.4.4] and about roles in [2.4.5].

The experiment protocol is based on the main studies in CIS listed in the Summary of main CIS systems in the literature [Table 2-5] and uses the guidelines from HCI experimentations references (Blandford, Cox and Cairns, 2008; Purchase, 2012). The protocol steps are listed below, followed by the experiment protocol diagram typical flow in [Table 7-2]

1. Greetings Participants are greeted and assigned desktop PC. 2. Participant Information Sheet (PIS) Participants were sent a copy of the PIS in the appointment email. A hard copy of the PIS was handed over at this stage as well. A copy of the PIS is found in [Appendix 2.2]. 3. Task Briefing This briefing was delivered verbally to ensure participants are aware of the all the experiment aspects.

138

Chapter 7 Experimental Study

4. Consent Form Participants had to sign a standard consent form provided as part of the Ethical approval for the experiment. A copy of which can be found in [Appendix 2.1]. 5. Demographics Questionnaire A copy of this is found in [Appendix 4.1] 6. Assign User accounts and experiment sequence The accounts and groups are already created for the participants. Participants first names are used as usernames in the web app, but these are then encoded in the database based on the session, e.g. 001A and 001B as the pair for trial 1. The trial table for the 18 experiments is detailed previously in [Table 7-7]. 7. SearchAware Tutorial (Demo) and Practice Trial The detailed demonstration video which is 5:33 minutes long is played for the participants. They were shown how to pause or rewind to any past point15. This is followed by a trial run where participants can familiarise themselves with the functionality and features of SearchAware. Moreover, the participants are encouraged to search for their own research references to be able to find the search engine functional and of use for the simulated search task. 8. Task Handout [Appendix 3.1-3.3] (× 3) The paper copy of the preassigned work task is handed to the participants. The tasks are assigned as shown in [Table 7-7] and detailed in Step 10 below. 9. Pre-Search Discussion (× 3) After individually reading the task, the participants are encouraged to discuss their search approach prior to the search session. A maximum of 5 minutes will be given before each task. 10. Search Work Task Session (× 3) The order of the search session trial, conditions, and tasks for each group are found in [Table 7-7] below. As noted, the order of the trials and conditions are rotated to ensure it covers all possible combinations of the number of participants. A break of 2 minutes is given between each trial. The details of the tasks are found in [7.3.1], and the tasks are in [Appendix 3.1 – 3.3]. Each session trial will last for a maximum 20 minutes. This was based on trial experiments where participants were asked to read the abstracts and skim through the content of the publications retrieved if deemed necessary. Furthermore, similar experiments that utilise the open web solutions in synchronous collaborative search systems typically set timings between 20-25 minutes (Paul and Morris, 2009).

15 SearchAware v2 tutorial video is available at: http://searchaware.alarrayed.info/video/Demo.mp4

139

Chapter 7 Experimental Study

Trial 1 Trial 2 Trial 3 Groups Condition Task Condition Task Condition Task

1 and 10 1 BUS 2 ICT 3 ORG

2 and 11 2 BUS 3 ICT 1 ORG

3 and 12 3 BUS 1 ICT 2 ORG

4 and 13 1 ICT 2 MGT 3 BUS

5 and 14 2 ICT 3 MGT 1 BUS

6 and 15 3 ICT 1 MGT 2 BUS

7 and 16 1 MGT 2 BUS 3 ICT

8 and 17 2 MGT 3 BUS 1 ICT

9 and 18 3 MGT 1 BUS 2 ICT

Task Code: Business (BUS; Appendix 3.1); Management (MGT; Appendix 3.2); Information Communication and Technology (ICT; Appendix 3.3). Conditions: Condition 1: IM Only; Condition 2: Cues Only; Condition 3: IM and Cues

Table 7-7: Experimental Trials Arrangement

11. Post-Task Questionnaire This includes handing the participants the following questionnaires: • Engagement Questionnaire, as explained in [7.2.2.1]; [Appendix 4.2.1]. • Activity Awareness Questionnaire, as explained in [7.2.2.2]; [Appendix 4.2.2]. • Cognitive Load (Task Load) Measurement (Post-Trial) in [7.2.2.2]; [Appendix 4.2.3]. 12. Post-Experiment Questionnaire This includes handing the participants the following questionnaire: • Usability Questionnaire, as explained in [7.2.37.2.2.2]; [Appendix 4.3]. 13. Reimbursements Participants are handed a £15 Amazon voucher after the completion of the task and signing an attendance sheet.

140

Chapter 7 Experimental Study

Protocol Flow Steps (Duration) 1 Greeting 1 Greeting (≈ 2 minutes) 2 Participant Information 2 Participant Info Sheet Sheet (PIS) [Appendix 2.1]Appendix 2.1 [Appendix 2.1] Participant Information (≈ 2 minutes) 3 Task Briefing 3 Task Briefing (≈ 2 minutes) 4 Consent Form 4 Consent Form [Appendix 2.2] [Appendix 2.2] (≈ 1 minutes) 5 Pre experiment Demographics 5 Pre-Experiment Questionnaire questionnaire [Appendix 4.1.1] Questionnaire [Appendix 4.1.1] (≈ 5 minutes) 6 Assign user accounts and experiment sequence 6 Assign user accounts and experiment sequence 7 Video Tutorial and Practice Trial (≈ 10 minutes) 7 SearchAware Tutorial (Demo) and 8 Task Handout Practice Trial (Tasks BUS, ICT, or NPO) [Appendix 3.1.1] Engagement 8 Task Handout (≈ 2 minutes) Questionnaire [Appendix 3.n] 9 Pre-Search discussion [Appendix 4.2.1] (≈ 3 minutes) × n 9 Pre-Search Discussion 10 Search Session Activity Awareness (≈ 23 minutes) Questionnaire 2 min 10 Search Trask Session [Appendix 4.2.2] 11 Post-Task Questionnaire break [Appendix 4.2.1] [Appendix 4.2.2] Cognitive Load [Appendix 4.2.3] 11 Post-Task Questionnaire Index [Appendix 4.2.1 – 4.2.3] Questionnaire Break (≈ 2 minutes) [Appendix 4.2.3] 12 Post-Experiment Questionnaire 12 Post-Experiment [Appendix 4.3.1] Questionnaire Usability 13 Reimbursement (Form 7) Questionnaire [0Appendix 4.3] n = number of trials. 13 Reimbursements (n = 3 in this experiment)

Figure 7-2: Experiment protocol flow

141

Chapter 7 Experimental Study

7.4 Results

The first section [7.4.1] details the actual experiment method and provides general information on its conduct and administration. The remaining subsections [ 7.4.2], [ 7.4.3] and [ 7.4.4] lists the experimental results including the basic statistical results and the comparison tests related to the experimental setup.

7.4.1 Overview Before conducting the formal experiments, two pilot trials were done with two days. This allowed for further adjustments to the experiment protocol and changes to the work tasks. Participants, as discussed, had to meet criteria of being able to attend as a pair, and both should be least Master’s degree students and preferably had worked together in any assignment or research project.

Figure 7-3: SearchAware v2 – Experiment Participants16

16 All participants were formally asked for permission to be photographed.

142

Chapter 7 Experimental Study

The actual experiment was conducted in the premises of the University of Manchester. In total, 36 participants took part in the experiment; all were recruited through friends and email announcements distributed within the school mailing list.

Out of the total 36 participants, 34 were students of the University of Manchester and 2 were graduate PhD holders, also from the University of Manchester. 5 participants were MSc students from the Business school, and the remaining 29 were PhD students from Business School. The median age of the participants is 28, and their ages ranged from 23 till 35 with an average age of 29 years young. 20 Male participants joined (55.6%) and 16 females (44.4%). All participant pairs were acquainted with each other, and 28 (77%) reported that they have worked with each other before. All participants have used scholarly search engines before, with a frequency usage for at least several times a month.

7.4.2 Productivity and Performance Measurements Results These measures include the number of unique keyword queries, viewed abstracts and viewed page results, rated results and the redundant keyword queries.

To find if the measurements revealed any statistical significance, the proper statistical tests needs to be selected for the experiment type based on the number of variables and levels/conditions. In this experiment, all the applicable tests are for between groups repeated measures comparisons.

To achieve that, the data collect for normality to check if these values are to be analysed using parametric or non-parametric test. This can be achieved either by using Skewness and Kurtosis values able to perform parametric tests. These are suggested to range between -2 and +2 as a general rule to be acceptable as parameters even though these are sensitive enough for nonparametric values (Purchase 2012, pp.125–126). As these measurements include single scale item is used, thus the reliability test is inapplicable.

Next, the proper test from the parametric tests family needs to be applied. As the experiment is to compare between the different conditions and since this experiment contained one independent variable with three levels conditions then the proper test is a one-way analysis of variance (ANOVA) which involves one independent variable (referred to as a factor) which has a number of different levels. Moreover, as all the participants performed under all three conditions. Then the appropriate test considered is a one-way repeated measures ANOVA. (Pallant, 2013, p. 258). On the other hand, for non-parametric values, then the Friedman test is applicable (Pallant, 2013, p. 243).

The preliminary analysis revealed that there was no statistical significance in the differences between the tasks domains in terms of the selected Simulated Task Work, nor in the order of the tasks were

143

Chapter 7 Experimental Study handed to the participant groups. This allowed the analysis to be performed by collapsing that tasks as a single task. This is based on finding the Cronbach alpha coefficient, which ideally should be above 0.7, but can be accepted as 0.5 for scales with lower than 10 items (Pallant, 2013, p. 101).

7.4.2.1 Unique Keyword Queries As explained in [7.2.1], the complete number of queries by all pairs of participants (n=18) is 873 queries, of which 819 are unique to each search session, which is almost 93.8% of the total number of queries. These are the unique keyword combination use per group. The descriptive statistics of the unique queries is shown in [Table 7-8] below:

Awareness Cues Mean Std. Deviation C1: IM Only 14.17 4.048 C2: Cues Only 15.11 7.128 C3: Cues and IM 16.22 8.314 N = 18

Table 7-8: Descriptive Statistics for unique queries across groups

The analysis shows that while there is an increase in the number of unique search queries in the third condition with the availability of awareness cues and IM. However, there was no significant effect on the number of the different awareness cues, with a Wilks’ Lambda of 0.937, F(2,16) = 0.535 with a p- value .596 > .05.

7.4.2.2 Viewed Results The Viewed Results measurement, as explained in [7.2.1] consists of two types of results views in SearchAware v2. Abstract views, which are abstract button is clicked [Figure 6-2 B] and Page view, which is when the Title of the result is clicked [Figure 6-2 C]. When both are combined in a single scale, the descriptive statistical results are:

Awareness Cues Mean Std. Deviation C1: IM Only 13.36 6.875 C2: Cues Only 15.58 5.310 C3: Cues and IM 14.69 6.727 N=36

Table 7-9: Total Viewed Results - descriptive results

As these values do not meet the normal distribution assumption and is based on aggregated value of two aspects. However, conducting the analysis using each separately allows the use of the parametric analysis for the abstract views, as shown below.

144

Chapter 7 Experimental Study

Awareness Cues Mean Std. Deviation C1: IM Only 7.69 8.940 C2: Cues Only 9.61 8.076 C3: Cues and IM 10.53 7.948 N = 36

Table 7-10: Abstract Viewed Results - descriptive results

After performing the analysis, the ANOVA test results Wilks’ Lambda = 0.819, F(2,34) = 375 and a significance of .034, meaning p < .05. Additionally, the partial eta squared value = 0.181, which suggests a small effect size. ANOVA test reveals a significant change in the number of Abstract views across the three conditions, with the condition C3 having a higher number of views with a mean of 10.53 views.

As for the Page views, the following table shows the descriptive results:

Awareness Cues Mean Std. Deviation C1: IM Only 5.67 5.611 C2: Cues Only 5.06 5.616 C3: Cues and IM 5.08 5.886 N=36

Table 7-11: Page Viewed Results - descriptive results

As the collected data do not meet the normality assumption, then the Friedman test is used. The results of the test are χ2 (2, n = 36) = 1.754, p = .416 > .05, therefore no statistical significance is found in the number of page views between the three conditions. Nevertheless, for this group of participants, there is a slight advantage to the number views in the first condition as shown in [Table 7-11].

7.4.2.3 Rated Results The descriptive statistics of the rated results (n = 36) are shown in [Table 7-12] below.

Awareness Cues Mean Std. Deviation C1: IM Only 11.41 7.905 C2: Cues Only 11.32 6.489 C3: Cues and IM 10.15 7.003 N = 36

Table 7-12: Rated Results - descriptive results

145

Chapter 7 Experimental Study

After an ANOVA is performed to compare between the three shows that while there is an increase in the number of unique search queries in the third condition with the availability of awareness cues and IM. However, there was no significant effect on the number of the different awareness cues, with a Wilks’ Lambda of 0.937, F(2,16)=0.535 with a p value .596 >.05.

7.4.2.4 Redundant queried keyword As explained in [7.2.1], the redundant keywords are calculated for each participant if the keyword used is used a second time or more. The descriptive is shown in below.

Condition Mean Std. Deviation C1: IM Only .78 1.692 C2: Cues Only .67 1.656 C3: IM and Cues .89 1.214 N = 36

Table 7-13: Redundant queried keyword - descriptive results

As the collected data did not meet the normality assumption, then the Friedman test is used. The test revalued no statistical significance in the number of Redundant queried keyword between the three conditions. The results of the test are χ2 (2, N=36) = 2.220, p = 0.330 > .05.

Therefore, there is no significance in the number of redundant keyword between the three conditions.

7.4.3 Interactivity Measurements Results As with the productivity and performance measures discussed above [7.4.2], each of the interactivity measures is tested for normality by identifying the Skewness and Kurtosis values able to determine if the parametric tests are possible or not.

In addition, since all of these tests used scaled questionnaire items with multiple subscales, each was tested for reliability using the most common indicator of internal consistency which is Cronbach's Alpha coefficient, which ideally should be > .7.

7.4.3.1 Engagement Measurement Results As explained in the experiment protocol section in [ 7.3.2], the engagement measurement questionnaire is administered on paper as the first questionnaire after each experimental trial, a copy of the questionnaire is found in [Appendix 4.2.1 Engagement in SearchAware (Post-Trial)]. The table below shows the descriptive results as follows:

146

Chapter 7 Experimental Study

Condition Mean Std. Deviation

C1: IM Only 5.684 0.527

C2: Cues Only 5.142 0.600

C3: IM and Cues 5.722 0.6709 N = 36

Table 7-14: Engagement index descriptive results

Before performing any significance check, the normality check is performed, followed the reliability check of the 7-index Engagement measurement scale is checked.

Cronbach's Alpha α Number of Items

.785 8

Table 7-15: Engagement index reliability check

The reliability check for the Engagement index is .785, which suggested an acceptable index consistency. Therefore, a One-Way repeated measures ANOVA analysis is valid for this measures. ANOVA test reveals a significant effect for awareness cues with Wilks’ Lambda = 0.487, F(2,34) = 17.886 and a significance of .000, meaning p < .05. Additionally, the partial eta squared value = 0.513, which suggests a very high effect size.

The descriptive results show that the lowest engagement mean is for the second condition with cues only, followed by the first cues only and then third condition with both cues and IM, this suggests that the combination of cues and IM achieves a better engagement in SearchAware v2.

7.4.3.2 Cognitive Load (Task Load) Measurement (Post-Trial) As explained in the experiment protocol section in [ 7.3.2], the cognitive load questionnaire is administered on paper as the third and final questionnaire after each experimental trial, a copy of the questionnaire is found in [Appendix 4.2.3].

A total of 18 groups × 2 participants per group × 3 post-trial questionnaires yields a total of 108 results. The descriptive results are as follows:

147

Chapter 7 Experimental Study

Condition Mean Std. Deviation

C1: IM Only 7.133 1.314

C2: Cues Only 6.873 1.002

C3: IM and Cues 6.680 1.130 N = 36

Table 7-16: Cognitive load measurement descriptive results

Before performing any significance check, the normality is checked, followed by the reliability test of the 5-index cognitive load measurement scale is checked, and the result is as follows:

Cronbach's Alpha α Number of Items

.735 5

Table 7-17: Cognitive load index reliability check

The reliability check for the cognitive load the whole experiment is 0.57 which suggested low value, but can be accepted as 0.5 for scales with lower than 10 items (Pallant, 2013, p. 101). Therefore, a One-Way repeated measures ANOVA analysis is valid for this measures. ANOVA reveals no significant effect for awareness cues with Wilks’ Lambda = 0.858, F(2,34) = 2.506 and a significance = .0749, meaning p > 0.05. Additionally, the multivariate partial eta squared value η2= 0.142.

Nevertheless, the third condition, which is the IM and cues means value achieved almost a midway point between the higher cognitive load with cues only, and the lower cognitive load with IM only, suggesting that the use of cues increased the cognitive load on the participants.

7.4.3.3 Activity Awareness As explained in the experiment protocol section in [7.3.2], the activity awareness questionnaire is administered on paper as the second questionnaire after each experimental trial, a copy of the questionnaire is found in [Appendix 4.2.2].

A total of 18 groups × 2 participants per group × 3 post-trial questionnaire set yields a total of 108 results. Moreover, the negatively worded question, which is the first question of this questionnaire, is reversed for this measurement. The results are as follows:

148

Chapter 7 Experimental Study

Condition Mean Std. Deviation

C1: IM Only 5.256 0.7505

C2: Cues Only 4.346 0.8998

C3: IM and Cues 5.722 0.7317 N = 36

Table 7-18: Activity awareness index descriptive results

Before performing any significance checks, the normality is checked using, followed by the reliability index of 13-index activity awareness measurement scale check, as shown below:

Cronbach's Alpha α Number of Items

.901 13

Table 7-19: Activity awareness index reliability check

The reliability check for the activity awareness index for each condition is 0.901, which suggested a very high index consistency. Therefore, a One-Way repeated-measures ANOVA analysis is valid for this measurement. ANOVA reveals a significant effect for awareness cues with Wilks’ Lambda = 0.327, F(2,34) = 34.986 and a significance = 0.000, meaning p < 0.05. Additionally, the partial eta squared value η2 = 0.673, which suggests a very high effect size.

The descriptive results table shows that the lowest activity awareness mean was for the second condition with cues only, followed by the first cues only and then third condition with both cues and IM. The IM and cues (C3) condition averaged higher means for the participants, which suggests an increase of awareness which matches the assumption that communication increases awareness, and its absence suggests lower awareness amongst the participants even with IM communication available.

7.4.4 Usability Measurements Results As explained in the hypothesis and measurements section in [7.2.3], the usability measurement questionnaire is administered as the final questionnaire after the end of all the experimental trials. The questionnaires listed the Group Awareness Mode Indicator [Figure 6-1 A]; which had three modes depending on the condition: None, Group and Full. The questionnaire is found in [Appendix 4.3].

A total of 18 groups × 2 participants per group × 3 of experiment questionnaires on each yields a total of 108 results. The descriptive results are as follows:

149

Chapter 7 Experimental Study

Perceived Usability Perceived Ease of Use

(PU) (PEoU)

Condition Mean Std. Deviation Condition Mean Std. Deviation

C1: IM Only 5.401 0.623 C1: IM Only 5.707 0.614

C2: Cues Only 5.246 0.715 C2: Cues Only 5.222 0.636

C3: IM and Cues 6.334 0.593 C3: IM and Cues 6.292 0.510 N = 36 N = 36

Table 7-20: Usability measurement descriptive results

Before performing any further analysis, the reliability check of each of the two part 12-item (each with six items) index scale is checked as mentioned in the previous scales. The obtained reliability values by PU and PEoU are listed below:

Perceived Usefulness (PU) Perceived Ease of Use (PEOU)

Cronbach's Alpha α Number of Items Cronbach's Alpha α Number of Items

.839 6 .799 6

Table 7-21: Usability Measurements

Thus, both scales are reliable and are checked to satisfy the normality assumption. Therefore, a One- way repeated measures ANOVA analysis is valid for both measures. For the PU measurement, the one- way repeated measures ANOVA reveals a significant effect with Wilks’ Lambda = 0.728, F(2,34) = 19.160 and a significance = 0.047, meaning p > .05. Additionally, the multivariate partial eta squared value = 0.794, which suggests a large effect size.

For the Perceived Ease of Use measurement, the one-way repeated measures ANOVA reveals a significant effect for the PEoU Perceived Ease of Use with Wilks’ Lambda = 0.382, F(2,34) = 3.942 and a significance = .039, meaning p > .05. Additionally, the multivariate partial eta squared value η2 = 0.794, which suggests a large effect size.

Therefore, both the usability measurements are found to have a significant effect for the awareness cues for the condition 3, i.e. cues with the IM functionality enabled. However, both the tests found that the cues only, i.e. condition 2, had the lowest usability means, implying the participants did not find the cues usable on their own.

150

Chapter 7 Experimental Study

7.5 Discussion

The first subsection focuses on the discussion of the hypothesis test results. In the second subsection, a closer look at some the communication messages and other experimental issues is presented.

7.5.1 Hypothesis Testing Results The following [Table 7-22] summarises the hypothesis testing results:

Measurement Hypothesis Result Hypothesis Type Item (Condition) H1a Unique Keyword Queries Positive Inf. No significance found Productivity and H1b Viewed Results Positive Inf. Partial significance (C3) Performance H1c Rated Results Positive Inf. No significance found Measurements H1d Redundant Queries Negative Inf. No significance found H2a Engagement Positive Inf. Significance (C3) Interactivity H2b Cognitive Load Positive Inf. No significance found Measurements H2c Activity Awareness Positive Inf. Significance (C3)

Usability H3a Perceived Usability (PU) Positive Inf. Significance (C3) Measurements H3b Perceived Ease of Use (PEoU) Positive Inf. Significance (C3)

Table 7-22: Summary of Hypothesis testing results

The results show a mixed view of the indicators and measures used in this experiment. While most of the productivity and performance hypotheses have not achieved any significance, most of the interactivity and usability hypotheses did achieve. The difference represents also the discrepancy between the log-based measurements and the user-evaluative and feedback measurements, which is discussed at the end of this section.

The first hypothesis suggests that the productivity and performance numbers will improve with the availability of the search awareness cues. Nevertheless, this has proven to be statistically insignificant when the conditions are compared, except for the Abstract Viewed results which was found to have significant increase with condition C3.

For the first sub-hypothesis, H1a, the unique keyword queries did not differ in any condition, although the mean for C3 in this experiment is slightly higher. Compared to similar experiments, Shah and Marchionini’s Coagmento study found that while the unique keyword queries have statistically increased, most of the other productivity measures used were found insignificant. However, it is worth noting here that one of the Coagmento conditions included a ‘Personal’ condition, i.e. solitary search.

151

Chapter 7 Experimental Study

Furthermore, it also based on the WWW search and included a prize for the highest numbers of snippets (thumbnail screen capture). Shah and Marchionini concluded that “it is hard to claim difference in productivity among different conditions.”, however, judging by the number of significant queries, they suggest that the participants “managed to explore more volume of information with the same amount of time and work.” (Shah and Marchionini, 2010, p. 1979).

The result for the second sub-hypothesis, H1b, prompted a further investigation about the relation between the Abstract Viewed results and the Page Viewed Results under each condition, both of which are used to view the results, albeit in concise or full manner. Therefore, a repeated measures paired-sampled t-test between each of these is performed for each of the three conditions. The results are below in [Table 7-23]:

Condition Mean Std. Deviation t df Sig. (2-tailed)

C1: IM Only 2.028 13.250 .918 35 .365

C2: Cues Only 5.472 12.698 2.586 35 .014

C3: IM and Cues 4.528 12.429 2.186 35 .036

Table 7-23: T-Test between Abstract Views and Page Views for all conditions

These results indicate significant tendency to view the abstracts, compared to viewing the full result page in the second and third condition, i.e. with the cues on. This implies that having the cues on increased the number of views of the shortened version of the results rather than the full details, implying a faster pace of working collaboratively. However, as the rated results suggest, that did not lead necessarily to increased results rating.

Focusing further on the snippets used in Coagmento, these can be compared with ratings for the results in the SearchAware v2, as they both represent a save, bookmark, like or favourite result, depending on the context. In the first iteration of Coagmento, snippets were found statistically insignificant, and these were attributed mainly to “personal motivations”. In a different study by Kelly and Payne study, using a later version of Coagmento, they suggested that these snippets needed to be fine-tuned and differentiated for different purposes, e.g. saving, sharing, or marking relevance (Kelly and Payne, 2014).

The ratings in SearchAware v2, which are tested as H1c, are aimed at marking relevance for search results. As discussed in [6.2.2], this is done by displaying the average ratings in the Avg column, and by that the relevance is shared with the collaborators. However, this did not translate to any significance in the analysis. On the contrary, the first condition rated results mean in [Table 7-12]

152

Chapter 7 Experimental Study indicated higher ratings that both the second and third conditions, for this experiment participant at least. Therefore, alternative methods for awareness should be sought.

A suggested alternative analysis method is to compare the between groups pairwise (Imazu, Nakayama and Joho, 2011; Villa and Jose, 2012). Therefore, comparing between C1 and C2, C1 and C3, and C2 and C3 using repeated measures paired-sampled t-test for parametric data and Wilcoxon signed rank test for non-parametric data (Pallant, 2013). For the above results, performing the corresponding tests (either parametric or non-parametric) also revealed the same results.

The second hypothesis is regarding the interactivity of the collaborators with systems and with other collaborators, which are subjective user-evaluative measurements. Two out of the three results suggested a positive influence of the cues. For the first sub-hypothesis H2a, the engagement measurement shows that the participants are more interested and absorbed, amongst other subscales this measurement collects. This is despite having an overhead of collaboration and a stream of visual awareness cues to be altered too. This indicates a subtle, yet effective approach for the awareness cues. Moreover, for sub-hypothesis H2c, the activity awareness measurement is also found to be significant, suggesting a successful approach to allow the participant to be aware of their group collaborator activities.

The cognitive load measure analysis, for sub-hypothesis H2b, is found to be insignificant, as the numbers suggest a decreased value for the last condition. However, despite not fulfilling the hypothesis, having is a good indication of an awareness design. Shah and Marchionini found the same result, but suggests that this is a good thing, as it does indicate increased awareness and engagement, as shown above, but without the increased cognitive load task on the collaborators. To explain that, they assert “this informs us that even those with a more complex interface, [with the awareness cues], felt no more mentally loaded than those with a simple interface [without the awareness cues]” (Shah and Marchionini, 2010, p. 1983). In light of that, it can be said that the design achieved an improved awareness of the search activities, thus a better search experience for the collaborators.

As for the third hypothesis H3, both the perceived usability, H3a, and perceived ease of use, H3b, were found to be statistically positivity influenced by all the awareness cues. SearchAware v2 is, therefore, found to be easy to use and usable for the context of collaborative information seeking. However, the participants’ informal feedback identified several important issues.

Overall, the participants found SearchAware fit to be used for the simulated work tasks, and when asked if they consider this web app for their actual research or within their search groups most responded positively. Nevertheless, their main identified issue is the slow response of the interface,

153

Chapter 7 Experimental Study which is due primarily to the refreshed interface for new cues. Although this refresh is designed to work when the user is not using the actual SERP, e.g. when a comment is written or when an abstract is viewed, occasionally the participants quickly close that window to get back in the SERP to find it still reloading. This has also been to found to be bit excessive due to the use of shared hosting plan, which limits the resources available to web apps. The performance is noticeably faster on the local development machine.

Another usability issue identifying was the keyword timeline, which is designed intentionally in a basic form in SearchAware. Some participants preferred to be able to see both their keywords and their collaborators’ keywords, which resembles CoSense approach, as shown in [Figure 2-12], but can be evolved and condensed using a multiple timeline approach. Furthermore, the use of awareness cues, e.g. the ‘Viewed’, ‘Viewed and Commented’ icon was found easy and innate to the participants. Using the bold font to highlight newer cues being added is easily recognisable by the participants with expressions like: “aha, just like Gmail”.

Finally, the discrepancy between most of the system-focused hypotheses (H1) and the user-focused hypotheses (H2 and H3) can indicate a design flaw in SearchAware v2 cues in aiding the productivity and performance of information search task. Also, it might indicate the setting of an incorrect hypothesis or a wrong use of the measurements, despite using similar, but not identical, previous experiments as a reference.

Other confounding variables such as the nature of the simulated work tasks (although that did not reveal any statistical difference), the duration of the trial, or the use of the external digital library of Mendeley might be attributed. This merits further future work in this aspect.

7.5.2 Communication The chat logs of the participants are saved in this experiment to provide a further understanding of the search process and the effect of the awareness cues. A detailed study focusing on the aspects of computer-mediated communication in CIS using Coagmento, identified three categories of communication messages particularly related to collaborative information seeking: Strategy, Information Seeking and Awareness, which were added to a previous list from the literature (González-Ibáñez, Haseki and Shah, 2013, p. 1173). While the communication analysis is not the focus of this study, some of the highlighted issues with regrades to the awareness communication messages are discussed next.

The need for awareness is very apparent in the exchanged communication between the participants, particularly for the groups that started with the IM option only. The attempted division of labour when

154

Chapter 7 Experimental Study no cues where available is apparent in this sample exchange of messages in [Figure 7-5].For the groups that started with the cues first and then were informed that these would disable in the next search task, the collaborators attempted to recreate the same sense of keyword and search activity sharing, indicating their importance to the search process, such as shown in [Figure 7-4].

Group 15; Trial 2; Condition 1; Task: MGT P015B: Hey [P015A]! Did you get the general idea? P015A: yes i did P015B: Shall we go to search for the specified keywords? P015A: i tried some new keyword combination which didn't work i used olympic event but the articles were not that much relevant P015B: right i got some good papers i'll start searching for facilities and others P015A: go ahead P015B: :) […] P015A: yeah have found anything particularly related to championships? P015B: no I did not use it as keyword

Figure 7-4: Partial IM log for Group 15; Trial 3; Condition 2; Task: MGT

Other issues identified are about the experimental setup. These were some issues related to the time constraints, which can be limiting in the amount and ability to interact. Furthermore, although a break was given between the trial sessions, few participants expressed fatigue, and to some extent boredom, particularly in the last session, which is expected in such experiments.

Group 1; Trial 1; Condition 1; Task: BUS P001B: what was your search term? P001A: Viral marketing P001B: Mechanisms, evaluation, how about your, how about yours????????????????????? P001A: I said it! social media marketing P001B: just this!ok

Figure 7-5: Partial IM log for Group 1; Trial 1; Condition 1; Task: BUS

155

Chapter 7 Experimental Study

7.6 Summary

The experiment provided insightful details on how to consider and support the collaborative aspect of information seeking with regards to the design and implementation of awareness cues through the result list. The visual awareness cues play a considerable impact on how the cues affected the search interface usability, and in turn the user experience of the search process.

By rotating the awareness mechanism between the awareness cues, and the CMC channel represented in the form of instant messaging communication, the experiment tested three hypotheses. The first was that the productivity and performance of the participants, the second was the level of interactivity, and the third was the level of usability.

For the former, most of the measures of user search productivity did not provide any significant improvements in the performance results, apart from a partial influence of the number of viewed results per group. This measurement was aligned with previous studies, but addressing these shortcomings might need further understanding of the measures and their suitability and expected results in collaborative information seeking tasks.

The second set of tests comprised of three interaction measurements in terms of engagement, cognitive load and activity awareness, the former and the later found that the introduction of awareness cues significantly influenced these measurements. Additionally, the cognitive load did not seem to increase with the awareness cues which is also an indication of the proper use of awareness cues.

The third set of measurements concerning usability revealed a significant improvement in both measures of the perceived ease of use and perceived usability by the participants. It correlates significantly with the introduction of the awareness cues and indicates that the participant’s user experience improved.

The findings of this study suggest that collaboration with awareness cues impact on collaborates search process goes beyond increasing the productivity as reported in earlier studies, and more into the user satisfaction in terms of collaboration (Capra et al., 2010; Shah, 2010b; Shah and González- Ibáñez, 2010). This aligned with the discussion in [ 2.2] that collaboration happens with the collaborators experience of solidarity (Denning and Yaholkovsky, 2008) and that “social behaviors that have governed groups and organizations for thousands of years again rise to prominence.” (Grudin and Poltrock, 2012, p. 29).

156

Chapter 8. Conclusion

This chapter starts with section [8.1] that presents a summary of the developed research artefact and the conducted user evaluations. Next, in section [8.2], the main contribution of the research is presented, including the implications for the design and implementation of collaborative search systems. The section also includes reflective remarks on how this research compares with prominent literature in the system-focused research field in CIS. This is followed by a final section [8.3] that identifies a list of the limitations of this research and the anticipated future work.

8.1 Summary

This research is motivated by the identified gap in the literature on the design and implementation of a proper and adequate set of awareness cues to aid the collaborators in collaborative information seeking (CIS) process. To summarise the results, the research and its objectives, detailed in [1.3.2], are listed next, and are followed by an explanation of the approach and process used to achieve these objectives. The key findings and results of each objective are also summarised below.

1. Development of the research artefact and the introduction of proper and adequate awareness cues.

To achieve that, the developed artefact should provide a functional collaborative search interface as a web app. This will provide the environment to test and manipulate a set of visual awareness cues in co-located or remote locations, as well as possibility of working in synchronous and asynchronous collaborations. The artefact should primarily, provide a mechanism display visual and contextual awareness cues of a curated set of the collaborators’ activities. These cues are associated with the search and results interfaces and are can be configured and exported to work and integrate with other third-party systems.

This objective is accomplished through the successful realisation of SearchAware. This IT software artefact is described as a mashup web application implemented provides a suitable platform to create a customizable Search User Interface (SUI). It acts as a search interface wrapper for a selected set of search engines and database. Moreover, SearchAware aims to be used for collaborative search, and its design follows the main design guidelines for SUIs and CIS systems. As a collaborative tool, SearchAware works in synchronous and asynchronous modes, and caters for the distributed nature of remote collaborators. It contains various features and functionalities aimed to assist and aid the

157

Chapter 8 Conclusion researcher to extend the search scope by adding external search engines and providers, and allow the record and monitoring of activities for analysis of the web application usage.

Two sets of novel awareness cues mechanisms are designed and implemented through SearchAware. SearchAware v1 supports the awareness through the integration of a Twitter social network timeline as the main visual timeline mode. This provides awareness to the collaborators from within SearchAware itself, and through following the designated timeline from Twitter main platforms. This approach enabled the search activities such as the: keywords queried, viewed results, rated result, and in addition the collaborators’ comments that appear in an automated manner on the group’s timeline.

SearchAware v2 is an overhaul of the SUI and the SERP. The placement of the awareness cues is refocused on the SERP list, which aims to increase the awareness exposure and reduces the clutter and number of clicks to reach the group’s search activities log. In addition, SearchAware 2 provides other forms of awareness to the typical user such as the personal search and view history logs. The main novel part is that the cues in SearchAware v2 are associated with each result through side visual cues and includes a collapsible hierarchal grid of the history associated with the results found and the collaborators’ activities. All these changes entailed extensive rework to the SearchAware interface and backend, yet it maintained a cohesive SUI design that follows prominent design guidelines stemming the literature. Therefore, SearchAware emphasized the collaborative search aspects and aimed towards achieving a seamless experience to the collaborators.

2. Evaluative studies to measure the effectiveness and usability of the awareness cues

Two user studies were performed, one for each version of the aretfact. Both included the collection of data; and analysis of the participants’ usage and their feedback. Through this iterative and incremental approach, the research followed the methodological approach of Design Science Research (DSR) (Vaishnavi and Kuechler, 2015) that emphasises the knowledge contribution throughout the processes of design, implementation and evaluation of the artefacts. This is a very similar approach to the key literature in this domain, on which this research was based on to formulate its approach.

Both studies were tested within an academic context with students. The first study used SearchAware v1 with groups of participants working on a shared undergraduate coursework. The naturalistic study results included the usage details and reflective feedback about the experience with the research artefact. These results were analysed and the results promoted further investigation to design and implement an adequate SUI to cater for the CIS process needs. The study provided an insight into the

158

Chapter 8 Conclusion challenges of CIS in both the technicality of the search system collaborative aspects of the search process. The main result was the need for an alternative approach to deploying an improved awareness mechanism in SearchAware, coupled with an alternative evaluation method that can provide a tighter control of the confounding factors.

In the second study, the second iteration of SearchAware was used. A controlled experiment, rather than a naturalistic study, was conducted for SearchAware v2, as this allowed for an in-depth insight of the effect of this newer form of awareness cues. Postgraduate students participated in the experiment as dyads to collaboratively and synchronously use SearchAware. The participants were exposed to a series of these cues that were systematically manipulated. This manipulation was in terms of the availability of the cues, with and without the communication channels, to understand and measure the cues effects on the participants.

The results of the second study suggest that these awareness cues influenced an improved search interface usability and that the collaborators felt a negligible impact from collaboration process, as suggested by the empirical work, implying that the main part of the evaluative feedback was aligned with the research expectations. Hence the research successfully achieved this part of the objective. However, the performance of the participants in the system performance measurements and productivity measurements did not meet the proposed expectations. Therefore, this suggests the need for further investigation of the productivity and performance aspect in terms of the measurements used, and the capabilities of the research artefact.

3. Reproducibility of the artefact

The details of the design of both versions of SearchAware are described and justfied across two chapters of this thesis in [3.3] and [Chapter 6]. These include a detailed description of the design concept, the mashup implementation approach, the development tools, the technologies used, as well the architectural design including the frontend and the backend. Moreover, the justification for the selection of design and implementation paths in each of these phases is provided along with the list of technical and implementation challenges faced during each of these phases.

While the integration with Twitter was removed in the second iteration of SearchAware to focus on the integration of the awareness cues within the search results, it can be considered as a proof of concept design. In that sense, Twitter or any other social network or platform can be integrated with SeearchAware, with minimal configuration effort. This is possible due to the modular design of SearchAware, which allows it to be reconfigured to post status updates such as search activities or comment to any third-party application.

159

Chapter 8 Conclusion

8.2 Contribution

The contribution of the research is presented in two subsections. The first directly answers the research question stated in [1.3.1] and highlights the intended contribution of the research. The second focuses on the software devised in this research beyond the focus on the awareness cues.

8.2.1 Implications for the Design and Implementation of Awareness Cues Reviewing the stated research question form [1.3.1]:

How can awareness cues be designed and implemented visually and contextually to aid explicit collaborative information seeking; and how can these cues affect the search interface usability?

The research question included two parts; the first part is concerned with the design and implementation of the awareness cues. Based on the user evaluative studies, the outcome of this research empirically demonstrated that applying and integration awareness cues within the search results could positively influence the collaborator's search experience in a shared search session. In particular, SearchAware v2 approach of associating each result hit with the collaborators’ search activities in an expandable form was statistically influential in terms of usability and interactivity. Furthermore, the awareness cues were contextually relevant to the domain in which the search process is being performed. As SearchAware uses scholarly search engines of academic publication databases, the search activities’ cues content included additional information such as whether a viewing was a publication or its abstract.

This aligns with the literature focused on CIS prototypes and systems, which was discussed detail in the Awareness Cues and Mechanisms subsection of the literature review in [2.5]. These studies also utilised other forms of collaborative activity awareness visual cues and mechanisms. Prominent examples of CIS software that includes the use of collaborative workspaces, e.g. Coagmento (Shah and Marchionini, 2010), timelines, e.g. SearchTogether (Morris and Horvitz, 2007) and CoSense (Paul and Morris, 2011), or sidebars such Querium v2 (Golovchinsky, Dunnigan and Diriye, 2012), and ResultsSpace (Capra et al., 2013). Other CIS research tools opted for associating partial activity awareness with the results, such as CollabSearch (Yue, Han and He, 2014) or ezDL (Böhm, Klas and Hemmje, 2014b) or simply provided a feed of the activities as a set of automated instant messages (Joho, Hannah and Jose, 2008). Most of these studies used the open web or TREC database as a provider of the data to be searched.

160

Chapter 8 Conclusion

SearchAware tested two approaches for delivering visual awareness mechanism. SearchAware v1 used a dual timeline of keywords and search activities where participants can view the timeline from within SearchAware or follow the stream on a social network feed. That approach revealed that participants felt that cues were helpful in remote collaboration, but SearchAware had some technical and usability issues. This particular issue is typically faced by other CIS prototypes, such as SearchTogether and Querium, and prompted redesigned artefacts to improve the usability and ultimately the collaboration aspect of the search process (Morris, 2013). In fact, such iterations are a core part of the DSR methodology used in this research.

Consequently, SearchAware v2 focused on the integration of contextually relevant awareness cues within the search results, therefore eliminating the need to check a continuous feedback of activities. This persistence set of cues in SERP, discussed briefly in SearchTogether and CoSense, was implemented in SearchAware further than any previous CIS system. The novelty in SearchAware that the search activities cues and comments are coupled with search results, so that the collaborators can access the search session at any time (or place) during the search to view their search group’s search activities.

As for the second part of the research question, the research provided empirical evidence suggesting that the integration of the search awareness cues can have further influence and better and seamless information seeking process if designed and implemented in a similar approach to SearchAware v2. The design of the cues from these studies demonstrated that the content and relevancy of the cues must be carefully selected to match the context in which the intended search will happen as it influences these awareness cues considerably. This was highlighted by Hyldegård, particularly in collaborative search (Hyldegård, 2009).

The focus on the context is also very relevant to understand which of the CIS aspects from the identified list, compiled earlier in section [2.4], are more relevant to the designated CIS. For example, in SearchAware, the aspect of division of labour was more apparent as the information needs can be considered inherently complex. A different example is a leisure travel booking, which is more about sharing a result and justify a selection of a hotel or a restaurant, in which the aspect of communication and preference proceeds that of the division of labour. This is detailed in a study of comparing and contrasting the subtle differences in the nature of the search context. Newman distinguishes between tightly-coordinated collaboration, which usually takes place in more complex collaborations, such as academic research and professional work context, and direct-coordinated collaboration, such as leisure and general search contexts in (Newman et al., 2015).

161

Chapter 8 Conclusion

8.2.2 SearchAware as a Research Software Artefact The design and implementation of SearchAware is acknowledged and inspired by the prominent and detailed work in system-focused CIS literature. Noticeably in this area: Coagmento (Shah and Marchionini, 2010; González-Ibáñez and Shah, 2011; González-Ibáñez, Haseki and Shah, 2013), SearchTogether and CoSense (Morris and Horvitz, 2007; Paul and Morris, 2011), Querium (Golovchinsky, Dunnigan and Diriye, 2012), and ResultsSpace (Capra et al., 2013). These experimental tools provided a comprehensive and detailed perspective on the design and implementation of CIS systems. However, SearchAware enhances a set of specific features that are beyond the research focus on awareness cues, these include advances in the technical aspects mainly focused on the web- based modular development, extensibility in terms of the addition of newer search engines, and the availability of a set of tools to capture and generate comprehensive usage logs. Other features focus on the design approach, and relative ease at which SearchAware is applicable and configurable to work with other search contexts, provided a configurable API search engine or databases are provided. These features that SearchAware include are detailed next.

Developed specifically as a mashup web app, as detailed in [4.3], SearchAware was tested thoroughly in the domain of academic publications, which is suitable for the expected participants in this research. As it is integrated with Mendeley and Microsoft Academic Search, both search engines are connected to large and frequently updated scholarly databases. Moreover, it provides a suitable and customizable alternative search aggregator interface that can be used for studies beyond the focus on awareness cues. The extensibility of SearchAware is technically possible with other database services due to its modular and mashup development approach. In essence, SearchAware can be extended to include other scholarly search engines or databases such as PubMed or Scopus with minimal reconfiguring to its backend code and frontend interface.

Furthermore, SearchAware could also be connected to other domains and contexts where a collaborative search process may occur, either in the workplace or lifestyle domains. These may include comparison websites including travel services such as flight, hotel and car rental bookings, utility and phone plan services, or shopping and retail services. These might require further changes to the interface to match the context of the domain and appropriate for the type of information needed in such domains.

The iterative and incremental developmental process in the design and implementation of SearchAware follows that of Design Science Research (DSR), an established and systematic approach for Information Systems (Vaishnavi and Kuechler, 2015). As detailed in [3.2], DSR emphasises that the lessons learned during this process are part of the contribution to the domain knowledge. Although

162

Chapter 8 Conclusion there are limited examples of the use of DSR in interactive systems (Adikari, McDonald and Campbell, 2009), this research contributes towards adding knowledge to that set of studies.

Finally, SearchAware includes an extensive user activities log system that generates a detailed record of all the activities user perform. The logging mechanism is included in all features of the web app and exported as a Comma-Separated Values (CSV) file which can be used for further analysis of the search process.

SearchAware, therefore, represents an appropriate tool to be considered in the limited number of collaborative search systems available for the CIS research domain community. Its design and implementation aspect are documented in this thesis for reproducibility. Moreover, the design and implementation lessons detailed here provide a resource for any future similar systems that can advance the utilisation of awareness cues or study. Furthermore, SearchAware can be used for further research to expand the understanding of collaborative search practices, such as synergic work (González-Ibáñez, Shah and Córdova-Rubio, 2011) or surveys of CIS practices (Aldosari et al., 2016); it can also be used for further comparisons of search behaviours and activities between it and other CIS system discussed earlier in this subsection such as the study of daily tasks (Kelly and Payne, 2014).

8.3 Limitations and Future Work

Some of the main limitations of this research can be addressed in a selected set of anticipated improvements to SearchAware. These can be in terms of the advances in its design and implementation and may be coupled with follow-up future studies using SearchAware. These improvements can include, but not limited to, the following changes, as descrivbed in the next subsections.

8.3.1 Extended Use of Interface Control and Input Features As noted in [2.4.3], the user interface plays a significant role in CIS and collaboration groupware in general. The focus on the Search User Interface (SUI) in this research was presented in [2.1.3]. The main issues related to the design of search user interfaces are discussed thoroughly in (Hearst, 2009; Morville and Callender, 2010; Wilson, 2011) and provide an extensive set of features to be included in these interactive interfaces.

In SearchAware, the implementation was focused on a selected set of major current techniques and features in the SUI. Nevertheless, some missing features could, if included, enhance the system usability; these include auto completion of queries, suggested keywords, similar results, and query reformulation features (Wilson, 2011). Although these are suggested to improve the solitary search

163

Chapter 8 Conclusion process only, they may improve the collaborative search process as well. Implementing these will require additional structural changes to how SearchAware is implemented. Moreover, they might be curtailed by the constrained support of the providing search engine or database.

8.3.2 Further Search Contexts For this research, SearchAware integrated the results of two main search engines linked to large dedicated academic publications databases. Nevertheless, even though most search engines of other databases use similar API connectivity, SearchAware was not extended to these other types of engines that provide other forms services and resources in this research.

Future research can utilise and explore further contextual domains, like price comparison websites of travel bookings or broadband services, as highlighted earlier in [8.2.1]. Although using the general web, i.e. the WWW, might not be suitable for this interface type due to the unlimited variety of web pages’ content and context. However, expanding SearchAware to contextual relevant search domains such as news articles or correspondences should require minimal reconfiguration.

8.3.3 Use of Modern Web Technologies The new advances in development tools technologies, such as ASP.NET Core (which is the successor for ASP.NET 4.0, used in this research) can provide improved dynamic pages and seamless refresh of individual controls. In addition, a new version of SignalR, a complementary technology for ASP.NET that provides simplicity in the development of real-time web communicating directly between clients, rather than relying on the server, which may lead to significant improvements in terms of speed of exchange and display of awareness cues.

Another aspect in the used of newer web technologies is the inclusion of integrated communication channel, mainly a built-in instant messaging channel instead of utilising on an available third-party provided such as Google Chat which was used in this research.

8.3.4 Limitations of System-focused Evaluation The measurements used to evaluate the user’s productivity and performance in SearchAware v2 experiment presented in [7.2.1] did not achieve significant results with the introduction of the awareness cues. The participants’ results did not show a statistical difference in terms of an increase of the results viewed or a decrease in the overlapped results read. This is contrary to the interactivity and usability measures, which found that awareness cues influenced the participants in a positive manner.

164

Chapter 8 Conclusion

Carefully examining the s research that utilised the similar set of measurements also exhibited similar results, in particular in the conditions which used collaborative search, as these typically included a solitary search condition as an independent variable e.g. (Shah and Marchionini, 2010) and (Capra et al., 2013). As these studies did not find statistical significance between the different experimental conditions, other research opted not to use these measures at all. Further studies can focus on a different set of measures or use an improved set of measures. For example, the IIR measures suggested in (Shah, 2014b) adapted the classical IR measures used primarily in solitary search to which take into consideration the use of the open web (Kelly, 2009). Therefore, there needs to be careful examination and consideration of the productivity and performance of IIR measurements for collaborative search.

8.3.5 Laboratory Experiments’ Constraints In the study chapters [Chapter 5] which introduced the naturalistic study and [Chapter 7] which presented the laboratory experiment, it was highlighted that there are constraints on the form of such naturalistic user studies that are performed in the context. The first user study was within an academic context and groups of undergraduate students as participants. The study did not enforce any strict time constraints in the duration (few weeks) and can be done online at any time and venue. However, several participants had limited or even no interaction with the research artefact, which rendered a solid analysis difficult to be performed.

As for the laboratory experiment that is, by definition, bound in an artificial context, several issues can skew the results of the participants. Noticeably in collaborative search studies, two factors have a noticeable impact, the first is the search task assigned, which was discussed thoroughly in [7.3.1], and the second is the task duration (Shah, 2014b). In the Coagmento study, three sessions of search tasks were performed, two of which being collaborative, and each session lasted for 25 minutes (Shah and Marchionini, 2010). For this study, approximately 23 minutes were given per session, also with a total of three sessions. This was based on the type of the search task and from conducting a couple of trial experiments.

As this might be a limiting factor for this study. The set of timing for each session in an artificial manner is mainly is due to practical issues in such contexts, but as Purchase highlights “there is no golden rule” in such scenarios (Purchase, 2012, p. 60). Limiting the time can force participants to finish hastily and may cause fatigue and boredom, and leaving the timing open is impractical and, as with limited timings, may also cause fatigue and boredom. Purchase suggests few tips to handle these limitations, but ultimately it depends on the type and task of the study (2012, p. 61),. The balance between these limited and unlimited duration is a delicate issue which can be explored further through the

165

Chapter 8 Conclusion manipulation of the duration or the number of sessions for collaborative information seeking laboratory experiments.

166

References

Adikari, S., McDonald, C. and Campbell, J. (2009) ‘Little Design Up-Front: A Design Science Approach to Integrating Usability into Agile Requirements Engineering’, in, pp. 549–558. doi: 10.1007/978-3- 642-02574-7_62.

Al-Thani, D., Stockman, T. and Tombros, A. (2015) ‘The Effects of Cross-modal Collaboration on the Stages of Information Seeking’, in Proceedings of the XVI International Conference on Human Computer Interaction - Interacción ’15. New York, New York, USA: ACM Press, pp. 1–8. doi: 10.1145/2829875.2829925.

Albar, A. A. (2015) ‘A Proposed CIS Interface Based on the Analysis of Problem-Solving in Online Technical Support’, in Hansen, P., Shah, C., and Klas, C.-P. (eds) Collaborative Information Seeking: Best Practices, New Domains and New Thoughts. Springer International Publishing, pp. 117–137. doi: 10.1007/978-3-319-18988-8_7.

Aldosari, M., Sanderson, M., Tam, A. and Uitdenbogerd, A. L. (2016) ‘Understanding collaborative search for places of interest’, Journal of the Association for Information Science and Technology, 67(6), pp. 1331–1344. doi: 10.1002/asi.23466.

Alexa (2016) Alexa Top 500 Global Sites. Available at: http://www.alexa.com/topsites (Accessed: 25 May 2016).

Alton-Scheidl, R., Bockle, H. and Schmutzer, R. (1997) ‘Web4Groups - A Tool for telematic collaboration’, in Database and Expert Systems Applications. 8th International Conference, DEXA ’97. Proceedings. IEEE Comput. Soc, pp. 425–430. doi: 10.1109/DEXA.1997.617328.

Amershi, S. and Morris, M. R. (2008) ‘CoSearch: A System for Co-located Collaborative Web Search’, in Proceeding of the twenty-sixth annual CHI conference on Human factors in computing systems - CHI ’08. New York, New York, USA: ACM Press, pp. 1647–1656. doi: 10.1145/1357054.1357311.

Amershi, S. and Morris, M. R. (2009) ‘Co-located collaborative web search: understanding status quo practices’, in Proceedings of the ACM Conference on Human factors in computing systems (CHI 2009). New York, New York, USA: ACM New York, NY, USA, pp. 3637–3642. doi: 10.1145/1520340.1520547.

Attfield, S., Blandford, A. and Makri, S. (2010) ‘Social and interactional practices for disseminating current awareness information in an organisational setting’, Information Processing & Management. Elsevier Ltd, 46(6), pp. 632–645. doi: 10.1016/j.ipm.2009.10.003.

‘awareness’ (2015) Oxford English Dictionary. Oxford University Press. Available at: http://www.oed.com/view/Entry/13894?redirectedFrom=awareness+.

Baeza-Yates, R. and Ribeiro-Neto, B. (1999) Modern Information Retrieval. First. Addison Wesley (ACM Press). Available at: http://www.mir2ed.org/.

Bagozzi, R. P. (2007) ‘The Legacy of the Technology Acceptance Model and a Proposal for a Paradigm Shift’, Journal of the association for information systems, 8(4), pp. 244–254. Available at: http://aisel.aisnet.org/jais/vol8/iss4/3/ (Accessed: 3 January 2015).

Bates, M. J. (1989) ‘The design of browsing and berrypicking techniques for the online search interface’, Online Information Review, 13(5), pp. 407–424. doi: 10.1108/eb024320.

167

References

Bates, M. J. (2002) ‘Toward an integrated model of information seeking and searching’, New Review of Information Behaviour Research, 3(1), pp. 1–15. Available at: http://ptarpp2.uitm.edu.my/silibus/TOWARDANINTEGRATEDMODEL.pdf (Accessed: 24 November 2011).

Batrinca, L., Khan, M. T., Billman, D., Aydemir, B. and Convertino, G. (2013) ‘A Timeline Visualization for Multi-team Collaborative Planning’, CHI ’13 Extended Abstracts on Human Factors in Computing Systems, pp. 157–162. doi: 10.1145/2468356.2468385.

Beal, V. (no date) SOA - Service-Oriented Architecture. Available at: http://www.webopedia.com/TERM/S/Service_Oriented_Architecture.html.

Blake, C. and Pratt, W. (2006a) ‘Collaborative information synthesis I: A model of information behaviors of scientists in medicine and public health’, Journal of the American Society for Information Science and Technology, 57(13), pp. 1740–1749. doi: 10.1002/asi.20487.

Blake, C. and Pratt, W. (2006b) ‘Collaborative information synthesis II: Recommendations for information systems to support synthesis activities’, Journal of the American Society for Information Science and Technology, 57(14), pp. 1888–1895. doi: 10.1002/asi.20486.

Blandford, A. and Attfield, S. (2010) ‘Interacting with Information’, Synthesis Lectures on Human- Centered Informatics, 3(1), pp. 1–99. doi: 10.2200/S00227ED1V01Y200911HCI006.

Blandford, A., Cox, A. and Cairns, P. (2008) ‘Controlled experiments’, in Research Methods for Human- Computer Interaction, pp. 1–16.

Böhm, T., Klas, C.-P. and Hemmje, M. (2013) ‘Supporting Collaborative Information Seeking and Searching in Distributed Environments’, in LWA 2013. Bamberg, Germany. Available at: http://www.minf.uni-bamberg.de/lwa2013/FinalPapers/lwa2013_submission_2.pdf (Accessed: 4 May 2014).

Böhm, T., Klas, C.-P. and Hemmje, M. (2014a) ‘Collaborative Information Seeking in Professional Work- Settings: A Study of Equipment Utilization’, Datenbank-Spektrum, 14(1), pp. 29–38. doi: 10.1007/s13222-014-0145-2.

Böhm, T., Klas, C.-P. and Hemmje, M. (2014b) ‘ezDL: Collaborative Information Seeking and Retrieval in a Heterogeneous Environment’, Computer, 47(3), pp. 32–37. doi: 10.1109/MC.2013.447.

Bohøj, M., Borchorst, N. G., Bouvin, N. O., Bødker, S. and Zander, P.-O. (2010) ‘Timeline Collaboration’, in Proceedings of the 28th international conference on Human factors in computing systems CHI 10. Atlanta, GA, USA, pp. 523–532. doi: 10.1145/1753326.1753404.

Borlund, P. (2000) ‘Experimental components for the evaluation of interactive information retrieval systems’, Journal of Documentation, 56(1), pp. 71–90. doi: 10.1108/EUM0000000007110.

Borlund, P. (2003) ‘The IIR evaluation model: a framework for evaluation of interactive information retrieval systems’, Information Research. T.D. Wilson, 8(3). Available at: http://informationr.net/ir/8- 3/paper152.html (Accessed: 19 June 2012).

Borlund, P. and Ingwersen, P. (1997) ‘The development of a method for the evaluation of interactive information retrieval systems’, Journal of Documentation, 53(3), pp. 225–250. doi: 10.1108/EUM0000000007198.

168

References

Borlund, P. and Schneider, J. W. (2010) ‘Reconsideration of the simulated work task situation’, in Proceeding of the third symposium on Information interaction in context - IIiX ’10. New Brunswick, New Jersey, USA: ACM Press, pp. 155–164. doi: 10.1145/1840784.1840808.

Brin, S. and Page, L. (1998) ‘The anatomy of a large-scale hypertextual Web search engine’, Computer Networks and ISDN Systems, 30(1–7), pp. 107–117. doi: 10.1016/S0169-7552(98)00110-X.

Broder, A. (2002) ‘A taxonomy of web search’, SIGIR Forum. ACM, 36(2), pp. 3–10. doi: 10.1145/792550.792552.

Burghardt, M., Heckner, M. and Wolff, C. (2012) ‘The Many Ways of Searching the Web Together : A Comparison of Social Search Engines’, in Lewandowski, D. (ed.) Web Search Engine Research. Emerald Group Publishing Limited (Library and Information Science), pp. 19–46. doi: 10.1108/S1876- 0562(2012)002012a004.

Cairns, P. and Cox, A. L. (2008) Research Methods for Human-Computer Interaction. Cambridge University Press. Available at: http://www.cambridge.org/ir/academic/subjects/psychology/applied- psychology/research-methods-human-computer-interaction.

Capra, R., Arguello, J., Chen, A. T., Hawthorne, K., Marchionini, G. and Shaw, L. (2012) ‘The ResultsSpace collaborative search environment’, in Proceedings of the 12th ACM/IEEE-CS joint conference on Digital Libraries - JCDL ’12. New York, New York, USA: ACM Press, p. 435. doi: 10.1145/2232817.2232929.

Capra, R., Chen, A. T., Hawthorne, K. and Arguello, J. (2012) ‘ResultsSpace: An Experimental Collaborative Search Environment’, Proceedings of the American Society for Information Science and Technology, 49(1), pp. 1–4. doi: 10.1002/meet.14504901371.

Capra, R., Chen, A. T., Hawthorne, K., Arguello, J., Shaw, L. and Marchionini, G. (2012) ‘Design and Evaluation of a System to Support Collaborative Search’, Proceedings of the American Society for Information Science and Technology, 49(1), pp. 1–10. doi: 10.1002/meet.14504901181.

Capra, R., Chen, A. T., McArthur, E. and Davis, N. (2013) ‘Searcher actions and strategies in asynchronous collaborative search’, Proceedings of the American Society for Information Science and Technology. Montreal, Quebec, Canada, 50(1), pp. 1–10. doi: 10.1002/meet.14505001077.

Capra, R., Marchionini, G., Velasco-Martin, J. and Muller, K. (2010) ‘Tools-at-hand and learning in multi-session, collaborative search’, in Proceedings of the 28th international conference on Human factors in computing systems - CHI ’10. New York, New York, USA: ACM Press, pp. 951–960. doi: 10.1145/1753326.1753468.

Carroll, J. M., Neale, D. C., Isenhour, P. L., Rosson, M. B. and McCrickard, D. S. (2003) ‘Notification and awareness: synchronizing task-oriented collaborative activity’, International Journal of Human- Computer Studies. Academic Press, Inc., 58(5), pp. 605–632. doi: 10.1016/S1071-5819(03)00024-7.

Carroll, J. M., Rosson, M. B., Convertino, G. and Ganoe, C. H. (2006) ‘Awareness and teamwork in computer-supported collaborations’, Interacting with Computers, 18(1), pp. 21–46. doi: 10.1016/j.intcom.2005.05.005.

Carroll, J. M., Rosson, M. B., Farooq, U. and Xiao, L. (2009) ‘Beyond being aware’, Information and Organization. Elsevier Ltd, 19(3), pp. 162–185. doi: 10.1016/j.infoandorg.2009.04.004.

Chakrabarti, S. (2003) ‘Mining the Web: Discovering Knowledge from Hypertext Data’, Online Information Review. First. Morgan Kaufmann, 27(3), p. 344. doi: 10.1108/14684520310489113.

169

References

Cheverst, K., Dix, A., Fitton, D., Graham, C. and Rouncefield, M. (2009) ‘Situatedness of Awareness Information: Impact on the Design and Usage of Awareness Systems’, in Markopoulos, P., De Ruyter, B., and Mackay, W. (eds) Awareness Systems: Advances in Theory, Methodology and Design. London: Springer-Verlag (Human-Computer Interaction Series). doi: 10.1007/978-1-84882-477-5_17.

Chi, E. H. (2009) ‘Information Seeking Can Be Social’, Computer. IEEE, 42(3), pp. 42–46. doi: 10.1109/MC.2009.87.

Chi, E. H. and Pirolli, P. (2006) ‘Social Information Foraging and Collaborative Search’, in Human Computer Interaction Consortium HCIC Workshop. Winter Park, CO, p. 5. Available at: http://www2.parc.com/istl/groups/uir/publications/items/UIR-2006-06-Chi-SocialForaging.pdf (Accessed: 24 May 2013).

Clark, H. H. (1996) Using Language. Cambridge: Cambridge University Press. doi: 10.1017/CBO9780511620539.

Clark, H. H. and Brennan, S. E. (1991) ‘Grounding in Communication’, in Resnick, L. B., Levine, J. M., and Teasley, S. D. (eds) Perspectives on Socially Shared Cognition. Washington DC, USA: American Psychological Association.

Collins, L. M., Mane, K. K., Martinez, M. L. B., Hussell, J. A. T. and Luce, R. E. (2005) ‘ScienceSifter: Facilitating Activity Awareness in Collaborative Research Groups through Focused Information Feeds’, in First International Conference on e-Science and Grid Computing (e-Science’05). IEEE, pp. 40–47. doi: 10.1109/E-SCIENCE.2005.72.

Computer-supported cooperative work (2014) Wikipedia. Available at: https://en.wikipedia.org/wiki/Computer-supported_cooperative_work (Accessed: 10 May 2014).

Convertino, G., Neale, D. C., Hobby, L., Carroll, J. M. and Rosson, M. B. (2004) ‘A Laboratory Method for Studying Activity Awareness’, Proceedings of the third Nordic conference on Human-computer interaction - NordiCHI ’04. New York, New York, USA: ACM Press, (1), pp. 313–322. doi: 10.1145/1028014.1028063.

Coyle, M. and Smyth, B. (2008) ‘(Web Search) shared: Social Aspects of a Collaborative, Community- Based Search Network’, in Nejdl, W., Kay, J., Pu, P., and Herder, E. (eds) Adaptive Hypermedia and Adaptive Web-Based Systems. Springer (LNCS), pp. 103–112. Available at: http://www.springerlink.com/index/e7588604m274760q.pdf (Accessed: 13 December 2011).

Crumlish, C. and Malone, E. (2009) Designing Social Interfaces. First. Edited by M. Treseler. O’Reilly Media, Inc.

Cutrell, E. and Guan, Z. (2007) ‘What are you looking for? An eye-tracking study of information usage in web search’, Proceedings of the SIGCHI conference on Human factors in computing systems - CHI ’07, p. 407. doi: 10.1145/1240624.1240690.

Czerwinski, M., Cutrell, E. and Horvitz, E. (2000) ‘Instant messaging: Effects of Relevance and Timing’, in People and computers XIV: Proceedings of HCI. Vol. 2. British Computer Society, 2000. British Computer Society, pp. 71–76. Available at: http://research.microsoft.com/en- us/um/people/marycz/hci2000.pdf (Accessed: 10 April 2014).

Davis, F. D. (1989) ‘Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology’, MIS Quarterly. JSTOR, 13(3), p. 319. doi: 10.2307/249008.

Denning, P. J. and Yaholkovsky, P. (2008) ‘Getting to “We”’, Communications of the ACM, 51(4), p. 19. doi: 10.1145/1330311.1330316.

170

References

Dervin, B. (1998) ‘Sense-making theory and practice: an overview of user interests in knowledge seeking and use’, Journal of Knowledge Management, 2(2), pp. 36–46. doi: 10.1108/13673279810249369.

Diriye, A. and Golovchinsky, G. (2012) ‘Querium: A Session-Based Collaborative Search System’, in Advances in Information Retrieval. Springer-V, pp. 583–584. doi: 10.1007/978-3-642-28997-2_72.

Dix, A., Finlay, J., Abowd, G. D. and Beale, R. (2003) Human Computer Interaction. Third Edit. Essex, England: Prentice Hall. Available at: www.hcibook.com/e3/.

Dourish, P. and Bellotti, V. (1992) ‘Awareness and Coordination in Shared Workspaces’, Proceedings of the 1992 ACM conference on Computer-supported cooperative work (CSCW 1992). New York, New York, USA: ACM Press, pp. 107–114. doi: 10.1145/143457.143468.

Dupont, C. (2008) SearchWiki: make search your own, Google Offical Blog. Available at: https://googleblog.blogspot.com/2008/11/searchwiki-make-search-your-own.html.

Ellis, C. A., Gibbs, S. J. and Rein, G. (1991) ‘Groupware: some issues and experiences’, Communications of the ACM, 34(1), pp. 39–58. doi: 10.1145/99977.99987.

Endsley, M. R. (1995) ‘Toward a Theory of Situation Awareness in Dynamic Systems’, Human Factors: The Journal of the Human Factors and Ergonomics Society, 37(1), pp. 32–64. doi: 10.1518/001872095779049543.

Endsley, M. R. and Jones, D. G. (2011) Designing for Situation Awareness: An Approach to User- Centered Design. Second Edi. CRC Press.

Evans, B. M. and Chi, E. H. (2008) ‘Towards a model of understanding social search’, in Proceedings of the 2008 ACM conference on Computer supported cooperative work (CSCW 2008). San Diego, California, USA: ACM, p. 485. doi: 10.1145/1460563.1460641.

Evans, B. M. and Chi, E. H. (2010) ‘An elaborated model of social search’, Information Processing & Management. Elsevier Ltd, 46(6), pp. 656–678. doi: 10.1016/j.ipm.2009.10.012.

Evans, B. M., Kairam, S. and Pirolli, P. (2010) ‘Do your friends make you smarter?: An analysis of social strategies in online information seeking’, Information Processing & Management. Elsevier, 46(6), pp. 679–692. doi: 10.1016/j.ipm.2009.12.001.

Farooq, U., Ganoe, C. H., Carroll, J. M., Councill, I. G. and Lee Giles, C. (2008) ‘Design and evaluation of awareness mechanisms in CiteSeer’, Information Processing & Management, 44(2), pp. 596–612. doi: 10.1016/j.ipm.2007.05.009.

Fernández-Luna, J. M., Huete, J. F., Pérez-Vázquez, R. and Rodríguez-Cano, J. C. (2010) ‘CIRLab: A groupware framework for collaborative information retrieval research’, Information Processing & Management, 46(6), pp. 749–761. doi: 10.1016/j.ipm.2009.10.009.

Fidel, R. (2012) Human Information Interaction. Cambridge, Massachusetts, USA: The MIT Press. Available at: http://mitpress.universitypressscholarship.com/view/10.7551/mitpress/9780262017008.001.0001/ upso-9780262017008 (Accessed: 8 November 2014).

Filho, F. F., Olson, G. M. and de Geus, P. L. (2010) ‘Kolline: a task-oriented system for collaborative information seeking’, in Proceedings of the 28th ACM International Conference on Design of Communication - SIGDOC ’10. New York, New York, USA: ACM Press, pp. 89–94. doi: 10.1145/1878450.1878465.

171

References

Flickner, M., Sawhney, H., Niblack, W., Ashley, J., Huang, Q., Dom, B., Gorkani, M., Hafner, J., Lee, D., Petkovic, D., Steele, D. and Yanker, P. (1995) ‘Query by image and video content: the QBIC system’, Computer, pp. 23–32. doi: 10.1109/2.410146.

Foley, C. and Smeaton, A. F. (2009) ‘Synchronous Collaborative Information Retrieval: Techniques and Evaluation’, in Boughanem, M., Berrut, C., Mothe, J., and Soule-Dupuy, C. (eds) European Conference on Information Retrieval. Berlin, Heidelberg: Springer Berlin Heidelberg (Lecture Notes in Computer Science), pp. 42–53. doi: 10.1007/978-3-642-00958-7_7.

Foley, C. and Smeaton, A. F. (2010) ‘Division of labour and sharing of knowledge for synchronous collaborative information retrieval’, Information Processing & Management. Elsevier Ltd, 46(6), pp. 762–772. doi: 10.1016/j.ipm.2009.10.010.

Foster, J. (2007) ‘Collaborative Information Seeking and Retrieval’, in Blaise Cronin (ed.) Annual Review of Information Science and Technology. Medford, New Jersey: Wiley Periodicals, pp. 329–356. doi: 10.1002/aris.1440400115.

Freyne, J. and Smyth, B. (2006) ‘Cooperating search communities’, in Wade, V., Ashman, H., and Smyth, B. (eds) Proceedings of the 2006 4th Adaptive Hypermedia and Adaptive Web-Based Systems. Springer, pp. 101–110. doi: 10.1007/11768012_12.

Fuks, H., Raposo, A., Gerosa, M. A., Pimental, Pimental, Mariano and Mariano (2008) ‘The 3C Collaboration Model’, in Encyclopedia of E-Collaboration. IGI Global, pp. 637–644. doi: 10.4018/978- 1-59904-000-4.ch097.

Ganoe, C. H., Somervell, J. P., Neale, D. C., Isenhour, P. L., Carroll, J. M., Rosson, M. B. and McCrickard, D. S. (2003) ‘Classroom BRIDGE’, in Proceedings of the 16th annual ACM symposium on User interface software and technology - UIST ’03. New York, New York: ACM Press, pp. 21–30. doi: 10.1145/964696.964699.

Garcia-Molina, H., Koutrika, G. and Parameswaran, A. (2011) ‘Information Seeking: Convergence of Search, Recommendations, and Advertising’, Communications of the ACM, 1 November, p. 121. doi: 10.1145/2018396.2018423.

Ghani, J. A., Supnick, R. and Rooney, P. (1991) ‘The experience of flow in computer-mediated and in face-to-face groups’, in Proceedings of the twelfth international conference on Information systems (ICIS ’91). New York; New York; USA, pp. 229–237. Available at: http://dl.acm.org/citation.cfm?id=126686.150736 (Accessed: 5 March 2012).

Gianoutsos, S. and Grundy, J. (1996) ‘Collaborative work with the World Wide Web: Adding CSCW support to a Web browser’, in Proceedings of the Oz-CSCW. Brisbane, Australian, pp. 14–21. Available at: http://www.ict.swin.edu.au/personal/jgrundy/papers/ozcscw96.pdf (Accessed: 13 May 2014).

Golovchinsky, G., Adcock, J., Pickens, J., Qvarfordt, P. and Back, M. (2008) ‘Cerchiamo: A Collaborative Exploratory Search Tool’, in Proceedings of Computer Supported Cooperative Work 2008 Workshops (CSCW 2008). ACM Press, pp. 4–5. Available at: http://fxpal.fxpal.com/publications/FXPAL-PR-08- 470.pdf.

Golovchinsky, G., Diriye, A. and Pickens, J. (2011) ‘Designing for Collaboration in Information Seeking’, in The 5th Workshop on Human-Computer Interaction and Information Retrieval. Mountain View, California. Available at: http://collab.infoseeking.org/resources/papersCIS11/GolovchinskyPaper.pdf (Accessed: 9 May 2012).

172

References

Golovchinsky, G., Dunnigan, A. and Diriye, A. (2012) ‘Designing a tool for exploratory information seeking’, in Proceedings of the 2012 ACM annual conference extended abstracts on Human Factors in Computing Systems Extended Abstracts - CHI EA ’12. New York, New York, USA: ACM Press, p. 1799. doi: 10.1145/2212776.2223712.

Golovchinsky, G., Pickens, J. and Back, M. (2009) ‘A Taxonomy of Collaboration in Online Information Seeking’, in Presented at 1st Intl Workshop on Collaborative Information Seeking at JCDL 2008, pp. 1– 3. Available at: http://arxiv.org/abs/0908.0704 (Accessed: 7 December 2011).

Golovchinsky, G., Qvarfordt, P. and Pickens, J. (2009) ‘Collaborative Information Seeking’, Computer, 42(3), pp. 47–51. doi: 10.1109/MC.2009.73.

González-Ibáñez, R., Haseki, M. and Shah, C. (2013) ‘Let’s search together, but not too close! An analysis of communication and performance in collaborative information seeking’, Information Processing & Management, 49(5), pp. 1165–1179. doi: 10.1016/j.ipm.2012.12.008.

González-Ibáñez, R. and Shah, C. (2011) ‘Coagmento: A System for Supporting Collaborative Information Seeking’, in The 74th Annual Meeting of the American Society for Information Science and Technology (ASIST 2011). New Orleans, LA, USA: ASIST. Available at: http://www.asis.org/asist2011/posters/336_FINAL_SUBMISSION.pdf (Accessed: 29 September 2011).

González-Ibáñez, R., Shah, C. and Córdova-Rubio, N. (2011) ‘Smile! Studying expressivity of happiness as a synergic factor in collaborative information seeking’, Proceedings of the American Society for Information Science and Technology, 48(1), pp. 1–10. doi: 10.1002/meet.2011.14504801171.

Gray, B. (1989) Collaborating: Finding Common Ground for Multiparty Problems. First Edit. Jossey- Bass.

Gregor, S. and Hevner, A. R. (2013) ‘Positioning and presenting design science research for maximum impact’, MIS Quarterly, 37(2), pp. 337–355. doi: 10.2753/MIS0742-1222240302.

Gross, T. and Prinz, W. (2003) ‘Awareness in Context: A Light-Weight Approach’, in Kuutti, K., Karste, E. H., Fitzpatrick, G., Dourish, P., and Schmidt, K. (eds) Proceedings of the Eighth European Conference on Computer Supported Cooperative Work 2003. Helsinki, Finland: Springer, pp. 14–18. doi: 10.1007/978-94-010-0068-0_16.

Grudin, J. (1994a) ‘Computer-Supported Cooperative Work: History and Focus’, Computer, 27(5), pp. 19–26. doi: 10.1109/2.291294.

Grudin, J. (1994b) ‘Groupware and Social Dynamics: Eight Challenges for Developers’, Communications of the ACM, 37(1), pp. 92–105. doi: http://doi.acm.org/10.1145/175222.175230.

Grudin, J. and Poltrock, S. E. (2012) ‘Taxonomy and Theory in Computer Supported Cooperative Work’, in The Oxford Handbook of Organizational Psychology, pp. 1–47. doi: 10.1093/oxfordhb/9780199928286.013.0040.

Gutwin, C. A. and Greenberg, S. (2002) ‘A Descriptive Framework of Workspace Awareness for Real- Time Groupware’, Computer Supported Cooperative Work (CSCW). Springer, 11(3–4), pp. 411–446. doi: 10.1023/A:1021271517844.

Gutwin, C. and Greenberg, S. (2004) ‘The Importance of Awareness for Team Cognition in Distributed Collaboration’, in Salas, E. and Fiore, S. M. (eds) Team Cognition: Understanding the Factors That Drive Process and Performance, p. 268. doi: 10.1.1.17.6293.

173

References

Haas, K., Mika, P., Tarjan, P. and Blanco, R. (2011) ‘Enhanced results for web search’, in Proceedings of the 34th international ACM SIGIR conference on Research and development in Information - SIGIR ’11. New York, New York, USA: ACM Press, p. 725. doi: 10.1145/2009916.2010014.

Halvey, M., Vallet, D., Hannah, D., Feng, Y. and Jose, J. M. (2010) ‘An asynchronous collaborative search system for online video search’, Information Processing & Management. Elsevier Ltd, 46(6), pp. 733– 748. doi: 10.1016/j.ipm.2009.11.007.

Halvey, M., Vallet, D., Hannah, D. and Jose, J. M. (2009) ‘ViGOR: A Grouping Oriented Interface for Search and Retrieval in Video Libraries’, in Proceedings of the 2009 joint international conference on Digital libraries - JCDL ’09. Austin, Texas: ACM Press, pp. 87–96. doi: 10.1145/1555400.1555415.

Hansen, P. and Järvelin, K. (2005) ‘Collaborative Information Retrieval in an information-intensive domain’, Information Processing & Management, 41(5), pp. 1101–1119. doi: 10.1016/j.ipm.2004.04.016.

Harr, R., Wiberg, M. and Whittaker, S. (2011) ‘Understanding Interaction Search Behavior in Professional Social Networks’, Human Technology. University of Jyväskylä, Agora Center, 7(3), pp. 194–215. Available at: http://urn.fi/URN: NBN:fi:jyu-2011112211712 (Accessed: 20 May 2014).

Hart, S. G. (2006) ‘Nasa-Task Load Index (NASA-TLX); 20 Years Later’, Proceedings of the Human Factors and Ergonomics Society 50th Annual Meeting, 50(9), pp. 904–908. doi: 10.1177/154193120605000909.

Hart, S. G. and Staveland, L. E. (1988) ‘Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research’, Human Mental Workload, 1(3), pp. 139–183. doi: 10.1016/S0166- 4115(08)62386-9.

Hearst, M. A. (2009) Search User Interfaces. First. New York, New York, USA: Cambridge University Press. Available at: http://searchuserinterfaces.com/.

Hearst, M. A. (2014) ‘What’s Missing from Collaborative Search?’, Computer, March, pp. 58–61. doi: 10.1109/MC.2014.77.

Heath, C. and Luff, P. (1991) ‘Collaborative activity and technological design: Task coordination in London Underground control rooms’, in Proceedings of the Second European Conference on Computer-Supported Cooperative Work (ECSCW ’91). Amsterdam, The Netherlands: Springer Netherlands, pp. 65–80. doi: 10.1007/978-94-011-3506-1_5.

Hertzum, M. (2008) ‘Collaborative information seeking: The combined activity of information seeking and collaborative grounding’, Information Processing & Management, 44(2), pp. 957–962. doi: 10.1016/j.ipm.2007.03.007.

Hertzum, M. (2010) ‘Breakdowns in collaborative information seeking: A study of the medication process’, Information Processing & Management. Elsevier Ltd, 46(6), pp. 646–655. doi: 10.1016/j.ipm.2009.10.005.

Hevner, A. and Chatterjee, S. (2010) ‘Design Science Research in Information Systems’, in Design Research in information Systems, pp. 9–22. doi: 10.1007/978-1-4419-5653-8_2.

Hevner, A. R., March, S. T., Park, J. and Ram, S. (2004) ‘Design Science in Information Systems Research’, MIS Quarterly, 28(1), pp. 75–105. doi: 10.2307/25148625.

Hjørland, B. (2009) ‘The Foundation of the Concept of Relevance’, Journal of the American Society for Information Science and Technology. Wiley Interscience, 61(2), pp. 217–237. doi: 10.1002/asi.21261.

174

References

Honeycutt, C. and Herring, S. C. (2009) ‘Beyond Microblogging: Conversation and Collaboration via Twitter’, in 2009 42nd Hawaii International Conference on System Sciences. IEEE, pp. 1–10. doi: 10.1109/HICSS.2009.89.

Hughes, D. J., Rowe, M., Batey, M. and Lee, A. (2012) ‘A tale of two sites: Twitter vs. Facebook and the personality predictors of social media usage’, Computers in Human Behavior. Elsevier Ltd, 28(2), pp. 561–569. doi: 10.1016/j.chb.2011.11.001.

Hyldegård, J. (2006) ‘Collaborative information behaviour: Exploring Kuhlthau’s Information Search Process model in a group-based educational setting’, Information Processing & Management, 42(1), pp. 276–298. doi: 10.1016/j.ipm.2004.06.013.

Hyldegård, J. (2009) ‘Beyond the search process – Exploring group members’ information behavior in context’, Information Processing & Management. Elsevier Ltd, 45(1), pp. 142–158. doi: 10.1016/j.ipm.2008.05.007.

Imazu, M., Nakayama, S. and Joho, H. (2011) ‘Effect of Explicit Roles on Collaborative Search in Travel Planning Task’, in Salem, M. V. M., Shaalan, K., Oroumchian, F., Shakery, A., and Khelalfa, H. (eds) Information Retrieval Technology. Springer Berlin Heidelberg (Information Retrieval Technology), pp. 205–214. doi: 10.1007/978-3-642-25631-8_19.

Ingwersen, P. (1992) Information Retrieval Interaction. Taylor Graham Publishing. Available at: www.db.dk/pi/iri/.

Ingwersen, P. and Järvelin, K. (2005) The Turn: Integration of information seeking and retrieval in context. Springer (The Kluwer International Series on Information Retrieval). Available at: http://www.springer.com/computer/book/978-1-4020-3850-1 (Accessed: 18 October 2013).

Ishii, H., Kobayashi, M. and Arita, K. (1994) ‘Iterative design of seamless collaboration media’, Communications of the ACM, 37(8), pp. 83–97. doi: 10.1145/179606.179687.

Johansen, R. (1988) Groupware: Computer Support For Business Teams. New York, New York, USA: The Free Press.

Johansen, R., Sibbet, D., Benson, S., Martin, A., Mittman, R. and Saffo, P. (1991) Leading Business Teams: How Teams Can Use Technology and Group Process Tools to Enhance Performance. Edited by R. Johansen. Addison-Wesley.

Joho, H., Hannah, D. and Jose, J. M. (2008) ‘Comparing collaborative and independent search in a recall-oriented task’, in Proceedings of the Second international symposium on Information interaction in context (IIiX ’08). London, UK: ACM Press, p. 89. doi: 10.1145/1414694.1414715.

Joho, H., Hannah, D. and Jose, J. M. (2009) ‘Revisiting IR techniques for collaborative search strategies’, Advances in Information Retrieval. Springer, pp. 66–77. Available at: http://www.springerlink.com/index/366763754ur25155.pdf (Accessed: 27 May 2011).

Kang, R., Fu, W.-T. and Kannampallil, T. G. (2010) ‘Exploiting knowledge-in-the-head and Knowledge- in-the-social-web: Effects of Domain Expertise on Exploratory Search in Individual and Social Search Environments’, Proceedings of the 28th international conference on Human factors in computing systems (CHI 2010). ACM, pp. 393–402. Available at: http://portal.acm.org/citation.cfm?id=1753386 (Accessed: 4 November 2010).

Karamuftuoglu, M. (1998) ‘Collaborative information retrieval: Toward a social informatics view of IR interaction’, Journal of the American Society for Information Science, 49(12), pp. 1070–1080. doi: 10.1002/(SICI)1097-4571(1998)49:12<1070::AID-ASI3>3.0.CO;2-S.

175

References

Karunakaran, A. and Reddy, M. C. (2012) ‘Barriers to collaborative information seeking in organizations’, Proceedings of the American Society for Information Science and Technology, 49(1), pp. 1–10. doi: 10.1002/meet.14504901169.

Karunakaran, A., Reddy, M. C. and Spence, P. R. (2013) ‘Toward a model of collaborative information behavior in organizations’, Journal of the American Society for Information Science and Technology, 64(12), pp. 2437–2451. doi: 10.1002/asi.22943.

Kelly, D. (2009) ‘Methods for Evaluating Interactive Information Retrieval Systems with Users’, Foundations and Trends in Information Retrieval, 3(1–2), pp. 1–224. doi: 10.1561/1500000012.

Kelly, R. and Payne, S. J. (2013) ‘Division of Labour in Collaborative Information Seeking: Current Approaches and Future Directions’, in CSCW ’13 Workshop on Collaborative Information Seeking: Consolidating the Past, Creating the Future. San Antonio, Texas. Available at: http://collab.infoseeking.org/resources/papers/cis2013/KellyPaper.pdf (Accessed: 25 September 2013).

Kelly, R. and Payne, S. J. (2014) ‘Collaborative Web Search in Context: A Study of Tool Use in Everyday Tasks’, in Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing - CSCW ’14. New York, New York, USA: ACM Press, pp. 807–819. doi: 10.1145/2531602.2531617.

Kock, N. (2008) Encyclopedia of E-Collaboration. Hershey, Pennsylvania: IGI Global.

Kolfschoten, G. L., Herrmann, T. and Lukosch, S. (2013) ‘Differentiated Awareness-Support in Computer Supported Collaborative Work’, Computer Supported Cooperative Work (CSCW), 22(2–3), pp. 107–112. doi: 10.1007/s10606-012-9185-z.

Konstan, J. A., Miller, B. N., Maltz, D., Herlocker, J. L., Gordon, L. R. and Riedl, J. (1997) ‘GroupLens: applying collaborative filtering to Usenet news’, Communications of the ACM, 40(3), pp. 77–87. doi: 10.1145/245108.245126.

Kuhlthau, C. C. (1991) ‘Inside the search process: Information seeking from the user’s perspective’, Journal of the American Society for Information Science, 42(5), pp. 361–371. doi: 10.1002/(SICI)1097- 4571(199106)42:5<361::AID-ASI6>3.0.CO;2-#.

Kuhlthau, C. C., Heinström, J. and Todd, R. J. (2008) ‘The “information search process” revisited: is the model still useful?’, Information Research, 13(4). Available at: http://www.informationr.net/ir/13- 4/paper355.

Kuhlthau, C. C. and Tama, S. L. (2001) ‘Information search process of lawyers: a call for ’just for me’ information services’, Journal of Documentation, 57(1), pp. 25–43. doi: 10.1108/EUM0000000007076.

Kules, B., Capra, R., Banta, M. and Sierra, T. (2009) ‘What do exploratory searchers look at in a faceted search interface?’, Proceedings of the 2009 joint international conference on Digital libraries - JCDL ’09. New York, New York, USA: ACM Press, p. 313. doi: 10.1145/1555400.1555452.

Lee, K. J. (2006) ‘What Goes Around Comes Around: An Analysis of Del.Icio.Us As Social Space’, Proceedings of the 2006 20th Anniversary Conference on Computer Supported Cooperative Work, pp. 191–194. doi: 10.1145/1180875.1180905.

Legris, P., Ingham, J. and Collerette, P. (2003) ‘Why do people use information technology? A critical review of the technology acceptance model’, Information & Management, 40(3), pp. 191–204. doi: 10.1016/S0378-7206(01)00143-4.

176

References

Li, H., Councill, I. G., Lee, W.-C. and Giles, C. L. (2006) ‘CiteSeerx: an architecture and web service design for an academic document search engine’, in Proceedings of the 15th international conference on World Wide Web (WWW 2006). ACM, pp. 883–884. Available at: http://portal.acm.org/citation.cfm?id=1135926 (Accessed: 2 July 2011).

Liechti, O. and Sumi, Y. (2002) ‘Editorial: Awareness and the WWW’, International Journal of Human- Computer Studies, 56(1), pp. 1–5. doi: 10.1006/ijhc.2001.0512.

London, S. (1995) Collaboration and Community. Charlottesville, Virginia, USA. Available at: http://www.upperskeena.ca/storytellers/CCL research/ccl/themes/micro-macro/collaboration.pdf (Accessed: 8 May 2014).

Malone, T. W. and Crowston, K. (1990) ‘What is coordination theory and how can it help design cooperative work systems?’, in Proceedings of the 1990 ACM conference on Computer-supported cooperative work - CSCW ’90. New York, New York, USA: ACM Press, pp. 357–370. doi: 10.1145/99332.99367.

Manning, C. D., Raghavan, P. and Schütze, H. (2008) Introduction to Information Retrieval. Cambridge University Press. Available at: http://nlp.stanford.edu/IR-book/html/htmledition/irbook.html.

Mao, Y., Shen, H. and Sun, C. (2010) ‘Supporting Exploratory Information Seeking by Epistemology- based Social Search’, Proceedings of the 15th international conference on Intelligent user interfaces (IUI ’10), pp. 353–356. Available at: http://dl.acm.org/citation.cfm?id=1720029 (Accessed: 28 May 2012).

Marchionini, G. (1995) Information Seeking in Electronic Environments. Cambridge: Cambridge University Press (Cambridge Series on Human-Computer Interaction). doi: 10.1017/CBO9780511626388.

Marchionini, G. (2006) ‘Exploratory Search: From Finding to Understanding’, Communications of the ACM, 1 April, p. 41. doi: 10.1145/1121949.1121979.

Marchionini, G. (2008) ‘Human–information interaction research and development’, Library & Information Science Research, 30(3), pp. 165–174. doi: 10.1016/j.lisr.2008.07.001.

Marchionini, G. and White, R. W. (2007) ‘Find What You Need, Understand What You Find’, International Journal of Human-Computer Interaction, 23(3), pp. 205–237. doi: 10.1080/10447310701702352.

Markopoulos, P., De Ruyter, B. and Mackay, W. (2009) ‘Awareness Systems: Advances in Theory, Methodology and Design’, in Markopoulos, P., De Ruyter, B., and Mackay, W. (eds) Human-Computer Interaction Series. London: Springer (Human-Computer Interaction Series), pp. v–xiii. doi: 10.1007/978-1-84882-477-5.

McNally, K., O’Mahony, M. P., Smyth, B., Coyle, M. and Briggs, P. (2011) ‘Social and Collaborative Web Search: An Evaluation Study’, in Proceedings of the 15th international conference on Intelligent user interfaces - IUI ’11. New York, New York, USA: ACM Press, p. 387. doi: 10.1145/1943403.1943472.

Morris, M. R. (2007) Collaborating alone and together: Investigating persistent and multi-user web search activities. Available at: http://research.microsoft.com/pubs/70402/tr-2007-11.pdf (Accessed: 30 June 2011).

Morris, M. R. (2008) ‘A survey of collaborative web search practices’, Proceeding of the 26th annual SIGCHI conference on Human factors in computing systems (CHI 2008). New York, New York, USA: ACM Press, pp. 1657–1660. doi: 10.1145/1357054.1357312.

177

References

Morris, M. R. (2013) ‘Collaborative Search Revisited’, in Proceedings of the 2013 conference on Computer supported cooperative work - CSCW ’13. San Antonio, Texas: ACM Press, pp. 1181–1191. doi: 10.1145/2441776.2441910.

Morris, M. R., Fisher, D. and Wigdor, D. (2010) ‘Search on surfaces: Exploring the potential of interactive tabletops for collaborative search tasks’, Information Processing & Management. Elsevier Ltd, 46(6), pp. 703–717. doi: 10.1016/j.ipm.2009.10.004.

Morris, M. R. and Horvitz, E. (2007) ‘SearchTogether: An Interface for Collaborative Web Search’, in Proceedings of the 2007 ACM Symposium on User interface software and technology (UIST 2007). Newport, Rhode Island, USA: ACM Press, pp. 3–12. doi: 10.1145/1294211.1294215.

Morris, M. R., Lombardo, J. and Wigdor, D. (2010) ‘WeSearch: Supporting Collaborative Search and Sensemaking on a Tabletop Display’, in Proceedings of the 2010 ACM conference on Computer supported cooperative work - CSCW ’10. New York, New York, USA: ACM Press, p. 401. doi: 10.1145/1718918.1718987.

Morris, M. R. and Teevan, J. (2008) ‘Understanding groups’ properties as a means of improving collaborative search systems’, in Proceedings of the JCDL’08 Workshop on Collaborative Information Retrieval. Available at: http://arxiv.org/pdf/0908.0586.

Morris, M. R. and Teevan, J. (2009) Collaborative Web Search: Who, What, Where, When, and Why, Synthesis Lectures on Information Concepts, Retrieval, and Services. Morgan & Claypool (Synthesis Lectures on Information Concepts, Retrieval,and Services). doi: 10.2200/S00230ED1V01Y200912ICR014.

Morville, P. and Callender, J. (2010) Search Patterns. Sebastopol, California, USA: O’Reilly Media, Inc.

Newman, K., Knight, S., Hansen, P. and Elbeshausen, S. (2015) ‘Situating CIS: The Importance of Context in Collaborative Information Seeking’, in Hansen, P., Shah, C., and Klas, C.-P. (eds) Collaborative Information Seeking: Best Practices, New Domains and New Thoughts. Springer International Publishing, pp. 37–54. doi: 10.1007/978-3-319-18988-8_3.

Nichols, D. M. and Twidale, M. B. (2011) ‘Recommendation, Collaboration and Social Search’, in Ruthven, I. and Kelly, D. (eds) Interactive Information Seeking, Behaviour and Retrieval. Facet Publishing, pp. 205–220. Available at: http://researchcommons.waikato.ac.nz/handle/10289/5659 (Accessed: 26 April 2012).

Nielsen, J. (1993) ‘Iterative User-Interface Design’, Computer, 26(11), pp. 32–41. doi: 10.1109/2.241424.

Nielsen, J. (2012) Usability 101: Introduction to Usability, Nielsen Norman Group. Available at: https://www.nngroup.com/articles/usability-101-introduction-to-usability/ (Accessed: 12 October 2014).

Norman, D. (2013) The Design of Everyday Things: Revised and Expanded.

O’Brien, H. L., Toms, E. G., Kelloway, E. K. and Kelley, E. (2009) ‘Developing and evaluating a reliable measure of user engagement’, Proceedings of the American Society for Information Science and Technology, 45(1), pp. 1–10. doi: 10.1002/meet.2008.1450450258.

Olson, G. M. and Olson, J. S. (2000) ‘Distance Matters’, Human-Computer Interaction. Taylor & Francis, 15(2), pp. 139–178. doi: 10.1207/S15327051HCI1523_4.

178

References

Olson, J. S. and Olson, G. M. (2013) ‘Working Together Apart: Collaboration over the Internet’, Synthesis Lectures on Human-Centered Informatics, 6(5), pp. 1–151. doi: 10.2200/S00542ED1V01Y201310HCI020.

Olson, J. S., Olson, G. M., Storrøsten, M. and Carter, M. (1992) ‘How a group-editor changes the character of a design meeting as well as its outcome’, in Proceedings of the 1992 ACM conference on Computer-supported cooperative work - CSCW ’92. New York, New York, USA: ACM Press, pp. 91–98. doi: 10.1145/143457.143466.

Orlikowski, W. J. and Iacono, C. S. (2001) ‘Research Commentary: Desperately Seeking the “IT” in IT Research: A Call to Theorizing the IT Artifact’, Information Systems Research, 12(2), pp. 121–134. doi: 10.1287/isre.12.2.121.9700.

Pallant, J. (2013) SPSS Survival Manual: A Step by Step Guide to Data Analysis Using IBM SPSS.

Papagelis, A. and Zaroliagis, C. (2007) ‘Searchius: A Collaborative Search Engine’, in Eighth Mexican International Conference on Current Trends in Computer Science (ENC 2007). IEEE, pp. 88–98. doi: 10.1109/ENC.2007.19.

Paul, R. (2009) Google Sidewiki interesting, but real utility unclear. Available at: http://arstechnica.com/business/2009/09/google-introduces-sidewiki-invites-users-to-annotate- web/ (Accessed: 17 March 2012).

Paul, S. A. and Morris, M. R. (2009) ‘CoSense: Enhancing Sensemaking for Collaborative Web Search Sharoda’, in Proceedings of the 27th international conference on Human factors in computing systems - CHI 09. Boston, Massachusetts: ACM Press, pp. 1771–1780. doi: 10.1145/1518701.1518974.

Paul, S. A. and Morris, M. R. (2011) ‘Sensemaking in Collaborative Web Search’, Human-Computer Interaction, 26(1), pp. 72–122. doi: 10.1080/07370024.2011.559410.

Paul, S. A. and Reddy, M. C. (2010) ‘Understanding Together: Sensemaking in Collaborative Information Seeking’, in Proceedings of the 2010 ACM conference on Computer supported cooperative work - CSCW ’10. Savannah, Georgia: ACM Press, p. 321. doi: 10.1145/1718918.1718976.

Perez, J. R., Leelanupab, T. and Jose, J. M. (2012) ‘CoFox : A Synchronous Collaborative Browser’, in Hou, Y., Nie, J.-Y., Sun, L., Wang, B., and Zhang, P. (eds) Information Retrieval Technology, pp. 262– 274. doi: 10.1007/978-3-642-35341-3_22.

Perez, J. R., Whiting, S. and Jose, J. M. (2011) ‘CoFox: A visual collaborative browser’, in Proceedings of the 3rd international workshop on Collaborative information retrieval - CIR ’11. New York, New York, USA: ACM Press, p. 29. doi: 10.1145/2064075.2064084.

Pickens, J., Golovchinsky, G., Shah, C., Qvarfordt, P. and Back, M. (2008) ‘Algorithmic mediation for collaborative exploratory search’, in Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval - SIGIR ’08. New York, New York, USA: ACM Press, p. 315. doi: 10.1145/1390334.1390389.

Pigneur, Y. (2013) Design science research in information systems. Available at: http://www2.hec.unil.ch/wpmu/ypigneur/wp- content/uploads/sites/15/2012/12/syllabusDRIS2013.pdf (Accessed: 15 June 2016).

Pinelle, D., Gutwin, C. and Greenberg, S. (2003) ‘Task Analysis for Groupware Usability Evaluation: Modeling Shared-Workspace Tasks With the Mechanics of Collaboration’, ACM Transactions on Computer-Human Interaction (TOCHI’03), 10(4), pp. 281–311. doi: http://doi.acm.org/10.1145/966930.966932.

179

References

Pirolli, P. (2009) ‘An Elementary Social Information Foraging Model’, in Proceedings of the 27th international conference on Human factors in computing systems - CHI 09. New York, New York, USA: ACM Press, p. 605. doi: 10.1145/1518701.1518795.

Pirolli, P. and Card, S. K. (1999) ‘Information Foraging’, Psychological Review, 106(4), pp. 643–675. doi: 10.1037/0033-295X.106.4.643.

Pirolli, P. and Fu, W.-T. (2003) ‘SNIF-ACT: A Model of Information Foraging on the World Wide Web’, User Modeling 2003, pp. 45–54. doi: 10.1007/3-540-44963-9_8.

Poltrock, S., Grudin, J., Dumais, S., Fidel, R., Bruce, H. and Pejtersen, A. M. (2003) ‘Information Seeking and Sharing in Design Teams’, in Proceedings of the 2003 international ACM SIGGROUP conference on Supporting group work - GROUP ’03. New York, New York, USA: ACM Press, p. 239. doi: 10.1145/958160.958198.

Prekop, P. (2002) ‘A qualitative study of collaborative information seeking’, Journal of Documentation. Emerald Group Publishing Limited, 58(5), pp. 533–547. doi: 10.1108/00220410210441000.

Purcell, K., Brenner, J. and Rainie, L. (2012) Search Engine Use 2012, Pew Internet & American Life Project. Available at: http://pewinternet.org/Reports/2012/Search-Engine-Use-2012.aspx.

Purchase, H. C. (2012) Experimental Human-Computer Interaction: A Practical Guide with Visual Examples. Cambridge University Press. Available at: http://www.cambridge.org/gb/knowledge/isbn/item6813346/.

Quan-Haase, A., Cothrel, J. and Wellman, B. (2005) ‘Instant Messaging for Collaboration: A Case Study of a High-Tech Firm’, Journal of Computer-Mediated Communication, 10(4), pp. 00–00. doi: 10.1111/j.1083-6101.2005.tb00276.x.

Rädle, R., Jetter, H. and Reiterer, H. (2013) ‘TwisterSearch A Distributed User Interface for Collaborative Web Search’, in Lozano, M. D., Gallud, J. A., Tesoriero, R., and Penichet, V. M. R. (eds) Distributed User Interfaces: Usability and Collaboration. London: Springer London (Human–Computer Interaction Series), pp. 53–67. doi: 10.1007/978-1-4471-5499-0.

Reddy, M. C. and Dourish, P. (2002) ‘A Finger on the Pulse: Temporal Rhythms and Information Seeking in Medical Work’, in Proceedings of the 2002 ACM conference on Computer supported cooperative work - CSCW ’02. New York, New York, USA: ACM Press, p. 344. doi: 10.1145/587078.587126.

Reddy, M. C. and Jansen, B. J. (2008) ‘A model for understanding collaborative information behavior in context: A study of two healthcare teams’, Information Processing & Management. Elsevier, 44(1), pp. 256–273. Available at: http://portal.acm.org/citation.cfm?id=1314720.1315087 (Accessed: 2 October 2010).

Reddy, M. C., Jansen, B. J. and Krishnappa, R. (2009) ‘The role of communication in collaborative information searching’, Proceedings of the American Society for Information Science and Technology, 45(1), pp. 1–10. doi: 10.1002/meet.2008.1450450259.

Reddy, M. C., Jansen, B. J. and Spence, P. R. (2010) ‘Collaborative Information Behavior: Exploring Collaboration and Coordination during Information Seeking and Retrieval Activities’, in Foster, J. (ed.) Collaborative Information Behavior: User Engagement and Communication Sharing. IGI Global, pp. 73– 88. doi: 10.4018/978-1-61520-797-8.ch005.

180

References

Richter, F. (2013) Social Networking Is the No. 1 Online Activity in the U.S. Available at: http://www.statista.com/topics/1164/social-networks/chart/1238/digital-media-use-in-the-us/.

Rittenbruch, M. and McEwan, G. (2009) ‘An Historical Reflection of Awareness in Collaboration’, in Awareness Systems, pp. 3–48. doi: 10.1007/978-1-84882-477-5_1.

Rodden, T. (1991) ‘A survey of CSCW systems’, Interacting with Computers, 3(3), pp. 319–353. doi: 10.1016/0953-5438(91)90020-3.

Ruotsalo, T., Peltonen, J., Eugster, M. J. A., Glowacka, D., Reijonen, A., Jacucci, G., Myllymäki, P. and Kaski, S. (2014) ‘IntentRadar: Search User Interface that Anticipates User’s Search Intents’, in Proceedings of the extended abstracts of the 32nd annual ACM conference on Human factors in computing systems - CHI EA ’14. New York, New York, USA: ACM Press, pp. 455–458. doi: 10.1145/2559206.2574807.

Russell-Rose, T. and Tate, T. (2012) Designing the Search Experience: The Information Architecture of Discovery. Morgan Kaufmann.

Ruthven, I. (2009) ‘Interactive Information retrieval’, Annual Review of Information Science and Technology, 42(1), pp. 43–91. doi: 10.1002/aris.2008.1440420109.

Saldaña, J. (2009) The Coding Manual for Qualitative Researchers. Los Angeles, CA, USA: Sage Publications.

Saunders, M., Lewis, P. and Thornhill, A. (2009) Research Methods for Business Students. Fifth Edit, Research methods for business students. Fifth Edit. Pearson Education Limited.

Schmidt, K. (2002) ‘The Problem with “Awareness”: Introductory Remarks on “Awareness in CSCW”’, Computer Supported Cooperative Work (CSCW). Citeseer, 11(3), pp. 285–298. Available at: http://scholar.google.com/scholar?hl=en&btnG=Search&q=intitle:The+Problem+with+?Awareness? #0 (Accessed: 2 February 2011).

Schmidt, K. (2011) Cooperative Work and Coordinative Practices: Contributions to the Conceptual Foundations of Computer-Supported CooperativeWork (CSCW). London: Springer London (Computer Supported Cooperative Work). doi: 10.1007/978-1-84800-068-1.

Schmidt, K. and Bannon, L. (1992) ‘Taking CSCW seriously’, Computer Supported Cooperative Work (CSCW), 1(1–2), pp. 7–40. doi: 10.1007/BF00752449.

Schmidt, K. and Bannon, L. (2013) ‘Constructing CSCW: The First Quarter Century’, Computer Supported Cooperative Work (CSCW), 22(4–6), pp. 345–372. doi: 10.1007/s10606-013-9193-7.

Shah, C. (2008) ‘Toward collaborative information seeking (CIS)’, in Proceedings of 1st International Workshop on Collaborative Information Seeking. Pittsburgh, Pennsylvania, USA: ACM. Available at: http://arxiv.org/abs/0908.0709 (Accessed: 9 June 2011).

Shah, C. (2010a) A Framework for Supporting User-Centric Collaborative Information Seeking. University of North Carolina at Chapel Hill. Available at: http://www.unc.edu/~chirags/papers/Shah_Dissertation.pdf.

Shah, C. (2010b) ‘Advances in Librarianship Collaborative Information Seeking : A Literature Review’, in Woodsworth, A. (ed.) Exploring the Digital Frontier (Advances in Librarianship). Emerald Group Publishing Limited, pp. 3–33. doi: 10.1108/S0065-2830(2010)0000032004.

181

References

Shah, C. (2012) Collaborative Information Seeking: The Art and Science of Making the Whole Greater than the Sum of All. Edited by C. Zhai, M. de Rijke, N. J. Belkin, and C. Clarke. Berlin, Heidelberg: Springer Berlin Heidelberg (The Information Retrieval Series). doi: 10.1007/978-3-642-28813-5.

Shah, C. (2013a) ‘Collaborative Information Seeking (CIS): Challenges and Opportunities’, in Proceedings of the Third Workshop on Collaborative Information Seeking at CSCW 2013 Conference. Available at: http://collab.infoseeking.org/resources/papers/cis2013/ShahRutgers.pdf.

Shah, C. (2013b) ‘Effects of awareness on coordination in collaborative information seeking’, Journal of the American Society for Information Science and Technology. John Wiley & Sons, Inc., 64(6), pp. 1122–1143. doi: 10.1002/asi.22819.

Shah, C. (2014a) ‘Collaborative Information Seeking’, Journal of the Association for Information Science and Technology, 65(2), pp. 215–236. doi: 10.1002/asi.22977.

Shah, C. (2014b) ‘Evaluating collaborative information seeking - synthesis, suggestions, and structure’, Journal of Information Science, (April). doi: 10.1177/0165551514530651.

Shah, C. (2015) ‘Collaborative Information Seeking: From “What?” and “Why?” to “How?” and “So What?”’, in Hansen, P., Shah, C., and Klas, C.-P. (eds) Collaborative Information Seeking: Best Practices, New Domains and New Thoughts. Springer International Publishing, pp. 3–16. doi: 10.1007/978-3- 319-18988-8_1.

Shah, C., Capra, R. and Hansen, P. (2014) ‘Collaborative Information Seeking [Guest editors’ introduction]’, Computer, March, pp. 22–25. doi: 10.1109/MC.2014.54.

Shah, C. and González-Ibáñez, R. (2010) ‘Exploring information seeking processes in collaborative search tasks’, Proceedings of the American Society for Information Science and Technology, 47(1), pp. 1–7. doi: 10.1002/meet.14504701211.

Shah, C. and González-Ibáñez, R. (2011) ‘Evaluating the synergic effect of collaboration in information seeking’, Proceedings of the 34th international ACM SIGIR conference on Research and development in Information - SIGIR ’11. New York, New York, USA: ACM Press, p. 913. doi: 10.1145/2009916.2010038.

Shah, C. and Marchionini, G. (2010) ‘Awareness in Collaborative Information Seeking’, Journal of the American Society for Information Science and Technology, 61(10), pp. 1970–1986. doi: 10.1002/asi.21379.

Shah, C., Pickens, J. and Golovchinsky, G. (2010) ‘Role-based results redistribution for collaborative information retrieval’, Information Processing & Management. Elsevier Ltd, 46(6), pp. 773–781. doi: 10.1016/j.ipm.2009.10.002.

Shneiderman, B. and Plaisant, C. (2005) Designing the User Interface: Strategies for Effective Human- Computer Interaction. Fourth Edi. Pearson Education.

Simone, C. and Schmidt, K. (2000) ‘Mind the gap! Towards a unified view of CSCW’, Designing Cooperative Systems: The Use of Theories and Models. Proceedings of the 4th International Conference on the Design of Cooperative Systems (COOP 2000). Edited by R. Dieng-Kuntz, A. Giboin, L. Karsenty, and G. De Michelis, pp. 23–26. Available at: http://scholar.google.com/scholar?hl=en&btnG=Search&q=intitle:Mind+the+gap!+Towards+a+unifi ed+view+of+CSCW#1 (Accessed: 21 February 2014).

182

References

Sinclair, J. and Cardew-Hall, M. (2007) ‘The folksonomy tag cloud: when is it useful?’, Journal of Information Science, 34(1), pp. 15–29. doi: 10.1177/0165551506078083.

Smeaton, A. F., Lee, H., Foley, C. and McGivney, S. (2006) ‘Collaborative video searching on a tabletop’, Multimedia Systems, 12(4–5), pp. 375–391. doi: 10.1007/s00530-006-0064-7.

Smeaton, A. F., Lee, H., Foley, C., McGivney, S. and Gurrin, C. (2006) ‘Físchlár-DiamondTouch: collaborative video searching on a table’, in Chang, E. Y., Hanjalic, A., and Sebe, N. (eds) SPIE 6073, Multimedia Content Analysis, Management, and Retrieval. San Jose, CA, pp. 607308-607308–11. doi: 10.1117/12.643077.

Smyth, B. (2007) ‘A Community-Based Approach to Personalizing Web Search’, Computer, 40(8), pp. 42–50. doi: 10.1109/MC.2007.259.

Smyth, B., Balfe, E., Boydell, O., Bradley, K., Briggs, P., Coyle, M. and Freyne, J. (2005) ‘A Live-User Evaluation of Collaborative Web Search’, in Nineteenth International Joint Conference on Artificial Intelligence (IJCAI 05), pp. 1419–1424.

Smyth, B., Balfe, E., Freyne, J., Briggs, P., Coyle, M. and Boydell, O. (2005) ‘Exploiting Query Repetition and Regularity in an Adaptive Community-Based Web Search Engine’, User Modeling and User- Adapted Interaction, 14(5), pp. 383–423. doi: 10.1007/s11257-004-5270-4.

Sonnenwald, D. H. (1996) ‘Communication roles that support collaboration during the design process’, Design Studies, 17(3), pp. 277–301. doi: 10.1016/0142-694X(96)00002-6.

Sonnenwald, D. H. and Lievrouw, L. A. (1997) ‘Collaboration during the design process: a case study of communication, information behavior, and project performance’, in Vakkari, P., Savolainen, R., and Dervin, B. (eds) Information Seeking in Context, pp. 179–204. Available at: http://www.researchgate.net/publication/234813706_Collaboration_during_the_design_process_a _case_study_of_communication_information_behavior_and_project_performance/file/32bfe512fdc 355a7c0.pdf (Accessed: 2 January 2014).

Sonnenwald, D. H., Maglaughlin, K. L. and Mary C. Whitton (2004) ‘Designing to support situation awareness across distances: an example from a scientific collaboratory’, Information Processing & Management, 40(6), pp. 989–1011. doi: 10.1016/j.ipm.2003.10.002.

Sonnenwald, D. H., Marchionini, G., Wildemuth, B. M., Dempsey, B. J., Viles, C. L., Tibbo, H. R. and Smith, J. B. (1999) ‘Collaboration Services in a Participatory Digital Library: An Emerging Design’, in Proceedings of Third International Conference on Conception of Library and Information Science (CoLIS). Dubrovnik, Croatia. Available at: http://eprints.rclis.org/handle/10760/7968 (Accessed: 24 February 2014).

Sonnenwald, D. H. and Pierce, L. G. (2000) ‘Information behavior in dynamic group work contexts: interwoven situational awareness, dense social networks and contested collaboration in command and control’, Information Processing & Management, 36(3), pp. 461–479. doi: 10.1016/S0306- 4573(99)00039-4.

Sperber, D. and Wilson, D. (1995) Relevance: Communication and Cognition. Second. Blackwell Publishers Ltd.

Stahl, G. (2011) ‘Theories of cognition in CSCW’, in Bødker, S., Bouvin, N. O., Wulf, V., Ciolfi, L., and Lutters, W. (eds) Proceedings of the 12th European Conference on Computer Supported Cooperative Work (ECSCW 2011). Aarhus, Denmark: Springer-Verlag, pp. 193–212. doi: 10.1007/978-0-85729-913- 0_11.

183

References

Strauss, A. (1988) ‘The Articulation of Project Work: An Organizational Process’, The Sociological Quarterly, 29(2), pp. 163–178. doi: 10.1111/j.1533-8525.1988.tb01249.x.

Strauss, A. L. (1985) ‘Work and the Division of Labor’, The Sociological Quarterly, 26(1), pp. 1–19. doi: 0.1111/j.1533-8525.1985.tb00212.x.

Talja, S. and Hansen, P. (2006) ‘Information Sharing’, in Spink, A. and Cole, C. (eds) New Directions in Human Information Behavior. Heidelberg, Germany: Springer-Verlag (Information Science and Knowledge Management), pp. 113–134. doi: 10.1007/1-4020-3670-1_7.

Taylor-Powell, E., Rossing, B. and Geran, J. (1998) Evaluating collaboratives: Reaching the Potential. Wisconsin. Available at: http://learningstore.uwex.edu/Evaluating-Collaboratives-Reaching-the- Potential-P1032.aspx.

Twidale, M. B. and Nichols, D. M. (1998) ‘Designing interfaces to support collaboration in information retrieval’, Interacting with Computers, 10(2), pp. 177–193. doi: 10.1016/S0953-5438(97)00022-2.

Twidale, M. B., Nichols, D. M. and Paice, C. D. (1997) ‘Browsing is a collaborative process’, Information Processing & Management. Elsevier, 33(6), pp. 761–783. doi: 10.1016/S0306-4573(97)00040-X.

Vaishnavi, V. K. and Kuechler, W. (2015) Design Science Research Methods and Patterns: Innovating Information and Communication Technology. Second. CRC Press.

Vakkari, P., Pennanen, M. and Serola, S. (2003) ‘Changes of search terms and tactics while writing a research proposal’, Information Processing & Management, 39(3), pp. 445–463. doi: 10.1016/S0306- 4573(02)00031-6.

Venkatesh, V. and Davis, F. D. (1996) ‘A Model of the Antecedents of Perceived Ease of Use: Development and Test’, Decision Sciences, 27(3), pp. 451–481. doi: 10.1111/j.1540- 5915.1996.tb00860.x.

Venkatesh, V. and Davis, F. D. (2000) ‘A Theoretical Extension of the Technology Acceptance Model: Four Longitudinal Field Studies’, Management Science, 46(2), pp. 186–204. doi: 10.1287/mnsc.46.2.186.11926.

Villa, R., Gildea, N. and Jose, J. M. (2008) ‘A study of awareness in multimedia search’, in Proceedings of the 8th ACM/IEEE-CS joint conference on Digital libraries - JCDL ’08. New York, New York, USA: ACM Press, p. 221. doi: 10.1145/1378889.1378926.

Villa, R. and Jose, J. M. (2012) ‘A study of awareness in multimedia search’, Information Processing & Management. Elsevier Ltd, 48(1), pp. 32–46. doi: 10.1016/j.ipm.2011.03.005.

Wasko, M. M. and Faraj, S. (2005) ‘Why Should I Share? Examining Social Capital and Knowledge Contribution in Electronic Networks of Practice’, MIS Quarterly, 29(1), pp. 35–57. Available at: http://www.jstor.org/stable/25148667.

Wasson, M. (2013) ASP.NET - Single-Page Applications: Build Modern, Responsive Web Apps with ASP.NET. Available at: https://msdn.microsoft.com/en-us/magazine/dn463786.aspx.

Welch, C. and Popper, B. (2015) Twitter reaches 300 million active users, but the stock crashes after earnings leak early, The Verge. Available at: http://www.theverge.com/2015/4/28/8509855/twitter- earnings-q1-2015-leak-selerity.

White, R. W. (2016) Interactions with Search Systems. Cambridge: Cambridge University Press. doi: 10.1017/CBO9781139525305.

184

References

White, R. W., Kules, B., Drucker, S. M. and Schraefel, M. c. (2006) ‘Introduction: Supporting Exploratory Search’, Communications of the ACM. ACM New York, NY, USA, 49(4), p. 36. doi: 10.1145/1121949.1121978.

White, R. W. and Marchionini, G. (2007) ‘Examining the effectiveness of real-time query expansion’, Information Processing & Management, 43(3), pp. 685–704. doi: 10.1016/j.ipm.2006.06.005.

White, R. W. and Roth, R. A. (2009) Exploratory Search: Beyond the Query-Response Paradigm, Synthesis Lectures on Information Concepts, Retrieval, and Services. Morgan & Claypool (Synthesis Lectures on Information Concepts, Retrieval, and Services). doi: 10.2200/S00174ED1V01Y200901ICR003.

Wilson, J. M., O’Leary, M. B., Metiu, A. and Jett, Q. R. (2008) ‘Perceived Proximity in Virtual Work: Explaining the Paradox of Far-but-Close’, Organization Studies, 29(7), pp. 979–1002. doi: 10.1177/0170840607083105.

Wilson, M. L. (2011) Search User Interface Design, Synthesis Lectures on Information Concepts, Retrieval, and Services. Morgan & Claypool (Synthesis Lectures on Information Concepts, Retrieval, and Services). doi: 10.2200/S00371ED1V01Y201111ICR020.

Wilson, M. L. and Schraefel, M. c. (2010) ‘Evaluating collaborative information-seeking interfaces with a search-oriented inspection method and re-framed information seeking theory’, Information Processing and Management. Elsevier Ltd, 46(6), pp. 718–732. doi: 10.1016/j.ipm.2009.10.001.

Wilson, T. D. (1999) ‘Models in information behaviour research’, Journal of Documentation. MCB UP Ltd, 55(3), pp. 249–270. doi: 10.1108/EUM0000000007145.

Wilson, T. D. (2000) ‘Human Information Behavior’, Informing Science: The International Journal of an Emerging Transdiscipline (InformingSciJ). The Informing Science Institute (ISI), 3(2), pp. 49–56. Available at: http://www.inform.nu/Articles/Vol3/v3n2p49-56.pdf (Accessed: 27 February 2012).

Yu, J., Benatallah, B., Casati, F. and Daniel, F. (2008) ‘Understanding Mashup Development’, IEEE Internet Computing, 12(5), pp. 44–52. doi: 10.1109/MIC.2008.114.

Yue, Z., Han, S. and He, D. (2012) ‘A Comparison of Action Transitions in Individual and Collaborative Exploratory Web Search’, in Proceedings of 8th Asia Information Retrieval Societies Conference, AIRS 2012, Tianjin, China, December 17-19, 2012., pp. 52–63. doi: 10.1007/978-3-642-35341-3_5.

Yue, Z., Han, S. and He, D. (2014) ‘Modeling search processes using hidden states in collaborative exploratory web search’, in Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing - CSCW ’14. New York, New York, USA: ACM Press, pp. 820–830. doi: 10.1145/2531602.2531658.

Yue, Z., Han, S., He, D. and Jiang, J. (2014) ‘Influences on Query Reformulation in Collaborative Web Search’, Computer, 47(3), pp. 46–53. doi: 10.1109/MC.2014.62.

Zheng, K., Mei, Q. and Hanauer, D. A. (2011) ‘Collaborative search in electronic health records.’, Journal of the American Medical Informatics Association (JAMIA), 18(3), pp. 282–91. doi: 10.1136/amiajnl-2011-000009.

Zimmerman, J., Forlizzi, J. and Evenson, S. (2007) ‘Research Through Design as a Method for Interaction Design Research in HCI design research in HCI’, Proceedings of the SIGCHI conference on Human factors in computing systems, pp. 493–502. doi: http://doi.acm.org/10.1145/1240624.1240704.

185

Appendix 1. User Study Participants’ Reflections

Sentence Feature / ID P# Category (Highlighted coded Words) Aspect 1 A1 reduces duplicate of efforts Positive Productivity

2 A1 and saves time Positive Productivity

3 A1 the timeline was not functioning until after two days Negative Error

4 A3 provided me with a wide range of knowledge. Positive Search Aggregation 5 A4 half of the team were accidently registered under Negative Error another group’s shared workspace 6 A5 which as mentioned above we all had it as problem, it Negative Error kept showing few or no results to the search made, 7 A5 and we were able to see the search made by other Negative Limitation groups which I think didn’t help a lot 8 B2 SearchAware was a key tool used in my part of the Positive Search project, which was to research our topic, and Aggregation therefore facilitated my individual productivity fantastically. 9 B3 Search Aware was a fantastic way of facilitating Positive Search individual productivity as the tool left a time line Aggregation feature for us all to see.

10 B3 However the tools did have errors quite frequently. Negative Error 11 B3 Search Aware technology could be improved by Commentary goal setting up a goal orientated feature orientated feature 12 B4 we were affected by the sheer usability of the Negative Usability technology. For example, none of us had used SearchAware before, so individually it took us quite a while to use it properly.

186

Appendix 2

13 B4 However, having time to reflect individually on our Positive Timeline work and be able to see very fast what everyone else Awareness was doing, which is not similar or non-virtual teams, definitely improved our performance as individuals.

14 B4 Moreover, since we could supervise everyone’s Positive Timeline progress regarding research, using SearchAware, we Awareness did not experience Social Loafing

15 B5 Search Aware also enabled individual productivity Positive Search because each person could use this as a means to Aggregation obtain research articles 16 B5 Moreover, this was a way of monitoring other team Positive Timeline members’ individual progress by reviewing the Awareness timeline. 17 B6 Therefore I could find various literatures quickly Positive Search through this technology without visiting each site. Aggregation 18 C1 This helps build workspace awareness of team, and Positive Timeline prevent repeat same research. Awareness 19 C1 Information also could be found through following this Positive Timeline share link. SearchAware was really useful tools for Awareness virtual team. 20 C1 SearchAware enhanced productivity of work Positive Productivity 21 C1 Only problem was that people have their own research Negative Usability method when they do research therefore they tend to follow their habit not via SearchAware. 22 C4 Search Aware gave me advantage to identify useful Positive Productivity information which influenced my performance thoroughly. 23 D1 in SearchAware, another group’s searches were Negative Usability coming up. This made the task very confusing of seeing who in our group had actually viewed what. 24 D3 The group found the volume of data provided by Negative Usability SearchAware to be of little relevance: displayed the viewed links from search results, but provided no means to see which search results were of high or low pertinence 25 D3 made greater use of SearchAware, the removal of the Positive Timeline simultaneity requirement would likely have enabled Awareness communication at times when all team members were not 26 D4 The group experienced problems using the Negative Usability collaborative search tool ‘SearchAware’. This was due

187

Appendix 2

to a lack of communication as to which group number we were 27 E1 as the tool seemed to be buggy at times, we came to a Negative Usability consensus it was easier not to use it 28 E3 which greatly benefited the process, as members were Positive Timeline not always available at the same time but could Awareness however view all searched documents 29 E3 Members of the team and myself all experience bugs Negative Error whilst using the system 30 E3 As the system made use of twitter, it made it possible Positive Social Network for use to message one another with particular areas Integration of interest 31 E3 which we would then later post on the Facebook wall Commentary Social Network to ensure all members had been made aware of the Integration findings. 32 E4 Finally, SearchAware was used to compile our Positive Usability research and although new to me I found it very user friendly. 33 F3 I liked the fact that SearchAware allows me to see Positive Timeline what other members were researching in real-time so Awareness I could look at it if I needed to; this made group research more efficient and effective. It was also an indication to the team about how much research effort everyone did contribute. 34 F3 However, I had a bad user experience using Negative Error SearchAware. It often goes into an error page when I wished to browse the search results that were returned in later pages; unfortunately, I could only see the first page 35 F3 besides, it was a bit confusing to me to find the correct Negative Usability link I should click to read the entire article instead of just the preview pages 36 F3 If SearchAware was embedded in Facebook group, say Commentary Social Network one tab named as SearchAware in the group page Integration showing members’ research history, the experience of technology integration would be more enjoyable. 37 F4 It reduced our duplication of efforts by showing each Positive Timeline other’s research articles and keywords simultaneously. Awareness

188

Appendix 2. Participant Information Sheet and Consent Form

Appendix 2.1 Participant Information Sheet

Appendix 2.2 Consent Form

189

Appendix 2

Appendix 2.1 Participant Information Sheet

MBS Ethics Approval Reference Number: MBSPGR/N379

You are being invited to take part in a research study of search awareness activity in collaborative research activities.

Before you decide it is important for you to understand why the research is being done and what it will involve. Please take time to read the following information carefully and discuss it with others if you wish. Please ask if there is anything that is not clear or if you would like more information. Take time to decide whether or not you wish to take part. Thank you for reading this.

Who will conduct the research?

Researcher: Hussain Al-Arayedh ([email protected]) Room 2.26 Crawford House, Manchester Business School M13 9QS Telephone: 0161-306-2098

Supervisory Team: Dr. Oscar De Bruijn ([email protected]) Room F38, MBS East, Manchester Business School M13 9QS University of Manchester

Prof. Andrew Howes ([email protected]) Room 135 (Y9 Computer Science Building), School of Computer Science University of Birmingham

Title of the Research

Search Awareness in Collaborative Research

What is the aim of the research?

This study aims to understand what impact will automated Search Awareness (which is sharing of keywords searched, results viewed and highlighted results between collaborators) contributes to the progress and experience of collaborative research. The study will test if search collaborators will prefer search awareness cues, as oppose to traditional digital communication, and if it will reduce the need for explicit communication regarding the progress of the research.

Why have I been chosen?

The main reason is that you, as a student, graduate or employee of a higher education institute or other research-related division in organisations and companies, will have been performing some collaborative search activities in your academic research with other colleagues or persons. As you have been using traditional communication channels to exchange the progress, we would like to test some different approaches and compare between them.

As the research aims suggests, we would like to understand the impact of this collaborative search activity using certain technologies and cues, we name it here as Search Awareness, which will be used amongst collaborators like you in a research activity.

190

Appendix 2

What would I be asked to do if I took part?

You will be asked, along with other participants as a group (two to three more), to work together on searching and selecting academic references on a given specific topic. To find these references you will be given a special web site to search through; this site uses scholarly search engines to search through a wide range of scholarly libraries.

Each group will be asked to briefly discuss their approach before starting and then each participant will be performing their search separately with the ability to exchange their progress. Each group will be given instructions on what and how they can communicate during the search session.

You will then be asked to complete a questionnaire your experience with shared search results with the other members and how useful it was.

What happens to the data collected?

The date collected will be in three forms:

(1) Statistics and contents (the logs) from the research instrument (the research website) and the reference management system.

(2) The questionnaires and/or quick interviews conducted during the study, in addition to any feedback handed regarding the study.

(3) Observational data and screenshots from the current usage.

Your computer activity during the experiment in the instructed websites will be logged. The data and usage collection will be used solely for research purposes on collaborative information search that is the search logs, shared bookmarks and comments.

How is confidentiality maintained?

All data will be coded and any relevant personal information (names, emails and any sort of identifying information) will be securely kept during the administration of the study. All personal data will be deleted once the study is completed.

If you would like to be contacted about the study results or academic dissemination of the study, then please provide alternative means of contact. These will not be associated with the identity details of the study itself.

What happens if I do not want to take part or if I change my mind?

It is up to you to decide whether to take part after reading the information sheet. If you do decide to take part, you will be given this information sheet to keep and be asked to sign a consent form. If you decide to take part, you are still free to withdraw at any time without giving a reason and without detriment to yourself.

Will I be paid for participating in the research?

Once you have completed all the tasks required from the experiment you will be reimbursed as advertised.

191

Appendix 2

What is the duration of the research?

The whole experiment should take 1and half hour and should never exceed more than 2 hours. You will be advised on the time left on regular intervals.

Where will the research be conducted?

The research will be conducted in the University of Manchester premises.

Will the outcomes of the research be published?

The research is primarily used for the doctoral research and any publications or articles solely for academic purposes.

Contact for further information

Please raise any questions to the researcher mentioned above. You can further raise any questions or concerns to the supervisor team (mentioned above) as well.

What if something goes wrong?

The research team will be glad to assist you with any help and answer any questions and address any concerns or doubts you might have on the research process and its outcome.

If for any reason you felt unsatisfied with the research team, you can pursue a formal complaint about the conduct of the research you can contact the Head of the Research Office, Christie Building, University of Manchester, Oxford Road, Manchester, M13 9PL

192

Appendix 2

Appendix 2.2 Consent Form

Search Awareness in Collaborative Research

Consent Form (Controlled Experiment)

MBS Ethics Approval Reference Number: MBSPGR/N379

If you are happy to participate, please complete and sign the consent form below:

1 I confirm that I have read the attached information sheet on the above project and  have had the opportunity to consider the information and ask questions and had these answered satisfactorily.

2 I understand that my participation in the study is voluntary and that I am free to  withdraw at any time without giving a reason and without detriment to any treatment/service

3 I understand that the usage statistics during the study, including the content and  frequency, will be collected for dissemination in academic and scholarly work. In addition, screen captures of the activity might be used.

4 I understand that researcher will provide the needed privacy and confidentiality for  any personal information and contact information provide for the purpose of this study.

I agree to take part in the above project

Name of participant Date Signature

Name of experiment organiser Date Signature

Experiment Path Details (to be filled by experiment organizer)

Group Number: ______

Start Time: ______

Location: ______

193

Appendix 3. Experimental Tasks

Three tasks were handed to the experiment participants in various order. All the tasks were in the domain of Business and Information Technology

• Task 1 • Task 2 • Task 3

194

Appendix 3

Appendix 3.1 Task 1: Social Media Marketing (BUS) Setting

You and your colleague are working for the marketing research team in a production company that specialises in ready-made meals. To advertise their new branded range of meals with local and international flavours, the company is launching a promotional campaign using social media and social network sites like Facebook, Instagram and Twitter. This campaign aims to promote the new range of meals and get potential customers aware and engaged about the new meals range, nutritional information, prices and other information. Task

Your company’s management had already hired a digital advertising agency; however, your manager would prefer an academic perspective on the use of social networks sites for marketing and promotional purposes, especially in the food production, and retail sector in general. Collect relevant publications on the use and adoption of social media and social networks sites for marketing and promotional purposes. They can include empirical studies on social network marketing, pros and cons of marketing products and services on social networks sites, methods and techniques used. Studies on the use and analysis of social networks, and the measures of success or failure in marketing campaigns are also important to your manager. You can also search for any relevant research that you deem can contribute to the report. Process

For this task, your company had bought access to a major scholarly database for this session using SearchAware. You and your partner should compile list of relevant research articles on that you deem worthy of considering for further reading. You compile the list by rating and commenting on publications. Use the publication title and abstract to decide on which articles you believe can be considered for further reading, and view the full article details if needed. A full five stars rank means the article is highly relevant to the topic, a one star rank means that might be of relevance to the topic. Use this table as guidance: 5 stars Highly relevant 4 stars Very relevant 3 stars Somewhat relevant 2 stars Slightly relevant 1 star Might be of relevance Conditions Before you can search using SearchAware, you will have 3 minutes to discuss with your partner the approach to this task. Once you start searching, please refrain from speaking during the search session and use the provided communication channels as necessary.

195

Appendix 3

Appendix 3.2 Task 2: Major Sports Event Hosting (MGT) Setting

Major global cities are competing to host global and continental sporting events like football championships and Commonwealth Games. As members of a research team in the city council, you and your partner are asked by the head of Manchester City council to find research articles and studies that discuss the various aspects of hosting major sporting events. The Manchester city council had already liaised with a public relations company to launch a campaign to gather reactions. Nevertheless, the city mayor would like to have a thorough scholarly research on the matter to complement the campaign before starting a bidding process for such events. Task

The mayor you like to prepare a list of academic articles references covering the social and economic aspects of hosting major sporting events. Issues of interest to the city council will focus on public and private engagement like promoting public attendance to sporting events, and/or funding and business partnerships like advertising and sponsorship. Moreover, studies on the supporting issues such as facilities, venues, transportation, infrastructure, accommodation and media rights are of relevance to the mayor.

You can also search for any relevant research that you deem can contribute to the report. Process

For this task, the city council had bought access to major scholarly database access for this session using SearchAware. You and your partner are asked to compile list of relevant research articles on that you deem worthy of considering for further reading.

You compile the list by rating and commenting on publications. Use the publication title and abstract to decide on which articles you believe can be considered for further reading, and view the full article details if needed. You can rate the results from the results hit and add any comments to it. A full five stars rank means the article is highly relevant to the topic, a one star rank means that might be of relevance to the topic. Use this table as guidance:

5 stars Highly relevant 4 stars Very relevant 3 stars Somewhat relevant 2 stars Slightly relevant 1 star Might be of relevance Conditions

Before you can search using SearchAware, you will have 3 minutes to discuss with your partner the approach to this task. Once you start searching, please refrain from speaking during the search session and use the provided communication channels as necessary.

196

Appendix 3

Appendix 3.3 Task 3: Mobile Retail Payment (ICT) Setting

Mobile payment is the use of smartphones that are linked with bank accounts to pay for goods or services. Due to the rapid availability of technological solutions and supporting financial institutions, this payment form is gaining popularity amongst consumers and business alike.

The major grocery chain you and your partner are working for is considering utilising this technology across its branches. The Information Technology department in your organisation is already planning a feasibility study and a test trial. Nevertheless, your manager would like to learn about the latest academic research on mobile payment solutions and gain a scholarly perspective on the issue.

Task

Your manager asked you to put together a list of academic references that includes scholarly articles were forms of mobile payments have been researched from a technological and/or business aspects.

These include introducing and managing mobile payments, business and consumer impact, the technology used, and the business side of using mobile technology for purchases. Publications that have considered interfaces, devices, and the process of payment and consumer studies of mobile payments will be of value to this list your manager asked for.

You can also search for any relevant research that you deem can contribute to the report. Process

For this task, the grocery chain had bought access to major scholarly database access for this session using SearchAware. You and your partner are asked to compile list of relevant research articles on that you deem worthy of considering for further reading.

5 stars Highly relevant 4 stars Very relevant 3 stars Somewhat relevant 2 stars Slightly relevant 1 star Might be of relevance Conditions

197

Appendix 4. Questionnaires

Three sets of questionnaires were administered in each experiment.

Set 1: Pre-Experiment

1. Appendix 4.1: Demographics and Background Questionnaire

Set 2: Post-Trial

2. Appendix 4.2.1: Engagement in SearchAware Questionnaire (Post-Trial). 3. Appendix 4.2.2: Awareness Questionnaire 4. Appendix 4.2.3: Task Load Measurement Questionnaire (Post-Trial). 5. Appendix 4.3: Usability (Post-Experiment).

198

Appendix 4

Appendix 4.1 Demographics and Background (Pre-Experiment)

A) Demographics

A.1 How old are you? ______Years.

A.2 What is your gender?  Male  Female

A.3 Are you currently a student?  Yes, Postgraduate taught student.  Yes, Postgraduate research student.  No, I have completed my studies. My degree is: (Please mention :______.)

A.4 What is your current or final academic study field? (e.g. Marketing, Accounting) (Please mention :______.)

A.5 Do you know your experiment partner?  No  Yes, but we had never worked together on any project / assignment.  Yes, we have worked together on one or more projects / assignments.

B) Experience with Search Engines

B.1 On average, how many times do you  Almost Never use general web search engines?  Several times a month (like Google Search, Bing, Yahoo?)  Several times a week  Every day, with 10 or less a day  Every day, with more than 10 times a day

B.2 On average, of the times in B.1, how  Almost Never many times do you use them for your  Less than quarter of time work and/or studies?  Almost half of the time  Over half of the time

B.3 Out of those times in B.2, how many  Almost Never times do you use web search with  Less than quarter of time people at the same place and/or time  Almost half of the time on the same or related topic?  Over half of the time

C) Scholar Search Engines, databases and reference management systems

C.1 On average, how many times do you  I don’t use scholarly search engines or databases use scholarly search engines and  Several times a month databases? (like Google Scholar,  Several times a week ScienceDirect or JSTOR)  Several times a day

199

Appendix 4

C.2 If you have worked on a research project,  Almost Never how many times have you used reference  Less than quarter of time management software? (like Endnote,  Almost half of the time Zotero or Mendeley)  Over half of the time

D) Communication

D.1 Regarding your work and/or university- Check all that apply related academic study  Short Messaging Services (SMS), iMessages.  Skype, Google Talk / Hangout, Yahoo Messenger Which of the following communication or or other instant messaging instant messaging services have you used or  WhatsApp, Line, Viber, WeChat or other mobile currently using instant messaging services.  Facebook Chat Messenger, Instagram Direct or other social network chat services  Others (please mention:______)

D.2 In a the work environment or university- Check all that apply related work  Browser (e.g. Chrome, Firefox, Internet Explorer)  Dedicated software (i.e. a chatting client, Which of the following platforms / devices program) do you currently use?  Dedicated Mobile or Tablet app

E) Social Networks

E.1 Which of the following social network sites Check all that apply have you used or currently using actively?  Facebook  Instagram  Twitter  LinkedIn  Others (please mention:______)

E.2 On average, how many social networks posts  I do not use social networks do you perform?  Once or few times a month (for example: status updates / likes /  Once or more times a week comments or replies)  One or more times a day

200

Appendix 4

Appendix 4.2.1 Engagement in SearchAware (Post-Trial)

A1) Using the SearchAware website was…

Neither Strongly Somewhat Somewhat Strongly Uninteresting interesting nor Interesting Uninteresting Uninteresting Interesting interesting uninteresting       

A2) Using the SearchAware website was…

Strongly not Not Somewhat not Neither enjoyable Somewhat Strongly Enjoyable enjoyable enjoyable enjoyable nor not enjoyable Enjoyable enjoyable       

A3) Using the SearchAware website was…

Somewhat Neither exciting Somewhat Very Dull Dull Exciting Very exciting dull nor dull exciting       

A4) Using the SearchAware website was…

Not fun at Somewhat Neither fun nor Not fun Somewhat fun Fun A lot of fun all not fun not fun       

A5) How did you feel while collaborating with colleagues using this website?

Not Somewhat Neither Somewhat Absorbed absorbed Not absorbed not absorbed nor Absorbed absorbed intensely intensely absorbed not absorbed       

A6) How did you feel while collaborating with colleagues using this website?

Attention was Attention Attention was Attention was Attention was Attention Attention was not fully was not somewhat neither focused somewhat was focused fully focused focused focused not focused nor not focused focused       

A7) How did you feel while collaborating with colleagues using this website?

Neither Did not Somewhat Did Did not concentrated Somewhat did Did concentrate did not concentrate concentrate nor not concentrate concentrate fully concentrate fully concentrated       

A8) How did you feel while collaborating with colleagues using this website?

Not deeply Somewhat Neither engaged Somewhat Deeply Not engaged Engaged engaged not engaged nor not engaged engaged engaged       

201

Appendix 4

Appendix 4.2.2 Awareness of Collaborators in SearchAware (Post-Trial)

B1) I found it difficult to tell what work my partners had done using SearchAware.