Enhancing Access to Peer Support Through Technology for Recovery from Substance Use Disorders

A Dissertation SUBMITTED TO THE FACULTY OF THE UNIVERSITY OF MINNESOTA BY

Sabirat Rubya

IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY

Svetlana Yarosh

May your memory forever be a comfort and a blessing. I am proud to have fulfilled your dream, as you are the one who inspired me to take risks to pursue my dreams.

i Acknowledgement

Before starting grad school, being “Dr. Rubya” was a dream, and then it became a journey; an amazing and eventful journey. Throughout the journey I have been lucky enough to get help, support, and inspiration from many people. I would like to express my sincere gratitude to all of them. I want to begin by thanking my adviser, Lana Yarosh. She has been a supportive mentor and an inspiring teacher from the beginning of my PhD. I consider myself lucky and always keep saying that she is the “best” and the “coolest” adviser one could ever have. Every meeting and interaction with her was an opportunity for me to learn. The best thing I learned from her is how to handle rejections and see them as opportunities to improve. I cannot thank her enough for all the guidance to become a better researcher. A special thanks for wishing me luck with the “I believe in you” sticker as I was going to teach my first class at Macalster – one of the many times of helping me believe in myself. My sincere gratitude goes to professors Loren Terveen, Jaideep Srivastava, and Brenna Greenfield for serving on my PhD committee. They have inspired me with their ownre- search trajectories and their valuable feedback and comments have made my research so much richer. I would also like to thank Professors Shashi Shekhar and Rosalie Kane for serving on the committee for my thesis proposal and preliminary exams. Thank you to my friends and colleagues in GroupLens for helping me refine my ideas and providing me feedback on my papers and talks. Thank you to Dan Kluver, Max Harper and Vikas Kumar for their help and guidance during the initial years of my Ph.D. A special thanks to Baris Unver and Jasmine Jones for all the conversations we had about research, and about life. It is great to have an opportunity of working in such a friendly and supportive environment.

ii Many thanks to the undergrads who have contributed in different stages of my research and helped me develop and enhance my mentoring skill. I would not be able to complete this work without the help of the study participants and community collaborators–my heartfelt gratitude to them. I want to thank the National Science Foundation for funding my research (grants 1464376 and 1651575), and many anonymous reviewers who criticized my work and made it better. I don’t know how to express just how much appreciation I hold in my heart for my mother and my father. I am so grateful to them for supporting me, and praying for my well-being all along, for all the hard work they had done to bring me up and make sure I have the best of education. I feel for my mother who couldn’t imagine passing a day without seeing me, and had to bear with all the pain of living far away from me. I hope I can make my parents proud in every possible way. My thanks also to my brother for helping me address my weaknesses and giving me unusual ideas for overcoming them. Thank you to my uncle for helping me understand the importance of hard work to succeed in life. I would particularly like to acknowledge the support of my husband, Tahmid, who has provided much encouragement and helped me refine my ideas and their presentation. He has helped me go through some of my most diﬀicult times. He always inspires me to bea better person. I am thankful that we have each other. Finally, my most special thanks to the Almighty for giving me the blessing, the strength, the chance and endurance to complete this journey.

iii Abstract

More than 23.5 million people in the United States (and a lot more worldwide) suffer from substance use disorders (SUDs). SUDs represent one of the most widespread and haz- ardous public health issues. Even after following a crisis medical treatment intervention (e.g., detox, rehab), up to 75% of individuals have recurrence of the symptoms within one year. In order to reduce the risk of relapse, healthcare providers recommend going through a maintenance program that includes continued abstinence and peer support. The most common approach to practicing recovery maintenance is by participating in “12-step” programs, such as Alcoholics Anonymous (AA). Although members worry about achieving anonymity, access, equality, etc., as technology interacts with the program, there are opportunities for designing technology to help the peer support process in these special groups. However, it is vital to understand the challenges faced by the members of these groups to inform design of technologies for them. This dissertation focuses on understanding the opportunities and challenges for technology to amplify peer support in recovery communities and developing new interface technologies to enhance their reach to the help, support, and connection when they need it most. In the course of developing an application that help people find the right AA meetings for them, I explore the opportunity of applying human-in-the-loop information extraction techniques in extracting unstructured data with better accuracy. This work also explores the performance trade-offs in crowdsourcing in terms of task completion time, cost, and accuracy, while applying different crowd-work techniques to recruit generic workers vs. community members from different platforms.

iv Contents

List of Tables viii

List of Figures x

1 Introduction 1

2 Background 4

2.1 Substance Use Disorders ...... 4 2.2 Treatment and Recovery Options for SUDs ...... 5 2.3 Twelve Step Fellowships ...... 7 2.4 Online 12-step Recovery Communities ...... 8

3 Related Work 9

3.1 The Values and Traditions of the Twelve Step Programs ...... 9 3.2 Technology Design for SUDs ...... 10 3.3 Online Health Peer-Support and Anonymity in HCI ...... 12

4 Formative Investigation: Analysis of Online Peer Support in Recovery from SUDs 14

4.1 Related Work ...... 15 4.2 Video-Mediated Peer-Support in an Online Recovery Community . . . . . 18

v Contents

4.3 Ethical Considerations ...... 20 4.4 Two Synergistic Investigations ...... 21 4.5 Conclusion ...... 46

5 Formative Investigation: Understanding Interpretations of Online Anonymity in 12-Step Recovery Communities 47

5.1 Related Work ...... 48 5.2 Methods ...... 52 5.3 Results ...... 60 5.4 Discussion ...... 80 5.5 Conclusion ...... 84

6 Prototype Development of “AA Meeting Finder” through HAIR 85

6.1 Related Work ...... 87 6.2 Methods ...... 91 6.3 Results ...... 98 6.4 Discussion ...... 102 6.5 Conclusion ...... 104

7 Generic and Community-Situated Crowdsourcing in HAIR 108

7.1 Related Work ...... 110 7.2 Methods ...... 114 7.3 Results ...... 125 7.4 Discussion ...... 136 7.5 Limitations ...... 144 7.6 Conclusion ...... 144

vi Contents

8 Towards Making HAIR Scalable and Sustainable 146

8.1 Updating the List at a Regular Interval ...... 146 8.2 AA Meetings Public API ...... 148 8.3 Guide and Documentation for using HAIR ...... 148

9 Conclusion and Future Work 150

9.1 Summary ...... 150 9.2 Future work ...... 153 9.3 Closing statement ...... 153

A Online Recovery Interview Protocol Guide 155

B Online Recovery Questionnaire 161

C Email/ Private message Sent to Recruit Interview Participants 174

D Screening Survey and Task Instructions on Amazon Mechanical Turk 175

D.1 Screening survey questions ...... 175 D.2 Task instruction screenshots ...... 175

Bibliography 179

vii List of Tables

4.1 Statistics of meeting attendees from survey ...... 24 4.2 Participants and features of their online recovery. Types of meetings included: AA (Alcoholics Anon.), NA (Narcotics Anon.), OA (Overeaters Anon.), CoDA (Co-Dependents Anon.), Al-Anon (Families of Alcoholics), ACA (Adult Child in Alcoholics), Dual Diagnosis (for mood disorder), Women’s Meeting, and PTSD (Post Traumatic Stress Disorder). Selection criteria included: 1. Recent active participation in online meetings, 2. Recent active participation in f2f meetings, 3. Frequent activity on ITR, 4. High number of friends on ITR, and 5. Chairing meetings on ITR ...... 28

5.1 Participants’ Demographics. Fellowships included: AA, NA, OA (Overeaters Anon.), EA (Emotions Anon.), ACA (Adult Child in Alcoholics), CoDA (Co- Dependents Anon.), Al-Anon (Families of Alcoholics), and CMA (Crystal Meth Anon.)...... 53 5.2 Frequency of themes in the questionnaire responses (number and percent of responses with a particular code, applied non-exclusively)...... 62 5.3 Name and photo sharing behavior of ITR users ...... 75

viii List of Tables

5.4 Comparison of Recovery Time, Time in ITR, Age, and Number of friends between the ITR population sharing and not sharing faces in profile images. A different N is provided for each variable since not every member shared the information of interest on his or her profile. We calculated “ITR time” from the first wall post from another member welcoming them toITR...... 78

6.1 Performance of the page classification ...... 100 6.2 Number of pages needing human input ...... 100 6.3 % (number) of meetings from the Minnesota domains ...... 100

7.1 Worker costs and total costs for generic and community-situated crowdsourcing. Worker cost is the portion of the total cost paid to only the workers for their task completion, and not to the platform...... 126

7.2 푝-values from the Kruskal-Wallis tests for the differences in task completion time and accuracy among the three tasks ...... 127 7.3 Means and standard deviations of task completion time (in seconds) of generic and community-situated crowd workers for three different tasks ...... 129 7.4 Statistical comparisons of average task completion time and accuracy between: 1) Generic crowdsourcing and community-situated crowdsourcing from MTurk, 2) Generic crowdsourcing and community-situated crowdsourcing from MTurk,

and 3) Community-situated crowdsourcing from MTurk and ITR. 푝-values are reported using Wilcoxon Rank-Sum Tests and adjustment with Holm-Bonferroni

method. (* 푝 < 0.05, ** 푝 < 0.01, *** 푝 < 0.001) ...... 129

ix List of Figures

4.1 The interface and key features of an InTheRooms online 12-step meeting . . . . 19 4.2 Statistics of meeting attendance for each type of attendees ...... 25 4.3 Usefulness of f2f and online meetings ...... 26

5.1 . Interpretations of Anonymity with respect to Time in Recovery (participants are divided in equal-frequency bins)...... 73

6.1 Different structures of meeting pages on different regional domains ...... 105 6.2 Pipeline of HAIR (the page classification phase is on left, and the meeting information retrieval phase is on right) ...... 106 6.3 Summary of results) ...... 107

7.1 Recruiting workers from InTheRooms through advertisement ...... 118 7.2 Interfaces of the three tasks (a) meeting page validation, (b) meeting validation, and (c) meeting identification ...... 121 7.3 Boxplots of accuracy achieved by three different types of crowd workers for: (a) page validation, (b) meeting validation, and (c) meeting identification . . . . 130

x List of Figures

7.4 Different types of errors in the page validation and the meeting validation tasks; the first three groups from the left respectively show the average accuracy of editing the day, time, and address information correctly in the meeting validation task. The rightmost group shows the the average percentage (and the differences between groups) of selecting the “not sure” option in the pageval- idation task ...... 132

D.1 Screenshot of the instructions for the task of page validation ...... 175 D.2 Screenshot of the instructions for the task of meeting validation ...... 176 D.3 Screenshot of the instructions for the task of meeting identification ...... 176 D.4 Screenshot of the consent form shown to the MTurk and ITR participants . . . 178

xi Chapter 1

Introduction

Substance use disorders (SUDs) are characterized by the condition of clinically significant impairment caused by extensive use of alcohol or other non-medically used psychoactive substances. According to a study by Substance Abuse and Mental Health Services Admin- istration (SAMHSA), more than 23.5 million Americans abused psychoactive substances and alcohol in 2009, which is more than nine percent of the population over the age of 12 [3]. These disorders are a medical condition which is estimated to cost the United States $223.5 billion per year [32]. SUDs can contribute to social, familial and financial diﬀicul- ties, health complications, and even death for the individual. Common treatments to support initial abstinence from substance use and to prevent subsequent recurrence of symptoms are detox and hospital/treatment center intervention with professional-led after-care programs. However, after this initial treatment (generally, 1-6 months) patients typically rely on one or more maintenance programs such as replacement therapy, cognitive behavioral therapy (CBT), SMART recovery, or a 12-step program for long-term recovery [114]. Twelve-step fellowships (such as Alcoholics Anonymous (AA) and Narcotics Anonymous (NA)) have been shown to be equally effective as other maintenance approaches, but have the benefit of being free of cost and widely available anywhere in the world. AA and NA focus on providing face-to-face meetings where people share and provide support. Recently 12-step recovery approach has been expanded to online communities. Online 12-step fellowships use a number of unique peer-support practices to reach

1 and help each other. Although there have been a number of studies examining twelve-step communities from the psychological, sociological, cultural, and clinical perspectives, very little research has been conducted to understand the role of technology in these groups. Whereas many researchers have designed and evaluated technology-based intervention techniques for recovery from substance addiction, only one study has conducted formal investigations of how technology actually fits (or does not fit) into the culture of twelve-step recovery programs. Yarosh described in that work how technology may introduce tensions including challenges in achieving anonymity, consensus, autonomy, etc. as it interacts with recovery [241]. Therefore, it is imperative to understand the needs and challenges of these groups in order to address those challenges through technology. This work seeks to understand the needs and challenges of 12-step recovery communities with an aim to pointing out opportunities for designing technology to enhance their reach for peer-support. In this proposal, I will describe my completed work including formative investigations to understand how technology interacts with anonymity and peer support in these groups, and a novel human-aided information retrieval approach to design a global self-updating peer-support meeting list. I will propose future work on evaluating the feasibility of my proposed method of combining community-situated crowdsourcing with automated information retrieval techniques in generating and maintaining the meeting list. Specifically, I investigate the following research questions:

• RQ1: How does technology interact with the in-person support for recovery?

• RQ2: Does technology interfere with the “anonymity” rooted in 12-step programs?

• RQ3: How can we effectively extract grassroots data and utilize community-situated crowdsourcing to maintain a sustainable global self-updating meeting list for AA?

2 I have conducted a series of mixed-methods studies including in-depth interviews and online surveys with longtime AA community members and newcomers to recovery to find out the opportunities to design technology that they will find useful. The formative investigations pointed out that, attending peer-support meetings is a fundamental component of successful recovery. However, due to preference of regional autonomy in AA, there is currently no global up-to-date AA meeting list, which is a fundamental infrastructural component and can address a lot of the existing challenges of the recovery communities. I have proposed HAIR (Human-Aided Information Retrieval) - that combines crowdsourcing with information retrieval techniques to generate the meeting list from grassroots meeting data on different websites. I discuss the results of an empirical study to compare the performance of generic crowd workers with community volunteers and provide implications to make the human computation component of HAIR more accurate and eﬀicient.

3 Chapter 2

Background

In this chapter, I describe terms and concepts that are important for better understanding of this work and the experience of people in recovery from SUDs.

2.1 Substance Use Disorders

Substance use disorders are characterized by needing increasing amounts of a chemical substance to achieve desired effect, consistent use of larger amounts than intended, and persis- tent unsuccessful attempts to cut down or stop use despite increasingly severe consequences to the user. These disorders are a medical condition, which are estimated to cost the United States $374 billion per year [3]. The most common types of substance use disorders in the U.S. include the use of alcohol, opioids, stimulants, and marijuana. 2004 estimates show that 67% of Americans drink alcohol, with 11.9% developing dependence to the substance; 45.8% of Americans try illicit substances during their lifetime, with rates of dependence between 10.3% and 67.8%, depending on the substance [32]. Substance use disorders, also referred to as drug addiction, can start with experimental use of a recreational drug in social situations, and then, for some people, the drug use becomes more frequent. Particularly with opioids, drug addiction begins with exposure to prescribed medications, or receiving medications from a friend or relative who has been prescribed the medication. These substances can cause harmful changes in how the brain

4 2.2. Treatment and Recovery Options for SUDs functions. Intoxication, or the immediate effects of the drug causes intense pleasure, calm, increased senses or a high and the effects vary based on the substance. However, in general, compulsive and/or uncontrollable drug seeking and use can have harmful consequences and changes in the brain, which can be long lasting. Uncontrolled substance use leads to other problems as well, such as violence, crime, depression, family deprivation, and even sui- cides [140]. Approximately 22% of deaths by suicide in 2008 in the United States involved alcohol intoxication, with a blood-alcohol content at or above the legal limit [100].

2.2 Treatment and Recovery Options for SUDs

Substance use disorders are prevalent, high-impact health conditions that are associated with significant personal and societal costs, but there is always hope for recovery. An important delineation should be made between treatment for SUDs and long-term recovery support. Though interchanged often, treatment should ultimately result in reduction or elimination of the negative symptoms of the disorder, while recovery support should help encourage and maintain that improvement, build protective factors and, in the case of SUD treatment, strengthen sobriety or abstinence over longer periods of time by preventing recurrence of symptoms. Common initial treatments to support abstinence and prevent subsequent recurrence of symptoms are detoxification (detox) and hospital/treatment center intervention with professional- led aftercare programs (e.g., rehab). There is a variety of such programs and they are not very regulated, but they typically include interaction with therapists, doctors, and social support groups. A common stay is between 1 and 6 months. The next stages are the ones where supportive technologies for recovery become increasingly relevant. The person may elect to continue with out-patient treatment, where interaction with doctors decreases and

5 2.2. Treatment and Recovery Options for SUDs groupwork is more common. The typical length of such a program is 1 to 2 months. Then, he/she may choose to live in a sober home where they are in the company of other recovering people and held accountable for rules such as attending social support groups and having random drug tests. People typically stay between 2 months and 2 years, with 6 months being the recommended length of time. Finally, they return to independent living and have to maintain their recovery on their own for the rest of their lives. Most people do this through regular attendance of peer support groups like Alcoholics Anonymous (AA) and Narcotics Anonymous (NA). This work is focused on the last stage where people are particularly vulnerable to recurrence of symptoms and where various forms of social support have been found to be particularly helpful to recovery. There are many great opportunities for making technology that supports communication with a support network and behavior tracking/change, both of which are active research areas in HCI. In this work, I particularly focus on designing technology for supporting 12- step maintenance programs because:

• Even though there are conflicting viewpoints about the effectiveness of the 12-step programs [63, 126], Various investigations have shown these interventions to be as effective than alternative approaches [174, 181],

• They are frequently the intervention recommended by the medical community [217], and

• They have the benefit of being free of cost and widely available almost anywhere in the world [193]

6 2.3. Twelve Step Fellowships

2.3 Twelve Step Fellowships

There are over 200 different types of 12-step programs (known as fellowships) docusing on specific issues of substance dependence (e.g., Crystal Meth Anonymous) and behavioral compulsion (e.g., Gamblers Anonymous). Though in most of my work the focus is on the two most established twelve-step groups (NA and AA have the most meetings worldwide and thus provided ample opportunities for participatory observation), all twelve-step fellowships advocate a similar process for recovery, which includes the following elements:

• Abstaining from the Problematic Behavior: twelve-step programs are based on the idea that recovery requires complete abstinence from the problem behavior. Substance- based programs advocate abstinence from all mind-altering substances (e.g., an NA member in recovery does not drink alcohol). Milestones in recovery (e.g., three months without using a substance) are celebrated with a public gifting of a small token such as a poker chip or a keychain. Abstaining is seen as a byproduct of addressing underlying addiction issues (e.g., resentments, loneliness) by following the three classes of suggestions below.

• Meeting Attendance: regularly (daily to weekly) attending meetings where time is devoted to reading the fellowship-approved literature, listening to a member speaker share, and sharing experiences in recovery. Each group meeting is autonomous and nonprofessional (ran by current members of the fellowship). The groups’ relationship to individuals and other groups is governed by the Twelve Traditions (more i next chapter).

• Sponsorship and Service: working with a member of the program who has had a longer time in recovery (a sponsor) to get an outside perspective one’s recovery pro-

7 2.4. Online 12-step Recovery Communities

cess and providing the same service to newer members in the program (sponsees). Additionally, service involves participating in outreach meetings at hospitals and in- stitutions, helping with the logistics of running a meeting, and participating at the regional and national levels of the fellowship.

• Stepwork: working with guidance of a sponsor and the fellowship literature to continually progress through the Twelve Steps1 of the program (after doing Step Twelve, focus again shifts to Step One). The steps include suggestions for admitting the problem, establishing a relationship with a “higher power” (e.g., God), understanding personal character defects through a written inventory, making amends to others, establishing spiritual maintenance practices, and reaching out to newcomers.

Twelve-step programs are considered to be programs of suggestions (rather than rules or requirements) and the only requirement for membership is “a desire to stop” the behavior being addressed. Thus, each individual member’s program may include some or all of these elements to varying degrees.

2.4 Online 12-step Recovery Communities

It is only recently that 12-steps recovery approach has been expanded to online communities. Online 12-step fellowships use a number of unique peer-support practices to reach and help each other. One of such communities is IntheRooms.com (more on this in Chapter 4). These online communities provide online peer-support through profile creation, making friends, wall posts, private messaging, blogs, online meetings, etc.

8 Chapter 3

Related Work

I start this chapter with a discussion on the traditions and dynamics of the 12-step communities, since the use of technology by these groups are greatly influenced by these values rooted in the programs. I then describe related research and developed technologies for this population contributed by different domains. I conclude this chapter with a description of how people seek for peer-support and maintain anonymity in online health communities.

3.1 The Values and Traditions of the Twelve Step Programs

Twelve-step fellowships (e.g., Alcoholics Anonymous and Narcotics Anonymous) are the most widely available addiction recovery resource in the US [193]. Some estimates show that as much as 3.1% of the U.S. population may be involved in Alcoholics Anonymous (AA) [193]. Despite the recent increased focus on formal evidence-based care (e.g. CBT) [181], 12-step groups such as AA and NA are the most commonly used form of long-term continuing care program among individuals with SUDs in the United States [114]. Var- ious investigations have shown twelve-step interventions to be as or more effective than alternative approaches and they are frequently the intervention recommended by the medical community [212]. Prior research indicates that up to 80% of treatment centers refer patients to 12-step groups following acute treatment [212, 180] At the core of the 12-step fellowships are the “twelve steps” and “twelve traditions.” The

9 3.2. Technology Design for SUDs twelve steps form a program of long-term sobriety and a blueprint for spiritual development of individuals. The twelve traditions are the bylaws and code of ethics of the programs, provide guidelines that govern the entire fellowship, and center on a number of “spiritual principles,” such as anonymity, equality, unity, etc. [234]. Members of twelve step fellowships rely on forming and engaging in mutual-help support communities. Privacy is of utmost importance in these communities, as there is significant social stigma associated with substance use disorders [58] and many recovering persons are reconstructing a life after years of antisocial behavior. The importance of spirituality, anonymity, and mutual help rooted in the twelve traditions, and the sensitivity and vulnerability of the members make this community a unique context to investigate the potential role of technology. Prior research in HCI has shown that there are challenges in technology research in sensitive settings like this [228], and it is critical to understand the values and the traditions rooted in the community in order to identify current opportunities for, and barriers to, the design of social computing systems in such contexts. As pointed by prior work on 12-step fellowships, in-person peer-support and anonymity are two critical components of these programs that need to be investigated in more details [241]. Therefore, we start with a formative investigation with an aim to understand what technology may be acceptable to these community members, by addressing and unfolding the concerns brought up by prior work.

3.2 Technology Design for SUDs

Although we are approaching the problem of substance use from a social computing perspective (i.e., to design technology to facilitate peer-support in recovery), we have taken into

10 3.2. Technology Design for SUDs account insights from existing work in this domain conducted by broader research community. Twelve-step fellowships have been of interest to both clinical studies of recovery and ethnographic investigations of these fellowships as social structures. Project MATCH, a clinical investigation of recovery from alcoholism, found that AA was no less effective than other interventions on any measures and significantly more effective on the measure ofthe percentage of participants maintaining abstinence over three years [175]. Other clinical investigations have focused on the effectiveness of twelve-step fellowships for specific subsets of the population [81] and how twelve-step fellowships can be combined with professional interventions for best patient outcomes [230]. Ethnographic investigations of twelve-step fellowships have focused on issues of language and meaning [170], social agency [193], and identity [37]. All of these emphasize that 12-step fellowships develop a distinct culture, language, and practices that are adopted by new members through participation in these communities. Only four studies have looked at recovering people’s perceptions of the role of technology in how they recover and connect with their support networks. In the first study, Campbell and Kelley showed that members of AA value mobile phones as a way of connecting with each other outside of meetings [38]. In a second study, participants who used SoberDiary—a mobile sensing system to track recurrence of symptoms, found that technology could provide an objective way of legitimizing their initial efforts to remain sober [244]. The third study showed that 12-step group members use a variety of technologies including phone, video-chat, and messaging technologies to augment face-to-face meeting attendance [241]. A survey study by Bergman et al. showed that, online users generally endorsed InTheRooms as a helpful platform, particularly with respect to increased abstinence/recovery motivation and self-eﬀicacy [22]. All of these studies point to the continued

11 3.3. Online Health Peer-Support and Anonymity in HCI importance of the in-person meetings as a primary venue of peer-support, but none investigated the challenges in finding and exchanging this support and how technology canbe utilized to address some of these challenges. We approach substance use disorders from a social computing perspective, since peer- support has been proved effective in recovery from similar health issues (more in nextsec- tion). Our formative investigation methods have been inspired by existing work in HCI on the role of peer-support and anonymity in similar health-related communities.

3.3 Online Health Peer-Support and Anonymity in HCI

More recently 12-step fellowship approaches have been expanded to online communities. Online 12-step fellowships use a number of unique peer-support practices to reach and help each other (i.e., online video meetings). Online recovery communities offer a context where both the priorities of privacy/anonymity and the priority of engaging form strong tie support networks are essential high-stakes part of everyday life and must be negotiated both at the individual and at the group levels. Investigating these types of communities can provide a new perspective on online peer support and interpretations and practices around anonymity in both recovery communities and in other health support social networks in general. Significant body of work has explored and analyzed different aspects of online health communities including types and roles of users [143, 208], impact and effectiveness of exchanged information [249], relationship between peer support and well-being [71], and importance and enactment of privacy and anonymity [67, 36], etc. HCI researchers have shown the eﬀicacy of formative investigations that utilize qualitative methods like interviews [241], participatory design [204], discourse analysis [], etc. to better understand needs of communities focusing on different health contexts (e.g., depression, mental health,

12 3.3. Online Health Peer-Support and Anonymity in HCI pregnancy etc). We adopt some of these approaches to expand our understanding of the critical components and aspects of 12-step recovery communities pointed to by prior work. We investigate these aspects and discuss what technology may (or may not) work for these communities. We propose a human-aided information retrieval technique drawing from traditional information retrieval algorithms and human computation approaches to build a technology that may have potential to enhance reach to in-person peer support in recovery. As future work, we propose conducting studies that can provide better insights on the feasibility of our proposed approach in developing the system and usability and acceptability of the developed system.

13 Chapter 4

Formative Investigation: Analysis of Online Peer Support in

Recovery from SUDs

Previous work on 12-step recovery pointed out that peer-support is a critical component for maintaining sobriety. The first aim of our formative investigations was to have a broader understanding of online peer support for recovery and to find out the challenges in finding and exchanging peer support. One example of innovation in online communities for recovery from substance use disorders to provide peer-led synchronous social support is the use of video-mediated meetings . In this chapter, we provide details of our findings about these meetings through two synergistic investigations:

• Questionnaire distributed to members of an online community (InTheRooms1) to understand how video-mediated meetings fit into their support ecosystem.

• In-depth interviews with active members of this online community to understand their lived experience with online meetings and how they manage inherent tensions involved in this style of recovery.

Previous CSCW health support community investigations have focused on text-based support (e.g., [143, 216, 250]). Clinical communities have examined video-based support 1http://www.intherooms.com

14 4.1. Related Work in the past, but only as part of professionally-delivered clinical interventions (e.g., [146, 147, 148, 156]). However online recovery communities are unique in that they provide us with the opportunity to investigate synchronous video-mediated peer support in a naturalistic context. We aim to understand the role of this video-mediated peer support and identify underlying tensions in this type of communication to inform design of other peer support health communities. We begin this chapter by situating our work in the context of previous research on online health communities and video-mediated synchronous communication. Next, we describe the specific online community and the video-meeting feature that are the settings forour investigations, as well as outline the situating quantitative characteristics of this community. Then, we discuss the methods and results of two synergistic investigations, including five vital tensions inherent in synchronous communication in this community. We conclude with a discussion connecting this work to our next investigations and by providing implications for other online peer-support communities.

4.1 Related Work

We built on the significant body of work regarding how people find peer support inonline health communities. We discuss this body of work and the new perspectives that our study offers by focusing specifically on remote synchronous video-mediated peer-support for recovery from substance use disorders.

4.1.1 Online Peer-Support Health Communities

Dedicated online peer-support health communities are important because people may not be comfortable discussing health conditions on general-purpose social networking sites [166].

15 4.1. Related Work

There have been many investigations of online peer-support health communities focused on issues as diverse as weight loss [142], cancer [71], and infertility [229]. Such communities may provide participants with information and trustworthy advice (e.g., [211]), as well as social support, friendship, and encouragement (e.g., [75, 143]). However, there are also a number of common challenges as some of these communities struggle to maintain active engagement [153], find a good balance between those who provide support and those who receive it [186], negotiate divergent viewpoints and models of the condition in question [144], and maintain members’ privacy [189]. One limitation of this corpus is that the majority of this diverse body of work has focused on text-based forums, chat rooms, or discussion boards. We build our formative investigation on this diverse body of work by considering many of the same underlying threads of anonymity, self-disclosure, identity, and mutual support in the unique context of online recovery from substance use disorders. Additionally, this chapter focuses on synchronous video-based peer support groups—a technology that has not been explored yet in other online peer-support health community in a naturalistic context.

4.1.2 Remote Synchronous Communication

Researchers have explored synchronous computer-mediated communication tools exten- sively. Much of the work has focused on the use of video-chat and novel synchronous communication technologies in the workplace (e.g., [29, 223]) and home (e.g. [12, 112, 242]). In the health and wellness domain, CSCW explorations of synchronous remote communication tools have mostly focused on collaborations between medical professionals (e.g., [155, 129]) and patient-doctor telemedicine consultations (e.g., [129]). Similarly, related fields outside of CSCW have generally focused on telemedicine and remote professional counseling interactions when considering the role of remote synchronous communication

16 4.1. Related Work in health and wellness (e.g. [8, 40]). Though the role of remote synchronous peer support has remained largely unexamined in communities like CSCW, implications from these other contexts sensitized us to potential challenges for video-mediated peer support, for example: people are less trusting over video-chat [20]; people are more likely to give riskier advice [129]; joint attention is diﬀicult [173]; and misbehavior is common in anonymous video-chat [238]. Outside of CSCW, two sets of investigations have considered the potential eﬀicacy of video-based support interventions. One set of video-based investigations delivered the Healthy Relationships behavioral intervention to a group of women living with HIV [146, 147, 148]. Another video-based intervention examined the deployment of a 10-session manualguided psychosocial support group for caregivers of older adults with neurodegenerative disease [151, 156]. Both of these investigations found clinical benefits of such social support interventions deployed over video-chat. Our investigation contributes to this body of work but differs in three important ways. First of all, we are able to examine video-based social support in a naturalistic long-term context rather than as a short-term controlled intervention. Second, both of these investigations involved interventions that were delivered by a trained professional, whereas this investigation is able to provide insight on entirely peer-led support (thus, highlighting a different set of social support issues). Third,12-step programs are an exceptional cultural context [233], worthy of investigation in their own right. This context can provide unique insight, for example, our participants are able to explicitly compare in-person and video-chat meetings (the majority having tried both). These differences allow us to consider the novel perspectives that video-based 12-step meetings can offer regarding video-based peer support.

17 4.2. Video-Mediated Peer-Support in an Online Recovery Community

4.2 Video-Mediated Peer-Support in an Online Recovery Com-

munity

We investigated video-meetings hosted by InTheRooms. InTheRooms (ITR) is the largest online community for recovering individuals, and their friends and families, hosting over 500 thousand members. This is a free public social networking site focused on connecting people in recovery and others affected by substance use disorders with peer support. Like other online communities, the ITR platform provides many features for social communication like profiles, wall posts, status updates, private messaging, instant messaging, blogs, discussion boards, and more. However, it also provides a unique mechanism for hosting and attending video-mediated peer support groups and meetings. ITR hosts more than 100 video meeting groups at regularly scheduled weekly times. These meeting span 18 recovery fellowships (e.g., AA, NA, Al-Anon, etc.). Each meeting is led by a chairperson (a volunteer peer) who selects the topic and readings for each (typically) 60-minute meeting and guides others in sharing their stories. Figure 4.1 shows the interface of an in-progress 12-step video meeting. In order to learn more about this website and its users, we created a timed web crawling framework to scrape data from ITR throughout a period of three months (February 2016 to April 2016). While the full discussion of this process and findings is outside of the scope of this dissertation, some parts of this process are important to situate our work. We collected a list of online users (as pseudonyms) every hour, a list of meeting attendees and chairpersons for every online video meeting, and users’ profile information. During the three-month data collection period, 16,452 different users accessed the website. These users’ profiles revealed that these participants were more likely to be women, with 57% of those who revealed their gender reporting being female (overall statistics: 39% female, 29% male, 32% unspecified).

18 4.2. Video-Mediated Peer-Support in an Online Recovery Community

Figure 4.1: The interface and key features of an InTheRooms online 12-step meeting

They were likely to be middle-aged (M=46, SD=13.6). Participants self-reported an average of 7.4 years (SD=9.3) in continuous recovery. We were able to gather meeting attendance data for 1,142 total meetings in the three-month period. Of the 16,452 users who accessed ITR during the study, 5,508 (33%) attended in at least one video meeting. On average, each meeting had 99 audience participants (SD=39). For each person who attended at least one meeting, the average number of meetings attended in the three-month period was 8 (SD=19.33). This situating quantitative process provided us with the context to conduct our mixed-methods and qualitative investigations of online video meetings. There are two reasons why this is a community of interest to us. First, this community is a source of sensitized participants who are able to discuss and articulate core topics, such as enacting anonymity, forming and maintaining strong tie support networks, and group self-

19 4.3. Ethical Considerations organization around the socio-technical issues of privacy, enforcing social norms, and more. Second, this is the first opportunity to study active video-mediated peer support meetings in an online health community in a naturalistic (rather than clinical intervention) setting. Understanding video-mediated meeting practices in this community can inform the design of technologies in other health contexts.

4.3 Ethical Considerations

Recovery from substance use is a personal and private undertaking for most people and we took steps to consider the ethical implications of our work and to protect the rights of ITR users. All research activity on this project was reviewed and approved by our university IRB as an investigation where potential benefits of the scientific work outweighed the risks to the participants. The approval covered three phases. First, in order to scrape basic quantitative characteristics of website use (described above), we created an account marked as a research account and unaﬀiliated with any fellowship. We confirmed that such scraping activity did not violate terms of service. ITR meetings are considered public online spaces since they are listed as “open” meetings (i.e., which may be attended by non-members as observers2). Since personal anonymity is part of the site’s terms of service (i.e., users are not to reveal their full names or phone numbers in public sections of the site, such as profiles or usernames), we were able to collect lists of pseudonymous usernames and timestamps for specific activities of interest without asking for informed consent from each site user (as that would have been impractical). Second, we worked with the ITR website founders and owners to reach out directly to ITR users with a questionnaire. A link to the questionnaire was distributed as a paid banner 2e.g., http://aavirginia.org/hp/meetings/ocm.html

20 4.4. Two Synergistic Investigations advertisement on the ITR homepage for one month (August 2015), as well as advertised in the weekly ITR newsletter with a message from the researchers explaining that we were interested in learning more about the role of technology in recovery. All users who completed the questionnaire, viewed and responded to an online informed consent form, but documentation of informed consent was waived in order to preserve participant anonymity. Third, we reached out to potential interview participants via email or on-site private message. Participants were provided with the opportunity to volunteer to be interviewed over phone or Skype with a broad solicitation to share their experience on ITR with a researcher. All participants viewed and verbally consented to an informed consent form, though again, documentation of informed consent was waived in order to preserve anonymity. All interviews were audio recorded only with the participant’s permission.

4.4 Two Synergistic Investigations

In order to understand how video-based online 12-step meetings are used by recovery communities, we conducted two synergistic investigations: an online questionnaire to understand how video meetings fit into the participants’ recovery ecosystems and an in-depth interview study to understand the lived experience and inherent tensions experienced by the participants of these online meetings. Both of these investigations focus on understanding participant perceptions of their naturalistic experience on ITR, rather than as feedback on a targeted intervention.

4.4.1 Study 1: Online Questionnaire of ITR Users

Previous studies have pointed out community worries about how video-mediated meetings may affect commitment to recovery and in-person meeting attendance [241]. ITR provides

21 4.4. Two Synergistic Investigations an excellent opportunity to investigate how online meeting may fit into the participants’ other recovery activity, as well as provide initial insight as to why they chose to attend (or not attend) online meetings. We conducted an online questionnaire to answer the following research questions:

• RQ1: How do ITR users combine online and face-to-face (f2f) 12-step meeting attendance in their recovery?

• RQ2: What characteristics differentiate participants who currently attend online only, attend f2f only, or at-tend both types of meetings?

Methods

We designed and deployed an online questionnaire to understand how ITR users combined online and f2f meetings. Recruitment and Procedure

As discussed earlier, a link to the questionnaire was distributed as a paid banner advertisement on the ITR homepage for one month, as well as advertised in the weekly ITR newsletter. Participants were asked to complete a 10-minute questionnaire about their recovery. The survey included the following information requests:

• Basic demographic information

• Recovery history, including self-reported continuous clean/sober time

• Commitment to recovery as measured by a modified 7-item AAI (Alcoholics Anony- mous Involvement) scale [98] (modification converted the word “alcohol” tothe phrase “alcohol /drugs/ my problem behavior”).

22 4.4. Two Synergistic Investigations

• F2f and online meeting attendance frequency

• Perceptions of one’s anonymity in (f2f, online, or both) meetings measured using a 5-item scale (See Appendix B)

We also included an opportunity for the survey taker to share additional contact information if they were willing to be interviewed over phone or Skype about their experience. Overall, 285 members of ITR completed the entire questionnaire (another 100 started but did not complete the questionnaire). The age (M=49, SD=11.5) gender (61% F, 39% M), and geographic (72% from US and 28% from 15 other countries) breakdown of the participants was similar to the site use demographics scraped in the situating quantitative investigation. Survey participants had an average of 8.5 years of recovery time (SD=9.5 years), which also seems consistent with scraped data. Limitations

While the ITR questionnaire allowed us to answer some important questions about the role of online meetings in participants’ recovery, this method also had inherent limitations. As with all questionnaires, there was self-selection bias to the participants who chose to respond. For example, respondents with strong opinions were more likely to want to participate, whereas respondents earlier in or less confident in their recovery may have chosen not to respond. Since our questionnaire and most of ITR meetings are in English, most of our participants were from the United States, Canada, or Western Europe, thus limiting our potential for cross-cultural insights. Additionally, the ITR website is an existing community which people use in a naturalistic setting as they see fit, rather than a controlled clinical intervention. Therefore, while we list self-reported continuous recovery time for each participant, these should not be interpreted as outcome variables of a controlled intervention,

23 4.4. Two Synergistic Investigations

Table 4.1: Statistics of meeting attendees from survey

since a number of con-founding factors likely affect both meeting attendance and recovery time (e.g., disorder severity, commitment to recovery, etc.).

Results

Our survey results revealed three types of ITR participants: those who currently attend f2f meetings only (22%), those who currently attend online meetings only (15%), and those who attend both types of meetings (52%). Since only 15% of people exclusively attend online meetings, it was important to understand how and why ITR participants made use of both online and f2f recovery meetings. Table 4.1 presents the descriptive statistics for each type of meeting attendee on several key characteristics. Figure 4.2 provides meeting attendance statistics for each of the three types of ITR participants. We report the outcome statistic of average self-reported continuous recovery time for each group, which is highest (12.2 years) for people who only attend f2f meetings and lowest for people who attend only online ones (7.7 years). However, given that these are high averages with large variability, and a number of confounding variables (e.g., ability to drive, severity of condition), we recommend continued quantitative investigation of the eﬀicacy of online meetings. We do not conduct statistical hypothesis testing since this work was exploratory and not driven by a priori hypotheses. However, it does suggest some possible hypotheses for future

24 4.4. Two Synergistic Investigations

Figure 4.2: Statistics of meeting attendance for each type of attendees

exploration:

• People who attend online meetings may be more likely to be new to recovery (but not younger in age).

• People who attend only online meetings may have an increased chance of recurrence of symptoms (as they report less time in continuous recovery).

• Participants are more likely to remain more anonymous in online meetings than in f2f meetings (which echo other studies of online anonymity (e.g., [166, 178])).

• People who attend both online and f2f meetings may at-tend more meetings total per month than people who specialize and attend only one type of meeting.

We asked people who have attended at least one meeting online and f2f about the perceived usefulness of each meeting type. Figure 4.3 reveals a distribution of responses where the mode describes the two types of meetings as equally helpful, though a greater number of participants describe online meetings as less helpful than f2f meetings. Given that f2f meetings are still seen as an important part of people’s recovery, we also provided a free

25 4.4. Two Synergistic Investigations

Figure 4.3: Usefulness of f2f and online meetings response field to allow people who did not attend these meeting regularly to share theirrea- sons. On the questionnaire, 60 people mentioned that they no longer attend f2f meetings at all and 38 mentioned that they were temporarily unable to attend meetings. We conducted an inductive analysis to cluster the provided reasons into categories. The most common reasons were:

• Lack of transportation (30% of the 98 participants),

• Health problems and/or disability (19%),

• Work scheduled during meeting times (12%)

Other reasons were less common and included lack of child-care (5%), disliking the people in local f2f meetings (8%), and philosophical disagreements with local 12-step approaches (9%). Clearly, online meetings filled an important gap for these participants, allowing them access to recovery options that otherwise would not have been available to them. Our finding suggested that online meetings served an important role in the recoveryof the participants we surveyed. This lies in contrast to previous work which suggests that 12-step members are wary of video-mediated and online meeting participation [241]. To

26 4.4. Two Synergistic Investigations understand the lived experience of online recovery among ITR members, we conducted an in-depth interviewing investigation.

4.4.2 Study 2: In-depth Interviews with ITR Members

To understand the lived experience of ITR members and the role that video-mediated meetings play in their recovery, we conducted in-depth interviews with influential members of ITR. We aimed to answer the following research questions:

• RQ1: What tensions do participants face in their experiences of meeting attendance

• RQ2: What strategies do they appropriate to address these tensions?

Methods

We conducted in-depth semi-structured phone/video-chat interviews with fourteen members of ITR. We describe the participants, procedure, analysis, and limitations below: Participants

We followed an “abductive” qualitative sampling approach, by selectively reaching out to participants who may have interesting perspectives on online recovery [164]. Reasons for selecting a potential participant included:

1. Recent active participation in f2f meetings (from responses on study 1 questionnaire)

2. Specific characteristics of ITR participation determined from the preliminary quantitative analysis, such as frequent online activity, large number of ITR friends, and recent participation or chairing of an ITR meeting.

Our goal was to be able to gather a diverse set of perspectives on online meetings and we actively sought divergent viewpoints on online recovery through this selective recruitment

27 4.4. Two Synergistic Investigations

Table 4.2: Participants and features of their online recovery. Types of meetings included: AA (Alcoholics Anon.), NA (Narcotics Anon.), OA (Overeaters Anon.), CoDA (Co-Dependents Anon.), Al-Anon (Families of Alcoholics), ACA (Adult Child in Alco- holics), Dual Diagnosis (for mood disorder), Women’s Meeting, and PTSD (Post Traumatic Stress Disorder). Selection criteria included: 1. Recent active participation in online meetings, 2. Recent active participation in f2f meetings, 3. Frequent activity on ITR, 4. High number of friends on ITR, and 5. Chairing meetings on ITR

process. Table 4.2 provides a description of the participants and the selection criteria for each participant. We continued recruiting participants until deemed that we had reached data saturation on the specific topic of interest. Procedure

We reached out to potential participants via private message on ITR, email, or by phone. Potential participants were referred to the previous research activity of our group on ITR (the questionnaire and associated newsletter post) and asked to volunteer to be interviewed about “how to improve online recovery communities” and “how places like ITR.com can help people in recovery.” Interviews were conducted through November 2015 and January 2016. Eleven of the interviews were conducted via video-chat and the other three via phone (based on participant preferences). All interviews were audio recorded with participants’ permission. Interviews were each conducted by one of the two investigators, following a

28 4.4. Two Synergistic Investigations common protocol. Each interview lasted between 40 and 110 minutes with an average of 53 minutes. Interviews were open-ended, prompting participants to share personal narratives about their online recovery. Participants’ time was compensated with a $25 gift card. Analysis

We transcribed all interviews and analyzed the transcripts using an inductive data-driven approach [138, 207]. Analysis consisted of two passes. In the first pass we generated open codes (over 1,500 total) to capture diverse individual view-points. This high number of open codes is due to our effort to capture the nuance in the specific statements of each participant and was significantly reduced through the process of memoing and clustering [138]. In the second phases of analysis, we memoed and clustered the codes using a constant comparison method operationalized as aﬀinity mapping. Each open code was compared to others and positioned to reflect its aﬀinity to emerging themes and clusters. Through this process,we identified some major tensions in online meetings as themes which are presented inthe following section. The reported themes consisted both of ones that appeared consistently across multiple interview participants, and also the ones that came from points of contention and represented divergent responses and opinions. Limitations

We acknowledge that there is an unfortunate implicit sampling bias in our recruiting approach as the people who volunteered for the interviews were more likely to want to share their opinions about their experiences and thus perhaps hold stronger opinions (either positive or negative) than the larger body of their fellowships. Again, since the interviews were conducted in English, our participants represented mostly the United States, Canada, and Western Europe, limiting potential for cross-cultural comparison (we did not find any difference between tensions reported by US and non-US participants in this study). Addi- tionally, nine of the fourteen partic-ipants were female, presenting a gender bias. However,

29 4.4. Two Synergistic Investigations this may be close to a true representation of ITR members’ gen-der ( 60% of ITR members are female based on the quantitative scraping in the situating study) and females are more likely to participate in voluntary service [231].

Results

Our analysis revealed five major tensions inherent in synchronous communication inthis community.

Self-Disclosure vs. Anonymity The choice between disclosing identifiable information and remaining anonymous is an important tension in attending online meetings. Many participants may want to keep their recovery private from employers and others, but at the same time need to share details about their personal lives in order to receive the support they need. This was an inherent tension found in many participants’ interviews and they were able to reflect on their choices and compromises. There were several views of anonymity espoused by the participants. Some saw it as a spiritual principle embodied in the 12th tradition as “principles before personalities,” meaning that all people in the room are equal and do not bear names. Others viewed it as a personal choice: “it’s okay for me to break mine” (P4) and “anonymity is not a big deal for me” (P2), but also clarified the importance of maintaining others’ anonymity: “it’s not okay for me to break somebody else’s” (P4) and “it’s also about sponsees’ anonymity” (P8). Given the public nature of ITR’s video meetings, participants enacted their anonymity in different ways. Six participants reported that they do not share key aspects of their experience on ITR. They were aware of the fact that ITR has a lot of members that they do not know and who may violate anonymity:

30 4.4. Two Synergistic Investigations

If I have an argument with my sponsor, I wouldn’t come to the ITR meetings and talk about that. Even if she hadn’t been in the room, for sure somebody could

be spreading it. (P9)

One always needs to be mindful of anonymity being respected. So many people there that would take something that was perhaps shared in the meeting under the 12th tradition of anonymity and often share with someone else. For the joy of gossip, or trying to bad-mouth somebody, they may break your anonymity.

(P8) Moreover, they were concerned about other possible harms caused to their and others’ offline lives by self-disclosure, such as“losing a job if employer knows about recovery” (P11), “losing self-esteem to other circle of friends who are not in recovery” (P9), “seeing guys in meetings who I have dated before” (P3) or, “celebrities who are members of the community, it is very important for them to be anonymous” (P2). P5 thinks non-recovering people (referring to them as “normies”) cannot comprehend what the addicts have gone through and may judge an addict’s past behavior. On the other hand, other participants considered self-disclosure as a way of getting help and as a responsibility to let newcomers know that they are not alone. Many chose to share and be open about recovery as a way of letting others know that help is available and there are ways out of addiction: I want them to know that I’m in AA and I want them to know that if they would ever get in same situation there is something they can turn to for help. And I try to set that example in all my actions, in all my daily affairs, just like it says in

step 12. (P2)

31 4.4. Two Synergistic Investigations

I feel like there’s so much addiction in the world and if we don’t start coming out of our anonymity, nobody is gonna know that. There are addicts everywhere. They are not just you know, some bomb on the street. It’s like illness, and it’s a

problem and that we need to fix it. (P3) P1 described her life to be “kind of an open book,” so that anyone can ask questions and she can answer. While most participants chose to either emphasize anonymity or self-disclosure in their use of ITR, some participants described a mixed approach. For example, they only shared detailed stories and personal information with their sponsors or close friends on ITR, or through private messaging on or off the site, but not in public video meetings. To summarize, participants were cautious and made explicit decisions about their self- disclosure practices in ITR video meetings, balancing the desire to help others and to seek support with the risks of breaking own or others’ anonymity.

Diversity vs. Like-Mindedness ITR attendees have the opportunity to interact with 12- step members around the world, as ITR hosts over 100 different online meetings with attendees from more than 70 countries. Our interview participants described an inherent tension be-tween wanting to be exposed to people with diverse perspectives on recovery and wanting to hear from like-minded others who follow familiar practices. The diversity of personalities, people, and experiences makes online meetings different from f2f ones, as P5 high-lighted: I guess the thing I really dislike about f2f is if you go to the same meeting week after week, you pretty much got their stories down. On ITR, you get the ability to hear from people from east to west coast, all over the world, different perspectives about the program as it is in different places. The variety of experience,

32 4.4. Two Synergistic Investigations

strength, and hope is far greater. (P5) However, as a divergent narrative, P12 saw this level of diversity sometimes manifesting itself with people “jumping from one meeting to another and making the meetings a blurry mess.” Even P5 (who was generally positive about meeting diversity) admitted that it could be overwhelming or intimidating for a newcomer, causing an “information over-load”. In- deed, several participants mentioned that they preferred to hear message of recovery from “other people having similar problem” (P12), “somebody in their area” (P4), and “like- minded people that [she] trusted and shared with” (P6). There were several strategies participants used for managing this tension. Many of the participants mentioned specifically listening for the consistent purpose and message ofre- covery and support across the diverse meetings and members: So many different people and that’s what interesting for me that, all the different people that come together for one single purpose: to get sober, to stay sober.

(P2) I mean feeling like that every day… I can sit and be alone in my trailer. But NO, I’m not ALONE because I have all these people all over the world. They know

who I am and they care about me. (P1) As a second strategy, participants sought out alternative viewpoints and approaches and then picked the message that was right for them in their situation: It’s very interesting to see the perspectives of someone from Germany and what their f2f meetings are like, talking to them. Then hearing to somebody in Japan, it’s just what their experiences have been in recovery is… when you have that

group of people to choose from and to learn from. (P5) There’s this is old saying, and I’ll just say this, ’how many meetings do you need?’ I need one a week. The problem is I don’t know which one it is. You’ve

33 4.4. Two Synergistic Investigations

got to hear which you need to hear. Just don’t know which one it at be, so hit

all. (P9) In summary, video-mediated meetings on ITR provide their members with an opportunity to attend a 12-step meeting with diverse people with varied experiences. While this may be overwhelming to newcomers, participants dealt with this tension by listening for the common purpose and picking and choosing approaches from the diverse set of ideas that could best support their particular situations.

Convenience vs. Contact While ITR members can attend a variety of online meetings easily, in online meetings they miss out on the f2f benefits of being together physically. The tension between convenience and physical contact is a vital aspect of how our participants managed their recovery. Nine participants expressed that the most positive and compelling feature of f2f meetings is its physicality, for example: “smelling coffee” (P1, P2, P3, P7), “seeing others’ body language” (P10), “touch, hugs, and handshakes” (P4, P7), “more personable” (P13), “better sense of emotion” (P2), etc. P6 considers this something that can never be achieved through computer-mediated communication: I wasn’t brought up with the technology. To me that physical contact is really

really important. It makes me feel human. (P6) Nonetheless, the flexibility and convenience of being able to attend meetings fromthe comfort of one’s home was frequently mentioned as a reasonable trade-off: But there happen days when the weather is really bad and I don’t feel comfortable driving because I have to drive at least 25 minutes to get to a meeting in the morning. I’ll get what I need for my recovery that day through the online

meetings. (P12)

34 4.4. Two Synergistic Investigations

The ability to go online, the ability to introduce yourself from a camera, from the comfort of your own home, from a place of familiarity, makes someone much more susceptible, even if they choose not to show their face, or comment on a

board. (P11) Online meetings are also more convenient in terms of time and can be portable, since one can join on a laptop, tablet, or smart phone: But there are few meetings at least [when I travel], I can attend the meetings

in my browser on the tablet. (P9) Sometimes I’ll be out in sporting events, like with my husband and if I get bored and I know there’s a meeting coming or you know…when I’ve started to not feel comfortable with people I’m around, then I just take my iPad with me, and login, connect to people. It’s easy and available.

(P12) These findings echo those of other online health communities (e.g.,[73, 97]), but important to establish for video-mediated meetings given that video may introduce new constraints on convenience. Individual preferences for f2f versus online meetings seemed to be influenced by the particular importance of physicality for the individual participant and their experiences of intimacy in f2f meetings. For example, P7 attends only f2f meetings and facetiously said he would attend online meetings only if technology can make his computer emit the smell of coffee.” Some participants also resisted the necessity to“become more computer-literate” (P10) or “adopt new technology” (P14). On the other hand, P3 has decided not to go to anymore f2f meetings since “the intimacy isn’t there, like it is online, which is really odd.” However, many of the participants who preferred online meetings also acknowledged a potential danger, as going online to online meetings could “turn [us] into hermits” (P4), or “feed into [my] laziness” (P13). There were several strategies participants used to deal with this tension. The first strat-

35 4.4. Two Synergistic Investigations egy was to mix attending f2f and online meetings: I go to their online meetings, when I’m sick or can’t get to f2f meeting. (P4) You’ll find a good meeting f2f, then a good meeting online. But you canalso find a bad meeting f2f, and a bad meeting online … I think a combination ofthe

two really is the best. Then you get the best of both worlds. (P8) As another strategy, people who attended only online meetings sought specific opportunities to deepen their contact with their online support network, such communicating via other channels (e.g., one-on-one Skype) and taking opportunities to meet in-person when possible. For example, one participant mentioned attending the AA international conven- tion to meet his ITR connections: We went to the conference and it was great because I actually met about ten

people there from InTheRooms. And it was fantastic! (P5) To summarize, f2f meetings provide a physicality that is diﬀicult to replicate online and is important to some participants. On the other hand, others build strong online connections and appreciate the convenience of attending meetings from home. Participants managed this tension by combining f2f and online meetings when possible and seeking opportunities for strengthening online connections through other channels.

Immediacy vs. Awareness Online meetings are available on ITR almost every hour of every day, allowing participants to receive immediate support, but also creating meetings with a less consistent attendance base and making it more diﬀicult to recognize if a person is in need of help. On one hand, the number and universal availability of online meetings allow newcomers to find immediate support in meetings “available around the clock” (P8). Additionally, meeting people online allowed participants to form a support network that could field a

36 4.4. Two Synergistic Investigations phone call for help even at really late or early local times: I needed people to call that would be available at anytime. A lot of the people around here aren’t available at any time. In Arizona when it would be ten o clock in Arizona, versus being mid-night in Georgia. So there were a lot of

people that reached out and gave me their numbers. (P6) These findings replicated the findings of some previous online health community work (e.g., [73, 97]), however they are also different from those of other online health support groups (e.g., [152]) – this may point to “immediacy” as a dimension that has to be established and measured for different online health communities. While immediacy is generally positive, in this community it seemed to conflict with the goal of awareness. F2f meetings typically have a consistent set of attendees, making it easy for attendees to become aware of a newcomer or somebody who is struggling: They might go to an online meeting and get great support. But if they don’t share, if they are not recognized, people can’t approach them, whereas in a f2f meeting people will typically know if somebody is new and will go up and say

“Hi how are you doing? Are you new or new to the area?” (P11) You shake somebody’s hand and it’s a sweaty hand, they’ve got stuff going on

in their mind, they’re trying to get sober. (P7) This awareness is not possible in online meetings unless an individual chooses to identify as a newcomer in need of help, as the audience is abstracted as a set of still icons (i.e., “the individual is not visible, it’s a steady picture” (P10)). Even with known individuals, online meetings may provide fewer opportunities to be aware of their state on any given day since their video is only shown during their share. As P3 stated, “you only get small amount of time with them,” in contrast to the easy ability to gaze over at a friend during a f2f meeting.

37 4.4. Two Synergistic Investigations

To resolve this tension, participants had to be willing to reach out to others outside of the video meetings to develop an awareness of others in their social circles. Nobody mentioned doing this for newcomers who did not self-identify. However, a few participants with established ITR friend groups mentioned checking in with each other during a meeting through back-channels (e.g., IM and Skype text chat) or through private messaging after the meeting.

Norms vs. Inclusion Twelve step fellowships generally follow a codified set of norms known as the “Twelve Traditions,” however specific interpretations and practices regarding the traditions may vary, leading to a tension between enforcing consistent norms versus being inclusive to diverse interpretations. One concrete example is in identifying oneself at the start of the meeting. Some fellowships interpret the traditions to mean that everybody should use the same language and refer to the common condition when identifying (i.e., “I am an alcoholic” in AA, “I am an addict” in NA). However, this interpretation is more flexible in some geographic regions than others (e.g., some AA meetings are fine with an attendee identifying as an addict, etc.). While the same tension exists in f2f meetings and online ones, the geographic diversity of norms can cause friction in meetings. Some participants described getting corrected (or wanting to correct others) on language and program-specific norms: In NA, we have our own program and we don’t want you to come in and start talking about what stuff you do in AA and reading from AA literature, CoDA or

any other fellowships … drives me nuts! (P12) Especially the old-timers, they are very hard core crusty, can be aggressive. It turns people off, it used to turn me off quickly. I think people are really sensitive

when they are new and coming in. (P3)

38 4.4. Two Synergistic Investigations

I thought they were much better off when they just had a meeting without at-

taching a fellowships name to it, just InTheRoom meetings. (P4) There were other examples of divergent norms that reflected geographic differences in the interpretations of the traditions, such as the use of the donation basket, cross-gender sponsorship, etc. However, there were also some clear-cut examples of norm violation that were not specific to the interpretation of the traditions. Several participants who served as the chairperson of a meeting described a number of uncomfortable situations, such as people not wear- ing a shirt while sharing video, people treating the meeting as an opportunity to flirt, and people not sharing on the topic of recovery. For example, some chairpersons mentioned: People get up there and they won’t shut up. Where five people are waiting on the box that want to talk, this person just talks about that really isn’t recovery

related. (P6) Suddenly they all talk about relationship or something. What is the purpose of

that meeting? Meeting should be literature and topic related. (P5) Each chairperson faced a trade-off. On one hand, they could allow the person tocon- tinue sharing (in the name of inclusion but at the cost of maintaining a focus on recovery). Frequently, this led to complaints from others after the meeting: When you don’t take other attendees’ advice and put it into use or into action,

they get offended by that. (P2) On the other hand, they could ban the participant from the meeting which could result in a person in need not getting the help that they may desperately require. This approach was also criticized by participants: It does bother me that they take it to upon themselves to close the doors of AA to certain individuals over certain things and it’s not their group, it’s their site.

39 4.4. Two Synergistic Investigations

Now I have never known a church to come up and say, so and so can’t come to

our meetings. (P5) Unlike other tensions, there seemed to be no established and clear-cut way for online meeting participants to determine appropriate norms and strategies around consistently enforcing them. The tension of norms versus inclusion was left almost entirely up to the chairpersons and there seemed to be no solution that was acceptable to all members.

4.4.3 Discussion

We conducted two synergistic investigations of video-mediated peer support in an online recovery community. In this section, we bring together insights from these two studies to provide recommendations and future directions for online health communities in this and other contexts.

Opportunity for Video-Mediated Support

Our investigation points to the potential of video-mediated meetings to enhance the social peer support exchanged in online health communities. Our questionnaire revealed how these meetings fit into the participants’ recovery ecosystem. Many participants whoused ITR combined f2f and online meetings, allowing them to increase the number of meetings attended monthly. Online meetings also expanded access to 12-step meetings for participants who temporarily or permanently could not attend f2f meetings in their area because of transportation, health/disability, or work schedule. Those who had tried both online and f2f meetings, generally reported that online meetings were almost as useful as f2f ones. In interviews with participants, we learned that while online meetings were not always non- problematic, participants saw a number of advantages to having this opportunity to connect online. These advantages included the diversity of viewpoints provided, the immediacy of

40 4.4. Two Synergistic Investigations support and feedback, and the convenience of being able to attend without physically going to a specific location. Overall, this points to significant potential benefits of including synchronous video-mediated meetings as a component in other peer support health communities (as has already been shown for clinician-led social support interventions [e.g., [147, 148, 146, 156]]). As most online peer-led health communities currently focus on text-based interaction, video-mediated support group meetings may provide a promising compromise between the intense interaction of f2f support groups and the diversity, convenience, and immediacy of online support.

Future Work While our investigations point to the potential of online video-mediated peer support, there are many unanswered questions about the eﬀicacy of online meetings. For example, due to a number of confounding variables, it is diﬀicult to conduct a meaningful comparison between the self-re-ported outcome measures (continuous recovery time) of participants who do and those who do not use online meetings. One initial step in this direction could be conducting a larger scale propensity matching analysis on our scraped data set to better quantify the effect of online meetings on recovery time. A more rigorous and less confounded investigation would include a controlled trial with random assignment of participants to attend f2f meetings only, online meetings only, or both. However, given that we found that online meetings may be perceived as less effective by many participants who have attended both online and offline meetings, it may be ethically questionable toassign participants to the “online only” condition.

Opportunity for Synergy with Face-to-Face Support

It is important to emphasize that participants did not generally think that online meeting should replace f2f interaction. In our questionnaire, we saw that a relatively small minority

41 4.4. Two Synergistic Investigations

(15%) attended online meetings exclusively. Many of those participants were unable to attend f2f meetings because of life circumstances, rather than by choice. In interviews, many of the participants seemed conflicted about online meetings, citing worries about self-disclosing online, a lack of physicality, and loss of awareness of others’ states. When possible, many participants preferred hybrid strategies rather than relying exclusively on online meetings. They limited self-disclosure in video meetings, but sought out additional support through more private channels. They found opportunities to meet online friends in-person and to connect through other channels that provided more time and awareness than a brief share in a meeting would allow. Finally, many seemed to prefer to combine both online and f2f meeting attendance as part of their recovery to get “the best of both worlds” (P8). Taken together these findings highlight that online meetings should not try to replace f2f ones. For online peer support groups that currently meet exclusively online (as many of the groups studied in previous literature (e.g., [142, 208, 216])), this may point to the importance of providing periodic opportunities for f2f reunions, meetings, and conventions to create opportunities for the physicality and awareness that may otherwise be lacking in online groups.

Future Work Our studies asked participants to reflect and discuss their cur-rent experience on ITR and in f2f recovery. One important aspect that was not investigated in our study is that of recovery as an ongoing and evolving journey. There may be phases in one’s recovery where f2f contact is more important (e.g., immediately following a recurrence of symptoms, during major life changes). On the other hand, it is possible that online support may in-crease participants’ willingness to seek f2f support if they seek a more anonymous venue before being willing to self-identify as an addict/alcoholic in f2f meetings. Under- standing the role of f2f versus online support and recommending the appropriate solution

42 4.4. Two Synergistic Investigations given a person’s stage of recovery, personality, privacy concerns, etc. are important questions for supporting individual recovery.

Challenge of Transparency of Norms

Norms seemed to lie at the core of the conflict experienced by online video meeting participants. Our questionnaire revealed that participants may be less likely to self-disclose in-formation that could lead to a loss of anonymity in online meetings rather than f2f ones. Our interviews continued to probe this point, with some participants revealing that they were hesitant to self-disclose because it was not clear that other video meeting participants would follow the anonymity traditions and norms of 12-step recovery. Norms were also at the core of a tension that did not seem to have a clear resolution. Some online participants would be-have in clearly inappropriate ways, such as soliciting dates or sharing non-recovery topics in meetings. Other tensions were subtler. Geographic differences in the interpretation of the traditions of each program led to strongly conflicting opinions as these variances intersected on the international stage of online meetings. Online meetings provided a diversity of perspectives that was appreciated, but also a polyvocality on norms that led to a loss of unity. Our participants did not have immediate strategies for resolving these problems, but this issue does lead to two implications for these and other online health support spaces. First, online health support groups need opportunities and spaces to discuss, work out conflict, and develop a consensus on specific group norms and policies (i.e., “stormand norm” [66]). While ITR currently does host two “business” video meetings, this is significantly less time than is typically spent in similar activities by f2f groups (which typically hold group business meeting monthly for each meeting and regular other meetings at the area, region, state, and national levels, with a clear path for information in both directions).

43 4.4. Two Synergistic Investigations

Other online health communities frequently provide little (if any) opportunity for meta- discussion and consensus building. As a second implication, once norms are established, they need to be made as transparent as possible to all of the participants in the community, as intuitions participants develop in local f2f communities of the same fellowship may not translate to the online space. For example, as a person joins an online meeting, they may be asked to read the list of policies out loud and state that they agree (similar to the reading portion in f2f meetings).

Future Work The CSCW community has conducted significant work on how collaborative practices emerge, evolve, and get rein-forced in peer production communities (e.g., [Geiger2010Work, 72, 110, 124]). There is also a fair amount of work at CSCW that focuses on identify specific norms expressed in online health communities (e.g.,[143]). However, there is a need for additional re-search in how norms in online communities are negotiated over time and made visible to new participants. One possible direction for this work in health communities is applying analysis methods used on Wikipedia discussion pages (e.g., [124]) to examining visible artifacts developed in group, area, and regional “business” meetings of 12-step fellowships (e.g., meeting notes, meeting logistics online forums). This may lead to a more process-oriented understanding of norm formation in health peer support groups which may inform the design of digital artifacts in this context.

Challenge of Constructive Moderations

Our interviews revealed that currently, the burden for forming, interpreting, and enforcing the norms in online video meetings fell almost exclusively to the chairperson of each meeting. Several of the chairpersons shared that they felt caught between the desire to maintain the norms and recovery atmosphere of a meeting and the desire to be inclusive to all

44 4.4. Two Synergistic Investigations participants. This tension may be an especially significant one in health peer-support communities, as the decision to ban an individual may result in that individual’s death. On the other hand, the decision to allow an individual to break the norms may reduce that group’s helpfulness to other participants leading to them leaving the community. Online communities frequently maintain norms and reduce vandal-isms through techniques like censorship and banning (e.g., [Geiger2010Work]) but these may not be appropriate in life-critical online spaces like peer-support health communities. Certainly, it seems that site chairpersons, moderators, and administrators should receive significant training and support in making decisions, if bans are employed in the community. However, we believe that this also points to the necessity of considering peer moderation that goes beyond bans.

Future Work There is significant potential for future socio-technical work exploring constructive moderation in online communities. Constructive moderation approach in online peer support should consider a norm violation to be an opportunity to educate the offending individual in the norms of the community and provide them with an opportunity to participate in more productive ways. Constructive moderation approaches have been previously used and found to be effective in online games (e.g., [117, 191]). One concrete idea for supporting constructive moderation in online 12-step communities is to en-courage groups to create explicit group policy documents (many in-person groups do so) and allow for anonymously referring individuals to specific sections of the policy (e.g., NA group policy may include the clarity statement4, so a per-son using AA language may be sent a pointer to this policy). This and other approaches point to interesting potential directions for constructive moderation in online peer-support communities.

45 4.5. Conclusion

4.5 Conclusion

From our findings, we conclude that video-mediated meetings may be a promising addition to online recovery communities as well as other health peer-support communities, especially if combined with periodic face-to-face opportunities. However, we also found that two challenges will need to be addressed to support such meetings in this and other health communities: enabling forming and making visible of community norms and supporting constructive moderation. One of the important tensions in the peer-support process for recovery is to balance the need of self-disclosure and anonymity. Moreover, previous research on understanding role of technology in 12-step programs indicated anonymity as one of the critical issues that arise when technology and these groups interact. Therefore, as the next step of our formative investigations, we aim to understand the interpretations of online anonymity in 12-step communities, like AA and NA.

46 Chapter 5

Formative Investigation: Understanding Interpretations of

Online Anonymity in 12-Step Recovery Communities

To answer RQ3 of my research agenda mentioned in Chapter 1, we adopted mixed-methods research techniques in understanding how individuals in 12-step fellowships like AA and NA interpret and enact [19]. In face-to-face communities (i.e., traditional AA and NA meetings), the priorities of privacy and mutual support have been negotiated through the “spiritual principle” of anonymity [169]. However, true online anonymity may be hard or impossible to achieve [167] and recovery groups must consider new practices and shifting dynamics introduced by technology [241]. Online recovery communities offer a context where both the priorities of privacy and anonymity and the priority of engaging in the community to form strong tie support networks are essential high-stakes part of everyday life and must be negotiated both at the individual and at the group levels. As a result, members of these communities are sensitized to the concept of “anonymity” and able to discuss it with nuance and depth. To understand the concept of anonymity in these online recovery communities, we present our findings for the following two questions in this chapter:

• RQ1: How do members of Alcoholics and Narcotics Anonymous interpret anonymity?

• RQ2 How do members of Alcoholics and Narcotics Anonymous enact anonymity online?

47 5.1. Related Work

“Anonymity” is an important topic in Social Computing (e.g., [128, 172, 178]), and investigating groups like AA and NA, where anonymity is a central norm or value, can provide a new perspective on anonymity that can be applied more broadly in sensitive online contexts. We approach these questions through a synergistic mixed-methods investigation, combining qualitative data from interviews and an online questionnaire with quantitative data of traces of activity from InTheRooms. Our qualitative investigations reinforce that anonymity is a core principle in AA and NA, but show that only one common interpretation of anonymity (“unidentifiability”) is currently captured in previous computing literature. Our study reveals alternative interpretations of anonymity as a “social contract” between members and a prioritizing of “program over individual” needs. Our quantitative investigation shows that while many individuals follow practices to ensure unidentifiability in their online participation, there is a significant portion of people who do not. Members with more recovery time, more time in the online community, and more friends in a particular online community are more likely to reveal identifiable information in their profile there such as a photo with a visible face. We begin this chapter by situating our investigation in previous work on anonymity and online health communities. We discuss our mixed-methods approach and report findings from both the qualitative and quantitative investigations. Finally, we discuss implications for research and design for recovery communities based on each of the three interpretations of anonymity in AA and NA.

5.1 Related Work

This formative investigation builds on prior research on online anonymity and extends the understanding of anonymity in recovery communities. We discuss the definitions and the-

48 5.1. Related Work ories of anonymity from computing and social science, its interpretations in different face- to-face and online contexts, and its roles in other online health communities.

5.1.1 Theories of Anonymity

Most of the research on anonymity in group contexts uses the definition by Gary T. Marx [150], which focuses on whether a person can be identified using any of the seven dimensions of identity: the person’s legal name, location, traceable pseudonyms, untraceable pseudonyms that provide clues about identity, revealing patterns of behavior, membership in social groups, or any information, skills or items that indicate personal characteristics. Marx’s definition is essentially “social,” requiring “an audience of at least one person.” With multiple group members, people’s interpretation of anonymity in different face-to- face and online contexts in fact go far beyond being nameless and often depend on the interaction with each other [68, 127, 222]. Anonymity has long been investigated with regard to its roles and effects in social behavior [188], which gave birth to Deindividualization Theory, asserting that the immersion of an individual within a group or crowd results in a loss of self-awareness [60]. This lack of self-identity is more likely to influence different types of aggressive and negative social behaviors when they are in group settings, such as stealing more candy with con- cealed identity while trick-or-treating [61], administering electric shocks to their victims for longer time period with face covered [251], or allocating less money for opponents in dictator game in anonymous condition [43]. Alternatively, instead of exploring the role of anonymity in uninhibited behaviors, the Social Identity Model of Deindividualization Ef- fects (SIDE) seeks to explain crowd behavior by an individual’s perceived social identity within the group [190]. SIDE assumes that anonymity within a group shifts the awareness of distinct individual more towards the identity of the group as a whole [42].

49 5.1. Related Work

Some prior work focusing on the relationship of anonymity in face-to-face communication and anti- or pro-social behavior considers anonymity as an objective phenomenon [17, 183]. Treating anonymity in this manner does not take into account the possibility of dis- crepancies between actual anonymity and individual perceptions of anonymity, neither does it consider the correlation of varying levels of perceptions with varying levels of behaviors [188, 128]. As online 12-step communities are originated from their face-to-face counterparts, the root of anonymity in these groups may be grounded on different classic theories and studies that treat anonymity objectively. However, people’s perceptions of anonymity in face-to-face groups may not necessarily apply to online spaces. Therefore, we consider anonymity in online 12-step groups as a subjective component and seek to determine how it compares and contrasts to other interpretations of online anonymity.

5.1.2 Anonymity in Online Spaces

As discussed, the way anonymity is enacted in online spaces may differ from in-person forms of anonymity. This has been the topic of a number of investigations in CSCW and related fields, as diverse as anonymity in data collection [132, 203], anonymity in social mobile applications [107, 27], and anonymity in online communities [171, 177, 224]. Dif- ferent interpretations of online anonymity is often discussed in terms of a “spectrum” or “shades” of anonymity [253], use of pseudonyms [103], using temporary and/or multiple accounts over different social networks [128, 178], and even IP masking [132, 77]. In case of Computer-Mediated Communication, researchers frequently emphasize the “unidentifiability” aspect of anonymity, because it may be possible to determine aname from other identifiable information (e.g., [125]). Whereas some investigations have identified negative effects of unidentifiability, such as negative disinhibition that causes usersto deviate from normative behavior of the community by lying [221], posting malicious com-

50 5.1. Related Work ments [125, 214], or harassing others [214, 221], other research in this space has discovered a number of benefits of unidentifiability, including increases in self-disclosure [176, 15], more willingness to discuss potentially embarrassing topics [26, 106, 185], and more honesty and intimacy in communication [141]. Some studies showed that prohibiting anonymous participation in fact decreases negative disinhibition and toxic behavior in some communities [111, 50]. Although 12-step communities were formed long before any of this research was conducted, their inclusion of anonymity as a program principle may reflect an intuitive understanding of the potential positive aspects of anonymity. AA and NA considers anonymity to be a “spiritual foundation” of the program [7], but it is not known how interpretations of anonymity in these groups relate to previous theoretical or empirical work or how those interpretations are enacted by the members, in-person or online.

5.1.3 Anonymity in Sensitive Online Contexts

Twelve-step communities allow their members to discuss highly sensitive topics, such as alcoholism, illegal drug use, and consequences stemming from those activities. Anonymity may be particularly important in these communities and others where participation may carry stigma. Anonymity (or pseudonymity) facilitates the discussion of sensitive topics [79, 198, 205] (e.g., health problems, eating disorder, substance use, LGBT, etc.) for users who feel stigmatized by their conditions [54] or by their socio-economic differences with other members of the community [237]. Often anonymity provided by temporary identities and the absence of or reduced visual and social cues allow people to feel more comfortable talking about their conditions without the fear of being judged [236, 16]. Another study identified perceived effectiveness of pseudonyms as an important factor to shape theextent to which drug forum users discuss their own drug use online [19]. Furthermore, exchang-

51 5.2. Methods ing help and support has been identified as one of the primary reasons of seeking online anonymity [106]. People find it easier to discuss personal problems online than face-to-face because of the affordance of anonymity that many online communities provide [120]. Al- though they often have to balance between sharing information related to specific needs and the desire to manage self-presentation [166], self-disclosure around stigmatized conditions like mental illness or sexual abuse tend to be higher in platforms that allow anonymous or semi-anonymous communication (e.g., Instagram, Tumblr, or Reddit) [16, 54]. Not only that, anonymous contents gather feedback that is more involving and emotionally engaging [54], and garner relevant responses with little cyberbullying or negativity from others [26]. These investigations indicate that there may be potential to achieve greater anonymity online than in person in some contexts, based on perceptions of anonymity in those particular settings. Prior work specifically with 12-step communities found that members may actually worry more about whether anonymity can be enacted online in a way that respects the traditions of their programs [241]. Similarly, an interview study on use of online communities and online video for AA and NA meeting participation, highlighted anonymity as an important tension in how members use these tools [198]. We build on this significant previous work to further unpack the understanding of interpretations of anonymity in 12-step communities, triangulate those findings with multiple methods, and identify the implications for designing online spaces according to the specific needs of 12-step recovery.

5.2 Methods

In order to understand how AA and NA members perceive and enact anonymity, we conducted a mixed methods investigation with data from three sources: secondary analysis of

52 5.2. Methods

Table 5.1: Participants’ Demographics. Fellowships included: AA, NA, OA (Overeaters Anon.), EA (Emotions Anon.), ACA (Adult Child in Alcoholics), CoDA (Co-Dependents Anon.), Al-Anon (Families of Alcoholics), and CMA (Crystal Meth Anon.).

two in-depth interview studies, qualitative and quantitative analysis of an online questionnaire, and a quantitative analysis of scraped data from a large online recovery community. We describe each of these and how each method complemented and extended the work.

5.2.1 Secondary Analysis of In-Depth Interviews

We conducted a secondary analysis of data from two previous in-depth interview studies [198, 241] on technology use by members of 12-step fellowships. Together, these previous

53 5.2. Methods studies included 26 participants from a variety of 12-step fellowships, with diverse times in recovery, and representing an even gender split (see Table 5.1). These interview studies were conducted to answer different primary research questions. Our previous study [198] focused on the role of video-mediated meetings in online recovery, while work by Yarosh [241] discussed general attitudes towards the use of supporting technologies by recovery communities. Participants of [241] were recruited from a participatory observation of 12-step face-to-face meetings and an announcement before the meetings. Participants of our previous study, as described in 4,[198] were contacted via the private messaging option of an online recovery community. Each interview lasted between 40 and 130 minutes. These included the following descriptive questions about anonymity:

1. Describe the importance of anonymity in your recovery.

2. How do you maintain your anonymity in your online participation?

3. Describe a situation where you did not share details of your story because of privacy/anonymity. How did that affect your credibility?

4. Describe a situation, if any, when you think anonymity made it harder for you to connect with others or to get support.

We chose to conduct a secondary analysis of these interviews because both featured “anonymity” as a core theme of their findings, though neither was able to expand upon this theme nor sought to achieve a generalizable understanding of its interpretation adopted by recovery communities. We secured the anonymized transcripts of the in-depth interviews from both studies directly from the research team that carried out those work. We conducted additional passes through the interview transcripts using a directed qualitative analysis [95]. As is key to this

54 5.2. Methods type of analysis, we began with the grounding provided by previous work, developing codes both prior and concurrent with the data analysis. Several themes were already made clear in the previous work, such as the role of anonymity for individual privacy, for protecting the program, for establishing an atmosphere of equality, and as a spiritual principle [198, 241]. Additionally, our previous investigation pointed to a number of tensions between anonymity and online self-disclosure, such as the desire to maintain personal anonymity conflicting with the desire to better help newcomers [198]. Taking these existing themes into account, our secondary analysis focused on identifying specifically how people interpret and enact anonymity and uncovering themes that were excluded from the primary analysis because they did not relate specifically to the narrative of those investigations.

5.2.2 Qualitative and Quantitative Analysis of an Online Questionnaire

While the secondary analysis mentioned above was able to provide us with rich description and lived experience of anonymity in 12-step groups, “abductive methods” [164] like qualitative interview analysis typically focus on theoretical saturation rather than generaliz- ability or sampling. In order to understand to what extent identified themes were common or uncommon in 12-step communities and to be able to compare specific perceptions of anonymity in online and offline 12-step groups, we conducted a questionnaire study including specific questions focused on the perceptions of anonymity. A link to the questionnaire was distributed as a paid banner advertisement on the homepage of a large online recovery community (InTheRooms2) for one month, as well as advertised in the weekly newsletter of the site. InTheRooms (ITR) is the largest online social network (453,715 members as per April 2017) for people in recovery from substance use disorders, hosting a number of 12-step fellowships, meetings, and forums. Members of this site create a profile and interact with others through private messages, public “wall” posts, group forums, and

55 5.2. Methods video-mediated 12-step meetings. Participants were asked to complete a 10-minute questionnaire about their recovery. A complete version of the questionnaire is provided in Appendix B. All questions were optional to answer to allow participants to skip questions that they felt were too personal. The questionnaire included questions about basic demographics (i.e., age, location) and recovery related information (i.e., time in recovery, frequency of meeting attendance, etc.). We included the following questions focused specifically on perceptions of anonymity:

1. “To what extent does this statement describe you: ’Anonymity is part of the spiritual foundation of my recovery’?” (5-point Likert scale answer from “strongly disagree” to “strongly agree”)

2. A modified version of Yun’s scale for measuring perception of one’s anonymity [248]. This scale included five questions for measuring perceived anonymity in face-to-face meetings and seven questions for measuring perceived anonymity in video-mediated ITR meetings. (Each question was a 5-point Likert scale from “strongly disagree” to “strongly agree”).

3. “In your own words, what does anonymity mean to you?” (Provided a paragraph text field for answer)

Anonymity is frequently discussed in 12-step programs (e.g., AA’s pamphlet#47) [220] and we could reasonably expect that participants would respond to question #3 vis-à-vis their interpretations of anonymity in recovery. Overall, 285 participants (61% female, 39% male) completed the entire questionnaire (another 100 started but did not complete it). The average age of the participants was 49 years (SD = 11.5) with an average of 8.5 years of recovery time (SD = 9.5 years). The

56 5.2. Methods majority (72%) of the survey participants were from United States, while the other 28% represented a total of 15 other countries worldwide. Quantitative questions were analyzed using appropriate descriptive statistics. Quali- tative questions (like #3 above) were analyzed using thematic data-driven analysis [164]. We began with a focused reading of questionnaire responses, applying open-codes to capture individual units of meaning. Next, we combined open codes from this process, with those previously developed from the secondary interview analysis. We clustered open codes based on thematic aﬀinity. Through multiple such clustering passes we arrived at asetof themes that comprehensively captured each reported perspective on anonymity in 12-step communities. We then developed a codebook where each theme was described by a single code and a long-form description of the sorts of responses which that code may encom- pass. As the authors applied the codebook, additional small refinements were made to the long-form descriptions to enhance consistency in applying the codes. Once the codebook was complete, two investigators independently coded a sample of 20 randomly selected answers (20% of total number of 101 answers that were given to this question), and discussed each answer and corresponding codes to establish a common vocabulary. Following this process of establishing a common ground, we coded another set of 20 randomly selected answers to establish inter-rater reliability. This yielded an overall Cohen’s Kappa coeﬀi- cient of 0.91 (over 20 answers, coded in 11 non-exclusive categories), which is considered “almost perfect” agreement [123]. Given this high level of agreement, the lead investigator then assigned categories to the remaining 81 answers.

5.2.3 Quantitative Analysis of Scraped Data

There are a number of limitations of self-reported data like interviews and questionnaire responses described above. First of all, how people enact or practice anonymity may not

57 5.2. Methods match their stated descriptions. Second, self-reported data is often impractical to gather at a substantial scale, making it diﬀicult to identify and interpret specific patterns that mayap- ply only to subsets of the population. To address these limitations, we sought to triangulate this data with a quantitative investigation of automatically collected traces of participant behavior on ITR. ITR lets users control how much information they share within the community, and thus provides us with an opportunity to examine some of the aspects of enacting anonymity online from behavior traces of a large user base. The following descriptive data was collected from ITR throughout a period of one year (November 2015 to October 2016):

1. Once every hour: a list of usernames who have loaded an ITR page within the past hour (available on the ITR homepage as an “Online Now” list).

2. One time: Self-reported demographic and recovery-related information on user profiles, such as first name, last name, screenname, number of friends, age, recovery time, and URL of the profile picture.

We considered the set of categories developed in the previous phases to decide which codes may be operationalized in a way that could be measured through traces of online activities. Although, not all perceptions are quantifiable, we could test a subset of the categories with data from ITR users’ profiles and we could triangulate this data with other traces to understand specific patterns in interpretations of anonymity across members of 12-step communities. Specifics of additional data collected and statistical methods employed to understand specific patterns are discussed in section 5.3.

5.2.4 Ethical Considerations

Due to sensitive nature of the data we analyzed, we took steps to consider the ethical implications of our work and to protect the rights of the ITR users. All research activity on

58 5.2. Methods this project was reviewed and approved by our university IRB as an investigation where potential benefits of the scientific work outweighed the risks to the participants. Toscrape basic quantitative characteristics (described in section 5.3.3) of the community users, we created an ITR account marked as a research account unaﬀiliated with any fellowship. We confirmed that such scraping activity did not violate ITR’s terms of service. Since personal anonymity is part of the site’s terms of service (i.e., users are instructed to choose carefully the personal information they post like full names or phone numbers in public sections of the site, such as profiles), we were able to collect lists of users and related profile information for specific activities of interest without asking for informed consent from each ITR user(as that would have been impractical). As scraping ITR profiles does not violate ITR’s terms of service we did not require the permission from the site owners and it was carried out without their direct involvement. As the work progressed, we were able to structure a more formal collaboration with the ITR founders. The questionnaire study was discussed with them and was distributed with their permission and support. All users who completed the questionnaire viewed and responded to an online informed consent form, but documentation of informed consent was waived in order to preserve participant anonymity.

5.2.5 Limitations

The interview transcripts from previous studies and the online questionnaire allowed us to answer some important questions about interpretations of anonymity in AA and NA. How- ever, these methods had inherent limitations. First, there was a self-selection bias to the participants of the online questionnaire. For example, members with more time and experience in recovery were more likely to want to participate. Second, both the questionnaire and interview participants represented mostly the United States, Canada, and Western Eu- rope. Future work could consider a cross-cultural comparison of different perceptions of

59 5.3. Results anonymity. Finally, we collected online traces of user behavior from a representative online recovery community, whereas 12-step members in different communities may enact anonymity differently. Still, we believe our approach is suﬀiciently robust to providea foundation for understanding perceptions of anonymity online.

5.3 Results

We begin by concurrently reporting the results of both the interview analysis and the online questionnaire (given the high synergy in these two investigations). We then expand on and triangulate aspects of these findings with our quantitative investigation of online trace data from ITR.

5.3.1 Role of Anonymity in AA and NA

Since the foundation of AA in 1935, the concept of anonymity was described as a core principle of the community [169]. The 12 traditions are the guidelines for how 12-step groups and members should interact with each other, the public, and AA as a whole. Two of the traditions are particularly relevant: the 12th tradition (“Anonymity is the spiritual foundation of all our traditions, ever reminding us to place principles before personalities”), and the 11th tradition (“We need always maintain personal anonymity at the level of press, radio, and films”) [234]. However, there has been much debate about the exact definition of anonymity and how the Internet fits into these traditions, as Yarosh observed some level of anxiety among her study participants about the interaction of internet and anonymity [241]. Again, work by Rubya and Yarosh pointed out that despite these worries a lot of people use digital technologies as part of their recovery [198]. The technologies deemed acceptable to people varied based on the individual interpretations of and commitment to recovery.

60 5.3. Results

By deploying our questionnaire to members of ITR, we were able to see how members of this online community reasoned about the role of anonymity. We found that 75% questionnaire responders agreed or strongly agreed that anonymity was “the spiritual foundation” of their recovery, indicating that the majority people using online tools for recovery still considered anonymity as an important part of their recovery. We attempted to measure participants’ perception of their own anonymity and compare their perception of anonymity online to that offline. We used a modified version ofYun’s perceived anonymity scale [248], which is a common instrument deployed in computing investigations (e.g., [128]). This instrument focuses on facets of perceived anonymity such as whether people in a community know your name, place of work, where you live, etc. We observed that the mean of aggregated perceived online anonymity score is higher (M = 19.62, SD = 5.97) than perceived offline anonymity score (M = 16.21, SD = 6.68)out of 35, providing indication that people think they are more anonymous in online settings. We used a paired t-test to compare perceived anonymity between face-to-face and online meeting for the participants who reported attending both types of meetings (N = 148). This difference was statistically significant at a .01 level (p < 0.000). Previous work on 12-step recovery communities showed that people worry more about anonymity online than in traditional offline meetings [241]. Thus, we would have expected participants to rate online contexts lower than offline ones on the Yun’s anonymity scale. However, our questionnaire revealed the opposite—participants’ scores on this scale were higher for online rather than offline contexts. This surprising reversal led us to believe that scales established in computing investigations might not fully capture the perceptions of anonymity in 12-step online communities. Therefore, one of the main goals of this research is to develop a better understanding of how anonymity is both perceived and enacted in 12- step communities, as a necessary step in formation of guidelines for anonymity-preserving

61 5.3. Results

Table 5.2: Frequency of themes in the questionnaire responses (number and percent of responses with a particular code, applied non-exclusively). technologies in this critical context.

5.3.2 Interpretations of Anonymity

We investigated self-reported interpretations of anonymity through a secondary analysis of in-depth interviews and a qualitative analysis of the online questionnaire. Our analysis identified ten themes within these self-reported interpretations. We broadly characterize these ten themes into three overarching categories (themes and broad categories listed in Table 5.2), which we report in this section with examples from the data.(footnote)

Unidentifiability

Most previous investigations conflate anonymity with “unidentifiability”—the absence of identifiable personal information. Three themes in the interviews and questionnaire seemed to reflect a similar set of ideas and priorities: not disclosing one’s full name, not disclosing that one is in recovery, and having control over one’s recovery. Overall, we observed that AA and NA members wanted to remain in control over the information disclosed about their

62 5.3. Results identities and recoveries. To many, this meant an absolute decision, such as not sharing full name or photo online and keeping their recovery histories private. Others simply wanted to have case-by-case control over to whom they disclosed. We also saw examples of people who acknowledged these common definitions of anonymity but chose to take the opposite approach (primarily to make it easier for them to help others). Not disclosing full name. Many of the interview participants stated that they preferred not to associate their full names or faces with their online participation. They did not share their last names on ITR profile, though many did mention using real first names, but avatars with photos: My first name is there, and an avatar but my last name isn’t. (P22) Oneofthe interviewees confirmed not using a full name, but also mentioned using a pseudonym asa first name: It’s unusual for people to give their full name in their bio. I don’t even use my

real name there. (P16) This may represent an unusual practice in the context of AA and NA (though common in some other online communities [19]), as in our observations of face-to-face meetings people tend to give real first names. We investigate this idea quantitatively in more detail in section 5.3.4 Online, participants mentioned taking extra care not to associate other aspects of their identity that may link their full name with their ITR accounts. P24 reported buying online gift cards for making donation instead of using credit cards or PayPal so that her real name cannot be traced from her accounts. Members discussed their fear of “not understanding technology and what people can do with it” (P22), and “some people being predatory online” (P14) as reasons of not sharing full names or faces. However, as a contrary viewpoint, some others were not uncomfortable sharing their full names within the community, and often in public. They pointed out the importance of

63 5.3. Results sharing identifiable information to help others and to carry out service work, which isan important step in fellowships like AA and NA. I have put my full name there. I put my business card right on there before,

which used to, had my old cell phone number, my e-mail. (P19) Questionnaire responses reflected these conflicting approaches as well. There were18 answers in total (17.8%) that reported interpreting anonymity as being “nameless.” Sixteen of these (15.8%) preferred to maintain unidentifiability, while two of them contained the divergent perspective of “I don’t care much about it” and “they need to know who I am in case they need my help.” Not disclosing that I’m in recovery.Twelve step group members’ denotation of remaining “unidentifiable” went beyond withholding names and faces from profiles. Interview participants expressed that, to them anonymity meant not disclosing to anyone that they were in recovery. For these participants, it was particularly important to keep their meeting attendance and recovery activities (online and offline) separate from other areas of their lives due to the fear of “losing job or self-esteem” (P2, P16), “having social stigma” (P17), or others “lacking open-mindedness” (P22). I don’t want the general public knowing that I am an alcoholic. It would affect

my job. (P11) Again, this was not a priority for every participant and others expressed divergent opinions: Everybody here where I live could know that I’m addict. So, I don’t really care.

(P21) I am public about being clean. (Q18) In the questionnaire, we found 10 (9.9%) answers focusing on interpreting anonymity as not disclosing about their recovery to others. On the other hand, 2 (1.9%) participants explicitly stated that while others may view anonymity that way, they “didn’t mind people

64 5.3. Results knowing.” Personal control over disclosure. While the previous two themes focus on all or none disclosure of name and recovery status, a significant number of participants reported, anonymity is about having the personal control to make disclosure decisions on a case-by-case basis. Participants wanted full control to decide if and with whom they shared their recovery history, and would expect that only they be able to break their anonymity. I can break my own anonymity to you, to my boss, to my neighbor, my family,

my best friends. It’s my decision. (P1) I don’t mind anybody knowing that I am a recovering alcoholic, as long as I do

the telling. (Q52) We found strong support for this interpretation in the questionnaire data as we assigned 16 (15.8%) responses to this code.

Social Contract

Three diverse, but closely related, views of anonymity focused on the overarching theme of anonymity as a “social contract.” These codes discuss anonymity as mutual trust and implicit agreement for protecting each other’s privacy and anonymity. Unlike the previous theme, the focus of this one is not on privacy through “unidentifiability” but rather on privacy through social norms. This theme comprised the largest number of questionnaire responses (N=50, 49.5%) and included aspects of not disclosing meeting (both physical and online) information, of respecting others’ privacy, and of trusting other people to respect theirs. Not disclosing things seen/heard in meetings. Most twelve-step meetings start or end with a reading that says, “What you see here, who you see here, when you leave here, let it stay here” (frequently followed by a chorus of agreement from the group: “Hear! Hear!”)

65 5.3. Results

[1]. Many of the interview participants interpreted anonymity as the confidentiality of the meetings they attended. It is very important for me that, what is said in the meeting stays in the meeting.

(P21) You cannot take what’s shared in a meeting and share it with someone else.

(P12) This code was particularly prominent in the questionnaire responses. It was assigned to 30 (29.7%) responses. For example: “privacy and confidentiality regarding the shares,”“not to divulge who go to my meetings,” etc. Many considered it taboo to discuss meeting topics outside of the meeting. However, one participant admitted that they would “discuss others’ recovery experience outside of the meetings without associating any names if the story is important” (Q33). This is interesting because it seems that interpretation of what exactly can be shared outside of a meeting may be different for different participants. Respect for others’ anonymity. In groups like AA and NA, anonymity is not merely about preserving own privacy, members tend to view anonymity as a communal responsibility to keep others’ recovery protected. Interview participants described a moral obliga- tion to respect others’ choices of being anonymous. While they could decide to break own anonymity (as pointed out in section 5.3.2, members were mindful not to break others’. You are not supposed to break someone’s anonymity. That’s the only rule I can

think of. (P15) You cannot talk about other people, [whether they] are in or not in recovery. (P22) Participants were aware of the harms that might be caused to other people if their anonymity was violated: I know about many instances where other people would break other people’s anonymity on Facebook because ‘well we’re both alcoholics’. But this other

66 5.3. Results

person’s circle of friends is not necessarily alcoholic, isn’t necessarily aware

about this person’s problem. (P17) Our questionnaire results were consistent with this finding, with 36 (35.6%) participants describing their perception of anonymity as “to protect others’ privacy.” While this theme echoes many of the concerns mentioned in section 5.3.2, it was interesting that participants framed their understanding of “anonymity” in terms of protecting the anonymity of others in addition to seeking their own privacy only. Trusting others to respect your anonymity. For many of our participants, “anonymity” was synonymous with “trust.” AA and NA members in our study felt safe sharing their recovery history in the meetings due to the mutual respect and trust they had developed in the community. If I want someone to know I am in recovery, I can trust that I’ll be the one that

tells them. (P22) Others will respect my anonymity, so it’s an opportunity to get recovery without it effecting my work, family relationships. It makes me feel safe tospeakmy

mind. (Q66) In fact, some of the participants mentioned that it may be easier to have this kind of trust in an online setting (perhaps because of fewer identifiability concerns): It’s much easier to trust online because they can’t do very much to me. So, I

kind of initially trust everyone until I have a reason not to. (P15) On the other hand, others worried more online: You got to be careful online because as soon as it is shared, they could do

whatever they want. (P20) On ITR I don’t trust that what I say in the meeting will stay in the meeting. I cannot control who is in the meetings. You can login as guest for instance.

67 5.3. Results

(P21) Twenty-one (20.8%) of the questionnaire participants reflected similar perceptions of anonymity as “feeling safe and secure while sharing.”

Program over Individual

A 12-step “program” is a set of guiding principles outlining a course of action for recovery from addiction, compulsion, or other behavioral problems. The twelve steps were created by the founders of Alcoholics Anonymous (AA) with a primary purpose to ““help someone stay sober and help others achieve sobriety” [234] by effecting enough change in the individual’s thinking through a spiritual awakening. Sobriety is fostered by regular meeting attendance, or contact with a sponsor or other members of the program. There are principles that correspond to each of the twelve Steps of AA and NA (e.g., Step 1: “We admitted that we were powerless over alcohol—that our lives had become unmanageable”—refers to honesty) [7]. In our analysis, many of the participants’ interpretations of anonymity ran deeper than unidentifiability or social norms around privacy— participants spoke about anonymity in terms of the program’s principles of humility, justice, and spirituality and emphasized on putting the principles ahead of their own ego. We associated four codes regarding equality and humility to fall into the high-level category “program over individual” and noticed a substantial number of questionnaire response (32.7% of 101 total quotes) fitting into this category. Humility. The principle of humility is associated with step 7 of AA and NA, which is about asking for higher power’s help in forgiving self and embracing pursuit of modesty as a fundamental aspect of staying sober. Several participants related anonymity to this concept of being humble. Some of them took a spiritual approach, saying “everything is being done by them by a higher power” (P20).

68 5.3. Results

Another sense of anonymity is trying to be dependent on God and do my God’s

will. And in that sense, I try to act anonymously. (P11) One of the questionnaire participants described anonymity as a way of maintaining the integrity of the group: Anonymity is practices/beliefs that remove the individual from the circumstance

and allow Higher Power and/or Group Conscience to prevail. (Q41) In the process of practicing humility and modesty, they learned to be anonymous in helping others in the program, rather than seeking to “get any recognition” (P26) for those activities: I don’t need a whole lot of patting on the back from that. God provides a lot of

that to me. (P19) Thus, maintaining anonymity in carrying out service for the program or in carrying the message of recovery to other members of the program enable the members to do their higher power’s will, and to give up their natural desire for distinction, forsaking all rewards and credit. Twelve (11.9%) quotes from the questionnaire explained anonymity with respect to humility, for example, “service without expectations” (Q34), “way to cultivate humility” (Q65), “relinquishment of ego” (Q7), and “not taking credit for recovery” (Q88). Equal rights in the rooms. Twelve-step groups serve diverse memberships of people of various backgrounds, demographics, and personal characteristics. Our participants considered anonymity as a mean of ensuring equality. Several participants mentioned the importance of paying attention to the principles or ideas shared by individuals, rather than being distracted by the specifics of their personalities: Anonymity helps us all to remember to put the principles first before our personalities. It is about, the individuals only get better if we, practice the principles.

69 5.3. Results

(P22) In particular, this was important to apply to the idea that everybody deserved to receive the benefits of the program regardless of who they are: Because anybody and everybody can come into these groups and you aren’t gonna get along with everybody. But they deserve to be clean and sober today

just as much as the next person does. (P22) Anonymity also meant that no participants received access to special rights or accolades based on who they were: We have no leaders, presidents, ceo’s, etc. We have trusted servants, addicts

just like me, who know that to keep what we have we must give it away. (Q53) All human beings. That I am not more or less special than anyone else. (Q78) The questionnaire responses also reflected the interpretation that anonymity is about equality regardless of recovery age and social status. These responses included under- standings of anonymity like “unconditional compassion” (Q68), “no one is better or worse” (Q39), etc. We assigned 11 responses (10.9%) to this category. Right of having own understanding of recovery. Several participants interpreted anonymity as the right to have own understanding of recovery and how it works. While we did not see any examples of this theme in the interviews, there were a few questionnaire participants whose answers fit into this theme. In our questionnaire, these participants described anonymity as “not pushing my own understanding of recovery to anyone else” (Q17), “freedom to be myself” (Q25), and “not [having] to gain acceptance of the choice of channels I implement to heal from addiction” (Q30). This means any individual can become a member of the program with just the desire to stay sober—it is the spirit of recovery that matters. It is okay to practice the program with own understanding and anonymity gives the freedom of not caring for gaining acceptance of this understanding as individuals can decide with

70 5.3. Results who and what they want to share. Five (4.9%) of statements fit into this theme. Not affiliating recovery identity at a public level. The last code in this category focused on protecting the program by not affiliating with it at a public level. The 11th tradition in both AA and NA requests members “maintain personal anonymity at the level of press, radio, and film.” While on the surface, this may appear to refer to “unidentifiability,” the specific explanation of this tradition given by participants highlighted an alternative view. Participants worried that the reputation of the program may be affected if someone iden- tifies as a member publicly. Therefore, personal anonymity makes it harder for individuals to bring the whole group into disrepute by prohibiting public affiliation with a particular program. If I become the face of NA in a movie or on TV, and then I get loaded, then a whole group of people who have seen that movie or have seen me on TV saying, I’m an addict, and look at how NA has changed my life, well then they see NA didn’t work. Which is not the truth at all, but that’s the perception of people

who have seen me as the face of getting loaded. (P1) Participants extended their understanding of this tradition to online social networks and recovery communities as well. My personal anonymity ‘...need always be maintained at the level of press, radio

& film...’ which now most emphatically includes online forums. (Q65) However, at least some of the participants mentioned making the explicit decision to “break anonymity on social media” (P16). Although the importance of personal anonymity at a public level was mentioned by many interview participants, only seven of the questionnaire responses (6.9%) reflected this view.

71 5.3. Results

5.3.3 Observed Patterns in Questionnaire Responses

As participants in the questionnaire represented diverse views of anonymity, we sought to identify more specific patterns that could account for these divergent interpretations. For example, demographics like age and gender and recovery-related information like time in recovery and time in the community may all influence interpretations of anonymity. As we conducted open coding of questionnaire responses, we noted possible patterns of interest to investigate in a more quantitative way. One of the possible patterns that caught our attention was a potential shift in participants’ interpretations of anonymity as they gained more time and experience in recovery communities. Indeed, this would be consistent with previous research which shows that community members tend to develop a “sense of community” over time and participate in a way that contributes to the group’s purpose [28]. To investigate this possibility, we constructed a histogram measuring the frequency of mentions of each of the three overarching themes by participants with different lengths of recovery time (see Figure 5.1). We grouped these members in four quartiles according to their time in recovery (each quartile representing one fourth of the participants). Each bucket had 24 participants (5 participants did not report their recovery time in the questionnaire and were excluded from this analysis). We found out that members who interpreted anonymity as absence of “identifiability” concentrated around the first two buckets representing recovery month less than 90 months. Sixty-three percent of 37 participants who talked about “unidentifiability” had recovery time in this range, “program over individual” respectively, and this percentage for “identifiability” gradually decreased with recovery time (First series in Figure 5.1). However, given the small number of participants per quartile and the relative lack of people in early recovery in the questionnaire dataset, we wanted to see whether a statistically significant

72 5.3. Results

Figure 5.1: . Interpretations of Anonymity with respect to Time in Recovery (participants are divided in equal-frequency bins). pattern would be present in the larger online population.

5.3.4 Quantitative Study of Enacted Anonymity on ITR

Both the interview and the questionnaire responses were self-reported data. As self-reported data has its inherent limitations, and driven by the findings mentioned in section 5.3.2 and 5.3.3, we triangulated this data with traces of user behavior on ITR to observe if and how people enacted anonymity online by being “unidentifiable” and if the perceptions of anonymity varied with other demographic or recovery-related information. To collect traces of online behavior from members who actively utilized the community as a tool for recovery, we considered only the “active users” of ITR for this analysis. We use the term “active users” to refer to the subset of users who were online on ITR for at least three different sessions in the one year period of our data collection (N =28,490).

Quantification of “Unidentifiability”

It is important to note that, conceptions like “social contract” and “program over individual” are not quantifiable from traces of users’ online behaviors. The theme “social contract” in our findings implied viewing anonymity as mutual respect and trust. While there areex- isting self-report scales to measure perceived trust and social support [252], these cannot

73 5.3. Results be readily inferred from traces of online activity. Same argument applies for the practice of humility and the sense of being treated equally on ITR—the core concepts of “program over individual.” We acknowledge the limitation of not triangulating our qualitative findings about anonymity as these two ideas with any other data source, although these are important and novel concepts of anonymity in recovery communities. However, an analysis of the “unidentifiability” perception in ITR’s large user base can provide us with insights ofhow their interpretations and enactment of anonymity match or differ. Some (though not all) aspects of “unidentifiability” can be seen through traces of online activities. We analyzed three quantifiable practices that many participants reported as ways of enacting “unidentifiability:” not sharing full names, using pseudonyms instead of real names, and not aﬀiliating their recovery accounts with images of their faces. Analysis of “sharing name” behavior. ITR enables its users to adopt the strategy of not sharing first or last names (an approach mentioned by many of our participants), asitpro- vides the option of participating in the community by just selecting a unique “screenname” identifier. We examined the proportion of users who entered more than a single character into the first or last name field in their ITR profile. We observed that a large proportionof ITR users shared neither first name nor last name directly in their profile (N = 23,620; 83%). Of the remaining 4,870 active users, 4490 (15.8%) had only first names and 380 (1.3% of the active members) had both first and last names on their profiles. No active members were found to have shared only last name. This low proportion of sharing both first and last names indicates that a lot of users enact online anonymity by not disclosing full names. However, these proportions alone do not make it clear whether people are sharing actual first and last names or pseudonyms in these fields, and we discuss this next. Analysis of “using pseudonym” behavior. It is not immediately clear whether people would use pseudonyms in online recovery spaces. From our experience of informal par-

74 5.3. Results

Table 5.3: Name and photo sharing behavior of ITR users

ticipatory observation of dozens of open AA and NA meetings, 12-step members tend to share their real first names in face-to-face meetings. On the other hand, in our interviews we had one participant discussing her choice of using a pseudonym instead of her real name on the online profile. Previous research in health domain also points to potential benefitsof anonymity through using pseudonyms in online health communities [111]. Our quantitative analysis provided us with the opportunity to empirically address this question. One of the aspects we noticed in our data set is that some people fill in the first or last name fields, but with something other than a name. For example, we noticed recovery slogans like “As Bill Sees It” and “Keep It Simple” as first or last names in our scraped data. To understand whether this behavior was commonplace, we came up with an approach to detect possible use of non-names in this field. If people were using real first or last names, then we would expect the proportion of each most common name to be similar to that of the general population. For example, 1% of the US population has the last name “Smith” and we would expect that proportion to be similar among users of ITR who report their last name (as most of ITR members are from US). To test this idea in our data, we used the 2014 census data for the proportion of the population with 1,000 most common first names and 1,000 most common last names in the US population [220]. For each name, we calculated the percentage of all ITR users who shared a name who reported this particular name. To compare the proportion of people with that name in the general population versus those

75 5.3. Results reported on ITR, we conducted paired Wilcoxon Rank-Sum tests (instead of paired t-tests as none of the populations was normally distributed) on both first names and last names. We also report the total proportion of the population accounted for by those names, both in the general population and on ITR. Table 5.3 reports these statistics. We found 1,000 most common last names accounted for fairly different proportions of the population in the general population and on ITR (43.37% and 28.23% respectively). We also found that the Wilcoxon Rank-Sum test of difference in proportion for each last name was statistically significantly different (p < 0.000). These results may suggest that a significant portion of ITR members who fill in the last name field actually use something other than their actual last names. Thus, while 1.3% of active users put something in the last name field, the percentage of people sharing their actual real last name is likely lower than even that 1.3%. This is confirming that the practice of sharing last names is uncommon in online recovery (just as it may be discouraged in face-to-face meetings). On the other hand, we found 1,000 most common first names accounted for very similar proportions of the population in the general population and on ITR (80.87% and 75.23% respectively). We also found that the Wilcoxon Rank-Sum test of difference in proportion for each name was not statistically significantly different (at a 0.05 level). While this cannot be interpreted to mean the two populations are the same, we know that the practice of using non-name slogans is not as prevalent for first names. There are two alternative explanations for the similarity of common first name proportions accounted for in the general and ITR populations: one may be that people are using common names as pseudonyms, while the other that people are actually using their real first names. While there are limitations to our approach, thereis more evidence to support the latter explanation. The evidence lies in the relative proportion of each name. Since the ten most common names are least identifiable, if people were using first name pseudonyms, we would expect to see inflated use of the top tennamesin

76 5.3. Results the ITR population. For example, Mary and John are two of the most common names in the US population, accounting for 1.32% and 1.64% of people respectively. If taking common names as pseudonyms was a frequent practice, these numbers should be inflated in the ITR population. However, instead these accounted for a fairly similar 0.57% and 1.55% of the ITR population respectively. We did not observe inflated percent for any of the ten most common names in the population, providing some degree of evidence against the possible explanation of first name pseudonym use. We tentatively propose that people inonline recovery use their real first name if they choose to share one. Analysis of “sharing face” behavior. Participants mentioned taking extra care to protect their anonymity as “unidentifiability” by not sharing information that could be connected to their identity (section 5.3.2). One example of such information that we can readily detect in traces of their online activity is whether they choose to use a photograph with a face as a profile image. We would expect that participants who worry about identifiability would choose one of the default avatars (ITR provides a library of dozens of such avatars) or select a non-face image (e.g., photo of a cat). We analyzed active ITR members’ profile pictures to see if they used avatars or photos not containing faces. We could differentiate a user-uploaded photo and a default avatar from the URLof the image, as all avatar profile pictures had a common prefix “/avatars” in their URLs.To differential between user-uploaded photos that show a face versus those that do not, weused the Face API from Microsoft Azure [157]. Our analysis confirmed that, almost half of active ITR members (N=13,455; 47.23%) used default avatars as their profile pictures. For an additional 30.22% of users the APIwas not able to detect any face, leaving only 6,425 (22.55%) active ITR users with profile photo containing faces (see Table 5.4). However, there are two limitations of this process: we assumed a picture detected with a face to be the user’s own photo, which might not be the

77 5.3. Results

Table 5.4: Comparison of Recovery Time, Time in ITR, Age, and Number of friends between the ITR population sharing and not sharing faces in profile images. A different N is provided for each variable since not every member shared the information of interest on his or her profile. We calculated “ITR time” from the first wall post from another member welcoming them to ITR.

case for every user, and the API may not be 100% accurate. Even with these limitations, it is safe to conclude that a substantial portion of ITR users did not associate their faces with their recovery profiles.

Relating “Unidentifiability” to Time in Recovery

Participants reported that behaviors of sharing last names or photos of their faces would break anonymity given the “Unidentifiability” interpretation. Our analysis confirmed that many members of ITR chose to reduce identifiability by not sharing this information. We sought to use this analysis to confirm or disconfirm a possible pattern we observed inthe qualitative data for a small set of 96 participants—people with more time in recovery may be less concerned with “Unidentifiability” and more concerned with other interpretations (section 5.3.3). If this is true, we would expect that ITR members who share their last names or faces to have more recovery time than those who do not. We can then examine how people who share these aspects versus those who don’t differ on other possible measures of temporal difference. For example, it’s possible that older (physical age) members care less about identifiability, or those who have been in the community for more time, orsimply those who are better connected to others within that community (as these are likely to vary

78 5.3. Results positively with recovery time). We compared all four of these variables (recovery time, time on ITR, age, and number of friends) between active members who shared their faces in their profile images and who did not, using Wilcoxon Rank-Sum test (as these distributions were not normal). Table 5.2 summarizes the result. We found out the two populations to be different at a 0.01 significance level in terms of time in recovery (p = 0.002), which supports the pattern found from our questionnaire analysis (section 5.3.3). We also found that active users sharing photos with face had significantly (at a 0.001 significance level) more time onITR(p < 0.000) and more friends on ITR (p < 0.000) than users who preferred not to share faces. However, there was no statistically significant difference in physical age between the two groups. Since section 5.3.4 indicated that a significant number of users adopted pseudonyms, slogans, etc. instead of real last names in their accounts, conducting a similar test for relating the variables to sharing last name behavior might not provide accurate results and hence, we exclude that from consideration. One other important caution in interpreting these results is that, they demonstrate a relationship rather than causality. For example, it is not clear whether people are more likely to share a photo of their face because they have more friends on ITR or whether they have more friends on ITR because they shared a photo of their face (or whether some third confounding variable is affecting both of these aspects). However, it is clear that there isa relationship between how embedded somebody is in the recovery community (i.e., time in recovery, time in community, and number of friends in that community) and whether or not they choose to enact their anonymity as “unidentifiability.”

79 5.4. Discussion

5.4 Discussion

In this section, we present insights from our analysis to consider in future research regarding anonymity, privacy, and self-disclosure, and in designing for online recovery communities and sensitive online contexts.

5.4.1 Implication for Research: Interpretation of Anonymity in Specific Context

Anonymity beyond “Affordance”

Research focused on online anonymity and self-disclosure should address anonymity beyond an affordance provided by online platforms or interfaces. Many 12-step community members in our study viewed anonymity as a mutual agreement and social equalizer, which represent anonymity’s socially constructed nature, beyond the concept of how much private they wanted to be online. Whereas many computing investigations have highlighted anonymity as an affordance, and discussed how not providing the choice of hiding one’s offline identity makes a site “anti-anonymous” [52], or how perceived anonymity influences people’s decision to use temporary identities online [128], there are online groups interacting with their face-to-face counterparts and interpretations of anonymity among the members of these groups may be constructed through their cultural or social practices from offline lives, and not merely by the features provided by the online interfaces theyuse[35]. Interpretations and enactment of, and expectations around anonymity may be different to the members of these communities. It is important to identify specific conceptions of anonymity adopted by the members of a community in order to decide how anonymity has to be af- forded in that particular context. For example, providing the option of not sharing real first name may not be as valuable to the members of online AA and NA groups as in other sensitive online communities, since our findings suggest that they use real first names online,

80 5.4. Discussion a trend possibly adopted from the physical AA meetings.

Anonymity beyond “Unidentifiability”

There is an additional need for research to understand anonymity beyond being unidentifiable or untraceable, as pointed out by our findings. The “unidentifiability” theme accounted for 37% of our questionnaire responses. It is closely related to interpretations of anonymity in many other social networks and online communities which allow for anonymity by providing choices of using pseudonyms [26, 103], temporary identities [57, 128], and anonymous networks [77]. Rest of the responses revealed other interpretations of anonymity (e.g. “social contract” and “program over individual”). Some aspects of these interpretations res- onate only with very recent HCI and CSCW research [202], and some others are community specific. Traditional anonymity preserving tools may not be able to meet the needs ofthe groups where anonymity means beyond “being untraceable”, pointing to the importance of context-specific research to understand anonymity.

5.4.2 Implication for Design: Supporting All or None, and Selective Disclosure

Future design in sensitive online spaces should consider supporting different levels of self- disclosure, as the decision of disclosing information in these communities may be influenced by different factors [141, 176]. Previous work focused on individual considerations of costs and benefits when making privacy decisions [79]. In contrast, our work foregrounded that individuals in AA and NA consider costs and benefits at a higher group level. Our participants considered how their self-disclosure behavior may affect the whole program, rather than only them individually, both in terms of costs (e.g., public aﬀiliation may harm the program) and benefits (e.g., greater self-disclosure may make it easier to help newcomers). Future research should continue to investigate the reasons affecting self-disclosure

81 5.4. Discussion decisions in different mutual health support groups, and the groups should allow for“allor none” or “selective” disclosure accordingly to balance the members’ privacy concerns and their needs of connecting with others. As our findings suggest, members are more likely to be less unidentifiable whenthey have spent more time and have more friends in the community. An implication for such communities is to provide users with the opportunity of “no disclosure” when they join the community and later by allowing them to “selectively disclose”. In fact, designing appropriate options for selective-disclosure can enable oldtimers to share information that newcomers will benefit from—through a system that asks newcomers to rate different pieces of information they find useful and later utilizes the highly rated information to helpthe oldtimers’ disclosure decisions. Even in the case of selective disclosure, it is important to make the members aware of the consequences they might not otherwise consider. Even though our participants decided with who and to what extent they self-disclose, the co-ownership of shared information can cause a “boundary turbulence” as discussed in Petronio’s Communication Privacy Manage- ment theory [179]. For example, if an individual (say P1) shares his recovery story with one of his friends (P2), and P2 shares this story with P3 who personally knows P1, but not his recovery status, P3 can possibly identify P1 from other information in the story. This is a breach of P1’s anonymity even though his name was not associated. A set of design interventions can be considered to prevent such turbulence. For example, a real-time detection technique can identify re-sharing of others’ private information from conversations. It can then discourage users by reminding them that the re-sharing behavior would violate the anonymity of the primary sharer and thus would go against the norm of the community. Such systems are diﬀicult to implement in practice and there are chances thatusers may consider such interventions as intrusions in their private conversations. Nonetheless,

82 5.4. Discussion designing and deploying such systems to support varying levels of disclosure may prove to be beneficial in sensitive contexts.

5.4.3 Implication for Design: Supporting Anonymity as an Equalizer

Our investigation points to the importance of considering anonymity as a social equalizer while designing for online peer support groups, and particularly for online recovery communities. Recently a fair amount of work in CSCW and CHI has considered designing better technology for these groups [19, 198, 241]. Interpretations of anonymity related to the program’s spiritual principles of humility and equality, as adopted by a significant portion of our participants, can inform some of those design decisions. Eighteen percent of our participants viewed anonymity as a social equalizer enabling them to obscure information that could cause bias or discrimination. While this finding is in a way consistent with prior work showing anonymity to have an equalizing effect due to absence of visual and social cues [13, 51, 106], most of these prior work focus on the equalization of gender or social status [64]. In communities following 12-step practices, equality has many more aspects than these two, as our findings imply. Equality to our participants meant equal rights for everyone, no leader or superstar, practice of humility in all activities, etc. Hence, although some features from other online groups may benefit users of 12-step communities, there is a need for careful consideration in following designs from those groups. For example, Wikipedia offers digital badges named as ‘barnstar’ to itsedi- tors having substantial contribution [232]. Similarly, some online physical activity games provide virtual rewards to its users for achieving particular goals [23, 24]. If similar rewarding practices are implemented in recovery communities, it would entitle some users as “recovery superstars” which would clearly go against the traditions of the program. Al- though rewarding has been proved beneficial to encourage participation and collaboration

83 5.5. Conclusion in many online groups [90, 163], AA and NA must decide on rewarding policies that uphold the norms regarding equality. This consideration applies for other design decisions as well.

5.5 Conclusion

Members of online recovery communities practicing 12-step fellowships like AA and NA offer an opportunity to provide a nuanced interpretation of anonymity. We conducteda mixed-method analysis to understand how AA and NA members interpret and enact online anonymity. First, through a secondary analysis of interviews from two previous studies and a qualitative investigation of a questionnaire we find that these members had three diverse interpretations of anonymity: “identifiability,” “social contract,” and “program over individual.” Second, we analyze traces of user behavior in a large online recovery community showing that, members with more recovery time, more time in the community, and more friends in that community are less likely to enact anonymity through “unidentifiability.” From our findings, we conclude that interpretations of anonymity insensitive online health communities can go beyond traditional computing definitions of anonymity as unidentifiability. Considering this, we provide implications for research on anonymity and self-disclosure, and discuss design implications in online recovery spaces and other sensitive online contexts. In summary, the two key insights from our formative investigations are to:

1. Focus on amplifying face-to-face support,

2. Design for “anonymity” as “spirital principle” of the program

In the following chapters, we present a design idea for people recovering from alcoholism and explain the technical challenges to implement it.

84 Chapter 6

Prototype Development of “AA Meeting Finder” through

HAIR

Drawing insights from the formative investigations, I came up for a possible design idea in the context of addiction recovery: a global self-updating meeting list for AA. An up-to- date comprehensive list of meetings can be useful in helping meet the following challenges frequently brought up by our study participants and in previous research work in this space:

1. In the interviews, participants (Chapter 4) mentioned that While new in recovery, or while traveling, they found it challenging to find out a meeting to attend. Indi- vidual AA groups value autonomy, and they typically operate at a local level (e.g. county, city, town, etc.). One consequence of this level of autonomy is inconsistency in how local groups make their information available and in how their meetings are catalogued. The current online structure of AA maintains meeting information in regional levels and there is no updated list of all the meetings happening throughout the United states. Outdated meeting lists from online search results and static apps often misdirected users to drive for hours to locations where there wasn’t any meeting, which may be very frustrating for newcomers. A global “self-updating” meeting list can help people recovering from substance use find more easily the peer-support they may desperately need.

85 2. Many participants in Chapter 5 interpreted “anonymity” as a “social contract” among members. As a part of enacting anonymity, they would share their personally identifiable information only with other members of recovery who go to the same in-person meetings. Moreover, participatory design with people recovering from substance use conducted by Zachary et al revealed that, sponsees would be interested in choosing sponsors from members who would go to the same meetings [204] Accurate time and location of the meetings are required to identify group of people attending the same meetings.

While a complete up-to-date meeting list is crucial in helping people find the peer- support they need, there isn’t any straightforward solution to develop this list. Since the regional AA websites keep meeting information (time and location) in different formats and structures, detection of webpages containing meetings and retrieving relevant information from those pages become extremely diﬀicult. Moreover, merely designing a static global meeting finder will not serve the purpose since it would soon become outdated whenever any of the websites updates any meeting details. Therefore, our goal is to solve this problem by retrieving meeting information from all the regional webpages, making them available in one place, and having a systematic way for the list to self-update automatically in a regular interval (i.e., to create a global “self-updating” meeting list). In this chapter, I approach the technical challenges in extracting meeting information from all regional websites to achieve this goal. Although there has been extensive research on information retrieval from heterogeneous unstructured sources, the automated techniques may fall short in this context due to the diversity in the format and structure of the regional websites. Additionally, retrieving complete information often requires contextual knowledge. Integration of human computation (crowdsourcing) with automated IR

86 6.1. Related Work approaches have produced better-quality results in knowledge acquisition tasks, and thus have the potential to be applied in solving the problem under consideration. In particular, I investigate the following research questions:

• RQ1: What are the information retrieval challenges of developing a global self- updating AA meeting list?

• RQ2: How can we combine human computation with existing IR algorithms to provide a more accurate and scalable solution to develop the list?

I answer these questions by proposing and evaluating a semi-automated Human-Aided Information Retrieval (HAIR) approach to information retrieval in community contexts like AA, that may empower the members of the community to work in concert with automated classification and retrieval algorithms.

6.1 Related Work

The process of retrieving meetings from regional AA websites included classification to identify webpages that contain meeting lists and information retrieval to extract meeting details. I incorporated human computation techniques with the automated processes of classification and information retrieval. In this section, I describe relevant research inthese three fields (classification, information retrieval, and human computation), and situate the opportunities for expanding on and combining these approaches in a single solution.

6.1.1 Classification of Webpages

The proposed method includes classifying a regional AA website page as a “meeting page” (a page containing information about one or more AA meeting) or a “non-meeting page” (a

87 6.1. Related Work page that does not). There have been extensive investigations in classifying different types of webpages using machine learning [158, 218]. Most common models built to identify, categorize, and filter information from webpages use supervised learning techniques (neural networks [14], support vector machines [102, 200], etc.). In service of better online search and web content management, researchers constructed features based on the text content of URL addresses [59], structure and content of webpages (e.g., [44, 192]), and aspects of neighboring pages [187, 213]. We primarily followed methods from the first two categories to generate features for our machine learning model, as those are more relevant in this context. For the classification task (i.e., distinguishing a webpage containing a meeting list from a pagethat does not) we applied Random Forest with Bagging, since research suggests that this might be the most accurate technique in this kind of imbalanced binary classification problems [2]. However, applying these techniques off-the-shelf is unlikely to provide expected accuracy in developing a self-updating global meeting list due to two reasons. The first reason is the diversity in the structure of local AA websites. There are many different examples of structures, such as regional AA domains having seven different meeting pages (one for each day of the week) that need to be classified correctly to get a complete set of meeting pages for that area. However, these pages may seem dissimilar when fed to a model for classification, since there may be a day with dozens of meetings versus another day with just one or two meetings. The second reason involves contextual challenges in identifying meeting pages. AA meetings are not the only type of information page that includes addresses and times 1 (e.g., oﬀice hours, non-recurring organizational meetings which are not open toall, etc.). Automated processes do not contain domain knowledge and may fail to classify these 1Example: http://www.utahaa.org/calendar/calendar.php

88 6.1. Related Work pages accurately.

6.1.2 Information Extraction from Unstructured and Semi-Structured Sources

To extract meeting information (e.g., day, time, address) from meeting pages, we built on prior work from Information Retrieval (IR), particularly pattern detection techniques. Re- gional pages vary drastically in the type and structure of meetings on them (see Figure 6.1 for examples), necessitating our work to be situated in research on IR techniques from heterogeneous unstructured sources. Previously researchers have used natural language processing (NLP) [82], pattern matching [154], and machine learning [41, 78, 130], etc. for extracting data from heterogeneous semi-structured or unstructured sources. Popular applications of these techniques, “named entity recognition” [159] and “temporal information extraction” [136] are relevant here, since we want to extract specific temporal entities (i.e., address, day, and time) associated with each AA meeting. Researchers have recently used natural language to extract temporal information (e.g., events) from unstructured text in social networking sites [192] or Wikipedia [122, 225]. We followed and extended a modified version of the algorithm developed by Chang et al. [41], in which the treelike structure of webpages and repetitive patterns were utilized to extract multiple records of similar type. The above techniques may be used as a potential solution for retrieving meeting information, however two challenges reduce the success of off-the-shelf solutions. The first challenge is in the diversity in the format structures of each “meeting page.” Meetings may be represented in different structures on different pages, such as an HTML table rowor combination of table rows, an event on a calendar, or even text on a scanned image. Auto- mated algorithms may struggle to find and relate the correct details to a specific meeting. The second challenge involves the role of context. Complete retrieval often depends on

89 6.1. Related Work knowing how the page was accessed. For instance, some meeting pages provide an option of a dynamic search; relevant information about a meeting (e.g., time, area, type, etc.) may come from a selected option of a dropdown list. Existing methods may fail to understand this context.

6.1.3 Combining Human and Machine Intelligence

Due to the limitations discussed in the previous subsections, we propose to combine human computation and machine intelligence to get a better result than either could achieve alone. To leverage the “wisdom of the crowd,”we suggest recruiting crowd workers from a crowdsourcing marketplace to verify both the classified webpages and the automatically extracted meeting information and to identify missing information [206]. Human computation is a computer science technique in which a computational process performs its function by out-sourcing certain steps to humans [239]. Recent research has been particularly interested in systems that leverage human and ma-chine intelligence independently and/or in sequence, combining results from both sources [209, 240]. For example in the field of machine learning, crowd workers have been employed to generate ground truth data and features and to validate results after classification [45, 47]. Alterna- tively, other systems have utilized machine learning and classification techniques to filter low quality crowdsourced data [135, 215, 235]. However, it is not until recently that crowdsourcing has been combined with machine computations to improve accuracy of tasks that are diﬀicult for either to perform independently [48]. This combination has been proved ef- ficient in solving problems like video-captioning [96], cheating detection [134], acquiring street-level accessibility information [89], and building intelligent environments [122]. Re- searchers have also considered utilizing crowdsourcing to improve results in solving problems like data wrangling [101], collating data from multiple sources [46], and knowledge

90 6.2. Methods acquisition [137, 240]. This process of pipelining computation results with human verifica- tion is similar to the concept of “centaur chess” where each human player reviews possible results of candidate moves generated by a machine intelligence “teammate” before making the final decision and selecting the team’s move[39]. Centaur chess teams outperform both humans and computers acting alone. The limitations of existing automated IR techniques make our problem a good fit for combining human computing with machine intelligence. Classifying webpages and retrieving information from probable meeting pages through algorithms may not always provide accurate and expected results, when such information is crucial. Moreover, incorporating human input with machine results has proven effective in cases where contextual knowledge is important. Therefore, I adopted and implemented HAIR: a human-aided information retrieval technique to address information retrieval challenges in the context of developing a global self-updating peer support meeting list for AA.

6.2 Methods

In this section, I describe the process of obtaining the initial dataset (i.e., pages from regional AA websites), followed by the two steps of HAIR to develop the meeting list: classification of pages and retrieval of meeting records from meeting pages.

6.2.1 Obtaining the Initial Webpages

The oﬀicial website of AA lists the homepage URLs for all regional AA domains bythe US state names [7]. Each of these domains usually consists of pages describing the AA program, weekly meetings in that region, non-recurring events, resources for newcomers in recovery, etc. I wrote a web scraping script using the Python Selenium library that:

91 6.2. Methods

1. Extracts homepage URLs of the regional websites

2. Recursively traverses through the links referenced by each homepage using depth- first search up to three levels. By manual inspection, I determined that meeting pages appeared within three levels of the homepage

3. Determines the MIME type for each of the pages and saves the URLs along with their types in a MySQL database

Following this method, I obtained over 10,000 URLs by scraping 385 domains. This number does not match with the total number of AA regions throughout the US, since 207 regions listed their hotline numbers only. After filtering and removal of duplicates, I ended up with 9468 pages in total.

6.2.2 Classification of Pages

This phase consists of automatic classification of pages as “meeting page” or “non-meeting page” using a machine learning classifier and integrating human computation with the output of the classification model.

Classification of Pages using Machine Learning

Since AA websites contain other resources for recovery, only a small subset of the extracted URLs would refer to a meeting list. In order to eﬀiciently reduce the number of pages to look for meeting records I developed a binary classification model. Ground Truth and Features. I selected a random subset containing 642 pages to train the classifier, making sure to pick at least one page from each regional domain. Eachpage was classified manually and independently by at least two researchers, with a 0 (non-meeting

92 6.2. Methods page) or a 1 (meeting page) value as the class. Cases of disagreement were discussed among researchers until consensus was reached. 452 and 190 pages were classified as non-meeting pages and meeting pages respectively. From our experience of manually classifying hundreds of pages, I developed a list of possible page features to use in the classification. The set of final features included:

• presence of the word “meeting” in the URL (e.g., http://aaminneapolis.org/ meetings)

• number of occurrences of word “meeting” in the text

• proportion of occurrences of text structured as a time

• proportion of occurrences of text structured as an address

• number of occurrences of weekday names in the text

I considered the proportions of days, times, and addressed since meeting pages often contain a larger number of times and addresses than the non-meeting pages regardless the format of the page. The proportions instead of raw numbers were considered since number of meetings on a page can vary from very few to a few hundreds. A Python script extracted these features from the 9468 pages and saved them in the database. I applied appropriate methods for scraping different formats of webpages based on the MIME type. We considered eight different formats: HTML, PDF, Google Calendar, Google Map, Document (.doc, .docx, .xls, or .xlsx), Google doc, Google sheet, and images. Classification. This is an imbalanced classification problem [131] and I determined that the recall of the model is more important to consider in this context than precision. Misclassifying a meeting page may result in multiple meetings missing in the final global

93 6.2. Methods list. If a meeting page is misclassified with a high confidence (more on section 6.2.2), this page will not further be sent for crowdsourcing and hence this error cannot be recovered in latter stages of the pipeline. On the other hand, a misclassified non-meeting page will be either sent to crowd workers (if confidence is low) or checked for presence of meetings in the next stage and eventually removed from consideration. Therefore, we cannot overlook the cost of misclassifying the meeting pages which are a small subset of the test instances. I tried out several classification algorithms [2] using the Python WEKA wrapper to determine which one performs better with respect to recall and overall F1 score, selecting random bagging with forest.

Integrating Human Computation

Determining Which Pages to Send to Crowd Workers. I used a prediction margin with each classified instance, rating the confidence of the classifier associated with that prediction. I selected a cutoff value where the set of instances below the margin were sent for crowdsourcing and the remaining were accepted as correct classification. I sorted the pages by prediction margin, inspected them to determine a threshold confidence that minimizes both the size of the set below the threshold (needing crowdsourcing) and the number of errors above the threshold (number of missed meetings), and selected the cutoff to be 90%. Interface of the Page Classification Task. I created a HIT (Human Intelligence Task) on Amazon Mechanical Turk (mTurk) [11], a crowdsourcing internet marketplace. The HIT description included instructions, examples of meeting and non-meeting pages, and a link to the web interface that we developed for the crowd workers to label meeting pages (see the interface for page classification task in Figure 6.2). The web interface was developed using the Python Flask framework. For each HIT, it showed 20 random pages sequentially and asked the worker if it was a meeting page with “yes”, “no”, and “not sure” options. If

94 6.2. Methods workers selected “not sure,” they had to answer two or three additional questions to help us determine the label. Recruiting Workers and Filtering Results. I conducted several pilot experiments (both with mTurkers and non-mTurkers who were undergraduate students) to refine the task design and description, to determine the pay per task, and to define a reasonable minimum time of task completion. Based on the number of pages needing to be crowdsourced (section 6.3.1), I had 971 assignments of the task. Each HIT had a payment of 50 cents to fulfill the minimum state wage requirement. We followed the majority voting scheme, with five different workers labeling each page to determine the final class [48]. For ensuring label quality, I used best practices from previous work [247]. We only accepted answers from workers with greater than 50% accuracy. Additionally, I considered the amount of time (a minimum of 120 seconds based on the pilot experiments) a worker took to complete a HIT to avoid answers from untrustworthy workers. I disregarded and reassigned 51 HITs that did not meet this requirement.

6.2.3 Retrieval of Meeting Information

This phase consists of an automated pattern detection approach to retrieve meeting records from the pages obtained from the previous phase and crowdsourcing to identify missed meetings if any and to validate the retrieved meetings. There is an absence of confident ground truth of meeting records for the global list, since there are about 60 thousand meetings overall. As a proof of concept, I applied the retrieval and crowdsourcing to the meeting pages from five different regional websites in Minnesota. The ground truth consisted of 1892 meetings from eight meeting pages in these domains.

95 6.2. Methods

Automated Approaches for Meeting Retrieval

I applied a pattern detection-based approach for identifying meeting location on meeting pages and a regular expression approach to extract corresponding meeting records. Identification of Meeting Location on Meeting Pages. I used a combination of pattern detection techniques from existing Python libraries and from prior literature. This step was necessary for pinpointing individual meeting records in a list of meetings. For the webpages identified as meeting pages, we applied repetitive pattern detection technique described in [41]. The HTML content of the webpages was represented as a tree of HTML tags and similar patterns were extracted. About 60% of the meeting pages were of this type. For PDFs I used the Python PDFMiner library to determine text segments along with their coordinates. Any other types of document files were converted to PDF and the same approach was followed. I utilized the Calendar API to extract multiple events from the meeting pages where a Google Calendar was located. I analyzed the image files using the Python Tesseract library and extracted co-ordinates of text segments along with their content. Extraction of Meeting Information. I checked for presence of regular expressions of time, day, or address in the repetitive patterns. If any of these aspects was missing from a pattern (e.g., the page may be accessed by selecting a particular day from a dropdown and I have to associate the day record with each of the meetings on the page), I searched that aspect in the siblings and then in the parent of that pattern recursively in the pattern tree. All events from Google Calendars were initially considered as meetings. The text segments were checked with regular expressions from PDFs and images. I considered the coordinates of the neighboring text segments for missing aspect and assigned it from the segment that is in nearest (Euclidean) distance. All records were saved in the database.

96 6.2. Methods

Integrating Human Computation

I integrated crowdsourcing to edit incorrectly fetched information by the automated approach (validation) and to retrieve meetings that were not detected (identification). I created two different HITs on mTurk and developed corresponding interfaces using the Python Flask framework. Meeting Validation. The interface sequentially showed 10 random meetings highlighted in yellow on corresponding webpages and asked the worker if it was a meeting record, with “yes” and “no” options (Figure 6.2). If workers selected “no” for a meeting, they were advanced to the next record. Otherwise, they were prompted to edit the automatically extracted details of the meeting if these did not match with highlighted record. Meeting Identification. The interface sequentially showed 20 random meeting pages with the retrieved meetings highlighted in yellow and asked the workers if they noticed any meeting as not highlighted, with “yes” and “no” options. If workers selected “no” for a page, they were advanced to the next page. Otherwise, they would be asked to select a rectangle around any unhighlighted single meeting. I then retrieved corresponding information from the answers and fed this record to the validation task. Therefore, the next worker who seeing this page would have this meeting highlighted. We continued the task until three workers had selected “no” for all the meeting pages, meaning all meetings on those pages were retrieved. Recruiting Workers and Filtering Results. After a round of pilot experiments we established the pay per task to be 60 cents for both types of HIT, and the minimum HIT completion time to accept an answer to be 150 seconds for the meeting validation HIT and 120 seconds for the meeting identification HIT. We had 410 assignments of the meeting validation HIT and 75 assignments for the meeting identification HIT. I followed majority voting

97 6.3. Results scheme where three different workers labeled each meeting. However, for edited meeting information from the validation task, I considered both majority votes and completeness (e.g., a meeting address with vs without a zip). I had to disregard and reassign answers from 20 validation HIT and 31 identification HIT that did not meet this requirement.

6.3 Results

In this section, I discuss the results of classification of pages and retrieval of meeting information and compare the results before and after enhancing with crowdsourcing. Addition- ally, I explain the results in terms of the effects in the context of AA.

6.3.1 Classification of Pages

The classifier reduced the number of pages to send to the crowd workers for meetingre- trieval, while crowdsourcing substantially improved the accuracy of the classifier. I classified 8826 test pages (since 642 of the 9468 pages were used as training instances) with the supervised machine learning model (ML). The classifier output 4268 pages as

“meeting pages” (with ≥90% confident about 699 pages). The accuracy of the model isnot as high as recall since we tried to maximize recall. Similarly, the classifier labels pages as non-meeting with more confidence so that there is a very low probability of misclassifying a meeting page (Table 6.2). Nonetheless, when I looked for sources of errors I noticed that many of the misclassified meeting pages had very few meeting records and the proportions of time and address present in the text content of those pages were lower compared to other meeting pages. Moreover, non-meeting pages listing events or other details that look like meetings (e.g., oﬀice hours) were misclassified as meeting pages.

98 6.3. Results

For enhancing the model with human computation (HC), I selected the 3569 meeting pages and 312 non-meeting pages for which the classifier’s confidence was less than 90%. I noticed that the “not sure” option was very infrequent in the answers and I did not observe any cases where I had to consider this option. After filtering the crowdsourced answers, the number of total meeting pages was reduced to 1275. Therefore, the workers labeled 576 pages out of 3881 pages as meeting pages. The performance of the combined approach showed significantly better results in terms of recall and F1 score (see Table 6.1). However, I found out that the list of meeting pages still included pages with non-recurring organizational event pages, or statistics. Possible reasons for this are workers not being enough careful, or their lack of contextual knowledge. There was a synergistic effect of combining the ML and HC. First, the automated classification reduced the number of pages needing human input from 8826 to 3881 (byabout 56%), thus reduced the cost of crowdsourcing. If we reduced the confidence threshold, there would be even fewer pages, however, we may have missed more meeting pages. Second, the ML only approach would result in misclassification of more than 30% of the pages, resulting in missing a relatively high number of meetings. Explanation in terms of the Context of Recovery

The ML only approach would have missed more than 800 meeting pages, completely making their meetings unavailable to people who need the help desperately. Although these pages usually consist of relatively small number of meetings, even pages with one meeting is important since it might be the only meeting happening in a rural area. Crowdsourcing helped identify many of these pages. On the other hand, many non-meeting pages were misclassified with ML only approach, potentially confusing, frustrating, and misdirecting people at a critical stage of their recovery. Crowdsourcing reduced this impact substantially by detecting 2993 such non-meeting pages.

99 6.3. Results

Table 6.1: Performance of the page classification

Table 6.2: Number of pages needing human input

Table 6.3: % (number) of meetings from the Minnesota domains

6.3.2 Retrieval of Meeting Information

I applied pattern-detection and regular-expression-based approaches to identify and retrieve meeting records from pages obtained from the previous phase. Crowdsourcing complemented the automated approaches by validating information and identifying missed records. Websites of the five regions from Minnesota had almost same variety in format ofpages as the set of all pages and thus can be considered as a representative sample. Page classification phase produced 11 meeting pages from these five. However, while extracting ground truth meetings, I discarded three pages that had zero meetings. The pattern detection approach (IR) was able to detect 1572 meetings (correctly or with partially correct information) out of 1892 meeting records. However, while looking for sources of probable errors, I noticed that in the cases of attaching information present in nearby co-ordinates or in a neighboring node in the HTML tree (e.g., meetings listed by days on a page) the pattern

100 6.3. Results detection often failed resulting in 16.9% of the meetings unidentified (Table 6.3). Due to the variety in representation of time, day, and address and due to meetings with incomplete address information or simply listing a church as the address, regular expression did not work accurately in every case, resulting the number of correctly identified meetings to be 1365. Although only 11% of the meetings fell in the category of partially identified, it is very high if considered in the context of recovery, which I describe shortly. Human computation (HC) substantially improved the percentage of correctly identified meetings by recognizing 282 additional meetings and reduced the percentage of meetings with wrong information by updating 146 meeting records (see the bottom row of Table 6.3). There was a synergistic effect of combining the two techniques. First, the results from the combined techniques could enhance the accuracy of the page classification step. For example, there were pages where no record was marked in the IR or HC as meeting information. Clearly these are misclassified meeting pages and there is a potential for feeding the pages back to the previous phase. Second, the IR only approach would have missed relatively high number of meetings and included many meetings with potentially wrong information. Explanation in terms of the Context of Recovery

The 3rd column of Table 6.3 may be considered as the most important metrics to eval- uate the proposed approach in terms of effect on people in recovery. For example, the unidentified meetings could include the only meeting or a very popular meeting inapartic- ular area and failure to find that is frustrating. However, the 2nd column refers to the fact that, after automated extraction about 11% meetings would consist of wrong information, creating confusion and potentially sending people to a location where there is no meeting. This result implies that a person suffering from alcoholism has a good chance of ending up in a wrong location regardless. The results from both phases are summarized in Figure 6.3

101 6.4. Discussion

6.4 Discussion

In this section, I conceptualize this work in a broader IR context and discuss future directions towards developing the global self-updating meeting list.

6.4.1 Leveraging the Specifics of Context

Prior work has investigated different IR techniques that consider human-in-the-loop (e.g., [46, 137]). However, most work takes a context-agnostic approach to include human computation and make decisions about information retrieval strategies. In contrast, recovery is a high-impact context where specific decisions and approaches may lead to significant positive or negative impact on people’s lives. We considered the context in: Prioritizing errors. The classification model was selected to prioritize recall over precision. This idea is espoused in many imbalanced classification problems particularly involving fraud detection (e.g., [182]). In the context of AA, obtaining all the meeting pages was essential, even though it added extra pages to send for human input. The latter phases were designed in a way that eventually discarded those misclassified non-meeting pages. Addi- tionally, I considered what the percentage of unidentified meetings and the percentage of meetings identified with partially correct information would mean to someone in recovery. The latter one (while conceptually “partially correct”) may have a bigger negative impact on members of AA, since it might send someone at a time or to a place where there is no meeting. That is why I took both into consideration and designed the “meeting validation” and the “meeting identification” tasks separately to ensure correctness of the extracted meetings as well as to minimize the number of unidentified meetings. Designing for eventual user value. My long-term goal is to provide substantial value to AA members. While in a less critical context, it would be reasonable to simply identify and

102 6.4. Discussion catalog local pages, that may not provide suﬀicient value for newcomers who may already be experiencing information overload [198, 195]. Our motivation for including the meeting retrieval phase is to produce a complete list of searchable meetings and to present them in a navigational interface that provides information in a more scalable, geographically, and temporally relevant way. Overall, this points to the idea of moving away from context-agnostic IR methods towards ones bespoke to work better in this specific high-impact context. Building on this design insight, I plan to involve people in recovery instead of mTurk workers for human computation. One significant source of error in page classification was both the classifier and the mTurkers failing to distinguish organizational (area) meetings and the actual weekly AA meetings. Since people in recovery are more familiar with the distinction between these two, involving them will potentially improve the performance of the crowdsourcing step.

6.4.2 Assumptions about Ground Truth

Currently, we assume that the meeting information on any regional website is accurate, while reality may not reflect this. One possible future direction to solve this problem is to extend the human-in-the loop part of the HAIR into the physical world. Although this dissertation does not focus on the sustainability of the developed meeting list (more in next chapter), we hope to develop a meeting finder app from the extracted meetings and make it public and available for use by people in recovery. We will then have opportunities to send the app users on physical world “mission” in their local area to identify and modify possible inaccuracies on regional pages. For instance, while periodically extracting meeting information, we may note that a page has not been updated in over six months, pointing to the possibility of the page being outdated. The app may ask users searching for meetings in that area to volunteer to validate information by going to a listed meeting and confirming

103 6.5. Conclusion if that is still existent. The idea of this future work implies that human-aided information retrieval can be implemented not only online but also in the real world.

6.5 Conclusion

Due to tradition of regional autonomy in its organizational structure, AA currently lacks a global or national meeting list. This makes it diﬀicult for newcomers in recovery or people traveling to a new area to find the support they need. With a goal of creating a navigational meeting finder app which includes all AA meetings throughout the United States, Ihave proposed HAIR: a semi-automated information retrieval approach that classifies webpages of different structures and formats and extracts meeting information from them combining machine learning, pattern detection, and crowdsourcing techniques. I conclude that crowdsourcing has potential to be applied in concert with IR techniques in this high-impact context, as it substantially improves the accuracy of the automated approaches by being able to identify many meeting pages and to modify wrong meeting information. Additionally, a major implication of my work is pointing to the importance of context-specific rather than context-agnostic semi-automated IR methods.

104 6.5. Conclusion

Figure 6.1: Different structures of meeting pages on different regional domains

(a) Meetings represented as markers on a map. Clicking on a marker provides detailed information about the meeting (b) Meeting list on a calendar

(c) List of meetings on a webpage (each meeting (d) Scanned image of a meeting list (uploaded as consists of multiple rows of an HTML table) an image on one of the regional websites)

105 6.5. Conclusion

Figure 6.2: Pipeline of HAIR (the page classification phase is on left, and the meeting information retrieval phase is on right)

106 6.5. Conclusion

Figure 6.3: Summary of results)

107 Chapter 7

Generic and Community-Situated Crowdsourcing in HAIR

To develop a centralized countrywide AA meeting list, I adopted a human-aided information retrieval approach for this purpose where automated machine learning and pattern detection approaches extract information about possible meetings listed on different AA websites. Then, these meeting webpages and meeting information were validated through paid human workers on an online crowdsourcing platform. We refer to this crowdsourcing technique, where a wide variety of “crowds” (without a particular set of assumed skills) can be recruited online or offline to complete tasks, as “generic crowdsourcing.” However, amajor source of error in this technique came from the workers providing poor quality answers and failing to identify webpages and content that provided information about other AA events rather than actual AA meetings. One reason of the poor performance of the generic crowd workers may be due to the lack of their contextual knowledge. Additionally, unlike non- members, AA and similar peer-support group members are motivated to perform service work to help other newcomers in the community [198]. Previous research has shown effectiveness of crowdsourcing with “right workers” (e.g., who are present at a particular location or time, have expertise in performing particular tasks, etc.) [18, 104, 56, 246]. Building and expanding on this idea, I utilized the context knowledge of AA members and their intrinsic motivation to help their peers in achieving better quality results in crowdsourcing meeting information. Here, we define the technique of seeking workers whose membership in particular communities of practice provides them with a unique perspective on relevant

108 background and norms of that community as “community-situated crowdsourcing”. Community-situated crowds can be recruited from different platforms and may work as paid or unpaid participants. In general, unpaid crowdsourcing leads to a potentially un- predictable workforce and indeterminable task completion time due to the lack of financial incentive[30]. In other words, there are performance trade-offs in crowdsourcing in terms of task completion time, cost, and accuracy, both while recruiting generic crowd workers vs. community members, and while recruiting community members through paid vs. unpaid platforms. I analyze these performance trade-offs by answering the following research questions:

• RQ1: How do community-situated crowd workers perform compared to generic crowd workers in terms of task completion time, accuracy, and cost?

• RQ2: What are the benefits and trade-offs of recruiting paid vs. unpaid community- situated crowd workers?

I approach these questions through an empirical comparison of generic crowdsourcing and community-situated crowdsourcing with workers from two different platforms in terms of time, accuracy and cost. I found that community-situated workers recruited from an online community can achieve better accuracy in AA meeting information identification and validation than generic crowd workers, though it may take substantially longer to recruit community-situated crowd workers, particularly if they are recruited as unpaid volunteers. Additionally, I found evidence in the data pointing out that the community-situated workers’ expertise and familiarity with the context may have helped them in achieving better accuracy. From these findings, I provide implications for effectively utilizing community-

109 7.1. Related Work situated crowdsourcing in this and other domains, as well as discuss the implications in the broader context of other crowdsourcing ICT systems.

7.1 Related Work

7.1.1 Situated Crowdsourcing

In this dissertation work, I refer to the crowdsourcing technique where a wide variety of “crowd” (without a particular set of assumed skills) can be recruited online or offline to complete tasks as “generic crowdsourcing.” In this case, crowd feedback is solicited from crowds driven by financial motivations [243]. However, crowd workers can also be motivated by intrinsic aspects other than monetary renumeration, such as enjoyment, or community based motivation [93, 109]. While online crowdsourcing markets make it convenient to pay for workers willing to solve a range of different tasks, they suffer from limitations like not attracting enough workers with desired background or skills [133, 9, 69]. For example, it can be a challenge to recruit workers for a task that requires workers speaking a specific language or living in a certain city [104, 10] . Situated crowdsourcing can help fill in the gaps in these scenarios where the crowd needs to be associated with some context. Situated crowdsourcing consists of embedding input mechanisms (e.g., public displays, tablets) into a physical space and leveraging users’ serendipitous availability [13] or idle time (“cognitive surplus”) [210]. It allows for a geo-fenced and more contextually controlled crowdsourcing environment, thus enabling targeting certain individuals, leveraging people’s local knowledge or cognitive states, or simply reaching an untapped source of potential workers [84, 85, 86, 92]. Researchers have discussed benefits of targeting geographically or temporally situated crowd over generic crowd in scenarios like providing emergency services in disasters [139],

110 7.1. Related Work geotagging photos [104], etc. Additionally, the idea of situated crowdsourcing can be extended to capture the context of selecting the “right crowd” who are better suited for a task (e.g., with a particular skill or quality), are more reliable, and can provide better quality results [53, 133]. Most prior work focuses on temporally-situated or location-situated crowdsourcing— seeking workers who are at a time or place to best be able to do the task. Building on these ideas, I coin the term “community-situated crowdsourcing”—seeking workers whose membership in particular communities of practice provides them with a unique perspective on relevant background and norms of that community. For example, members of 12-step communities like AA are familiar with the format of recovery meeting and governing structure meetings. A major source of error in HAIR came from workers being unable to distinguish area or district meetings from open weekly AA meetings. However, we can assume that actual members of this community would not make the same mistakes, thus being the “right crowd” with the required skill (familiarity with the context) for the tasks of 12-step meeting identification and validation. The primary contribution of this work is empirically comparing two alternative approaches for community-situated crowdsourcing. The secondary contribution is quantitatively establishing the benefits of community-situated crowdsourcing in the recovery contexts.

7.1.2 Community-Situated Crowdsourcing: Generic Platforms versus Targeted Online Communities

There may be multiple reasonable approaches for recruiting a community-situated crowd. For example, when using a generic platform like Amazon Mechanical Turk, membership in a particular community can be self-reported and can serve as a qualification for completing

111 7.1. Related Work the task. Alternatively, one may be able to solicit members of a desired group directly through online spaces dedicated to those communities (e.g., Facebook recovery groups, InTheRooms.com). Prior work has incorporated elements of both approaches, but without systematically separating and comparing the two. For example, friendsourcing leverages one’s social network to gather information that might be unavailable or less trustworthy if obtained from other sources [21, 25, 149, 199]. Researchers have attempted to build systems that, for example, use friendsourcing for social tagging of images and videos [25], to seek out personalized recommendations, opinions, or factual information [25, 160], etc., to help blind communities get answers to questions about the world around them captured by cameras [34], or to provide cognitive aids to people with dementia [149]. Friendsourcing removes the financial cost of the service and improves the quality and trustworthiness of the answers received [87, 161] . Several studies applying friendsourcing in health contexts (e.g., dementia, Alzheimer’s disease, etc.) suggest that friendsourcing can encourage engagement and provide more emotional and informational support compared to crowdsourcing [21]. Friendsourcing is a specific example of leveraging an existing known community and soliciting members through existing or novel online channels to complete desired tasks. As another example, altruistic crowdsourcing refers to cases where unpaid tasks are carried out by a large number of volunteer contributors [10]. This form of crowdsourcing often utilizes members of same community to complete collective tasks or getting better quality and more trustworthy information [10, 84]. However, some situations require altruistic contribution from out-group crowd, for example, in providing directions to people with visual impairments [33], or in generating valuable daily life advice for people with autism [91]. Altruistic crowdsourcing may both leverage existing social networks or may serve as a filter when seeing contributions from the larger community (in the sense that those willing

112 7.1. Related Work to complete unpaid tasks in a particular context are likely to have a personal connection or interests in that context). Both friendsourcing and altruistic crowdsourcing approaches rely heavily on social motivators such as social reciprocity, the practice of returning positive or negative actions in kind, reinforcing social bonds, the opportunity of showing off expertise, etc. [25, 53, 34]. Systems that utilize members of same community or known channels as the crowd often leverage their intrinsic interest in a particular domain and their sense of belonging to the community [108, 201]. This form of crowdsourcing have been proved to receive better quality feedback for class projects from classroom peers [227, 55]. Moreover, researchers have discussed the eﬀicacy of crowdsourcing peer-based altruistic support in critical contexts, such as to reduce depression and to promote engagement [162], for generating behavior change plans [4, 56], to exchange health information [194] , etc. Although friendsourced answers often contain personal or contextual information that improves their quality, in some cases people do not consider social network as an appropriate venue for asking questions due to high perceived social costs, limiting the potential benefit of friendsourcing [199, 21, 34]. This body of work provides compelling examples of benefits of community-situated crowdsourcing but is largely opportunistic about how such a situated crowd is recruited. Obviously, friendsourcing could not be reasonably accomplished through filtering workers on generic platforms. However, most other cases, both qualifying members on existing platforms and targeting specific online communities may be reasonable approaches and may have different costs and benefits. The primary contribution is providing an empirical comparison of accuracies, costs, and time trade-off in these two approaches to community- situated crowdsourcing, by comparing tasks completed by workers on MTurk who self- identify as members of 12-step programs and those done by member of the targeted online

113 7.2. Methods recovery community InTheRooms. Based on the findings, I provide recommendations for community-situated crowdsourcing in other contexts.

7.2 Methods

I conducted an experiment to understand the trade-offs of generic vs. community-situated crowdsourcing. For generic crowdsourcing, I recruited workers from Amazon Mechanical Turk (MTurk) and for community-situated crowdsourcing I solicited workers from both a filtered subset of MTurk and an online community for people in recovery. I performed statistical comparisons to understand the differences in performances of these three different populations.

7.2.1 AA Meeting Dataset

The essential steps in developing a comprehensive AA meeting list include identifying different regional webpages on different domains that contain one or multiple AA meetings, and locating all the meetings and their corresponding information (e.g., day, time, and address) on those meeting pages. As described in Chapter 6, I gathered ground truth data labels for 964 webpages, each with a label of 1 or 0 indicating whether it contains a list of meetings or not, and location of 1892 meetings from a subset of the meeting pages. This subset was selected from 18 regional websites from three different states in the USA. This was saved in a MySQL database on a secure server obtained from the Computer Science department.

7.2.2 Participants

Since I already had data for one of the three populations (generic MTurk) and I wanted to minimize the cost of recruiting workers, I utilized the saved data to perform a power analysis.

114 7.2. Methods

I found that for the statistical comparisons the desired sample sizes (the minimum number of participants) for the other two populations (community-situated MTurk and community- situated ITR workers) should be 400 to achieve a power of 0.8 at the 95% confidence level.

Generic Crowd Workers from MTurk

Amazon Mechanical Turk or MTurk1 is a popular online crowdsourcing marketplace where “requesters” can hire remotely located “crowd workers” (MTurkers) to perform discrete on-demand tasks known as Human Intelligence Tasks (HITs). The generic crowd sample in the previous work consisted of a total of 1060 individuals recruited from MTurk [196]. I created a HIT type of survey link where the purpose of the study was described briefly and a link of the website where they had to complete the tasks was provided. After they completed all three tasks, they were shown a unique survey code which they had to copy and paste on the MTurk platform so their responses could be tracked and associated with corresponding worker ids to approve or disapprove their HITs. Participants were recruited during the month of May 2018.

Community-situated Crowd Workers from MTurk

For recruiting MTurk workers who are practicing their recovery through one or more 12- step fellowships, I designed a screening survey and explicitly mentioned that only members of 12-step programs should attempt the questions. The screening survey consisted of three basic questions about recovery practices in 12-step fellowships (see below). 660 workers attempted the survey, with 480 of them answering all three questions correctly. We accepted all the HITs for the screening survey, but only these 480 workers were selected as “qualified” to perform the tasks. The actual HIT with the link to the tasks (described in Section 7.2.3) 1https://mturk.com

115 7.2. Methods was made visible only to the qualified workers. Participants were recruited during the month of December 2019. It is worth mentioning again that the term “community-situated crowdsourcing” is a shorthand for the process of recruiting workers who are members of particular groups or communities related to the context of the research questions. Thus all the community- situated workers in this study have self-reported to be members of 12-step fellowships. We cannot, however, make a certain claim about all of them actually being in recovery. Screening Survey Questions: The screening survey included the following three questions about recovery in 12-step fellowships (Appendix D.1):

• What is your primary 12-step fellowship?

• How many meetings do you attend weekly?

• Fill in the blank: The 12th tradition says, “______is the spiritual foundation of all our Traditions, ever reminding us to place principles before personalities.”

Ensuring Quality of the Workers: Previous work has pointed out that paid crowd workers recruited from online platforms often produce low quality results [80] to maximize their earnings by completing tasks quickly. In order to assure high quality recruitment of participants, we used several strategies. First, the recruitment was limited to MTurkers whose average HIT acceptance rate was 90% or higher. The second strategy was to embed a custom script on the HIT page that limits the number of times that a single worker may work on this study. By implementing this code, MTurkers could take part in this survey only once, reducing the effect of noise in the results from worker experience over repeated tasks. Lastly, all the response patterns were reviewed on a daily basis and all speeders and straight-liners were removed every day. For instance, if a respondent completed the survey

116 7.2. Methods too quickly (i.e., speeders who took less than five minutes in total starting from reading the consent form to finishing the last task) without carefully reading the question items, there- sponses from that respondent were excluded from the analysis. Also, if a respondent rushed through the tasks by clicking on the same response to multiple questions (i.e., straight-liners for all ten different answers for a particular task), those cases were also filtered outfromdata analysis. As a result of such data cleaning procedures, different participant answers were removed for different tasks, which resulted in an unequal number of participants forthree different tasks. For community-situated MTurkers, the sample sizes for page validation, meeting validation, and meeting identification were 423, 406, and 435 respectively.

Community-situated Crowd Workers from ITR

We worked with the InTheRooms website founders and owners to reach out directly to ITR members to participate in the study. A link to the survey was distributed as a paid banner advertisement on the ITR homepage for two months (October 2019-November 2019), as well as advertised in their weekly ITR newsletter with a message from the researchers explaining the purpose of the study (Figure 7.1). Similar to the workers from MTurk, all ITR members who responded to the advertisement viewed and agreed to an online informed consent. Again, the community-situated crowd workers from ITR were recruited based on the assumption that the online community members are in fact 12-step fellowship members, which may not be true for all members of ITR, since previous work has established that online communities may have spammers or non-members [219, 76]. However, we did not ask the filtering questions to the ITR members and we assume that this difference inscreen- ing did not generate considerable biases to the results based on a few assumptions from our research experience with this community for past couple years. First, ITR is primar-

117 7.2. Methods

(a) Study advertisement on ITR homepage

(b) Study advertisement on ITR newsletter Figure 7.1: Recruiting workers from InTheRooms through advertisement ily an online peer support network for 12-step fellowship members and their friends and families. A spammer in this community does not have much to gain from frequent interactions with others. Second, even if we assume that there is a considerable number of non-12 step fellowship members in the community, the probability of them responding to a study without any monetary reward should be very low. Finally, since we were not providing any

118 7.2. Methods reward to the ITR members for participating in the survey, providing additional questions for screening might have reduced the turnover of the responses, increasing the time and the cost of recruitment. In addition, on MTurk, since the questions in the screening survey were relatively basic, workers who were not actual 12-step members might have managed to answer them correctly. I took several additional steps to reduce this effect, if present, and to ensure worker quality, as described in the previous subsection. I believe that these filtering techniques should considerably mitigate the impact of asking not so hard screening questions to the MTurk workers and of not asking the screening questions to the ITR workers on the study outcome.

Ethical Considerations

Recovery from substance use is a personal and private undertaking for most people and we took steps to consider the ethical implications of this work and to protect the rights of the community-situated workers who are members of different 12-step programs. All research activity on this project was reviewed and approved by University if Minnesota IRB as an investigation where potential benefits of the scientific work outweighed the risksto the participants. The approval covered two phases. First, all users who completed the tasks viewed and responded to an online informed consent form, but documentation of informed consent was waived in order to preserve participant anonymity. Second, I made sure that the screening survey and the tasks do not ask the participants for any personally identifiable information.

7.2.3 Tasks

I created responsive web interfaces for each task using the Python Flask framework. All workers were assigned to complete all three tasks, but I randomized the order of the tasks

119 7.2. Methods

(a)

(b) 120 7.2. Methods

(c) Figure 7.2: Interfaces of the three tasks (a) meeting page validation, (b) meeting validation, and (c) meeting identification shown to them to reduce the learning effect on the results. From both MTurk and ITR platforms, they were redirected to the task website and were shown a consent form. Doc- umentation of informed consent was waived in order to preserve participant anonymity. After they agreed to participate in the study, I showed them brief descriptions for each of the tasks. Prior to starting each task, there was a popup page with detailed instructions and examples (See Appendix D.2).

Meeting Page Validation

The interface for page validation showed 10 webpages (selected randomly from the ground truth data set of pages) sequentially and asked the worker if it was a meeting page with “yes”, “no”, and “not sure” options. If workers selected “not sure,” they had to answer two or three additional questions to help us determine the label (Fig. 7.2a).

121 7.2. Methods

Meeting Information Validation

The interface sequentially showed 10 meetings (selected randomly from the ground truth data set of meetings) highlighted in yellow on corresponding webpages and asked the worker if it was a meeting record, with “yes” and “no” options (Fig. 7.2b). If workers selected “no” for a meeting, they were advanced to the next record. Otherwise, they were prompted to edit the automatically extracted details of the meeting if these did not match with the highlighted record.

Meeting Information Identification

The interface sequentially showed 10 meeting pages (selected randomly from the ground truth data set of pages) with the retrieved meetings highlighted in yellow and asked the workers if they noticed any meeting as not highlighted, with “yes” and “no” options. If workers selected “no” for a page, they were advanced to the next page. Otherwise, they would be asked to draw a rectangle around any one unhighlighted meeting (Fig. 7.2c). Prior to recruitment, I conducted several pilot experiments: two on MTurk and two with undergraduate volunteers, to refine the interfaces of the tasks and task instructions and descriptions, to determine the appropriate pay per worker ($2.5), and to define a reasonable minimum time of task completion (five minutes).

7.2.4 Platform Cost

The cost of task completion for different worker types is the total cost paid to the workers only for completing the tasks, since this fraction of the total cost would more likely influence the workers’ task completion time and accuracy. However, for this study, we had to pay additional costs for recruiting and filtering workers. In the interest of keeping the dis-

122 7.2. Methods cussion about cost of the task completion consistent, we separate the platform cost that was associated with the use of particular crowdsourcing platforms to recruit workers, from the actual worker cost that was the amount of money that went to the workers who completed all the tasks. However, I acknowledge that in reality the required number of participants might not have been obtained without the advertising that caused the platform cost for recruiting the community-situated crowd. Therefore, this chapter also discusses the total and average worker cost including all the expenses spent for the study (Section 7.3.1). On MTurk, the price a requester has to pay for a HIT is comprised of two components: the amount to pay workers plus a fee to pay MTurk. The usual MTurk fee is 20% of the worker fee for any HIT. For each added worker qualification in this study (i.e., workers with greater than 90% HIT acceptance and workers with correct answers on the screening survey) an additional 5% of the worker fees was added to the platform cost. For generic crowdsourcing on MTurk, the researchers paid a total platform cost of $543.75 (25% of the total worker cost described in the Results Section). This was a sum of the usual 20% of the worker cost and the 5% fee for an additional criterion of making the HIT visible only to workers having average HIT acceptance rate of 90% or higher. For community-situated crowdsourcing on MTurk, the platform cost was a total of $274.95, including the MTurk fees of $4.95 for the screening survey and $270 for the actual HIT. The additional cost of the workers filling out the screening survey could be reduced by including the survey as a part of the actual tasks, and then considering the HITs from only those workers with correct answers to the survey questions. However, that might not be a good idea for this research, assuming that plenty of workers who are not members of a 12- step fellowship would attempt the HITs but would get rejected due to low quality answers, increasing both worker dissatisfaction and our labor for data management and filtering. Con- sidering the small amount of money and design effort introduced by the screening survey,

123 7.2. Methods we decided to keep it separate from the actual HIT. We paid $6000 to ITR to run the advertisement for two months. In addition to putting up the banner on their homepage and on the website’s weekly newsletters, the website founders guided us in improving the banner design and the task interfaces to attract more participants, and provided continuous support in recruiting participants through batch emailing community members about the research asking for their participation. For other domains, however, the platform cost would largely depend on the type of platform to recruit participants from and would vary based on the type of the tasks in question and the required community to perform them.

7.2.5 Measures

For comparing the performance of different types of crowdsourcing, the variables I calculated and the statistical tests I conducted are as follows:

• Cost: I calculated unit and total costs paid to the workers and to the platforms.

• Time of recruitment: We report the amount of time required to recruit the required number of valid participants from each platform.

• Average accuracy and average task completion time: For the three different groups of workers, generic crowd, community-situated crowd from MTurk, and community- situated crowd from ITR, I at first conducted Kruskal-Wallis tests for mean accuracy and mean completion time for page validation, meeting validation, and meeting identification tasks to find out if there was a difference between at least one groupandthe other groups of workers. Afterwards, I performed between-subjects Wilcoxon Rank- Sum Tests (as some of those distributions were not normal) for only the tasks that had p-values less than 0.05 from the Kruskal-Wallis tests to calculate the difference

124 7.3. Results

in mean accuracy and mean completion time for the three pairs: generic crowd and community-situated crowd from MTurk, generic crowd from MTurk and community- situated crowd from ITR, and community situated crowd from MTurk and ITR. To

account for multiple comparisons, I adjusted the 푝-values using the Holm-Bonferroni method. Since the type of data to identify and validate was different in each task, I calculate the accuracy differently.

# of pages correctly answered with yes/no Accuracy = 푝푎푔푒_푣푎푙푖푑푎푡푖표푛 total # of pages validated with yes/no

# of meetings correctly validated (labeled & edited) Accuracy = 푚푒푒푡푖푛푔_푣푎푙푖푑푎푡푖표푛 total # of meetings attempted

# of meetings correctly identified Accuracy = 푚푒푒푡푖푛푔_푖푑푒푛푡푖푓 푖푐푎푡푖표푛 total # of meeting pages attempted

• To compare performance for specific portions of the tasks requiring workers’ expertise that come from membership in the community, I conducted appropriate statistical analyses. For instance, to calculate the difference in average number of meetings validated with a “not sure” option, I conducted a Chi-Square test, where the categories were represented as whether a particular page was classified with a “not sure” option by a particular type of crowd worker.

7.3 Results

I analyzed the performance of generic crowdsourcing and community-situated crowdsourcing with workers from two different platforms and discuss the trade-offs in this sectionin terms of recruitment time and cost, task completion time, and accuracy of the workers.

125 7.3. Results

Worker cost only Total worker cost including platform cost (average cost per worker) (average cost per worker) Generic: MTurk $2175 ($2.5) $2718.75 ($3.125) Community-situated: MTurk $1125 ($2.5) + $19.80 ($0.03) = $1099.80 $1431 ($3.18) Community-situated: ITR $0 ($0) $6000 ($13.33) Table 7.1: Worker costs and total costs for generic and community-situated crowdsourcing. Worker cost is the portion of the total cost paid to only the workers for their task completion, and not to the platform.

Additionally, we were interested in understanding if community-situated crowd workers performed better in detecting and validating particular types of information due to their familiarity with the context.

7.3.1 Cost of the Workers

As mentioned earlier, worker cost is the portion of the total cost that went to the workers, and not to the platform. For the actual HIT to complete the tasks, $2.50 was paid to each MTurk worker. For generic crowdsourcing in the previous work, 870 HITs were accepted and we accepted 426 HITs from the community-situated crowd answers. Since the total number of participants in these two samples are different and there was an additional cost of $19.80 for the 660 accepted HITs of the screening survey in the MTurk community-situated crowdsourcing, there is a difference in the total worker cost between generic workers and community-situated MTurk workers. The total worker cost is $2175 ($2.50 per worker) and $1099.80 ( $2.57 per worker) for generic MTurk and community-situated MTurk workers respectively. We did not pay any monetary rewards to the ITR workers. Therefore, considering the average cost per worker, community-situated crowdsourcing cost the least (zero), followed by generic crowdsourcing from MTurk, and community-situated crowdsourcing from MTurk respectively. Table 7.1 reports these worker costs. Ideally, if the same number of paid generic and community-situated crowd workers are

126 7.3. Results

Time Accuracy <.001 <.001 Page validation *** *** <.001 <.001 Meeting validation *** *** .003 Meeting identification .172 ** Table 7.2: 푝-values from the Kruskal-Wallis tests for the differences in task completion time and accuracy among the three tasks recruited to perform the same number of tasks, the cost would be about the same. In contrast to the paid community-situated MTurkers, the community-situated worker cost can potentially be very low or even zero, irrespective of the number of recruited workers, if we choose to recruit volunteers from online groups/communities relevant to the context in question. However, these calculations disregard the platform costs which can significantly increase the average cost per worker for community-situated workers in many studies similar to this one. I discuss the total cost and the average worker cost including all platform costs in the next subsection.

Total Cost

Even though in this study the platform cost ideally did not impact the workers’ task completion time and accuracy, many studies may require high advertising costs to ensure enough participation and/or to obtain quality results. The third column of Table 7.1 represents the total and average costs including both worker and platform costs. The average cost per worker for generic and community-situated crowdsourcing were $3.125 and $3.18 respectively. The additional average worker cost for community-situated MTurk is due to the HIT used for filtering workers. The average worker cost for ITR participants was $13.33 con-

127 7.3. Results sidering the platform cost, which is very expensive compared to the other two populations. Including platform cost in fact reorders the worker cost by type required for this study. How- ever, for tasks related to other contexts/domains, or even in the context of AA, the platform cost can be reduced by recruiting workers from other platforms with large user base. I return to this point in Section 7.4.3.

7.3.2 Time to Recruit Workers

MTurk and ITR both have over 500,000 members, although not all of them are active [168]. Paid online platforms like MTurk have been shown to be an effective way of quickly recruiting workers (both experts and novices) [31]. I found evidence for this in our study, as it took the least time to recruit generic workers from MTurk. For generic crowdsourcing, the researchers aimed to recruit participants until they achieved validation results for a predetermined number of meeting pages and meetings [196]. They published HITs in batches and received the desired number of responses in about a week (푛 = 1060). For community-situated MTurkers, it took two weeks to recruit 450 participants, starting from the time the screening survey HIT was published. For the ITR advertisement, I achieved a total of 460 complete responses which was about the same number as the targeted number of participants for this comparison. It took, however, additional time for us prior to recruitment to negotiate with the website founders about the logistics of the advertisement. Therefore, time to recruit generic crowd workers was substantially less than both types of community-situated workers. Additionally, recruitment time of unpaid volunteers from ITR was about four times slower than paid community-situated workers on MTurk. In general for other domains, recruiting volunteers from online communities might consist of similar steps and might take substantially more time than recruiting paid community-

128 7.3. Results

Page validation Meeting validation Meeting identification M SD M SD M SD Generic: MTtuk 237.61 162.11 318.37 255.50 268.50 212.78 Community: MTurk 315.12 181.70 555.73 332.46 309.49 313.62 Community: ITR 343.23 265.59 507.27 360.37 329.26 316.08 Table 7.3: Means and standard deviations of task completion time (in seconds) of generic and community-situated crowd workers for three different tasks

Page validation Meeting validation Meeting identification Time Accuracy Time Accuracy Time Accuracy <.001 .021 <.001 .002 Generic and community: MTtuk .18 .15 *** * *** ** <.001 <.001 <.001 <.001 .030 Generic and community: ITR .15 *** *** *** *** * <.001 Community: MTurk and community: ITR .97 .163 .051 .25 .87 *** Table 7.4: Statistical comparisons of average task completion time and accuracy between: 1) Generic crowdsourcing and community-situated crowdsourcing from MTurk, 2) Generic crowdsourcing and community-situated crowdsourcing from MTurk, and 3) Community- situated crowdsourcing from MTurk and ITR. 푝-values are reported using Wilcoxon Rank- Sum Tests and adjustment with Holm-Bonferroni method. (* 푝 < 0.05, ** 푝 < 0.01, *** 푝 < 0.001) situated workers. On the contrary, depending on the type of data validation, other platforms with more volunteers (e.g., Facebook groups) may be leveraged to reduce this time. There- fore, the researchers/recruiters have to consider the trade-off between time and number of participants for different platforms based on the emergency and frequency of the crowdsourcing task in question (more about this in the discussion section). % starting tables for time comparison

7.3.3 Task Completion Accuracy of Workers

ITR members completed the tasks with highest accuracy regardless of the type of the task, followed by community-situated MTurkers and generic MTurkers, respectively (refer to the

129 7.3. Results

(a) (b) (c) Figure 7.3: Boxplots of accuracy achieved by three different types of crowd workers for: (a) page validation, (b) meeting validation, and (c) meeting identification boxplots in Fig. 7.3). Kruskal-Wallis tests showed significant difference in mean accuracy except for the task of meeting identification (Table 7.2). For page validation and meeting validation, the differences between generic crowd workers and the community-situated crowd workers were statistically significant at the 95% confidence level. Additionally, mean accuracy of the community-situated workers from ITR was significantly higher for page validation than community-situated workers from MTurk. For meeting validation, accuracy of both types of community-situated workers were significantly higher than generic crowd workers. I discuss these accuracy and their interpretations in more details in the following subsections.

Page Validation

The mean accuracy of community-situated crowdsourcing from ITR (푀 = 72.1%, 푆퐷 = 19.18%) was significantly higher than both the mean accuracy of community-situated crowdsourcing from MTurk (푀 = 60.52%, 푆퐷 = 18.95%)(푝 < .0001) and generic crowdsourcing from MTurk (푀 = 56.40%, 푆퐷 = 16.24%)(푝 < .0001) (Fig. 7.3a). This dif-

130 7.3. Results ference in accuracy in the context of creating an AA meeting list means that recruiting from community-situated sites can increase the accuracy of meeting page validation by 16% compared to a generic platform like MTurk, resulting in correct label for about one in six additional meeting pages in the data set.

Meeting Validation

Similar to the task of page validation, the highest mean accuracy (푀 = 76.94%, 푆퐷 = 17.33%) was achieved by the community-situated workers from ITR, followed by community- situated MTurkers (푀 = 73.54%, 푆퐷 = 15.45%) and generic MTurkers (푀 = 71.54%, 푆퐷 = 20.51%). This accuracy was significantly higher than generic crowdsourcing from MTurk (푝 < .0001) (Fig. 7.3b). In addition, community-situated MTurkers’ accuracy was significantly higher (푝=.002) than the accuracy of generic MTurkers. In the context of creating an AA meeting list, this finding implies that the ITR members can provide accurate day, time, and address information for about 5% more meetings than the generic crowd workers. AA hosts more than 50,000 meetings throughout the United States and community-situated volunteers can substantially improve the reliability of the meeting list by validating these meetings accurately.

Meeting Identification

Community situated ITR workers achieved the highest mean accuracy of meeting identification task (푀 = 81.80%, 푆퐷 = 21.24%), followed by community-situated MTurk workers (푀 = 81.36%, 푆퐷 = 25.39%) and generic MTurk workers (푀 = 79.76%, 푆퐷 = 23.36%) respectively (Fig. 7.3c). There was no significant difference in accuracy between any ofthe three pairs. I assume that, since meeting identification involved only segmenting a part ofa webpage if there was a meeting that was not highlighted (i.e., drawing a rectangle around an

131 7.3. Results

Figure 7.4: Different types of errors in the page validation and the meeting validation tasks; the first three groups from the left respectively show the average accuracy of editingthe day, time, and address information correctly in the meeting validation task. The rightmost group shows the the average percentage (and the differences between groups) of selecting the “not sure” option in the page validation task unhighlighted meeting record), it is comparable to the task of segmenting images for object detection and is a common type of HIT on crowdsourcing platforms. In fact, generic crowd workers completed this task with comparable accuracy in less time on average.

7.3.4 Evidence for Reasons of Difference in Accuracy

We were interested to further investigate the possible reasons for the significant difference in accuracy between community-situated crowdsourcing and generic crowdsourcing. Through post-hoc analyses and a closer look into the data, I found out that this difference in accuracy could be contributed by one or more of the following factors:

132 7.3. Results

Task Completion Time of Workers

The average time taken to complete the tasks by generic MTurkers (page validation: 푀 = 237.61푠, 푆퐷 = 162.11푠, meeting validation: 푀 = 318.37푠, 푆퐷 = 255.50푠, meeting identification: 푀 = 268.50푠, 푆퐷 = 212.78푠) was less than both the community-situated MTurkers (page validation: 푀 = 315.12푠, 푆퐷 = 181.70푠, meeting validation: 푀 = 555.73푠, 푆퐷 = 332.46푠, meeting identification: 푀 = 309.49푠, 푆퐷 = 313.62푠), and the ITR workers (page validation: 푀 = 343.23푠, 푆퐷 = 265.59푠, meeting validation: 푀 = 555.73푠, 푆퐷 = 332.46푠, meeting identification: 푀 = 329.26푠, 푆퐷 = 316.08푠). Generic crowd workers completed the tasks of page validation and meeting validation significantly faster than both the paid (푝 < .001) and unpaid (푝 < .001) community-situated crowds. For meeting identification, this difference was not significant. There was nosig- nificant difference in completion time of any of the three tasks between paid andunpaid community-situated worker samples (see Time columns in Table 7.4). In terms of average completion time by type of task, page validation took the least amount of time for all types of workers and meeting validation took the most. This is expected, since for validating a meeting workers often have to scroll through the webpage shown to them to confirm that the time and location of the highlighted meeting is accurate, and edit the information if necessary. On the other hand, for validating a meeting page they mostly need to skim through the page shown to them for a list of meetings and determine if those are open AA meetings. Unlike editing and typing texts in meeting validation task, for page validation they had to answer one (a maximum of 2-3 if they selected “not sure”) multiple choice question. These results indicate that workers being paid for task completion were probably performing the tasks very quickly and paying minimal attention to each one, causing the lowest

133 7.3. Results accuracy for generic MTurkers. This finding also resonates with results from many previous studies on paid crowdsourcing [145, 31]. For the community-situated crowd samples, volunteers from ITR usually took slightly more time on average than the paid MTurkers. This could probably be attributed to the different motivations of the community-situated crowd from these platforms: For ITR members the only reason of participation was their personal motivations to be of service to the community, whereas MTurk members were both members of 12-step fellowships (supposedly) who wanted to help other community members, and also belonged to online crowd workers seeking to maximize their earnings through completing HITs quickly.

Knowing When “Not Sure”

I calculated the reported accuracy of page validation considering only those workers who were certain in labeling most of the webpages shown to them. To be more specific, if a worker selected “not sure” for more than three pages, I did not include his/her response to calculate the accuracy for this particular task. I assumed that the accuracy difference may occur partly because of the willingness of different types of workers in choosing the “not sure” option when they were actually not sure. To find out if this assumption was true, we calculated the average number of pages where workers selected the “not sure” option, and this was highest for the ITR members (푀 = 9.48%) and lowest for the generic MTurkers (푀 = 2.1%) (Fig. 7.4). This measure was calculated by counting the number of pages where not sure option was selected by a particular type of workers and dividing that number by the total number of pages shown to all workers of that type. Since ITR workers achieved the highest average accuracy for page detection, we assume that the community members are more confident of what they do not know and they did not want to provide incorrect answers (more on Section 7.4.3). A Chi-Square test showed that this number was significantly higher

134 7.3. Results for ITR than generic MTurkers (푝 = .002). In the cases where workers were not sure, they could choose to answer a 2-3 additional questions that could guide us in determining the probable label of the webpage. Through an analysis of the answers to those additional questions I identified that ITR workers were more willing and accurate in answering those additional questions. In other words, to calculate average accuracy, if we consider the pages where we can obtain a label through workers’ answers to additional questions, we get a 1% increase in the accuracy for ITR workers, whereas the accuracy of generic and community-situated MTurkers do not change at all.

Better Context Interpretations

We suspected that the community members could perform better in differentiating meeting pages from other pages due to their background knowledge of the AA program’s structures and norms. For example, the findings from the previous work with generic crowdsourcing suggested for page validation that, many generic workers labeled webpages with events or office hours (that have times and locations) inaccurately as meeting pages[196]. We assumed that the community-situated workers would label pages more accurately in these cases. However, we did not have explicit ground truth data for a set of pages that listed events or office hours, and therefore, we cannot claim that ITR workers were significantly more accurate in differentiating meeting and event pages than other types for workers. How- ever, I manually checked some of the pages that were labeled differently by generic and community-situated workers and found that some event pages were correctly classified as non-meeting pages by community-situated workers only, possibly contributing to a higher accuracy of the ITR workers. Similarly, based on my observation, community-situated workers performed better than generic crowd workers in differentiating single event information (e.g., day, time, address of picnics or service meetings, office hours etc.) from

135 7.4. Discussion meeting information. For the task of meeting validation, total accuracy of a worker was attributed to being able to identify whether the highlighted information is a single meeting (i.e., it includes only one day, one time, and one address of an AA meeting) and to edit the day, time, and address accurately to match the ground truth. Due to better context interpretations of the community-situated workers, we assumed that there may be difference in how different types of workers edit different information. Fig. 7.4 shows the average percentage of day, time, and address information edited by the workers. The higher accuracy of the community workers than the generic workers in editing the day information might occur due to workers being more familiar with the structure of the webpages and taking more time to skim through the pages. This is because from observations, I had seen many meeting websites that listed the meetings per day of the week. For these types of pages, workers had to scroll through the top of the page to edit the day information.

7.4 Discussion

The implications of these results point to next steps in my research on providing a reliable information source for meetings for people in recovery. In addition, I consider and discuss practical implications for technology design given the impacts of worker type on the quality of outcomes of data validation, as well as the research opportunities for the CSCW and the Human Computation domains revealed by this study.

7.4.1 Implications for the Context of Recovery

The investigations point to the importance of considering the trade-offs of time, cost, and accuracy while adapting crowdsourcing for AA meeting data validation. Finding a meeting

136 7.4. Discussion is currently a challenge for many AA members, as a reliable list of meetings in an area largely depends on local AA groups continually updating changes to meetings, which makes the current meeting finders outdated for different regions. Crowdsourcing, when combined with automated information retrieval techniques, can help periodically extract and validate meeting information from different websites and provide a reliable up-to-date source of information. I used the context of recovery as an example to broadly answer the research questions regarding community-situated crowdsourcing applicable to other domains as well. Consequently, the results had many interesting implications for the next step in utilizing crowdsourcing effectively in this particular domain. Community-situated workers from ITR achieved the highest accuracy for most of the tasks (including the comparatively longer task of meeting validation) among the three different worker samples. This finding implies that we can rely on volunteers for data validation for this particular problem. This resonates with the findings of many other studies showing that community-based crowdsourcing results in more accurate and reliable outcomes [citizenscience, 245, 88]. Keeping the meeting lists up-to-date would require help from workers in periodic intervals and community-situated crowdsourcing can ensure a stream of continuous workers, potentially reducing the worker cost to zero or close to that. This particular study caused us a higher platform cost for recruiting ITR members than both generic and community-situated MTurkers. Although the platform cost does not impact worker performance directly, this is part of the total cost required for the study. This platform cost can be significantly reduced both by using other online platforms and apps, andby developing our own system to recruit community members. For example, with the meeting data initially extracted and validated, we can build an up-to-date meeting finder application targeted for AA members. When the app is in use, we can potentially ask users of the app to volunteer in helping us improve the meeting list through validating meeting information.

137 7.4. Discussion

However, relying on a system or app in its initial phase with a small number of users may be questionable to recruit volunteers, since depending on the volume of data requiring validation, recruiting the desired number of participants can take a substantial amount of time. For example, we recruited from an online community with more than 500,000 members, but the recruitment time was about two months for 450 participants. As a counter-argument, this study limited the participants to perform the tasks only once in order to reduce the learning by repetition effect, whereas in a real-time system, each worker would be ableto perform the tasks as many times as possible, minimizing the trade-off in recruitment time, and improving worker expertise at the same time. In summary, volunteers from recovery communities can be relied upon for validating AA meeting webpages and meetings, and the cost of recruitment can be substantially reduced by careful consideration in adapting different approaches to attract more volunteers.

7.4.2 Implications for Design of Crowdsourcing ICT Systems

These findings suggest that many design decisions of crowdsourcing ICT systems, suchas the task interfaces, the choice of platform, etc. should consider the trade-offs in cost and accuracy. I found statistically significant difference in the mean percentage of accuracy between the generic and the community-situated crowd workers. However, “statistically significant” differences in time, accuracy, or cost between different types of crowdsourcing may not be practically significant for many other domains. Careful considerations haveto be made regarding these trade-offs to select the type of workers and the platform depending on the monetary budget, urgency of retrieving crowdsourced answers, and the accepted threshold of errors in those. For example, the choice of a platform to solicit the community-situated workers from largely depends on the budget for the platform cost, existing collaboration with the plat-

138 7.4. Discussion form’s owners, and the sensitivity of the context. In the case of recovery, members are particularly vulnerable and anonymity is of utmost importance to them [197]. I made sure to design the tasks in a way such that no identifiable personal information has to be revealed by the participants. If the information being validated requires crowdsourcing from people with a stigmatized health condition, members of online communities where real identity is associated (e.g., Facebook health support groups) may not show enough interest in participation. Additionally, if the data requires validation in frequent interval like the context of this study, requesters have to choose a platform from where they can recruit the expected number of workers within the time constraints. While designing tasks in this type of recruitment, one or more existing techniques of minimizing errors while getting rapid answers [113, 165, 118] should be adopted. About 4% of the participants from ITR completed the tasks partially (e.g., completed one or two tasks). The order of the tasks was randomized but a close look at the data revealed that many of them left in the middle when the task of meeting validation was shown to them last. From the results of this study, I found out that validating meetings took the longest time on an average among the three tasks for both types of crowdsourcing. Making the overall task more granular (i.e., one task per worker instead of three) would probably yield more workers completing the assigned task and increase engagement, as suggested by previous work [62, 119]. However, in the case of ITR workers, who were redirected to the task interfaces from clicking on the advertisement, dividing the tasks would require three times the number of interested participants (about 450 participants per task). While designing task interfaces, these trade-offs are important to consider.

139 7.4. Discussion

7.4.3 Implications for Research

Findings from this study provide practical implications for further research regarding the trade-offs and benefits of generic vs. paid community-situated vs. unpaid community- situated crowdsourcing in different contexts. One of the important factors influencing this choice is the type of task to be crowdsourced. For example, generic MTurkers in this study achieved about the same accuracy for meeting identification, but were significantly quicker in completing the task. However, for the other two tasks, their accuracy was way lower than the community-situated crowd. Therefore, piloting the tasks with different types of workers prior to running the actual experiments may provide a guideline for determining which tasks are more suitable for a particular type of crowdsourcing vs. the others. Similarly, the choice of recruiting community-situated workers from online marketplaces like MTurk rather than online communities can be influenced by the actual proportion of the community members in the general population. For instance, about 1% Americans are members of AA [6], and MTurk typically represents similar proportions of particular community members as the general US population. In many other domains, however, the required community members may represent a larger proportion of the general us population (e.g., students). Researchers conducting experiments related to those communities may seek community- situated crowd workers from MTurk or other online marketplaces, where they can recruit a large number of people very quickly, instead of selecting online groups with comparatively smaller number of people (e.g., online communities of students or public Facebook groups, etc.). The choice of platform has an impact on the platform cost as well. The study incurred a high platform cost for advertising on ITR. As discussed earlier in this chapter (Section 7.4.1), the survey can be integrated in external apps and websites related to recovery re-

140 7.4. Discussion ducing the total cost per worker. Similarly, for other studies researchers can optimize the platform cost by carefully choosing a platform that has no or little impact on other performance measures (e.g., time of recruitment, answer quality, etc.). Let’s take Facebook advertising as an example. The average cost per click for Facebook ads is $1.72 [70], and thus using Facebook ads instead of ITR would possibly reduce the platform cost for this study. However, it might have impacted the response rate, as AA members are particularly concerned about their anonymity [197], and there may not be a large number of targeted 12-step members on Facebook who would participate. This may not be an issue for studies requiring other types of community-situated workers, and those may benefit from recruiting workers through platforms with lower advertising costs. Future research should aim to understand the relationship between platform cost and worker performance, and provide a comprehensive guideline on selecting the right platform for a study. Research focused on online community-based crowdsourcing should address particular motivations and expertise of the community members to figure out the feasibility and effectiveness of recruiting such workers. CSCW and HCI researchers have pointed out that community-based intrinsic motivations such as altruism, collectivism, reciprocity, etc. play an important role in worker participation and engagement [83, 115, 18, 116, 5, 109]. Simi- larly, worker expertise and knowledge about the problem domain have proven to be directly related with the quality of crowdsourced data [246, 56, 226]. In many other contexts like recovery, the expertise can come from particular community members who are willing to volunteer for social reasons, such as to achieve recognition in the community or to provide service. Researchers should investigate generic and community-situated crowdsourcing in those domains to understand the benefits and the trade-offs in time, quality, cost, etc.For example, members of an online community of cancer caregivers may provide better answers to a care-giving related question than a non-caregiver.

141 7.4. Discussion

ITR members in this study did not receive any monetary incentives. I assume a major motivation behind their participation was to provide service to the community by helping us create a useful resource for the newcomers in the community. I provided an option for the workers to leave us comments on the task descriptions and the design, and some of the comments from the ITR participants actually revealed this “sense of providing service” to the community by helping others (e.g., “thank you for allowing me to help out.I believe it’s our purpose to help the newcomer if anyone needs help.”). This intrinsic motivation probably also encouraged them to take suﬀicient time to select accurate answers for as many questions as they could. One might expect the community-situated MTurkers to perform better than ITR members, as they should have the same intrinsic motivation along with monetary remuneration for task completion. However, they might not have paid enough attention in order to complete more tasks in less time, resulting in lower accuracy of task completion than the ITR participants. Prior research has established the impact of incentives and motivations on not only who participates in a particular task [94], but also on their attention, dropouts, and performance [99, 105]. Based on prior work and the findings that different types of motivations and incentives may affect workers’ attention and question-answering behavior differently, we recommend future research to use motivation and incentives toana- lyze reliability of the answers. As an example, whenever possible, researchers can compute worker motivation using established scales [184] and figure out if workers with particular motivation types provide quality answers and further target workers with similar motivation. Using the wisdom and motivations from community-situated crowds in this context produced high quality results. This provides implications to develop approaches for collaborative crowdsourcing, pairing generic workers with community-situated workers. In the context of paid microtasks, collective crowdsourcing with the paired-worker model has led to better accuracy and increased output, which, in turn, has translated into lower costs [74,

142 7.4. Discussion

121]. Moreover, different types of workers play different roles and exhibit different potentials in crowdsourcing [49, 96]. Researchers can build on the findings of these previous works and this study to create coordination, where community-situated workers help set directions and guide the non-expert generic workers. The number of ITR members selecting the “not sure” option for the page validation task was significantly higher than the other two populations. This finding echoes thewell known Dunning–Kruger effect, which is defined in psychology as a cognitive bias inwhich low-ability individuals suffer from illusory superiority, mistakenly assessing their ability as much higher than it really is [65]. In other words, experts tend to know what they do not know. In the context of crowdsourcing, this may affect answer quality tremendously if the task comprises many diﬀicult or confusing questions. Therefore, further research needsto be done to find out ways to minimize this effect. Overall, my findings and collective observations highlight multiple opportunities and directions with pursuing deeper understanding of effective applications of community-situated crowdsourcing in different domains, not compromising the quality of the crowdsourced results.

7.4.4 Broader Impact and Takeaways

The study results have practical implications regarding performance trade-offs for different types of crowdsourcing that may be helpful for other HCI and CSCW researchers. It discusses important factors to consider in determining platforms, worker types, and incentives for crowdsourcing. The results suggest that generic crowdsourcing platforms are good for some tasks on a specific domain while other seemingly similar tasks will benefit fromdo- main knowledge. Perhaps this can inform the creation of more complicated setups, where an initial screening can help direct the workers to the right kind of task, even in the context of the same task. Future research should also benefit from the implications regarding the

143 7.5. Limitations two types of crowdsourcing working in tandem.

7.5 Limitations

We assumed that the community members from InTheRooms are in fact members of 12-step fellowships and did not provide the screening questions for them. Although seemingly there is no particular reason for a non-member to join this online community and take part in the study (as discussed in Section 7.2.2), using the same screening criterion for both platforms would make the study conditions fairer. Similarly, even though the three questions used here for screening were not too easy to bypass and I adopted additional filtering criteria to approve answers, I cannot certainly claim that all of the community-situated MTurkers were in fact 12-step fellowship members. In addition, the motivation and the incentive (i.e., pay for the MTurk community-situated workers, altruism and sense of belonging to the community for ITR workers, etc.) may have impacted the outcome of the study, and therefore, follow-up experiments should be conducted to rule out the impact of incentive and motivation. In particular, comparisons of task completion time, accuracy, cost, etc. should be carried out with sample workers recruited from ITR with monetary incentive and generic workers recruited for free.

7.6 Conclusion

In the context of crowdsourcing data validation for Alcoholics Anonymous meetings, this chapter presents an empirical comparison of accuracy, cost, and time trade-offs in generic crowdsourcing from MTurk, community crowdsourcing from MTurk, and community-situated crowdsourcing from an online recovery community, InTheRooms. Community-situated workers from the online recovery community achieved significantly higher accuracy in the

144 7.6. Conclusion validation tasks than the other two types of crowd workers. I further investigated the possible reasons for this difference in accuracy and found that task completion time, knowing when “not sure”, better context interpretations, etc. may have influenced the accuracy of different types of crowd workers. From the findings, I have provided practical implications for crowdsourcing in the recovery context, as well as for research and design in other domains. I have discussed the factors that should be considered while selecting community- situated workers from a particular platform vs. the others, including platform cost, urgency and frequency of crowdsourcing, type of the crowdsourced tasks, etc.

145 Chapter 8

Towards Making HAIR Scalable and Sustainable

The “Meeting Finder” application developed through HAIR requires an update in frequent interval to reflect the changes in local AA websites. Additionally, the code and algorithms for HAIR are required to be maintained with clear documentation so other researchers or developers can use them. This chapter describes the steps taken to make sure that HAIR is scalable and sustainable.

8.1 Updating the List at a Regular Interval

We are not only interested in getting a snapshot of meetings at a single point in time, but also plan to enable it to sustainably self-update. The meeting information is saved in a secure MySQL database server and is designed to be updated in a frequent interval. The processes for scraping local AA websites, filtering the meeting pages, and extracting meeting from the meeting pages can be run in a periodic interval (i.e., once every 3 months). We also plan to apply the following approaches that reduce the cost of computation and crowdsourcing:

1. Instead of classifying all webpages, we will first look for new pages added to the regional websites and pages that have changed since the last update, and send only those pages to the ML model.

2. Instead of all the pages from the previous version of the list, we will send only the set of pages for crowdsourcing that are obtained from step 1 above.

146 8.1. Updating the List at a Regular Interval

3. After automated extraction of meeting records we will match with crowd answers from the previous version and select appropriate subset to crowdsource.

4. Before crowdsourcing, we will check the day, time, address of all meetings in a region to remove possible duplicates.

8.1.1 Consequences of Self-Updating

Some errors may disappear with time when continual updates are performed (e.g., if the crowd workers identify a wrong meeting page or a wrong meeting once, the next time the same page or meeting will possibly be evaluated by a different set of workers, creating a probability of amending the error). On the contrary, if a meeting page remains unidentified and does not change frequently, we will not consider it in subsequent self-updating processes, and the error will compound. As we transition to involve volunteers (AA members) in crowdsourcing reducing the monetary cost to zero, the projected cost must be calculated in terms of human time. If we continue to update the list in a one-month interval, considering the current average task completion time of mTurkers, the cost would be about 96 hours human-time per update. However, it will be significantly reduced when we incorporate the steps described abovein subsequent updates. AA members who have many years in recovery perform service work for the community, which creates opportunities for us to offer the crowdsourcing as a service that could enhance their recovery while also providing valuable information for newcomers. In addition, community volunteers can validate and identify meeting information more accurately than generic crowd workers, as discussed in the previous chapter.

147 8.2. AA Meetings Public API

8.2 AA Meetings Public API

API (Application Program Interface) refers to how multiple applications can interact with and obtain data from one another. In this case, the local meeting lists obtained through here can be utilized by other applications or services. For example, an application for recommending potential sponsors to newcomers may need this information to accurately find out which participants go to the same meeting. I developed an API through which other apps can access the extracted meeting information programmatically. The API fetches data from the meeting database and returns in JSON format upon particular forms of HTML GET requests. The API currently supports the following operations:

1. Given a particular state’s name, it can return a list of area names in that state.

2. Given the name of a particular AA area name, it can return a list of probable meeting pages extracted from the local AA websites for that area.

3. Given the name of a particular AA area name, it returns a list of probable meetings (day, time, and address) in that area.

4. Given a meeting ID, it can return details of that meeting (day, time, and address).

The API is developed using Python and is hosted on the http://cs-aameeting.cs. umn.edu server. A guide to use this API can be found here.

8.3 Guide and Documentation for using HAIR

HAIR combined a classification algorithm, a pattern matching algorithm, and crowdsourcing to create the global meeting list. All the steps of HAIR are documented through a GitHub repository. The guide can be utilized by other researchers to replicate the technique

148 8.3. Guide and Documentation for using HAIR and apply it in other contexts, if necessary. In addition, the human-in-the loop computing techniques have the potential to be extended in other 12-step fellowships for solving relevant problems.

149 Chapter 9

Conclusion and Future Work

The contributions of this thesis include: two formative investigations forging a better understanding of the critical components in recovery communities and the opportunities of designing technology to facilitate peer-support in those communities; the design, and im- plementation of HAIR in developing a global auto-updating peer-support meeting finder as a useful resource to find social support; an empirical comparison of two alternative approaches for community-situated crowdsourcing. In this chapter, I summarize the contributions introduced in this work, and discuss future research avenues to explore.

9.1 Summary

My work combined mixed-methods formative investigations and social computing design techniques (including crowdsourcing, machine learning, and pattern detection) to understand the needs of addiction recovery communities and create tools to enhance their reach. In the context of addiction recovery, I have pointed out situations where current approaches fail, potential areas of caution for the developers of new systems, and try to measure both positive and negative outcomes of novel interventions. I also applied a human-in-the-loop information retrieval technique for data validation in the high-impact context of recovery and pointed out the potential for adopting similar approaches to address various challenges in extracting data from unstructured sources.

150 9.1. Summary

In order to understand the experience and the impact of using video-mediated meetings for peer-led synchronous social support, I conducted a mixed-method analysis. Through online questionnaire and in-depth interviews, I found out that video-mediated meetings may be a promising addition to many health peer-support communities, especially if combined with periodic face-to-face opportunities. However, it is important to enable forming and making visible of community norms and supporting constructive moderation in order to support such synchronous communication in recovery and other health communities. I sought to develop a better understanding of how anonymity is both perceived and enacted in 12-step communities, as a necessary step in formation of guidelines for anonymity- preserving tech-nologies in this critical context. I found out that only one common interpretation of anonymity (“unidentifiability”) is currently captured in previous computing literature. There are alternative interpretations of anonymity as a “social contract” between members and a prioritizing of “program over individual” needs in these support groups. An analysis of online traces of user activity showed that while many individuals follow practices to ensure unidentifiability online, there is a significant portion of people whodonot. Members with more recovery time, more time in the online community, and more friends in that community are more likely to reveal identifiable information in their profile such as a photo with a visible face. This study provided an implication of considering community- specific interpretations of anonymity while designing in sensitive online contexts. Recovery community members consider face-to-face and online meetings as useful meanings of getting support, but from my formative investigations I found out that finding the right meetings often become a challenge, particularly when new to recovery. To address this problem, I created a countrywide list of all AA meetings by retrieving meeting information from the regional AA websites, making it available in a searchable format, and having a systematic way for the list to self-update automatically at a regular interval. Bring-

151 9.1. Summary ing human in the loop for extracting and validating has significantly improved the accuracy of meeting data retrieved. With all the extracted information, I developed a responsive web application, “meeting finder”, where users can search for and to navigate to meetings inany area. Additionally, I developed an API to provide meeting data in a structured format to other apps and websites that support people in recovery. I conducted an empirical comparison of accuracy, cost, and time trade-offs in generic and community-situated crowdsourcing. Volunteers recruited from an online recovery community outperformed generic crowd workers in terms of cost and task completion accuracy. The study provides implications for utilizing intrinsic motivation of members for community-related crowdsourcing tasks.

9.1.1 Lessons learned from conducting research with a vulnerable population

Recovery from substance use is a personal and private undertaking for most people and we took steps to consider the ethical implications of our work and to protect the rights of the users of the online community (InTheRooms) where we conducted our research. We were cautious of protecting the privacy ad confidentiality of the participants in each step: recruitment, payment, extracting data from online sources, data analysis, etc. Moreover, we had to be careful about the norms of the community that are enacted to protect anonymity. For example, it may seem normal to take notes during an observation study. However, taking notes in an AA meeting might not be acceptable to the attendees who are sharing their stories. Taking these additional measures often posed challenges in data collection and analysis, such as lower response rate from the community members, or excluding “ethically controversial” data from analysis.

152 9.2. Future work

9.2 Future work

The work covered in this thesis provides implication for future work in understanding peer- support in a broader context of mental health, and in designing and implementing technologies that have meaningful impact on health informatics. A lot of important questions emerged from my research that can help address challenges and concerns in other health communities. For example, how can technology help in improving community experience for newcomers? How can the intrinsic motivation of community members to help others be best utilized in creating resources for them? I will leverage my experience of using mixed-methods investigations in answering these questions. Next, I believe there are a lot of limitations in existing systems that can be addressed through analyzing and incorporating user feedback. In addition, outcome of this work shows opportunities for designing technology in providing healthcare support in conservative cultures. For example, in developing countries there are stigma and superstitions associated with mental health conditions and it may be interesting to apply value-sensitive design techniques to address challenges and build systems that will work in those settings.

9.3 Closing statement

This work adds to our understanding of the relationship between people and technology by contributing to two related core Social Computing open problems: enacting anonymity online and managing strong tie support in online communities. Working closely with existing communities to create new technologies has established the significance of considering the values, practices, and traditions of these groups in each step of research and design. Design investigations of online communities, personal health informatics systems, and persuasive technologies in this space have the potential to extend availability and transform the way

153 9.3. Closing statement people recover from addiction and alcoholism. Additionally, the application of community- situated crowdsourcing informs various aspects of the design of task interfaces and selection of platforms or workers. Overall, my research provides important and novel contributions in the domains of online health communities, human computation, and human-centric information retrieval.

154 Appendix A

Online Recovery Interview Protocol Guide

Pre-study checklist:

□ Email the consent to the participant or direct them to the online version of the informed consent form.

□ Check batteries in audio-recorder and set it up near phone or computer that will be used for the interview

□ Print this protocol and print a notebook for notes

□ Confirm date, time, and contact information with participant

Study Protocol:

□ Introduce yourself

□ Review the consent form

□ Ask for and answer any questions regarding the consent form

□ Get verbal consent for study participation

□ Ask for permission to audio record

□ Part 0: gather basic background. Questions:

155 ∘ Describe your recovery history. How long have you been in recovery?

∘ Describe the aspects of your recovery program that help you stay clean/sober today (e.g., sponsorship, service, fellowship, etc.).

∘ What kinds of recovery meetings do you attend (e.g., AA, NA, or both)?

∘ What is the role of spirituality in your recovery?

∘ How do you combine online and offline recovery?

∗ What are the differences to you between online and offline recovery?

□ Part 1: Understanding current information and communication technology (ICT) use. Example questions: (At this point, describe “technology” to the participant, i.e., use of text, phone, computer, online resource)

∘ What are some ICTs that you regularly use to support your recovery?

∘ What are some technologies that you tried but ended up not working out?

∘ Describe a situation where an ICT was helpful in your recovery.

∘ Describe a situation where an ICT was unhelpful or harmful to your recovery.

∘ What were some challenges you faced as a newcomer that may have been supported with ICTs?

∘ Can you describe a situation when watching vlogs or reading blogs seemed affective to you? Which one is more affective for you? Why?

∘ How have you used ICT for each of these aspects of your recovery:

∗ Working the steps?

∗ Personal reflection, meditation, or prayer?

156 · Working step 11 and 12 that involve spirituality (online books, prayer, meditation etc)

∗ Communicating with fellow recovering addicts/alcoholics?

∗ Carrying the message of recovery to the still suffering addict/alcoholic?

∗ In communicating with your sponsor or your sponsees?

∗ Participating in service to the NA/AA community?

∗ Finding or attending meetings? H I meetings or service?

∗ Accessing program literature or daily reflections?

∗ Keeps you away from negative thoughts/ distracts you from your thoughts of addiction?

∘ How did the role of ICTs in your recovery change as you got more clean/sober time?

∗ How did your use of technology evolve as you were going through the steps?

∘ Describe the role of your personal devices (cellphones) in recovery (use of the iPhone app)

□ Part 2: use of intherooms.com Example questions:

∘ What led you to start using intherooms.com?

∘ How do you currently use intherooms.com for your recovery?

∗ What are your main motives now in using the website – how did they change since you first started?

∗ What inspires you to use intherooms.com?

157 ∘ What type of personal information you share on your profile?

∘ How did your personal information sharing change since you first started?

∘ Describe a situation where somebody provided you with support on in the- rooms.com. What was most helpful about that interaction?

∘ Describe a situation where you provided somebody with support on in the- rooms.com. How did that go? What was hard about it? What was fulfilling about it?

∘ How do you choose people to be your friend on intherooms.com?

∘ How do you choose which groups to join?

∘ Describe your closest relationships on intherooms.com. How did these relationships form? How do you maintain these relationships?

∗ Describe your experience of social communication with your peers/ sponsor outside of intherooms.com.

∗ What are the factors affecting “trust” in your online social group? (Who do you trust?)

∘ Describe any situations on intherooms.com where you were unable to maintain a supportive relationship. Why do you think that happened? What would you do differently?

∗ Can you give an example of any positive comment or an example of a negative comment you received from your online peers? How did these comments affect your recovery?

∘ Describe the importance of anonymity to your recovery. How do you maintain your anonymity on intherooms.com?

158 ∗ Describe a situation where you did not share details of your story because of privacy/anonymity. How did that affect your credibility?

∗ Describe a situation, if any, when you think anonymity made it harder for you to connect with others or get support.

∘ Describe a situation on intherooms.com where a user violated the rules or social norms of the community. What happened? How was the situation handled? How would you handle the situation differently? What can be done to prevent this situation in the future?

∘ What do you get out of an online/offline support (and follow up with meeting)?

∘ What are the factors affecting your decision whether or not to share your web- cam in the online meetings?

∘ Describe your experience of status updates/ blog posts, if any.

∘ How would you improve intherooms.com?

∗ Describe a situation, if any, when technology got in the way of your getting support online

∗ Describe a situation, if any, when you wanted to get some help to get the technology working right for your recovery

∗ What are some of the good and bad features of intherooms.com website? (tell participant about examples of good and bad)

∗ Can you describe any experience when your anonymity was violated? What do you think the admins should do to strengthen privacy and anonymity in intherooms.com?

159 ∗ Do you use any other online support groups except intherooms.com? How is that helpful?

□ Part 3: unstructured feedback

∘ What is something I should have asked you, but I didn’t?

∘ Please, email me if there is anything else you would like to share with me.

□ Thank the participant for his or her time

□ Ask the participant to pass your information to other potential participants (remind them not to break the anonymity of others)

Post-Study Checklist:

□ Upload audio files to secure computer

□ Transcribe and delete audio file

160 Appendix B

Online Recovery Questionnaire

1. Where do you live? (state, city, zip)

2. What is your gender?)

a) Female

b) Male

c) Other

3. How old are you?

4. To what extent does this statement describe you: ”I am comfortable with technology.”)

a) Strongly disagree

b) Disagree

c) Neither disagree nor agree

d) Agree

e) Strongly agree

5. What is your education level?

a) No schooling completed

161 b) Nursery school to 8th grade

c) Some high school, no diploma

d) High school graduate, diploma or the equivalent (for example: GED)

e) Some college credit, no degree

f) Trade/technical/vocational training

g) Associate degree

h) Bachelor’s Degree

i) Master’s Degree

j) Doctorate degree

6. Approximately how many free hours (time that you are not sleeping, working, or providing family care) do you have on a typical week day?

7. Approximately how many free hours (time that you are not sleeping, working, or providing family care) do you have on a typical weekend day?

8. What do you consider your primary recovery community? (Check or write in multiple options if you consider yourself a member of multiple communities)

a) AA

b) NA

c) Other

9. What is your recovery date (e.g., ”sober date”)? (Day/Month/Year)

10. Which of the following options best describes your recovery history?)

162 a) I have not had any slips / relapses since first getting into recovery.

b) I have had 1 slip / relapse since first getting into recovery.

c) I have had 2 or 3 slips / relapses since first getting into recovery.

d) Agree

e) I have had more than 3 slips / relapses since first getting into recovery.

11. To what extent does this statement describe you: ”I have had a spiritual awakening.”)

a) Strongly disagree

b) Disagree

c) Neither disagree nor agree

d) Agree

e) Strongly agree

12. To what extent does this statement describe you: ”Anonymity is part of the spiritual foundation of my recovery.”)

a) Strongly disagree

b) Disagree

c) Neither disagree nor agree

d) Agree

e) Strongly agree

13. In your own words, what does anonymity mean to you?

14. Do you currently have a sponsor in your fellowship?

163 a) Yes

b) No, and I am not looking for one

c) No, but I am looking for one

15. Do you currently ”work the steps” of your fellowship?

a) Yes, regularly

b) Yes, but not regularly

c) No, but I’m trying to get into it

d) No, and it’s not for me

16. Do you interact with people in your fellowship outside of the context of meetings (e.g., call each other, get coffee together, etc.)?

a) Yes, regularly

b) Yes, but not regularly

c) No, but I’m trying to get into it

d) No, and it’s not for me

17. Staying in recovery is the most important thing in my life.

a) Strongly disagree

b) Disagree

c) Neither disagree nor agree

d) Agree

e) Strongly agree

164 18. I am totally committed to staying off alcohol / drugs / my problem behavior.

a) Strongly disagree

b) Disagree

c) Neither disagree nor agree

d) Agree

e) Strongly agree

19. I will do whatever it takes to recover from my using / drinking / my problem behavior.

a) Strongly disagree

b) Disagree

c) Neither disagree nor agree

d) Agree

e) Strongly agree

20. I never want to return to alcohol / drugs / my problem behavior again.

a) Strongly disagree

b) Disagree

c) Neither disagree nor agree

d) Agree

e) Strongly agree

21. I have had enough of alcohol / drugs / my problem behavior.

a) Strongly disagree

165 b) Disagree

c) Neither disagree nor agree

d) Agree

e) Strongly agree

22. Do you attend in-person (offline) meetings of your primary fellowship?

a) Yes, I currently attend in-person meetings.

b) No, but I have attended in-person meeting(s) in the past.

c) No, I have never attended in-person meetings.

d) Yes, but I am temporarily unable to regularly make in-person meetings.

23. If you answered (b) in Q 22, briefly, describe why you no longer attend in-person meetings.

24. If you answered (c) in Q 22, briefly, describe why you have never attended an in- person meeting.

25. If you answered (d) in Q 22, briefly, describe the situation that is temporarily stopping you from being able to regularly make in-person meetings.

26. On average, how often do you attend in-person meetings of your primary fellowship?

a) Multiple meetings per day

b) Daily

c) 4-6 times a week

d) 2-3 times a week

166 e) Once a week

f) 2-3 times a month

g) 2-3 times a month

27. People in offline meetings know my full name.

a) Strongly disagree

b) Disagree

c) Neither disagree nor agree

d) Agree

e) Strongly agree

28. People in offline meetings know the town and state where I live.

a) Strongly disagree

b) Disagree

c) Neither disagree nor agree

d) Agree

e) Strongly agree

29. People in offline meetings are aware of my recovery history (e.g., sober date, relapse history, etc.).

a) Strongly disagree

b) Disagree

c) Neutral

167 d) Agree

e) Strongly agree

30. People in offline meetings are aware of my professional life (e.g., what Idofora living, where I work).

a) Strongly disagree

b) Disagree

c) Neutral

d) Agree

e) Strongly agree

31. People in offline meetings are aware of my personal life (e.g., my relationship status, my family relationships, etc.).

a) Strongly disagree

b) Disagree

c) Neutral

d) Agree

e) Strongly agree

32. Do you attend online meetings of your primary fellowship?

a) Yes, I currently attend online meetings.

b) No, but I have attended online meeting(s) in the past.

c) No, I have never attended online meetings.

168 d) Yes, but I am temporarily unable to regularly make online meetings.

33. If you answered (b) in Q 32, briefly, describe why you no longer attend online meetings.

34. If you answered (c) in Q 32, briefly, describe why you have never attended an online meeting.

35. If you answered (d) in Q 32, briefly, describe the situation that is temporarily stopping you from being able to regularly make online meetings.

36. On average, how often do you attend online meetings of your primary fellowship?

a) Multiple meetings per day

b) Daily

c) 4-6 times a week

d) 2-3 times a week

e) Once a week

f) 2-3 times a month

g) Less than once a month

37. Briefly describe how you were first introduced to online meetings.

38. People in online meetings know my full name.

a) Strongly disagree

b) Disagree

c) Neutral

169 d) Agree

e) Strongly agree

39. People in online meetings know what I look like (e.g., from profile photo, video, etc.).

a) Strongly disagree

b) Disagree

c) Neutral

d) Agree

e) Strongly agree

40. People in online meetings know my basic demographic characteristics (e.g., age, gender).

a) Strongly disagree

b) Disagree

c) Neutral

d) Agree

e) Strongly agree

41. People in online meetings know the town and state where I live.

a) Strongly disagree

b) Disagree

c) Neutral

d) Agree

170 e) Strongly agree

42. People in online meetings are aware of my recovery history (e.g., sober date, relapse history, etc.).

a) Strongly disagree

b) Disagree

c) Neutral

d) Agree

e) Strongly agree

43. People in online meetings are aware of my professional life (e.g., what I do for a living, where I work).

a) Strongly disagree

b) Disagree

c) Neutral

d) Agree

e) Strongly agree

44. People in online meetings are aware of my personal life (e.g., my relationship status, my family relationships, etc.).

a) Strongly disagree

b) Disagree

c) Neutral

171 d) Agree

e) Strongly agree

45. In your own words, how does attending online meetings help your recovery?

46. What best describes your opinion on how online meetings compare to offline meetings?

a) Online meetings are not even worth attending

b) Online meetings are less helpful than offline meetings, but still worth it ifit’s the only option.

c) Online meetings are about as helpful as offline meetings.

d) Offline meetings are less helpful than online meetings, but still worth itifit’s the only option.

e) Offline meetings are not even worth attending

47. What best describes your perception of your anonymity in online meetings compared to offline meetings?

a) Online meetings do much worse at preserving my anonymity than offline meetings.

b) Online meetings do a little worse at preserving my anonymity than offline meetings.

c) Online meetings do about as well at preserving my anonymity as offline meetings.

d) Offline meetings do a little worse at preserving my anonymity than online meetings.

172 e) Offline meetings do much worse at preserving my anonymity than online meetings.

48. Other than online meetings, in which features of online recovery do you participate regularly (check all that apply)?

a) Reading or contributing to forums or discussion boards

b) Checking online recovery friends’ profiles / updates

c) Posting directly to online recovery friends (such as, on their walls or in private messages)

d) Reading or contributing to online chat rooms

e) Reading or listening to speaker stories / tapes

f) Finding offline meetings to attend

g) Other

49. Please, share any general comments about your online recovery experience or about this survey.

50. We may reach out to a few participants by phone or video-chat for a more detailed interview. You will be compensated for your time and your anonymity will remain protected. If this is something you may be interested in doing, please share your contact information with us and we will reach out to you with more information.

173 Appendix C

Email/ Private message Sent to Recruit Interview Participants

Dear [ITR pseudonym], My name is Sabirat Rubya and I am a researcher at the University of Minnesota working with Prof. Svetlana Yarosh on understanding how to improve online recovery communities. We have been working with ITR.com and you may have seen one of our ads on the site recently. We would like to interview influential members of intherooms.com and we think that you fit the bill because [specific selection criterion]. Would you be willing to participate in a phone or Skype interview with me (less than 90 minutes of your time) to help me better understand how places like ITR.com can help other people in recovery? We will compensate you for your time with a $25 Amazon.com gift card delivered to an email address of your choice. Additionally, protecting your anonymity is our highest priority, so we will never ask you for your last name or other identifying information and you won’t have to sign anything. If you are interested or have any questions about this study, please reply to this message or email me at [email protected]. Sincerely, University of Minnesota Recovery Team

174 Appendix D

Screening Survey and Task Instructions on Amazon

Mechanical Turk

D.1 Screening survey questions

1. What is your primary fellowship?

2. How many meetings do you attend weekly?

3. Fill in the blank: The 12th tradition says, “______is the spiritual foundation of all our Traditions, ever reminding us to place principles before personalities.”

D.2 Task instruction screenshots

D.2.1 Page validation

Figure D.1: Screenshot of the instructions for the task of page validation

175 D.2. Task instruction screenshots

D.2.2 Meeting validation

Figure D.2: Screenshot of the instructions for the task of meeting validation

D.2.3 Meeting identification

Figure D.3: Screenshot of the instructions for the task of meeting identification

176 D.2. Task instruction screenshots

D.2.4 Consent form

177 D.2. Task instruction screenshots

Figure D.4: Screenshot of the consent form shown to the MTurk and ITR participants

178 Bibliography

[1] Aa meeting structure. URL: http://www.darvsmith.com/dox/aa.html. [On- line; accessed 2017-04-21]. [2] A. A. AbdulHussien. Comparison of machine learning algorithms to classify web pages. en. International Journal of Advanced Computer Science and Applications (ijacsa), 8(11), 2017. DOI: 10.14569/IJACSA.2017.081127. [3] N. I. o. D. Abuse. Trends & Statistics. en. URL: https://www.drugabuse.gov/ related-topics/trends-statistics. [4] E. Agapie, L. Colusso, S. A. Munson and G. Hsieh. Plansourcing: generating behavior change plans with friends and crowds. in CSCW ’16, pages 119–133, New York, NY, USA. ACM, 2016. ISBN: 978-1-4503-3592-8. DOI: 10.1145/2818048. 2819943. URL: http://doi.acm.org/10.1145/2818048.2819943. event-place: San Francisco, California, USA. [5] S. L. Alam and J. Campbell. Crowdsourcing Motivations in a not-for-profit GLAM context: The Australian Newspapers Digitisation Program. en. in ACIS 2012 : Pro- ceedings of the 23rd Australasian Conference on Information Systems. Australasian Conference on Information Systems, 2012. ISBN: 978-1-74156-172-2. URL: https: //openresearch-repository.anu.edu.au/handle/1885/154077 (urlseen 11/01/2020). [6] Alcoholics Anonymous : Estimates Worldwide A.A. Individual and Group Mem- bership. URL: https://www.aa.org/assets/en_US/aa-literature/smf- 132 - estimates - worldwide - aa - individual - and - group - membership (urlseen 16/01/2020). [7] Alcoholics anonymous : twelve steps and twelve traditions. URL: http://www. aa.org/pages/en_US/twelve-steps-and-twelve-traditions. [Online; accessed 2017-07-07]. [8] L. Alem, S. Hansen and J. Li. Evaluating clinicians’ experience in a telemedicine application: a presence perspective. in OZCHI ’06, pages 47–54, New York, NY, USA. ACM, 2006. ISBN: 978-1-59593-545-8. DOI: 10.1145/1228175.1228187. URL: http://doi.acm.org/10.1145/1228175.1228187. [Online; accessed 2016-04-26].

179 Bibliography

[9] O. Alonso and M. Lease. Crowdsourcing 101: putting the wsdm of crowds to work for you. in WSDM ’11, pages 1–2, New York, NY, USA. ACM, 2011. ISBN: 978- 1-4503-0493-1. DOI: 10.1145/1935826.1935831. URL: http://doi.acm.org/ 10.1145/1935826.1935831. event-place: Hong Kong, China. [10] Altruistic crowdsourcing for arabic speech corpus annotation. URL: https://www. sciencedirect.com/science/article/pii/S1877050917321592. [11] Amazon mechanical turk. URL: https://www.mturk.com/. [Online; accessed 2018-09-23]. [12] M. G. Ames, J. Go, J. J. Kaye and M. Spasojevic. Making love in the network closet: the benefits and work of family videochat. in CSCW ’10, pages 145–154, New York, NY, USA. ACM, 2010. ISBN: 978-1-60558-795-0. DOI: 10.1145/1718918. 1718946. URL: http://doi.acm.org/10.1145/1718918.1718946. [Online; accessed 2016-05-14]. [13] Y. Amichai-Hamburger. The Social Net: Understanding our online behavior. en. OUP Oxford, 07 february 2013. ISBN: 978-0-19-164822-9. [14] I. Anagnostopoulos, C. Anagnostopoulos, V. Loumos and E. Kayafas. Classifying web pages employing a probabilistic neural network. IEE Proceedings - Software, 151(3):139–150, june 2004. ISSN: 1462-5970. DOI: 10.1049/ip-sen:20040121. [15] N. Andalibi. Social media for sensitive disclosures and social support: the case of miscarriage. in GROUP ’16, pages 461–465, New York, NY, USA. ACM, 2016. ISBN: 978-1-4503-4276-6. DOI: 10.1145/2957276.2997019. URL: http://doi. acm.org/10.1145/2957276.2997019. [Online; accessed 2017-03-29]. [16] N. Andalibi, O. L. Haimson, M. De Choudhury and A. Forte. Understanding social media disclosures of sexual abuse through the lenses of support seeking and anonymity. in CHI ’16, pages 3906–3918, New York, NY, USA. ACM, 2016. ISBN: 978-1-4503-3362-7. DOI: 10.1145/2858036.2858096. URL: http://doi.acm. org/10.1145/2858036.2858096. [Online; accessed 2017-03-29]. [17] Anonymous. To reveal or not to reveal: a theoretical model of anonymous communication. en. Communication Theory, 8(4):381–407, 01 november 1998. ISSN: 1468-2885. DOI: 10.1111/j.1468-2885.1998.tb00226.x. [18] D. Auferbauer and H. Tellioundefinedlu. Centralized crowdsourcing in disaster management: findings and implications. in Proceedings of the 8th International Conference on Communities and Technologies, Camp;T ’17, pages 173–182, Troyes, France. Association for Computing Machinery, 2017. ISBN: 9781450348546. DOI: 10.1145/3083671.3083689. URL: https://doi-org.ezp2.lib.umn.edu/ 10.1145/3083671.3083689.

180 Bibliography

[19] M. J. Barratt. Discussing illicit drugs in public internet forums: visibility, stigma, and pseudonymity. in CT ’11, pages 159–168, New York, NY, USA. ACM, 2011. ISBN: 978-1-4503-0824-3. DOI: 10.1145/2103354.2103376. URL: http://doi. acm.org/10.1145/2103354.2103376. [Online; accessed 2017-03-30]. [20] A. L. Batcheller, B. Hilligoss, K. Nam, E. Rader, M. Rey-babarro and X. Zhou. Testing the technology: playing games with video conferencing. in pages 849–852, 2007. ISBN: 978-1-59593-593-9. [21] D. R. Bateman, E. Brady, D. Wilkerson, E.-H. Yi, Y. Karanam and C. M. Calla- han. Comparing crowdsourcing and friendsourcing: a social media-based feasibility study to support alzheimer disease caregivers. en. JMIR Research Protocols, 6(4):e56, 2017. DOI: 10.2196/resprot.6904. [22] B. G. Bergman, N. W. Kelly, B. B. Hoeppner, C. L. Vilsaint and J. F. Kelly. Digital recovery management: Characterizing recovery-specific social network site participation and perceived benefit. Psychology of Addictive Behaviors, 31(4):506–512, 2017. ISSN: 1939-1501(Electronic),0893-164X(Print). DOI: 10.1037/adb0000255. Place: US Publisher: American Psychological Association. [23] S. Berkovsky, M. Coombe, J. Freyne, D. Bhandari and N. Baghaei. Physical activity motivating games: virtual rewards for real activity. in CHI ’10, pages 243–252, New York, NY, USA. ACM, 2010. ISBN: 978-1-60558-929-9. DOI: 10.1145/1753326. 1753362. URL: http://doi.acm.org/10.1145/1753326.1753362. [Online; accessed 2017-04-23]. [24] S. Berkovsky, J. Freyne and M. Coombe. Physical activity motivating games: be active and get your own reward. ACM Trans. Comput.-Hum. Interact., 19(4):32:1– 32:41, december 2012. ISSN: 1073-0516. DOI: 10.1145/2395131.2395139. [25] M. S. Bernstein, D. Tan, G. Smith, M. Czerwinski and E. Horvitz. Personaliza- tion via friendsourcing. ACM Trans. Comput.-Hum. Interact., 17(2):6:1–6:28, may 2008. ISSN: 1073-0516. DOI: 10.1145/1746259.1746260. [26] J. Birnholtz, N. A. R. Merola and A. Paul. ”is it weird to still be a virgin”: anonymous, locally targeted questions on facebook confession boards. in CHI ’15, New York, NY, USA. ACM, 2015. DOI: 10 . 1145 / 2702123 . 2702410. URL: http : //doi.acm.org/10.1145/2702123.2702410. [27] E. W. Black, K. Mezzina and L. A. Thompson. Anonymous social media – understanding the content and context of yik yak. Computers in Human Behavior, 57:17–22, april 2016. ISSN: 0747-5632. DOI: 10.1016/j.chb.2015.11.043. [28] A. Blanchard and M. Markus. Sense of virtual community-maintaining the experience of belonging. in HICSS ’02, pages 270.2–, Washington, DC, USA. IEEE Computer Society, 2002. ISBN: 978-0-7695-1435-2. URL: http://dl.acm.org/ citation.cfm?id=820746.821187. [Online; accessed 2017-04-21].

181 Bibliography

[29] S. A. Bly, S. R. Harrison and S. Irwin. Media spaces: bringing people together in a video, audio, and computing environment. Commun. ACM, 36(1):28–46, 1993. [30] R. M. Borromeo, T. Laurent and M. Toyama. The influence of crowd type and task complexity on crowdsourced work quality. in Proceedings of the 20th Inter- national Database Engineering Applications Symposium, IDEAS ’16, pages 70– 76, Montreal, QC, Canada. Association for Computing Machinery, 2016. ISBN: 9781450341189. DOI: 10.1145/2938503.2938511. URL: https://doi.org/ 10.1145/2938503.2938511. [31] R. M. Borromeo and M. Toyama. An investigation of unpaid crowdsourcing. Human- centric Computing and Information Sciences, 6(1):11, 2016. [32] E. E. Bouchery, H. J. Harwood, J. J. Sacks, C. J. Simon and R. D. Brewer. Economic Costs of Excessive Alcohol Consumption in the U.S., 2006. en. American Journal of Preventive Medicine, 41(5):516–524, november 2011. ISSN: 07493797. DOI: 10. 1016/j.amepre.2011.06.045. URL: https://linkinghub.elsevier.com/ retrieve/pii/S0749379711005381 (urlseen 07/11/2019). [33] E. Brady. Getting fast, free, and anonymous answers to questions asked by people with visual impairments. SIGACCESS Access. Comput., (112):16–25, july 2015. ISSN: 1558-2337. DOI: 10.1145/2809915.2809918. [34] E. L. Brady, Y. Zhong, M. R. Morris and J. P. Bigham. Investigating the appropri- ateness of social network question asking as a resource for blind users. in CSCW ’13, pages 1225–1236, New York, NY, USA. ACM, 2013. ISBN: 978-1-4503-1331- 5. DOI: 10.1145/2441776.2441915. URL: http://doi.acm.org/10.1145/ 2441776.2441915. event-place: San Antonio, Texas, USA. [35] C. F. Brooks. Toward ‘hybridised’ faculty development for the twenty‐first century: blending online communities of practice and face‐to‐face meetings in instructional and professional support programmes. Innovations in Education and Teaching In- ternational, 47(3):261–270, 01 august 2010. ISSN: 1470-3297. DOI: 10 . 1080 / 14703297.2010.498177. [36] M. Burke, R. Kraut and D. Williams. Social Use of Computer-mediated Communi- cation by Adults on the Autism Spectrum. in Proceedings of the 2010 ACM Confer- ence on Computer Supported Cooperative Work, CSCW ’10, pages 425–434, New York, NY, USA. ACM, 2010. ISBN: 978-1-60558-795-0. DOI: 10.1145/1718918. 1718991. URL: http://doi.acm.org/10.1145/1718918.1718991 (urlseen 12/11/2019). event-place: Savannah, Georgia, USA. [37] C. Cain. Personal Stories: Identity Acquisition and Self-Understanding in Alco- holics Anonymous. Ethos, 19(2):210–253, 1991. ISSN: 0091-2131. URL: https : //www.jstor.org/stable/640431 (urlseen 12/11/2019).

182 Bibliography

[38] S. W. Campbell and M. J. Kelley. Mobile phone use among Alcoholics Anony- mous members: new sites for recovery. en. New Media & Society, 10(6):915–933, december 2008. ISSN: 1461-4448. DOI: 10.1177/1461444808096251. URL: https: //doi.org/10.1177/1461444808096251 (urlseen 26/09/2019).

[39] M. Cassidy. Centaur chess shows power of teaming human and machine. en푈푆, 30 december 2014. URL: https://www.huffingtonpost.com/mike-cassidy/ centaur-chess-shows-power_b_6383606.html. [Online; accessed 2018-08- 13]. [40] R. Chandwani and R. De. Doctor-patient interaction in telemedicine: linking the structurational aspects to institutionalization. in ICTD ’13, pages 17–20, New York, NY, USA. ACM, 2013. DOI: 10.1145/2517899.2517934. URL: http://doi. acm.org/10.1145/2517899.2517934. [Online; accessed 2016-04-26]. [41] C.-H. Chang and S.-C. Lui. Iepad: information extraction based on pattern dis- covery. in WWW ’01, pages 681–688, New York, NY, USA. ACM, 2001. ISBN: 978-1-58113-348-6. DOI: 10.1145/371920.372182. URL: http://doi.acm. org/10.1145/371920.372182. [42] J. Chang. The role of anonymity in deindividuated behavior: a comparison of deindividuation theory and the social identity model of deindividuation effects (side). The Pulse, 6(1):2–8, 2008. [43] G. Charness and U. Gneezy. What’s in a name? anonymity and social distance in dictator and ultimatum games. Journal of Economic Behavior Organization, 68(1):29–35, october 2008. ISSN: 0167-2681. DOI: 10.1016/j.jebo.2008.03. 001. [44] M. Chau and H. Chen. A machine learning approach to web page filtering using content and structure analysis. Decis. Support Syst., 44(2):482–494, january 2008. ISSN: 0167-9236. DOI: 10.1016/j.dss.2007.06.002. [45] Y. Chen, P. L. Bogen II, H. Hsieh, E. A. Fox and L. N. Cassel. Categorization of computing education resources with utilization of crowdsourcing. in JCDL ’12, pages 121–124, New York, NY, USA. ACM, 2012. ISBN: 978-1-4503-1154-0. DOI: 10.1145/2232817.2232840. URL: http://doi.acm.org/10.1145/2232817. 2232840. [46] Z. Chen. A semiautomatic approach for accurate and low-effort spreadsheet data extraction. in 2014. [47] J. Cheng and M. S. Bernstein. Flock: hybrid crowd-machine learning classifiers. in CSCW ’15, pages 600–611, New York, NY, USA. ACM, 2015. ISBN: 978-1-4503- 2922-4. DOI: 10.1145/2675133.2675214. URL: http://doi.acm.org/10. 1145/2675133.2675214.

183 Bibliography

[48] E. H. Chi. Technical perspective: humans and computers working together on hard tasks. Commun. ACM, 60(9):92–92, august 2017. ISSN: 0001-0782. DOI: 10.1145/ 3068614. [49] P. H. Chia and J. Chuang. Community-based web security: complementary roles of the serious and casual contributors. in Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work, CSCW ’12, pages 1023–1032, Seattle, Washington, USA. Association for Computing Machinery, 2012. DOI: 10.1145/ 2145204.2145356. URL: https://doi.org/10.1145/2145204.2145356. [50] D. Cho, S. Kim and A. Acquisti. Empirical analysis of online anonymity and user behaviors: the impact of real name policy. in pages 3041–3050. 2012 45th Hawaii International Conference on System Sciences, january 2012. DOI: 10.1109/HICSS. 2012.241. [51] K. M. Christopherson. The positive and negative implications of anonymity in internet social interactions: ”on the internet, nobody knows you’re a dog”. Comput. Hum. Behav., 23(6):3038–3056, november 2007. ISSN: 0747-5632. DOI: 10.1016/ j.chb.2006.09.001. [52] A. M. Cirucci. Facebook’s affordances, visible culture, and anti-anonymity. in SM- Society ’15, 11:1–11:5, New York, NY, USA. ACM, 2015. ISBN: 978-1-4503-3923- 0. DOI: 10.1145/2789187.2789202. URL: http://doi.acm.org/10.1145/ 2789187.2789202. [53] F. Curmi, M. A. Ferrario, J. Whittle and F. F. Mueller. Crowdsourcing synchronous spectator support: (go on, go on, you’re the best)n-1. in CHI ’15, pages 757–766, New York, NY, USA. ACM, 2015. ISBN: 978-1-4503-3145-6. DOI: 10 . 1145 / 2702123.2702338. URL: http://doi.acm.org/10.1145/2702123.2702338. event-place: Seoul, Republic of Korea. [54] K. P. Davison, J. W. Pennebaker and S. S. Dickerson. Who talks? the social psychology of illness support groups. eng. The American Psychologist, 55(2):205–217, february 2000. ISSN: 0003-066X. PMID: 10717968. [55] L. de Alfaro and M. Shavlovsky. Crowdgrader: a tool for crowdsourcing the evaluation of homework assignments. in SIGCSE ’14, pages 415–420, New York, NY, USA. ACM, 2014. ISBN: 978-1-4503-2605-6. DOI: 10.1145/2538862.2538900. URL: http://doi.acm.org/10.1145/2538862.2538900. event-place: Atlanta, Georgia, USA. [56] R. A. J. de Vries, C. Zaga, F. Bayer, C. H. C. Drossaert, K. P. Truong and V. Ev- ers. Experts get me started, peers keep me going: comparing crowd- versus expert- designed motivational text messages for exercise behavior change. in Pervasive- Health ’17, pages 155–162, New York, NY, USA. ACM, 2017. ISBN: 978-1-4503-

184 Bibliography

6363-1. DOI: 10.1145/3154862.3154875. URL: http://doi.acm.org/10. 1145/3154862.3154875. event-place: Barcelona, Spain. [57] M. De Choudhury and S. De. Mental health discourse on reddit: self-disclosure, social support, and anonymity. in 2014. URL: http : / / www . aaai . org / ocs / index.php/ICWSM/ICWSM14/paper/download/8075/8107. [Online; accessed 2017-07-05]. [58] J. C. Dean and F. Rud. The drug addict and the stigma of addiction. eng. Int J Addict, 19(8):859–869, december 1984. DOI: 10.3109/10826088409061991. [59] M. I. Devi, R. Rajaram and K. Selvakuberan. Machine learning techniques for automated web page classification using url features. in volume 2, pages 116–120. International Conference on Computational Intelligence and Multimedia Applica- tions (ICCIMA 2007), december 2007. DOI: 10.1109/ICCIMA.2007.342. [60] E. Diener. Deindividuation, self-awareness, and disinhibition. English. Journal of Personality and Social Psychology, 37(7):1160–1171, 1979. ISSN: 1939-1315 0022- 3514. DOI: 10.1037/0022-3514.37.7.1160. [61] E. Diener, S. C. Fraser, A. L. Beaman and R. T. Kelem. Effects of deindividuation variables on stealing among halloween trick-or-treaters. English. Journal of Personality and Social Psychology, 33(2):178–183, 1976. ISSN: 1939-1315 0022- 3514. DOI: 10.1037/0022-3514.33.2.178. [62] D. E. Difallah, M. Catasta, G. Demartini, P. G. Ipeirotis and P. Cudré-Mauroux. The dynamics of micro-task crowdsourcing: the case of amazon mturk. in Pro- ceedings of the 24th International Conference on World Wide Web, WWW ’15, pages 238–247, Florence, Italy. International World Wide Web Conferences Steer- ing Committee, 2015. ISBN: 9781450334693. DOI: 10.1145/2736277.2741685. URL: https://doi-org.ezp2.lib.umn.edu/10.1145/2736277.2741685. [63] L. A. K. DrPH. Alcoholics Anonymous Effectiveness: Faith Meets Science. Journal of Addictive Diseases, 28(2):145–157, april 2009. ISSN: 1055-0887. DOI: 10.1080/ 10550880902772464. URL: https://doi.org/10.1080/10550880902772464 (urlseen 07/11/2019). [64] V. J. Dubrovsky, S. Kiesler and B. N. Sethna. The equalization phenomenon: status effects in computer-mediated and face-to-face decision-making groups. Human– Computer Interaction, 6(2):119–146, 01 june 1991. ISSN: 0737-0024. DOI: 10 . 1207/s15327051hci0602_2. [65] D. Dunning. Chapter five - The Dunning–Kruger Effect: On Being Ignorant of One’s Own Ignorance. en. in J. M. Olson and M. P. Zanna, editorsAdvances in Ex- perimental Social Psychology. volume 44, pages 247–296. Academic Press, january 2011. DOI: 10 . 1016 / B978 - 0 - 12 - 385522 - 0 . 00005 - 6. URL: http : / /

185 Bibliography

www.sciencedirect.com/science/article/pii/B9780123855220000056 (urlseen 11/01/2020). [66] D. B. Egolf. Forming Storming Norming Performing: Successful Communications in Groups and Teams. en. iUniverse, 2001. ISBN: 978-0-595-20444-1. [67] E. V. Eikey. Privacy and Weight Loss Apps: A First Look at How Women with Eat- ing Disorders Use Social Features. in Proceedings of the 19th International Con- ference on Supporting Group Work, GROUP ’16, pages 413–415, New York, NY, USA. ACM, 2016. ISBN: 978-1-4503-4276-6. DOI: 10.1145/2957276.2996282. URL: http://doi.acm.org/10.1145/2957276.2996282 (urlseen 12/11/2019). event-place: Sanibel Island, Florida, USA. [68] N. B. Ellison, L. Blackwell, C. Lampe and P. Trieu. “the question exists, but you don’t exist with it”: strategic anonymity in the social lives of adolescents. en. Social Media + Society, 2(4):2056305116670673, 01 november 2016. ISSN: 2056-3051. DOI: 10.1177/2056305116670673. [69] L. Erickson, I. Petrick and E. Trauth. Hanging with the right crowd: matching crowdsourcing need to crowd characteristics. AMCIS 2012 Proceedings, 29 july 2012. [70] Facebook Ad Benchmarks for YOUR Industry. URL: https://www.wordstream. com/blog/ws/2017/02/28/facebook- advertising- benchmarks. Library Catalog: www.wordstream.com. [71] S. Farnham, L. Cheng, L. Stone, M. Zaner-Godsey, C. Hibbeln, K. Syrjala, A. M. Clark and J. Abrams. Hutchworld: clinical study of computer-mediated social support for cancer patients and their caregivers. in CHI ’02, pages 375–382, New York, NY, USA. ACM, 2002. ISBN: 1-58113-453-3. DOI: 10.1145/503376.503444. URL: http://doi.acm.org/10.1145/503376.503444. [Online; accessed 2014-09- 18]. [72] R. Faulkner, S. Walling and M. Pinchuk. Etiquette in wikipedia: weening new ed- itors into productive ones. in WikiSym ’12, 5:1–5:4, New York, NY, USA. ACM, 2012. ISBN: 978-1-4503-1605-7. DOI: 10.1145/2462932.2462939. URL: http: //doi.acm.org/10.1145/2462932.2462939. [Online; accessed 2016-08-08]. [73] R. D. Ferguson, M. Massimi, E. A. Crist and K. A. Moffatt. Craving, creating, and constructing comfort: insights and opportunities for technology in hospice. in CSCW ’14, pages 1479–1490, New York, NY, USA. ACM, 2014. ISBN: 978-1- 4503-2540-0. DOI: 10.1145/2531602.2531631. URL: http://doi.acm.org/ 10.1145/2531602.2531631. [Online; accessed 2016-08-08]. [74] O. Feyisetan and E. Simperl. Social incentives in paid collaborative crowdsourcing. ACM Trans. Intell. Syst. Technol., 8(6), july 2017. ISSN: 2157-6904. DOI: 10.1145/ 3078852. URL: https://doi.org/10.1145/3078852.

186 Bibliography

[75] J. Finn. An exploration of helping processes in an online self-help group focusing on issues of disability. en. Health Social Work, 24(3):220–231, 01 august 1999. ISSN: 0360-7283, 1545-6854. DOI: 10.1093/hsw/24.3.220. PMID: 10505283. [76] K. E. Fisher, K. T. Unruh and J. C. Durrance. Information communities: characteristics gleaned from studies of three online networks. Proceedings of the American Society for Information Science and Technology, 40(1):298–305, 2003. [77] A. Forte, N. Andalibi and R. Greenstadt. Privacy, anonymity, and perceived risk in open collaboration: a study of tor users and wikipedians. in CSCW ’17, pages 1800– 1811, New York, NY, USA. ACM, 2017. ISBN: 978-1-4503-4335-0. DOI: 10.1145/ 2998181.2998273. URL: http://doi.acm.org/10.1145/2998181.2998273. [Online; accessed 2017-04-22]. [78] D. Freitag. Machine learning for information extraction in informal domains. en. Machine Learning, 39(2-3):169–202, 01 may 2000. ISSN: 0885-6125, 1573-0565. DOI: 10.1023/A:1007601113994. [79] J. Frost, I. E. Vermeulen and N. Beekers. Anonymity versus privacy: selective information sharing in online cancer communities. Journal of Medical Internet Re- search, 16(5), 14 may 2014. ISSN: 1439-4456. DOI: 10.2196/jmir.2684. URL: http : / / www . ncbi . nlm . nih . gov / pmc / articles / PMC4051744/. PMID: 24828114 PMCID: PMC4051744. [80] U. Gadiraju, R. Kawase, S. Dietze and G. Demartini. Understanding malicious behavior in crowdsourcing platforms: the case of online surveys. in Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, CHI ’15, pages 1631–1640, Seoul, Republic of Korea. Association for Computing Ma- chinery, 2015. ISBN: 9781450331456. DOI: 10 . 1145 / 2702123 . 2702443. URL: https://doi.org/10.1145/2702123.2702443. [81] E. R. Galaif and S. Sussman. For whom does Alcoholics Anonymous work? eng. Int J Addict, 30(2):161–184, january 1995. ISSN: 0020-773X. DOI: 10 . 3109 / 10826089509060740. [82] F. S. Gharehchopogh and Z. A. Khalifelu. Analysis and evaluation of unstructured data: text mining versus natural language processing. in pages 1–4. 2011 5th Inter- national Conference on Application of Information and Communication Technolo- gies (AICT), october 2011. DOI: 10.1109/ICAICT.2011.6111017. [83] V. Girotto. Collective creativity through a micro-tasks crowdsourcing approach. in Proceedings of the 19th ACM Conference on Computer Supported Coopera- tive Work and Social Computing Companion, CSCW ’16 Companion, pages 143– 146, San Francisco, California, USA. Association for Computing Machinery, 2016. ISBN: 9781450339506. DOI: 10.1145/2818052.2874356. URL: https://doi- org.ezp2.lib.umn.edu/10.1145/2818052.2874356.

187 Bibliography

[84] J. Goncalves, D. Ferreira, S. Hosio, Y.Liu, J. Rogstadius, H. Kukka and V.Kostakos. Crowdsourcing on the spot: altruistic use of public displays, feasibility, performance, and behaviours. in UbiComp ’13, pages 753–762, New York, NY, USA. ACM, 2013. ISBN: 978-1-4503-1770-2. DOI: 10.1145/2493432.2493481. URL: http : / / doi . acm . org / 10 . 1145 / 2493432 . 2493481. event-place: Zurich, Switzerland. [85] J. Goncalves, S. Hosio, D. Ferreira, T. Anagnostopoulos and V. Kostakos. Bazaar: a situated crowdsourcing market. in UbiComp/ISWC’15 Adjunct, pages 1385–1390, New York, NY, USA. ACM, 2015. ISBN: 978-1-4503-3575-1. DOI: 10 . 1145 / 2800835.2800974. URL: http://doi.acm.org/10.1145/2800835.2800974. event-place: Osaka, Japan. [86] J. Goncalves, S. Hosio, D. Ferreira and V. Kostakos. Game of words: tagging places through crowdsourcing on public displays. in DIS ’14, pages 705–714, New York, NY, USA. ACM, 2014. DOI: 10.1145/2598510.2598514. URL: http://doi. acm.org/10.1145/2598510.2598514. event-place: Vancouver, BC, Canada. [87] D. González, R. Motz and L. Tansini. Recommendations given from socially-connected people. en. in Y. T. Demey and H. Panetto, editorsLecture Notes in Com- puter Science, pages 649–655. Springer Berlin Heidelberg, 2013. ISBN: 978-3-642- 41033-8. [88] K. Gushima, M. Sakamoto and T. Nakajima. Community-based crowdsourcing to increase a community’s well-being. in Proceedings of the 18th International Con- ference on Information Integration and Web-Based Applications and Services, ii- WAS ’16, pages 1–6, Singapore, Singapore. Association for Computing Machin- ery, 2016. ISBN: 9781450348072. DOI: 10.1145/3011141.3011173. URL: https: //doi-org.ezp3.lib.umn.edu/10.1145/3011141.3011173. [89] K. Hara and J. E. Froehlich. Characterizing and visualizing physical world accessibility at scale using crowdsourcing, computer vision, and machine learning. SIGACCESS Access. Comput., (113):13–21, november 2015. ISSN: 1558-2337. DOI: 10.1145/2850440.2850442. [90] B. Hoisl, W. Aigner and S. Miksch. Social rewarding in wiki systems – motivating the community. en. in Lecture Notes in Computer Science, pages 362–371. Interna- tional Conference on Online Communities and Social Computing, Springer, Berlin, Heidelberg, 22 july 2007. ISBN: 978-3-540-73256-3. DOI: 10.1007/978-3-540- 73257-0_40. URL: https://link.springer.com/chapter/10.1007/978-3- 540-73257-0_40. [Online; accessed 2017-07-07].

188 Bibliography

[91] H. Hong, E. Gilbert, G. D. Abowd and R. I. Arriaga. In-group questions and out- group answers: crowdsourcing daily living advice for individuals with autism. in CHI ’15, pages 777–786, New York, NY, USA. ACM, 2015. ISBN: 978-1-4503- 3145-6. DOI: 10.1145/2702123.2702402. URL: http://doi.acm.org/10. 1145/2702123.2702402. [Online; accessed 2016-04-12]. [92] S. Hosio, J. Goncalves, V. Lehdonvirta, D. Ferreira and V. Kostakos. Situated crowdsourcing using a market model. in UIST ’14, pages 55–64, New York, NY, USA. ACM, 2014. ISBN: 978-1-4503-3069-5. DOI: 10.1145/2642918.2647362. URL: http://doi.acm.org/10.1145/2642918.2647362. event-place: Hon- olulu, Hawaii, USA. [93] M. Hossain. Users’ motivation to participate in online crowdsourcing platforms. in pages 310–315. 2012 International Conference on Innovation Management and Technology Research, may 2012. DOI: 10.1109/ICIMTR.2012.6236409. [94] G. Hsieh and R. Kocielnik. You get who you pay for: the impact of incentives on participation bias. in Proceedings of the 19th ACM Conference on Computer- Supported Cooperative Work Social Computing, CSCW ’16, pages 823–835, San Francisco, California, USA. Association for Computing Machinery, 2016. ISBN: 9781450335928. DOI: 10.1145/2818048.2819936. URL: https://doi.org/ 10.1145/2818048.2819936. [95] H.-F. Hsieh and S. E. Shannon. Three approaches to qualitative content analysis. en. Qualitative Health Research, 15(9):1277–1288, 01 november 2005. ISSN: 1049- 7323. DOI: 10.1177/1049732305276687. [96] Y. Huang, Y. Huang, N. Xue and J. P. Bigham. Leveraging complementary contributions of different workers for eﬀicient crowdsourcing of video captions. in CHI ’17, pages 4617–4626, New York, NY, USA. ACM, 2017. ISBN: 978-1-4503-4655- 9. DOI: 10.1145/3025453.3026032. URL: http://doi.acm.org/10.1145/ 3025453.3026032. [97] J. Huh. Clinical questions in online health communities: the case of ”see your doctor” threads. in CSCW ’15, pages 1488–1499, New York, NY, USA. ACM, 2015. ISBN: 978-1-4503-2922-4. DOI: 10.1145/2675133.2675259. URL: http://doi. acm.org/10.1145/2675133.2675259. [Online; accessed 2016-08-08]. [98] K. Humphreys, L. A. Kaskutas and C. Weisner. The alcoholics anonymous aﬀil- iation scale: development, reliability, and norms for diverse treated and untreated populations. en. Alcoholism: Clinical and Experimental Research, 22(5):974–978, 01 august 1998. ISSN: 1530-0277. DOI: 10.1111/j.1530-0277.1998.tb03691. x.

189 Bibliography

[99] K. Ikeda and M. S. Bernstein. Pay it backward: per-task payments on crowdsourcing platforms reduce productivity. in Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, CHI ’16, pages 4111–4121, San Jose, Cal- ifornia, USA. Association for Computing Machinery, 2016. ISBN: 9781450333627. DOI: 10.1145/2858036.2858327. URL: https://doi.org/10.1145/2858036. 2858327. [100] In Brief: Substance Use and Suicide: | SAMHSA Publications. URL: https : / / store.samhsa.gov/product/In- Brief- Substance- Use- and- Suicide- /sma16-4935 (urlseen 17/10/2019). [101] Z. Jin, M. R. Anderson, M. Cafarella and H. V. Jagadish. Foofah: transforming data by example. in SIGMOD ’17, pages 683–698, New York, NY, USA. ACM, 2017. ISBN: 978-1-4503-4197-4. DOI: 10.1145/3035918.3064034. URL: http: //doi.acm.org/10.1145/3035918.3064034. [102] T. Joachims. Text categorization with support vector machines: learning with many relevant features. en. in C. Nédellec and C. Rouveirol, editorsMachine Learning: ECML-98. volume 1398, pages 137–142. Springer Berlin Heidelberg, Berlin, Hei- delberg, 1998. ISBN: 978-3-540-64417-0. URL: http://link.springer.com/10. 1007/BFb0026683. DOI: 10.1007/BFb0026683. [103] D. G. Johnson and K. Miller. Anonymity, pseudonymity, or inescapable identity on the net (abstract). in ACM POLICY ’98, pages 37–38, New York, NY, USA. ACM, 1998. ISBN: 978-1-58113-038-6. DOI: 10 . 1145 / 276755 . 276774. URL: http : //doi.acm.org/10.1145/276755.276774. [Online; accessed 2017-03-29]. [104] C. Jonathan and M. F. Mokbel. Stella: geotagging images via crowdsourcing. in SIGSPATIAL ’18, pages 169–178, New York, NY, USA. ACM, 2018. ISBN: 978- 1-4503-5889-7. DOI: 10.1145/3274895.3274902. URL: http://doi.acm.org/ 10.1145/3274895.3274902. event-place: Seattle, Washington. [105] E. Jun, G. Hsieh and K. Reinecke. Types of motivation affect study selection, attention, and dropouts in online experiments. Proc. ACM Hum.-Comput. Interact., 1(CSCW), december 2017. DOI: 10.1145/3134691. URL: https://doi.org/ 10.1145/3134691. [106] R. Kang, S. Brown and S. Kiesler. Why do people seek anonymity on the internet?: informing policy and design. in CHI ’13, pages 2657–2666, New York, NY, USA. ACM, 2013. ISBN: 978-1-4503-1899-0. DOI: 10.1145/2470654.2481368. URL: http://doi.acm.org/10.1145/2470654.2481368. [Online; accessed 2017- 01-27].

190 Bibliography

[107] R. Kang, L. Dabbish and K. Sutton. Strangers on your phone: why people use anonymous communication applications. in CSCW ’16, pages 359–370, New York, NY, USA. ACM, 2016. DOI: 10.1145/2818048.2820081. URL: http://doi. acm.org/10.1145/2818048.2820081. [108] N. Kaufmann and T. Schulze. Worker motivation in crowdsourcing and human computation, 0. URL: http://www.humancomputation.com/Program.html. [Online; accessed 2019-05-13]. [109] N. Kaufmann, T. Schulze and D. Veit. More than fun and money. worker motivation in crowdsourcing – a study on mechanical turk. AMCIS 2011 Proceedings - All Submissions, 06 august 2011. URL: https://aisel.aisnet.org/amcis2011_ submissions/340. [110] C. Kiene, A. Monroy-Hernández and B. M. Hill. Surviving an ”eternal september”: how an online community managed a surge of newcomers. in CHI ’16, pages 1152– 1156, New York, NY, USA. ACM, 2016. ISBN: 978-1-4503-3362-7. DOI: 10.1145/ 2858036.2858356. URL: http://doi.acm.org/10.1145/2858036.2858356. [Online; accessed 2016-08-08]. [111] P. G. Kilner and C. M. Hoadley. Anonymity options and professional participation in an online community of practice. in CSCL ’05, pages 272–280, Taipei, Taiwan. International Society of the Learning Sciences, 2005. ISBN: 978-0-8058-5782-5. URL: http://dl.acm.org/citation.cfm?id=1149293.1149328. [Online; accessed 2017-03-29]. [112] D. S. Kirk, A. Sellen and X. Cao. Home video communication: mediating ’close- ness’. in pages 135–144. ACM, 2010. ISBN: 978-1-60558-795-0. [113] A. Kittur, E Chi and B. Suh. Crowdsourcing for usability: using micro-task markets for rapid, remote, and low-cost user measurements. Proc. CHI 2008, 2008. [114] H. D. Kleber, R. D. Weiss, R. F. Anton, T. P. George, S. F. Greenfield, T. R. Kosten, C. P. O’Brien, B. J. Rounsaville, E. C. Strain, D. M. Ziedonis, G. Hennessy, H. S. Connery, J. S. McIntyre, S. C. Charles, D. J. Anzia, I. A. Cook, M. T. Finnerty, B. R. Johnson, J. E. Nininger, P. Summergrad, S. M. Woods, J. Yager, R. Pyles, C. D. Cross, R. Peele, J. P. D. Shemo, L. Lurie, R. D. Walker, M. A. Barnovitz, S. H. Gray, S. Saxena, T. Tonnu, R. Kunkle, A. B. Albert, L. J. Fochtmann, C. Hart, D. Regier, W. G. on Substance Use Disorders, A. P. Association and S. C. on Practice Guidelines. Treatment of patients with substance use disorders, second edition. american psychiatric association. eng. The American Journal of Psychiatry, 164(4 Suppl):5–123, april 2007. ISSN: 0002-953X. PMID: 17569411.

191 Bibliography

[115] M. Kobayashi, S. Arita, T. Itoko, S. Saito and H. Takagi. Motivating multi- gen- erational crowd workers in social-purpose work. in Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work Social Computing, CSCW ’15, pages 1813–1824, Vancouver, BC, Canada. Association for Computing Ma- chinery, 2015. ISBN: 9781450329224. DOI: 10 . 1145 / 2675133 . 2675255. URL: https://doi-org.ezp2.lib.umn.edu/10.1145/2675133.2675255. [116] M. KOSONEN, C. GAN, M. VANHALA and K. BLOMQVIST. User motivation and knowledge sharing in idea crowdsourcing. International Journal of Innovation Management, 18(05):1450031, 2014. DOI: 10.1142/S1363919614500315. eprint: https://doi.org/10.1142/S1363919614500315. URL: https://doi.org/ 10.1142/S1363919614500315. [117] Y. Kow and B. Nardi. Regulating anti-social behavior on the internet: the example of league of legends, 2013. [118] R. A. Krishna, K. Hata, S. Chen, J. Kravitz, D. A. Shamma, L. Fei-Fei and M. S. Bernstein. Embracing error to enable rapid crowdsourcing. in Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, CHI ’16, San Jose, California, USA. Association for Computing Machinery, 2016. DOI: 10 . 1145 / 2858036.2858115. URL: https://doi.org/10.1145/2858036.2858115. [119] A. Kulkarni, M. Can and B. Hartmann. Collaboratively crowdsourcing workflows with turkomatic. in Proceedings of the ACM 2012 Conference on Computer Sup- ported Cooperative Work, CSCW ’12, pages 1003–1012, Seattle, Washington, USA. Association for Computing Machinery, 2012. ISBN: 9781450310864. DOI: 10.1145/ 2145204.2145354. URL: https://doi-org.ezp2.lib.umn.edu/10.1145/ 2145204.2145354. [120] P. E. Kummervold, D. Gammon, S. Bergvik, J.-A. K. Johnsen, T. Hasvold and J. H. Rosenvinge. Social support in a wired world: use of online mental health forums in norway. eng. Nordic Journal of Psychiatry, 56(1):59–65, 2002. ISSN: 0803-9488. DOI: 10.1080/08039480252803945. PMID: 11869468. [121] M. Kutlu, T. McDonnell, A. Sheshadri, T. Elsayed and M. Lease. Mix and match: collaborative expert-crowd judging for building test collections accurately and af- fordably. In Proceedings of the 1st Biannual Conference on the Design of Experi- mental Search Information REtrieval Systems (DESIRES):42–46, 2018. [122] E. Kuzey and G. Weikum. Extraction of temporal facts and events from wikipedia. in TempWeb ’12, pages 25–32, New York, NY, USA. ACM, 2012. ISBN: 978-1- 4503-1188-5. DOI: 10.1145/2169095.2169101. URL: http://doi.acm.org/ 10.1145/2169095.2169101.

192 Bibliography

[123] J. R. Landis and G. G. Koch. The measurement of observer agreement for cate- gorical data. Biometrics, 33(1):159–174, 1977. ISSN: 0006-341X. DOI: 10.2307/ 2529310. [124] D. Laniado, A. Kaltenbrunner, C. Castillo and M. F. Morell. Emotions and dia- logue in a peer-production community: the case of wikipedia. in WikiSym ’12, 9:1– 9:10, New York, NY, USA. ACM, 2012. ISBN: 978-1-4503-1605-7. DOI: 10.1145/ 2462932.2462944. URL: http://doi.acm.org/10.1145/2462932.2462944. [Online; accessed 2016-08-08]. [125] N. Lapidot-Lefler and A. Barak. Effects of anonymity, invisibility, and lack ofeye- contact on toxic online disinhibition. Computers in Human Behavior, 28(2):434– 443, march 2012. ISSN: 0747-5632. DOI: 10.1016/j.chb.2011.10.014. [126] P. I. A. B. Laudet. Attitudes and Beliefs About 12-Step Groups Among Addiction Treatment Clients and Clinicians: Toward Identifying Obstacles to Participation. Substance Use & Misuse, 38(14):2017–2047, january 2003. ISSN: 1082-6084. DOI: 10.1081/JA-120025124. URL: https://doi.org/10.1081/JA-120025124 (urlseen 07/11/2019). [127] M. Lea, R. Spears and D. de Groot. Knowing me, knowing you: anonymity effects on social identity processes within groups. en. Personality and Social Psy- chology Bulletin, 27(5):526–537, 01 may 2001. ISSN: 0146-1672. DOI: 10.1177/ 0146167201275002. [128] A. Leavitt. ”this is a throwaway account”: temporary technical identities and perceptions of anonymity in a massive online community. in CSCW ’15, pages 317– 327, New York, NY, USA. ACM, 2015. ISBN: 978-1-4503-2922-4. DOI: 10.1145/ 2675133.2675175. URL: http://doi.acm.org/10.1145/2675133.2675175. [Online; accessed 2017-02-01]. [129] M. K. Lee, N. Fruchter and L. Dabbish. Making decisions from a distance: the impact of technological mediation on riskiness and dehumanization. in CSCW ’15, pages 1576–1589, New York, NY, USA. ACM, 2015. ISBN: 978-1-4503-2922-4. DOI: 10.1145/2675133.2675288. URL: http://doi.acm.org/10.1145/ 2675133.2675288. [Online; accessed 2016-04-26]. [130] W. LEHNERT, S. SODERLAND, D. ARONOW, F. FENG and A. SHMUELI. In- ductive text classification for medical applications. Journal of Experimental Theo- retical Artificial Intelligence, 7(1):49–80, 01 january 1995. ISSN: 0952-813X. DOI: 10.1080/09528139508953800. [131] C. Lemnaru and R. Potolea. Imbalanced classification problems: systematic study, issues and best practices. en. in R. Zhang, J. Zhang, Z. Zhang, J. Filipe and J. Cordeiro, editorsLecture Notes in Business Information Processing, pages 35–50. Springer Berlin Heidelberg, 2012. ISBN: 978-3-642-29958-2.

193 Bibliography

[132] B. Li, E. Erdin, M. H. Gunes, G. Bebis and T. Shipley. An overview of anonymity technology usage. Computer Communications, 36(12):1269–1283, 01 july 2013. ISSN: 0140-3664. DOI: 10.1016/j.comcom.2013.04.009. [133] H. Li, B. Zhao and A. Fuxman. The wisdom of minority: discovering and targeting the right group of workers for crowdsourcing. in WWW ’14, pages 165–176, New York, NY, USA. ACM, 2014. ISBN: 978-1-4503-2744-2. DOI: 10.1145/2566486. 2568033. URL: http://doi.acm.org/10.1145/2566486.2568033. event-place: Seoul, Korea. [134] X. Li, K.-m. Chang, Y. Yuan and A. Hauptmann. Massive open online proctor: protecting the credibility of moocs certificates. in CSCW ’15, pages 1129–1137, New York, NY, USA. ACM, 2015. ISBN: 978-1-4503-2922-4. DOI: 10.1145/2675133. 2675245. URL: http://doi.acm.org/10.1145/2675133.2675245. [135] H. Lieberman, K. Dinakar and B. Jones. Crowdsourced ethics with personalized story matching. in CHI EA ’13, pages 709–714, New York, NY, USA. ACM, 2013. ISBN: 978-1-4503-1952-2. DOI: 10.1145/2468356.2468481. URL: http://doi. acm.org/10.1145/2468356.2468481. [136] X. Ling and D. S. Weld. Temporal information extraction. en. in Twenty-Fourth AAAI Conference on Artificial Intelligence, 05 july 2010. URL: https://www. aaai.org/ocs/index.php/AAAI/AAAI10/paper/view/1805. [Online; accessed 2018-09-21]. [137] H. Liu, N. Wang and X. Ren. Crowdsr: a crowd enabled system for semantic recovering of web tables. en. in X. L. Dong, X. Yu, J. Li and Y. Sun, editorsLecture Notes in Computer Science, pages 581–583. Springer International Publishing, 2015. ISBN: 978-3-319-21042-1. [138] J. Lofland, D. Snow, L. Anderson and L. H. Lofland. Analyzing Social Settings: A Guide to Qualitative Observation and Analysis. Wadsworth/Thomas Learning, Belmont, CA, 4th edition, 2006. [139] T. Ludwig, C. Kotthaus, C. Reuter, v. S. Dongen and V. Pipek. Situated crowdsourcing during disasters: managing the tasks of spontaneous volunteers through public displays. International Journal of Human-Computer Studies. Special Issue on Mobile and Situated Crowdsourcing, 102:103–121, 01 june 2017. ISSN: 1071- 5819. DOI: 10.1016/j.ijhcs.2016.09.008. [140] I. O. Lund, E. Sundin, C. Konijnenberg, K. Rognmo, P. Martinez and A. Fielder. Harm to Others From Substance Use and Abuse. Subst Abuse, 9(Suppl 2):119–124, may 2016. ISSN: 1178-2218. DOI: 10.4137/SART.S39722. URL: https://www. ncbi.nlm.nih.gov/pmc/articles/PMC4861007/ (urlseen 17/10/2019).

194 Bibliography

[141] X. Ma, J. Hancock and M. Naaman. Anonymity, intimacy and self-disclosure in social media. in CHI ’16, pages 3857–3869, New York, NY, USA. ACM, 2016. ISBN: 978-1-4503-3362-7. DOI: 10.1145/2858036.2858414. URL: http://doi. acm.org/10.1145/2858036.2858414. [Online; accessed 2017-03-27]. [142] X. Ma, G. Chen and J. Xiao. Analysis of an online health social network. in IHI ’10, pages 297–306, New York, NY, USA. ACM, 2010. ISBN: 978-1-4503-0030-8. DOI: 10.1145/1882992.1883035. URL: http://doi.acm.org/10.1145/1882992. 1883035. [Online; accessed 2016-04-12]. [143] D. Maloney-Krichmar and J. Preece. A Multilevel Analysis of Sociability, Us- ability, and Community Dynamics in an Online Health Community. ACM Trans. Comput.-Hum. Interact., 12(2):201–232, june 2005. ISSN: 1073-0516. DOI: 10 . 1145/1067860.1067864. URL: http://doi.acm.org/10.1145/1067860. 1067864 (urlseen 12/04/2016). [144] J. Mankoff, K. Kuksenok, S. Kiesler, J. A. Rode and K. Waldman. Competing online viewpoints and models of chronic illness. in CHI ’11, pages 589–598, New York, NY, USA. ACM, 2011. ISBN: 978-1-4503-0228-9. DOI: 10.1145/1978942. 1979027. URL: http://doi.acm.org/10.1145/1978942.1979027. [Online; accessed 2016-04-19]. [145] A. Mao, E. Kamar, Y. Chen, E. Horvitz, M. E. Schwamb, C. J. Lintott and A. M. Smith. Volunteering versus work for pay: incentives and tradeoffs in crowdsourcing. en. in First AAAI Conference on Human Computation and Crowdsourcing, 2013. URL: https://www.aaai.org/ocs/index.php/HCOMP/HCOMP13/paper/ view/7497. Retrieved January 9, 2020 from. [146] S. L. Marhefka, E. R. Buhi, J. Baldwin, H. Chen, A. Johnson, V. Lynn and R. Glueckauf. Effectiveness of healthy relationships video-group—a videoconferencing group intervention for women living with hiv: preliminary findings from a randomized controlled trial. Telemedicine and e-Health, 20(2):128–134, 16 november 2013. ISSN: 1530-5627. DOI: 10.1089/tmj.2013.0072. [147] S. L. Marhefka, H. J. Fuhrmann, P. Gilliam, B. Lopez and J. Baldwin. Interest in, concerns about, and preferences for potential video-group delivery of an effective behavioral intervention among women living with hiv. en. AIDS and Be- havior, 16(7):1961–1969, 24 september 2011. ISSN: 1090-7165, 1573-3254. DOI: 10.1007/s10461-011-0040-x. [148] S. L. Marhefka, S. Iziduh, H. J. Fuhrmann, B. Lopez, R. Glueckauf, V. Lynn and J. Baldwin. Internet-based video-group delivery of healthy relationships – a “preven- tion with positives” intervention: report on a single group pilot test among women living with hiv. AIDS Care, 25(7):904–909, 08 may 2013. ISSN: 0954-0121. DOI: 10.1080/09540121.2013.793266. PMID: 23713756.

195 Bibliography

[149] J. Martins, J. Carilho, O. Schnell, C. Duarte, F. M. Couto, L. Carriço and T. Guer- reiro. Friendsourcing the unmet needs of people with dementia. in W4A ’14, 35:1– 35:4, New York, NY, USA. ACM, 2014. ISBN: 978-1-4503-2651-3. DOI: 10.1145/ 2596695.2596716. URL: http://doi.acm.org/10.1145/2596695.2596716. event-place: Seoul, Korea. [150] G. T. Marx. What’s in a name? some reflections on the sociology of anonymity. The Information Society, 15(2):99–112, 01 may 1999. ISSN: 0197-2243. DOI: 10.1080/ 019722499128565. [151] E. Marziali and P. Donahue. Caring for others: internet video-conferencing group intervention for family caregivers of older adults with neurodegenerative disease. en. The Gerontologist, 46(3):398–403, 01 june 2006. ISSN: 0016-9013, 1758-5341. DOI: 10.1093/geront/46.3.398. PMID: 16731880. [152] M. Massimi. Exploring remembrance and social support behavior in an online be- reavement support group. in CSCW ’13, pages 1169–1180, 2013. ISBN: 978-1- 4503-1331-5. DOI: 10.1145/2441776.2441908. URL: http://doi.acm.org/ 10.1145/2441776.2441908. [Online; accessed 2014-09-19]. [153] M. Massimi, J. L. Bender, H. O. Witteman and O. H. Ahmed. Life Transitions and Online Health Communities: Reflecting on Adoption, Use, and Disengagement. in Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing, CSCW ’14, pages 1491–1501, New York, NY, USA. ACM, 2014. ISBN: 978-1-4503-2540-0. DOI: 10.1145/2531602.2531622. URL: http://doi.acm.org/10.1145/2531602.2531622 (urlseen 12/04/2016). [154] B. McLernon and N. Kushmerick. Transductive Pattern Learning for Information Extraction. en. techreport, april 2006. URL: http : / / www . dtic . mil / docs / citations/ADA456766. [Online; accessed 2018-08-18]. [155] H. M. Mentis, A. Rahim and P. Theodore. Crafting the image in surgical telemedicine. in CSCW ’16, pages 744–755, New York, NY, USA. ACM, 2016. ISBN: 978-1- 4503-3592-8. DOI: 10.1145/2818048.2819978. URL: http://doi.acm.org/ 10.1145/2818048.2819978. [Online; accessed 2016-04-26]. [156] E. Meyers, L. Garcia and E. Marziali. Using online videoconferencing for connecting caregivers of people with dementia. English. Journal of CyberTherapy and Rehabilitation, 4(2):223–225, 22 june 2011. ISSN: 17849934. [157] Microsoft cognitive services - face api. URL: https://www.microsoft.com/ cognitive-services/en-us/face-api. [Online; accessed 2017-04-27]. [158] D. Mladenic. Turning Yahoo into an Automatic Web-Page Classifier. 13th European Conference on Artificial Intelligence Young Researcher Paper. 1998.

196 Bibliography

[159] R. J. Mooney and R. Bunescu. Mining knowledge from text using information extraction. SIGKDD Explor. Newsl., 7(1):3–10, june 2005. ISSN: 1931-0145. DOI: 10. 1145/1089815.1089817. [160] M. R. Morris, K. Inkpen and G. Venolia. Remote shopping advice: enhancing in-store shopping with social technologies. in CSCW ’14, pages 662–673, New York, NY, USA. ACM, 2014. ISBN: 978-1-4503-2540-0. DOI: 10.1145/2531602. 2531707. URL: http://doi.acm.org/10.1145/2531602.2531707. event-place: Baltimore, Maryland, USA. [161] M. R. Morris, J. Teevan and K. Panovich. A comparison of information seeking using search engines and social networks. en. in Fourth International AAAI Con- ference on Weblogs and Social Media, 16 may 2010. URL: https://www.aaai. org/ocs/index.php/ICWSM/ICWSM10/paper/view/1518. [Online; accessed 2019-05-13]. [162] R. R. Morris, S. M. Schueller and R. W. Picard. Eﬀicacy of a web-based, crowdsourced peer-to-peer cognitive reappraisal platform for depression: randomized controlled trial. eng. Journal of Medical Internet Research, 17(3):e72, 30 march 2015. ISSN: 1438-8871. DOI: 10.2196/jmir.4167. [163] C. Moser, M. Peterhansl, T. Kargl and M. Tscheligi. The potentials of gamification to motivate older adults to participate in a p2p support exchange platform. in CHI PLAY ’15, pages 655–660, New York, NY, USA. ACM, 2015. ISBN: 978-1-4503- 3466-2. DOI: 10.1145/2793107.2810326. URL: http://doi.acm.org/10. 1145/2793107.2810326. [164] M. Muller. Curiosity, creativity, and surprise as analytic tools: grounded theory method. en. in J. S. Olson and W. A. Kellogg, editorsWays of Knowing in HCI, pages 25–48. Springer New York, 2014. ISBN: 978-1-4939-0377-1. URL: http : //link.springer.com/chapter/10.1007/978-1-4939-0378-8_2. [Online; accessed 2015-07-30]. [165] S. Nath, P. Dayama, D. Garg, Y. Narahari and J. Zou. Threats and Trade-Offs in Resource Critical Crowdsourcing Tasks Over Networks. en. in Twenty-Sixth AAAI Conference on Artificial Intelligence, july 2012. URL: https://www.aaai.org/ ocs/index.php/AAAI/AAAI12/paper/view/4807 (urlseen 11/01/2020). [166] M. W. Newman, D. Lauterbach, S. A. Munson, P. Resnick and M. E. Morris. ”it’s not that i don’t have problems, i’m just not putting them on facebook”: challenges and opportunities in using online social networks for health. in CSCW ’11, pages 341–350, New York, NY, USA. ACM, 2011. ISBN: 978-1-4503-0556-3. DOI: 10.1145/1958824.1958876. URL: http://doi.acm.org/10.1145/1958824. 1958876. [Online; accessed 2016-04-13].

197 Bibliography

[167] H. Nissenbaum. The meaning of anonymity in an information age. en. The Infor- mation Society, 15(2):141–144, may 1999. ISSN: 0197-2243, 1087-6537. DOI: 10. 1080/019722499128592. [168] . L. S. NW, S. 800Washington and D. U.-.-. |. M.-.-. |. F.-.-. |. M. Inquiries. How Me- chanical Turk Works. en-US. URL: https://www.pewresearch.org/internet/ 2016/07/11/research-in-the-crowdsourcing-age-a-case-study/pi_ 2016-07-11_mechanical-turk_0-01/ (urlseen 12/01/2020). [169] S. Nylander. Changing my life one step at a time: using the twelve step program as design inspiration for long term lifestyle change. in NordiCHI ’12, pages 711– 720, New York, NY, USA. ACM, 2012. ISBN: 978-1-4503-1482-4. DOI: 10.1145/ 2399016.2399125. URL: http://doi.acm.org/10.1145/2399016.2399125. [Online; accessed 2015-09-28]. [170] S. O’Halloran. Participant Observation of Alcoholics Anonymous: Contrasting Roles of the Ethnographer and Ethnomethodologist. The Qualitative Report, 8(1):81–99, june 2003. ISSN: 1052-0147. URL: https://nsuworks.nova.edu/tqr/vol8/ iss1/6. [171] E. Omernick and S. O. Sood. The impact of anonymity in online communities. in pages 526–535. 2013 International Conference on Social Computing, september 2013. DOI: 10.1109/SocialCom.2013.80. [172] A. Orita and H. Hada. Is that really you?: an approach to assure identity without revealing real-name online. in DIM ’09, pages 17–20, New York, NY, USA. ACM, 2009. ISBN: 978-1-60558-786-8. DOI: 10.1145/1655028.1655034. URL: http: //doi.acm.org/10.1145/1655028.1655034. [Online; accessed 2017-01-27]. [173] J. Ou, Y. Shi, J. Wong, S. R. Fussell and J. Yang. Combining audio and video to predict helpers’ focus of attention in multiparty remote collaboration on physical tasks. in ICMI ’06, pages 217–224, New York, NY, USA. ACM, 2006. ISBN: 978- 1-59593-541-0. DOI: 10.1145/1180995.1181040. URL: http://doi.acm.org/ 10.1145/1180995.1181040. [Online; accessed 2016-04-26]. [174] P. C. Ouimette, J. W. Finney and R. H. Moos. Twelve-step and cognitive-behavioral treatment for substance abuse: A comparison of treatment effectiveness. Journal of Consulting and Clinical Psychology, 65(2):230–240, 1997. DOI: 10.1037/0022- 006X.65.2.230. [175] M. Pagano, K. Friend, S. Tonigan and R. Stout. Helping other alcoholics in alcoholics anonymous and drinking outcomes: findings from project match. Journal of studies on alcohol, 65(6):766–773, november 2004. ISSN: 0096-882X. PMID: 15700515 PMCID: PMC3008319.

198 Bibliography

[176] C. Paine, A. N. Joinson, T. Buchanan and U.-D. Reips. Privacy and self-disclosure online. in CHI EA ’06, pages 1187–1192, New York, NY, USA. ACM, 2006. ISBN: 978-1-59593-298-3. DOI: 10.1145/1125451.1125674. URL: http://doi.acm. org/10.1145/1125451.1125674. [Online; accessed 2017-03-29]. [177] M. Paskuda and M. Lewkowicz. Anonymous quorans are still quorans, just anonymous. in CT ’15, pages 9–18, New York, NY, USA. ACM, 2015. ISBN: 978-1- 4503-3460-0. DOI: 10.1145/2768545.2768551. URL: http://doi.acm.org/ 10.1145/2768545.2768551. [Online; accessed 2017-03-29]. [178] U. Pavalanathan and M. De Choudhury. Identity management and mental health discourse in social media. in WWW ’15 Companion, pages 315–321, Republic and Canton of Geneva, Switzerland. International World Wide Web Conferences Steering Committee, 2015. ISBN: 978-1-4503-3473-0. DOI: 10.1145/2740908. 2743049. URL: http://dx.doi.org/10.1145/2740908.2743049. [Online; accessed 2015-12-17]. [179] S. Petronio. Communication privacy management theory. en. in The International Encyclopedia of Interpersonal Communication. John Wiley Sons, Inc., 2015. ISBN: 978-1-118-54019-0. URL: http://onlinelibrary.wiley.com/doi/10.1002/ 9781118540190.wbeic132/abstract. DOI: 10.1002/9781118540190.wbeic132. [180] J. F. K. PhD, J. D. Y. BA and M. G. M. PhD. Treatment Staff Referrals, Participation Expectations, and Perceived Benefits and Barriers to Adolescent Involvement in Twelve-Step Groups. Alcoholism Treatment Quarterly, 26(4):427–449, september 2008. ISSN: 0734-7324. DOI: 10.1080/07347320802347053. URL: https://doi. org/10.1080/07347320802347053 (urlseen 12/11/2019). [181] L. A. R. Ph.D, S. B. Ph.D, E. G. Ph.D, E. H. Ph.D, R. G. M.A, A. V. B.S, A. C. L. M.A, A. G. M.D and K. M. M.D. State-of-the-art behavioral and pharmacological treatments for alcohol use disorder. The American Journal of Drug and Alcohol Abuse, 45(2):124–140, march 2019. ISSN: 0095-2990. DOI: 10.1080/00952990. 2018.1528265. URL: https://doi.org/10.1080/00952990.2018.1528265 (urlseen 07/11/2019). [182] C. Phua, D. Alahakoon and V. Lee. Minority report in fraud detection: classification of skewed data. SIGKDD Explor. Newsl., 6(1):50–59, june 2004. ISSN: 1931-0145. DOI: 10.1145/1007730.1007738. [183] J. Piazza and J. M. Bering. The effects of perceived anonymity on altruistic pun- ishment. en. Evolutionary Psychology, 6(3):147470490800600314, 01 july 2008. ISSN: 1474-7049. DOI: 10.1177/147470490800600314.

199 Bibliography

[184] L. Posch, A. Bleier, C. M. Lechner, D. Danner, F. Flöck and M. Strohmaier. Mea- suring motivations of crowdworkers: the multidimensional crowdworker motivation scale. Trans. Soc. Comput., 2(2), september 2019. ISSN: 2469-7818. DOI: 10.1145/ 3335081. URL: https://doi.org/10.1145/3335081. [185] J. Powell, N. Inglis, J. Ronnie and S. Large. The characteristics and motivations of online health information seekers: cross-sectional survey and qualitative interview study. en. Journal of Medical Internet Research, 13(1):e20, 2011. DOI: 10.2196/ jmir.1600. [186] J. Preece, B. Nonnecke and D. Andrews. The top five reasons for lurking: improving community experiences for everyone. Computers in Human Behavior. The Com- pass of Human-Computer Interaction, 20(2):201–223, march 2004. ISSN: 0747- 5632. DOI: 10.1016/j.chb.2003.10.015. [187] X. Qi and B. D. Davison. Knowing a web page by the company it keeps. in CIKM ’06, pages 228–237, New York, NY, USA. ACM, 2006. ISBN: 978-1-59593-433-8. DOI: 10.1145/1183614.1183650. URL: http://doi.acm.org/10.1145/ 1183614.1183650. [Online; accessed 2018-08-13]. [188] H. Qian and C. R. Scott. Anonymity and self-disclosure on weblogs. en. Journal of Computer-Mediated Communication, 12(4):1428–1451, 01 july 2007. ISSN: 1083- 6101. DOI: 10.1111/j.1083-6101.2007.00380.x. [189] F. Rahman, I. D. Addo and S. I. Ahamed. Prisn: a privacy protection framework for healthcare social networking sites. in RACS ’14, pages 66–71, New York, NY, USA. ACM, 2014. ISBN: 978-1-4503-3060-2. DOI: 10.1145/2663761.2664199. URL: http://doi.acm.org/10.1145/2663761.2664199. [Online; accessed 2016-04-12]. [190] S. D. Reicher, R. Spears and T. Postmes. A social identity model of deindividuation phenomena. European Review of Social Psychology, 6(1):161–198, 01 january 1995. ISSN: 1046-3283. DOI: 10.1080/14792779443000049. [191] K. E. Ringland, C. T. Wolf, L. Dombrowski and G. R. Hayes. Making ”safe”: community-centered practices in a virtual world dedicated to children with autism. in CSCW ’15, pages 1788–1800, New York, NY, USA. ACM, 2015. ISBN: 978-1- 4503-2922-4. DOI: 10.1145/2675133.2675216. URL: http://doi.acm.org/ 10.1145/2675133.2675216. [Online; accessed 2016-05-27]. [192] A. Ritter, Mausam, O. Etzioni and S. Clark. Open domain event extraction from twitter. in KDD ’12, pages 1104–1112, New York, NY, USA. ACM, 2012. ISBN: 978-1-4503-1462-6. DOI: 10.1145/2339530.2339704. URL: http://doi.acm. org/10.1145/2339530.2339704.

200 Bibliography

[193] R. Room. Alcoholics Anonymous as a social movement. in Research on Alcoholics Anonymous: Opportunities and alternatives, pages 167–187. Rutgers Center of Al- cohol Studies, Piscataway, NJ, US, 1993. ISBN: 978-0-911290-24-0. [194] E. L. Rubenstein. Crowdsourcing health literacy: the case of an online community. in ASIST ’13, 120:1–120:5, Silver Springs, MD, USA. American Society for In- formation Science, 2013. ISBN: 978-0-87715-545-4. URL: http://dl.acm.org/ citation.cfm?id=2655780.2655900. event-place: Montreal, Quebec, Canada. [195] S. Rubya. Facilitating peer support for recovery from substance use disorders. in CHI EA ’17, pages 172–177, New York, NY, USA. ACM, 2017. ISBN: 978-1-4503- 4656-6. DOI: 10.1145/3027063.3048431. URL: http://doi.acm.org/10. 1145/3027063.3048431. [196] S. Rubya, X. Wang and S. Yarosh. Hair: towards developing a global self-updating peer support group meeting list using human-aided information retrieval. in Pro- ceedings of the 2019 Conference on Human Information Interaction and Retrieval, CHIIR ’19, pages 83–92, Glasgow, Scotland UK. Association for Computing Ma- chinery, 2019. ISBN: 9781450360258. DOI: 10 . 1145 / 3295750 . 3298933. URL: https://doi.org/10.1145/3295750.3298933. [197] S. Rubya and S. Yarosh. Interpretations of online anonymity in alcoholics anonymous and narcotics anonymous. Proc. ACM Hum.-Comput. Interact., 1(CSCW), december 2017. DOI: 10.1145/3134726. URL: https://doi.org/10.1145/ 3134726. [198] S. Rubya and S. Yarosh. Video-mediated peer support in an online community for recovery from substance use disorders. in CSCW ’17, pages 1454–1469, New York, NY, USA. ACM, 2017. ISBN: 978-1-4503-4335-0. DOI: 10.1145/2998181. 2998246. URL: http://doi.acm.org/10.1145/2998181.2998246. [Online; accessed 2017-03-30]. [199] J. M. Rzeszotarski and M. R. Morris. Estimating the social costs of friendsourcing. in CHI ’14, pages 2735–2744, New York, NY, USA. ACM, 2014. ISBN: 978-1- 4503-2473-1. DOI: 10.1145/2556288.2557181. URL: http://doi.acm.org/ 10.1145/2556288.2557181. event-place: Toronto, Ontario, Canada. [200] E. Saraç and S. A. Özel. Web page classification using firefly optimization. in pages 1–5. 2013 IEEE INISTA, june 2013. DOI: 10.1109/INISTA.2013.6577619. [201] E. Schenk and C. Guittard. Towards a characterization of crowdsourcing practices. en. Journal of Innovation Economics Management, n7(1):93–107, 06 april 2011.

201 Bibliography

[202] A. Schlesinger, E. Chandrasekharan, C. A. Masden, A. S. Bruckman, W. K. Edwards and R. E. Grinter. Situated anonymity: impacts of anonymity, ephemerality, and hyper-locality on social media. in CHI ’17, pages 6912–6924, New York, NY, USA. ACM, 2017. ISBN: 978-1-4503-4655-9. DOI: 10.1145/3025453.3025682. URL: http://doi.acm.org/10.1145/3025453.3025682. [203] W. C. Schmidt. World-wide web survey research: benefits, potential problems, and solutions. en. Behavior Research Methods, Instruments, Computers, 29(2):274– 279, 01 june 1997. ISSN: 0743-3808, 1532-5970. DOI: 10.3758/BF03204826. [204] Z. Schmitt and S. Yarosh. Participatory Design of Technologies to Support Recov- ery from Substance Use Disorders. Proc. ACM Hum.-Comput. Interact., 2(CSCW), november 2018. ISSN: 2573-0142. DOI: 10.1145/3274425. URL: http://doi. acm.org/10.1145/3274425 (urlseen 12/11/2019). [205] S. Y. Schoenebeck. The secret life of online moms: anonymity and disinhibition on youbemom. com. in Citeseer, 2013. URL: http://citeseerx.ist.psu.edu/ viewdoc/download?doi=10.1.1.592.7868&rep=rep1&type=pdf. [Online; accessed 2017-03-30]. [206] T. Schulze, S. Krug and M. Schader. Workers’ task choice in crowdsourcing and human computation markets. ICIS 2012 Proceedings, 14 december 2012. URL: http: //aisel.aisnet.org/icis2012/proceedings/ResearchInProgress/40. [207] I. Seidman. Interviewing as Qualitative Research: A Guide for Researchers in Edu- cation and the Social Sciences. en. Teachers College Press, 2006. ISBN: 978-0-8077- 4666-0. [208] Y. Setoyama, Y. Yamazaki and K. Namayama. Benefits of peer support in online Japanese breast cancer communities: differences between lurkers and posters. eng. Journal of Medical Internet Research, 13(4):e122, 2011. ISSN: 1438-8871. DOI: 10. 2196/jmir.1696. [209] V. Shashidhar, N. Pandey and V. Aggarwal. Spoken english grading: machine learning with crowd intelligence. in KDD ’15, pages 2089–2097, New York, NY, USA. ACM, 2015. ISBN: 978-1-4503-3664-2. DOI: 10.1145/2783258.2788595. URL: http://doi.acm.org/10.1145/2783258.2788595. [210] C. Shirky. Cognitive Surplus: How Technology Makes Consumers into Collabo- rators. en. Penguin, 10 june 2010. ISBN: 978-1-101-43472-7. Google-Books-ID: PjeTO822t4퐶. [211] E. Sillence. Seeking out very like-minded others: exploring trust and advice issues in an online health support group. Int. J. Web Based Communities, 6(4):376–394, october 2010. ISSN: 1477-8394. DOI: 10.1504/IJWBC.2010.035840.

202 Bibliography

[212] V. J. Slaymaker and T. Sheehan. The Impact of AA on Professional Treatment. en. in L. A. Kaskutas and M. Galanter, editorsRecent Developments in Alcoholism: Research on Alcoholics Anonymous and Spirituality in Addiction Recovery, Recent Developments in Alcoholism, pages 59–70. Springer, New York, NY, 2008. ISBN: 978-0-387-77725-2. DOI: 10 . 1007 / 978 - 0 - 387 - 77725 - 2 _ 4. URL: https : //doi.org/10.1007/978-0-387-77725-2_4 (urlseen 12/11/2019). [213] W. Sriurai, P. Meesad and C. Haruechaiyasak. Hierarchical web page classification based on a topic model and neighboring pages integration. arXiv:1003.1510 [cs], 07 march 2010. URL: http://arxiv.org/abs/1003.1510. arXiv: 1003.1510. [214] J. Suler. The online disinhibition effect. eng. Cyberpsychology Behavior: The Im- pact of the Internet, Multimedia and Virtual Reality on Behavior and Society, 7(3), june 2004. ISSN: 1094-9313. DOI: 10.1089/1094931041291295. PMID: 15257832. [215] C. Sun, N. Rampalli, F. Yang and A. Doan. Chimera: large-scale classification using machine learning, rules, and crowdsourcing. Proc. VLDB Endow., 7(13):1529– 1540, august 2014. ISSN: 2150-8097. DOI: 10.14778/2733004.2733024. [216] A. Tamersoy, M. De Choudhury and D. H. Chau. Characterizing Smoking and Drinking Abstinence from Social Media. in Proceedings of the 26th ACM Confer- ence on Hypertext & Social Media, HT ’15, pages 139–148, New York, NY, USA. ACM, 2015. ISBN: 978-1-4503-3395-5. DOI: 10.1145/2700171.2791247. URL: http://doi.acm.org/10.1145/2700171.2791247 (urlseen 26/09/2019). event-place: Guzelyurt, Northern Cyprus. [217] J. S. Tonigan, G. J. Connors and W. R. Miller. Alcoholics anonymous involvement (aai) scale: reliability and norms. en. Psychology of Addictive Behaviors, 10(2):75– 80, 1996. ISSN: 1939-1501, 0893-164X. DOI: 10.1037/0893-164X.10.2.75. [218] M. Tsukada, T. Washio and H. Motoda. Automatic web-page classification by using machine learning methods. in WI ’01, pages 303–313, London, UK, UK. Springer- Verlag, 2001. ISBN: 978-3-540-42730-8. URL: http://dl.acm.org/citation. cfm?id=645960.673927. [Online; accessed 2017-01-11]. [219] T. C. Turner, M. A. Smith, D. Fisher and H. T. Welser. Picturing usenet: mapping computer-mediated collective action. Journal of Computer-Mediated Communica- tion, 10(4):JCMC1048, 2005. [220] Understanding anonymity. URL: http : / / www . aa . org / assets / en _ US / p - 47_understandinganonymity.pdf. [221] M. Van Kleek, D. Smith, N. R. Shadbolt, D. Murray-Rust and A. Guy. Self curation, social partitioning, escaping from prejudice and harassment: the many dimensions of lying online. in WWW ’15 Companion, pages 371–372, New York, NY, USA. ACM, 2015. ISBN: 978-1-4503-3473-0. DOI: 10.1145/2740908.2745940. URL:

203 Bibliography

http://doi.acm.org/10.1145/2740908.2745940. [Online; accessed 2017- 03-29]. [222] A. B. Van Zant and L. J. Kray. “i can’t lie to your face”: minimal face-to-face interaction promotes honesty. Journal of Experimental Social Psychology, 55:234–238, november 2014. ISSN: 0022-1031. DOI: 10.1016/j.jesp.2014.07.014. [223] G. Venolia, J. Tang, R. Cervantes, S. Bly and G. Robertson. Embodied social proxy : mediating interpersonal connection in hub-and-satellite teams. in pages 1049– 1058, 2010. ISBN: 978-1-60558-929-9. [224] G. Wang, B. Wang, T. Wang, A. Nika, H. Zheng and B. Y. Zhao. Whispers in the dark: analysis of an anonymous social network. in IMC ’14, pages 137–150, New York, NY, USA. ACM, 2014. ISBN: 978-1-4503-3213-2. DOI: 10.1145/2663716. 2663728. URL: http://doi.acm.org/10.1145/2663716.2663728. [Online; accessed 2017-04-12]. [225] Y. Wang, M. Zhu, L. Qu, M. Spaniol and G. Weikum. Timely yago: harvesting, querying, and visualizing temporal knowledge from wikipedia. in EDBT ’10, New York, NY, USA. ACM, 2010. ISBN: 978-1-60558-945-9. DOI: 10.1145/1739041. 1739130. URL: http://doi.acm.org/10.1145/1739041.1739130. [226] S. C. Warby, S. L. Wendt, P. Welinder, E. G. Munk, O. Carrillo, H. B. Sorensen, P. Jennum, P. E. Peppard, P. Perona and E. Mignot. Sleep-spindle detection: crowdsourcing and evaluating performance of experts, non-experts and automated methods. en. Nature Methods, 11, 4:385–392, 2014. DOI: 10.1038/nmeth.2855. [227] H. Wauck, Y.-C. G. Yen, W.-T. Fu, E. Gerber, S. P. Dow and B. P. Bailey. From in the class or in the wild?: peers provide better design feedback than external crowds. in CHI ’17, pages 5580–5591, New York, NY, USA. ACM, 2017. ISBN: 978-1- 4503-4655-9. DOI: 10.1145/3025453.3025477. URL: http://doi.acm.org/ 10.1145/3025453.3025477. event-place: Denver, Colorado, USA. [228] J. Waycott, H. Davis, A. Thieme, S. Branham, J. Vines and C. Munteanu. Ethi- cal Encounters in HCI: Research in Sensitive Settings. in Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Sys- tems, CHI EA ’15, pages 2369–2372, New York, NY, USA. ACM, 2015. ISBN: 978-1-4503-3146-3. DOI: 10.1145/2702613.2702655. URL: http://doi.acm. org/10.1145/2702613.2702655 (urlseen 09/10/2019). event-place: Seoul, Re- public of Korea. [229] J. L. Welbourne, A. L. Blanchard and M. D. Boughton. Supportive communication, sense of virtual community and health outcomes in online infertility groups. in CT ’09, pages 31–40, New York, NY, USA. ACM, 2009. ISBN: 978-1-60558-713-4. DOI: 10.1145/1556460.1556466. URL: http://doi.acm.org/10.1145/ 1556460.1556466. [Online; accessed 2016-04-12].

204 Bibliography

[230] W. White. Sponsor, recovery coach, addiction counselor: the importance of role clarity and role integrity. Philadelphia Department of Behavioral Health and Men- tal Retardation Services, 2006. [231] Why don’t men volunteer as much as women? URL: http://priceonomics.com/ the-altruism-gender-gap/. [Online; accessed 2016-05-19]. [232] Wikipedia:awards. en, 30 january 2017. URL: https://en.wikipedia.org/w/ index.php?title=Wikipedia:Awards&oldid=762810081. Page Version ID: 762810081. [233] D. M. Wilcox. Alcoholic Thinking: Language, Culture, and Belief in Alcoholics Anonymous. Greenwood Publishing Group, 1998. [234] B. Wilson. Twelve steps and twelve traditions. The AA Grapevine, Inc. and Alco- holics Anonymous, 1953. [235] S. Wilson, F. Schaub, R. Ramanath, N. Sadeh, F. Liu, N. A. Smith and F. Liu. Crowdsourcing annotations for websites’ privacy policies: can it really work? in WWW ’16, pages 133–143, Republic and Canton of Geneva, Switzerland. Inter- national World Wide Web Conferences Steering Committee, 2016. ISBN: 978-1- 4503-4143-1. DOI: 10.1145/2872427.2883035. URL: https://doi.org/10. 1145/2872427.2883035. [236] K. Wright. Perceptions of on‐line support providers: an examination of perceived homophily, source credibility, communication and social support within on‐line support groups. Communication Quarterly, 48(1):44–59, 01 january 2000. ISSN: 0146-3373. DOI: 10.1080/01463370009385579. [237] K. B. Wright, S. B. Bell, K. B. Wright and S. B. Bell. Health-related support groups on the internet: linking empirical findings to social support and computer-mediated communication theory. eng. Journal of Health Psychology, 8(1):39–54, january 2003. ISSN: 1359-1053. DOI: 10.1177/1359105303008001429. PMID: 22113899. [238] X. Xing, Y.-L. Liang, H. Cheng, J. Dang, S. Huang, R. Han, X. Liu, Q. Lv and S. Mishra. Safevchat: detecting obscene content and misbehaving users in online video chat services. in WWW ’11, pages 685–694, New York, NY, USA. ACM, 2011. ISBN: 978-1-4503-0632-4. DOI: 10.1145/1963405.1963501. URL: http: //doi.acm.org/10.1145/1963405.1963501. [Online; accessed 2016-04-27]. [239] Y. Yang, B. B. Zhu, R. Guo, L. Yang, S. Li and N. Yu. A comprehensive human computation framework: with application to image labeling. in MM ’08, pages 479– 488, New York, NY, USA. ACM, 2008. ISBN: 978-1-60558-303-7. DOI: 10.1145/ 1459359.1459423. URL: http://doi.acm.org/10.1145/1459359.1459423. [Online; accessed 2018-08-13].

205 Bibliography

[240] D. Yao, H. Sun and X. Liu. Combining crowd contributions with machine learning to detect malicious mobile apps. in Internetware ’15, pages 120–123, New York, NY, USA. ACM, 2015. ISBN: 978-1-4503-3641-3. DOI: 10.1145/2875913. 2875941. URL: http://doi.acm.org/10.1145/2875913.2875941. [241] S. Yarosh. Shifting dynamics or breaking sacred traditions?: the role of technology in twelve-step fellowships. in CHI ’13, pages 3413–3422, New York, NY, USA. ACM, 2013. ISBN: 978-1-4503-1899-0. DOI: 10.1145/2470654.2466468. URL: http://doi.acm.org/10.1145/2470654.2466468. [Online; accessed 2016- 03-29]. [242] S. Yarosh, A. Tang, S. Mokashi and G. D. Abowd. ”almost touching”: parent-child remote communication using the sharetable system. in CSCW ’13, pages 181– 192, New York, NY, USA. ACM, 2013. ISBN: 978-1-4503-1331-5. DOI: 10.1145/ 2441776.2441798. URL: http://doi.acm.org/10.1145/2441776.2441798. [Online; accessed 2016-05-15]. [243] Y.-C. G. Yen, S. P. Dow, E. Gerber and B. P. Bailey. Social network, web forum, or task market?: comparing different crowd genres for design feedback exchange. in DIS ’16, pages 773–784, New York, NY, USA. ACM, 2016. ISBN: 978-1-4503- 4031-1. DOI: 10.1145/2901790.2901820. URL: http://doi.acm.org/10. 1145/2901790.2901820. event-place: Brisbane, QLD, Australia. [244] C.-w. You, K.-C. Wang, M.-C. Huang, Y.-C. Chen, C.-L. Lin, P.-S. Ho, H.-C. Wang, P. Huang and H.-H. Chu. SoberDiary: A Phone-based Support System for Assisting Recovery from Alcohol Dependence. in Proceedings of the 33rd Annual ACM Con- ference on Human Factors in Computing Systems, CHI ’15, pages 3839–3848, New York, NY, USA. ACM, 2015. ISBN: 978-1-4503-3145-6. DOI: 10.1145/2702123. 2702289. URL: http://doi.acm.org/10.1145/2702123.2702289 (urlseen 26/09/2019). event-place: Seoul, Republic of Korea. [245] Z. Yu, D. Zhang, D. Yang and G. Chen. Selecting the best solvers: toward community based crowdsourcing for disaster management. in 2012 IEEE Asia-Pacific Services Computing Conference, pages 271–277, 2012. DOI: 10.1109/APSCC. 2012.20. [246] A. Yuan, K. Luther, M. Krause, S. I. Vennix, S. P. Dow and B. Hartmann. Al- most an expert: the effects of rubrics and expertise on perceived value of crowdsourced design critiques. in Proceedings of the 19th ACM Conference on Computer- Supported Cooperative Work Social Computing, CSCW ’16, pages 1005–1017, San Francisco, California, USA. Association for Computing Machinery, 2016. ISBN: 9781450335928. DOI: 10.1145/2818048.2819953. URL: https://doi-org. ezp2.lib.umn.edu/10.1145/2818048.2819953.

206 Bibliography

[247] D. Yue, G. Yu, D. Shen and X. Yu. A weighted aggregation rule in crowdsourcing systems for high result accuracy. in pages 265–270. 2014 IEEE 12th International Conference on Dependable, Autonomic and Secure Computing, august 2014. DOI: 10.1109/DASC.2014.54. [248] H. Yun. The creation and validation of a perceived anonymity scale based on the social information processing model and its nomological network test in an online social support community, 2006. [249] M. Zhang and C. C. Yang. Classification of Online Health Discussions with Text and Health Feature Sets. en. in Workshops at the Twenty-Eighth AAAI Conference on Artificial Intelligence, june 2014. URL: http://www.aaai.org/ocs/index. php/WS/AAAIW14/paper/view/8739 (urlseen 12/04/2016). [250] M. Zhang and C. C. Yang. Classifying user intention and social support types in online healthcare discussions. in pages 51–60. 2014 IEEE International Conference on Healthcare Informatics (ICHI), september 2014. DOI: 10.1109/ICHI.2014. 15. [251] P. G. Zimbardo. The human choice: individuation, reason, and order versus deindividuation, impulse, and chaos. English. Nebraska Symposium on Motivation, 17:237– 307, 1969. ISSN: 0070-2099. [252] G. D. Zimet, N. W. Dahlem, S. G. Zimet and G. K. Farley. The multidimensional scale of perceived social support. Journal of Personality Assessment, 52(1):30–41, 01 march 1988. ISSN: 0022-3891. DOI: 10.1207/s15327752jpa5201_2. [253] A. G. Zimmerman and G. J. Ybarra. Online aggression: the influences of anonymity and social modeling. English. Psychology of Popular Media Culture, 5(2):181, april 2016. ISSN: 2160-4142 (Electronic), 2160-4134 (Print). DOI: 10.1037/ppm0000038.

207