The Writing Process in Online Mass Collaboration NLP-Supported Approaches to Analyzing Collaborative Revision and User Interaction Vom Fachbereich Informatik der Technischen Universität Darmstadt genehmigte Dissertation zur Erlangung des akademischen Grades Dr.-Ing. vorgelegt von Johannes Daxenberger, M.A. geboren in Rosenheim Tag der Einreichung: 28. Mai 2015 Tag der Disputation: 21. Juli 2015 Referenten: Prof. Dr. Iryna Gurevych, Darmstadt Prof. Dr. Karsten Weihe, Darmstadt Assoc. Prof. Ofer Arazy, Ph.D., Alberta Darmstadt 2016 D17 Please cite this document as URN: urn:nbn:de:tuda-tuprints-52259 URL: http://tuprints.ulb.tu-darmstadt.de/5225 This document is provided by tuprints, E-Publishing-Service of the TU Darmstadt http://tuprints.ulb.tu-darmstadt.de [email protected] This work is published under the following Creative Commons license: Attribution – Non Commercial – No Derivative Works 3.0 Germany http://creativecommons.org/licenses/by-nc-nd/3.0/de/deed.en Abstract In the past 15 years, the rapid development of web technologies has created novel ways of collaborative editing. Open online platforms have attracted millions of users from all over the world. The open encyclopedia Wikipedia, started in 2001, has become a very prominent example of a largely successful platform for collaborative editing and knowledge creation. The wiki model has enabled collaboration at a new scale, with more than 30,000 monthly active users on the English Wikipedia. Traditional writing research deals with questions concerning revision and the writing process itself. The analysis of collaborative writing additionally raises questions about the interaction of the involved authors. Interaction takes place when authors write on the same document (indirect interaction), or when they coordinate the collaborative writing process by means of communication (direct interaction). The study of collaborative writing in on- line mass collaboration poses several interesting challenges. First and foremost, the writing process in open online collaboration is typically characterized by a large number of revi- sions from many different authors. Therefore, it is important to understand the interplay and the sequences of different revision categories. As the quality of documents produced in a collaborative writing process varies greatly, the relationship between collaborative re- vision and document quality is an important field of study. Furthermore, the impact of direct user interaction through background discussions on the collaborative writing pro- cess is largely unknown. In this thesis, we tackle these challenges in the context of online mass collaboration, using one of the largest collaboratively created resources, Wikipedia, as our data source. We will also discuss to which extent our conclusions are valid beyond Wikipedia. We will be dealing with three aspects of collaborative writing in Wikipedia. First, we carry out a content-oriented analysis of revisions in the Wikipedia revision history. This includes the segmentation of article revisions into human-interpretable edits. We develop a taxonomy of edit categories such as spelling error corrections, vandalism or informa- tion adding, and verify our taxonomy in an annotation study on a corpus of edits from the English and German Wikipedia. We use the annotated corpora as training data to create iii models which enable the automatic classification of edits. To show that our model is able to generalize beyond our own data, we train and test it on a second corpus of English Wiki- pedia revisions. We analyze the distribution of edit categories and frequent patterns in edit sequences within a larger set of article revisions. We also assess the relationship between edit categories and article quality, finding that the information content in high-quality arti- cles tends to become more stable after their promotion and that high-quality articles showa higher degree of homogeneity with respect to frequent collaboration patterns as compared to random articles. Second, we investigate activity-based roles of users in Wikipedia and how they relate to the collaborative writing process. We automatically classify all revisions in a representative sample of Wikipedia articles and cluster users in this sample into seven intuitive roles. The roles are based on the editing behavior of the users. We find roles such asVandals, Watchdogs, or All-round Contributors. We also analyze the stability of our discovered roles across time and analyze role transitions. The results show that although the nature of roles remains stable across time, more than half of the users in our sample changed their role between two time periods. Third, we analyze the correspondence between indirect user interaction through col- laborative editing and direct user interaction through background discussion. We analyze direct user interaction using the notion of turns, which has been established in previous work. Turns are snippets from Wikipedia discussion pages. We introduce the notion of corresponding edit-turn-pairs. A corresponding edit-turn-pair consists of a turn and an edit from the same Wikipedia article; the turn forms an explicit performative and the edit corresponds to this performative. This happens, for example, when a user complains about a missing reference in the discussion about an article, and another user adds an appropri- ate reference to the article itself. We identify the distinctive properties of corresponding edit-turn-pairs and use them to create a model for the automatic detection of correspond- ing and non-corresponding edit-turn-pairs. We show that the percentage of corresponding edit-turn-pairs in a corpus of flawed English Wikipedia articles is typically below 5% and varies considerably across different articles. The thesis is concluded with a summary of our main contributions and findings.The growing number of collaborative platforms in commercial applications and education, e.g. in massive open online learning courses, demonstrates the need to understand the collab- orative writing process and to support collaborating authors. We also discuss several open issues with respect to the questions addressed in the main parts of the thesis and point out possible directions for future work. Many of the experiments we carried out in the course of this thesis rely on supervised text classification. In the appendix, we explain the con- cepts and technologies underlying these experiments. We also introduce the DKPro TC framework, which was substantially extended as part of this thesis. iv Zusammenfassung Die Weiterentwicklung von Webtechnologien in den vergangenen 15 Jahren hat vollkom- men neue Formen gemeinschaftlichen Schreibens im Web hervorgebracht. Open-Access Online-Plattformen haben Millionen Benutzer, die über die gesamte Erde verteilt sind.Die Online-Enzyklopädie Wikipedia, gegründet im Jahr 2001, hat sich zu einer der bekanntesten und erfolgreichsten Plattformen für gemeinschaftliches Schreiben und Wissensgenerierung entwickelt. Das Wiki-Modell macht Zusammenarbeit in einer neuen Dimension möglich, so dass bspw. in der englischen Wikipedia jeden Monat mehr als 30.000 Benutzer aktiv sind. Die traditionelle Schreibforschung setzt sich mit Fragen über Revision und den Schreib- prozess auseinander. Die Analyse gemeinschaftlichen Schreibens interessiert sich darüber hinaus für die Interaktion der beteiligten Benutzer. Solche Interaktion findet statt wenn Autoren am selben Dokument schreiben (indirekte Interaktion), oder wenn Autoren den gemeinschaftlichen Schreibprozess mittels mündlicher oder schriftlicher Kommunikation koordinieren (direkte Interaktion). Die Erforschung gemeinschaftlichen Schreibens unter massiver Zusammenarbeit auf Online-Plattformen beinhaltet mehrere interessante Heraus- forderungen. Der gemeinschaftliche Schreibprozess im Web ist gekennzeichnet durch eine typischer- weise sehr hohe Zahl von Änderungen, die von vielen verschiedenen Autoren stammen. Dementsprechend ist es unverzichtbar, den Zusammenhang und die Abfolge unterschied- licher Revisionstypen zu verstehen. Da die inhaltliche Qualität der Dokumente, die unter Zusammenarbeit erstellt werden, sehr unterschiedlich ist, ist außerdem die Erforschung der Korrelation zwischen gemeinschaftlichen Änderungen und Dokumentqualität ein wichti- ges Feld. Desweiteren ist der Einfluss direkter Benutzerinteraktion mittels Diskussionen im Hintergrund auf den gemeinschaftlichen Schreibprozess größtenteils unbekannt. In der vorliegenden Arbeit setzen wir uns mit diesen Herausforderungen im Kontext massiver Zusammenarbeit auf Online-Plattformen auseinander. Dabei verwenden wir Wikipedia, eine der größten gemeinschaftlich erstellen Online-Ressourcen, als Datengrundlage. Wir werden auch diskutieren, inwiefern unsere Erkenntnisse über Wikipedia hinaus Gültigkeit besitzen. v Drei Hauptaspekte gemeinschaftlichen Schreibens in Wikipedia stellen das Grundge- rüst dieser Arbeit dar. Als erstes führen wir eine inhaltliche Analyse von Revisionstypen in der Wikipedia Versionsgeschichte durch, wozu auch die Segmentierung von Artikelre- visionen in kleinere Edits, die einfacher zu interpretieren sind, zählt. Wir entwickeln eine Taxonomie für Edittypen, die bspw. Rechtschreibkorrekturen, Vandalismus oder Ergänzun- gen von Information beinhaltet. Die Taxonomie wird getestet in einer Annotationsstudie auf Edits aus der englischen und der deutschen Wikipedia. Wir verwenden die annotier- ten Korpora als Trainingsdaten zum Erstellen eines Modells für die automatischen Klas- sifikation