Modeling Intention in Email

Modeling Intention in Email Vitor R. Carvalho CMU-LTI-08-007 Language Technologies Institute School of Computer Science Carnegie Mellon University 5000 Forbes Ave., Pittsburgh, PA 15213 www.lti.cs.cmu.edu Thesis Committee: William W. Cohen, Chair Tom M. Mitchell Robert E. Kraut Lise Getoor, University of Maryland Submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy. Copyright c 2008 Vitor R. Carvalho Keywords: Machine Learning, Email Management, Ranking To Sarah iv Abstract Email management has a fundamental role in modern work productivity. In this thesis we present evidence that email management can be potentially improved by the effective use of machine learning techniques to model different aspects of user intention. We initially propose a taxonomy of user in- tentions in terms of Speech Acts applied to email communication, or “email acts”, and show that email act classification can be largely automated, potentially leading to better email prioritization and management. We then describe how machine learning can be used to reduce the chances of costly email addressing errors. One type of costly error is an “email leak”, i.e., mistakenly sending a message to an unintended recipient — a widespread problem that can severely harm individuals and corporations. Another type of addressing error is forgetting to add an intended collaborator as recipient, a likely source of costly misunderstandings and communication delays that can be potentially addressed with intelligent recipient recommendation. We propose several different approaches to address these problems, and show very positive experimental results in a large email collection. In addition, we describe a 4-week long user study based on the implementation of some of the proposed models in a popular email client (Mozilla Thunderbird). More than 15% of the human subjects reported that it prevented real email leaks, and more than 47% of them utilized recipient recommendations. Overall the study shows that recipient recommendation and email leak detection can be valuable additions to real email clients, with more than 80% of the subjects reporting that they would permanently use these models if a few interface/optimization changes were implemented. Finally, we introduce a new robust rank learning algorithm to further im- prove recipient recommendation predictions. The algorithm is essentially a non-convex optimization procedure over a sigmoidal loss function, in which any linear baseline ranking model can be used as starting point. This new learning method provides substantial rank performance improvements on recipient recommendation tasks, outperforming all previously introduced models, including well-known state-of-the-art ranking algorithms. vi Contents 1 Introduction 1 1.1 Motivation . 1 1.2 Overview and Contributions . 2 1.3 Organization . 5 2 Email “Speech Acts” 7 2.1 Introduction . 7 2.2 A Taxonomy of Email Acts . 8 2.3 Corpus . 10 2.4 Inter-Annotator Agreement . 12 2.5 Classifying Email into Acts . 14 2.6 Collective Classification of Email Acts . 15 2.6.1 Evidence for Sequential Correlation of Email Acts . 17 2.6.2 Predicting Acts from Surrounding Acts . 19 2.6.3 Collective Classification Algorithm . 20 2.6.4 Experiments . 22 2.6.5 Discussion . 28 2.7 Linguistic Analysis . 29 2.7.1 Preprocessing and N-gram Features . 29 2.7.2 Experiments . 33 2.8 Related Work . 36 vii 2.8.1 Speech Act Theory . 36 2.8.2 Dialog Act Tagging . 38 2.8.3 Email Acts and Other Applications . 39 2.8.4 Segmentation . 40 3 Email Information Leaks 43 3.1 Introduction . 43 3.2 The Enron Dataset . 44 3.3 Generating Synthetic Leaks . 47 3.3.1 Leak Criteria . 47 3.4 Methods . 49 3.4.1 Baselines: Using Textual Content . 49 3.4.2 Reranking with Social Network Information . 51 3.5 Finding Real Email Leaks . 55 3.5.1 Sampling from Seen and Unseen Recipients . 57 3.6 Leak Prediction Results . 58 3.7 Discussion and Related Work . 61 4 Recommending Email Recipients 63 4.1 Introduction . 63 4.2 Evidence of Message Addressing Problems . 63 4.3 Data Preprocessing and Task Definition . 64 4.4 Models . 65 4.4.1 Expert Search Model 1 . 66 4.4.2 Expert Search Model 2 . 68 4.4.3 TFIDF Classifier . 69 4.4.4 K-Nearest Neighbors . 69 4.4.5 Other Baselines: Frequency and Recency . 70 4.4.6 Threading . 70 viii 4.5 Results . 71 4.5.1 Initial results . 71 4.5.2 Rank Aggregation . 74 4.5.3 Email Auto-Completion . 75 4.6 Discussion and Related Work . 76 5 User Study 81 5.1 Introduction . 81 5.2 Cut Once: a Mozilla Thunderbird Extension . 82 5.2.1 Algorithms . 82 5.2.2 Training . 84 5.2.3 Prediction . 85 5.2.4 Logging . 86 5.3 Study Description . 88 5.4 Results . 90 5.4.1 Adoption . 90 5.4.2 Usage and Predictions . 91 5.4.3 Baseline Comparison . 97 5.5 Discussion . 100 5.6 Conclusions and Related Work . 101 6 Learning Robust Ranking Models 105 6.1 Introduction . 105 6.2 Learning to Rank . 106 6.2.1 Pairwise-preference Ranking . 106 6.2.2 Outliers in Pairwise Preference Ranking . 108 6.3 Robust Pairwise Ranking . 110 6.3.1 RankSVM . 111 6.3.2 Sigmoid Approximation . 111 ix 6.3.3 Learning . 113 6.4 Ranking Datasets . 114 6.4.1 Email Recipient Recommendation . 114 6.4.2 Other Ranking Datasets . 116 6.5 Experiments . 117 6.5.1 Performance . 117 6.5.2 Learning Curve . 121 6.5.3 Sigma Parameter . 122 6.6 Discussion and Related Work . 123 7 Conclusions 125 7.1 Summary . 125 7.2 Future Directions . 126 A Email Act Labeling Guidelines 129 A.1 Verbs . 129 A.2 Nouns . 130 B User Study Supporting Material 133 B.1 Recruiting Message . 133 B.2 Initial Questionnaire . 134 B.3 Final Questionnaire . 134 B.4 User Comments . 136 Bibliography 137 x List of Figures 2.1 Taxonomy of email acts used in most experiments. Shaded nodes are the ones for which classifiers were constructed. 11 2.2 Diagram of a “Conversation for Action” Structure from Winograd & Flo- res [1986] . 16 2.3 Transition Diagram for the four most common specific verbs. 18 2.4 Kappa Values on 1F3 using Relational (Context) features and Textual (Content) features. 20 2.5 Kappa versus iteration on 1F3, using classifiers trained on 3F2. 24 2.6 Plot of baseline Kappa (x-axis) versus Kappa (%) after iterative collective classification was performed. Points above the dotted line represent an improvement. 25 2.7 Plot of baseline Kappa (x-axis) versus Kappa(%) after iterative collective classification was performed. Performance improvement by groups of email acts. Groups were selected based on performance in the prelimi- nary tests. 26 2.8 Plot of baseline Kappa (x-axis) versus Kappa(%) after iterative collective classification was performed. Performance improvement for delivery- related and non-delivery related email acts. 27 2.9 Kappa values with and without collective classification, averaged over the four test sets in the leave-one-team-out experiment. 28 2.10 Error Rate on a 5-fold Cross-validation Experiment . 34 2.11 Precision versus Recall on Request Act Classification . 36 3.1 Time frames of different email collections. 46 xi 3.2 Performance of Real Leak Case Germany-c For Different Probabilities α. 58 3.3 Performance of Real Leak Cases Kitchen-l For Different Probabilities α.. 59 4.1 Auto-completion performance on the TOCCBCC task for different number of initial letters. 76 4.2 Auto-completion performance on the CCBCC task for different number of initial letters. ..

Load more