The Authorship of Audacity: Data Mining and Stylometric Analysis of Speeches

Jonathan Herz Abdelghani Bellaachia School of Engineering and School of Engineering and Applied Science Applied Science George Washington University George Washington University Washington, DC 20052 Washington, DC 20052 Email: [email protected] Email: [email protected]

Abstract—We explore the feasibility of identifying authorship Keenan, Adam Frankel, and Ben Rhodes. Jon Favreau among President Obama’s principal speechwriters with the was Obama’s lead speechwriter from 2005 - 2013, and interesting result that, yes, we can. This task is difficult because has the most speeches in the corpus attributed to him as there are few training examples, multiple authors (four), and primary author: 13 in total. Cody Keenan, the current lead because these authors consciously attempt to emulate a single speechwriter, joined the team in 2009. Five of the speeches in style - the President’s. Four corpuses are created to compare our corpus are attributed to him. Adam Frankel also worked different text pre-processing techniques. On each, function word frequencies are analyzed with ANOVA to select discriminating on the 2008 campaign, and has 9 speeches attributed in our feature vectors. Using leave-one-out cross-validation, K-nearest training set. Ben Rhodes, who wrote for the campaign and neighbors achieves the best classification accuracy, 78%. One now specializes in foreign policy speeches, has 10 attributions. interesting result is that the new head White House lead speechwriter, Cody Keenan, is not distinguishable from the other principal speechwriters beyond pure chance. Classification accuracy is improved to 90% after removing his work from our C. Obama’s Speechwriters corpus. In this paper, we will examine the President Barack Key words: authorship attribution, text mining, classification, Obama’s national speeches and remarks to attempt to statistical analysis, stylometry. determine which speeches were written by which of his principal speechwriters. It is too early to say with certainty I.INTRODUCTION what Obama’s place in history will be, but it is safe to say that he is a politician whose career was built on the strength A. Stylometry of his oratory. Obama catapulted to the national stage with his Stylometry is the study of identifying authorship of address to the 2004 Democratic National Convention, before texts based on information contained in the text itself. he had even won national office. His viability as a presidential Many methods have been proposed, but they all share the candidate only four years later can only be explained by the assumption that every author leaves behind quantifiable and strength of that speech, and the widespread national attention distinctive markers that allow them to be identified from that it garnered. Although a gifted orator in his own right, a pool of possible authors [1]. Some of the more popular even Obama has had plenty of help from others. Beginning features that have been used in the problem domain include with the start of his Senate career in 2009, Obama began to various types of word frequencies[2, 3], n-gram frequencies, assemble a speechwriting team that would follow him to the punctuation[4], and aggregate measurements such as average White House and beyond. sentence length [5]. Unlike many text classification problems, Obama has four principal speechwriters with identifiable authorship identification uses function word frequencies as speeches: Jon Favreau, Cody Keenan, Ben Rhodes, and feature vectors rather than ignoring them as stop words. Adam Frankel. They are, as a group, remarkably young for presidential speechwriters. After five years of the Obama administration, most are now in their late 20s or early 30s. B. The Data They joined the team at different times, and some have recently left the administration. Unfortunately, information We assembled a corpus of 37 speeches and addresses about the specific speeches these writers worked on is very delivered by President Obama for which we can assign sparse. Out of hundreds of speeches delivered by Barack primary authorship to one of four principal speechwriters. Obama over the course of his national political career, we These speeches span both his time as candidate and sitting were only able to attribute 37 to individual speechwriters president. One of the challenges in this experiment was finding through interviews available in the public domain. attributions of speeches, since presidential speechwriters are, Jon Favreau was Obama’s lead speechwriter until early in the words of FDR, expected to “have a passion for 2013. In 2005, Favreau began working for then-Senator anonymity.” Obama as his speechwriter. Some of President Obama’s The four principal speechwriters are Jon Favreau, Cody most defining speeches were written in conjunction with Jon Favreau. After the Jeremiah Wright controversy engulfed the analysis[4], and support vector machines [3,5]. Beginning 2008 Obama presidential campaign, Jon Favreau worked on in the 1980s, J. F. Burrows began to explore the use the key speech “A More Perfect Union,” which began by stylometric techniques to analyze Jane Austen’s novels and contextualizing the remarks of that controversial preacher other English authors of the Victorian era. Like Mosteller and against the backdrop of race relations in the United States, Wallace, Burrows analyses used frequencies of function words and ended with a call to work together to address social such as by, the, from, etc. as discriminant features to classify problems[6]. The speech became popular very quickly on authorial style. Burrows even used stylometric techniques YouTube, with 1.2 million views in the first 24 hours after to study character dialogue in some of Jane Austen’s books release. Favreau also worked on President Obama’s “Nobel to identify the stylometric fingerprint of different fictional Peace Prize acceptance speech”[6]. Other important speeches characters. Burrows’ primary contribution to the field was the that were identified as Favreau’s work include both inaugural pioneering use of Principal Component Analysis to reduce speeches[6,7], the “2008 Jefferson-Jackson Day Dinner the high dimensionality inherent in using the frequencies of address”[8], and, more recently, the President’s “remarks at dozens of function words to categorize text[14]. Principal the prayer vigil for victims of the Sandy Hook Elementary Component Analysis takes a correlation or covariance matrix school shooting”[6]. of the dimensions being used in the analysis in order to Cody Keenan joined the presidential speechwriting team find the highest eigenvalues, which correspond to the most in 2009, and succeeded Jon Favreau as the new Director significant variables. Usually, the marginal amount of variance of Speechwriting in 2013. Keenan wrote the President’s explained by the dimensions or variables after the first few “eulogy for Senator ”[9], and the president’s drops off dramatically. “remarks upon the signing of the ‘Edward M. Kennedy Serve The combination of feature selection of function words America Act”’[9]. Keenan also worked on the “2009 White introduced by Mosteller and Wallace, and the generalization House Correspondents’ Association dinner remarks”[10], of that approach to multivariate methods using Principal the President’s “speech at the Tucson shooting memorial Component Analysis for dimensional reduction by Burrows, service”[10], and “the 2013 State of the Union Address”[10]. set a standard that has been followed by many stylometric Ben Rhodes’ official job title is deputy national security studies. For example, In the paper Who wrote the 15th Book adviser for strategic communications. His speeches focus of Oz, Jose Binongo uses exactly the approach described on foreign policy. Speeches that have been attributed in the above to determine whether popular sequels to The Wizard press to Rhodes include the New Beginning speech made of Oz by Frank Baum had been written by Baum himself by Obama in June 2009 in Egypt as an attempt to improve or by Ruth Plumly Thompson, an associate of Baum’s who America’s image in the Middle East[11], a speech given is known to have written many of the later Oz books[15]. to a large public audience in Israel in March 2013[12], a First, like Mosteller and Wallace, Binongo tallied the February 2009 speech on ending the Iraq War[11], a “speech frequency of function words. Next, like Burrows’ work, the to the Ghanaian Parliament in July 2009”[11], and a “speech high-dimensional 50 most frequently occurring words were delivered to the New Economic School in Moscow”[11]. reduced to 2 using Principal Component Analysis. Adam Frankel, like Jon Favreau, also started in the Kerry Similar methods have also been used to attribute campaign. He was Favreau’s first hire in 2007 after Obama presidential addresses. In the 2006 paper Who Wrote Ronald announced his candidacy for the presidency, but left in 2011. Reagan’s Radio Addresses?, Airoldi, Anderson, Fienberg, Speeches that are known to have been written by him include and Skinner use function word frequency counts, Principal the President’s “eulogy for Senator Robert C. Byrd”[13], Component Analysis, and other methods to attempt to the President’s “address to the Upper Big Branch coal mine determine authorship of several hundred of Ronald Reagan’s disaster memorial service”[13], the President’s “address at radio addresses that were delivered in the late 1970s, as Reagan the National Peace Officers’ memorial”[13], the President’s prepared to run for the White House[3]. The Reagan study “eulogy for civil rights activist Dr. Dorothy Height”[13], also tried many different methods of stylistic fingerprinting in and the President’s 2009 speech to the American Medical addition to function word frequencies, including delta-squared Association[13]. He also prepared remarks for the president measures, SMART list words, semantic features, information at a National Prayer Breakfast after his first inauguration[8]. gain, and n-gram methods. In all cases, Principal Component Analysis was used for dimensionality reduction. The Reagan radio address paper used several methods for classification as well. Naive Bayes, LDA, Logistic Regression, Unit-Weight Models, SVM, k-Nearest Neighbors, CART, Random Forests, II.RELATED WORK Majority Voting, and Maximum Likelihood approaches were Mosteller and Wallace pioneered the modern study of all tried. The authors settled on using Naive Bayes to classify stylometry with their work on the Federalist Papers[2]. word features selected by the delta-squared measure. “Mosteller and Wallace employed numerical probabilities to The Reagan radio address study shares many of the express degrees of belief about propositions such as ‘Hamilton problems inherent in our study; the line between a president’s wrote paper No. 52’ and then used Bayes’ theorem to adjust words and his aides is often unclear, the number of addresses these probabilities for the evidence in hand. They attributed known to be attributed to particular aides is small relative to all 12 disputed papers to Madison, a conclusion broadly in the number of addresses of unknown origin, and there is no agreement with historical scholarship” [1]. Mosteller and way to know for certain whether or not our predictions for Wallace used a Naive Bayes classifier for the classification authorship are accurate. The Reagan study has the advantage task, but many other algorithms have been used with success, of being written over a decade after Ronald Reagan’s for example: Naive Bayes, neural nets, linear discriminant presidency ended. The intervening years give time for aides to report what they wrote, for historians and archivists to but not in all documents (inverse document frequency). For examine the documents produced by the administration, and the purposes of our stylometric analysis, we were primarily for criticism, praise, and attribution of a President’s speeches interested in the words that appear most frequently in each and addresses delivered for purely political reasons to recede speech, as these are most likely to be the function words. behind the curtain of history and rational analysis. We did not use any document weighting whatsoever, since While, to our knowledge, there have been no other most document weighting schemes are designed specifically studies of authorship attribution applied specifically to to reduce the weight of function words in the term-document Obama’s speeches, Savoy studied the use of classification matrix. We tried two term frequency weighting schemes. One techniques including support vector machines (SVM) and was augmented term frequency, which allowed us to normalize Nave Bayesian classifiers to identify speeches belonging word frequencies between large and small speeches. Aug- delivered on the campaign trail as Candidate Obama, and mented term frequency, originally introduced by the SMART those delivered while in office as President Obama[16]. information retrieval system is: While that study shares many similarities to ours, as it also 0.5 ∗ tf uses individual word frequencies as feature vectors to a 0.5 + t,d classification problem, it focuses on contextual words that max(tft,d) reveal the subject matter of the speech. Our study, which Where tft,d is the raw term frequency of term t in document d. attempts to solve a slightly different problem, restricts itself to function words that have little or no contextual value. The other term frequency weighting scheme we used A relatively recent branch of stylometric research has was a simple normalization, interesting implications for our experiment. Adversarial stylometry attempts to measure the ability of authors to tft,d/Td preserve their anonymity by consciously obfuscating their where tft,d is the raw term frequency of term t in document own writing style, often by emulating another authors style. d and T is the total number of terms in document d. Brennan, Afroz and Greenstadts 2012 study used a pool of volunteers who were able to reduce the accuracy of several If we had used raw word counts, of course, very long popular stylometric classification techniques to the level speeches would have much greater weight than shorter ones, of random chance[5]. Obama’s speechwriters, although not and our final results might have simply clustered speeches by participating in a study, do try to speak as one voice that length rather than by the stylistic fingerprints we hoped to of the President of the United States. Obama is known for uncover. having a distinctive rhetorical style, and it is the job of his To summarize, the four pre-processing methods used speechwriters to try to emulate it. were: Stemmed / Augmented term weighting, Unstemmed / Augmented term weighting, Stemmed / Normalized term weighting, and Unstemmed / Normalized term weighting. III.EXPERIMENTAL APPROACHAND RESULTS These will be referred to as the S/A, U/A, S/N, and U/N sets, respectively. All experiments were performed on each of the A. Pre-Processing four resulting data sets. Our raw corpus of speeches consisted of the 37 major speeches whose authorship we could verify. B. Feature Selection Four different pre-processing approaches were used on this collection of speeches. In each case, we followed standard text For each of the preprocessed sets, individual analysis of processing steps of removing numbers, removing punctuation, variance (ANOVA) was conducted where either the augmented converting to lower case. At this point, in many studies, or normalized term frequency was the response variable and the corpus is stemmed. Stemming is a process by which the speechwriter was the classification variable. Table 1 below words with the same root such as happy and happiness are summarizes the number of statistically significant (p < 0.05) converted, or stemmed, into their same common root. For ANOVA results in other words, the number of function English language documents, the Porter Stemming algorithm is words whose between-speechwriter frequency distributions the most commonly used algorithm for the task. However, we were least likely to differ from each other from chance and were concerned that stemming the corpus could remove some the 10 most significant discriminators (lowest p-value) for stylometric markers, if one author used certain verb forms or each of the 4 pre-processing groups. It is not surprising tenses more often than others. So, we performed all of our that unstemmed data sets should have more discriminating experiments on both a stemmed and an unstemmed corpus. function word frequencies, as there are more words to choose from since the stemmer summarizes different versions of the After these pre-processing steps were completed, we com- same root as a single word. bined all of the stemmed words and documents of the corpus into one Document Term Matrix. At this point, there are many Since the feature lists were restricted to function words only options for normalizing term and document frequencies. For (using the well-known Snowball stemmers stop word list as a many text mining applications, a TF-IDF (for Term Frequency, list of function words), it is hard to imagine how many of the Inverse Document Frequency) matrix is used. This kind of words, such as “has”, “got”, “was”, that were selected using matrix brings out the words most likely to capture the semantic the ANOVA tests outlined above could have any contextual essence of each document by finding the words that appear meaning whatsoever. Some other words, such as “youngest” most often in individual documents (the term frequency) or “she” could possibly be found to have some contextual Augmented Augmented Normalized Normalized Stemmed Unstemmed Stemmed Unstemmed (37-1 = 36), and then using the data from the class label (28 total) (35 total) (33 total) (48 total) being predicted as a test set. word p word p word p word p between .0008 does .0005 has .0001 has .0001 cannot .0009 between .0008 must .0001 must .0001 Predicted Author youngest .0016 cannot .0009 have .0001 have .0001 Favreau Keenan Rhodes Frankel area .0024 youngest .0016 clear .0001 clear .0001 Favreau 6 1 6 0 was .0045 areas .0042 will .0002 will .0002 Actual Keenan 2 2 1 0 has .0058 was .0045 end .0007 areas .0004 Author Rhodes 1 0 9 0 use .0112 right .0056 younger .0008 find .0006 Frankel 3 2 0 4 end .0119 has .0058 got .0011 parts .0006 Augmented, Stemmed Naive Bayes Cross-Validation Predictions (56.8% accuracy) she .0126 use .0090 was .0011 younger .0008 just .0129 she .0126 just .0013 end .0009 Predicted Author Favreau Keenan Rhodes Frankel TABLE I. LOWEST 10 P-VALUES FOR EACH SET Favreau 5 3 5 0 Actual Keenan 2 3 0 0 Author Rhodes 1 0 9 0 Frankel 1 2 0 6 value, but it is more likely that such word would be common Augmented, Unstemmed Naive Bayes Cross-Validation Predictions (62.2% accuracy) in several different categories of speeches. Predicted Author Favreau Keenan Rhodes Frankel Favreau 11 0 0 2 Actual Keenan 2 0 1 2 C. classification Author Rhodes 3 0 7 0 Frankel 1 0 0 8 Four different off-the-shelf classification methods were Normalized, Stemmed Naive Bayes Cross-Validation Predictions (70.3% accuracy) explored: Nave Bayes, K-nearest neighbors on the projections Predicted Author of Principal Component Analysis (PCA), Linear Discriminant Favreau Keenan Rhodes Frankel Analysis (LDA), and Feed-Forward Neural Networks. Each Favreau 11 0 0 2 Actual Keenan 3 0 0 2 classification method was used with all possible combinations Author Rhodes 1 0 9 0 of relevant parameters (for example, k for K-nearest Frankel 1 0 0 8 Normalized, Unstemmed Naive Bayes Cross-Validation Predictions (75.7% accuracy) neighbors). For each classifier, we chose to use leave-one-out cross-validation was used to calculate classifier accuracy due TABLE III. CONFUSION MATRICES FOR EACH PRE-PROCESSINGSET to the very small size of the training set only 37 speeches. In leave-one-out cross-validation, for each speech, a classifier is built using the other 36 speeches, and the speechwriter Since most text data, including stylometric data, is highly for that speech is predicted using that classifier. The three dimensional, it makes sense to try techniques to reduce the non-linear classifiers, Nave Bayes, PCA+Knn, and Neural dimensionality of the data. Principal Component Analysis Nets all achieved accuracy ratings around 60-70%, while (PCA) is a method that summarizes high-dimensional data by LDA achieved no better than random chance on some of finding the eigenvectors of covariance matrices of the original the pre-processing sets. These results make intuitive sense; feature vectors. The first principal component summarizes one would not expect a high-dimensional, messy data set of the data in the direction of the highest variance, and each political speeches to be easily linearly separable. The best successive principal component summarizes the highest cross-validated for each classifier on each pre-processing set amount of variance possible subject to the constraint that it is shown on Table 2. is orthogonal with preceding components. Figure 1 shows the projections of the first two principal components for each pre-processing set. Just from this simple visualization, there does appear to be some degree of authorial structure LOOCV Augm. Augm. Norm. Norm. a conclusion borne out in our cross-validation prediction Accuracy Stem. Unst. Stem. Unst. Naive Bayes 57% 62% 70% 76% accuracy of about 70% for most of the pre-processing groups. PCA + Knn 70% 78% 68% 70% LDA 57% 32% 24% 43% Neural Nets 60% 70% 65% 68% While some stylometric studies have used Linear Discriminant TABLE II. LOOCV ACCURACYFOREACHPRE-PROCESSINGSET, FOR EACH CLASSIFICATION ALGORITHM Analysis, or LDA, with success[5], in our experiment they produced the highest error rate of any of the classifiers that we tried. For the normalized, stemmed pre-processing set, LDA achieved the same performance as random guessing: 25%. Naive Bayes classifiers are commonly used for a These results confirm what our visualizations above suggest wide range of classification problems, like e-mail spam the data are not linearly separable, and the other 3 classifiers detection. Nave Bayes classifiers assign category labels used, which are all capable of modeling complex decision using Bayes theorem with the simplifying assumption that boundaries, had much higher cross-validated prediction all observations are independent (hence the naivete in Nave accuracy. Bayes). For each pre-processing set, we used the function Authorial structure was also evident using neural nets. We words and their frequencies from our analysis of variance as used feed-forward neural nets with back-progagation using feature vectors to train Nave Bayes classifiers. The leave-one- a single hidden layer. We tried different numbers of hidden out cross-validation prediction confusion matrices are shown neurons, from 1 to 36 for each pre-processing set. In each below. Each of these predictions was made by creating a case, cross-validated prediction accuracy did not vary by classifier using all of the other speeches in the training set more than 1 or 2 correct classifications for 4 or more hidden Fig. 2. Normalized, unstemmed scatterplot of top 5 most relevant features. Different colors correspond to different authors. Not lack of linear separability, though complex structure is apparent in the data

neurons. Neural nets have several advantages, all of which apply to our authorship study of Obama’s speeches[17]: they can adaptively learn from the data itself, they can generalize on a limited training set, they are good at capturing non-linear interactions between input variables, and they are tolerant of individual data points that exhibit some unusual characteristics compared to other members of the set. In the course of our experiments, we discovered that one author’s work was consistently more difficult to identify than his peers: Cody Keenan (Table 4). Cody Keenan served as an assistant speechwriter during the period of time that the other three were active, so it is possible that the speeches attributed to him may have been more collaborative than the speeches written by the more senior speechwriters. Furthermore, we also have the fewest number of speeches - 5 - for Mr. Keenan out of any of the authors, so it is also possible that our sample size is simply too small to accurately evaluate the distinguishing characteristics of his writing style. Removing Mr. Keenan’s speeches from our corpus

LOOCV Augm. Augm. Norm. Norm. Accuracy Stem. Unst. Stem. Unst. Favreau 46% 38% 85% 85% Keenan 40% 60% 0% 0% Rhodes 90% 90% 70% 90% Frankel 44% 67% 89% 89%

LOOCV Augm. Augm. Norm. Norm. Accuracy Stem. Unst. Stem. Unst. Favreau 62% 54% 85% 77% Keenan 20% 20% 0% 20% Rhodes 70% 80% 60% 60% Frankel 50% 89% 67% 78%

TABLE IV. LOOCV ACCURACIESFORINDIVIDUALAUTHORSFOR NAIVE BAYES (TOP) AND NEURAL NETS (BOTTOM) CLASSIFIERS

Fig. 1. Projections of first two principal components for each set. From improved accuracy significantly. The results for all pre- top: augmented, stemmed; augmented, unstemmed; normalized, stemmed; processing techniques and classification methods without normalized, unstemmed Keenan speeches are in Table 5. Intuitively, since Keenan recognizably President Obama’s. Together, their writings are was an assistant speechwriter for much of his tenure, we the voice of the President of the United States, but individually, speculate that his speeches were edited by the other authors, presidential speechwriters are expected to avoid excessive confusing the issue of authorship. The fact that there are so publicity. As Franklin Roosevelt put it, they are expected few speeches attributed to him, and that he did not take an to have a passion for anonymity. These results suggest that active role in the presidential campaign, suggests that his stylometry can, in fact, be used to differentiate authors who role in presidential speechwriting was quite limited compared are actively attempting to write in a similar style. to the other 3 authors. It is known that some speeches Our findings also suggest that avoiding the use of are more or less collaborative efforts[8], although principal stemming algorithms during text pre-processing can increase authors are usually identified. In the case of Mr. Keenan’s the accuracy of stylometric techniques. With the exception speeches, it would seem that his speeches were much more of linear discriminant analysis, each of our classification collaborative. To show how much clarity of results was techniques performed better on the unstemmed data sets improved by removing Keenan’s speeches from our data set, than on their corresponding stemmed data sets. This finding we also present confusion matrices for each pre-processing provides further evidence that word forms can be a source of set for Naive Bayes classification in Table 6. The difference stylistic discrimination, and therefore should not be discarded. is striking.

LOOCV Augm. Augm. Norm. Norm. V. REFERENCES Accuracy Stem. Unst. Stem. Unst. Naive Bayes 72% 72% 84% 91% 1. Holmes, D. and Kardos, J. (2003), “Who Was the Author? PCA + Knn 91% 81% 84% 81% An Introduction to Stylometry”. Chance, vol. 16, no. 2, pp. LDA 70% 80% 60% 60% Neural Nets 78% 84% 75% 75% 5-8. TABLE V. RESULTS WITHOUT KEENAN 2. Mosteller, F. and Wallace, D. (1964), “Applied Bayesian and Classical Inference: The Case of the Federalist Papers,” Reading MA: Addison-Wesley.

Predicted Author Favreau Rhodes Frankel 3. Airoldi, E., Anderson, A., Fienberg, S., and Skinner, Favreau 9 3 1 K. (2006), “Who Wrote Ronald Reagan’s Radio Addresses?” Actual Rhodes 1 9 0 Author Frankel 4 0 5 Bayesian Analysis, vol. 1, no. 2., pp. 289-320. Augmented, Stemmed Naive Bayes Cross-Validation Predictions (71.9% accuracy) 4. Baayen, R.H., Van Halteren, H., Nejit, A., and Tweedie, Predicted Author Favreau Rhodes Frankel F. (2002) “An Experiment in Authorship Attribution.” In Favreau 7 5 1 Proceedings of the 6th International Conference on the Actual Rhodes 1 9 0 Statistical Analysis of Textual Data (JADT 2002). Author Frankel 2 0 7 Augmented, Unstemmed Naive Bayes Cross-Validation Predictions (71.9% accuracy) 5. Brennan, M., Afroz, S., and Greenstadt, R. (2011), Predicted Author Favreau Rhodes Frankel “Adversarial Stylometry: Circumventing Authorship Favreau 12 0 1 Recognition to Preserve Privacy and Anonymity.” ACM Actual Rhodes 3 7 0 Trans. Info. Syst. Sec. 15, 3, Article 12, 22 pages. Author Frankel 1 0 8 Normalized, Stemmed Naive Bayes Cross-Validation Predictions (84.4% accuracy) 6. Peterson, H. (2013, Feb. 28), “Outgoing speechwriter Predicted Author reveals how Obama casually asked him to spike a bin Laden Favreau Rhodes Frankel Favreau 13 0 0 joke on the night of the Pakistan raid as he reflects on his 8 Actual Rhodes 2 8 0 years with the president.” The Daily Mail [Online]. Available Author Frankel 1 0 8 Normalized, Unstemmed Naive Bayes Cross-Validation Predictions (90.6% accuracy) at: www.dailymail.co.uk/news/article-2286126/Jon-Favreau- reveals-Obama-asked-spike-Osama-bin-Laden-joke-Pakistan- TABLE VI. CONFUSIONMATRICESAFTEREXCLUDING KEENAN raid.html SPEECHES 7. Stein, S. (2013, Jan. 24), “Obama’s Inaugural Address ‘One of the Hardest Speeches I’ve Written,’ Jon Favreau Says.” Huffington Post [Online]. Available IV. CONCLUSION at: www.huffingtonpost.com/2013/01/24/obamas-inaugural- Our study shows evidence that authors can still be identified address-jon-favreau n 2545236.html when they consciously attempt to speak in a unified voice, although with somewhat diminished accuracy. Similar ap- 8. Berry, M., and Gottheimer, J. (2010), “Power in Words: proaches to the ones presented here yielded superior accuracy the Stories Behind Barack Obama’s Speeches.” Boston, MA: in [2,3,15,17], but we did not experience the same degra- Beacon Press. dation in accuracy as the adversarial stylometry experiment conducted in [3]. Although the individual speechwriters are 9. Kohut, M. (2010, Jan. 11), “Alumnus Cody Keenan not attempting to specifically defeat stylometric analyses, they MPP 2008: White House Wordsmith.” Harvard are attempting to achieve a consistent rhetorical tone that is Kennedy School Alumni Stories [Online]. Available at: www.hks.harvard.edu/news-events/news/alumni/alum-cody- keenan-jan10

10. Franke-Ruta, G. (2013, Feb. 12), “Who is Cody Keenan, Obama’s SOTU Speechwriter?” [Online]. Avail- able at: www.theatlantic.com/politics/archive/2013/02/who-is- cody-keenan-obamas-sotu-speechwriter/272938/

11. Brotzen, F. (2009, Aug. 6), “Rice alum helps craft Obama’s international message.” Rice University News [Online]. Available at: news.rice.edu/2009/08/06/rice-alum- helps-craft-obamas-international-message/

12. Landler, M. (2013, Mar. 15), “Worldly at 35, and Shaping Obama’s Voice.” [Online]. Available at: www.nytimes.com/2013/03/16/world/middleeast/benjamin- rhodes-obamas-voice-helps-shape-policy.html

13. Nicholas, P. (2010, Sep. 3), “The brain behind Obama’s speeches.” The Los Angeles Times [Online]. Available at: articles.latimes.com/2010/sep/03/nation/la-na-speechwriter- 20100903

14. Burrows, J.F. (1992), “Not Unless You Ask Nicely: The Interpretative Nexus Between Analysis and Information,” Literary and Linguistic Computing, vol. 7, pp. 91-109.

15. Binongo, J. (2003), “Who Wrote the 15th Book of Oz? An Application of Multivariate Analysis to Authorship Attribution.” Chance, vol. 16, no. 2, pp. 9-17.

16. Savoy, J., and Zubaryeva, O. (2011). “Classification based on Specific Vocabulary.” In 2011 IEEE/WIC/ACM International Confences on Web Intelligence and Intelligent Agent Technology.

17. Tweedie, F. J., Singh, S., and Holmes, D. I. (1996), “Neural Network Applications in Stylometry: the Federalist Papers.” Computers and the Humanities vol 30. pp. 1-10.