sign in sign up
<<
Home , Philocles, Philopoemen

Stylometric Classification of Literary Texts by Genre

Efthimios Tim Gianitsos Thomas J. Bolt Pramit Chaudhuri Department of Computer Science Department of Department of Classics University of Texas at Austin University of Texas at Austin University of Texas at Austin Joseph P. Dexter Neukom Institute for Computational Science Dartmouth College

Abstract analyses of literary genre have been reported, us- ing both English and non-English corpora such Classification of texts by genre is an impor- as classical Malay poetry, German novels, and tant application of natural language process- Arabic religious texts (Tizhoosh et al., 2008; Ku- ing to literary corpora but remains understud- mar and Minz, 2014; Jamal et al., 2012; Hettinger ied for premodern and non-English traditions. et al., 2015; Al-Yahya, 2018). However, computa- We develop a stylometric feature set for an- cient Greek that enables identification of texts tional prediction of even relatively coarse generic as prose or verse. The set contains over 20 distinctions (such as between prose and poetry) re- primarily syntactic features, which are calcu- mains unexplored for classical Greek literature. lated according to custom, language-specific Encompassing the epic poems of , the heuristics. Using these features, we classify tragedies of , , and , almost all surviving classical Greek literature the historical writings of , and the phi- as prose or verse with >97% accuracy and F1 losophy of and , the surviving lit- score, and further classify a selection of the verse texts into the traditional genres of epic erature of is foundational for the and drama. Western literary tradition. Here we report a com- putational analysis of genre involving the whole of 1 Introduction the classical Greek literary tradition. Using a cus- tom set of language-specific stylometric features, Classification of large corpora of documents into we classify texts as prose or verse and, for the coherent groups is an important application of nat- verse texts, as epic or drama with >97% accuracy. ural language processing. Research on document An important advantage of our approach is that all organization has led to a variety of successful of the features can be computed without syntactic methods for automatic genre classification (Sta- parsing, which remains in an early phase of de- matatos et al., 2000; Santini, 2007). Computa- velopment for ancient Greek. As such, our work tional analysis of genre has most often involved illustrates how computational modeling of liter- material from a single source (e.g., a newspaper ary texts, where research has concentrated over- corpus, for which the goal is to distinguish be- whelmingly on modern English literature (Elson tween news articles and opinion pieces) or from et al., 2010; Elsner, 2012; Bamman et al., 2014; standard, well-curated test corpora that contain Chaturvedi et al., 2016; Wilkens, 2016), can be ex- primarily non-literary texts (e.g., the Brown cor- tended to premodern, non-Anglophone traditions. pus or equivalents in other languages) (Kessler et al., 1997; Petrenz and Webber, 2011; Amasyali 2 Stylometric feature set for ancient and Diri, 2006). Greek Notions of genre are also of substantial impor- tance to the study of literature. For instance, ex- The feature set is composed of 23 features cov- amination of the distinctive characteristics of var- ering four broad grammatical and syntactical cat- ious forms of poetry dates to and egories. The majority of the features are func- Rome (for instance, by Aristotle and Quintilian) tion or non-content words, such as pronouns and and remains an active area of humanistic research syntactical markers; a minority concern rhetorical today (Frow, 2015). A number of computational functions, such as questions and uses of superla-

52 Proc. of the 3rd Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature, pp. 52–60 Minneapolis, MN, USA, June 7, 2019. c 2019 Association for Computational Linguistics Feature Feature Genre Precision Recall Pronouns and non-content adjectives 4 verse 0.96 0.96 1 ἄλλος 4 prose 0.97 1 2 ἀυτός 10 verse 1 0.93 3 demonstrative pronouns 10 prose 1 1 4 selected indefinite pronouns 14 verse 0.97 0.96 5 personal pronouns 14 prose 1 1 6 reflexive pronouns 19 verse 1 0.89 Conjunctions and particles 19 prose 1 1 7 conjunctions 20 verse 1 0.85 8 μέν 20 prose 1 1 9 particles Subordinate clauses Table 2: Error analysis of non-exact features. The fea- 10 circumstantial markers tures are numbered as in Table 1. 11 conditional markers 12 ἵνα 13 ὅπως 14 sentences with relative pronouns in standard studies of English stylistics, such as 15 temporal and causal markers pronouns, others are specific to ancient Greek. At- 16 ὥστε not preceded by ἤ tention to language-specific features enhances sty- 17 mean length of relative clauses lometric methods developed for the English lan- Miscellaneous guage and not directly transferable to languages 18 interrogative sentences possessing a different structure (Rybicki and Eder, 19 superlatives 2011; Kestemont, 2014). Greek particles, for ex- 20 sentences with ὦ exclamations ample, are uninflected adverbs used for a wide 21 ὡς range of logical and emotional expressions; in En- 22 mean sentence length glish their equivalent meaning is often expressed 23 variance of sentence length by a phrase or, in speech, tone. In order to avoid significant problems arising from dialectical vari- Table 1: Full set of ancient Greek stylometric features. ation, including a large increase in homonyms, we restrict features to the Attic dialect, in which the majority of classical Greek texts were composed. tive adjectives and adverbs. Function words are Many features are computed by counting all in- standard features in stylometric research on En- flected forms of the appropriate word(s), which glish (Stamatatos, 2009; Hughes et al., 2012) and can be found in any standard ancient Greek text- have also been used in studies of ancient Greek lit- book or grammar such as Smyth (1956). A de- erature (Gorman and Gorman, 2016). Our feature tailed description of the methods for computing selection is not drawn from a prior source but has the features is given in Appendix A. been devised based on three criteria: amenabil- ity to exact or approximate calculation without Calculation of five features relies on heuris- use of syntactic parsing, substantial applicability tics to disambiguate between words of similar to the corpus, and diversity of function. The fea- morphology. (All other features can be calcu- ture set is listed in Table1. The first restric- lated exactly.) To assess the effectiveness of tion is necessary because a general-purpose syn- these heuristics, we hand-annotate the five features tactic parser remains to be developed for classi- in a representative sub-corpus containing three cal Greek (notwithstanding promising early-stage verse (Homer’s Odyssey 6, Quintus of Smyrna’s research through the open-source Classical Lan- Posthomerica 12, and Euripides’ Cyclops) and two guage Toolkit and other projects). All features are prose (Lysias 7 and ’s Caius Gracchus) per-character frequencies with the exception of a texts. Table2 lists the precision and recall of each handful that are normalized by sentence (indicated feature on the aggregated verse and prose texts. In in the table by “sentences with...”). every instance, the precision is > 0.95 and the re- Although some features overlap with those used call is > 0.85.

53 3 Experimental setup Accuracy (%) Weighted F1 (%) Fold 1 98.0 98.0 3.1 Dataset Fold 2 100 100 We use a corpus of ancient Greek text files, which Fold 3 99.3 99.3 was assembled by the Perseus Digital Library Fold 4 98.7 98.7 and further processed by Tesserae Project (Crane, Fold 5 100 100 1996; Coffee et al., 2012). A full list of texts is Mean 99.2 99.2 provided in Appendix B. Each file typically con- S.D. 1.9 1.9 tains either an entire work of literature (e.g., a play Overall 98.9 98.9 or a short philosophical treatise) or one book of a S.D. 0.8 0.8 longer work (e.g., Book 1 of Homer’s Iliad). 29 files are composites of multiple books included Table 3: Performance of prose vs. verse classifier for elsewhere in the Tesserae corpus and are omitted ancient Greek literary texts. from our analysis, leaving 751 files. In total, this corpus contains essentially all surviving classical Feature Gini S.D. Greek literature and spans from the 8th century ἀυτός 0.209 0.074 BCE to the 6th century CE. conjunctions 0.159 0.062 For our first experiment, we hand-annotate the demonstrative pronouns 0.121 0.057 full set of texts as prose (610 files) or verse reflexive pronouns 0.118 0.049 (141 files) according to standard conventions (Ap- μέν 0.0623 0.029 pendix B). For the second experiment, we hand- annotate the verse texts as epic (82 files) and Table 4: Feature rankings for prose vs. verse classifier. drama (45 files), setting aside 14 files that contain poems of other genres (Appendix C). 4 Results 3.2 Feature extraction All text processing is done using Python 3.6.5. 4.1 Prose vs. verse classification We first tokenize the files from the Tesserae cor- pus into either words or sentences using the Nat- Using the workflow described in Section 3.3, we ural Language Toolkit (NLTK; v. 3.3.0) (Bird classify each of the literary texts in the corpus et al., 2009). For sentence tokenization, we as prose or verse. Table3 lists the accuracy and use the PunktSentenceTokenizer class of NLTK weighted F1 score for a sample cross-validation Greek (Kiss and Strunk, 2006). After tokeniza- trial, along with the mean for that trial and overall tion, the features are calculated either by tabu- mean across the 400 trials. We find that the texts lating instances of signal n-grams or (for length- can be classified as prose or verse with extremely based features) counting characters exclusive of high accuracy using the set of 23 stylometric fea- whitespace, as described in Appendix A. tures and that, despite the small size of the corpus, classifier performance is robust to the choice of 3.3 Supervised learning cross-validation partition. The five highest-ranked All supervised learning is done using Python features are given in Table4. Outside of these five, 3.6.5. For each experiment, we use the scikit-learn no other feature has a Gini importance of > 0.05. (v. 0.19.2) implementation of the random forest All five features predominate in prose rather than classifier. A full list of hyperparameters and other poetry, of which three are pronouns or pronom- settings is given in Appendix D. For each binary inal adjectives. The sustained discussions com- classification experiment (prose vs. verse and epic monly found in various prose genres may favor vs. drama), we perform 400 trials of stratified 5- the use of pronouns to avoid extensive repetition of fold cross-validation; each trial has a unique com- nouns and proper names. The high ranking of con- bination of two random seeds, one used to initial- junctions is plausibly connected to the longer sen- ize the classifier and the other to initialize the data tences characteristic of most prose (mean length splitter. Feature rankings are determined by the 205 characters, compared to 166 characters for po- average Gini importance across the 400 trials. etry).

54 Accuracy (%) Weighted F1 (%) Feature Gini S.D. Fold 1 92.3 92.0 mean sentence length 0.186 0.12 Fold 2 100 100 demonstrative pronouns 0.155 0.095 Fold 3 100 100 interrogative sentences 0.127 0.12 Fold 4 100 100 ὡς 0.117 0.11 Fold 5 100 100 variance of sentence length 0.0952 0.075 Mean 98.5 98.4 S.D. 3.4 3.6 Table 6: Feature rankings for epic vs. drama classifier. Overall 99.8 99.8 S.D. 0.9 0.9 demonstrative pronouns in succession). Another typical characteristic of dramatic plot and dialogue Table 5: Performance of epic vs. drama classifier for accounts for the third highly-ranked feature - in- ancient Greek poetry. terrogative sentences - since both tragedies and comedies often show characters in a state of uncer- 4.2 Classification of poems as epic or drama tainty or ignorance, or making inquiries of other The genres of epic and drama are in certain re- characters. Although many of the features in the spects quite distinct: they differ in length and po- full set are correlated (e.g., sentence length and etic meter, and the vocabulary of ’ various markers of subordinate clauses), none of comic plays is unlike either epic or tragedy. In the top 5 plausibly are, suggesting that the analy- other aspects of form and content, however, they sis identifies a diverse set of stylistic markers for have much in common, including passages of di- epic and drama. rect speech, high register diction, and mytholog- 4.3 Misclassifications ical subject matter. The playwright Aeschylus is even reported to have described his tragedies For epic vs. drama, no text is misclassified in as “slices from the great banquets of Homer” more than 12% of the trials. For prose vs. verse, (Athenaeus, Deipnosophistae 8.347E). The sim- only five texts are misclassified in >50% of the tri- ilarities between epic and drama thus present an als (Demades, On the Twelve Years; Dionysius of intuitively greater challenge for classification. Halicarnassus, De Antiquis Oratoribus Reliquiae Table5 summarizes the results of the epic vs. 2; Plato, Epistle 1; Aristotle, Virtues and Vices; drama experiment, for which we achieve perfor- Sophocles, Ichneutae). Most of the common mis- mance comparable to that of the prose vs. verse classifications result from highly fragmentary or experiment. Table6 lists the top features, which short texts. Almost half the speech of Demades, reflect several important differences between the for example, contains short or incomplete sen- genres. The most important feature - sentence tences. The misclassified text of Dionysius of length - highlights the relatively shorter sentences Halicarnassus amounts to only a few unconnected of drama compared to epic, which can be ex- sentences; Sophocles’ Ichneutae (the only verse plained at least in part by the rapid exchanges be- text misclassified in over half the trials) is also tween speakers that occur throughout both tragedy fragmentary. The third most frequently misclas- and comedy. Although sentence length is a fea- sified text, Plato’s First Epistle, in fact highlights ture that can be affected by modern editorial prac- the classifier’s effectiveness, as it contains several tice, the difference between drama and epic on verse quotations, which (given the short length of this score is sufficiently large that it cannot be ex- the text) plausibly account for the error. plained by variations in editorial practice alone (< 5 Conclusion 80 characters/sentence on average across dramatic texts, > 150 characters/sentence for epic). The im- In this paper, we demonstrate that ancient Greek portance of demonstrative pronouns, ranked sec- literature can be classified by genre using a ond, plausibly captures a different side of drama straightforward supervised learning approach and - the habit of characters referring, often indexi- stylometric features calculated without syntactic cally, to persons or objects in the plot (e.g., ἐκ῀εινος parsing. Our work suggests a number of natu- ῟ουτόςἐιμι, ekeinos houtos eimi, “I am that very ral follow-up analyses, especially extension of the man,” Euripides, Cyclops 105, which uses two experiments to encompass the full range of tradi-

55 tional prose genres (such as historiography, philos- Gregory Crane. 1996. Building a digital library: The ophy, and oratory) and application of the feature Perseus Project as a case study in the humanities. In set to other questions in classical literary criticism. Proceedings of the First ACM International Confer- ence on Digital Libraries, pages 3–10. In addition, we hope that our heuristic approach will motivate and inform analogous work on other Micha Elsner. 2012. Character-based kernels for nov- premodern traditions for which natural language elistic plot structure. In Proceedings of the 13th Conference of the European Chapter of the Associa- processing research remains at an early stage. tion for Computational Linguistics, pages 634–644. Acknowledgments David K. Elson, Nicholas Dames, and Kathleen R. McKeown. 2010. Extracting social networks from This work was conducted under the auspices of literary fiction. In Proceedings of the 48th Annual the Quantitative Criticism Lab (www.qcrit.org), Meeting of the Association for Copmutational Lin- guistics, pages 138–147. an interdisciplinary group co-directed by P.C. and J.P.D. and supported by a National Endowment for John Frow. 2015. Genre. Routledge, London and New the Humanities Digital Humanities Start-Up Grant York. (grant number HD-10 248410-16) and an Ameri- Vanessa B. Gorman and Robert J. Gorman. 2016. Ap- can Council of Learned Societies (ACLS) Digital proaching questions of text reuse in ancient greek Extension Grant. T.J.B. was supported by an En- using computational syntactic stylometry. Open gaged Scholar Initiative Fellowship from the An- Linguistics, 2:500–510. drew W. Mellon Foundation, P.C. by an ACLS Lena Hettinger, Martin Becker, Isabella Reger, Fotis Digital Innovation Fellowship and a Mellon New Jannidis, and Andreas Hotho. 2015. Genre classi- Directions Fellowship, and J.P.D. by a Neukom fication on German novels. In 2015 26th Interna- Fellowship. tional Workshop on Database and Expert Systems Applications, pages 138–147.

James M. Hughes, Nicholas J. Fotia, David C. References Krakauer, and Daniel N. Rockmore. 2012. Quan- titative patterns of stylistic influence in the evolution Maha Al-Yahya. 2018. Stylometric analysis of classi- of literature. Proceedings of the National Academy cal Arabic texts for genre detection. The Electronic of Sciences USA, 109:7682–7686. Library, 36:842–855. Noraini Jamal, Masnizah Mohd, and Shahrul Azman M. Fatih Amasyali and Banu Diri. 2006. Automatic Noah. 2012. Poetry classification using support Turkish text categorization in terms of author, genre vector machines. Journal of Computer Science, and gender. In Christian Kop, Gunther¨ Fliedl, Hein- 8:1411–1416. rich C. Mayr, and Elisabeth Mtais, editors, Natu- ral Language Processing and Information Systems, Brett Kessler, Geoffrey Numberg, and Hinrich Schutze.¨ pages 221–226. Springer-Verlag, Berlin. 1997. Automatic detection of text genre. In Pro- ceedings of the 35th Annual Meeting of the Associa- David Bamman, Ted Underwood, and Noah A. Smith. tion for Computational Linguistics and Eighth Con- 2014. A Bayesian mixed effects model of literary ference of the European Chapter of the Association character. In Proceedings of the 53nd Annual Meet- for Computational Linguistics, pages 32–38. ing of the Association for Computational Linguis- tics, pages 370–379. Mike Kestemont. 2014. Function words in authorship attribution. From black magic to theory? In Pro- ceedings of the 3rd Workshop on Computational Lin- Steven Bird, Ewan Klein, and Edward Loper. guistics for Literature @ EACL 2014 2009. Natural Language Processing with Python. , pages 59–66. O’Reilly Media. Tibor Kiss and Jan Strunk. 2006. Unsupervised mul- tilingual sentence boundary detection. Computa- Snigdha Chaturvedi, Hal Daume´ III, Shashank Srivas- tional Linguistics, 32:485–525. tava, and Chris Dyer. 2016. Modeling evolving rela- tionships between characters in literary novels. In Vipin Kumar and Sonajharia Minz. 2014. Poem clas- Proceedings of the Thirtieth AAAI Conference on sification using machine learning approach. In Pro- Artificial Intelligence, pages 2704–2710. ceedings of the Second International Conference on Soft Computing for Problem Solving, pages 675– Neil Coffee, J.-P. Koenig, Shakthi Poornima, Roelant 682. Ossewaarde, Christopher Forstall, and Sarah Jacob- son. 2012. Intertextuality in the digital age. Trans- Philipp Petrenz and Bonnie Webber. 2011. Stable clas- actions of the American Philological Association, sification of text genres. Computational Linguistics, 142:383–422. 37:385–393.

56 Jan Rybicki and Maciej Eder. 2011. Deeper Delta • Reflexive pronouns are computed by count- across genres and languages: Do we really need the ing all inflected forms of ἐμαυτοῦ (emautou, most frequent words? Literary and Linguistic Com- “he himself”). puting, 26(3):315–321. Marina Santini. 2007. Automatic genre identification: A.2 Conjunctions and particles Towards a flexible classification scheme. In Pro- • Conjunctions are computed by counting all ceedings of the 1st BCS IRSG Conference on Future Directions in Information Access, page 1. instances of the common conjunctions τε, τ΄ (te or t, “and”), καί, καὶ (kai, “and”), ἀλλά, Herbert Weir Smyth. 1956. Greek Grammar. Revised ἀλλὰ (alla, “but”), καίτοι (kaitoi, “and in- by Gordon M. Messing. Harvard University Press. deed”), οὐδέ, οὐδὲ, οὐδ΄ (oude or oud, “and Efstathios Stamatatos. 2009. A survey of modern au- not”), μηδέ, μηδὲ, μηδ΄ (mede or med, “and thorship attribution methods. Journal of the Amer- not”), οὔτε, οὔτ΄ (oute or out, “and not”), ican Society For Information Science and Technol- μήτε, μήτ΄ (mete or met, “and not”), and ἤ, ogy, 60:538–556. ἢ (e, “or”). Efstathios Stamatatos, Nikos Fakotakis, and George Kokkinakis. 2000. Automatic text categorization in • μέν (men, “indeed”) is computed by counting terms of genre and author. Computational Linguis- all instances of μέν and μὲν. tics, 26:471–495. • Particles are computed by counting all in- Hamid Tizhoosh, Farhang Sahba, and Rozita Dara. 2008. Poetic features for poem recognition: A com- stances of ἄν, ἂν (an, a particle used to ex- parative study. Journal of Pattern Recognition Re- press uncertainty or possibility), ἆρα (ara, search, 3:24–39. “then”), γέ, γ΄ (ge or g, “at least”), δ΄, δέ, Matthew Wilkens. 2016. Genre, computation, and the δὲ (d or de, “but”), δή, δὴ (de, “indeed”), varieties of twentieth-century U.S. fiction. Journal ἕως (heos, “until”), κ΄, κε, κέ, κὲ, κέν, κὲν, of Cultural Analytics. κεν (k, ke, ken, a particle used to express uncertainty or possibility), (ma, used in A Details of stylometric features for μά oaths and affirmations, “by”), μέν, μὲν (men, ancient Greek “indeed”), μέντοι (mentoi, “however”), μὴν, A.1 Pronouns and non-content adjectives μήν (men, “truly”), μῶν (mon, “surely not”), • ἄλλος (allos, “other”) is computed by count- νύ, νὺ, νυ (nu, “now”), οὖν (oun, “so”), περ ing all inflected forms of ἄλλος, -η, -ο. (per, an intensifying particle, “very”), πω (po, “yet”), and τοι (toi, “let me tell you”). • αὐτός (autos, “self” or “him/her/it”) is com- puted by counting all inflected forms of A.3 Subordinate clauses αὐτός, -ή, -ό. • Circumstantial markers are computed by counting all instances of ἔπειτα, ἔπειτ΄ (epeita • Demonstrative pronouns are computed by or epeit, “then”), ὅμως (homos, “all the counting all inflected forms of the three same”), ὁμῶς (homos, “equally”), καίπερ Greek demonstrative pronouns οὗτος, αὕτη, (kaiper, “although”), and ἅτε, ἅτ΄ (hate or hat, τοῦτο (houtos, haute, touto, “this”), ὅδε, “seeing that”). ἥδε, τόδε (hode, hede, tode, “this”), and ἐκεῖνος, ἐκεῖνη, ἐκεῖνο (ekeinos, ekeine, • Conditional markers are computed by count- ekeino, “that”). ing all instances of εἰ, εἴ, εἲ, ἐάν, and ἐὰν (ei, ei, ei, ean, ean, all translated “if”). • Selected indefinite pronouns are computed by counting all inflected forms of τις, τις, τι (tis, • ἵνα (hina, an adverb of place often translated tis, ti, “any”) in non-interrogative sentences. “where” or a conjunction indicating purpose Interrogative sentences are excluded because often translated “in order that”) is computed the Greek interrogative pronoun (τίς) is often by counting all instances of ἵνα and ἵν΄ (hin). identical in form to the indefinite pronoun. • ὅπως (hopos, an adverb of manner often • Personal pronouns are computed by counting translated “how” or a conjunction indicating all inflected forms of the pronouns ἐγώ (ego, purpose often translated “in order that”) is “I”) and σύ (su, “you”). computed by counting all instances of ὅπως.

57 • Fraction of sentences with a relative clause is other possibilities) is computed by counting determined by counting sentences that have all instances of ὡς. one or more of the inflected forms of the • Greek relative pronouns ὅς, ἥ, ὅ (hos, he, ho, Mean and variance of sentence length is de- “who” or “which”). termined by counting the number of char- acters in each tokenized sentence (see Sec- • Temporal and causal markers are computed tion 3.2 of main paper). by counting all instances of μέκρι (mekri, B List of ancient Greek literary texts “until”), ἕως (heos, “until”), πρίν (prin, “be- fore”), ἐπεί (epei, “when”), ἐπειδή (epeide, Verse texts: Aeschylus, Agamemnon, Eumenides, “after” or “since”), ἐπειδάν (epeiden, “when- Libation Bearers, Persians, Prometheus Bound, ever”), ὅτε (hote, “when”), and ὅταν (hotan, Seven Against Thebes, and Suppliant Women; “whenever”). Apollonius, Argonautica; Aristophanes, Acharni- ans, Birds, Clouds, Ecclesiazusae, Frogs, Knights, • ὥστε (hoste, a conjunction used to indicate a Lysistrata, Peace, Plutus, , result, “so as to”) not preceded by ἤ is cal- and Wasps; , Dithyrambs and Epini- culated by counting all instances of ὥστε not cians; Bion of Phlossa, Epitaphius, Epithala- immediately preceded by ἤ. This limitation is mium, and Fragmenta; Callimachus, Epigrams imposed to exclude instances in which ὥστε and Hymns; Colluthus, Rape of Helen; Euripides, is part of a comparative phrase. Alcestis, , Bacchae, Cyclops, Electra, Hecuba, Helen, Heracleidae, Heracles, Hippoly- • The mean length of relative clauses is deter- tus, Ion, Iphigenia at Aulis, Iphigenia in Tauris, mined by counting the number of characters Medea, , Phoenissae, Rhesus, Suppliants, between each relative pronoun and the next and Trojan Women; Homer, Iliad and Odyssey; punctuation mark. , Podraga; Lycophron, Alexandra; Non- A.4 Miscellaneous nus of Panopolis, Dionysiaca; Oppian, Halieu- tica; Oppian of Apamea, Cynegetica; , Isth- • Interrogative sentences are computed by means, Nemeans, Olympians, and Pythians; Quin- counting all instances of “;” (the Greek ques- tus Smyrnaeus, Fall of ; Sophocles, Ajax, tion mark). Antigone, Electra, Ichneutae, at Colonus, Oedipus Tyrannus, , and Trachiniae; • Regular superlatives adjectives are computed Theocritus, Epigrams; Tryphiodorus, The Taking by counting all instances of -τατος, -τάτου, of Ilios. -τάτῳ, -τατον, -τατοι, -τάτων, -τάτοις, - Prose texts: Achilles Tatius, Leucippe et Cli- τάτους, -τάτη, -τάτης, -τάτῃ, -τάτην, - tophon; Aelian, De Natura Animalium, Epistu- τάταις, -τάτας, -τατα, -τατά, and τατε at lae Rusticae, and Varia Historia; Aelius Aris- word end. One inflected form, -ταται, is tides, Ars Rhetorica and Orationes; , excluded so as to avoid confusion with the Against Ctesiphon, Against Timarchus, and On Homeric third person singular middle/passive the Embassy; Andocides, Against , On indicative verb ending -αται. This method His Return, On the Mysteries, and On the Peace; does not detect certain irregular superlatives, Antiphon, Against the Stepmother for Poisoning, such as ἄριστος (aristos, “best”) or πρῶτος First Tetralogy, Second Tetralogy, Third Tetralogy, (protos, “first”), which would be significantly On the Murder of Herodes, and On the Choreutes; harder to disambiguate from non-superlative Apollodorus, Epitome and Library; Appian, Civil forms. Wars; Aretaeus, Curatione Acutorum Morbum • Sentences with ὦ exclamations is determined and Signorum Acutorum Morbum; Aristotle, Con- by identifying sentences that have at least one stitution, Economics, Eudemian Ethics, Meta- instance of ὦ (o, “O”), a Greek exclamation. physics, Nicomachean Ethics, Poetics, Politics, Rhetoric, and Virtues and Vices; Athenaeus, Deip- • ὡς (hos, an adverb of manner often trans- nosophists; Barnabas, Barnabae Epistulae; Basil lated “how” or a conjunction often translated of Caesarea, De Legendis and Epistulae; Calli- as “that,” “so that,” or “since,” among several stratus, Statuarum Descriptiones; Chariton, De

58 Chaerea; Clement, Exhortation, Protrepticus, and tione, De Syria Dea, Dearum Iudicium, , Quis Dis Salvetur; Demades, On the Twelve Deorum Consilium, Dialogi Deorum, Dialogi Years; Demetrius, Elocutione; , Marini, Dialogi Meretricii, Dialogi Mortuorum, Against Androtion, Against Apatourius,Against Dipsades, Electrum, Eunuchus, Fugitivi, Gallus, Aphobus, Against Aristocrates, Against Aristogi- Harmonides, Hercules, Hermotimus, Herodotus, ton, Against Boeotus, Against Callicles, Against , Hippias, Icaromenippus, Imagines, Iu- Callippus, Against Conon, Against Dionysodorus, dicium Vocalium, Iuppiter Confuatus, Iuppiter Against Eubulides, Against Evergus and Mnesibu- Tragoedus, Lexiphanes, Macrobii, Muscae En- lus, Against Lacritus, Against Leochares, Against comium, Navigium, Necyomantia, Nigrinus, Pa- Leptines, Against Macartatus, Against Midias, triae Encomium, Phalaris, Philopseudes, Pis- Against Nausimachus and Xenopeithes, Against cator, Pro Imaginibus, Pro Lapsu Inter Salu- Neaera, Against Nicostratus, Against Olympi- tandum, Prometheus, Prometheus Es In Verbis, odorus, Against Onetor, Against Pantaenetus, Pseudologista, Quomodo Historia Conscribenda Against Phaenippus, Against Phormio, Against Sit, Rhetorum Praeceptor, Saturnalia, Scytha, Polycles, Against Spudias, Against Stephanus, Soleocista, Somnium, Symposium, Timon, Toaxris Against Theocrines, Against Timocrates, Against vel Amicitia, Tyrannicida, Verae Historiae, Vi- Timotheus, Against Zenothemis, Erotic Essay, Ex- tarum Auctio, and Zeuxis; Lycurgus, Against ordia, For Phormio, For the Megalopitans, Fu- Leocrates; Lysias, Speeches; Marcus Aurelius, M. neral Speech, Letters, Olynthiac,On Organiza- Antoninus Imperator Ad Se Ipsum; , De- tion, On the Accession of Alexander, On the Cher- scription of Greece; Philostratus the Athenian, sonese, On the Crown, On the False Embassy, De Gymnastica, Epistulae et Dialexeis, Hero- On the Halonnesus, On the Liberty of the Rho- icus, Vita Apollonii, and Vitae Sophistarum; Philo- dians, On the Navy, On the Peace, On the Tri- stratus the Lemnian, Imagines; Plato, Alcibi- erarchic Crown, Philip, Philippic, and Reply to ades, Apologia, Charmides, Cleitophon, Cratylus, Philip; Dinarchus, Against Aristogiton, Against Critias, Crito, Epinomis, Epistles, Erastai, Eu- Demosthenes, and Against ; Dionysius thydemus, Euthyphro, , , Hip- of Halicarnassus, Ad Ammaeum, Antiquitates Ro- pias Maior, Hippias Minor, Ion, Laches, Leges, manae, De Antiquis Oratoribus, De Composi- Lovers, Lysis, Menexenus, Meno, Minos, Par- tione Verborum, De Demosthene, De Dinarcho, menides, Phaedo, Phaedrus, Philebus, Protago- De Isaeo, De Isocrate, De Lysia, De Thucydide, ras, Respublica, Sophista, Statesman, Symposium, De Thucydidis Idiomatibus, Epistula ad Pom- Theaetetus, Theages, and Timaeus; Plutarch, Ad peium, and Libri Secundi de Antiquis Oratoribus Principem Ineruditum, Adversus Colotem, Aemil- Reliquiae; Epictetus, Discourses, Enchiridion, ius Paulus, Agesilaus, Agis, Alcibiades, Alexan- and Fragments; , Elements; Eusebius of der, Amatoriae Narrationes, Amatorius, An Recte Caesarea, Historia Ecclesiastica; Flavius Jose- Dictum Sit Latenter Esse Vivendum, An Seni Re- phus, Antiquitates Judaicae, Contra Apionem, De spublica Gerenda Sit, An Virtus Doceri Possit An Bello Judaico, and Vita; Galen, Natural Faculties; Vitiositas Ad Infelicitatem Sufficia, Animine An Herodotus, Histories; , De Aere Aquis Corporis Affectiones Sint Piores, Antony, Apoph- et Locis, De Alimento, De Morbis Popularibus, thegmata Laconica, Aquane An Ignis Sit Utilior, De Prisca Medicamina, and Jusjurandum; Hyper- Aratus, , Artaxerxes, Bruta Animalia Ra- ides, Against Athenogenes, Against Demosthenes, tione Uti, Brutus, Caesar, Caius Gracchus, Caius Against Philippides, Funeral Oration, In Defense Marcius Coriolanus, Caius Marius, Camillus, of Euxenippus, and In Defense of Lycophron; Cato Minor, , , Cleomenes, Compa- Isaeus, Speeches; Isocrates, Letters and Speeches; rationis Aristophanes et Menandri Compendium, Lucian, Abdicatus, Adversus Indoctum et Libros Comparison of Aegisalius and , Compar- Multos Ementem, Alexander, Anacharsis, Apolo- ison of Agis Cleomenes and Gracchi, Compar- gia, Bacchus, Bis Accusatus Sive Tribunalia, Ca- ison of Alcibiades and Coriolanus, Comparison lumniae Non Temere Credundum, Cataplus, Con- of Aristides and Cato, Comparison of Demetrius templantes, De Astrologia, De Domo, De Luctu, and Antony, Comparison of Demosthenes with De Mercede, De Morte Peregrini, De Parasito Sive Cicero, Comparison of Dion and Brutus, Com- Artem Esse Parsiticam, De Sacrificiis, De Salta- parison of and Cimon, Comparison of

59 Lycurgus and Numa, Comparison of C Genre labels for verse texts and , Comparison of and Crassus, Epic: Apollonius, Argonautica; Colluthus, Rape Comparison of and Marcellus, Com- of Helen; Homer, Iliad and Odyssey; Nonnus of parison of and Fabius Maximus, Com- Panopolis, Dionysiaca; Oppian, Halieutica; Op- parison of and Titus, Comparison pian of Apamea, Cynegetica; Quintus Smyrnaeus, of Sertorius and , Comparison of Fall of Troy; Tryphiodorus, The Taking of Ilios. and Publicola, Comparison of and Ro- mulus, Comparison of and Aemilius, Drama: Aeschylus, Agamemnon, Eumenides, Conjugalia Praecepta, Consolatio ad Apollonium, Libation Bearers, Persians, Prometheus Bound, Consolatio ad Uxorem, Crassus, De Alexandri Seven Against Thebes, and Suppliant Women; Magni Fortuna aut Virtute, De Amicorum Mul- Aristophanes, Acharnians, Birds, Clouds, Ec- titudine, De Amore Prolis, De Animae Procre- clesiazusae, Frogs, Knights, Lysistrata, Peace, atione in Timaeo, De Capienda Ex Inimicis Util- Plutus, Thesmophoriazusae, and Wasps; Euripi- itate, De Cohibenda Ira, De Communibus Noti- des, Alcestis, Andromache, Bacchae, Cyclops, tiis Adversus Stoicos, De Cupiditate Divitiarum, Electra, Hecuba, Helen, Heracleidae, Heracles, De Curiositate, De Defectu Oraculorum, De E Hippolytus, Ion, Iphigenia at Aulis, Iphigenia Delphos, De Esu Carnium, De Exilio, De Fa- in Tauris, Medea, Orestes, Phoenissae, Rhesus, ciae Quae in Orbe Lunae Apparet, De Fato, De Suppliants, and Trojan Women; Sophocles, Ajax, Fortuna, De Fortuna Romanorum, De Fraterno Antigone, Electra, Ichneutae, Oedipus at Colonus, Amore, De Garrulitate, , De Oedipus Tyrannus, Philoctetes, and Trachiniae. Gloria Atheniensium, De Herodoti Malignitate, De Invidia et Odio, De Iside et Osiride, De Liberis Other: Bacchylides, Dithyrambs and Epinicians; Educandis, De Primo Frigido, De Pythiae Ora- Bion of Phlossa, Epitaphius, Epithalamium, and culis, De Recta Ratione Audiendi, De Se Ip- Fragmenta; Callimachus, Epigrams and Hymns; sum Citra Invidiam Laudando, De Sera Numi- Lucian, Podraga; Lycophron, Alexandra; Pindar, nis Vindicta, De Sollertia Animalium, De Sto- Isthmeans, Nemeans, Olympians, and Pythians; icorum Repugnantis, De Superstitione, De Tran- Theocritus, Epigrams. quillitate Animi, Demetrius, Epitome Argumenti Stoicos, Epitome Libri de Animae Procreatione, D Parameters for random forest models Fabius Maximus, , Instituta Laconica, La- caenarum Apophthegmata, Lucullus, Lycurgus, For all experiments, the parameters for the Marcellus, Marcus Cato, Maxime Cum Prin- scikit-learn random forest classifier are set cibus Philosopho Esse Diserendum, Mulierum Vir- to ‘bootstrap’: True, ‘class weight’: None, tutes, Nicias, Non Posse Suaviter Vivi Secun- ‘criterion’: ‘gini’, ‘max depth’: None, dum Epicurum, Numa, , Parallela Minora, ‘max features’: ‘auto’, ‘max leaf nodes’: Pelopidas, Pericles, Philopoemen, , Pla- None, ‘min impurity decrease’: tonicae Quaestiones, Pompey, Praecepta Geren- 0.0, ‘min impurity split’: None, dae Reipublicae, Publicola, Pyrrhus, Quaestiones ‘min samples leaf’: 1, ‘min samples split’: Convivales, Quaestiones Graecae, Quaestiones 2, ‘min weight fraction leaf’: 0.0, ‘n estimators’: Naturales, Quaestiones Romanae, Quomodo Ado- 10, ‘n jobs’: 1, ‘oob score’: False, ‘ran- lescens Poetas Audire Debeat, Quomodo Adula- dom state’: 0, ‘verbose’: 0, ‘warm start’: False. tor ab Amico Internoscatur, Quomodo Quis Suos in Virtute Sentiat Profectus, Regum et Impera- torum Apophthegmata, , Septem Sapien- Convivium, Sertorius, Solon, Sulla, Themis- tocles, Theseus, , Timoleon, Titus Flamininus, and Vitae Decem Oratorum; , Histories; Pseudo-Plutarch, De Musica and Placita Philosophorum; Strabo, Geography; , Peloponnesian War; , An- abasis.

60