STYLOMETRIC ANALYSES OF THE : A SHORT HISTORY

Photograph by Mark Philbrick.

FROM THE EDITOR: What value do computerized studies of author styles contribute to the polemics and irenics that seem to per- petually swirl around the Book of Mormon? In this article, authors Roper, Fields, and Schaalje take a few short steps back to take a long look at what such studies can and cannot contribute, including the latest twist, nearest shrunken centroid (NSC) classification. The authors present eight serious flaws with the NSC study and then offer the results of their recent study using extended nearest shrunken centroid (ENSCl classification, which overcomes those flaws. Long-time readers of FARMS publications and those of the Neal A. Maxwell Institute will enjoy this short history.

28 VOLUME 21 • NUMBER 1 • 2012 MATTHEW ROPER, PAUL J. FIELDS, AND 6. BRUCE SCHAALJE

laims about the authorship of the Book of Mor- in a text of unknown or questioned authorship and mon have a history as long as the book has been then comparing and contrasting those patterns to the Caround. To discredit 's descrip- patterns in texts of known authorship, the similari- tion of the book's origin, skeptics started proposing ties and dissimilarities between the textual patterns theories about who had written it even before it was can provide supporting evidence for or contradicting pubhshed in 1830.^ In 1834 Eber Howe proposed the evidence against an assertion of authorship. Spalding-Rigdon theory of Book of Mormon author- ship, which asserts that plagiarized an unpublished fictional work by Solomon Spalding Anonymous writing, plagiarism, and the to produce the Book of Mormon. He made this asser- consequent debates about the authorship tion even though the Book of Mormon was printed before Rigdon joined the church. Similar allegations of texts have a long history. . . . Ancient and variations on that theme continue today, despite catalogs of Aristotelian writings disagree .. solid historical evidence that the theory is a baseless fabrication.' Another way to look for evidence that as to which works Aristotle actually wrote. supports or does not support specific claims of au- thorship is to examine the writing styles in a text, Anonymous writing, plagiarism, and the con- specifically by identifying word-use patterns. In this sequent debates about the authorship of texts have article, we look at the strengths and weaknesses of a long history, perhaps extending back to the ad- various word studies that have attempted to deter- vent of writing itself. For example, three ancient mine who wrote the Book of Mormon. We conclude catalogs of Aristotelian writings disagree with each with the results of our own study of Book of Mor- other as to which works Aristotle actually wrote.' mon authorship. The authorship of Shakespeare's plays has been a topic of extensive debate and research, as has the Stylo m et ry authorship of the biblical epistles historically attrib- When reading a written text, a reader may often uted to the apostle Paul.^ In the sixteenth century in identify words and phrases that seem to ring with England and Wales, a series of anonymous religious a familiar voice, such that he or she may say, "This writings known as the Martin Marprelate tracts gen- sounds like it was written by Mark Twain (or Ernest erated a great deal of controversy, including specu- Hemingway or William Shakespeare)." But this is a lation about their authorship. Common Sense, pub- very subjective judgment. On the other hand, sty- lished anonymously hy Thomas Paine in January lometry, also known as computational stylistics, is 1776, was the most influential tract of the American a method of authorship attribution that uses far less Revolution and became an instant best seller, both subjective criteria—namely, statistical techniques— in the colonies and in Europe. To promote ratifica- to infer the authorship of texts based on writing pat- tion of the United States Constitution, eighty-five terns. It tries to describe an author's conscious and short essays signed with the pseudonym "Publius" unconscious creative actions with quantifiable mea- were published during 1787-88 in various New York sures such as the frequency with which an author City newspapers. They were later reprinted collec- uses certain words or groupings of words. tively as The Federalist. Although it was revealed in Stylometric analysis is based on the fundamental 1807 that the essays had been written by Alexander premise that authors write with distinctive, repeated Hamilton, James Madison, and John Jay, the specific patterns of word use. According to English professor authorship of twelve essays remained in dispute for John Burrows, written texts have a particular style over 150 years until statistical analyses would show and inherently display the intellectual propensities of strong support for Madison as their author.^ their authors.^ By identifying the word-use patterns

JOURNAL OF THE BOOK OF MORMON ANO OTHER RESTORATION SCRIPTURE 29 A Brief History of Stylomett7 Internet, development of ultrahigh dimensional sta- The use of statistical tools to test questions of au- tistical tools, and advances in Bayesian statistics. thorship in such situations goes back at least to 1851, when mathematician Augustus de Morgan proposed Limitations of Stylometry using average word length to numerically character- Stylometry is a useful tool in authorship attribu- ize authorship style.^° In 1887 Thomas Mendenhall, tion, but several limitations are important to keep in a physicist, proposed that an author has a "charac- mind when interpreting the results of a stylometric teristic curve of composition" determined by how analysis. Although stylometry is sometimes referred frequently an author uses words of different lengths. to as wordprint analysis (implying that it is a linguis- He applied this approach to compare the works of tic equivalent to fingerprint analysis), it does not Shakespeare and Francis Bacon, for example. In have the same identifying capability. The descrip- 1888 William Benjamin Smith, a mathematician writ- tion of stylometry as verbal DNA is an even less ap- ing under the pseudonym Conrad Mascol, published plicable overstatement. With stylometrics there is two papers describing a "curve of style" based on no way to perform population studies to determine average sentence lengths to distinguish authorial the general prevalence of word-use patterns. Conse- styles, which technique he applied to the Pauline quently, all probability assessments in stylometrics are relative only to the specific authors and the texts included in the study. A few advances in stylonnetry were made Although a person's fingerprint and DNA are in the first half of the twentieth century, unchangeably unique to that person, a writer is at liberty to adapt his or her style to a particular topic, but the most significant step was the Land- audience, and genre; to use artistic license to try new mark publication in 1964 of statisticians styles or even imitate others' styles; and to modify his or her own style over time as writing skills in- Frederick Mosteller and David Wallace. crease or falter. Shakespeare, for example, was fa- mously diverse in his writing style—an ability that is Epistles.^^ Then in 1893, Lucius Sherman, a profes- one of the hallmarks of a great author and also one sor of English, found that average sentence length of the things that makes stylometry a challenging could be used as an indicator of changes in writing methodology to apply successfully. styles over time.'' Further, writing style is not singularly specific A few advances in stylometry were made in the to a person. Stylometry can assess the similarity of first half of the twentieth century, but the most sig- writing styles among authors, but it cannot prove nificant step was the landmark publication in 1964 of personal identification of an author. Not only is statisticians Frederick Mosteller and David Wallace. there variation in an author's word-use patterns, but In their study they innovatively applied Bayesian authors can write sufficiently unlike themselves and statistical principles to investigate the authorship of sufficiently like each other at times that there are not the twelve disputed essays in The Federalist.^'' From clear boundaries between them, leaving fuzzy areas the late 1980s to the early 2000s, John Burrows made where their styles can overlap. So even though an seminal contributions to stylometric methodology. author's style may be distinctive, it is not distinct He introduced the "delta score" to measure word enough to be considered unique to that author to the frequency difterences among texts that varied by exclusion of all other authors in the world. author, in genre, or even across time periods.'' His Stylometric characteristics can provide a gen- method is now considered a benchmark for author- eral comparative description of an author's style, but ship attribution studies. Burrows also started a trend the writing style exhibited in a text is an indirect and of using principal components analysis in stylom- uncertain measure of an author's identity. Autho- etry.'' rial style is indistinct enough that one can say only, Today, the field of stylometrics is growing rap- "Based on these style characteristics, this text could idly due to the confluence of exponentially increas- have been written by author X, and it was more likely ing computing power, ubiquitous availability of the written by author X than by author Y." Thus, stylom-

30 VOLUME 21 • NUMBER 1 • 2012 etry can assess the probability of similar writing styles Wayne Larsen, Alvin Rencher, and Tim Layton— among texts, but that is not the same as the prob- examined the frequencies of noncontextual words in ability oí authorship of those texts. Stylometry is only a precedent-setting analysis of the Book of Mormon one source of evidence to support a claim of possible in 1980. Noncontextual words are function words authorship. Other evidence—such as historical and that have a grammatical role forming the structure of 1R a message, but they do not provide information about biographical evidence—becomes essential. the message. These are words such as a, an, but, how- In the context of what stylometry is and what ever, the, to, with, without, and so on. Mosteller and it is not, let us now consider the applications of the Wallace had shown that the way an author uses non- stylometric analyses that have been made regarding contextual words could be a means of characterizing the question of authorship of the Book of Mormon. the author's literary style independent of the author's Stylometric Analyses of the Book of Mormon message. For example, they found that Hamilton Since 1980, four major stylometric analyses of frequently used enough while Madison never used the Book of Mormon have been published—two by enough in his essays. Conversely, Madison frequently researchers at Brigham Young University,^' another used whilst, and Hamilton never used that term. Mo- by a doctoral student at Bristol Polytechnic,^" and steller and Wallace referred to such disparately used yet another by researchers at Stanford University.^^ words as "markers" that could be used to distinguish Each of these studies applied stylometry in different between the writings of Hamilton and Madison—a ways, seeking to address differing research ques- process of authorial discrimination. tions, but all aimed at testing claims of Book of Mor- In the Larsen study the researchers carefully mon authorship. constructed 2,000-word text blocks for each of the major purported authors in the Book of Mormon. Thie Larsen Study Then they tested whether the text blocks displayed Inspired by the Mosteller and Wallace study, evidence of a consistent style across the blocks, in- three statisticians at Brigham Young University- dicative of one author for all the texts, or whether

Organization of the church on 6 April 1830. Joseph Smith holds a copy of the Book of Mormon. © Robert T. Barrett.

JOURNAL OF THE BOOK OF MORMON AND OTHER RESTORATION SCRIPTURE 31 there was evidence of differing styles, congruent We can see that the text blocks attributed to with the claim that the Book of Mormon texts came Nephi, Alma, Mormon, and Moroni in the Book from different writers. of Mormon are consistently similar within authors Applying linear discriminant analysis^^ based on (tight grouping of texts by author) but consistently the frequencies of noncontextual words occurring in different among purported Book of Mormon au- each text block, the researchers used this technique thors (distinct cluster for each author, with some to compare the authors specified internally in the overlap). Joseph Smith's texts are clearly separated Book of Mormon to a set of nineteenth-century au- from the Book of Mormon texts. thors external to the Book of Mormon. The statisti- There is, of course, no statistical way to prove cal evidence of differences among the writings of the that the actual authors for the specific text blocks purported authors was overwhelming: were Nephi, Alma, Mormon, and Moroni. But who- ever the authors were, each one consistently wrote 1. "Distinct authorship styles can be readily distin- within his or her same style, and the styles differed guished within the Book of Mormon." from each other. If one person wrote the whole 2. "The nineteenth-century authors do not resem- Book of Mormon, he or she possessed an unusual ble Book of Mormon authors in style." and uncanny ability to write in different styles and to switch back and forth consistently between those A summary plot of their findings in figure l styles. shows how the texts form clusters for each of the Although somewhat overstated, it is hard to dis- four major authors identified internally in the Book agree with the Larsen study's main conclusion that of Mormon with a separate cluster for Joseph Smith "our study has shown conclusively that there were as an external author; his personal writings were many authors who wrote the Book of Mormon."^'* used in the comparison. The Hilton Study Skeptical of, but intrigued by, the results of the • Larsen study, John Hilton—a physicist at the Law- 9.0 rence Livermore National Laboratory in California

Joseph Smith and later a researcher at Brigham Young University- • • 7.0 decided in 1982 to test the reproducibility of the Larsen study results since a fundamental tenet of scientific research is that results of a study must be •'S 5.0 C reproducible by other researchers. In so doing, Hil- ton took a different approach than Larsen.^' Rather m 3.0 than using noncontextual word frequencies as sty-

I Alma • listic features, Hilton used sixty-five noncontextual ü 1.0 word-pattern ratios suggested by Andrew Morton, • • •Q a mathematician and religious studies scholar. Word- 1 •1.0 pattern ratios measure the rate of word use in four • • categories: • Nephi

• Moroni 1. Specific words in key positions of sentences, e.g., "the as the first word of a sentence," -6.0 2. Specific words adjacent to certain parts of •6.0 -5.0 -4.0 -3.0 -2.0 -1.0 0.0 1.0 2.0 3.0 4.0 5.0 speech, e.g., "and followed by an adjective," Second Discriminant Function 3. Collocations of words, e.g., "and followed by the," and Figure 1. Text clusters of major Book of Mormon authors and Joseph Smith. Linear discriminant analysis indicates that the writing styles of the major 4. Proportionate pairs of words, e.g., "no and not," Book of Mormon authors are distinguishable from each other and highly "all and any." distinctive from Joseph Smith's writing style.

32 VOLUME 21 • NUMBER 1 • 2012 tMutnber of Pairw¡se Reject¡ons Text vs. Text 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Nephi vs. Nephi 1 1 1 s c i S Alma vs. Alma 1 1 1 Smith vs. Smith

pa r 1 2 a-A i Cowdery vs. Cowdery 1 Spalding vs. Spalding 1 Nephi vs. Alma 1 2 2 1 1 1 Smith vs. Nephi 1 2 1 1 1 Smith vs. Alma 2 1 1 2 Cowdery vs. Nephi A 1 r-A i 2 1 1 ipa r "c o Cowdery vs. Alma 4 1 1 — fi Spalding vs. Nephi 1 1 1 1 2 Spalding vs. Alma 3 2 1 Evidence of Similar Styles -> Evidence of Different Styles

Figure 2. Rejections of pairwise comparisons of texts from the Hilton study Pairwise rejections fewer than seven of the possible sixty-five word pattern ratios in each text vs. text comparison indicate evidence of similar authorial style. The intra- author comparisons tend to show similar styles while the inter-author comparisons tend to show dissimilar styles.

Hilton's idea was that these ratios might be mini- he excluded quotations from the Bible and the dis- mally affected by unique phrases in the texts or by tinctive phrase and it came to pass. topic and genre differences among the texts and thus Hilton then made various comparisons among might be better detectors of an author's unconscious Book of Mormon texts attributed to Nephi and Alma word-use preferences. In agreement with Mor- and non-Book of Mormon texts known to have been ton, Hilton reasoned that these word-pattern ratios authored by Joseph Smith, Oliver Cowdery, and would be useful since they provide a nonambiguous Solomon Spalding. Specifically, he compared each count, occur frequently, have common alternative author to the texts attributed to himself (intra-author expressions, and tend to be used habitually. comparisons) and then each author to every other In addition, he developed a stylometric measure author (inter-author comparisons). Figure 2 summa- used to differentiate between any two texts based on rizes his results in tabular form. The first line of the the number of word-pattern ratios judged to be sig- table (Nephi vs. Nephi) indicates, for example, that nificantly different than expected (called rejections) there were three 5,000-word Nephi texts, and pair- between texts purportedly alleged to be written in wise comparisons of these texts yielded two, four, the same authorial style. He calibrated and validated and five rejections for tests of the sixty-five word- his method by applying it to texts of undisputed au- pattern ratios. Further, in comparing six sets of texts thorship from the i8oos and 1900s. He determined by Cowdery and Alma, four showed seven pairwise that seven or more rejections provided evidence of rejections, one showed eight, and the other nine, differences of writing style indicative of different thus showing their dissimilarity. In figure 2, the authorship. intra-author comparisons show evidence of similar Using the oldest extant versions of the Book of style, while the inter-author comparisons show evi- Mormon—primarily the printer's manuscript^''—he dence of dissimilar styles. applied his procedure to 5,000-word blocks of text. The most important result was that all of the This provided high reliability since in larger text Nephi, Alma, Smith, Cowdery, and Spalding texts blocks an author's writing habits and stylistic pro- are each consistent within themselves but distinctly pensities should assert themselves more strongly different from one another. Thus, the evidence from than in smaller texts. In compiling the text blocks. the Hilton study argues strongly against the idea that

JOURNAL OE THE BOOK OF MORMON AND OTHER RESTORATION SCRIPTURE 33 Joseph Smith, Oliver Cowdery, or Solomon Spald- course versus historical narrative). Holmes extracted ing could be the author of the Nephi or Alma texts. fourteen approximately 10,000-word blocks assigned Hilton concluded: to six Book of Mormon authors, divided sections 1 through 51 of the Doctrine and Covenants into three We show that it is statistically indefensible to pro- 10,000-word blocks, combined the writings of Jo- pose Joseph Smith or Oliver Cowdery or Solomon seph Smith into three 6,000-word blocks, included Spaulding as the author of the 30,000 words from the Book of Abraham as one text, and extracted the Book of Mormon manuscript texts attributed to Nephi and Alma. Additionally these two Book of three 12,000-word blocks from Isaiah. Mormon writers have wordprints unique to them- As illustrated in figure 3, Holmes found that the selves and measure statistically independent from Joseph Smith texts clustered together, the Isaiah each other in the same fashion that other uncon- texts clustered together, and all but three of the other tested authors do. Therefore, the Book of Mormon measures [as being] multiauthored, with authorship texts clustered together. consistent to its own internal claims.^ Holmes concluded from this that he had de- finitively shown that the writings of Mormon, Lehi, Hilton's findings were congruent with the Lar- Nephi, Jacob, and Moroni were not stylometrically sen findings. In 2006 these results were reproduced different. He stated, "There appears to be no real dif- again by researchers at Utah State University using ference between Alma's richness of vocabulary and generalized discriminant analysis—an extension of Mormon's richness of vocabulary, ... a conclusion the linear discriminant analysis used in the Larsen in direct contradiction to the findings of Larsen." He study.'' continued, "This study has therefore not found any evidence of multiple authorship within the Book of The Holmes Study Mormon itself," to which he added, "We may con- Not all Book of Mormon stylometric studies sider the Book of Abraham, the purported authors of have reached the same conclusion as Larsen and the Book of Mormon and Joseph Smith's revelations Hilton.^" For his doctoral dissertation at Bristol to be of similar style, therefore, with all the implica- Polytechnic in 1985, David Holmes—now at the tions that this may have for Mormon doctrine."^' College of New Jersey but previously a professor at The first part of Holmes's statement is prima the University of the West of England—carried out facie false since the Larsen study utilized noncon- a stylometric analysis of the Book of Mormon and textual word frequencies and did not include any related texts based on five measures of vocabulary findings about vocabulary richness. The rest of the richness.'^ As styUstic features. Holmes computed statement is an example of the classic fallacy argu- a standardized measure of words used once in the mentum ad ignorantiam: "1 did not find a difference so text (R), a standardized measure of words used twice there must not be a difference." When a researcher (V2/V), a Poisson-based measure of lexical repetitive- does not find evidence of an effect, he or she can ness (K), and two estimated parameters of the Sichel only say, "I did not find evidence of an effect." The distribution (a and 6)—a theoretical distribution to researcher cannot say, "Therefore, the effect does model word frequencies in writing. The first three not exist." The effect could still exist; the researcher measures were calculated for the total vocabulary simply did not find it. In addition. Holmes overgen- in the texts, while the last two were calculated for eralized the usefulness of his methodology by fail- nouns only. ing to recognize that the successful application of a His motivation was his impression at the time technique in one instance does not indicate that it is that vocabulary richness was a "particularly effec- useful in all instances.''* Even if a method found a tive measure for discrimination between writers."^' large difference in one instance does not mean the Holmes used the 1980 editions of the Book of Mor- method will find smaller differences in other cases. mon, the Doctrine and Covenants, and the Book of A method's ability to find small differences that in Abraham from the Pearl of Great Price; the book of fact exist is referred to by statisticians as the meth- Isaiah from the King James Bible; and diaries and od's power. histories written or prepared by Joseph Smith be- Subsequent research by Schaalje, Hilton, and tween 1838 and 1843. Ignoring genre (doctrinal dis- Archer has shown that Holmes's stylistic measures

VOLUME 21 • NUMBER 1 '2012 0.45

• D&C 0.35

• Mormon 0.25 • D&C • Moroni • Smith Mormon d) 0.15 C _„ « • • Mormon O Q. Mormon « « Moroni Alma O 0.05 Smith • Smith U '-^'^' , • Abraham 75 Alma &• -0.05 • Nephi « Jacob -0.15 o Nephi o o • Isaiah CO -0.25 • Isaiah * '^^'^'^

-0.35

• Nephi -0.45

• Mormon

-0.55 -1.00 -0.90 -0.80 -0.70 -0.60 -0.50 ^0.40 -0.30 -0.20 •OIO 0.00 0.10 0.20 0.30 0.40 0.50 First Principal Component

Figure 3. Principal components analysis plot based on Holmes's vocabulary richness nneasures. Although texts from Joseph Smith and Isaiah are easily distinguishable from Book of Mormon, Doctrine and Covenants, and Pearl of Great Price texts, Holmes's method could not distinguish among the purported authors within the Book of Mormon nor in comparison to the other scriptural texts. have low power and are consequently weak discrimi- The skepticism of Schaalje, Hilton, and Archer nators of authorship. ^^ For example, when testing toward the effectiveness of Holmes's vocabulary texts of undisputed authorship by Samuel Clemens richness technique has been borne out in a more re- (Mark Twain) and Samuel Johnson (a British author cent study by David L. Hoover: and lexicographer), among others, correct classifica- tion rates were 96% using noncontextual word fre- Despite the attractiveness of measures of vocabulary quencies, 92% for noncontextual word-pattern ra- richness, and despite the fact that they are some- times effective in clustering texts by a single author tios, but only 23% for vocabulary richness measures. and discriminating those texts from other texts by Similar results were obtained consistently in other other authors, such measures cannot provide a con- tests on sets of texts from novels (translated from sistent, reliable, or satisfactory means of identifying German into English), the Book of Mormon texts an author or describing a style. There is so much in- tratextual and intertextual variation among texts and (translated from an unknown ancient language into authors that measures of vocabulary richness should English), and the King James New Testament (trans- be used with great caution, if at all, and should be lated from Greek into English). Later, in a reanalysis treated only as preliminary indications of author- of The Federalist essays. Holmes himself found vo- ship, as rough suggestions about the style of a text or cabulary richness measures to be comparatively less author, as characterizations of texts at the extremes of the range from richness to concentration. Perhaps effective discriminators of authorship than noncon- their only significant usefulness is as an indicator of textual word frequencies.^

JOURNAL OF THE BOOK OF MORMON AND OTHER RESTORATION SCRIPTURE 35 what texts or sections of texts may repay further method updates initial probability estimates (called analysis by more robust methods. Unfortunately, the long-cherished goal of a measure of vocabulary rich- "prior probabilities") to calculate final probability ness that characterizes authors and their styles ap- estimates (called "posterior probabilities") based on pears to be unattainable. The basic assumption that newly obtained sample information. For example, underlies it is false.^'' without the sample information, the prior probabil- The results of the Holmes study certainly do ity estimates would be that all candidate authors are not nullify the results of the Larsen and Hilton stud- equally likely to be the author of a text of unknown ies nor portend any grave implications for Mormon authorship. But after the writing style in the text doctrine, as Holmes suggested. The Holmes study (sample information) is compared with the writing shows only that the Book of Mormon texts, although style of each candidate author, the posterior prob- consistently distinct in terms of noncontextual word ability estimates might show that one author is more usage and word-pattern ratios, display similar vo- likely the author of the text than the other candi- cabulary richness. This might reflect simply that the dates because of closer similarity of writing style. It Book of Mormon texts are the work of a single trans- is vitally important to note that NSC is a closed-set lator, as Joseph Smith claimed, and thus were limited method, which means it assumes the set of candi- by his vocabulary. date authors definitely includes the true author to the exclusion of any other possible candidates. The Jockers Study In the Jockers study, the researchers' hypothesis The weakest of the four major Book of Mor- was that the Book of Mormon is the collaborative mon stylometric studies is presented in a recent pa- work of multiple nineteenth-century authors. They per by Matthew Jockers, Daniela Witten, and Craig specifically sought to find support for the Spalding- Criddle^ —respectively an English lecturer, a statis- Rigdon theory. Therefore their set of candidate au- tics graduate student, and a civil engineering profes- thors included text blocks by Solomon Spalding, sor at Stanford University. Their study is innovative Sidney Rigdon, Oliver Cowdery, and Parley P. Pratt. in that the statistical method they used was "nearest Biblical texts by Isaiah and Malachi (combined as shrunken centroid classification" (NSC), a multivari- one author) were included as a positive control, and ate classification method based on Bayesian statistics contemporary nineteenth-century texts by Henry developed for the classification of tumors in genom- Longfellow and Joel Barlow were included as nega- ics research.'^ tive controls. The texts varied greatly in size, ranging In statistics, shrinkage is a way to reduce the from 114 to 17,797 words in length. uncertainty about an estimated quantity by combin- Even though chapter designations were not ing information from multiple sources in making the added to the Book of Mormon until 1879 (when all estimate. The more information that is included in of their candidate authors were dead), Jockers chose making an estimate, the less uncertainty there will to use the current chapter structure to define the be about that estimate. A centroid is the center of test text blocks for the Book of Mormon, reasoning a multidimensional cluster of data points. Think of rather dubiously that the chapters might have been it as the center of gravity of a disperse collection of contributed individually by their panel of suspected related items with varying sizes. When applied to authors and thus might provide evidence of "cor- stylometry, the NSC method uses the stylistic char- rect" authorship. The Book of Mormon chapters acteristics (such as word frequencies) found in the also varied widely in length from 95 to 3,752 words. texts of a set of candidate authors to create a rule for As stylistic features, Jockers used relative fre- determining the authorship of unknown texts. That quencies of the most common 110 words in the Book rule is then used to assign a text of questioned au- of Mormon that were used at least once by each thorship to the author whose cluster of texts has the purported author. From this list they removed four nearest centroid. The closer a test text of an unknown words that they felt were contextual in relating to author is to the centroid of a known author's texts, biblical subject matter {God, ye, thy, and behold), but the greater the likelihood that the style of the test without justification they retained fifteen other con- text matches the writing style of the known author. textual nouns: children, day, earth, father, hand, king, Using Bayes' theorem from statistics, the NSC land, man, men, name, people, power, son, time, and

36 VOLUME 21 • NUMBER 1 • 2012 words. For some unknown reason they apparently closed-set technique, a researcher must be assured wanted their definition of authorial style to include by external evidence such as well-established, non- some lexical words—other than biblical-sounding controversial historical information that all possible words—rather than just function words. candidate authors have been identified and included. The results of Jockers et al.'s application of NSC For The Federalist studies, there was no question that classification to assigning Book of Mormon chapters to their set of candidate authors are tabulated in figure 4. There are eight serious flaws with the Jockers Their cancJidate set cannot be considered study methodology that render the results moot. First to be connprehensive since the styles of and most obviously, Joseph Smith was excluded as a candidate author, even though as the book's transla- the vast majority of Book of Mormon tor he is the most likely author. His candidacy was chapters differ markedly from the styles considered in each of the previous studies. The Jock- ers researchers incorrectly claim that Joseph Smith of any of Jockers et al.'s candidates. could not be included because he frequently used scribes when preparing written documents and left inadequate samples of his personal writings. Dean the true author was included as a candidate. The Jessee has compiled a comprehensive set of Joseph question was only whether the writing style of a spe- Smith's writings, many of which are holographic cific paper favored Hamilton or Madison; there were (written solely in his own hand).'*" Because NSC is no other possible candidates. However, for the Book designed to pick one of the members of a closed set of Mormon the situation is not so simple—there is of candidates, excluding Joseph Smith from the anal- no substantiating historical or biographical informa- ysis seems like an attempt to stack the deck in favor tion to justify a constrained set of candidates. In fact, of the Spalding-Rigdon theory authors. the principal components plot of the Jockers study'*^ Second, and even more important, the set of shown in figure 5 provides confirming evidence that candidate authors for the Book of Mormon can- their candidate set cannot be considered to be com- not reasonably be considered closed. To employ a prehensive since the styles of the vast majority of Book of Mormon chapters differ markedly from the Nearest Shrunken Centroids (NSC) styles of any of Jockers et al.'s candidates. Because Classification Results of the dispersion of the data points (with very little overlap in the Book of Mormon clusters and the can- Percentage of Book didate authors' clusters), it is obvious that the possi- of iVlormon Chapters bility of other authors than were allowed in the Jock- Candidate Author Assigned to an Author ers study must be included in an analysis of Book of Mormon writing styles. Sidney Rigdon 39% Third, Jockers et al. assert that since twenty of Isaiah/iVIalachi the twenty-one chapters from Isaiah/Malachi were Solomon Spalding 22% correctly attributed, "this is evidence for the effec- Oliver Cowdery 8% tiveness of NSC classification."**^ Yet they ignore the Parley Pratt 4% forty-two Book of Mormon chapters that are known not to have been authored by Isaiah and Malachi but Henry Longfellow 1% that NSC mistakenly attributed to them. That means Joel Bariow 0% that NSC made twice as many incorrect attributions Joseph Smith Not Included to Isaiah and Malachi as correct attributions. The statement Jockers et al. should have made is "This is Figure 4. Percentage of Book of Mormon chapters assigned to each author hy Jockers et al. based on nearest shrunken evidence for the ineffectiveness of NSC classification." centroid INSCl classification probability estimates, including With two-thirds errors, this should have alerted Isaiah/Malachi as positive controls and Longfellow and Jockers et al. that their naïve application of NSC was Barlow as negative controls, but not including Joseph Smith. producing unreliable results. This should have also

JOURNAL OF THE BOOK OF MORMON AND OTHER RESTORATION SCRIPTURE 37 38 VOLUM E 21•NUMBER'201 2 Just asstoppedclockirightwiceday , ing dartsattheLisofcandidatauthors . NSC shouldbevieweasperformingno better thanattributingchaptersbythrow - tributions toRigdonmusalsbegrosslyoverstated. made themverysuspiciousthatthninety-threat- in thirty-fourchaptersattributedtoeachauthor.^' random assignmentofchapterswouldhaveresulte one chapters.AsJockertalpoinout,amer rect rateofattributionwouldbonlyabouthirty- for RigdonasIsaiahandMalachi,thethecor- If thesamproportionomisattributionsoccurred throwing dartsatthelisofcandidat e authors. thirty-one chaptersthatmighthavebeencorrectly pected byrandomassignment . Justasstopped attributed toRigdonwereonlywhawoulbex- Jockers etal.shouldhavrealizethereforthath tify thecluster oftextsatesttexirelativel y closest performing nobetterthaattributin g chaptersby clock isrighttwiceaday,NS C shouldbevieweas ters ¡blackdots)iclearlyseparatefromthclusterfocandidatauthors'text(reddotsi. Figure 5.PrincipalcomponentsanalysiplotforJockeral.'datashowingthathclusteBookofMormonchap- Second Príncipai Co Fourth, eventhoughtheNSC methodcaniden- 0-8-6 Candidate Authors • • * ••; •i • ^1 -4 •-•-f First PrincipalComponen —*n —«- • • ••^ V • • great distanceapart.Thiswouldallowforthpos- to, thatdoesnomeanicloseaabsolutesense. The testtexandthclosesclustercoulstillba Vegas, TucsonSanDiegoandsoon.Treliablyuse only relativelyclosetoLosAngeles—relativChi- answer thatSalLakeCityisclosest.Bu city isclosestoLoAngeles?"Wecouldcorrectl cities NewYork,ChicagoandSaltLakeCitywhich an analogy,letusaskthequestion"Considering sibility thatanexcludedauthorisactuallcloser.A the setofcandidatestested.Becaus e notallpossibl there arnootherpossiblcandidatauthors.With- Many citieswerenotincludedacandidates—La closest cityofallcitiesintheUnitedStatesofAmerica. City istheclosestofcandidatset,no cago andNewYork.Further,eventhoughSaltLake City issevenhundredmilefromLoAngeles,ot the closestofallpossiblcandidate s wouldbeun- candidates wereincludedinth e Jockersstudy,state- drawn iswhichcandidateth e closestfromamong out suchassurance,theonlyconclusionthacab searcher mustknowwithreasonablecertaintytha a closed-setmethodsuchsNSCinstylometry,re- substantiated extrapolation s andwouloverste p the ments thatmakeclaimabou t whichcandidateis • \ *• •«• >•( V • l4*V^'6r.« 8•1) - Book ofMormonChapters • * bounds of the evidence. In addition, just because San Naïve or inexperienced analysts frequently make the Diego is close to Los Angeles, that does not mean it is mistake of overlooking the effects of multiplicity— the same as Los Angeles. To claim they are the same that is, claiming that a random event has meaning city requires more evidence than just a measure of when in reality it is just the result of normal varia- relative proximity. Likewise, in stylometry, relative tion in a process.^ proximity only connotes similarity of style, not nec- Eighth, Jockers et al. represented Rigdon's writ- essarily the same authorship. ing style using fourteen articles published in news- Fifth, the NSC probabilities are presented by papers between 1833 and 1835, as well as nine reve- Jockers et al. as absolute probabilities. This is mis- lations authored by Rigdon beginning in 1863. The leading since, in fact, they are relative probabilities problem is that the styles of these two sets of writ- related only to the specific set of candidate authors ings show evidence of being distinctly different, as tested.^^ Suppose that for some Book of Mormon shown in figure 6, which is based on Jockers et al.'s chapter Rigdon's probability is calculated as 80%, data. Pratt's probability is calculated as 20%, and each To confirm this observation, we took all news- remaining candidate's probability is calculated as paper articles and pamphlets known to have been nearly 0%. The most that can be concluded from authored by Sidney Rigdon between 1831 and 1846 these numbers is that Rigdon's probability of a to create twenty-five composite texts ranging in size matching style is four times greater than Pratt's. One from 2,214 to 8,747 words. We also created fifteen could say that the odds are "four to one" (4:1) in favor composite texts ranging in size from 3,678 to 6,784 of Rigdon over Pratt, but one could not meaningfully words from all of the sections authored by Sidney state Rigdon's calculated likelihood without a com- Rigdon or jointly by Sidney and Phebe Rigdon in the parison to Pratt's. While in a relative sense the prob- Book of the Revelations of Jesus Christ to the Children ability calculated for Rigdon might be 80% within a of Zion through Sidney Rigdon, Prophet, Seer and Reve- limited set of authors, in an absolute sense it might lator.'*' The texts were combined in chronological be only 8%, for example, if «//possible authors were included. Sixth, the NSC procedure assumes that the variation of the word frequencies in the text blocks is the same for all text blocks. This requirement of equal variance—called homogeneity of variances- is grossly violated in the Jockers study due to the highly disparate sizes of the text blocks. It is com- pletely unreasonable to assume that the variances of word frequencies in text blocks of 100 words are the same as the variances of word frequencies in text blocks of 5,000 words or 15,000 words. Hence the au- thorship probabilities calculated by NSC make even less sense. Seventh, the authorship probabilities have still less meaning individually since so many texts (239 chapters) are classified simultaneously in a single sta- tistical procedure. When making a multitude of com- parisons within a single test procedure, some of the calculated probabilities will appear to indicate items that are significantly different from each other even though their difference occurred simply by chance. These differences can be spurious and signify noth- ing. This is a well-known hazard in statistical prac- tice and is referred to as the multiplicity problem.'*' Sidney Rigdon.

JOURNAL OF THE BOOK OF MORMON ANO OTHER RESTORATION SCRIPTURE 39 -tí-

Early Rigdon « o a. E o (J o o o° o o TB o o a. o o n 1 -6 -4 -2 • 2 4 6 í o o •o o u « Late Rigdon •

• ••

•

First Principal Component

Figure 6. Principal components analysis plot of early and late Rigdon texts. The early Rigdon texts were written from 1833 to 1835 and are shown as solid red dots. The late Rigdon texts were written after 1863 and are shown as open red dots. The distinctness of the two clusters suggests strongly that Rigdon s early writing style had evolved into another style later in his life.

order, and no section was split between two text completely unsuitable for inferring authorship of blocks. Figure 6 shows the distinct difference in the Book of Mormon; the full results for the control style between the two sets of texts. It is unknown texts were ignored; the calculated probabilities were whether Rigdon's style actually changed over the misinterpreted; the chapter-by-chapter probabilities seventeen intervening years, or whether his revela- of authorship are not even useful as relative prob- tions refiect the contributions of others such as his abilities; the effect of hugely different sample sizes wife. In any case, in a study of Book of Mormon was disregarded; the multiplicity effect of multiple authorship, Rigdon's style should be characterized simultaneous testing was ignored; and, finally, Rig- only by documents written in his early style—the don's two differing writing styles were confounded time period closest to the publication of the Book of into one composite style. Mormon. The Rigdon texts used in the Jockers study The only idea in the Jockers study that is of confound the two Rigdon styles. some value regarding the Book of Mormon is actu- The Jockers study concluded: ally not in their paper, but is based on data listed on their website that we used to produce figure 6. How- Our analysis supports the theory that the Book ever, it points to a very different conclusion from of Mormon was written by multiple nineteenth- century authors, and more specifically, we find that drawn by Jockers et al. strong support for the Spalding-Rigdon theory of authorship. In all the data, we find Rigdon as a unify- Most Recent Study Using ENSC ing force. His signal dominates the book, and where In response to the Jockers study, we recently other candidates are more probable, Rigdon is often conducted a new study correcting the methodologi- hiding in the shadows.'* cal fiaws in the Jockers study.^^ Most important, In actual fact, the Jockers study has shown noth- we developed a modification to the closed-set near- ing. The study design was biased to produce a desired est shrunken centroid (NSC) classification method result; the closed-set classification methodology is to enable it to be applied to open-set classification

Í0 VOLUME 21 • NUMBER 1 • 2012 problems.^" We refer to this method as extended assigned 28% of the chapters to Isaiah/Malachi. How- nearest shrunken centroid (ENSC) classification. In ever, the ENSC-assigned percentage of 15% (fig. 7b) is doing so, we modified the NSC formulas to allow for much closer to the correct percentage (12%). some other author—that is, to allow for the possibil- Considering Spalding without Joseph Smith as ity that an excluded author might have written the a candidate author, NSC assigned 22% of the chap- text whose authorship is in question. This open-set ters to Spalding, yet only 15% to him when Joseph modification allows for the existence of an uniden- Smith was included (fig. 7a). Obviously, when Jo- tified author with writing characteristics nominally seph Smith is included in the analysis, any supposed consistent with the test text and incorporates this support for the Spalding-Rigdon theory diminishes. possibility into the probability calculations. Without With Joseph Smith in the candidate author set, we including the possibility of someone else as the au- see that NSC assigned 12% of the chapters to Joseph thor, if the candidate set does not include the true Smith because of chapter reassignment away from author (using a closed-set approach for an open-set Spalding, Cowdery, and Pratt. This seems consistent situation), the probability of similar writing style can with the claim that Joseph Smith, as translator, dic- be grossly overstated and lead to entirely erroneous tated the text of the Book of Mormon, and in doing interpretations.'^ so perhaps had some influence on the structure of For purposes of comparability with the Jock- language in the document. ers study, we used the same list of no characteristic In contrast, when applying ENSC, the com- words as Jockers et al. as well as their chapter-by- bined total for Spalding-Rigdon drops to only 8%, chapter designation of text blocks from the Book of with ENSC assigning a mere 3% to Joseph Smith Mormon. We first reproduced the Jockers study re- (fig. 7b). The few chapters that ENSC indicated to be sults using the same set of candidate authors to con- closest to Rigdon, Spalding, Cowdery, and Smith are firm that our implementation of NSC was consistent randomly dispersed throughout the 239 chapters. with theirs. We then repeated the NSC analysis in- cluding Joseph Smith in the set of candidate authors. Finally, we applied the open-set ENSC technique al- lowing for the possibility of some other author. In addition, when we used the ENSC method, we took into account differences in sample sizes, adjusted for multiplicity, and recognized the distinction between Rigdon's time-separated writing styles. Figure 7a displays the results of applying NSC per the Jockers study and applying NSC with Joseph Smith included but without the possibility of someone else as the au- thor. Eigure 7b displays the results of applying ENSC allowing for the possibility of some other author. First, examining the NSC graph in figure 7a, we notice that the percentage of chapters NSC assigned to Rigdon is about the same with or without Joseph Smith in the candidate set (39% and 40%, respectively), while the ENSC graph in figure 7b shows far fewer chapters assigned to Rigdon (7%). Interestingly, the ENSC per- centage for Rigdon is the sum of roughly equal percent- ages for early and late Rigdon sample texts. Next we notice that the percentage for Isaiah/ Malachi (26%) as assigned by NSC (fig. 7a) is obvi- ously much too large (as discussed earlier), and the misattribution to Isaiah/Malachi actually increased when Joseph was a candidate author since then NSC Book of Mormon, 1st edition (18301. © 1995 IRI.

JOURNAL OF THE BOOK OF MORMON ANO OTHER RESTORATION SCRIPTURE NSC Classifications i« 100%

• I" 5 90% • NSC based on Jockers et al. with Joseph Smith and someone else excluded as possible authors = 1 80% • NSC with Joseph Smith included, but someone else 0 70% excluded as a possible author o -o • 'o 10 60%

I¿o 50% 40%

30% 1

20% Perc e 1 Id. 10% Ilk 0% J 1 Rigdon Isaiah/ Spaldng Cowdeiy Pratt Lon^ellow BailOM Smith Someone Malachi Else

ENSC Classifications 100% S s. 90% lENSC with Joseph Smith and someone else included as possible authors 80% li lENSC with Joseph Smith and someone else included as 70% possible authors, but without Isaiah/Malachi as controls o -a

o (0 60% j£ U °: 50% CQ S 'S "O 40%

30% 0) i u 20%

10%

0% Rigdon Isaiah/ Spaldng Cowdeiy Pratt Lon^dlow Barlow Smith Someone Malachi Else

Figures 7a and 7b. Nearest shrunken centroid INSC) and extended nearest shrunken centroid (ENSCl classification methods applied to Book of Mormon authorship. Although the closed-set NSC technique assigns a majority of chapters to Spalding and Rigdon within a constrained set of candidate authors, when allowing for the possibility that the candidate set is incomplete, the open-set ENSC technique assigns an even larger majority of the chapters to an unidentified author who was not included in the NSC candidate set. Percentages are based on the number of chapters that are deemed closest to a candidate authors style.

42 VOLUME 21 • NUMBER 1 »2012 indicating that they should be considered random amining all the evidence, our overall conclusion is misclassifications. that the Book of Mormon displays multiple writing Most interesting, though, the ENSC method (fig. styles throughout the text consistent with the book's 7b) assigned 73% of the chapters to "Someone Else." claim of multiple authors and that the evidence does Further, excluding the Isaiah/Malachi chapters and not show the writing styles of alleged nineteenth- looking only at the non-Isaiah/Malachi Book of Mor- century authors to be similar to those in the Book of mon chapters, ENSC assigned 93% of those chapters Mormon. Further, the claims thus far put forward for to "Someone Else" with a few chapters randomly alternative authorship of the Book of Mormon, other assigned to Oliver Cowdery and Joseph Smith, as than as described by Joseph Smith, are untenable. • would be expected if Joseph Smith had translated the text with Oliver Cowdery as his scribe. Clearly Jockers et al.'s claim of astronomical prob- Matthew Roper (MA, Brigham abilities in support of the Spalding-Rigdon theory is Young University] is a research scholar for the Neal A. Maxwell a great exaggeration. The ENSC results confirm our Institute for Religious Scholar- analysis that the Jockers study was fatally flawed in ship, Brigham Young University, concept and execution. Contrary to their contention, the evidence does not provide credible support for the claim that the writing styles exhibited in the Book of Mormon match any of their candidate authors— Spalding, Rigdon, Cowdery, or Pratt. In fact, the PaulJ. Fields (PhD, Pennsylvania State University] is a consultant evidence from a correctly conducted analysis clearly specializing in research methods supports the claim that someone other than their set of and statistical analysis. He has candidate authors wrote the book. Therefore, based on extensive experience in textual these findings, we conclude that stylometric evidence analysis and linguistic computing. does not support the Spalding-Rigdon theory of Book of Mormon authorship. G. Bruce Schaalje (PhD, North Conclusion Carolina State University) is a Stylometric analyses of the Book of Mormon professor of statistics at Brigham have generated much interest over the past thirty Young University. years. Some of these analyses have produced inter- esting information, but some of the studies have been characterized by hyperbole, faulty reasoning, and misapplication of statistical methods. When ex-

NOTES 3. Matthew Roper, "The Mythical speare's Works Were Written by Others 'Manuscript Found,'" FARMS Review (Bloomington: Indiana University 17/2 (2005): 7-140; and Roper, "Myth, Press, 1958); James G. McManaway, Louis C. Midgley, "Who Really Memory, and 'Manuscript Found,'" The Authorship of Shakespeare (Wash- Wrote the Book of Mormon? The FARMS Review 21/2 (2009): 179-223. ington, DC: Folger Shakespeare Critics and Their Theories," in Book 4. John F. Burrows, "Computers and Library, 1962); and Hugh Craig and of Mormon Authorship Revisited: the Study of Literature," in Comput- Arthur F. Kinney, Shakespeare, Com- ers and Written Texts, ed. Christo- puters, and the Mystery of Authorship The Evidence for Ancient Origins, pher S. Butler (Oxford: Blackwell, (Cambridge: Cambridge University ed. Noel B. Reynolds (Provo, UT: 1992), 167-204. Press, 2009). FARMS, 1997), 101-39. 5. Carnes Lord, "On the Early History 7. J. D. James, The Genuineness and Eber D. Howe, , of the Aristotelian Corpus" American Authorship of the Pastoral Epistles or, a Faithful Account of That Singular Journal of Philology 107/2 (1986): 137-61. (London: Longmans, Green, 1906); Imposition and Delusion, from Its 6. Reginald C. Churchill, Shakespeare Percy N. Harrison, The Problem of Rise to the Present Time (Painesville, and His Betters: A History and a the Pastoral Epistles (London: Oxford OH: Printed and Published by the Criticism of the Attempts Which Have University Press, 1921). Author, 1834). Been Made to Prove That Shake- 8. Joseph Black, "The Rhetoric of

JOURNAL OF THE BOOK OF MORMON AND OTHER RESTORATION SCRIPTURE Reaction: The Martin Marprelate DNA profile do not vary. Typographical Facsimile of the Tracts (1588-89), Anti-Martinism, 18. Harold Love, Attributing Authorship: Entire Text in Two Parts (Provo, UT: and the Uses of Print in Early An Introduction (Cambridge: Cam- FARMS, 2001). Modern England," Sixteenth Century bridge University Press, 2002). 28. Hilton, "On Verifying Wordprint Journal 2%/3, (1997): 707-25. 19. Wayne A. Larsen, Alvin C. Rencher, Studies," in Reynolds, Book of Mor- 9. See Frederick Mosteller and David and Tim Layton, "Who Wrote the mon Authorship Revisited, 241. L. Wallace, Inference and Disputed Book of Mormon? An Analysis of 29. Todd K. Moon, Peg Howland, and Ja- Authorship: "The Eederalist" (Reading, Worà^ï'mts" BYU Studies 20/3 (1980): cob H. Günther, "Document Author MA: Addison-Wesley, 1964). 225-51; reprinted by Wayne A. Classification Using Generalized Dis- 10. David I. Holmes, "The Evolution of Larsen and Alvin C. Rencher in Book criminant Analysis," in Proceedings of Stylometry in Humanities Scholar- of Mormon Authorship: New Light on the Fourth Workshop on Text Mining, ship," Literary and Linguistic Comput- Ancient Origins, ed. Noel B. Reynolds Sixth SLAM International Conference on ing 13/3 (1998): 112. (Provo, UT: BYU Religious Studies Data Mining (paper presented at the 11. T. C. Mendenhall, "The Characteris- Center, 1982), 157-88; John L. Hil- 2006 SIAM Conference on Data Min- tic Curves of Composition," Science ton, "On Verifying Wordprint Stud- ing, Bethesda, Maryland, 20-22 April 214 (11 March 1887): 237-46. ies: Book of Mormon Authorship," 2006). 12. C. Mascol, "Curves of Pauline and BYU Studies 30/3 (1990): 89-108; re- 30. As a side note, we mention another Pseudo-Pauline Style I," Unitarian printed in Reynolds, Book of Mormon Book of Mormon stylometry study Review 30 (November 1888): 452-60; Authorship Revisited, 225-53. reported in 1984 by psychiatrist Mascol, "Curves of Pauline and 20. David I. Holmes, "A Stylometric Ernest H. Taves: Trouble Enough: Pseudo-Pauline Style II," Unitarian Analysis of Mormon Scripture and Joseph Smith and the Book of Mormon Review 30 (December 1888): 539-46. Related Texts," Journal of the Royal (Buffalo, NY: Prometheus Books, 13. L. A. Sherman, Analytics of Litera- Statistical Society A 155, part 1 (1992): 1984). In his Book of Mormon study ture: A Manual for the Objective Study 91-120. Taves used word-pattern ratios of English Prose and Poetry (Boston: 21. Matthew L. Jockers, Daniela M. similar to the Hilton study but Ginn, 1893). Witten, and Craig S. Criddle, "Reas- concluded that "the texts provide no 14. Mosteller and Wallace, Inference and sessing Authorship of the Book of evidence for multiple authorship" in Disputed Authorship. Mormon Using Delta and Nearest the Book of Mormon. In a thorough 15. John F. Burrows, "Word Patterns Shrunken Centroid Classification," and systematic review of Taves's and Story Shapes: The Statistical Literary and Linguistic Computing study, Hilton showed that the study Analysis of Narrative Style," Literary 23/4 (2008): 465-91. was fatally fiawed from improper and Linguistic Computing 2/1 (1987): 22. Discriminant analysis is a statisti- text sampling, misapplication of the 61-70; Burrows, "'An Ocean Where cal procedure that uses multiple chi-square test statistic, and other Each Kind . . .': Statistical Analysis variables (in this case, word fre- design weaknesses. John L. Hilton, and Some Major Determinants quencies) to classify new obser- review oí Book of Mormon Stylometry, of Literary Style," Computers and vations (text block of unknown by Ernest Taves (FARMS Prelimi- the Humanities 23 (1989): 309-21; authorship) using mathematical nary Report, 1986), 16; in light of the Burrows, "'Delta': A Measure of functions that weight each of the amateurish nature of Taves's study, Stylistic Difference and a Guide to variables to maximize the differ- Hilton advised—and we agree—that Likely Authorship," Literary and Lin- ences between known groups (text its usefulness is limited to show- guistic Computing 17/3 (2002): 267-87; blocks of known authorship) while ing how stylometry "should not be Burrows, "Questions of Authorship: minimizing the differences within done." See Kenneth H. Godfrey, Attribution and Beyond," Computers the groups. Thus, dissimilar groups "Not Enough Trouble," review of and the Humanities 37 (2003): 5-32. can be distinguished (discriminated) Trouble Enough: Joseph Smith and the Book of Mormon, by Ernest Taves, 16. Principal components analysis is from similar groups. Dialogue 19/3 (1986): 139-44. a multivariate statistical technique 23. Larsen, Rencher, and Layton, "Who that can reduce a large set of cor- Wrote the Book of Mormon?," 240. 31. David I. Holmes, "Vocabulary related variables to a smaller set 24. Larsen, Rencher, and Layton, "Who Richness and the Prophetic Voice," of uncorrelated variables that are Wrote the Book of Mormon?," 245. Literary and Linguistic Computing 6/4 linear combinations of the original The reprinted article toned down (1991): 259-68. variables and are arranged in order the claim to read, "The evidence 32. Holmes, "Stylometric Analysis of of decreasing importance. to date is that many authors wrote Mormon Scripture," 91. 17. While fingerprints and DNA profiles the Book of Mormon." Larsen and 33. Holmes, "Stylometric Analysis of can be precisely measured, an Rencher, "Who Wrote the Book of Mormon Scripture," 117,118. author's word-use preferences are Mormon?," 180. 34. This was pointed out to Holmes in a far more variable and nebulous. For 25. Hilton, "On Verifying Wordprint letter to the editor saying, "In general, example, when counting how fre- Studies," 89-108. conclusions about statistical similar- quently an author uses the word the 26. Andrew Q. Morton, Literary Detec- ity require knowledge of the power in multiple samples of the author's tion: How to Prove Authorship and of the statistical procedure to detect writing, the counts form a cluster Eraud in Literature and Documents differences." G. Bruce Schaalje, letter rather than a single dot, since an (New York: Scribner's Sons, 1978). to the editors. Journal of the Royal author does not use the with exactly 27. This has now been published as Statistical Society A156, part 1 (1993): the same frequency in every text. In Royal Skousen, ed.. The Printer's 115. Holmes replied, "My research contrast, a person's fingerprints and Manuscript of the Book of Mormon: showed that the multivariate ap-

VOLUME 21 • NUMBER 1 «2012 proach to measuring vocabulary rich- sessing Authorship of the Book of ity to produce misleading results ness successfully discriminates be- Mormon," 472. indicates that the researchers were tween samples from within the same 43. Jockers, Witten, and Criddle, "Reas- unfamiliar with the tool (NSC) they genre. Both the personal writings and sessing Authorship of the Book of were using and unskilled in its the prophetic voice of Joseph Smith Mormon," 473, table 1. proper use. differ from those of Joanna Southcott 44. A probability can be absolute only 46. Please note that the presence of a in the element of style represented if it includes all possible outcomes. few extreme values—either high by vocabulary richness." David I. The probability of rolling a 2 on or low—would be expected in any Holmes, author's reply. Journal of the a six-sided die is 1 out of 6 (ap- data set, even if the data come from Royal Statistical Society A 156, part 1 proximately 17%). This is an absolute a process that is truly random. (1993): 116. Apparently Holmes was probability since the event of rolling Their presence does not necessar- unwilling to acknowledge that one a 2 is one of six possible outcomes ily signify anything unusual in the successful application of a method among the complete set of events- data. Researchers who ignore the in a specific situation does not prove rolling a 1, 2, 3, 4, 5, or 6. But when multiplicity problem are prone to the method to be generally powerful considering the events of rolling a 1 finding evidence in the data that in all situations. See Holmes, "Vo- or rolling a 2 as the only admissible supports their preconceived notion cabulary Richness and the Prophetic outcomes and thereby ignoring the of what "should be" in the data. This Voice," 259-68. events of rolling a 3, 4, 5, or 6, the is often referred to as data snooping, 35. G. Bruce Schaalje, John L. Hilton, and probability of rolling a 2 is 1 out of 2 in contrast to data analysis. Such re- John B. Archer, "Comparative Power (50%). While the absolute probabil- searchers are determined to find in of Three Author-Attribution Tech- ity of rolling a 2 among all possible the data what they want to be in the niques for Differentiating Authors," events is 17%, the relative probability data regardless of facts and reason. Journal of Book of Mormon Studies 6/1 of rolling a 2 among the limited set 47. Located in Stephen Post Papers, (1997): 47-63. Comparing the discrim- of events of rolling a 1 or rolling a 2 folders 11 and 12, L. Tom Perry inating power of Holmes's method is 50%. Confusing an event's relative Special Collections, Harold B. Lee to other methods when applied to probability with its absolute prob- Library, Brigham Young University, texts of undisputed authorship, the ability can greatly exaggerate the Provo, Utah. perceived likelihood ofthat event. discriminating power of vocabulary 48. Jockers, Witten, and Criddle, "Reas- richness measures is much lower 45. When making multiple comparisons sessing Authorship of the Book of than that of noncontextual word simultaneously, a researcher can Mormon," 483. frequencies or word-pattern ratios. make an error in statistical inference 49. G. Bruce Schaalje, Paul J. Fields, R. Harald Baayen, Word Frequency by claiming two things are signifi- Matthew Roper, and Gregory L. Distributions (Boston: Klumer Aca- candy different when in fact they are Snow, "Extended Nearest Shrunken demic, 2001), 214. not. When viewed individually, a pair Centroid Classification: A New 36. David I. Holmes and R. S. Forsyth, of items may appear to be different, Method for Open-Set Authorship At- "The Federalist Revisited; New Direc- yet when viewed in the full context of tribution of Texts of Varying Sizes," tions in Authorship Attribution," all possible comparisons in the study, Literary and Linguistic Computing 26/1 Literary and Linguistic Computing 10/2 their difference can be negligible. The (2011): 71-88. (1995): 111-27. probability of making such an error 50. G. Bruce Schaalje and Paul J. Fields, 37. David L. Hoover, "Another Perspec- increases as the number of multiple "Open-Set Nearest Shrunken Cen- tive on Vocabulary Richness," Com- comparisons increases. Consequently, troid Classification," Communications puters and Humanities 37 (2003): 173. the more complex a problem is, the in Statistics: Theory and Methods 41 38. Jockers, Witten, and Criddle, "Reas- greater the potential to make such an (2012): 63B-52. sessing Authorship of the Book of error. Standard statistical techniques 51. The fundamental fallacy in the Mormon," 465-91. have been developed and are used by Jockers study was that they equated 39. Robert Tibshirani, Trevor Hastie, competent researchers to compensate genomics problems to stylometry Balasubramanian Narasimhan, and for the effects of multiplicity and to problems. Although NSC has proven Gilbert Chu, "Class Prediction by guard against making such inferential to be highly successful in genomics Nearest Shrunken Centroids, with errors. The multiplicity problem is a classification problems, stylometric Applications to DNA Microarrays," major issue to guard against in genetic problems are much different: a large Statistical Science 18/1 (2003): 104-17. association studies wherein millions set of texts is usually the subject of genetic markers can be analyzed 40. Personal Writings of Joseph Smith, simultaneously. Researchers in one of classification, the sample sizes comp. and ed. Dean C. Jessee, rev. study may report statistically signifi- vary over a wide range, and most ed. (Salt Lake City: Deseret Book, cant results, yet subsequent research- important, the set of candidate au- 2002). Holographic material of Jo- ers will not be able to reproduce thors usually cannot be assumed to seph Smith is highlighted in bold in those results. One of the possible be closed. These characteristics are this collection. reasons for such conflicting outcomes not present in the typical genom- 41. See supplementary data for Jockers, is the failure of the first researchers to ics analysis. Consequently, naïve Witten, and Criddle, "Reassessing account for multiplicity. Since NSC is application of NSC, as in the Jockers Authorship of the Book of Mormon," a classification procedure originally study, can produce highly mislead- 465-91, at http://purLstanford.edu/ intended for use in genomics applica- ing results. A reanalysis of the Book ir:rs276tc2764 (accessed 29 February tions, the failure of Jockers et al. to of Mormon using ENSC produced 2012). recognize the potential for multiplic- dramatically different results from 42. Jockers, Witten, and Criddle, "Reas- the NSC method.

JOURNAL OF THE BOOK OF MORMON AND OTHER RESTORATION SORIPTURE 45 Copyright of Journal of the Book of Mormon & Restoration Scripture is the property of Brigham Young University, The Neal A. Maxwell Institute for Religious Scholarship and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.