How Convinced Should We Be by Negative Evidence? Ulrike Hahn ([email protected]) School of Psychology, Cardiff University, Tower Building, Park Place Cardiff CF10 3AT, UK Mike Oaksford ([email protected]) School of Psychology, Cardiff University, Tower Building, Park Place Cardiff CF10 3AT, UK Hatice Bayindir ([email protected]) School of Psychology, Cardiff University, Tower Building, Park Place Cardiff CF10 3AT, UK Abstract satisfactory account. Testament to this is the fact that fallacies figure in logic textbooks under the header of ‘informal Since John Locke, the so-called argument from ignorance has reasoning fallacies’ (see e.g., Hamblin, 1970) – an been considered to be a fallacy, and is widely represented in acknowledgement of the absence of a sufficient formal informal logic textbooks as an example of incorrect reasoning. logical treatment. In particular, logical accounts have proved This might seem surprising to researchers in many scientific disciplines who routinely draw inferences from negative unable to capture the seeming exceptions to fallacies that evidence. Oaksford and Hahn (2004) argued that this arise with simple changes in content that leave the structure of discrepancy can be explained within a Bayesian framework. the argument unaffected. This suggests that either it is not We present here experimental evidence for this view. formal aspects of fallacies that make them fallacious, or else that the relevant formal aspects are not being tapped into by Introduction classical logics. Fallacies, or arguments that seem correct but aren’t, have The so-called pragma dialectical approach (see e.g., been a longstanding focus of debate. Catalogues of reasoning van Eemeren & Grootendorst, 2003; Walton, 1995) is a more and argumentation fallacies originate with Aristotle and recent approach to the fallacies which eschews the idea that populate books on logic and informal reasoning to this day. fallacies can be explained purely through reference to their One such classic fallacy, which dates back to John Locke, is inherent structure. Rather, fallacies need to be viewed within the so-called argument from ignorance, or argumentum ad the wider argumentative context in which they are embedded. ignorantiam: Arguments are fallacies because they fall short of standards of rational discourse. (1) Ghosts exist, because nobody has proven that they This approach has its roots in pragmatics (e.g. Grice, don’t Searle) and seeks to distinguish different types of argumentative discourse (e.g. ‘information seeking’) for This argument does indeed seem weak, and one would want which normative rules are then established. An example of to hesitate in positing the existence of all manner of things such a rule is: “the discussant who has called into question the whose non-existence simply had not been proven, whether standpoint of the other in the confrontation stage is always these be UFO’s or flying pigs with purple stripes. entitled to challenge the discussant to defend his standpoint” However, is it really the general structure of this (Rule 2, van Eemeren & Grootendorst, 2003). The argument argument that makes it weak, and if so what aspect of it is from ignorance on this account then, is fallacious wherever, responsible? Other arguments from negative evidence are and because, it violates the discourse rules of the current routine in scientific and everyday discourse and seem context. acceptable: What such an account cannot explain, however, is variations in the strength of different arguments from (2) This drug is safe, because no-one has found any side ignorance within the same type of discourse context. effects Oaksford & Hahn (2004) provide evidence of such variation and put forth an alternative, Bayesian account: individual Should all arguments from negative evidence be avoided, or arguments such as (1) and (2) are composed of a conclusion can a systematic difference between the two examples be and evidence for that conclusion. Both conclusion and recognized and explained? evidence have associated probabilities which are viewed as The classic tool brought to the analysis of fallacies expressions of subjective degrees of belief. Bayes’ theorem such as the argument from ignorance is formal logic and it is then provides an update rule for the degree of belief widely acknowledged to have failed in providing a associated with the conclusion in light of the evidence. Argument strength, then, on this account is a function of the 887 degree of prior conviction, the probability of evidence, and Let n denote sensitivity, i.e., n = P(e|T), l denote the relationship between the claim and the evidence, in selectivity, i.e., l = P(¬e|¬T), and h denote the prior particular how much more likely the evidence would be if the probability of drug A being toxic, i.e., h = P(T), then positive claim were true. test validity is greater than negative test validity as long as the A Bayesian account captures, among other things, following inequality holds: the difference between positive and negative evidence and allows one to capture the intuition that the positive argument h 2 (n − n 2 ) > (1− h) 2 (l − l 2 ) (6) (3a) is stronger than the negative argument (3b): Assuming maximal uncertainty about the toxicity of drug A, (3a) Drug A is toxic because a toxic effect was observed i.e., P(T) = .5 = h, this means that positive test validity, (positive argument). P(T|e), is greater than negative test validity, P(¬T|¬e), when selectivity (l) is higher than sensitivity (n) and n + l > 1. As (3b) Drug A is not toxic because no toxic effects were Oaksford and Hahn (2004) argue, these are conditions often observed (negative argument, i.e., the argument met in practice for a variety of clinical and psychological from ignorance). tests. Therefore, in a variety of settings, positive arguments are stronger than negative arguments. However, (3b) too can be acceptable where a legitimate test Oaksford and Hahn (2004) also provide has been performed, i.e., experimental evidence to the effect that positive arguments such as (3a) are indeed viewed as more convincing than their If drug A were toxic, it would produce toxic effects in negative counterparts under the conditions just described. The legitimate test. evidence from their experiment further shows that people are Drug A has not produced toxic effects in such tests sensitive to manipulations in the amount of evidence (one Therefore, A is not toxic versus 50 studies or tests) as predicted by the account. Finally, participants were sensitive to the degree of prior Demonstrating the relevance of Bayesian inference for belief a character in a dialogue initially displayed toward the negative vs. positive arguments involves defining the conclusion, as the Bayesian account predicts. This finding conditions for a legitimate test. Let e stand for an experiment captures the ‘audience dependence’ of argumentation where a toxic effect is observed and ¬e stand for an assumed in the rhetorical research tradition (e.g., Perelman & experiment where a toxic effect is not observed; likewise let T Olbrechts-Tyteca, 1969). stand for the hypothesis that the drug produces a toxic effect Though these results are encouraging, they were and ¬T stand for the alternative hypothesis that the drug does drawn from a single experiment using only two topics of not produce toxic effects. The strength of the argument from argument. It is consequently important to test the generality ignorance is given by the conditional probability that the of the account with other materials. We do this in Experiment hypothesis, T, is false given that a negative test result, ¬e, is 1. Experiment 2 then examines further structural variants of found, P(¬T|¬e). This probability is referred to as negative arguments with negative evidence. test validity. The strength of the argument we wish to The experimental tests of the Bayesian account we compare with the argument from ignorance is given by provide here have a dual role. With regards to the positive test validity, i.e., the probability that the hypothesis, development of a normative account of argument strength, T, is true given that a positive test result, e, is found, P(T|e). participants’ data provide basic modal intuitions about These probabilities can be calculated from the sensitivity argument strength to supplement our own. This is important (P(e|T)) and the selectivity (P(¬e|¬T)) of the test and the as it is only too easy to mistake one’s own judgments for prior belief that T is true (P(T)) using Bayes’ theorem: universal. At the same time, our Bayesian account provides (only) a computational level theory. A detailed psychological P(e | T)P(T ) P(T | e) = (4) account of how people actually evaluate arguments is still P(e | T)P(T) + P(e | T )(1− P(T)) required. Experimental data is essential for such an account as P(e | T )(1− P(T )) well. To this latter end, it is of interest not only whether or not P(T | e) = (5) people are sensitive to the basic factors posited by the − + P(e | T )(1 P(T )) P(e | T )P(T ) account, but also how sensitive they are to their interactions and what limitations people show in practice. There are Sensitivity corresponds to the “hit rate” of the test and 1 numerous finer interactions between prior belief, polarity and minus the selectivity corresponds to the “false positive rate.” evidence predicted by the Bayesian account (for details see There is a trade-off between sensitivity and selectivity which Oaksford and Hahn, 2004, in particular Figure 1); however, is captured in the receiver operating characteristic curve as these are of interest primarily for more detailed modeling (Green & Swets, 1966) that plots sensitivity against the false they will not be considered here.