<<

Proc. Natl. Acad. Sci. USA Vol. 94, pp. 13046–13050, November 1997

A –culture coevolutionary model for brother–sister mating ( taboo͞norm of reaction͞evolutionarily stable strategies͞cultural transmission͞inbreeding depression)

KENICHI AOKI† AND MARCUS W. FELDMAN‡

†Department of Biological Sciences, University of Tokyo, Hongo 7-3-1, Bunkyo-ku, Tokyo 113, Japan; and ‡Department of Biological Sciences, Stanford University, Stanford, CA 94305

Communicated by Paul R. Ehrlich, Stanford University, Stanford, CA, September 10, 1997 (received for review June 18, 1997)

ABSTRACT We present a gene–culture coevolutionary depending on the cultural pressures its carrier experiences. For model for brother–sister mating in the human. It is shown that example, whether the parents are sibs or unrelated may directly cultural—as opposed to innate—determination of mate pref- affect the probability that their offspring mate with one erence may evolve, provided the depression is another. sufficiently high. At this coevolutionary equilibrium, sib mat- Here we present a coevolutionary model in which the ing is avoided because of cultural pressures. probability of sib mating is determined by an individual’s genotype and also by whether or not its parents were sibs. Westermarck (1) proposed that ‘‘there is an innate aversion to Analysis of this model yields the particularly interesting result sexual intercourse between persons living closely together that a wide norm of reaction may be evolutionarily stable (see from early youth’’ (italics added). In particular, sibs reared qualification below). When gene–culture coevolution has re- together are expected to avoid mating with one another. sulted in fixation of a genotype with a wide norm of reaction, Studies on the Chinese custom of sim-pua (adoption of a future avoidance of sib mating will not be innate. Nevertheless, the daughter-in-law) (2) and on patterns among adults evolved biases in cultural transmission and the selective dis- reared together as children in Israeli kibbutzim (3) seem to advantage of inbred offspring will cause sib mating to be support Westermarck’s proposition. eliminated from the population. At this coevolutionary equi- Throughout the , invertebrate, and plant king- librium, individuals are genetically capable of mating with sibs, doms, the propensity to inbreed varies widely (4). In plants, but do not because of cultural pressure. both self-fertilization and self-incompatibility are widely rep- resented. From simple population genetic models, an Model that increases the level of may succeed if the depression of that results from inbreeding is less than 50% (5–9). With genetic determination of the inbreeding rate, it is well In and , experiments designed to detect inbreed- known that there is an evolutionary trade-off between the ing avoidance have produced mixed results (10). In Japanese transmission of identical by descent to inbred offspring quail, for example, females reared with siblings chose the and the inbreeding depression suffered by those offspring. In company of first cousins over that of both sibs and third cousins the case of sib mating in an outbreeding population, the (11). Such kin recognition mechanisms would permit inbreed- relatedness of a parent to inbred and outbred offspring is in the ing to be avoided without the necessity of dispersal, which is ratio of 3:2. Hence, using an inclusive fitness argument, if the Ϫ commonly evoked as a major means of inbreeding depression is d, sib mating will invade if 3 (1 d) Ͼ Ͻ 1 in many mammals. 2ord ⁄3. This was also observed in a formal dynamical An ‘‘innate’’ behavior is understood as one that has the same analysis (7). This argument holds under both diploidy and expression across the normal range of environments, so that haploidy. For mathematical simplicity, we assume haploidy in within the species, there is little variation. Humans, however, what follows. Specifically, we posit two haploid genotypes A1 are subject to an extraordinarily wide array of social environ- and A2. ments, partly as a result of cultural differences among ethnic Experimental data suggest that mate choice is more strongly groups. Whereas sib mating has been banned in most societies exercised by the female of a species. Therefore, we posit that that we know about, past and present, it was institutionalized the occurrence of sib mating is dependent on female choice. If in some (12, 13). Cousin mating, a milder form of inbreeding, she is of genotype Ai and her parents are sibs, the probability is often preferred over complete outbreeding (14). she will want to mate with her brother is bis, whereas if her In the case of institutionalized sib mating in Roman Egypt, parents are unrelated, the corresponding probability is bir.In the innate avoidance mechanism postulated by Westermarck this context we should note, however, that marital dissatisfac- was readily overridden. Wolf’s studies (15, 16) of minor tion in the studied by Wolf is not necessarily marriage in Taiwan are usually regarded as providing evidence expressed more strongly in the sim-pua (i.e., wife). for the . But matings between the foster Mating is conditional on survival to reproductive age. Each sibs did in fact occur. individual, male or female, whose parents are sibs has a Archaeological demonstration of lithic traditions suggests viability of 1 Ϫ d relative to an individual with unrelated that cultural transmission has been an important determinant parents. One girl and one boy are born in each family, and of behavior in hominids for more than two million years. In a deaths occur independently. A surviving female may want to cultural species, it is more appropriate to frame the discussion mate with her brother, but can only do so if he also survives. in terms of a ‘‘norm of reaction’’ rather than narrowly innate If he dies, she mates with an unrelated male with probability behavior. Thus, a genotype may respond quite differently h. The remaining fraction 1 Ϫ h of incestuous females who have lost their brothers refrain from mating. The publication costs of this article were defrayed in part by page charge Define Sij to be the frequency of ordered sib matings payment. This article must therefore be hereby marked ‘‘advertisement’’ in between an Ai female and an Aj male, with Rij the correspond- accordance with 18 U.S.C. §1734 solely to indicate this fact. ing frequency of random matings between an Ai female and an © 1997 by The National Academy of Sciences 0027-8424͞97͞9413046-5$2.00͞0 PNAS is available online at http:͞͞www.pnas.org. Abbreviation: ESS, evolutionarily stable strategies.

13046 Downloaded by guest on September 30, 2021 Evolution: Aoki and Feldman Proc. Natl. Acad. Sci. USA 94 (1997) 13047

Aj male. Among the newborns of the next generation, let Xij be ͕1 Ϫ ͑1 Ϫ d͓͒1 Ϫ b d͑1 Ϫ h͔͖͒Sˆ ϩ ͑1 Ϫ d͒2b Ϫ b ␭ ϭ 1s 11 1s 1r the frequency of ordered sibships, comprising an Ai sister and Ϫ ͕ Ϫ ͑ Ϫ ͓͒ Ϫ ͑ Ϫ ͔͖͒ˆ 1 1 1 d 1 b1sd 1 h S11 an Aj brother and derived from a sib mating. Similarly, Yij is the frequency of ordered sibships of the same composition but ϭ 1 ϩ fЈ͑Sˆ ͒͞Tˆ. [7] derived from unrelated parents. In this haploid model, 11 ϩ ϩ (Here we are still assuming fixation on the genotype A1. S12 S21 R12 R21 ϭ ϩ ϭ ϩ Stability to introduction of the allele A2 will be considered X11 S11 , Y11 R11 ␭ ϾϪ 4 4 later.) Clearly, 1. Hence, the equilibrium Sˆ11 is stable if fЈ (Sˆ ) Ͻ 0—i.e., when Sˆ is the smaller root of f(S ) ϭ 0, and ϩ ϩ 11 11 11 S12 S21 R12 R21 Sˆ is unstable when fЈ (Sˆ ) Ͼ 0. Thus, when there is only one X ϭ X ϭ , Y ϭ Y ϭ 11 11 12 21 4 12 21 4 valid equilibrium, it is always stable, and when there are two, the smaller is stable. S ϩ S R ϩ R ϭ ϩ 12 21 ϭ ϩ 12 21 X22 S22 , Y22 R22 . [1] 4 4 Stability to Introduction of New Allele A2

Finally, let zi be the frequency of genotype Ai among the We next consider stability of the genetic monomorphism in A1 random mating males. We assume that all males, whether derived above to introduction of a new allele A2. Standard mated to their sisters or not, compete equally for random linear stability analysis shows that the nonzero eigenvalues are mating females. Hence given by the roots of the characteristic polynomial ⌽͑␭͒ ϭ ͓͑ Ϫ ͒2͑ ϩ ͒͑͞ ˆ͒ Ϫ ␭͔͓͑ Ϫ ͒2 ͞ˆ Ϫ ␭͔ S ϩ S R ϩ R 1 d b1s b2s 4T 1 d b2s T Ј ϭ ͑ Ϫ ͒ͩ ϩ 12 21ͪ ϩ ϩ 12 21 Wz1 1 d S11 R11 , 2 2 ⅐ ͑ Ϫ ␭͒ ϩ ͑ ͞ ˆ͕͒͑ ϩ ͒␭ Ϫ ͑ Ϫ ͒2͑͞ ˆ͒ m* m 8T b1r 3b2r 1 d 2T ϩ ϩ S12 S21 R12 R21 ⅐ ͓b ͑b ϩ b ͒ ϩ b ͑b ϩ b ͔͖͒, [8a] WzЈ ϭ ͑1 Ϫ d͒ͩS ϩ ͪ ϩ R ϩ , [2] 2r 1s 2s 2s 1r 2r 2 22 2 22 2 where where a prime indicates adults of the next generation and ϭ ͑ Ϫ ͕͒͑ Ϫ ͓͒ Ϫ ͑ ϩ ͒͑ Ϫ ͔͒ˆ m 1 d 1 d 2 b1s b2s 1 dh S11 ϭ ͑ Ϫ ͒ ͸͸ ϩ ͸͸ W 1 d Sij Rij. [3] ϩ ͓ Ϫ Ϫ ͑ Ϫ ͔͒ˆ ͖͑͞ˆ ˆ ͒ i j i j 2 b1r b2s 1 dh R11 TW , [8b] ϭ ͕͑ Ϫ ͓͒ Ϫ ͑ Ϫ ͒ Ϫ ͔ˆ Our assumptions imply the following recursions in the basic m* 1 d 2 b1s 1 dh b2r S11 variables S and R , where all summations are over 1 and 2: ij ij ϩ ͑ Ϫ Ϫ ͒ ˆ ͖͑͞ ˆ ˆ ͒ 2 b1r b2r R11 2TW , [8c] TSЈ ϭ ͑1 Ϫ d͒2X b ϩ Y b , [4a] ij ij is ij ir ˆ ϭ ͑ Ϫ ͓͒ Ϫ ͑ Ϫ ͔͒ˆ ϩ ˆ T 1 d 1 b1sd 1 h S11 R11, Ј ϭ ͸͑ Ϫ ͒ ͑ ϩ Ϫ ͒ Ј TRij 1 d Xik bisdh 1 bis z j ˆ ϭ ͑ Ϫ ͒ˆ ϩ ˆ k W 1 d S11 R11. [8d] ϩ ͸ ͑ Ϫ ͒ Ј Yik 1 bir z j, [4b] ⌽(␭) is a cubic of the form k ⌽͑␭͒ ϭ ͑␭ Ϫ ␭͒͑␭ Ϫ ␭͒͑␭ Ϫ ␭͒ ϩ k͑␭ Ϫ ␭*͒, [9] where 1 2 3 ␭ Յ ␭ Յ ␭ Appendix 1 ϭ ͑ Ϫ ͒͸͸ ͓ Ϫ ͑ Ϫ ͔͒ ϩ ͸͸ where we may assume 1 2 3.In we show that, T 1 d Xij 1 bisd 1 h Yij . [4c] provided ␭* Յ ␭ , the monomorphic equilibrium is locally i j i j 3 stable if ⌽(1) Ͻ 0 and locally unstable if ⌽(1) Ͼ 0.

Genetic Monomorphism in A1 Norms of Reaction and Evolutionarily Stable Strategies (ESS) Assume that the genotype A1 is fixed. Dynamics of the ϭ frequency of sib matings, S11 (and of random matings, R11 Ϫ Each genotype Ai is defined by its norm of reaction specified 1 S11) are then determined by the vertical transmission rates, by bir, bis. A genetic monomorphism in A1 will be evolutionarily b1s and b1r, and by against inbred offspring, d. stable (17, 18) if the parameters b1r and b1s are such that A2 The equilibrium frequency of sib matings, Sˆ11, satisfies cannot invade whatever the values of b2r and b2s. ϭ ͑ ͒ ϭ ͓ ϩ ͑ Ϫ ͒͑ Ϫ ͒ ͔ 2 Our first major result is that a fully internal norm of 0 f S11 d 1 1 d 1 h b1s S11 Ͻ Ͻ reaction—i.e., 0 b1r, b1s 1—cannot be an ESS. The proof, Ϫ ͓ ϩ Ϫ ͑ Ϫ ͒2 ͔ ϩ which is detailed in Appendix 2, relies on small perturbations 1 b1r 1 d b1s S11 b1r . [5] ϭ ϩ ␰ ϭ ϩ ␩ of the parameters. Set b2r b1r and b2s b1s , where ␰ ␩ ␭ Ͻ ␭ Ն ϭϪ Ϫ Ϫ Ϫ Յ and are small. Then * 3 in which case linear stability Because b1r 0 and f(1) (1 d)[1 (1 hd) b1s] 0, Ϫ Ϫ Ͼ Ͼ is assured if and only if ⌽(1) Ͻ 0. In general, the expansion of there is only one valid equilibrium if 1 (1 hd) b1s 0(d Ϫ Ϫ ϭ ⌽(1) will take the form 0 is assumed). If 1 (1 hd) b1s 0, Eq. 5 factors to yield ⌽͑ ͒ ϭ ␰ ϩ ␩ ϩ ␰2 ϩ ␰␩ ϩ ␩2 ϩ ͑ ͒ ϭ ͑ Ϫ ͕͓͒ Ϫ ͑ Ϫ ͒2 ͔ Ϫ ͖ 1 c␰ c␩ c␰␰ c␰␩ c␩␩ third order terms. f S11 S11 1 1 1 d b1s S11 b1r . [6] [10] ˆ ϭ ͞ Ϫ Ϫ 2 Hence, two valid equilibria, S11 b1r [1 (1 d) b1s ] and For a fully internal norm of reaction to be an ESS, we require ϭ Ͻ Ϫ Ϫ 2 ϭ Sˆ11 1, exist if b1r 1 (1 d) b1s, but Sˆ11 1 is the only the linear terms c␰ ϭ c␩ ϭ 0 and the quadratic form to be valid equilibrium when this inequality is reversed. negative definite. However, as shown in Appendix 2, c␰␰ ϭ 0, 2 For small perturbations from the equilibrium, Sˆ11, stability whence c␰␰c␩␩ Ϫ (c␰␩͞2) Յ 0 and, even if c␰ ϭ c␩ ϭ 0, the is governed by the eigenvalue quadratic form is not negative definite. Downloaded by guest on September 30, 2021 13048 Evolution: Aoki and Feldman Proc. Natl. Acad. Sci. USA 94 (1997)

Ͻ Ͻ Thus, for any norm of reaction with 0 b1r, b1s 1, there is always another norm of reaction b2r, b2s that can invade. Յ Under the reasonable assumption that b1r b1s, an evolution- ϭ ͞ ϭ arily stable norm of reaction entails b1r 0 and or b1s 1.

0 ؍ Norms of Reaction with b1r

In view of the often-made claim that avoidance of sib mating ϭ is innate, the case b1r 0 deserves special attention. When b1r ϭ ϭ ϭ ϭ 0, we have Sˆ11 0 and Tˆ Wˆ 1 at the genetic ϭ ϭ monomorphism. (If, in addition, h 0 and b1s 1, the unstable ϭ equilibrium Sˆ11 1 also exists.) From Eqs. 8a and 8c, because ␭ ϭ Ϫ 2 ϩ ͞ Ͻ ͞ ϭ Ϫ ͞ Ն ͞ * (1 d) (b1s 2b2s) 6 1 2 and m* (2 b2r) 2 1 2, ␭ Ͻ ␭ ϭ we have * 3. Hence, linear stability of Sˆ11 0 to invasion by A2 is determined by the sign of ⌽͑ ͒ ϭϪ͑ ͞ ͕͒ ͓ Ϫ ͑ Ϫ ͒2͑ ϩ ͒͞ ͔ 1 b2r 8 4 1 1 d b1s b2s 4 ⅐ ͓ Ϫ ͑ Ϫ ͒2 ͔ Ϫ ͑ Ϫ ͓͒ Ϫ ͑ Ϫ ͔͒ 1 1 d b2s 1 d 2 b2s 1 dh ⅐ ͓ Ϫ ͑ Ϫ ͒2͑ ϩ ͒͞ ͔ 3 1 d b1s 2b2s 2 }. [11]

Notice first that b2r does not affect stability provided it is Ͼ positive. Here we assume b2r 0, reserving the analysis of the ϭ case b2r 0 for the next section. Second, a norm of reaction ϭ ϭ of the form b ϭ 0, b Ն 0 will be an ESS only if ⌽(1) Ͻ 0 for FIG. 1. The curves labeled b1s 0 and b1s 1 give the minimum 1r 1s values of the inbreeding depression, d, for which the two norms of all b2s. Rewrite Eq. 11 as reaction b1r ϭ b1s ϭ 0 and b1r ϭ 0 with b1s ϭ 1 will be evolutionarily Ͼ ⌽͑1͒ ϭϪ͑b ͞8͒⌿͑b ͒, [12a] stable against all alternatives that satisfy b2r 0. The horizontal axis 2r 2s measures the probability, h, that an incestuous female mates with an unrelated male when her brother dies. For intermediate values of b1s, where there is a family of nonintersecting curves that lie between these two ϭ ⌿͑ ͒ ϭ ͑ Ϫ ͞ ͒ Ϫ ͑ Ϫ ͒2 ϩ ͑ Ϫ ͓͒Ϫ ͑ Ϫ ͒ curves. The curve labeled b1s 0 meets the d axis at 1/3. b2s 6 d 1 3 d 1 d b1s d 1 d 2 1 d ϭ ϩ 3͑1 Ϫ h͔͒b ϩ ͑1 Ϫ d͒3͓1 Ϫ d Ϫ d͑1 Ϫ h͔͒ both characterized by bir 0 differing only in bis, which will do 2s better? ⅐ ͞ Ϫ ͑ Ϫ ͒3͑ Ϫ ͒ 2 11 b1sb2s 2 d 1 d 1 h b2s. [12b] From Eq. we expect that the dominant eigenvalue will be one, necessitating a second order analysis. This is outlined in ⌿ ϭ Hence, (b2s) is linear in b2s if h 1 and takes the form of a Appendix 3. Applying the method of Nagylaki (21), we obtain quadratic that is convex upward if 0 Յ h Ͻ 1. In either case, the interesting result that, even to second order, the genetic ⌿ Ͼ ⌽ Ͻ Յ Յ ⌿ Ͼ (b2s) 0[ (1) 0] for 0 b2s 1 if and only if (0) 0 monomorphism in A1 is neutrally stable. Hence, natural se- and ⌿(1) Ͼ 0. lection on the alternative norms will be very weak, and In particular, from the requirement that ⌿(0) Ͼ 0, a transitions may occur by random drift. In particular, a wide ϭ necessary condition for evolutionary stability of b1r 0is norm of reaction is just as likely to evolve as a narrow one. ͑ Ϫ ͞ ͒ Ϫ ͑ Ϫ ͒2 Ͼ 1 ؍ d 1 3 d 1 d b1s 0. [13] Norms of Reaction with b1s 6 Hence, for a given value of d Ͼ 1͞3, inequality 13 is more likely ϭ A norm of reaction with b1r 0 implies no sib matings at to be satisfied the smaller is the value of b1s, i.e., the narrower equilibrium. However, sibling incest is in fact observed at low, is the norm of reaction. but nonnegligible, frequencies in modern human societies. In Fig. 1 we graph the minimum values for d for which the ϭ ϭ ϭ ϭ Because sibling incest is rare, we should consider the evolu- two extreme cases, b1r b1s 0 and b1r 0 with b1s 1, will Ͼ tionary stability of norms of reaction with b1r small. To this be evolutionarily stable (subject to the condition b2r 0). The end, expand the characteristic polynomial Eq. 8b evaluated at horizontal axis measures h. For intermediate values of b1s we ␭ ϭ 1 in powers of b , giving have a family of nonintersecting curves that lie between these 1r two curves. The dependence on h and b1s is not pronounced, ⌽͑1͒ ϭϪ͑b ͞8͒⌿˜ ͑b ͒ ϩ ͑b ͞8͓͒⌿˜ ͑b ͒ ϩ c ͑1 Ϫ b ͒ ϭ ϭ 2r 2s 1r 2s 1 2s and we see that the norm of reaction b1r 0 with b1s 1 can Ͼ ϩ ͔ ϩ ͑ ͒ be an ESS if d 0.4. If the inbreeding depression from sib c2b2r o b1r , [14] matings in humans has consistently exceeded 40%, then a wide Ϫ ⌿˜ norm of reaction, i.e., a large difference (b1s b1r), may have where (b2s) is given by Eq. 12b with b1s set equal to 1, and c1 evolved. Based on Seemanova´’sstudy (19) of nuclear family and c2 are functions of the parameters. ⌽ Ͻ incest, Durham (20) estimates the inbreeding depression re- Evolutionarily stability obtains when (1) 0 for all b2r and ⌿ Ͻ sulting from death plus major defect to be about 45%. b2s. However, from Eq. 14 this cannot occur: If ˜ (b2s) 0 for ⌽ Ͼ ⌿ Ͼ some b2s, then (1) 0 for sufficiently large b2r.If ˜ (b2s) ⌽ Ͼ ϭ ϭ .for all b2s, then (1) 0 when b2r 0 and b2s 1. From Eq 0 0 ؍ b2r ؍ Case of b1r ⌿ 12b, we see that (b2s) can vanish at most twice for b2s in [0,1]. ϭ Recall that the norm of reaction we consider in this paper is Hence, we conclude that a strategy with b1r small and b1s 1 defined by the pair of probabilities, bir and bis. We showed in cannot be an ESS. ϭ the previous section that any norm of reaction with b1r 0 can On the other hand, numerical work suggests that a norm of be stable against all alternatives that satisfy the restriction b2r reaction where b1r is large can be evolutionarily stable to all Ͼ Ͻ ϭ 0. To complete the analysis, let us consider the fate of alternative strategies that satisfy b2s 1. When b2s 1, this ϭ mutants with b2r 0. In other words, when two strategies are norm of reaction is neutrally stable to second order. An Downloaded by guest on September 30, 2021 Evolution: Aoki and Feldman Proc. Natl. Acad. Sci. USA 94 (1997) 13049

ϭ ϭ example of such a norm of reaction is b1r 0.5 and b1s 1 individuals are genetically capable of sib mating, but do not when d ϭ 0.2 and h ϭ 0. However, in this case the frequency because of cultural pressures. of sib matings at equilibrium is predicted to be one, which is An interesting relation exists between the two norms of ϭ ϭ unrealistically high. reaction b1r,b1s, and b2r,b2s, where b1r b2r 0 but b1s and b2s differ. Both are neutrally stable to second order to invasion by Discussion the other. In fact, numerical iteration of Eq. 4 reveals that the ϩ ϩ ϩ ϭ subspace R11 R12 R21 R22 1 is neutrally stable. Hence, ϭ In our model, if bir bis the female’s mating preference is two such norms of reaction may coexist, implying a genetic completely determined by her genotype. The particular case bir polymorphism in the probability that a daughter from a ϭ ϭ bis 0 corresponds to innate avoidance of brother–sister brother–sister union will herself be incestuous. Moreover, in a Ͻ mating. On the other hand, if bir bis parental mating behavior finite population transitions between the two norms may occur directly affects mate choice by their daughter. In particular, if by random drift. ϭ ϭ ϭ Ͼ bir 0 and bis 1 then we can say that mate choice is culturally We now argue that the norm of reaction, b1r 0 with b1s determined. 0, may be consistent with the known facts and suggest a way in A norm of reaction provides for phenotypic plasticity of a which this may be tested. First, in apparent contradiction, sib genotype in a range of environments. Sensitivity to variation in matings are observed at low, but nonnegligible, frequencies in parental mating type may then be regarded as a norm of modern societies. However, nuclear family incest often in- reaction mediated by cultural transmission. In our model the volves the mentally retarded (19), so that this is unlikely to be norm of reaction for genotype Ai is defined by the pair bir, bis. an evolved response. Second, about one-sixth of all marriages Ϫ It is natural to refer to the difference bis bir as the width of in one region of Roman Egypt were between full sibs (13). this norm. These socially recognized unions were recorded in the census We have attempted to identify parameter sets b1r, b1s that are documents and persisted for several centuries before the unbeatable by any alternative set b2r, b2s. Although there are custom eventually disappeared. Hence, the institution is best no norms of reaction that are truly evolutionarily stable, the regarded a transient phenomenon, initiated by forces extra- approach has proved informative. First we showed that a fully neous to our model. We may be able to test our prediction of Ͻ Ͻ Ͼ internal norm of reaction 0 b1r, b1s 1 cannot be an ESS. b1s 0 (or, for example, that incest results from being Յ On the reasonable assumption b1r b1s (otherwise a daughter homozygous for a recessive gene) if sufficient trans- Ͼ is more likely to be incestuous when her parents are unrelated generational data are available. If b1s 0, brother–sister than when they are themselves incestuous), evolutionary sta- marriages should occur recurrently in the descendants of an ϭ ͞ ϭ bility entails b1r 0 and or b1s 1. Next we showed that a incestuous pair. What is transmitted vertically may not be an ϭ norm of reaction where b1r is positive and small with b1s 1 ‘‘inbreeding meme,’’ but rather economic circumstances that ϭ can also be rejected. Hence, the remaining candidates for an favor sib mating. However, the result will be the same. If b1s ϭ unbeatable parameter set are (i) b1r 0 with b1s arbitrary; and 0, on the other hand, occurrence of sib mating should be ϭ ϭ Ͼ (ii) b1r positive and large with b1s 1. sporadic. Third, the norm of reaction b1r 0 with b1s 0 Concerning alternative ii, numerical work suggests that the cannot be an ESS unless the inbreeding depression, d, exceeds ϭ norm of reaction b1r positive and large with b1s 1 may be 1͞3. As noted above, Durham’s (20) estimate of d based on evolutionarily stable. However, the only cases we have been Seemanova´’s(19) data is about 45%. However, this estimate able to identify entail an equilibrium frequency of one for sib combines the effects of mortality and morbidity, on the matings. Even if we introduce forms of cultural transmission assumption that serious abnormalities are the equivalent of other than the vertical (parent–offspring) assumed here (22), death. The inbreeding depression as computed from differen- it would seem difficult to reduce the predicted frequency to the tial mortality alone is about 10%. low levels observed in most societies past and present. Ralls et al. (24) estimated inbreeding depression in 38 ϭ A norm of reaction where b1r 0 may be evolutionarily captive species of mammals. As measured by survivorship to a Ͼ stable against all alternatives that satisfy b2r 0. Here, the specified age, the average inbreeding depression for sib mat- other parameter defining the norm, b1s, may take any value ings is 33%. They suggest that, under natural as opposed to between 0 and 1. In Fig. 1 we graph the minimum values of the captive conditions, the cost of inbreeding may be exacerbated. inbreeding depression, d, for which the two extreme cases, b1r In the case of humans, it is arguable whether inbreeding ϭ ϭ ϭ ϭ b1s 0 and b1r 0 with b1s 1, are evolutionarily stable. depression would have been more severe without modern For intermediate values of b1s, we have a family of noninter- medical care. Although this seems likely, reproductive com- secting curves that lie between these two curves. pensation coupled with selective infanticide of malformed When d is sufficiently large and h is relatively small—e.g., if children would partly reduce the disparity in the number of d Ͼ 0.358 and h Ͻ 0.617, any such norm of reaction will be surviving children. We conclude that the inbreeding depres- Ͼ evolutionarily stable (subject to the condition b2r 0). For sion for sib matings may have been about 45% during most of smaller d or larger h, Fig. 1 shows that a small value of b1s is history and prehistory, if serious congential abnormalities more likely to produce evolutionary stability. Note that a norm resulted in death, infertility, or the inability to mate. ϭ of reaction where b1r 0 and b1s is small involves relatively Finally, a few words on the . Why sex and strong genetic determination of inbreeding avoidance, whereas marriage within the nuclear family are prohibited by custom, ϭ if b1r 0 and b1s is large, there is a significant cultural rule, or law is a major conundrum that continues to puzzle component in the determination of the daughter’s preference. anthropologists, and even some jurists (ref. 25; see also ref. 20 There is no theoretical reason to prefer one norm over the for a recent review). The present paper only scratches the other on the basis of our model, although the latter norm surface of the problem, but we believe that it suggests one seems more realistic because human mate choice is sensitive to direction in which a solution may be sought. In the context of cultural pressures. That the former will evolve in a larger a gene–culture coevolutionary model, the establishment of an region of the space of parameters d and h is consistent with the incest taboo may be equated with the spread of an ‘‘outbreed- proposal of Cavalli-Sforza and Feldman (23) that genetic ing meme.’’ The meme spreads in our model because of determination will usually prevail over the cultural. evolved biases in cultural transmission and natural selection With this norm of reaction, regardless of the value of b1s, the against inbred offspring. There is no need to invoke rational frequency of sib mating will converge to zero at the gene– choice based on a realization of the deleterious consequences Ͼ culture coevolutionary equilibrium. In particular, if b1s 0, of inbreeding. Downloaded by guest on September 30, 2021 13050 Evolution: Aoki and Feldman Proc. Natl. Acad. Sci. USA 94 (1997)

0 0 0 0 0 ء ء Appendix 1 0 0 0 0 0 ء 0 0 0 0 0 0 ء Our demonstration assumes that in Eq. 9 we have the strict 0 inequalities 0 Ͻ ␭ Ͻ ␭ Ͻ ␭ , but the proof can be generalized. [A3-1] . 0 0 0 ء 0 ء M ϭ 0 3 2 1 8e ˆ Ն Ϫ 2 ␭ 2 0 1 ء 0 ء From Eq. , T (1 d) . Hence, two of the i’s have the 0 bounds 0 0 0 ء 0 ء 0 Յ ͑ Ϫ ͒2͑ ϩ ͒͑͞ ˆ͒ Յ ͞ 0 0 0 0 0 0 0 0 1 d b1s b2s 4T 1 2, [A1-1]

0 Յ ͑1 Ϫ d͒2b ͞Tˆ Յ 1, [A1-2] The asterisks indicate nonnegative terms. In particular, the 2s three diagonal asterisks are less than unity, whence the dom- and ␭* has the bounds inant eigenvalue one is nondegenerate. The right eigenvector corresponding to this dominant eig- Յ ͑ Ϫ ͒2͓ ϩ ͑ ϩ ͔͒͞ T 0 1 d b1rb2s b2r b1s 2b2s envalue (0,0,0,0,1,0,0) , where T denotes the transpose. Hence, according to the method of Nagylaki (21) we need consider ⅐ ͓ ˆ͑ ϩ ͔͒ Յ ͞ ϩ 2 2T b1r 3b2r 1 2. [A1-3] only the terms in (R12 R21) among the second-order terms. ϩ Ј Ј Such terms appear in the expansions of (R12 R21) and R22, Ϫ ϩ 2͞ ϩ 2͞ We wish to prove that the dominant eigenvalue, which is where they contribute (R12 R21) 2 and (R12 R21) 4, positive, is less than unity if ⌽(1) Ͻ 0, and is greater than unity respectively. ⌽ Ͼ ␭ Յ ␭ if (1) 0. Consider three cases. First assume * 1. Then Next, the left eigenvector corresponding to the eigenvalue ⌽ ␭ Ͼ ␭ Յ ␭ Յ ␭ ␭ Յ ( ) 0 in the interval 2 3, where 2 1 from Eqs. one is A1-1 A1-2 ⌽Љ ␭ ϭ ␭ Ϫ ␭ ϩ ␭ Ϫ ␭ ϩ ␭ ͒ ء ء ͑ and . Moreover, ( ) 2[( 1 ) ( 2 ) ( 3 Ϫ ␭ Ͻ ␭ Ն ␭ ⌽ ␭ ␭ 0, ,0, ,1,0,2, [A3-2] )] 0 for 3. Hence, ( ) intersects the -axis only ␭ Ͼ ␭ ␭ Ͻ ␭ Յ ␭ ⌽ ␭ Ͻ Յ once when 3. Next, if 1 * 2, then ( 1) 0 ⌽ ␭ ␭ Ͻ ␭ Յ ␭ Ͻ ␭ Յ where the asterisks here indicate positive terms. Then the ( 2) and we also have 1 2 1. Similarly, if 2 * ␭ ⌽ ␭ Ͻ Յ ⌽ ␭ ␭ Ͻ ␭ Յ ͞ product of the vector of second order terms and Eq. A3-2 3, then ( 1) 0 ( *) where 1 * 1 2. In all cases, ϭ ϭ ␭ Յ ␭ ⌽ ␭ vanishes. Thus, if b1r b2r 0, then to second order, the therefore, if * 3, then the largest root of ( ) is less than genetic monomorphism in A is neutrally stable to the intro- unity if ⌽(1) Ͻ 0 and greater than unity if ⌽(1) Ͼ 0. It is clear 1 ␭ Ͼ ␭ ⌽ Ͼ duction of A2. that if * 3 and (1) 0, the largest eigenvalue is greater than unity. 1. Westermarck, E. (1894) The History of Human Marriage (Mac- millan, London), 2nd Ed. Appendix 2 2. Wolf, A. P. (1970) Am. Anthropol. 72, 503–515. 3. Shepher, J. (1971) Arch. Sex. Behav. 1, 293–307. Set 4. Maynard Smith, J. (1978) The Evolution of Sex (Cambridge Univ. Press, Cambridge). ϭ ϩ ␰ ϭ ϩ ␩ b2r b1r , b2s b1s , [A2-1] 5. Lloyd, D. G. (1979) Am. Nat. 113, 67–79. 6. Charlesworth, B. (1980) J. Theor. Biol. 84, 655–671. ␰ ␩ ␭ Ϸ Ϫ 2 ͞ Ͻ Ϫ 7. Feldman, M. W. & Christiansen, F. B. (1984) Am. Nat. 124, where and are small. Then * (1 d) b1s (2Tˆ) (1 2 ͞ Ϸ Ϫ 2 ͞ ␭ Ͻ ␭ 624–653. d) b1s Tˆ (1 d) b2s Tˆ, which implies * 3. Substitution ␭ ϭ 8. Holsinger, K. E., Feldman, M. W. & Christiansen, F. B. (1984) of Eq. A2-1 and 1inEq.8b yields Am. Nat. 124, 446–453. 9. Nagylaki, T. (1976) J. Theor. Biol. 58, 55–58. ⌽͑ ͒ ϭ ͓͑ Ϫ ͒2 ͑͞ ˆ͒ Ϫ ϩ ͑ Ϫ ͒2␩͑͞ ˆ͔͒ 1 1 d b1s 2T 1 1 d 4T 10. Bateson, P. (1983) in Mate Choice, ed. Bateson, P. (Cambridge Univ. Press, Cambridge), pp. 257–277. ⅐ ͓͑ Ϫ ͒2 ͞ˆ Ϫ ϩ ͑ Ϫ ͒2␩͞ˆ͔ 1 d b1s T 1 1 d T 11. Bateson, P. (1982) Nature (London) 295, 236–237. 12. Hopkins, K. (1980) Comp. Stud. Soc. Hist. 22, 303–354. ⅐ ͓͕͑ Ϫ ͓͒ Ϫ ͑ Ϫ ͒ Ϫ ͔ˆ 1 d 2 b1s 1 dh b1r S11 13. Bagnall, R. S. & Frier, D. W. (1994) The Demography of Roman Egypt (Cambridge Univ. Press, Cambridge). ϩ ͑ Ϫ ͒ ˆ ͖͑͞ ˆ ˆ ͒ Ϫ Ϫ ␰͑͞ ˆ͔͒ 2 1 b1r R11 2TW 1 2T 14. Goody, J. (1990) The Oriental, the Ancient and the Primitive (Cambridge Univ. Press, Cambridge). ϩ ͑1 Ϫ d͒͑͞8Tˆ 2͓͕͒2͑1 Ϫ d͓͒1 Ϫ b ͑1 Ϫ dh͔͒Sˆ 15. Wolf, A. P. & Huang, C. S. (1980) Marriage and Adoption in China 1s 11 1845–1945 (Stanford Univ. Press, Stanford, CA). ϩ ͓2 Ϫ b Ϫ b ͑1 Ϫ dh͔͒Rˆ ͖͞Wˆ 16. Wolf, A. P. (1995) Sexual Attraction and Childhood Association 1r 1s 11 (Stanford Univ. Press, Stanford, CA). 156, Ϫ ͑ Ϫ ͒␩͔⅐ ͕ ͓ Ϫ ͑ Ϫ ͒2 ͑͞ ˆ͔͒ 17. Hamilton, W. D. (1967) Science 477–488. 1 dh 4b1r 1 1 d b1s 2T 18. Maynard Smith, J. & Price, G. R. (1973) Nature (London) 246, 15–18. ϩ ͓ Ϫ ͑ Ϫ ͒2 ͑͞ ˆ͔͒␰ Ϫ ͓ ͑ Ϫ ͒2 ͞ 3 1 1 d b1s 2T 3 1 d b1r 19. Seemanova´,E. (1971) Hum. Hered. 21, 108–128. 20. Durham, W. H. (1991) Coevolution (Stanford Univ. Press, Stan- ⅐ ͑2Tˆ͔͒␩ Ϫ ͓͑1 Ϫ d͒2͞Tˆ͔␰␩͖. [A2-2] ford, CA). 21. Nagylaki, T. (1977) Selection in One- and Two- Systems, Expanding Eq. A2-2 in ␰ and ␩, there is no ␰2 term. Hence, c␰␰ Lecture Notes in Biomathematics (Springer, Berlin). ϭ 0inEq.10. 22. Cavalli-Sforza, L. L. & Feldman, M. W. (1981) Cultural Trans- mission and Evolution (Princeton Univ. Press, Princeton, NJ). Appendix 3 23. Cavalli-Sforza, L. L. & Feldman, M. W. (1983) Proc. Natl. Acad. Sci. USA 80, 4993–4996. ϩ Ϫ 24. Ralls, K., Ballou, J. D. & Templeton, A. (1988) Conserv. Biol. 2, Expansion of Eq. 4 in the small variables S11, S12 S21, S12 185–193. ϩ Ϫ S21, S22, R12 R21, R12 R21, and R22 yields the local stability 25. Glendon, M. A. (1989) The Transformation of Family Law (Univ. matrix: of Chicago Press, Chicago). Downloaded by guest on September 30, 2021