Heredity (2016) 117, 193–206 & 2016 Macmillan Publishers Limited, part of Springer Nature. All rights reserved 0018-067X/16 www.nature.com/hdy REVIEW Prediction and estimation of effective population size J Wang1, E Santiago2 and A Caballero3 Effective population size (Ne) is a key parameter in population genetics. It has important applications in evolutionary biology, conservation genetics and plant and animal breeding, because it measures the rates of genetic drift and inbreeding and affects the efficacy of systematic evolutionary forces, such as mutation, selection and migration. We review the developments in predictive equations and estimation methodologies of effective size. In the prediction part, we focus on the equations for populations with different modes of reproduction, for populations under selection for unlinked or linked loci and for the specific applications to conservation genetics. In the estimation part, we focus on methods developed for estimating the current or recent effective size from molecular marker or sequence data. We discuss some underdeveloped areas in predicting and estimating Ne for future research. Heredity (2016) 117, 193–206; doi:10.1038/hdy.2016.43; published online 29 June 2016 INTRODUCTION which correspond to the so-called inbreeding and variance effective The concept of effective population size, introduced by Sewall Wright sizes, respectively (Crow and Kimura, 1970). (1931, 1933), is central to plant and animal breeding (Falconer and Predictions of the effective population size can also be obtained Mackay, 1996), conservation genetics (Frankham et al.,2010; from the largest nonunit eigenvalue of the transition matrix of a Allendorf et al., 2013) and molecular variation and evolution Markov Chain which describes the dynamics of allele frequencies. (Charlesworth and Charlesworth, 2010), as it quantifies the magnitude Such derived effective size is called as eigenvalue effective size (see of genetic drift and inbreeding in real-world populations. A substantial Ewens, 1979, pp. 104–112), which is equivalent to the random number of extensions to the basic theory and predictions were made extinction effective size (Crow, 1954; see also Haldane, 1939). The since the seminal work of Wright, with main early developments by transition matrix can be written for many genetic models and is James Crow and Motoo Kimura (Kimura and Crow, 1963a; Crow and particularly useful for complex scenarios such as populations varying Kimura, 1970) and later by a list of contributors. Several review papers in size, having age structures or being subject to demographic changes (Crow and Denniston, 1988; Caballero, 1994; Wang and Caballero, (for example, Charlesworth, 2001; Pollak, 2002; Wang and Pollak, 1999; Nomura, 2005a) and population genetic books (Fisher, 1965; 2002; Engen et al., 2005). A less often used approach is that for the Wright, 1969; Ewens, 1979; Nagylaki, 1992) have summarised the mutation effective size, defined by the probability of identity in state of existing theory in predicting the effective size of a population at genes rather than identity by descent under an infinite allele model of different spatial and timescales under various inheritance modes mutations with a defined mutation rate (Whitlock and Barton, 1997). and demographies. Comparatively, methodological developments Later developments based on coalescence theory (Wakeley, 2008) (reviewed by Schwartz et al., 1999; Beaumont, 2003a; Wang, 2005; have also proved to be useful in the prediction of effective population Palstra and Ruzzante, 2008; Luikart et al., 2010; Gilbert and Whitlock, size, particularly in the evolutionary context for predicting genetic 2015) in estimating the effective size of natural populations from variability at the molecular level (Charlesworth, 2009; Nicolaisen and genetic data lag behind but are accelerating in the past decade, thanks Desai, 2012, 2013). The coalescence theory states that the chance of to the rapid developments of molecular biology. coalescence of any two random gene copies in one generation time is The classical developments of effective population size theory are 1/2N, which is the same as the rate of increase in identity by descent based on the rate of change in gene frequency variance (genetic drift) occurred from one generation to the next one. Thus, the probability of or the rate of inbreeding. The effective population size is defined in coalescence t generations ago is (1 − (1/2N))t − 1(1/2N). Therefore, the reference to the Wright–Fisher idealised population, that is, a average time of coalescence of two randomly chosen genes is T = 2N. hypothetical population with very simplifying characteristics where The coalescent effective population size refers to the expected time of genetic drift is the only factor in operation, and the dynamics of allelic coalescence T,ingenerations,ofgenecopiessuchthatT = 2Ne and genotypic frequencies across generations merely depend on the (Nordborg and Krone, 2002; Wakeley and Sargsyan, 2009). population census (N) size. The effective size of a real population is In this paper, we present a general overview of the main develop- then defined as the size of an idealised population, which would give ments for predicting the effective population size (Ne). The review rise to the rate of inbreeding and the rate of change in variance of gene does not attempt to be exhaustive, and some of the material frequencies actually observed in the population under consideration, mentioned in previous reviews will not be repeated. We mainly focus 1Institute of Zoology, Zoological Society of London, London, UK; 2Departamento de Biología Funcional, Facultad de Biología, Universidad de Oviedo, Oviedo, Spain and 3Departamento de Bioquímica, Genética e Inmunología, Facultad de Biología, Universidad de Vigo, Vigo, Spain Correspondence: Dr J Wang, Institute of Zoology, Zoological Society of London, Regent’s Park, London NW1 4RY, UK. E-mail: [email protected] Received 29 September 2015; revised 3 May 2016; accepted 16 May 2016; published online 29 June 2016 Predicting and estimating Ne JWanget al 194 on populations with different modes of reproduction, populations with under selection and populations under genetic management in captive "# 2 breeding conservation programmes, complementing previous reviews 2 ¼ Nf 2 þ Nm þ Nm 2 Sk Smm 2 Smm;mf Smf and adding material not covered or only partially covered by them. We Nm þ Nf Nf Nf also review the developments in estimating contemporary effective "# 2 sizes from genetic marker data, focussing on the estimation principles þ Nm 2 þ Nf þ Nf 2 ; ð Þ Sff 2 Sfm;ff Sfm 3 and ignoring the technical details that were covered in the original Nm þ Nf Nm Nm papers. The underlying assumptions, application scopes, robustness where S2 is the variance of the number of offspring of sex y from and accuracies of different estimation methods are discussed and xy parents of sex x and Sxm,xf is the covariance of the numbers of male compared. and female offspring from parents of sex x.Equation(2)isthesameas that derived by Hill (1979), although it is expressed in a different form. PREDICTION OF THE EFFECTIVE POPULATION SIZE It reduces to the classical equation of Wright (1933, 1939) for a Poisson In this section, we will summarise the main predictive equations for 2 ¼ = ; ¼ distribution of progeny number (that is, Sxy Ny Nx Sxm;xf 0for the asymptotic effective population size reached after a number of sex x, y = m, f), generations in a regular breeding system. In this case, all of the above approaches generally lead to the same predictive equations of Ne, 4NmNf Ne ¼ ; ð4Þ except for a few particular scenarios. For example, in a regular Nm þ Nf breeding system for an undivided population the asymptotic inbreed- which shows that unequal numbers of males and females in a ing and variance effective sizes converge. Only in situations such as population introduce a systematic variance in contribution between when the population is subdivided permanently in independent male and female parents and thus a reduction in effective size. sublines with completely independent pedigrees (Wang, 1997a, b) or Predictive formulae for the effective size of X-linked genes were when the population is decreasing or increasing in size, these types of originally given by Wright (1933) and later extended by other authors Ne will differ permanently. In fact, many of the equations shown (see Caballero, 1995). Later developments have also been made for below have been derived by two or more of the above approaches, and Y-linked and maternally transmitted genes (Charlesworth, 2001; we will only mention some of these. For clarity and a better Laporte and Charlesworth, 2002; Evans and Charlesworth, 2013). understanding of the main principles, several simplifying assumptions The generalisation of Equation (1) to the case of a partially selfed will also be made in this prediction section. Unless otherwise stated, population (in which there is partial selfing with proportion β, we will assume that populations do not change size through time and random mating otherwise) is are large enough so that second-order terms of 1/Ne can be safely 4N ignored. These terms are generally of little relevance but make the N ¼ ; ð5Þ e ðÞþÀ a 2ðÞþ a derivations and the Ne equations rather cumbersome. Finally, a single 21 Sk 1 undivided population with discrete generations under a regular (Crow and Morton, 1955), where breeding scheme will be generally assumed unless otherwise indicated, a ¼ b=ðÞ2b ðÞ so that prediction equations refer to asymptotic (Caballero, 1994) 2 6 effective population sizes. (Haldane, 1924) quantifies the deviation from Hardy–Weinberg equilibrium or the correlation of genes within individuals relative to Populations with different modes of reproduction the genes taken at random from the population (Wright, 1969). The As a starting point, we consider the simple equation derived by Wright value of α in a large random mating population is approximately 0 (1938), which takesÀÁ account of the variance of the contributions from (slightly negative when second-order terms are considered; see 2 parents to progeny Sk in a population of constant size N, Equation (23) below and Wang, 1996a).