File S1

Mathematical derivations

Identity by descent (IBD) and identity by state (IBS)

Consider the process g in the main text. As we assume that the genes under consideration are neutral, the expected allele frequency for any individual equals to that in the ancestral generation, = . As 𝐹𝐹 is an indicator function with values 0 or 1, it further holds that = , and = 0 for 𝐸𝐸�𝑥𝑥𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖 � 𝑝𝑝𝑗𝑗𝑗𝑗 𝑥𝑥𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖 . The probability that the alleles and in a gene j are of the same type u for the individuals i and i' is 𝐸𝐸�𝑥𝑥𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖 𝑥𝑥𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖 � 𝑝𝑝𝑗𝑗𝑗𝑗 𝐸𝐸�𝑥𝑥𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖 ′ 𝑥𝑥𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖 � given by . The probability that two randomly chosen alleles in a gene j are of the same type u for 𝑢𝑢 ≠ 𝑢𝑢′ 𝑘𝑘 𝑘𝑘′ the individuals i and i' is given by /4. This probability can be decomposed into two 𝐸𝐸�𝑥𝑥𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖 𝑥𝑥𝑖𝑖′𝑗𝑗𝑗𝑗 ′𝑢𝑢 � , components. First, the alleles may be identical by descent, the ancestral allele being of type u. The probability ∑𝑘𝑘 𝑘𝑘′ 𝐸𝐸�𝑥𝑥𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖 𝑥𝑥𝑖𝑖′𝑗𝑗𝑗𝑗 ′𝑢𝑢 � of this event is . The second option is that the alleles are not identical by descent, but the alleles in the ancestral generation were both of type . As the probability of the latter event is (1 ) 2 , we obtain 𝜃𝜃𝑖𝑖𝑖𝑖′ 𝑝𝑝𝑗𝑗𝑗𝑗 1 𝑝𝑝𝑗𝑗𝑗𝑗 − 𝜃𝜃𝑖𝑖𝑖𝑖′ 𝑝𝑝𝑗𝑗𝑗𝑗 E = + (1 ) 2 . (Eq. A1) 4 , � �𝑥𝑥𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖 𝑥𝑥𝑖𝑖′𝑗𝑗𝑗𝑗 ′𝑢𝑢 � 𝜃𝜃𝑖𝑖𝑖𝑖′ 𝑝𝑝𝑗𝑗𝑗𝑗 − 𝜃𝜃𝑖𝑖𝑖𝑖′ 𝑝𝑝𝑗𝑗𝑗𝑗 Similarly, for , 𝑘𝑘 𝑘𝑘′

≠ 𝑢𝑢′ 1 E = (1 ) . (Eq. A2) 4 , ′ � �𝑥𝑥𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖 𝑥𝑥𝑖𝑖′𝑗𝑗𝑗𝑗 ′𝑢𝑢′ � − 𝜃𝜃𝑖𝑖𝑖𝑖′ 𝑝𝑝𝑗𝑗𝑗𝑗 𝑝𝑝𝑗𝑗 𝑢𝑢 Thus, 𝑘𝑘 𝑘𝑘′

Cov[ , ] = E 4 = 4 , , , (Eq. A3) , , ′ ′ � 𝑥𝑥𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖 𝑥𝑥𝑖𝑖′𝑗𝑗𝑗𝑗 ′𝑢𝑢′ �� �𝑥𝑥𝑖𝑖𝑖𝑖𝑘𝑘𝑘𝑘 𝑥𝑥𝑖𝑖′𝑗𝑗𝑗𝑗 ′𝑢𝑢′ �� − 𝑝𝑝𝑗𝑗𝑗𝑗 𝑝𝑝𝑗𝑗𝑗𝑗 ′ 𝜃𝜃𝑖𝑖𝑖𝑖 ℎ𝑗𝑗 𝑢𝑢 𝑢𝑢 𝑘𝑘 𝑘𝑘′ 𝑘𝑘 𝑘𝑘′ where , , = , . As we assume no linkage among the genes, it holds that Cov , ′ ′= 0 for ′. ℎ𝑗𝑗 𝑢𝑢 𝑢𝑢 𝛿𝛿𝑢𝑢 𝑢𝑢 𝑝𝑝𝑗𝑗𝑗𝑗 − 𝑝𝑝𝑗𝑗𝑗𝑗 𝑝𝑝𝑗𝑗 𝑢𝑢 ′ ′ ′ ′ �𝑥𝑥𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖 𝑥𝑥𝑖𝑖 𝑗𝑗 𝑘𝑘 𝑢𝑢 � 𝑗𝑗 ≠ 𝑗𝑗′

IBD and IBS based definitions for FST, FIS and FIT

We denote by 0 the IBS probability for two alleles (same or different copy) in a randomly sampled gene j in a randomly sampled individual i in a randomly sampled subpopulation X. We denote by 1 the IBS probability for 𝑓𝑓 2 SI O. Ovaskainen et al. 𝑓𝑓 two alleles that are obtained by sampling two individuals (same or different) from a randomly chosen subpopulation, and then sampling alleles from a randomly chosen gene j of these two individuals. We denote by 2 the IBS probability for two alleles that are obtained by randomly selecting two subpopulations (same or different), then two individuals from these, and then two alleles of the same randomly selected gene j. 𝑓𝑓 2 We denote by = (1/ ) , the probability that two alleles from a randomly chosen gene j that are not identical by descent are identical by state. Note that IBS is possible when IBD does not hold although we ignore 𝜔𝜔 𝑛𝑛ℒ ∑𝑗𝑗 𝑢𝑢 𝑝𝑝𝑗𝑗𝑗𝑗 mutation, because multiple copies are available in the ancestral population. By Eq. A3 and the formulae for , , and (see Table I in the main text), 𝒮𝒮 𝜃𝜃 𝒫𝒫 𝜃𝜃 𝜃𝜃 1 1 1 1 = E = + 1 , (Eq. A4) 0 4 , ′ 𝒮𝒮 𝒮𝒮 𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖 𝑖𝑖𝑖𝑖 𝑘𝑘 𝑢𝑢 𝑓𝑓 𝒫𝒫 � 𝑋𝑋 � ℒ � � �′ �𝑥𝑥 𝑥𝑥 � 𝜃𝜃 � − 𝜃𝜃 � 𝜔𝜔 𝑛𝑛 1𝑋𝑋∈𝒫𝒫 𝑛𝑛 1𝑖𝑖∈𝑋𝑋 𝑛𝑛 1𝑗𝑗 𝑢𝑢 𝑘𝑘1𝑘𝑘 = E = + (1 ) , (Eq. A5) 1 2 4 , , 𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖 𝑖𝑖′𝑗𝑗𝑗𝑗 ′𝑢𝑢 𝑓𝑓 𝒫𝒫 � 𝑋𝑋 � ℒ � � � �𝑥𝑥 𝑥𝑥 � 𝜃𝜃 − 𝜃𝜃 𝜔𝜔 1 𝑛𝑛 𝑋𝑋∈𝒫𝒫1 𝑛𝑛 𝑖𝑖 𝑖𝑖′∈𝑋𝑋 𝑛𝑛 1 𝑗𝑗 𝑢𝑢 1𝑘𝑘 𝑘𝑘′ = E = + 1 . (Eq. A6) 2 2 4 , , , 𝒫𝒫 𝒫𝒫 𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖 𝑖𝑖′𝑗𝑗𝑗𝑗 ′𝑢𝑢 𝑓𝑓 � 𝑋𝑋 𝑌𝑌 � � � � �𝑥𝑥 𝑥𝑥 � 𝜃𝜃 � − 𝜃𝜃 � 𝜔𝜔 𝑛𝑛𝒫𝒫 𝑋𝑋 𝑌𝑌∈𝐵𝐵 𝑛𝑛 𝑛𝑛 𝑖𝑖∈𝑋𝑋 𝑖𝑖′∈𝑌𝑌 𝑛𝑛ℒ 𝑗𝑗 𝑢𝑢 𝑘𝑘 𝑘𝑘′ We note that in the notation of COCKERHAM and WEIR (1987), 1 = 0, 2 = 1, 3 = 2. COCKERHAM and WEIR 2 2 (1987) defined 0 = 1 1 as the variance within individuals, 1 = 1 2 as the variance among individuals 2 𝑄𝑄 𝑓𝑓 𝑄𝑄 𝑓𝑓 𝑄𝑄 𝑓𝑓 within subpopulations, and 2 = 2 3 as the variance among subpopulations within the population. 𝜎𝜎 − 𝑄𝑄 𝜎𝜎 𝑄𝑄 − 𝑄𝑄 Defining as the ratio of subpopulation variance to the total variance, we obtain 𝜎𝜎 𝑄𝑄 − 𝑄𝑄 𝑆𝑆𝑆𝑆 2 𝐹𝐹 2 1 2 = 2 2 2 = (Eq. A7) + + 1 2 0 𝜎𝜎1 2 𝑓𝑓 − 𝑓𝑓 𝐹𝐹𝑆𝑆𝑆𝑆 Similarly, 𝜎𝜎 𝜎𝜎 𝜎𝜎 − 𝑓𝑓

0 1 0 2 = , = . 1 1 1 2 𝑓𝑓 − 𝑓𝑓 𝑓𝑓 − 𝑓𝑓 𝐹𝐹𝐼𝐼𝐼𝐼 𝐹𝐹𝐼𝐼𝐼𝐼 measure the variance among individuals relative− to𝑓𝑓 that of the subpopulations− 𝑓𝑓 or the total variance. Combining the above, we obtain the IBD-based analogues

+ (1 ) 1 (1 ) = 𝒫𝒫 𝒫𝒫 = 𝒫𝒫 = , (Eq. A8) 1 1 (1 )(1 ) 1 𝒫𝒫 𝜃𝜃 − 𝜃𝜃 𝜔𝜔 − 𝜃𝜃 − � − 𝜃𝜃 � 𝜔𝜔 �𝜃𝜃 − 𝜃𝜃 � − 𝜔𝜔 𝜃𝜃 − 𝜃𝜃 𝐹𝐹𝑆𝑆𝑆𝑆 𝒫𝒫 𝒫𝒫 𝒫𝒫 𝒫𝒫 − 𝜃𝜃 − � − 𝜃𝜃 � 𝜔𝜔 − 𝜃𝜃 − 𝜔𝜔 − 𝜃𝜃 = , (Eq. A9) 1𝒮𝒮 𝜃𝜃 − 𝜃𝜃 𝐹𝐹𝐼𝐼𝐼𝐼 − 𝜃𝜃

O. Ovaskainen et al. 3 SI

= . (Eq. A10) 1𝒮𝒮 𝒫𝒫 𝜃𝜃 − 𝜃𝜃 𝐹𝐹𝐼𝐼𝐼𝐼 𝒫𝒫 − 𝜃𝜃 Additive genetic covariance among individuals

The covariance in the genetic value between traits m and in individuals i and is given by ′ ′ 𝑚𝑚 𝑖𝑖 Cov[ , , ] = Cov , = 4 , , (Eq. A11) , , , , ′ , , ′ 𝑖𝑖𝑖𝑖 𝑖𝑖′ 𝑚𝑚′ 𝑗𝑗𝑗𝑗𝑗𝑗 𝑗𝑗𝑗𝑗 ′𝑚𝑚′ 𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖 𝑖𝑖′𝑗𝑗𝑗𝑗 ′𝑢𝑢′ 𝑖𝑖𝑖𝑖 𝑗𝑗𝑗𝑗𝑗𝑗 𝑗𝑗𝑗𝑗 ′𝑚𝑚′ 𝑗𝑗 𝑢𝑢 𝑢𝑢 𝑎𝑎 𝑎𝑎 �′ 𝑣𝑣 𝑣𝑣 �𝑥𝑥 𝑥𝑥 � 𝜃𝜃 �′ 𝑣𝑣 𝑣𝑣 ℎ 𝑗𝑗 𝑘𝑘 𝑢𝑢 𝑘𝑘 𝑢𝑢′ 𝑗𝑗 𝑢𝑢 𝑢𝑢 To define the matrix, we construct a hypothetical individual representing the ancestral allele frequencies. We thus let 𝒜𝒜be a set of random vectors (independently for each j) that follow the multinomial distribution 𝐆𝐆 with parameters 1 and , so that E = . Let = … , be a random vector with = 𝑥𝑥𝑗𝑗𝑗𝑗 1, . Let be the covariance matrix with = 2 Cov[ ℒ , ]. We have , 𝑝𝑝𝑗𝑗𝑗𝑗 �𝑥𝑥𝑗𝑗𝑗𝑗 � 𝑝𝑝𝑗𝑗𝑗𝑗 𝛼𝛼 �𝛼𝛼 𝛼𝛼𝑛𝑛 � 𝛼𝛼𝑚𝑚 𝒜𝒜 𝒜𝒜 ∑𝑗𝑗 𝑢𝑢 𝑥𝑥𝑗𝑗𝑗𝑗 𝑣𝑣𝑗𝑗𝑗𝑗𝑗𝑗 𝐆𝐆 𝐺𝐺𝑚𝑚𝑚𝑚 ′ 𝛼𝛼𝑚𝑚 𝛼𝛼𝑚𝑚′ = 2 Cov , = 2 , , (Eq. A12) 𝒜𝒜 , , , , ′ 𝑚𝑚𝑚𝑚 ′ 𝑗𝑗𝑗𝑗𝑗𝑗 𝑗𝑗𝑗𝑗′𝑚𝑚′ 𝑗𝑗𝑗𝑗 𝑗𝑗𝑗𝑗′ 𝑗𝑗𝑗𝑗𝑗𝑗 𝑗𝑗𝑗𝑗′𝑚𝑚′ 𝑗𝑗 𝑢𝑢 𝑢𝑢 𝐺𝐺 � 𝑣𝑣 𝑣𝑣 �𝑥𝑥 𝑥𝑥 � �′ 𝑣𝑣 𝑣𝑣 ℎ 𝑗𝑗 𝑢𝑢 𝑢𝑢′ 𝑗𝑗 𝑢𝑢 𝑢𝑢

Combining the above shows that

Cov[ , ] = 2 (Eq. A13) ′ ′ 𝒜𝒜 𝒂𝒂𝑖𝑖 𝒂𝒂𝑖𝑖 𝜃𝜃𝑖𝑖𝑖𝑖 𝐆𝐆

We may consider Eq. A14 as the definition for the matrix. depends on the allele frequencies in the ancestral population ( ) and additive values of the𝒜𝒜 alleles ( )𝒜𝒜. Equivalently, as discussed above, we can 𝐆𝐆 𝐆𝐆 define. by constructing a hypothetical individual i that represents the allele frequencies in the focal 𝑝𝑝𝑗𝑗𝑗𝑗 𝑣𝑣𝑗𝑗𝑗𝑗 population,𝒜𝒜 i.e. by randomizing the alleles according to the allele frequencies independently for each allele 𝐆𝐆 in each gene. This leads to = 1/2, so the interpretation that is the variance in the additive value in 𝑝𝑝𝑗𝑗𝑗𝑗 such a hypothetical individual is consistent with Eq. A13. 𝒜𝒜 𝜃𝜃𝑖𝑖𝑖𝑖 𝐆𝐆

G in a local population

To compute for a particular population, we apply Eq. A12 with the allele frequencies corresponding to those present in that population. We denote the amount of additive variance present in 𝑗𝑗𝑗𝑗population by , 𝐆𝐆 1 𝑝𝑝 and the allele frequencies present in the population by = . ( ) 2 , 𝑋𝑋 𝐆𝐆𝑋𝑋 𝑋𝑋 𝑝𝑝 𝑋𝑋 𝑗𝑗𝑗𝑗 𝑛𝑛𝑋𝑋 ∑𝑖𝑖∈𝑋𝑋 𝑘𝑘 𝑥𝑥𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖 By the above it holds that

4 SI O. Ovaskainen et al.

, = 2 , ( ) ( ) ( ) ′ , , ′ ′ ′ ′ 𝑋𝑋�𝑚𝑚 𝑚𝑚 � 𝑗𝑗𝑗𝑗𝑗𝑗 𝑗𝑗 𝑢𝑢 𝑚𝑚 𝑢𝑢 𝑢𝑢 𝑋𝑋 𝑗𝑗𝑗𝑗 𝑋𝑋 𝑗𝑗𝑗𝑗 𝑋𝑋 𝑗𝑗𝑢𝑢 𝐆𝐆 �′ 𝑣𝑣 𝑣𝑣 �𝛿𝛿 𝑝𝑝 − 𝑝𝑝 𝑝𝑝 � 𝑗𝑗 𝑢𝑢 𝑢𝑢 = 2 ( ) ( ) ( ) . (Eq. A14) ′ , ′ � �� 𝑝𝑝 𝑋𝑋 𝑗𝑗𝑗𝑗 𝑣𝑣𝑗𝑗𝑗𝑗𝑗𝑗 𝑣𝑣𝑗𝑗𝑗𝑗 𝑚𝑚 − � 𝑝𝑝 𝑋𝑋 𝑗𝑗𝑗𝑗 𝑝𝑝 𝑋𝑋 𝑗𝑗𝑢𝑢 𝑣𝑣𝑗𝑗𝑗𝑗𝑗𝑗 𝑣𝑣𝑗𝑗𝑗𝑗 ′𝑚𝑚′ � 𝑗𝑗 𝑢𝑢 𝑢𝑢 𝑢𝑢′ The allele frequencies in the population and thus also depend on the realization of the process g. To compute the expectation of , we note that 𝑋𝑋 𝐆𝐆𝑋𝑋 𝐹𝐹 𝐆𝐆𝑋𝑋 1 E[ ] = E[ ] = (Eq. A15) ( ) 2 , 𝑋𝑋 𝑗𝑗𝑗𝑗 𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖 𝑗𝑗𝑗𝑗 𝑝𝑝 𝑋𝑋 � 𝑥𝑥 𝑝𝑝 𝑛𝑛 𝑖𝑖∈𝑋𝑋 𝑘𝑘 and

1 E[ ] = E[ ]. (Eq. A16) ( ) ( ) 4 2 , , , 𝑋𝑋 𝑗𝑗𝑗𝑗 𝑋𝑋 𝑗𝑗𝑗𝑗 ′ 𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖 𝑖𝑖′𝑗𝑗𝑗𝑗 ′𝑢𝑢′ 𝑝𝑝 𝑝𝑝 ′ � 𝑥𝑥 𝑥𝑥 𝑛𝑛𝑋𝑋 𝑖𝑖 𝑖𝑖 ∈𝑋𝑋 𝑘𝑘 𝑘𝑘′ By Eqs. A2 and A3,

1 2 2 E[ ( ) ( ) ] = 2 + (1 ) = + 1 (Eq. A17) , 𝒫𝒫 𝒫𝒫 𝑋𝑋 𝑗𝑗𝑗𝑗 𝑋𝑋 𝑗𝑗𝑗𝑗 𝑖𝑖𝑖𝑖′ 𝑗𝑗𝑗𝑗 𝑖𝑖𝑖𝑖′ 𝑗𝑗𝑗𝑗 𝑋𝑋 𝑗𝑗𝑗𝑗 𝑋𝑋 𝑗𝑗𝑗𝑗 𝑝𝑝 𝑝𝑝 �′ 𝜃𝜃 𝑝𝑝 − 𝜃𝜃 𝑝𝑝 𝜃𝜃 𝑝𝑝 � − 𝜃𝜃 � 𝑝𝑝 𝑛𝑛𝑋𝑋 𝑖𝑖 𝑖𝑖 ∈𝑋𝑋 and for

𝑢𝑢 ≠ 𝑢𝑢′ 1 E[ ( ) ( ) ] = 2 (1 ) = (1 ) . (Eq. A18) , 𝒫𝒫 𝑋𝑋 𝑗𝑗𝑗𝑗 𝑋𝑋 𝑗𝑗𝑢𝑢′ 𝑖𝑖𝑖𝑖′ 𝑗𝑗𝑗𝑗 𝑗𝑗𝑗𝑗 ′ 𝑋𝑋 𝑗𝑗𝑗𝑗 𝑗𝑗𝑗𝑗 ′ 𝑝𝑝 𝑝𝑝 �′ − 𝜃𝜃 𝑝𝑝 𝑝𝑝 − 𝜃𝜃 𝑝𝑝 𝑝𝑝 𝑛𝑛𝑋𝑋 𝑖𝑖 𝑖𝑖 ∈𝑋𝑋 Thus

E[ ( ) ( ) ] = + 1 , (Eq. A19) , 𝒫𝒫 𝒫𝒫 , ′ 𝑋𝑋 𝑗𝑗𝑗𝑗 𝑋𝑋 𝑗𝑗𝑗𝑗 ′ 𝑗𝑗𝑗𝑗𝑗𝑗 𝑗𝑗𝑗𝑗 ′𝑚𝑚′ 𝑋𝑋 𝑗𝑗𝑗𝑗 𝑗𝑗𝑗𝑗𝑗𝑗 𝑗𝑗𝑗𝑗𝑗𝑗 ′ 𝑋𝑋 𝑗𝑗𝑗𝑗 𝑗𝑗 𝑢𝑢 𝑗𝑗𝑗𝑗𝑗𝑗 𝑗𝑗𝑗𝑗 ′𝑚𝑚′ � 𝑝𝑝 𝑝𝑝 𝑣𝑣 𝑣𝑣 𝜃𝜃 � 𝑝𝑝 𝑣𝑣 𝑣𝑣 � − 𝜃𝜃 � �′ 𝑝𝑝 𝑝𝑝 𝑣𝑣 𝑣𝑣 𝑢𝑢 𝑢𝑢′ 𝑢𝑢 𝑢𝑢 𝑢𝑢 and the expectation of is

𝐆𝐆𝑋𝑋

E[ ( , )] = 2 2 2 1 ′ ′ 𝒫𝒫 𝒫𝒫 , ′ ′ 𝑋𝑋 𝑚𝑚 𝑚𝑚 𝑗𝑗𝑗𝑗 𝑗𝑗𝑗𝑗𝑗𝑗 𝑗𝑗𝑗𝑗 𝑚𝑚 𝑋𝑋 𝑗𝑗𝑗𝑗 𝑗𝑗𝑗𝑗𝑗𝑗 𝑗𝑗𝑗𝑗𝑗𝑗 ′ 𝑋𝑋 𝑗𝑗𝑗𝑗 𝑗𝑗 𝑢𝑢 𝑗𝑗𝑗𝑗𝑗𝑗 𝑗𝑗𝑢𝑢 𝑚𝑚′ 𝐆𝐆 � �� 𝑝𝑝 𝑣𝑣 𝑣𝑣 − 𝜃𝜃 𝑝𝑝 𝑣𝑣 𝑣𝑣 − � − 𝜃𝜃 � �′ 𝑝𝑝 𝑝𝑝 𝑣𝑣 𝑣𝑣 � 𝑗𝑗 𝑢𝑢 𝑢𝑢 𝑢𝑢 = 2 1 = 1 . (Eq. A20) 𝒫𝒫 ′ , ′ ′ 𝒫𝒫 𝒜𝒜 𝑋𝑋 𝑗𝑗𝑗𝑗 𝑗𝑗𝑗𝑗𝑗𝑗 𝑗𝑗𝑗𝑗 𝑚𝑚 𝑗𝑗𝑗𝑗 𝑗𝑗 𝑢𝑢 𝑗𝑗𝑗𝑗𝑗𝑗 𝑗𝑗 𝑢𝑢 𝑚𝑚′ 𝑋𝑋 𝑚𝑚𝑚𝑚′ � − 𝜃𝜃 � � �� 𝑝𝑝 𝑣𝑣 𝑣𝑣 − �′ 𝑝𝑝 𝑝𝑝 𝑣𝑣 𝑣𝑣 � � − 𝜃𝜃 � 𝐺𝐺 𝑗𝑗 𝑢𝑢 𝑢𝑢 𝑢𝑢 We note in passing that can be technically defined also for a "population" consisting of a single individual. Denoting for individual i by , we have E[ ] = (1 ) . Thus, the expected amount of additive 𝐆𝐆 𝒜𝒜 𝐆𝐆 𝐆𝐆𝑖𝑖 𝐆𝐆𝑖𝑖 − 𝜃𝜃𝑖𝑖𝑖𝑖 𝐆𝐆 O. Ovaskainen et al. 5 SI variation present in a local population represented by a single non-inbred individual ( = 1/2) is half of the additive variation that is present in the ancestral population. 𝜃𝜃𝑖𝑖𝑖𝑖

Additive genetic variation among populations

The additive variance-covariance among the populations is a random variable over the process g, and we next compute its expectation. We have 𝐃𝐃 𝐹𝐹 1 1 Cov , = Cov[ , ] = 2 = 2 , (Eq. A21) 𝒫𝒫 𝒫𝒫 , , 𝒜𝒜 𝒫𝒫 𝒜𝒜 𝑋𝑋 𝑌𝑌 𝑖𝑖 𝑗𝑗 𝑖𝑖𝑖𝑖 𝑋𝑋𝑋𝑋 �𝒂𝒂 𝒂𝒂 � 𝑋𝑋 𝑋𝑋 � 𝒂𝒂 𝒂𝒂 𝑋𝑋 𝑌𝑌 � 𝜃𝜃 𝐆𝐆 𝜃𝜃 𝐆𝐆 𝑛𝑛 𝑛𝑛 𝑖𝑖∈𝑋𝑋 𝑗𝑗 ∈𝑌𝑌1 𝑛𝑛 𝑛𝑛 𝑖𝑖∈𝑋𝑋2𝑗𝑗 ∈𝑌𝑌 Cov , = Cov , = , (Eq. A22) 𝒫𝒫 𝒫𝒫 𝒫𝒫 𝒫𝒫 𝒫𝒫 𝒜𝒜 𝑋𝑋 𝑋𝑋 𝑌𝑌 𝑋𝑋𝑋𝑋 �𝒂𝒂 𝒂𝒂 � 𝒫𝒫 � �𝒂𝒂 𝒂𝒂 � 𝒫𝒫 � 𝜃𝜃 𝐆𝐆 1 𝑛𝑛 𝑌𝑌∈𝒫𝒫 2 𝑛𝑛 𝑌𝑌∈𝒫𝒫 Var = Cov[ , ] = 2 = 2 . (Eq. A23) 𝒫𝒫 𝒫𝒫 𝒫𝒫 , 𝒫𝒫 𝒜𝒜 𝒫𝒫 𝒜𝒜 𝑋𝑋 𝑋𝑋𝑋𝑋 �𝒂𝒂 � 𝒫𝒫 � 𝒂𝒂 𝒂𝒂 � 𝜃𝜃 𝐆𝐆 𝜃𝜃 𝐆𝐆 𝑛𝑛 𝑋𝑋∈𝒫𝒫 𝑛𝑛𝒫𝒫 𝑋𝑋 𝑌𝑌∈𝒫𝒫 As the expectation of additive value is the same for each individual (and thus for and for ), it holds that 𝒫𝒫 𝒫𝒫 1 1 𝒂𝒂4𝑋𝑋 𝒂𝒂 [ ] = Var 2Cov[ , ] + Var = 2 + 2 𝒫𝒫 𝒫𝒫 𝒫𝒫 𝒫𝒫 𝒫𝒫 𝒫𝒫 𝒫𝒫 𝒜𝒜 𝑋𝑋 𝑋𝑋 𝑋𝑋 𝑋𝑋𝑋𝑋 𝐸𝐸 𝐃𝐃 𝒫𝒫 �� �𝒂𝒂 � − 𝒂𝒂 𝒂𝒂 �𝒂𝒂 �� 𝒫𝒫 � � 𝜃𝜃 − 𝒫𝒫 � 𝜃𝜃 𝜃𝜃 � 𝐆𝐆 𝑛𝑛 𝑋𝑋∈𝒫𝒫 = 2 . (Eq. A24) 𝑛𝑛 𝑋𝑋∈𝒫𝒫 𝑛𝑛 𝑌𝑌∈𝒫𝒫 𝒫𝒫 𝒜𝒜 �𝜃𝜃 − 𝜃𝜃 �𝐆𝐆 Covariance structure of breeding experiment data

The four random processes ( g , g′, s, e) described in the main text are disjoint, so that each of them can be applied in isolation or in combination with the others. We denote expectations (and related variances and 𝐹𝐹 𝐹𝐹 𝐹𝐹 𝐹𝐹 covariances) over combinations of these random processes by listing the processes over which the expectations are taking in the subscript, e.g. Cov(s,e)[ ]. When an expectation is taken over some of the random processes while others are kept at a fixed realization, the result is conditional on the realizations kept ∙ fixed.

We assume that the environmental effects are independent between individuals, and are independent of additive genetic effects. This gives Cov(e) , = E, where E denotes the environmental covariance matrix. The definition in left-hand side is conditional on the realizations of the processes , , , but in this �𝒆𝒆𝑖𝑖 𝒆𝒆𝑗𝑗 � 𝛿𝛿𝑖𝑖𝑖𝑖 𝐕𝐕 𝐕𝐕 g g′ s case the result in the right-hand side of is independent of these realizations. Similarly, we have Cov , = 𝐹𝐹 𝐹𝐹 𝐹𝐹(g) Cov , = Cov , = 0. (g ) (s) �𝒂𝒂𝒊𝒊 𝒆𝒆𝒋𝒋�

′ �𝒂𝒂𝑖𝑖 𝒆𝒆𝑗𝑗 � �𝒂𝒂𝑖𝑖 𝒆𝒆𝑗𝑗 �

6 SI O. Ovaskainen et al.

To proceed, we need to introduce some more notation. We denote co-ancestry coefficients for individual- 1 1 population pairs as = , so that = . We denote by s( ) and d( ) the sire and the 𝒾𝒾𝒾𝒾 𝒫𝒫 𝒾𝒾𝒾𝒾 𝑖𝑖𝑖𝑖 𝑗𝑗 ∈𝑌𝑌 𝑖𝑖𝑖𝑖 ( ( 𝑖𝑖∈𝑋𝑋 𝑖𝑖𝑖𝑖 dam of the individual𝜃𝜃 i, and𝑛𝑛 𝑌𝑌recall∑ that𝜃𝜃 ) and𝜃𝜃 𝑋𝑋𝑋𝑋 ) denote𝑛𝑛𝑋𝑋 ∑ the𝜃𝜃 local populations from𝑖𝑖 which𝑖𝑖 the sire and the dam of the individual i originate. 𝑆𝑆 𝑖𝑖 𝐷𝐷 𝑖𝑖

As shown by Eq. A13, the covariance among genetic effects for a pair of individuals is given by

Cov(g,g ) , = 2 , where i and j refer to any individuals in the field populations or the laboratory population′ . Thus, utilizing𝒜𝒜 the recursive formula for coancestry coefficients, 2 = + , we obtain �𝒂𝒂𝑖𝑖 𝒂𝒂𝑗𝑗 � 𝜃𝜃𝑖𝑖𝑖𝑖 𝐆𝐆 ( ) ( ) for laboratory individuals i and j 𝜃𝜃𝑖𝑖𝑖𝑖 𝜃𝜃𝑠𝑠 𝑖𝑖 𝑗𝑗 𝜃𝜃𝑑𝑑 𝑖𝑖 𝑗𝑗

1 1 Cov g,g , = Cov g,g [ , ] + Cov g,g [ , ] 2 ( ) 2 ( ) ′ ( ) ′ ( ) ′ � � 𝑖𝑖 𝑗𝑗 � � 𝑖𝑖 𝑗𝑗′ � � 𝑖𝑖 𝑗𝑗′ �𝒂𝒂 𝒑𝒑 � 1𝑆𝑆 𝑗𝑗 � 𝒂𝒂 𝒂𝒂1 𝐷𝐷 𝑗𝑗 � 𝒂𝒂 𝒂𝒂 = 𝑛𝑛 𝑗𝑗′∈𝑆𝑆 𝑗𝑗 2 + 𝑛𝑛 2 𝑗𝑗′∈𝐷𝐷 𝑗𝑗 2 ( ) 2 ( ) ( ) 𝒜𝒜 ( ) 𝒜𝒜 𝑖𝑖𝑖𝑖′ 𝑖𝑖𝑖𝑖′ 1𝑆𝑆 𝑗𝑗 � 𝜃𝜃 𝐆𝐆 𝐷𝐷 𝑗𝑗 � 1𝜃𝜃 𝑮𝑮 𝑛𝑛 𝑗𝑗′∈𝑆𝑆 𝑗𝑗 𝑛𝑛 𝑗𝑗′∈𝐷𝐷 𝑗𝑗 = [ ( ) + ( ) ] + [ ( ) + ( ) ] 2 ( ) 2 ( ) ( ) ′ ′ 𝒜𝒜 ( ) ′ ′ 𝒜𝒜 𝑠𝑠 𝑖𝑖 𝑗𝑗 𝑑𝑑 𝑖𝑖 𝑗𝑗 𝑠𝑠 𝑖𝑖 𝑗𝑗 𝑑𝑑 𝑖𝑖 𝑗𝑗 𝑆𝑆 𝑗𝑗 � 𝜃𝜃 𝜃𝜃 𝐆𝐆 𝐷𝐷 𝑗𝑗 � 𝜃𝜃 𝜃𝜃 𝐆𝐆 𝑛𝑛 𝑗𝑗′∈𝑆𝑆 𝑗𝑗 𝑛𝑛 𝑗𝑗′∈𝐷𝐷 𝑗𝑗 = + + + . (Eq. A25) 2𝒜𝒜 ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) 𝐆𝐆 𝒾𝒾𝒾𝒾 𝒾𝒾𝒾𝒾 𝒾𝒾𝒾𝒾 𝒾𝒾𝒾𝒾 �𝜃𝜃𝑠𝑠 𝑖𝑖 𝑆𝑆 𝑗𝑗 𝜃𝜃𝑑𝑑 𝑖𝑖 𝑆𝑆 𝑗𝑗 𝜃𝜃𝑠𝑠 𝑖𝑖 𝐷𝐷 𝑗𝑗 𝜃𝜃𝑑𝑑 𝑖𝑖 𝐷𝐷 𝑗𝑗 � For the field individual i’ and lab-individual j, it holds that

1 1 Cov g,g , = Cov g,g [ , ] + Cov g,g [ , ] 2 ( ) 2 ( ) ′ ( ) ′ ( ) ′ � � 𝑖𝑖′ 𝑗𝑗 � � 𝑖𝑖′ 𝑗𝑗′ � � 𝑖𝑖′ 𝑗𝑗′ �𝒂𝒂 𝒑𝒑 � 1 𝑆𝑆 𝑗𝑗 � 𝒂𝒂1 𝒂𝒂 𝐷𝐷 𝑗𝑗 � 𝒂𝒂 𝒂𝒂 𝑛𝑛 𝑗𝑗′∈𝑆𝑆 𝑗𝑗 𝑛𝑛 𝑗𝑗′∈𝐷𝐷 𝑗𝑗 = + = ( ) + ( ) . (Eq. A26) ( ) ( ) ( ) 𝒜𝒜 ( ) 𝒜𝒜 𝒾𝒾′𝒾𝒾 𝒾𝒾′𝒾𝒾 𝒜𝒜 𝑖𝑖′𝑗𝑗′ 𝑖𝑖′𝑗𝑗′ 𝑖𝑖 𝑆𝑆 𝑗𝑗 𝑖𝑖 𝐷𝐷 𝑗𝑗 𝑆𝑆 𝑗𝑗 � 𝜃𝜃 𝐆𝐆 𝐷𝐷 𝑗𝑗 � 𝜃𝜃 𝐆𝐆 �𝜃𝜃 𝜃𝜃 � 𝐆𝐆 𝑛𝑛 𝑗𝑗′∈𝑆𝑆 𝑗𝑗 𝑛𝑛 𝑗𝑗′∈𝐷𝐷 𝑗𝑗 Thus, we obtain for the laboratory individuals i and j

1 1 Cov g,g , = Cov g,g , + Cov g,g , 2 ( ) 2 ( ) ′ ( ) ′ ′ ( ) ′ ′ � � 𝑖𝑖 𝑗𝑗 � � 𝑖𝑖 𝑗𝑗 � � 𝑖𝑖 𝑗𝑗 �𝒑𝒑 𝒑𝒑 � 1𝑆𝑆 𝑖𝑖 ′� �𝒂𝒂 𝒑𝒑 � 𝐷𝐷 𝑖𝑖 1′ � �𝒂𝒂 𝒑𝒑 � 𝑛𝑛 𝑖𝑖 ∈𝑆𝑆 𝑖𝑖 𝑛𝑛 𝑖𝑖 ∈𝐷𝐷 𝑖𝑖 = ( ) + ( ) + ( ) + ( ) 2 ( ) 2 ( ) ( ) 𝒾𝒾′𝒾𝒾 𝒾𝒾′𝒾𝒾 𝒜𝒜 ( ) 𝒾𝒾′𝒾𝒾 𝒾𝒾′𝒾𝒾 𝒜𝒜 𝑖𝑖 𝑆𝑆 𝑗𝑗 𝑖𝑖 𝐷𝐷 𝑗𝑗 𝑖𝑖 𝑆𝑆 𝑗𝑗 𝑖𝑖 𝐷𝐷 𝑗𝑗 1 𝑆𝑆 𝑖𝑖 ′� �𝜃𝜃 𝜃𝜃 � 𝐆𝐆 𝐷𝐷 𝑖𝑖 ′ � �𝜃𝜃 𝜃𝜃 � 𝐆𝐆 = 𝑛𝑛 𝑖𝑖 ∈𝑆𝑆 𝑖𝑖+ + + 𝑛𝑛 𝑖𝑖 ∈𝐷𝐷 𝑖𝑖. (Eq. A27) 2 ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) 𝒫𝒫 𝒫𝒫 𝒫𝒫 𝒫𝒫 𝒜𝒜 �𝜃𝜃𝑆𝑆 𝑖𝑖 𝑆𝑆 𝑗𝑗 𝜃𝜃𝑆𝑆 𝑖𝑖 𝐷𝐷 𝑗𝑗 𝜃𝜃𝐷𝐷 𝑖𝑖 𝑆𝑆 𝑗𝑗 𝜃𝜃𝐷𝐷 𝑖𝑖 𝐷𝐷 𝑗𝑗 �𝐆𝐆 Combining the above, for lab-individuals i and j it further holds that

O. Ovaskainen et al. 7 SI

Cov g,g , = Cov g,g , Cov g,g , ′ 1 ′ ′ � ��𝒔𝒔𝑖𝑖 𝒑𝒑𝑗𝑗 �= � ��𝒂𝒂+𝑖𝑖 𝒑𝒑𝑗𝑗 � − +� ��𝒑𝒑𝑖𝑖 𝒑𝒑+𝑗𝑗 � 2 ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) 𝒾𝒾𝒾𝒾 𝒾𝒾𝒾𝒾 𝒾𝒾𝒾𝒾 𝒾𝒾𝒾𝒾 𝒜𝒜 1 𝑠𝑠 𝑖𝑖 𝑆𝑆 𝑗𝑗 𝑑𝑑 𝑖𝑖 𝑆𝑆 𝑗𝑗 𝑠𝑠 𝑖𝑖 𝐷𝐷 𝑗𝑗 𝑑𝑑 𝑖𝑖 𝐷𝐷 𝑗𝑗 �𝜃𝜃 + 𝜃𝜃 + 𝜃𝜃 + 𝜃𝜃 �𝐆𝐆 (Eq. A28) 2 ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) 𝒫𝒫 𝒫𝒫 𝒫𝒫 𝒫𝒫 𝒜𝒜 − �𝜃𝜃𝑆𝑆 𝑖𝑖 𝑆𝑆 𝑗𝑗 𝜃𝜃𝑆𝑆 𝑖𝑖 𝐷𝐷 𝑗𝑗 𝜃𝜃𝐷𝐷 𝑖𝑖 𝑆𝑆 𝑗𝑗 𝜃𝜃𝐷𝐷 𝑖𝑖 𝐷𝐷 𝑗𝑗 �𝐆𝐆 and that

Cov g,g , = Cov g,g , ′ = Cov ′ , Cov , Cov , + Cov , � ��𝒔𝒔𝑖𝑖 𝒔𝒔𝑗𝑗 � �g,g ��𝒂𝒂𝑖𝑖 − 𝒑𝒑𝑖𝑖 𝒂𝒂𝑗𝑗 − 𝒑𝒑g,𝑗𝑗g� g,g g,g 1 ′ 1 ′ ′ ′ = � ��𝒂𝒂𝑖𝑖 𝒂𝒂𝑗𝑗 � − +� ��𝒂𝒂𝑖𝑖 𝒑𝒑𝑗𝑗+� − � +��𝒑𝒑𝑖𝑖 𝒂𝒂𝑗𝑗 � � ��𝒑𝒑𝑖𝑖 𝒑𝒑𝑗𝑗 � 2 2 ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) 1 𝒜𝒜 𝒾𝒾𝒾𝒾 𝒾𝒾𝒾𝒾 𝒾𝒾𝒾𝒾 𝒾𝒾𝒾𝒾 𝒜𝒜 𝜃𝜃𝑖𝑖𝑖𝑖 𝐆𝐆 −+�𝜃𝜃𝑠𝑠 𝑖𝑖 𝑆𝑆 𝑗𝑗 + 𝜃𝜃𝑑𝑑 𝑖𝑖 𝑆𝑆 𝑗𝑗 + 𝜃𝜃𝑠𝑠 𝑖𝑖 𝐷𝐷 𝑗𝑗 𝜃𝜃𝑑𝑑 𝑖𝑖 𝐷𝐷 𝑗𝑗 �𝐆𝐆 2 ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) 𝒾𝒾𝒾𝒾 𝒾𝒾𝒾𝒾 𝒾𝒾𝒾𝒾 𝒾𝒾𝒾𝒾 𝒜𝒜 1 𝑠𝑠 𝑗𝑗 𝑆𝑆 𝑖𝑖 𝑑𝑑 𝑗𝑗 𝑆𝑆 𝑖𝑖 𝑠𝑠 𝑗𝑗 𝐷𝐷 𝑖𝑖 𝑑𝑑 𝑗𝑗 𝐷𝐷 𝑖𝑖 +− �𝜃𝜃 + 𝜃𝜃 + 𝜃𝜃 +𝜃𝜃 �𝐆𝐆 . (Eq. A29) 2 ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) 𝒫𝒫 𝒫𝒫 𝒫𝒫 𝒫𝒫 𝒜𝒜 �𝜃𝜃𝑆𝑆 𝑖𝑖 𝑆𝑆 𝑗𝑗 𝜃𝜃𝑆𝑆 𝑖𝑖 𝐷𝐷 𝑗𝑗 𝜃𝜃𝐷𝐷 𝑖𝑖 𝑆𝑆 𝑗𝑗 𝜃𝜃𝐷𝐷 𝑖𝑖 𝐷𝐷 𝑗𝑗 �𝐆𝐆 We next treat the randomness associated to sampling individuals from the field populations to form the parental generation for the laboratory population. As we assume that the individuals used in the breeding design are a random sample from the field populations, we obtain for laboratory individuals i and j 1 1 E( ) ( ) ( ) = E( ) ( ) ′ = ′ ′ = ( ) ( ) (Eq. A30) ( ) ( ) ( ) 𝒾𝒾𝒾𝒾 ( ) ( ), ( ) 𝒫𝒫 𝑠𝑠 𝑠𝑠 𝑖𝑖 𝑆𝑆 𝑗𝑗 𝑠𝑠 𝑠𝑠 𝑖𝑖 𝑗𝑗 𝑖𝑖 𝑗𝑗 𝑆𝑆 𝑖𝑖 𝑆𝑆 𝑗𝑗 �𝜃𝜃 � � 𝑆𝑆 𝑗𝑗 � 𝜃𝜃 � 𝑆𝑆 𝑖𝑖 𝑆𝑆 𝑗𝑗 � 𝜃𝜃 𝜃𝜃 𝑛𝑛 𝑗𝑗′∈𝑆𝑆 𝑗𝑗 𝑛𝑛 𝑛𝑛 𝑖𝑖′∈𝑆𝑆 𝑖𝑖 𝑗𝑗′∈𝑆𝑆 𝑗𝑗 Similarly,

E( ) ( ) ( ) = ( ) ( ), E( ) ( ) ( ) = ( ) ( ), E( ) ( ) ( ) = ( ) ( ). (Eq. A31) 𝒾𝒾𝒾𝒾 𝒫𝒫 𝒾𝒾𝒾𝒾 𝒫𝒫 𝒾𝒾𝒾𝒾 𝒫𝒫 𝑠𝑠 𝑠𝑠 𝑖𝑖 𝐷𝐷 𝑗𝑗 𝑆𝑆 𝑖𝑖 𝐷𝐷 𝑗𝑗 𝑠𝑠 𝑑𝑑 𝑖𝑖 𝑆𝑆 𝑗𝑗 𝐷𝐷 𝑖𝑖 𝑆𝑆 𝑗𝑗 𝑠𝑠 𝑑𝑑 𝑖𝑖 𝐷𝐷 𝑗𝑗 𝐷𝐷 𝑖𝑖 𝐷𝐷 𝑗𝑗 The expected�𝜃𝜃 sire-�dam𝜃𝜃 and dam-sire coancestry�𝜃𝜃 coefficients� 𝜃𝜃 are given �by𝜃𝜃 � 𝜃𝜃

E( ) ( ) ( ) = ( ) ( ), E( ) ( ) ( ) = ( ) ( ). (Eq. A32) 𝒫𝒫 𝒫𝒫 𝑠𝑠 𝑠𝑠 𝑖𝑖 𝑑𝑑 𝑗𝑗 𝑆𝑆 𝑖𝑖 𝐷𝐷 𝑗𝑗 𝑠𝑠 𝑑𝑑 𝑖𝑖 𝑠𝑠 𝑗𝑗 𝐷𝐷 𝑖𝑖 𝑆𝑆 𝑗𝑗 In case of sire-sire (or dam�𝜃𝜃 -dam)� combinations,𝜃𝜃 the two�𝜃𝜃 sires (dams)� 𝜃𝜃 may represent the same individual or different individuals. Whether they represent the same or different individuals is part of the study design and thus independent of the sampling process, giving

( ) ( ) ( ) ( ) E( ) ( ) ( ) = (Eq. A33) 𝒫𝒫 ( ) = ( ), 𝜃𝜃𝑆𝑆(𝑖𝑖)𝑆𝑆 𝑗𝑗 𝑠𝑠 𝑖𝑖 ≠ 𝑠𝑠 𝑗𝑗 and 𝑠𝑠 �𝜃𝜃𝑠𝑠 𝑖𝑖 𝑠𝑠 𝑗𝑗 � � 𝒮𝒮 𝜃𝜃𝑆𝑆 𝑖𝑖 𝑠𝑠 𝑖𝑖 𝑠𝑠 𝑗𝑗 ( ) ( ) ( ) ( ) E( ) ( ) ( ) = (Eq. A34) 𝒫𝒫 ( ) = ( ). 𝜃𝜃𝐷𝐷(𝑖𝑖)𝐷𝐷 𝑗𝑗 𝑑𝑑 𝑖𝑖 ≠ 𝑑𝑑 𝑗𝑗 𝑠𝑠 𝑑𝑑 𝑖𝑖 𝑑𝑑 𝑗𝑗 As �𝜃𝜃 � � 𝒮𝒮 𝜃𝜃𝐷𝐷 𝑖𝑖 𝑑𝑑 𝑖𝑖 𝑑𝑑 𝑗𝑗

8 SI O. Ovaskainen et al.

1 1 + , = 2 2 ( ) ( ) = (Eq. A35) 1 𝑠𝑠 𝑖𝑖 𝑑𝑑 𝑗𝑗 ( ) ( ) + ( ) ( )𝜃𝜃+ ( ) ( )𝑖𝑖+ 𝑗𝑗 ( ) ( ) , , 𝜃𝜃𝑖𝑖𝑖𝑖 �4 �𝜃𝜃𝑠𝑠 𝑖𝑖 𝑠𝑠 𝑗𝑗 𝜃𝜃𝑠𝑠 𝑖𝑖 𝑑𝑑 𝑗𝑗 𝜃𝜃𝑑𝑑 𝑖𝑖 𝑠𝑠 𝑗𝑗 𝜃𝜃𝑑𝑑 𝑖𝑖 𝑑𝑑 𝑗𝑗 � 𝑖𝑖 ≠ 𝑗𝑗 we obtain

1 1 + = 2 2 ( ) ( ) 1 𝒫𝒫 ⎧ +𝜃𝜃𝑆𝑆 𝑖𝑖 𝐷𝐷 𝑖𝑖 + + 𝑖𝑖 𝑗𝑗 , ( ) ( ), ( ) ( ) 4 ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ⎪ 𝒫𝒫 𝒫𝒫 𝒫𝒫 𝒫𝒫 ⎪1 𝑆𝑆 𝑖𝑖 𝑆𝑆 𝑗𝑗 𝑆𝑆 𝑖𝑖 𝐷𝐷 𝑗𝑗 𝐷𝐷 𝑖𝑖 𝑆𝑆 𝑗𝑗 𝐷𝐷 𝑖𝑖 𝐷𝐷 𝑗𝑗 E( )[ ] = ⎪ �𝜃𝜃 ( ) + ( 𝜃𝜃) ( ) + ( 𝜃𝜃) ( ) + ( 𝜃𝜃) ( ) � 𝑖𝑖 ≠ 𝑗𝑗,𝑠𝑠(𝑖𝑖) ≠= 𝑠𝑠(𝑗𝑗),𝑑𝑑(𝑖𝑖) ≠ 𝑑𝑑(𝑗𝑗) (Eq. A36) ⎪4 𝒮𝒮 𝒫𝒫 𝒫𝒫 𝒫𝒫 𝑠𝑠 𝑖𝑖𝑖𝑖 1 𝑆𝑆 𝑖𝑖 𝑆𝑆 𝑖𝑖 𝐷𝐷 𝑗𝑗 𝐷𝐷 𝑖𝑖 𝑆𝑆 𝑗𝑗 𝐷𝐷 𝑖𝑖 𝐷𝐷 𝑗𝑗 𝜃𝜃 �𝜃𝜃 𝜃𝜃+ 𝜃𝜃+ 𝜃𝜃+ � 𝑖𝑖 ≠ 𝑗𝑗, 𝑠𝑠(𝑖𝑖) 𝑠𝑠(𝑗𝑗), 𝑑𝑑(𝑖𝑖) =≠ 𝑑𝑑(𝑗𝑗) ⎨4 ( ) ( ) ( ) ( ) ( ) ( ) ( ) 𝒫𝒫 𝒫𝒫 𝒫𝒫 𝒮𝒮 ⎪1 𝑆𝑆 𝑖𝑖 𝑆𝑆 𝑗𝑗 𝑆𝑆 𝑖𝑖 𝐷𝐷 𝑗𝑗 𝐷𝐷 𝑖𝑖 𝑆𝑆 𝑗𝑗 𝐷𝐷 𝑖𝑖 ⎪ �𝜃𝜃 ( ) + ( 𝜃𝜃) ( ) + (𝜃𝜃) ( ) + (𝜃𝜃) � 𝑖𝑖 ≠ 𝑗𝑗,𝑠𝑠(𝑖𝑖) ≠= 𝑠𝑠(𝑗𝑗),𝑑𝑑(𝑖𝑖) = 𝑑𝑑(𝑗𝑗) ⎪4 𝒮𝒮 𝒫𝒫 𝒫𝒫 𝒮𝒮 ⎪ 𝑆𝑆 𝑖𝑖 𝑆𝑆 𝑖𝑖 𝐷𝐷 𝑗𝑗 𝐷𝐷 𝑖𝑖 𝑆𝑆 𝑗𝑗 𝐷𝐷 𝑖𝑖 ⎩ �𝜃𝜃 𝜃𝜃 𝜃𝜃 𝜃𝜃 � 𝑖𝑖 ≠ 𝑗𝑗 𝑠𝑠 𝑖𝑖 𝑠𝑠 𝑗𝑗 𝑑𝑑 𝑖𝑖 𝑑𝑑 𝑗𝑗 We next are now ready to extend the covariances over (g, g ) to covariances over (g, g , s). Since the sampling ′ ′ process s is independent of the processes g and g′, it holds for any random variables and ,

𝐹𝐹 𝐹𝐹 𝐹𝐹 𝐴𝐴 𝐵𝐵

Cov g,g ,s [ , ] = E g,g ,s [ ] E g,g ,s [ ]E g,g ,s [ ] ′ ′ ′ ′ � � 𝐴𝐴 𝐵𝐵 = E(�s) E �g,𝐴𝐴g 𝐴𝐴[ −] � E g�,g 𝐴𝐴[ �]E g,g� 𝐵𝐵[ ] + E(s) E g,g [ ]E g,g [ ] E g,g ,s [ ]E g,g ,s [ ] ′ ′ ′ ′ ′ ′ ′ = E(s) �Cov� g�,g𝐴𝐴𝐴𝐴( ,−) �+ E�(s𝐴𝐴) E� g,g � [𝐵𝐵]�E g,g [ �] � E� g𝐴𝐴,g ,s� [ ]�E𝐵𝐵g,g� −,s [ �] (Eq� 𝐴𝐴. A37� ) � 𝐵𝐵 ′ ′ ′ ′ ′ � � � 𝐴𝐴 𝐵𝐵 � � � � 𝐴𝐴 � � 𝐵𝐵 � − � � 𝐴𝐴 � � 𝐵𝐵 For any realization of the process s and any laboratory individual i,

𝐹𝐹

E g,g [ ] = E g,g [ ] = E g,g [ ] = 0, (Eq. A38) ′ ′ ′ � � 𝑖𝑖 � � 𝑖𝑖 � � 𝑖𝑖 and thus, for any laboratory individuals𝒂𝒂 i and j, 𝒑𝒑 𝒔𝒔

1 Cov , = E Cov , = + + + , (Eq. A39) g,g ,s (s) g,g 2 ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ′ ′ 𝒫𝒫 𝒫𝒫 𝒫𝒫 𝒫𝒫 𝒜𝒜 � ��𝒑𝒑𝑖𝑖 𝒑𝒑𝑗𝑗 � � � ��𝒑𝒑𝑖𝑖 𝒑𝒑𝑗𝑗 �� �𝜃𝜃𝑆𝑆 𝑖𝑖 𝑆𝑆 𝑗𝑗 𝜃𝜃𝑆𝑆 𝑖𝑖 𝐷𝐷 𝑗𝑗 𝜃𝜃𝐷𝐷 𝑖𝑖 𝑆𝑆 𝑗𝑗 𝜃𝜃𝐷𝐷 𝑖𝑖 𝐷𝐷 𝑗𝑗 �𝐆𝐆 Cov g,g ,s , = E(s) Cov g,g , = E(s) ( ) + ( ) ′ 1 ′ 𝒾𝒾𝒾𝒾 𝒾𝒾𝒾𝒾 𝒜𝒜 � ��𝒂𝒂𝑖𝑖 𝒑𝒑𝑗𝑗=� � �+ ��𝒂𝒂𝑖𝑖 𝒑𝒑𝑗𝑗+�� �𝜃𝜃+𝑖𝑖𝑖𝑖 𝑗𝑗 𝜃𝜃𝑖𝑖𝑖𝑖 𝑗𝑗 �𝐆𝐆 = Cov , , (Eq. A40) 2 ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) g,g ,s 𝒫𝒫 𝒫𝒫 𝒫𝒫 𝒫𝒫 𝒜𝒜 ′ �𝜃𝜃𝑆𝑆 𝑖𝑖 𝑆𝑆 𝑗𝑗 𝜃𝜃𝑆𝑆 𝑖𝑖 𝐷𝐷 𝑗𝑗 𝜃𝜃𝐷𝐷 𝑖𝑖 𝑆𝑆 𝑗𝑗 𝜃𝜃𝐷𝐷 𝑖𝑖 𝐷𝐷 𝑗𝑗 �𝐆𝐆 � ��𝒑𝒑𝑖𝑖 𝒑𝒑𝑗𝑗 � Cov g,g ,s , = Cov g,g ,s , = 0, (Eq. A41) ′ ′ � ��𝒔𝒔𝑖𝑖 𝒑𝒑𝑗𝑗 � � ��𝒂𝒂𝑖𝑖 − 𝒑𝒑𝑖𝑖 𝒑𝒑𝑗𝑗 � Cov g,g ,s , = E(s) Cov g,g , = 2E(s) ij , (Eq. A42) ′ ′ 𝒜𝒜 � ��𝒂𝒂𝑖𝑖 𝒂𝒂𝑗𝑗 � � � ��𝒂𝒂𝑖𝑖 𝒂𝒂𝑗𝑗 �� �𝜃𝜃 �𝐆𝐆

O. Ovaskainen et al. 9 SI and thus

Cov g,g ,s , = Cov g,g ,s , = Cov g,g ,s , Cov g,g ,s , ′ = 2 ′ (Eq. A43) ′ ′ � ��𝒔𝒔𝑖𝑖 𝒔𝒔𝑗𝑗 � � ��𝒂𝒂𝑖𝑖 − 𝒑𝒑𝑖𝑖 𝒂𝒂𝑗𝑗 − 𝒑𝒑𝑗𝑗 � � ��𝒂𝒂𝑖𝑖 𝒂𝒂𝑗𝑗 � − � ��𝒑𝒑𝑖𝑖 𝒑𝒑𝑗𝑗 � 𝒜𝒜 𝑓𝑓𝑖𝑖𝑖𝑖 𝐆𝐆 where 1 1 + = 2 4 ( ) ( ) ( ) ( ) 0 𝒫𝒫 𝒫𝒫 , ( ) ( ), ( ) ( ) ⎧ − �𝜃𝜃𝑆𝑆 𝑖𝑖 𝑆𝑆 𝑖𝑖 𝜃𝜃𝐷𝐷 𝑖𝑖 𝐷𝐷 𝑖𝑖 � 𝑖𝑖 𝑗𝑗 1 ⎪ , ( ) = ( ), ( ) ( ) = ⎪ 4 ( ) ( ) ( ) 𝑖𝑖 ≠ 𝑗𝑗 𝑠𝑠 𝑖𝑖 ≠ 𝑠𝑠 𝑗𝑗 𝑑𝑑 𝑖𝑖 ≠ 𝑑𝑑 𝑗𝑗 (Eq. A44) ⎪ 𝒮𝒮 𝒫𝒫 1 𝑆𝑆 𝑖𝑖 𝑆𝑆 𝑖𝑖 𝑆𝑆 𝑖𝑖 𝑖𝑖𝑖𝑖 �𝜃𝜃 − 𝜃𝜃 � 𝑖𝑖 ≠ 𝑗𝑗 ,𝑠𝑠 (𝑖𝑖 ) 𝑠𝑠 (𝑗𝑗 ),𝑑𝑑 (𝑖𝑖 )≠=𝑑𝑑 (𝑗𝑗 ) 𝑓𝑓 4 ( ) ( ) ( ) ⎨1 𝒮𝒮 𝒫𝒫 ⎪ �𝜃𝜃𝐷𝐷+𝑖𝑖 − 𝜃𝜃𝐷𝐷 𝑖𝑖 𝐷𝐷 𝑖𝑖 � 𝑖𝑖 ≠ 𝑗𝑗,𝑠𝑠(𝑖𝑖) ≠= 𝑠𝑠(𝑗𝑗),𝑑𝑑(𝑖𝑖) = 𝑑𝑑(𝑗𝑗) ⎪4 ( ) ( ) ( ) ( ) ( ) ( ) 𝒮𝒮 𝒮𝒮 𝒫𝒫 𝒫𝒫 ⎪ 𝑆𝑆 𝑖𝑖 𝐷𝐷 𝑖𝑖 𝑆𝑆 𝑖𝑖 𝑆𝑆 𝑖𝑖 𝐷𝐷 𝑖𝑖 𝐷𝐷 𝑖𝑖 ⎩ �𝜃𝜃 𝜃𝜃 − 𝜃𝜃 − 𝜃𝜃 � 𝑖𝑖 ≠ 𝑗𝑗 𝑠𝑠 𝑖𝑖 𝑠𝑠 𝑗𝑗 𝑑𝑑 𝑖𝑖 𝑑𝑑 𝑗𝑗

References

COCKERHAM, C. C., and B. S. WEIR, 1987 Correlations, descent measures - drift with migration and mutation. Proceedings of the National Academy of Sciences of the United States of America 84: 8512-8514.

10 SI O. Ovaskainen et al.

File S2

Parameter estimation

Given neutral molecular data and the breeding experiment data, we fitted a neutral model to the data using Bayesian inference, and then tested for deviation from neutrality using the approach described in the main paper. Here we describe the statistical model, the prior distributions and a Markov Chain Monte Carlo (MCMC) method that was used to parameterize the model.

The statistical model for neutral molecular markers

We model the neutral divergence of the local populations from a common ancestral population as a mixture of independent lineages. The model summarized in this section will be published later in 𝑛𝑛𝒫𝒫 more detail (KARHUNEN and OVASKAINEN 2011). 𝑛𝑛ℒ The allele frequencies in locus in lineage are distributed as

𝑗𝑗 𝑘𝑘~Dirichlet .

𝑘𝑘𝑘𝑘 𝑘𝑘 𝑗𝑗 where denotes the vector of allele frequencies𝒛𝒛 in the�𝑎𝑎 ancestral𝒒𝒒 � population, and measures the amount of drift experienced by lineage . The allele frequencies in locus in local population are 𝒒𝒒𝑗𝑗 𝑎𝑎𝑘𝑘 defined as a mixture of the lineage-specific frequencies, 𝑘𝑘 𝑗𝑗 𝐴𝐴 = . 𝐿𝐿 𝑛𝑛=1 𝐴𝐴 𝐴𝐴 𝑗𝑗 𝑘𝑘 𝑘𝑘𝑘𝑘 𝒑𝒑 �𝑘𝑘 𝜅𝜅 𝒛𝒛 Here, we assume that the mixture weights sum up to unity over the lineages, =1 = 1, so 𝐿𝐿 that vector is a proper frequency distribution.𝐴𝐴 The genotype (in terms of the neutral𝑛𝑛 marker𝐴𝐴 loci) 𝜅𝜅𝑘𝑘 ∑𝑘𝑘 𝜅𝜅𝑘𝑘 of each individual𝐴𝐴 in a population is a multinomial random variable of the allele frequencies, 𝒑𝒑𝑗𝑗 ~Multinomial 2, . . 𝐴𝐴 𝐴𝐴 𝑖𝑖𝑖𝑖 𝑗𝑗 𝑥𝑥The population-population� 𝒑𝒑 � coancestry coefficients depend on the model parameters as

= . 𝐿𝐿 𝐴𝐴 𝐵𝐵 𝑛𝑛=1 + 1 𝒫𝒫 𝜅𝜅𝑘𝑘 𝜅𝜅𝑘𝑘 𝜃𝜃𝐴𝐴𝐴𝐴 � Note that is defined through probability of IBD𝑘𝑘 for 𝑎𝑎neutral𝑘𝑘 loci, and thus, it does not depend on allele frequencies𝒫𝒫 in the ancestral generation. 𝜃𝜃𝐴𝐴𝐴𝐴 𝒒𝒒𝑗𝑗

O. Ovaskainen et al. 11 SI

The Directed Acyclic Graph (DAG) that describes the dependencies among model variables (including both neutral marker data and quantitative trait data) is shown in Supplementary Figure 1. While the framework develop in the paper and presented in the DAG allows for the inclusion of an arbitrary level of inbreeding ( ), we did not account for inbreeding in the present estimation scheme and thus 𝒮𝒮 assumed = 0.5(1 𝑋𝑋+ ). 𝒮𝒮 𝜃𝜃 𝒫𝒫 𝜃𝜃𝑋𝑋 𝜃𝜃𝑋𝑋 Prior distributions

To parameterize the model with Bayesian inference, prior distributions need to be defined for the primary parameters of the model: , , E , , and . We assume the priors 𝒜𝒜 𝝁𝝁 𝐆𝐆 𝐕𝐕 𝒒𝒒~𝒂𝒂N(0,100𝜿𝜿 ),

~Wishart𝜇𝜇𝑚𝑚( ( + 1) 1, + 1), 𝒜𝒜 − 𝑚𝑚 1 𝐆𝐆E~Wishart( 𝐈𝐈 ( 𝑚𝑚+ 1) , 𝑚𝑚+ 1), − 𝐕𝐕 ~Dirichlet𝐈𝐈𝑚𝑚 𝑚𝑚 , 𝑚𝑚 𝑞𝑞 𝑗𝑗 𝒒𝒒log ~N( �, 𝜷𝜷2𝑗𝑗 )�,

~𝑎𝑎Dirichlet𝑘𝑘 𝜇𝜇𝑎𝑎(𝜎𝜎𝑎𝑎 ). 𝜅𝜅 Here the indices , , and refer to loci, lineages𝜿𝜿𝐴𝐴 and subpopulations,𝜷𝜷𝐴𝐴 respectively. In the numerical examples of this paper, we assume the values = , = 1, 2 = 2. We set the number of 𝑗𝑗 𝑘𝑘 𝐴𝐴 𝑞𝑞 lineages equal to the number of populations, and assume𝑗𝑗 that lineage makes the dominant 𝜷𝜷𝑗𝑗 𝟏𝟏𝑛𝑛 𝜇𝜇𝑎𝑎 𝜎𝜎𝑎𝑎 contribution to population , i.e. that the matrix is diagonally dominant. To do so, we let 𝐴𝐴 𝐴𝐴 𝜿𝜿 0.2 = 0.8, and = for , 1 𝜅𝜅 𝜅𝜅 𝛽𝛽𝐴𝐴𝐴𝐴 𝛽𝛽𝐴𝐴𝐴𝐴 𝑘𝑘 ≠ 𝐴𝐴 and truncate the prior by the requirement that >𝑛𝑛𝐿𝐿 − for all . The latter specification ensures that label switching is not possible, which𝜅𝜅 property𝜅𝜅 improves the mixing of Markov Chain 𝛽𝛽𝐴𝐴𝐴𝐴 𝛽𝛽𝐴𝐴𝐴𝐴 𝑘𝑘 ≠ 𝐴𝐴 Monter Carlo (MCMC) algorithm (GELMAN et al. 2004).

The number of alleles in neutral marker locus in the ancestral generation is generally unknown, as some alleles may have disappeared after the lineages have diverged or are not present in the sampled 𝑛𝑛𝑗𝑗 𝑗𝑗 individuals. Due to mathematical properties of the Dirichlet distribution, we may pool all such unobserved alleles into one ‘meta-allele’. Thus, we define as the number of distinct alleles observed in locus plus one. 𝑛𝑛𝑗𝑗 𝑗𝑗 12 SI O. Ovaskainen et al.

Parameter estimation through MCMC

Each model parameter was sampled from its full conditional while keeping the other parameters fixed. Most parameters were updated using a random-walk Metropolis-Hastings (MH) algorithm, the proposal distributions given below. For these the variances of the proposal distributions were adjusted during the burn-in following Ovaskainen et al. (2008) to give an accept ratio of 0.44 for a single parameter or 0.22 for a vector of parameters.

• Sampling the additive genetic variance-covariance matrix was conducted using the MH algorithm. We used the proposal distribution Wishart( /𝒜𝒜, ), where the parameter was 𝐆𝐆 adjusted during the burn-in. 𝒜𝒜 𝐆𝐆 ν ν ν

• Sampling the residual variance-covariance matrix E was conducted using the MH algorithm. We used the proposal distribution Wishart( E / , ), where the parameter was adjusted during the 𝐕𝐕 burn-in. 𝐕𝐕 ν ν ν

• Sampling the overall mean and the population specific means . Conditional on the other parameters, the full conditional joint distribution of the parameters𝒫𝒫 ( , ) follows a multivariate 𝝁𝝁 𝐀𝐀 normal distribution. These parameters were thus updated directly (not using𝒫𝒫 the MH algorithm) 𝝁𝝁 𝐀𝐀 by drawing a random deviate from the full conditional.

• Sampling drift parameters . We used N( , 2 ) distributions separately for each to draw 2 proposals for log . The variance parameters 𝑘𝑘 were adjusted during the burn-in. 𝒂𝒂 𝑎𝑎𝑘𝑘 𝛿𝛿𝑎𝑎 𝑘𝑘 𝑘𝑘 𝑎𝑎𝑘𝑘 𝛿𝛿𝑎𝑎 • Sampling lineage loadings . We used truncated Dirichlet( ) distributions separately for

each and to draw proposals for . The ’s are proposal𝐴𝐴 parameters that were adjusted 𝜿𝜿 𝛿𝛿𝜿𝜿 𝜿𝜿𝐴𝐴 during the burn-in. 𝐴𝐴 𝐴𝐴 𝑗𝑗 𝜿𝜿𝐴𝐴 𝛿𝛿𝜿𝜿

• Sampling ancestral allele frequencies . We used truncated Dirichlet( ) distributions

separately for each to draw proposals for . The ’s are proposal parameters𝑗𝑗 that were 𝒒𝒒 𝛿𝛿𝑞𝑞 𝒒𝒒𝑗𝑗 adjusted during the burn-in. 𝑗𝑗 𝑗𝑗 𝒒𝒒𝑗𝑗 𝛿𝛿𝑞𝑞

• Sampling allele frequencies . We used Dirichlet( ) distributions separately for each

lineage and locus . The ’s parameters were adjusted𝑘𝑘𝑘𝑘 during the burn-in. 𝒛𝒛 𝛿𝛿𝒛𝒛 𝒛𝒛𝑘𝑘𝑘𝑘 𝑘𝑘𝑘𝑘 𝑘𝑘 𝑗𝑗 𝛿𝛿𝒛𝒛

O. Ovaskainen et al. 13 SI

We implemented the algorithm in Mathematica 7.1 (WOLFRAM RESEARCH 2008), and used 1,000 iterations for the burn-in and 2,000 for sampling the posterior distribution. We note that these sample sizes are rather small, and they were chosen due to logistic constrains (the estimation was conducted for 3,600 data sets in total). We tested for the sufficiency for mixing by examining the chains for convergence and for running a sample of the datasets for 10,000 + 20,000 iterations. Both tests (as well as the Results, which separated well the cases with and without selection, see main text) indicated that the length of the MCMC chain was sufficient for the present purpose. For the case of real data analysis, we naturally recommend a much longer MCMC chain.

Supplementary Figure 1. Directed acyclic graph (DAG) for the Bayesian model. The quantities in the ellipses are the parameter values being estimated. In the single boxes are the observations (the neutral data and the quantitative trait data ) and in the double boxes are the fixed quantities. The full arrows give stochastic relationships (i.e. those that have probability distributions associated with 𝑿𝑿 𝒁𝒁 them) and the dashed arrows are for deterministic relationships.

14 SI O. Ovaskainen et al.

References ALBERT, A. Y. K., S. SAWAYA, T. H. VINES, A. K. KNECHT, C. T. MILLER et al., 2008 The genetics of adaptive shape shift in stickleback: Pleiotropy and effect size. Evolution 62: 76-85. GELMAN, A., J. B. CARLIN, H. S. STERN and D. B. RUBIN, 2004 Bayesian Data Analysis. Chapman & Hall/CRC, Boca Raton. KARHUNEN, M., and O. OVASKAINEN, 2011 Defining and estimating FST through coancestry. Manuscript. WOLFRAM RESEARCH, I., 2008 Mathematica Edition: Version 7.1. Wolfram Research, Inc., Champaign, Illinois.

O. Ovaskainen et al. 15 SI

File S3

Generation of individual-based data

We assumed that in each generation of each population the breeding population consists of 20 individuals. We assumed a hermaphrodite species and randomized the two parents for each propagule independently of each other (selfing allowed). To model gene flow among the populations, we assumed that the parent is an immigrant from one of the other populations with probability =0.01.

𝜀𝜀 We assume a set of 32 loci determining the genotypic values among these traits, with 5 allelic variants in each locus, each having an equal frequency in the ancestral generation. We randomized the additive values of each allele in each locus from a two-dimensional multivariate normal distribution with mean zero and such a variance-covariance matrix (see Appendix A) that the expected amount of additive genetic variation in the ancestral generation was 1.0 0.9 E[ ] = . 0.9 1.0 𝒜𝒜 𝐆𝐆 � � The phenotypic value of individual was modeled as the sum of the additive component and the environmental effect, i.e. = + , where the environmental effect was randomized from the 𝑖𝑖 two dimensional multivariate normal distribution with mean zero and variance-covariance matrix 𝐳𝐳𝑖𝑖 𝐚𝐚𝑖𝑖 𝐞𝐞𝑖𝑖 𝐞𝐞𝑖𝑖

0.5 0 = . 0 0.5 𝐕𝐕𝐸𝐸 � � If modeling traits under selection, we let the populations become adapted locally to their environments by assuming that the environment experienced by a population favored an optimal phenotype . We assumed that the values are distributed (i.i.d. among the populations) multinormally with mean zero 𝑋𝑋 𝒒𝒒𝑋𝑋 and variance covariance𝑋𝑋 matrix 𝒒𝒒 5 4.5 = , 4.5 5 − 𝐕𝐕𝑆𝑆 � � and we thus have assumed that the expected direction− of population divergence through the selective gradient is not proportional to . 𝒜𝒜 𝐆𝐆 We assume that the populations undergo weak selection, so that the initial number of propagules produced is high (ten times the actual population size), and selection operates through competition influencing survival. We define the fitness of an individual with phenotype in population as

𝐳𝐳 𝑗𝑗 ( ) = exp( ( ) 1( )) − 𝑊𝑊𝑗𝑗 𝐳𝐳 − 𝐳𝐳 − 𝒒𝒒𝑗𝑗 𝐕𝐕𝐹𝐹 𝐳𝐳 − 𝒒𝒒𝑗𝑗

16 SI O. Ovaskainen et al. where the variance-covariance matrix

0.5 0 = 0 0.5 𝐕𝐕𝐹𝐹 � � describes the shape of the fitness landscape around the local optimum. We selected the 20 individuals that survived to form next generation among 200 propagules by sampling each individual with a probability that was proportional to its fitness value. In the case of no selection, the weights were set equal for all individuals.

To generate phenotypic data that allows for the estimation of genetic variability among and between populations, we assumed a breeding design in which five pairs of individuals were randomly sampled from each of the eight populations. We assumed a full-sib design with five offspring for each of the pairs, so that the total number of offspring was 8 x 5 x 5 = 200. These individuals were assigned environmental effects as for the wild individuals, and their phenotypes were then recorded.

For neutral molecular markers, we also assumed a set of 32 loci with 5 alleles in each locus with equal frequencies in the ancestral population. We randomized the flow of the neutral alleles from the ancestral to the present generation, sampled ten individuals from each population, and genotyped these for the neutral molecular markers.

O. Ovaskainen et al. 17 SI

File S4

Implementation of the method of Martin et al. (2008)

We analyzed the simulated data sets with Martin et al.’s method (MARTIN et al. 2008b). This method involves two separate tests: Comparing the estimates of the intra-population G matrix and the covariance matrix of population means for proportionality, and secondly, testing the proportionality coefficient of these matrices against the FST of molecular markers. The first test should see selection that has a magnitude comparable to random drift, but a direction incompatible with . The second test should see selection that has a magnitude incompatible with random drift, whether balancing𝒜𝒜 or 𝐆𝐆 disruptive. Both tests are based on the assumption that

2FST E[ ] = E[ ] 1 FST 𝐃𝐃 𝐆𝐆𝑋𝑋 under neutrality. We performed these tests following− the procedure described by the authors (CHAPUIS et al. 2008; MARTIN et al. 2008b). Thus, we estimated the G matrices using R’s MANOVA with three covariance components: inter-population, inter-family and inter-individual. The inter-population covariance matrix was used as an estimate of E[ ], whereas the inter-population covariance matrix was multiplied by two to yield an estimate of E[ ], which is appropriate for full-sib study design (LYNCH 𝐃𝐃 and WALSH 1998). Because the phenotypic data was known to be multivariate normal, it was not Cox 𝐆𝐆𝑋𝑋 transformed, but we followed the recommendation of standardization (CHAPUIS et al. 2008).

After this, the proportionality of E[ ] and E[ ] was tested, and a 95 % confidence interval was derived for the proportionality coefficient using code provided by MARTIN et al. (2008a). The Bartlett p 𝐃𝐃 𝐆𝐆𝑋𝑋 value of the proportionality test was recorded. To perform the second test of MARTIN et al. (2008b), a 95 % confidence interval was estimated for the FST using bootstrap over loci and the Weir-Cockerham estimator (WEIR and COCKERHAM 1984) implemented in Fstat (GOUDET 1995). The confidence interval 2FST of FST was transformed into the confidence interval of . This was compared with the confidence 1 FST interval of the proportionality coefficient obtained from −the phenotypic data. If these intervals did not overlap, this was taken as a signal of selection.

References

CHAPUIS, E., G. MARTIN and J. GOUDET, 2008 Effects of Selection and Drift on G Matrix Evolution in a Heterogeneous Environment: A Multivariate Q(st)-F-st Test With the Freshwater Snail Galba truncatula. Genetics 180: 2151-2161. GOUDET, J., 1995 FSTAT (Version 1.2): A computer program to calculate F-statistics. Journal of Heredity 86: 485- 486.

18 SI O. Ovaskainen et al.

LYNCH, M., and B. WALSH, 1998 Genetics and analysis of quantitative traits. Sinauer Associates Incorporated, New York. MARTIN, G., E. CHAPUIS and J. GOUDET, 2008a http://www.isem.cnrs.fr/spip.php?article934, pp. Institut des sciences de l'evolution montpellier, Montpellier. MARTIN, G., E. CHAPUIS and J. GOUDET, 2008b Multivariate Q(st)-F-st Comparisons: A Neutrality Test for the Evolution of the G Matrix in Structured Populations. Genetics 180: 2135-2149. WEIR, B. S., and C. C. COCKERHAM, 1984 Estimating F-Statistics for the Analysis of Population-Structure. Evolution 38: 1358-1370.

O. Ovaskainen et al. 19 SI