Available online at www.sciencedirect.com Intelligence 36 (2008) 574–583 Investigating the ‘g’-saturation of various stratum-two factors using automatic item generation ⁎ Martin E. Arendasy , Andreas Hergovich, Markus Sommer University of Vienna, Department of Psychology, Psychometric Technology Group, Austria Received 9 December 2006; received in revised form 27 November 2007; accepted 27 November 2007 Available online 4 February 2008 Abstract Even though researchers agree on the hierarchical structure of intelligence there is considerable disagreement on the g-saturation of the lower stratum-two factors. In this article it is argued that the mixed evidence in the research literature can be at least partially attributed to the construct representation of the individual tests used to measure the stratum-two factors. In the study described here two top-down approaches to automatic item generation were used to build construct representation directly into the items. This enabled a clearer substantive interpretation of the shared commonalities extracted by the various stratum-two factors and helped to rule out alternative, but mathematically equivalent, structural models based on independent empirical evidence. This approach was used to investigate the g-saturation of five stratum-two factors using a sample of 240 male and female respondents. The results support the assumption that fluid intelligence exhibits the highest g-saturation. © 2007 Elsevier Inc. All rights reserved. Keywords: CHC models; g-saturation; Construct representation; Automated item generation 1. Theoretical background and confirmatory factor analyses of Wechsler-like test batteries. For instance, Gignac (2006) investigated Despite a growing consensus on a hierarchical struc- the g-saturation of the subtests included in different ture of human intelligence with g at the apex and several Wechsler-like test batteries. The author used a single- broader stratum-two factors, researchers still disagree on trait-correlated uniqueness confirmatory factor analysis the g-saturation of the broader stratum-two factors. and specified a g-factor in addition to correlated resid- Recently some researchers (cf. Gignac, 2006; Robin- uals between subtests assumed to measure either (1) son, 1999) have argued that general intelligence is crystallized intelligence (Gc), (2) visual processing (Gv), more closely associated with crystallized intelligence or (3) fluid intelligence (Gf) in order to control for their (Gc). This claim is based on the results of exploratory shared commonalities when estimating their g-satura- tion. The results indicated that the crystallized intel- ligence (Gc) subtests had the highest average factor loadings on the g-factor. This is in contrast to the results ⁎ Corresponding author. University of Vienna, Department of Psychology, Liebiggasse 5, A-1010 Vienna, Austria. Tel.: +43 1 obtained by Roberts, Goff, Anjoul, Kyllonen, Pallier 4277 47848. and Stankov (2000) as well as Gustafsson (1984; 2002; E-mail address: [email protected] (M.E. Arendasy). Gustafsson & Balke, 1993; Undheim & Gustafsson, 0160-2896/$ - see front matter © 2007 Elsevier Inc. All rights reserved. doi:10.1016/j.intell.2007.11.005 M.E. Arendasy et al. / Intelligence 36 (2008) 574–583 575 1987), who reported that fluid intelligence (Gf)wasvir- researchers need to resort to supplementary empirical tually indistinguishable from the higher-order g-factor in evidence in order to come up with a substantive inter- their studies. pretation of the results obtained. Usually this empirical The picture is further complicated by results obtained evidence is derived from prior factor analyses conducted by Bickley, Keith and Wolfe (1995), who investigated the by other researchers or expert ratings of the cognitive factorial structure of 16 subtests of the revised Wood- demands of the individual subtests (cf. McGrew, 1997). cock–Johnson battery (WJ-R) using a hierarchical con- Neither of these two approaches is without difficulties. firmatory factor analysis. These researchers reported an When resorting to prior factor analytical research it must almost equal g-saturation of fluid intelligence (Gf:.88), be borne in mind that the assignment of a subtest to a quantitative reasoning (Gq: .86) and crystallized intelli- certain stratum-two factor may change depending on the gence (Gc: .87). This suggests that all three stratum-two other subtests included in the analysis (cf. McGrew, factors are equal with regard to their g-saturation. This 1997). conclusion was further supported by the finding that the Expert ratings, on the other hand, depend largely on model fit decreased significantly once the standardized the expertise of the raters and involve a certain amount of factor loadings of the three stratum-two factors on ‘g’ subjectivity in comparison to other methodological were set to 1. These results were also replicated by Carroll approaches. Furthermore, expert-novice research indi- (2003), who used a more extended set of variables taken cates that experts sometimes either disagree on the cog- from the WJ-R norm sample. nitive processes respondents use to solve a given item Taken together, the literature provides rather mixed set, or even fail to accurately identify them. This phe- evidence with regard to the g-factor saturation of the nomenon is commonly referred to as the experts' blind stratum-two factors fluid intelligence (Gf), crystallized spot (cf. Nathan & Petrosino, 2003). This problem is intelligence (Gc) and quantitative reasoning (Gq). In the highlighted in Ashton and Lee's (2006) criticism of the following section we discuss several reasons for this model specifications chosen by Gignac (2006). Using mixed evidence. the same data set and method of analysis, the authors specified alternative correlated uniqueness based on their 2. Reasons for the mixed evidence on the g-factor own task analysis and demonstrated that their model saturation not only fitted the data equally well but was in fact psychometrically identical to the model specified by The inconsistencies in the results on the g-saturation Gignac (2006). However, the model specified by Ashton of the stratum-two factors can be attributed to differences and Lee (2006) led to substantively different results with between the studies in: (1) the stratum-two factors and regard to the g-saturation of the individual subtests. Their (2) the homogeneity of the sample investigated, as well results indicated that the non-crystallized subtests had a as (3) the construct representation (Embretson, 1983)of higher average factor loading on the g-factor than the the individual subtests. In the following discussion we crystallized subtests. focus on the latter reason and outline how it affects Alternatively one could circumvent the problem as- results obtained with confirmatory factor analyses. sociated with the construct representation (Embretson, In several studies (e.g. Flanagan & McCrew, 1998; 1983) of the individual subtests by building the construct Johnson & Bouchard, 2005; McGrew, 1997) some representation directly into the items. However, this subtests have been observed to load on more than one requires a systematic theory-driven item construction stratum-two factor. This has been taken as evidence that approach that seeks to maximize the construct-related some subtests are far from pure measures of the intended variance in the item difficulties and simultaneously stratum-two factor and would be best described as mixed minimizes unwanted variance arising from non-con- measures. This claim is in line with research on the struct-related factors. In the following section we will cognitive processes respondents use to solve such subtests thus outline such a theory-driven item construction (e.g. Arendasy & Sommer, 2005; Embretson & Gorin, approach using the current debate on the g-saturation of 2001; Meo, Roberts & Marucci, 2007). However, the the broader stratum-two factors of the Cattell–Horn– use of mixed measures turns the substantive interpreta- Carroll model (CHC model) outlined above. However, tion of the extracted factors into a difficult enterprise and it should be noted that the approach could also be might even lead to biased results (cf. Ashton & Lee, 2006; applied to research on alternative models of intelli- Fischer, 1974). gence in order to circumvent problems arising from Since factor analyses do not establish the meaning issues of construct representation within the framework of the shared commonalities captured by each factor, of these models. 576 M.E. Arendasy et al. / Intelligence 36 (2008) 574–583 3. Automatic Item Generation as a tool for building either multiple regression analyses or the linear logistic construct representation into the item construction test model (LLTM: Fischer, 1973). If the radicals (Irvine, process 2002) derived from the theoretical model contribute at least R2 =.80 to the item parameters, the construct represen- In the last few years Automatic Item Generation (AIG) tation (Embretson, 1983) of the subtest can be assumed. has been introduced as a new item construction technology In this sense top-down approaches such as the auto- in applied psychometrics (Irvine, 2002). According to the matic min-max approach (Arendasy, 2005; Arendasy & various top-down approaches to automatic item genera- Sommer, 2007) outlined above serve to build construct tion, the construction process of the item generator starts representation directly into the item construction process with the definition of the