The Science Behind Predicting Job Performance at Recruitment
Wyn Davies, Global Product Manager, Pearson TalentLens Angus McDonald, Chartered Psychologist The Science Behind Predicting Job Performance at Recruitment
Recruitment is a game of risk, where the chances of selecting a candidate who will turn out to be an effective employee are balanced against the risks of identifying someone who will not work out as expected. Recruiters use a wide variety of tools and processes to help them manage the risks associated with hiring decisions. If risks are effectively managed, recruiters are more likely than not to identify candidates who perform well in the job, though even under optimal conditions this process is far from perfect.
The complexity of understanding candidates and evaluating whether they really have the capabilities, There is now over 100 years of scientific characteristics, and motivations necessary for any evidence on the effectiveness of different tools given job means that incorrect decisions are easy to that can be used in the recruitment process. make. Factor into this issues such as a candidate’s deliberate deception, and the difficulties for recruiters only increase. If recruiters get it wrong, they risk costs including the need to quickly replace employees and the potential disruption that poorly fitting employees may cause to the organization. The central concept of validity In response to this recruitment challenge, there is now In the context of recruitment, validity concerns the over 100 years of scientific evidence on the effectiveness propriety of a specific assessment tool for a defined of different tools that can be used in the recruitment purpose. Tools with higher validity enable recruiters to process. This evidence is based on the association more accurately identify those candidates who are likely between evaluations of candidates made during the to go on to perform well in the job. In doing so, they recruitment process, and how they subsequently substantially aid in the management of risk that is an perform in the role for which they are hired. In turn, inevitable part of each hiring decision. this evidence should feed back into the hiring process, guiding recruiters as to which tools will most effectively allow them to manage risk. This paper summarizes scientific research on the effectiveness of different recruitment tools, explains key concepts, and identifies considerations to be taken when using recruitment tools. It is recognized that the processes and legislative frameworks recruiters work within will vary between organizations and regions, so the information given here will need to be applied in a way that is consistent with those local frameworks.
888.298.6227 | US.TalentLens.com 2 The Science Behind Predicting Job Performance at Recruitment
Correlation and validity Many different statistical techniques can be used to determine how effective any assessment tool is for candidate selection, but the majority of these are based on the idea of correlation. Correlation is a statistical technique that looks at the strength of association between two variables, typically the score or scores obtained on an assessment tool, and some measure of job performance. The stronger the correlation, the more closely associated with each other the two measures are said to be, as illustrated in the two diagrams on the right. The charts in figure 1, on the right, show the association between tests scores and job performance. The one at the top shows there is a lower association between tests scores and job performance, whereas the chart below in figure 1 shows a much stronger association. A strong association is indicated by the majority of data Figure 1. Sample of weak (chart at top) points falling on or close to a “line of best fit”. The importance of this vs strong (chart at bottom) correlation association is that it more accurately allows us to identify candidates (validity) likely to perform well in the job. Having strong validity means that an assessment tool is able to predict job performance with a good degree of accuracy. By knowing this, it is possible to use assessment scores to identify those candidates most likely to perform well in the job. Consider the charts in figure 2 which plot the scores of two different assessments against job performance for 30 candidates. The horizontal green line shows the expected level of job performance, meaning we want to select those candidates whose performance will be above the green line. In the top chart in figure 2 which shows a weak association between assessment scores and job performance—or low validity—it is difficult to know which score to use to identify good performers. No matter how high you set your cut score on the assessment, illustrated on the horizontal axis, you will identify some people who go on to perform below the level expected. Conversely, low pass marks will also identify some good performers. The lower chart in figure 2 shows a stronger association between the assessment scores and subsequent job performance. This makes it much easier to set a cut score that identifies where the majority of candidates will perform at the required level. Having strong validity makes it easier to manage the risk associated with recruitment decisions, as assessment scores more accurately predict performance. It is worth noting that performance is affected Figure 2. Sample of weak (chart above) by a number of factors and there is no absolute measure or vs (strong)—chart below and the ideal test that predicts success in the job with 100% certainty. You can, level of job performance however, use a range of relevant and valid assessments to greatly increase the chances of predicting likely performance and reduce the risk of hiring the wrong person.
888.298.6227 | US.TalentLens.com 3 The Science Behind Predicting Job Performance at Recruitment
WHAT IS META-ANALYSIS? What does science tell us? For many years, organizations and academic In practice, recruiters do not know how employees researchers have been trying to understand will go on to perform in a job. This makes selecting the association between assessments made valid assessment tools and applying results from them as part of the recruitment process and appropriately prior to hiring critical. subsequent aspects of job performance. We know the principles behind validity and why it is Many thousands of studies exploring important, but which assessment tools are most valid? this link have been conducted with There is extensive research on the use of different inconsistent results. There are many assessment tools and techniques. More recently, this challenges in conducting this “real world” research has been synthesized to allow comparison of research including being able to follow the effectiveness of different methods. One of the main people from recruitment to a time when ways this has been done is through a technique called job performance can be meaningfully meta-analysis, which is a statistical technique that draws measured, while obtaining accurate together the findings. indicators of job performance and small A recent paper by Frank Schmidt and colleagues1 sample sizes. summarizes the effectiveness of 31 different assessment The last of these points is a particular issue methods for the prediction of job performance and 16 when assessing the validity of recruitment for the prediction of performance in job-related training. processes. For research results to be robust, The key finding from this research was that tests of it is important that sufficient people are general mental ability (GMA) were identified as having the included. However, many organizations highest predictive validity of any selection method. They have only a limited number of people in were also seen to be the best predictor of performance each role, so collecting adequate samples on job-related training. The extensive research takes a long time, if it is even possible. underpinning the use of GMA means a high level of Meta-analysis is a statistical technique that confidence can be placed in these findings. allows the findings from many individual studies to be combined. In doing so, issues General Mental Ability (GMA) Tests such as measurement errors and small As Schmidt and colleagues observe, “when an employer sample sizes are allowed for, meaning uses GMA to select employees who will have a high more robust estimates of the associations level of performance on the job, that employer is between different assessment methods and also selecting those who will learn the most from job job performance can be established. training programs and will acquire job knowledge faster Meta-analyses have consistently identified from experience on the job”. This last point touches that some types of assessment are far more on the primary reason why candidates higher in predictive of job performance than others. GMA perform better in the job; they acquire job In the absence of good local validation for knowledge more rapidly and more deeply than those a specific job role, these findings provide lower in GMA. recruiters with a sound basis for identifying Many tests have been constructed to assess aspects assessments that are likely to be the of mental ability, and there are numerous examples most effective. where these have been developed primarily for use in recruitment settings. Examples include the Watson- Glaser™ Critical Thinking Appraisal and RAVEN’s™ Progressive Matrices. As versions of these tests can be delivered unsupervised online or administered to groups of applicants at a time under supervised conditions, they have a high level of utility in recruitment settings. They
888.298.6227 | US.TalentLens.com 4 The Science Behind Predicting Job Performance at Recruitment
are relatively easy to use and interpret, do not take long It would be rare to be offered a job without some form to complete, and are generally inexpensive. of interview. Research supports the effectiveness of Another notable finding relating to GMA is the wide interviews, and after tests of GMA, they are one of the range of job roles for which it has validity. Typically most effective selection methods. Interestingly, there is these tests have been used in areas such as graduate little evidence to suggest that structured interviews are recruitment, where jobs are assumed to place more effective than unstructured interviews, however considerable cognitive demands on the job holders. structured phone-based interviews do tend to come out Tests of GMA are known to effectively predict graduate- as slightly less valid. level roles, but they also have considerable predictive Other methods that show at least modest predictive validity for virtually any role, including unskilled. GMA’s validity in Schmidt and colleagues’ research include ability to predict performance tends to increase as the integrity tests, biographical data, assessment centers, cognitive demands of jobs increase. Highest validities are grade point average (i.e. academic results at graduate seen for professional and managerial roles, where GMA level), peer ratings, work sample tests, job tryouts, is able to account for over 50 percent of the variability in behavioral consistency method, and job knowledge tests. job performance on average. However, not all of these methods can be readily applied in all recruitment settings. For example, job tryouts How do other measures compare to the cannot easily be used in high-risk jobs or jobs where effectiveness of GMA? considerable training is required, and peer ratings can There are many recruitment tools apart from GMA, only be used with internal candidates. and most selection processes use combinations of these Even when other assessment methods can be used, (See figure 3). it does not necessarily mean they will reduce the risk
S Ts
I te ests
G a uate Post a e u at o a es
B o a al ata
ob t out o e u e
I te t tests
ob o le e tests
Tele o e te e
Assess e t Pee at s
I te e s-u st u tu e
I te e s-st u tu e
Ab l t Test GMA
4 Correlation Coefficient
Figure 3. Job Predictiveness Correlation Coefficients vs. Job Performance
888.298.6227 | US.TalentLens.com 5 The Science Behind Predicting Job Performance at Recruitment
involved in hiring decisions. To determine this, we This is due to the overlap, or correlation, between GMA need to understand how additional measures provide and the additional measure being considerable. In “incremental validity” (see below). While measures the case on the right, the additional measure overlaps may be useful on their own, after the effects of GMA with GMA far less. In this case, using the additional have been allowed for, their impact may be much measure alongside GMA substantially adds to our overall less. Schmidt and colleagues consider GMA to be so understanding of job performance. fundamental in the prediction of job performance, they The extent to which using the additional measure evaluate the effectiveness of all other methods after provides additional information on job performance is allowing for GMA. referred to as “incremental validity”. Due to the large effect of GMA, it makes sense to understand how Incremental validity the incremental validity of other measures have Tests of GMA are the single best predictor of job been affected. performance but many other methods can also be effective. Given this, it might be tempting to use a number of high-validity assessment methods to reduce hiring risk. Unfortunately the situation is The highest combined predictive validity not that simple. The challenge is to understand how comes from the use of GMA and an each method uniquely adds to our understanding integrity test. of job performance—generally referred to as incremental validity. Let’s assume that the large blue circles below represent The effect of GMA job performance. Assessments are used to predict likely Once the predictive ability of GMA is allowed for, the job performance, and we know that GMA accounts for pattern of results changes somewhat. The highest about 50 percent of this (under optimal conditions). combined predictive validity comes from the use of This is illustrated by the gold “GMA” circle overlapping GMA and an integrity test. This is closely followed by job performance. What happens when we include an a combination of GMA and a structured interview, additional measure? then GMA and an unstructured interview. The use of integrity tests adds approximately 20 percent to the predictive ability of GMA alone, whereas structured and unstructured interviews add 18 and 13 percent respectively. The only other measure to add 10 percent or more to the predictive power of GMA are assessments of career interests.
Ab l t I te t test
Job Performance Ab l t test I te e st u tu e
dditiona ea ure Ab l t test I te e u st u tu e Figure 4. Sample of weak vs strong correlation between GMA, Ab l t test job performance, and one additional measure I te ests
This green measure has a similar relationship with job 0.66 0.68 0.7 0.72 0.74 0.76 0.78 0.8 performance to GMA, but in the case on the left it adds Figure 5. Predictiveness of Combined Assessment Methods little to the overall understanding of job performance.
888.298.6227 | US.TalentLens.com 6 The Science Behind Predicting Job Performance at Recruitment
Integrity tests The role of job analysis Since integrity tests add the most incremental validity Predictive validity research is a good guide to how to GMA, it is worth looking at these in more detail. helpful different types of assessments may be Two types of integrity tests can be identified: those in the recruitment process, but deciding exactly that ask directly about dishonesty (“overt” or “direct” what combination should be used relies on a clear assessments), and those that are more personality- understanding of job requirements. Without this, based (“indirect” assessments). Direct tests of integrity recruiters risk blindly applying assessments on the basis may ask about attitudes toward dishonesty, beliefs of data without a broader consideration of how they about the prevalence of dishonesty and, in some relate to job requirements. cases, past dishonest behavior. In contrast, indirect Job analysis details the requirements and tasks assessments attempt to identify aspects of a person’s associated with a specific role, and informs the job character that may underlie dishonest behaviors. Though description and person specification. A clear person personality-based assessments may be less indicative of specification identifies what aspects of candidates need actual behavior, they have the advantage of being less to be assessed and also provides a structure to the transparent, and therefore open to deliberate distortion assessment process. A structured recruitment process of responses. Both have been associated with behavioral has clearly defined stages with each stage providing outcomes in the workplace, but direct measures, despite successive information on the candidates. On the basis their apparent openness to faking, have been found to of this information, candidates may be rejected or 2 have slightly higher validity than indirect measures . progressed to the next stage. Personality measures The task for recruiters is to identify appropriate measures and when they are best used in the Personality measures are widely used in recruitment, recruitment process. For example, if a job requires but what about their validity as it pertains to job specific qualifications or for the job holder to have a performance? As shown in the graph below, although licence to practice or similar, this should be assessed the Big Five factors of Conscientiousness and Emotional very early on. There’s no point conducting other Stability are moderately predictive, the effects of assessments only to discover later on in the most personality factors are almost zero once GMA recruitment process that a candidate does not have has been allowed for. The exceptions to this are the required credentials. Conscientiousness, which as noted above is a significant part of many integrity measures, and to a slightly lesser Qualifications are an example of essential requirements, extent, Openness to Experience. with the exception of where organizations are willing to support candidates in working towards them. A general principle is to assess essential requirements before those that are desirable. Such essential requirements Co s e t ous ess may include qualifications or eligibility to work, but may also involve assessments such as tests of GMA O e ess to E e e e where minimum requirements for these have been established. As many tests of GMA are available online E t a e s o for unsupervised administration, this means that they can be used very early on in the selection process. A eeable ess Technology allows an increasing number of measures to be administered to candidates remotely, including E ot o al Stab l t tests of GMA, personality assessments, work samples, 0 0.05 0.1 0.15 0.2 0.25 and even interviews. Some of the most valid assessment O e at o al al t I e e tal al t a te allo o GMA methods can therefore be used without having any direct contact with candidates. Though this may be Figure 6. Validity of Big Five Personality Factors efficient in terms of processing applicants, organizations
888.298.6227 | US.TalentLens.com 7 The Science Behind Predicting Job Performance at Recruitment
GROUP DIFFERENCES AND BIAS concerned with candidate attraction may not choose this approach. Instead they may want to have early personal All assessments are expected to measure equally contact with candidates as a way of engaging them with for all groups. When group differences are seen, the organization. However, tests of GMA are generally this raises a concern that the assessment tool is well perceived by candidates3, despite the claims made in some way biased or is not measuring fairly for by developers of more game-like assessments. all groups. In summary, there is no single best way to design a Unlike physical attributes such as height or recruitment process. Each process should be tailored weight that can be directly measured, we do not to the specific needs of the organization, but key have direct access to the factors measured by considerations should be: psychometric assessments. Instead, assessments 1. What does the job analysis tell us is needed for ask respondents to solve problems, report on successful performance; their behavior, make judgements or similar, and from the information they provide we make 2. how do we assess the essential and desirable an inference about the degree to which the characteristics for the role in the most valid ways; respondents possess different qualities. In this 3. how are we going to structure the recruitment process process, it is quite possible that the assessments and use information gathered to successively filter are biased in some way—historically there are candidates; and certainly plenty of examples of this—but modern 4. how and when do we want to engage candidates with test development techniques have done much to the organization through the recruitment process. manage this source of bias. An alternative interpretation of group difference Adverse impact and diversity data is that it reflects real differences between Diversity is a key issue for many organizations, as the groups being assessed. There may be a they strive to address the uneven representation of tendency to believe that all people are equal different groups in their workforces. Many aspects of the and differences must be the result of biased recruitment process have been cited in diversity debates measurement. However, differences exist in the at various times. The judgements made by recruiters physical world, so why not the psychological? For are one of the areas that have received much attention, example, just because a measuring tape shows with research highlighting numerous forms of bias that men are taller on average than women, that may affect recruitment decisions based on does not mean that it is biased. This difference information from application forms, resumes, interviews is accepted as it is visible and we can directly or similar techniques. measure height, unlike many psychological Psychometric assessments have also long been cited in characteristics that have to be inferred from diversity discussions. The nature of the data produced other observations. from such assessments means that the extent to which Despite best practices in test development, they provide an objective assessment of different groups some group differences remain. An authoritative is relatively easy to study. As with validity research, many review of this area was conducted under studies have been conducted into group differences in the guidance of the American Psychological psychometric assessment scores and synthesis of these 4 Association . It concluded that while sex results has not always been straightforward. While there differences in performance on tests of GMA is general agreement that tests of cognitive ability show were generally seen to be small, this was not the some differences, especially when results are compared case for ethnic group differences. While various across different ethnic groups, they also predict job reasons had been offered to account for these performance equally for different groups5. This means differences, no adequate reason for them could that two people scoring the same on a test of GMA will be identified. show approximately the same level of job performance regardless of their group membership6.
888.298.6227 | US.TalentLens.com 8 The Science Behind Predicting Job Performance at Recruitment
Addressing adverse impact Employee engagement has become a key area of interest for many employers. Employees who are The ability of GMA tests to predict job performance engaged are energized by their work and committed to means their use is defensible from a legal perspective it and the organization. Engagement is also associated in most situations, but the results obtained by different with a range of positive workplace behaviors and groups can have the effect of reducing diversity in the reduced turnover. Gallup has been tracking employee workforce. This is clearly not desirable as organizations engagement in the US, typically finding that only about wish to ensure that their workforce is diverse. one-third of the workforce reports being engaged One practical impact of group differences is that adverse with their jobs. Worldwide, this figure is much lower8. impact tends to become greater as cut scores are set Disengaged employees are not only likely to have poorer higher. This is particularly relevant to the application performance, but they can also have a negative effect on of GMA. For GMA tests, in particular, setting a low cut colleagues, potentially causing productivity issues score—for example no higher than the 25th or 30th to spread. percentile—may help to minimize adverse impact. Organizations increasingly seek to understand more Adverse impact can be a complex area and we advise about candidates’ potential fit to the job role, a key that organisations consult Industrial Organizational determinant of engagement. In this context, fit is not Psychologists and other experts in this field to find out about having the qualifications, skills or knowledge to do more about the ways to moderate and minimize it. the job, or even the specific behaviors someone typically Psychometric test publishers strongly advise that displays, but whether the person’s values, attitudes GMA tests are not the sole source of assessment data and needs are aligned with those of the organization. upon which to base a hiring decision. When used Most often this is assessed at an organizational level with other relevant assessments, however, their high (e.g. “does this person share the values of the wider ability to predict job performance can be of great value organization?”), but they may also be at a more micro to recruiters. level, such as the work unit or specific team. Other forms of assessment such as interviews, integrity This focus on fit represents a shift from looking primarily tests, and situational judgement tests tend to show far at the “can do” to considering the “will do”. Schmidt and less evidence of group differences. Though they may be colleagues’ work concludes that person-organization less predictive than tests of GMA, employers increasingly fit assessments have an overall low predictive validity, see them as desirable as they maintain diversity in the but what exactly constitutes “fit” and what outcomes it applicant pool. Such assessments may be used as an affects are unclear. While fit may be weakly related to alternative to GMA or in combination with it. When performance, it does show stronger relationships with used in combination, GMA may be applied at a separate turnover intentions and positive work attitudes9. Fit is point from other assessments or alongside them, most often assessed by looking at the match between where results from all assessments can be combined to an individual’s values and those of the organization, but produce a weighted result which will have less adverse questions remain about how best to understand values impact. Such strategies can have a modest benefit in at an organizational level and how to relate these to reducing the impact of GMA on diversity, but do not offer individual candidate’s needs and expectations. a complete solution7.
Predicting for More Than Performance Values Values is another area that has become of increasing For much of the history of research into selection interest to organizations. As well as being involved in fit, methods, the key outcome has been job performance organizations seek to be more “values conscious” and or productivity. More recently, employers have shown look at an individual’s strengths. Prominent strengths- interest in additional areas that have not always been based approaches focus on values10, arguing that thoroughly understood. individual strengths result from the identification and application of values. In recruitment contexts, strengths-
888.298.6227 | US.TalentLens.com 9 The Science Behind Predicting Job Performance at Recruitment
based approaches tend to focus on each individual gathering and decision making. This process is not applicant’s unique pattern of strengths and how these adequately modeled in research that takes a “snapshot” might be utilized within the role and organization more of the association between scores on a measurement broadly. These tend not to replace more established instrument and job performance. Data may be collected selection methods, but are used in addition to them. early in the recruitment process or later on, possibly Organizations cite a number of reasons for using after some candidates have already been rejected. In strengths-based approaches including finding better other cases, data may be collected on people who are fit, creating organizational advantage, and increasing already working in a role, meaning no data are available diversity. It is notable that values and strength- on those already rejected. Decisions about when to use based approaches were not included in Schmidt and different assessment methods are critical in managing colleagues’ research, presumably due to research in risk, but research often fails to give sufficient detail to these areas still developing. adequately inform these decisions. Finally, validity is measured against “job performance”. Limitations of What We Know What constitutes effective job performance varies Much of this paper has been based around Schmidt considerably between roles and organizations, and and colleagues’ recent paper on the validity of different defining performance in a way that allows it to be readily assessment methods. This research is probably the and accurately measured is a challenge. Quite often, most comprehensive of its kind, but it is not without job performance relies simply on managerial ratings limitations. It is important that assessment users which can be influenced by multiple sources of bias. are aware of these limitations if they are to design As discussed above, other outcomes of interest to assessment processes with due considerations as to the employers are yet to be explored in the same depth as limits of our scientific knowledge. “job performance”. A significant limitation of all meta-analytic research is that the details of specific studies—such as the organization it was conducted in, job roles assessed for, sample Research consistently shows that the sensitive size etc—are often lost. While the best meta-analytic application of valid measures substantially studies may evaluate the quality of the research they reduces risks associated with recruitment use and weight results accordingly, this is not always and results in considerable gains in workforce the case. While local validation—establishing a link productivity. between assessments and job performance in a specific organization—is always preferable, opportunities for this are restricted to organizations recruiting a relatively large number of people into similar roles. It also dictates Though it is important to be mindful of these limitations, that employees are given sufficient time to establish research consistently shows that the sensitive application themselves in the role, meaning assessment of job of valid measures substantially reduces risks associated performance may not take place until at least 12 months with recruitment and results in considerable gains in after recruitment. workforce productivity. Recruiters can be confident Further consideration must be given to how assessment in generalizing from validity research for the majority results were applied and at what point in the selection of job roles, but need to apply this sensitively and process. It has already been noted that recruitment with due regard for the specific situation to gain is a process involving various stages of information maximum returns.
Want to be more confident in the tools you rely on to inform your recruitment decisions? We are here to help! Visit US.Talentlens.com or contact us at 888.298.6227 | [email protected]
888.298.6227 | US.TalentLens.com 10 The Science Behind Predicting Job Performance at Recruitment
References 1 Schmidt, F. L. (2016) The Validity and Utility of 6 Rothstein, H . R and McDaniel, M. A. (1992). Differential Selection Methods in Personnel Psychology: Practical validity by sex in employment settings. Journal of and Theoretical implications of 100 Years. Retrieved Business and Psychology, 7, 45-62. on 16 October 2017 from: researchgate.net/ 7 Sackett, P. R., Schmitt, N., Ellingson, J. E. and Kabin, publication/309203898 M. B. (2001). High-stakes testing in employment, 2 Van Iddekinge, C. H., Roth, P. L., Raymark, P. H. and credentialing, and higher education: Prospects in a Odle-Dusseau, H. N. (2012). The criterion-related post-affirmative-action world.American Psychologist, 56, validity of integrity tests: An updated meta-analysis. 302-318. Journal of Applied Psychology, 97, 499-530. 8 Gallup (2013). State of the Global Workplace: Employee 3 Anderson, N., Salgado, J. F. and Hülsheger, U. R. (2010). Engagement Insights for Business Leaders Worldwide. Applicant reactions in selection: Comprehensive Washington DC, Gallup. meta- analysis into reaction generalization versus 9 Arthur, W. Jr., Bell, S. T., Villado, A. J. and Doverspike, situational specificity.International Journal of Selection D. (2006). The use of person-organization fit in and Assessment, 18, 291-304. employment decision making: An assessment of its 4 Neisser, U., Boodoo, G., Bouchard, T. J., Boykin, A. W., criterion-related validity. Journal of Applied Psychology, Brody, N., Ceci, S. J., Halpern, D. F., Loehlin, J. C., Perloff, 91, 786-801. R.; Sternberg, R. J. and Urbina, S. (1996). Intelligence: 10 Peterson, C. and Seligman, M. E. P. (2004). Character Knowns and Unknowns. American Psychologist, 51, strengths and virtues: A handbook and classification. 77–101. New York: Oxford University Press. 5 Schmidt, F. L. and Hunter, J. E. (1981). Employment testing: Old theories and new research findings. American Psychologist, 36, 1128–1137.
888.298.6227 | US.TalentLens.com
Copyright © 2018 Pearson Education, Inc. or its affiliate(s). All rights reserved. Pearson, Watson-Glaser, and RAVENS are trademarks, in the U.S. and/or other countries, of Pearson Education, Inc. or its affiliates. TALEN8266 - 11412 SR 6/18