The future of HR is here (and it’s SHRM-certified)

Choosing Effective Talent Assessments to Strengthen Your Organization

This publication is designed to provide accurate and authoritative information regarding the subject matter covered. Neither the publisher nor the authors are engaged in rendering legal or other professional service. If legal advice or other expert assis- tance is required, the services of a competent, licensed professional should be sought. Any federal and state laws discussed in this report are subject to frequent revision and interpretation by amendments or judicial revisions that may significantly affect employer or employee rights and obligations. Readers are encouraged to seek legal counsel regarding specific policies and practices in their organizations.

This book is published by the SHRM Foundation, an affiliate of the Society for Human Resource Management (SHRM©). The interpretations, conclusions and recommendations in this report are those of the authors and do not necessarily represent those of the SHRM Foundation.

©2016 SHRM Foundation. All rights reserved. Printed in the United States of America.

This publication may not be reproduced, stored in a retrieval system or transmitted in whole or in part, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without the prior written permission of the SHRM Foundation, 1800 Duke Street, Alexandria, VA 22314.

Selection of report topics, treatment of issues, interpretation and other editorial decisions for the Effective Practice Guidelines series are handled by SHRM Foundation staff and the report authors. Report sponsors may review the content prior to publication and provide input along with other reviewers; however, the SHRM Foundation retains final editorial control over the reports. Editorial decisions are based solely on the defined scope of the report, the accuracy of the information and the value it will provide to the readers.

The SHRM Foundation does not explicitly or by implication endorse or make any representations or warranties of any kind regarding its sponsors or the products, services or claims made by its sponsors. The SHRM Foundation does not assume any responsibility or liability for the acts, omissions, products or services offered by its sponsors.

For more information, contact the SHRM Foundation at 703.535.6020. Online at shrmfoundation.org

16-0720 Table of Contents

iii Foreword v Acknowledgments vii About the Authors 1 The Business Case for Effective Talent Assessment 2 The Assessment Landscape 2 The Evolution of Assessment Technology 3 Future Trends 4 Tools of the Assessment Trade 4 Job Analysis 5 Assessment Methods 8 How to Select the Right Assessment Method 10 Validity 11 Adverse Impact 11 Tradeoffs Between Validity and Adverse Impact 12 Assessment Utility: Business Benefits 12 Costs 13 Candidate Reactions 14 Legal Issues 17 Conclusion 19 Sources and Suggested Readings 33 Endnotes

i

Choosing Effective Talent Assessments to Strengthen Your Organization

FOREWORD

Organizational success is largely driven by the acquisition of the best talent available. However, statistics reveal that 25 percent of mission-critical positions often remain vacant after five months. As organizations compete for the top candidates, they expend large sums of money, time and other resources in advertising and recruitment. Research shows that the careful use of talent assessments can result in increased productivity, cost savings and better retention of top employees.

To help you sort through the many assessments available and find those that will work best for you, the SHRM Foundation has created Choosing Effective Talent Assessments to Strengthen Your Organization. This guide focuses on the tools of the trade and how to select the right assessment to achieve your business goals. It helps employers better understand the value of assessment, as well as the legal and technical issues involved. In addition, it offers a clear roadmap to selecting the effective assessment methods to target top talent and boost productivity.

This report is part of the SHRM Foundation’s Effective Practice Guidelines series, which now includes more than 20 titles. Created in 2004 for busy HR professionals, the series integrates research findings with expert opinion on how to conduct effective HR practice. It provides the tools to successfully practice evidence-based management. Other recent reports include Talent Acquisition, Creating a More Human Workplace and The Aging Workforce. To ensure the material is both practical and research based, the reports are written by subject-matter experts and are then peer- reviewed by both academics and HR professionals.

The SHRM Foundation champions workforce and workplace transformation and inspires HR professionals to make it happen. Our educational resources are now used in hundreds of college classrooms worldwide. The SHRM Foundation also awards hundreds of scholarships annually to support education, conference attendance and professional certification for HR professionals. All this good work is made possible by the generous support of donors like you.

We encourage you to learn more. Please visit shrmfoundation.org to download other complimentary resources and to find out how you can get involved with the SHRM Foundation. Thank you!

Mark Schmit, Ph.D., SHRM-SCP Executive Director, SHRM Foundation iii

Choosing Effective Talent Assessments to Strengthen Your Organization

ACKNOWLEDGMENTS

The SHRM Foundation is grateful for the assistance of the following individuals in producing this report:

REVIEWERS PROJECT MANAGER Howard J. Klein, Ph.D. Dorothy Mebane, Ph.D. Professor of Management and Human Manager, Foundation Programs Resources SHRM Foundation Fischer College of Business The Ohio State University

Jim Link, SHRM-SCP Chief Human Resources Officer Randstad North America

Kameron M. Carter Ph.D. Candidate Department of Management and Organizations University of Iowa

The authors would like to thank Emmanuel Rilhac, head of EMEA & Global Functions, CEB, Arlington, Va., and Donna L. Weiss, director, Product Management, CEB, Arlington, Va., and two anonymous reviewers for their comments on and earlier draft of this report. Their comments contributed significantly to shaping the final product.

v

Choosing Effective Talent Assessments to Strengthen Your Organization

ABOUT THE AUTHORS

Elaine Pulakos, Ph.D., has spent her career working with organizations to design and implement talent management systems and processes in the areas of staffing and assessment, performance management, leadership development, and succession management. She is a recognized talent management thought leader and frequently invited speaker to address both practitioner audiences and the scientific community. She has authored numerous articles, book chapters and books, as well as three best-practice volumes for the Society for Human Resource Management (SHRM).

Dr. Pulakos’s work has been recognized with several awards, including the Society for Industrial and Organizational Psychology’s (SIOP’s) Distinguished Professional Contributions Award, the M. Scott Myers Award for Applied Research in the Workplace, and the William A. Owens Scholarly Achievement Award. Elaine is a fellow of the American Psychological Association and SIOP, for which she also served as president. Currently, she leads two groups within CEB: global measurement product development and PDRI, a CEB company.

Tracy Kantrowitz, Ph.D., is research director at CEB Talent Assessment. In this role, she is responsible for the development of assessment content and research related to the use of assessment tools for employee selection and development. Dr. Kantrowitz has published in leading journals and presented at national conferences on topics such as predictors of job performance, computer adaptive testing, unproctored Internet testing and mobile assessment. Tracy’s primary areas of expertise relate to test development, validation, research methodology, statistics and psychometrics, job analysis, and selection system design.

Tracy was previously chair of the Professional Practice Committee for the Society for Industrial and Organizational Psychology (SIOP). She has received awards from SIOP, where she was elected fellow, including the M. Scott Myers Award for applied research in the workplace and the Distinguished Early Career Contributions - Practice award.

Dr. Kantrowitz holds a Ph.D. in industrial/organizational psychology from the Georgia Institute of Technology.

vii SHRM Foundation’s Effective Practice Guidelines Series

Research reveals that careful talent assessment results in significant productivity increases, cost savings and decreased attrition. Choosing Effective Talent Assessments to Strengthen Your Organization

CHOOSING EFFECTIVE TALENT ASSESSMENTS TO STRENGTHEN YOUR ORGANIZATION

THE BUSINESS CASE FOR EFFECTIVE TALENT ASSESSMENT Organizations today compete fiercely for talent. Recent statistics indicate that 25 percent of critical positions remain unfilled after five months.1 Firms invest an enormous amount of money, time and other resources in advertising and recruiting strategies to attract the best candidates because today’s executives understand that one of the most important drivers of organizational success is outstanding talent. Research also reveals that careful talent assessment results in significant productivity increases, cost savings and decreased attrition.

Questions Leaders Are Asking Organization Challenges

On average, one in four new hires How can we reduce turnover? leaves within a year.2

Each position has many applicants, How can I find people who will but 65 percent of them do not meet succeed in specific positions? basic requirements.3

How can we identify effective Only 12 percent of today’s leaders? employees are enterprise leaders.4

Given the proven bottom-line value of attracting and retaining talent, why do so many organizations and HR professionals employ ineffective, haphazard approaches to selecting employees? This report addresses how and when to use assessments to improve hiring and retain the most valuable employees in a competitive market.

Although the term “assessment” can include a wide variety of tools, including resume reviews, applications, reference or background checks, and interviews, the focus here is on more formal scientific assessments. After a brief survey of the value of different types of assessments, we will focus on practical explanations of technical and legal issues and then offer a roadmap to choosing the methods that offer the best return on investment for your organization.

1 SHRM Foundation’s Effective Practice Guidelines Series

5 THE ASSESSMENT LANDSCAPE services. Approximately two- Assessment Trends thirds of HR professionals report ●● The nature and variety of Machine learning and that their organizations are assessment tools have evolved analytics are introducing putting a premium on positive considerably in recent years as new assessment applicant experiences, although analytics and data have become techniques. relatively few collect data to more important in the business ●● Globalization requires monitor reactions to assessment environment. Formal assessments 6 localized assessments procedures. are used to hire new employees, that can easily reach evaluate current employees One of the most dramatic overseas. for different roles, identify shifts involves the evolution ●● Technology is enabling employees with high leadership of the candidate-driven job on-demand, online potential, compare talent within market 7 , which has changed assessments with an organization against industry how organizations compete for or geographical benchmarks, immediate results and talent and the nature of their feedback. understand talent strengths and recruitment and assessment gaps, and develop employees’ processes. Candidates want brief, ●● More firms are using long-term value. Advances in informative, and engaging hiring immersive, compelling technology mean that more tools experiences and transparency assessment experiences. are accessible online, on-demand, so they know what to expect, ●● Organizations want and with immediate reporting and where they stand and how much personalized and feedback. effort to put into a hiring process. contextualized Technology has also given rise to All these new expectations are assessment. transforming assessment methods more available data on individuals ●● and practices. Candidate-centric hiring and new tools that enable requires shorter, more organizations to mine these data for transparent and engaging competitive advantage. For example, The Evolution of candidate experiences. social media and other data Assessment Technology collected via technology are used Unsupervised online testing for predictive analytics, modeling has become the standard for questions from a large set of items trends, passive assessment of assessment administration based on how each candidate is candidates and uncovering insights because it is so efficient, responding: If a candidate responds to enhance business performance. decreasing the time needed to fill correctly, the next item will be more The use of immersive, personalized vacancies and reducing costs.8 difficult; if a candidate responds assessment experiences is on the Initially, online testing prompted incorrectly, the next item will be rise. two major concerns: test security easier. A customized assessment and candidate integrity. In the hiring context, organizations with a unique combination of items are using these new tools and Concerns about security arose focuses on each candidate’s level of experiences to provide job because early versions of online ability and knowledge. CAT creates previews and sell the organization assessments were static, meaning more secure and more efficient and its brand. Some firms want that all candidates received assessments, with fewer items in to provide brand-enhancing the same items, which could less time, to meet the demands of hiring experiences that will attract be copied and shared easily.9 candidate-centric hiring. candidates and retain them If tests are copied and shared, CAT represents a critical as customers, even if they are they become invalid and must be innovation in online assessments10 not offered jobs. This trend is replaced, creating unsustainable but does not resolve the problem important because poor candidate costs for companies. experiences are broadly shared of candidate integrity or making throughout social networks and Computer adaptive testing (CAT) sure that the person taking can yield decreased use of the methods are now the norm because an online assessment is the organization’s products and CAT allows for dynamic selection of same person who is applying

2 Choosing Effective Talent Assessments to Strengthen Your Organization

Test Equivalence Is a Key Issue Technological Advances As assessment devices evolve from booklets to PCs to mobile phones in Assessment and tablets, we need to consider whether the mode of administration, ●● Unsupervised online rather than the assessment itself, is leading to lower or higher scores. testing. The ideal situation is that no matter what device is used, the candidate ●● earns the same score. Computer adaptive testing.

●● for a job. To ensure candidate as much as by their measurement Mobile assessment. integrity, some organizations and predictive qualities. ●● Low effort assessment. ask candidates who have progressed through the hiring Some assessments, including important in predicting future process to complete a brief onsite personality and past-experience performance. assessment at a convenient point assessments, show little to verify the candidate’s online difference in scores when completed on mobile devices Future Trends score. 12 or PCs. But research on Technology will continue to Mobile assessment now provides cognitive ability assessments is spark revolutionary changes even more convenience than inconsistent, with some studies in assessment methods and assessments via desktop finding equivalence across practices. For example, the ability computers. Consumer data 13 device types, and others finding to passively target candidates indicate that mobile devices differences—with lower scores from available information sources may be more readily available 14 on mobile phones. Overall, could lead to transformational to underrepresented groups, however, initial research shows changes in hiring processes. including African Americans, relatively small differences Today, applicants apply for Hispanic Americans and women.11 restricted to certain assessment positions based on job postings, Assessment administration on items. but tomorrow organizations may mobile devices may therefore search data for key talent in result in larger and more diverse Low-effort assessment uses new ways that bypass postings. This candidate pools. technology and machine learning to sift through large amounts of is uncharted territory and will Differences in mobile devices unstructured information, including require new policies on privacy, mean that mobile assessment resumes and applications, to legal and ethical considerations. can be complex, requiring collect predictive information As hiring practices span global more nuanced evaluation. For about candidates. The information markets, human resource example, larger screens on is coded and validated against professionals will also have to tablets allow some forms of work outcomes, such as job become familiar with additional assessment that will not work performance and retention, to laws and regulations. well on phones. How information produce algorithms that managers Perhaps most important, early is input, whether via touch can use to inform their hiring adopters of new technology screen or keyboard and mouse, decisions. These techniques should be aware that without presents unique challenges. automate identification and sufficient research behind it, A new class of assessments scoring of predictive information an innovation might undermine that use mobile-centric design so that large numbers of rather than help an organization principles has arisen to optimize candidates can be screened in to make effective, defensible the user experience. We are minimal time at low cost. seeing a demand for innovative hiring decisions. In some assessments that go beyond Low-effort assessment is still countries, litigation and audits of multiple choice and look and in its early stages, so all the hiring practices will lead to new feel more like games. The result advantages and disadvantages standards and precedents related of these innovations is that are not yet clear, but it is probably to new assessment technology, assessments are being judged best suited for situations in which and organizations must be alert by their design and accessibility past experience is especially for these changes. Later in this

3 SHRM Foundation’s Effective Practice Guidelines Series

report, we will look at the science Investigator Job: Sample Competencies behind assessments, legal ●● Communication: Speak and write clearly and concisely, with self- requirements and workers’ rights confidence and using effective inflection and gestures. to help organizations navigate these choppy new waters. ●● Critical thinking: Question assumptions and identify merits in logic. ●● Collaboration: Elicit cooperation from other individuals and organizations. TOOLS OF THE ASSESSMENT TRADE ●● Professionalism: Maintain a professional demeanor and appearance at all times. Organizations can use a broad array of techniques to make ●● Handling stress: Remain calm and levelheaded. selecting the right assessment tool easier. The first order of business is to do a thorough job Job Analysis Steps analysis, followed by a review of the various types of assessment STEP 1. Observe or interview job experts to develop a methods available. list of work activities and competencies workers need.

Job Analysis STEP 2. Ask job experts to rate which work activities and The first step in selecting an competencies are most critical for effective job performance. assessment tool is to understand what the target job or role requires STEP 3. Analyze the data to prioritize the most critical work the employee to do by conducting activities and competencies for your organization. a job analysis15 that focuses on identifying critical work activities and/or competencies. STEP 4. Select assessments that measure the performance of the most critical work activities or competencies required on the job. An in-depth discussion of job analyses is beyond the scope of this report, but they typically include four major steps, as to broader performance Investigator Job: Sample outlined above. requirements. Work Activities Every organization’s unique One useful source of job analysis ●● Provide testimony by character is reflected in how it information is O*NET, a database stating facts and answering performs a job analysis and then that is continually updated by questions. chooses assessments. Business surveying a broad range of workers ●● Gather and review information strategy, organizational culture from hundreds of occupations. to obtain evidence, or and contextual factors always play Employers and job seekers can develop background major roles. Contextual factors may leverage the O*NET database information on subjects. include high-versus-low growth, to obtain information for virtually high-versus-low change and high- any job in the global economy. ●● Integrate information to versus-low team interdependency. The database forms the heart uncover relationships A recent study of thousands of of O*NET OnLine, an interactive between individuals, events organizational leaders across 85 application for exploring and or evidence. companies showed the importance searching occupational information ●● Work in a team environment of organizational context factors (http://onetonline.org). Of course, as a team member or leader. in successful identification of it is always best to verify that job 16 ●● Calm and reassure victims leaders at all levels. These results information from O*NET fits the and others in tense situations. reinforce the value of thinking specific job and situation in your beyond job-specific competencies organization.

4 Choosing Effective Talent Assessments to Strengthen Your Organization

Work behavior is the tip of the iceberg—the observable Work Behavior Assessments performance on the job. Work ●● Behavioral interviews. behavior assessments require ●● Work simulations. Work candidates to perform simulated Behavior aspects of the job and provide ●● Physical ability tests. an observable measure of Acquired capabilities –knowledge, performance that should forecast skills, judgment, experience real on-the-job performance. Innate mental and physical Acquired Knowledge, abilities, personality, interests These assessments should correspond closely to the specific Skills and Experience work that candidates will perform Assessments and tend to be expensive, time- ●● Bio data/experience.

consuming and somewhat limited ●● Job knowledge. Assessment Methods in their use. ●● After you understand job Writing skills. Acquired knowledge, skills and requirements, you can select ●● Situational judgment. experience sit just below the assessments that measure the surface. Although less directly most critical work activities or observable than work behaviors, competencies. For example, is the assessments of job-relevant assessment goal to select someone Innate Ability and knowledge, skills and experience Attribute Assessments to perform a specific job in the are extremely effective predictors ●● Cognitive ability. near term or to predict employee of job performance. Because these potential for future leadership roles? assessments require measuring ●● Personality. In the former case, the assessment unique capabilities relevant to ●● Integrity. can be targeted to the most critical specific jobs, they are not generally ●● competencies for the job. In the applicable across jobs. Physical fitness. latter case, a broader and more comprehensive whole-person Innate ability and attributes, Below we discuss specific assessment will probably be more including personality and assessment methods, all of which useful. interests, lie deep below the surface and also have are backed by strong research Assessment types run along a been proven to be effective over decades. They are all readily continuum from more observable, predictors of job performance. available and effective for small direct measures of performance Because assessments of innate and large companies. We do not to less observable, indirect characteristics do not focus on include some newer assessment measures of performance. An job-specific traits, they are the methods with less research iceberg analogy helps illustrate most flexible and applicable for available to back them up. the different assessment types. use across different jobs. Cognitive ability assessments typically consist of multiple-choice Example of Cognitive Ability Test Items items that measure various mental abilities, such as verbal and Verbal ability Numerical ability Reasoning ability numerical ability, reasoning, and Innocuous means the 16% of 62.5 is: 1 3 2 4 3 5 4 6 5 reading comprehension. These same as: a. 844 a. 4 assessments have proven to be a. Harmless b. 8.44 b. 5 extremely useful predictors of b. Preventative c. 084 c. 6 job performance and are used c. Distasteful d. 8.4 d. 7 frequently in hiring for many types d. Futile of jobs.17 Some measure distinct abilities, whereas others measure a

5 SHRM Foundation’s Effective Practice Guidelines Series

combination to provide a measure of job performance,18 particularly Example of an Integrity of general mental ability or “g.”18 in combination with measures of Assessment Item cognitive ability. In work situations, It is okay to misrepresent Cognitive ability tests conscientiousness, extraversion, the truth if being completely administered online in agreeableness, openness to honest will create problems unsupervised testing situations experience and emotional that small “white lies” can often employ adaptive methods. stability are measured most solve. The mobile environment poses 20 frequently. Research suggests a. Strongly disagree some unique challenges that conscientiousness is the b. Disagree because the typical questions most robust and useful predictor c. Neither agree nor are too complex and lengthy of performance across jobs. disagree for mobile phones, although Achievement orientation is also a they may work well on tablets. strong predictor in many jobs.21 d. Agree Newer test designs will probably e. Strongly agree be optimized for mobile Personality assessments typically administration. consist of several multiple-choice or true/false items that measure Personality assessments a variety of personality factors. measure stable, job-relevant Examples of Physical Desirable answers are often quite Fitness Tests characteristics that have been obvious in this format, so there ●● Run a mile. shown to be effective predictors is considerable concern about how easy it is to “fake” a positive ●● Lift two 20-lb. weights for impression. On the other hand, 50 repetitions. Examples of Personality even when candidates do respond Test Items with what they assume are the Standard format: desired answers, personality perform a job. By nature, they don’t assessments are still useful for work in an online environment. I have remained calm in predicting job performance.22 Some situations in which others assessments avoid the problem Biographical data (bio data) have become upset. assessments ask candidates by using a format that presents questions about their past a. Definitely true equally desirable response options experience in areas that have b. Somewhat true representing different personality been shown to predict future c. Neither true nor false traits. This item format—asking 25 performance. For example, a candidates to choose which d. Somewhat false question about the number of answer is most like them—is less e. Definitely false volunteer organizations to which transparent than a simpler scale. someone has belonged could These assessments work well 23 indicate the extent to which he or Format that Mitigates across different devices. Intentional Desirable she is willing to pitch in and help Responses Integrity assessments measure others. A question about where Choose which item is most attitudes and experiences related one’s spending money came like you and which is least to honesty, trustworthiness from in high school could indicate 24 like you. and dependability. They a history of responsibility and are typically presented in a independence. a. I persist with tasks until multiple-choice format. Similar to completed. Some bio data assessments personality assessments, integrity are scored using data-driven b. I treat customer assessments can be easily techniques, meaning the points complaints with highest administered across different associated with each response priority. devices without problems. c. I deal well with extremely option are determined according stressful situations. Physical fitness tests measure to the relationship between overall fitness, strength, endurance that response and an important or other capabilities necessary to job outcome. Other bio data

6 Choosing Effective Talent Assessments to Strengthen Your Organization

automated scoring algorithms are Example of a Job assessments are rationally scored, used. Accomplishment records Knowledge Item meaning the test items are scored are effective predictors of job according to the degree to which The principle of the “lever” performance.28 is essential to the use of a: each response option measures a. Hydraulic jack an important competency for Job knowledge assessments a particular job. Biographical measure critical knowledge areas— b. Plow inventories consist of multiple- usually technical knowledge— c. Car steering wheel choice items, and like personality needed to perform a job d. Forklift assessments, they tend to be effectively.29 These assessments are equivalent across different modes of appropriate to use when candidates 26 administration. must possess a specific body of Another form of bio data is knowledge prior to job entry, not How Are Writing an assessment called an when candidates will be trained Assessments Created? on the job. Job knowledge tests accomplishment record. In this case, First, administrators prepare typically consist of multiple-choice candidates prepare a written account a written prompt and give it items, but they sometimes include of their best accomplishments in to about 200 candidates. At essay items. In addition to hiring key competency areas, such as least two human evaluators assessments, job knowledge tests planning and organizing, customer independently score service, and conflict resolution.27 are used often for certification and each writing sample using Candidates also provide the name licensing assessment. defined ratings scales and of an individual, such as a past Knowledge tests can be come to an agreement. supervisor, who can verify each administered well online using These scores and scales accomplishment. Evaluators are large banks of items and adaptive are used to train automated trained to score the accomplishments technology, but there is no definitive scoring algorithms, which in a consistent manner using research about whether they can are then used to score later standard rating criteria. Sometimes be administered effectively on writing samples. mobile devices. Certification and licensing assessments, especially Examples of Bio Data those that are high stakes, are still Assessment Items after the initial investment can be administered in a supervised testing significant. This type of assessment To how many volunteer environment for the most part. is suitable for online environments organizations do you as long as it is possible to confirm belong? Automated writing assessments that the candidate, and no one a. 0 can be developed to measure various competencies beyond the else, is the one providing the b. 1 writing itself, including a candidate’s writing sample. c. 2 to 4 persuasiveness, critical thinking, Situational judgment tests present d. 5 or more attention to detail, commercial focus candidates with situations just and political savvy. Technology is Where did you get your like those they would encounter used in lieu of human scoring with spending money in high on the job, with viable options a technique called latent semantic 30 school? for handling each situation. analysis. Writing assessments are Candidates might be asked to ●● Allowance from family especially useful in selecting people select the most effective, or the ●● to fill positions with significant Own earnings most and least effective, ways of writing demands, such as attorney, ●● handling a situation from a few Partly allowance, partly analyst or investigator. earnings options. These tests are more Development of the writing prompts expensive and complicated to ●● Other sources and scoring algorithms requires develop than many other types ●● Had no spending money quite a bit of time and resources, because developing scenarios but the efficiency and cost savings with several equally attractive,

7 SHRM Foundation’s Effective Practice Guidelines Series

Example Situational viable responses is difficult. They often use them to assess qualities Judgment Item are typically administered in written like teamwork, communication, You have just prepared or videotaped form, and both leadership, planning and a report and checked it formats can be administered online, adaptability. for accuracy. Before you but administering them on mobile attend a meeting to submit devices with smaller screens may be Work simulations measure how your report, you review challenging. a candidate performs activities the typed version and note Structured behavioral interviews— that the job requires, and of many serious errors. In this unlike more traditional, unstructured course, employers should use case, you would: interviews—consist of specific them only when knowing how to a. Show the original and questions that assess critical perform those activities is truly a the typed version to competencies required for specific prerequisite.32 These simulations the person in charge of jobs. can be time-consuming and typing and demand that Interviews are among the most expensive to develop. Sometimes, the errors be changed common hiring methods used trained evaluators do the scoring, before the meeting. in all sorts of organizations.31 but today assessment and b. Present the report at the Unfortunately, there are rarely any feedback are often automated. meeting, point out the agreed-on standards for evaluating errors and explain the responses, and unstructured Physical ability tests are used typist made them. interviews are not particularly useful to select people for physically for predicting job performance. demanding jobs, such as police c. Present the errors to the officer, firefighter or warehouse typist, ask him or her to An important characteristic of any worker.33 These assessments require make the corrections and effective interview is that it provides explain to those at the standardized rating criteria to help candidates to perform actual work meeting that the report is interviewers judge the quality tasks to determine whether they can still being typed. and effectiveness of candidate meet the physical requirements of a responses. Interviewers must learn job. The tests are often scored on a d. Present your report at to conduct an interview properly, the meeting without pass/fail basis and are often timed. probe for additional information and mentioning the errors. Physical ability tests are specific to apply the rating criteria accurately. Notify everyone of the work activities, whereas physical corrections after the Structured behavioral interviews fitness tests measure more general meeting. can assess almost any aspect of overall physical health, strength and competency, but employers most endurance.

Structured Behavioral Interview Question and Sample Rating Criteria

Tell me about a time when you were able to establish rapport with someone in a situation that made it difficult to do so. What were the circumstances? What did you do? What were the results?

Low Moderate High 1 3 5

Made minimal attempts to Attempted to understand the other Effectively reached out and sought understand the person’s person’s perspective. Developed a to understand the person’s views. perspective. Developed only a positive working relationship with Developed an extremely positive surface-level relationship in a the person in a moderately difficult relationship with the person in an simple situation. situation. extremely difficult situation.

8 Choosing Effective Talent Assessments to Strengthen Your Organization

Sample Physical Ability Tests HOW TO SELECT THE RIGHT ●● Candidate reactions: Do ASSESSMENT METHOD candidates react positively or Firefighter negatively to the assessment? In this section, we provide ●● Climb a ladder while guidelines to help human resource The results, explained in the carrying equipment. professionals make informed table below, demonstrate that ●● Drag a hose. decisions about assessments. We no simple formula for choosing will look at four important criteria, the right assessment exists ●● Carry a person down from a because each type measures building. and the answers will be based on solid research over time:34 specific aspects, and each has advantages and disadvantages. Warehouse Worker ●● Validity: Is the assessment Often, a combination of ●● Lift materials of weight that method useful for predicting job assessments works well to must be lifted on the job. performance? optimize results based on your ●● Drag materials for distances ●● Adverse impact: Do members organization’s specific goals required on the job. of protected groups, including and needs. Assessments can minorities, women and employees be phased in—using a shorter ●● Roll barrels. over age 40, systematically score one first, to screen out the most ●● Carry materials the distance lower than others? unqualified candidates, and then they must be carried on the using more complex methods for ●● Cost: What is the expense of job. higher-potential candidates. developing, administering and maintaining the assessment?

Evaluation of Assessment Methods on Four Key Criteria Costs to Develop/ Candidate Assessment Method Validity Adverse Impact Administer/ Reactions Maintain Cognitive ability High High (minorities) Low-high/low/high Somewhat favorable Personality Low-moderate Low Low-high/low/low More favorable if job-relevant Integrity Moderate-high Low Low/low/low Less favorable Physical fitness Moderate to high High (females and High/high/low More favorable over age 40) Biographical data Moderate Low-high for High/low/moderate More favorable if different types job-relevant Job knowledge High High (minorities) Low-high/low/high More favorable Writing High High (minorities) High/low Somewhat favorable Situational judgment Moderate Moderate High/low/moderate More favorable (minorities) Behavioral interviews High Low High/high/low More favorable Work simulations High Low-high, depends High/high/high More favorable on what is measured Assessment centers Moderate to high Low-high, depends High/high/high More favorable on what is measured Physical ability Moderate to high High (females and High/high/low More favorable those over age 40)

9 SHRM Foundation’s Effective Practice Guidelines Series

What is an assessment Validity: The Most Important Consideration center, and when do Does the assessment predict how individuals will perform on the job? you need one? An assessment center Criterion Validity offers a specific type of ●● Determines the strength of the relationship between scores on an work simulation, usually assessment and job performance effectiveness, tenure or other related to executive, important outcomes. managerial, or supervisory ●● jobs. Candidates respond Validity is measured using a correlation coefficient, which ranges to situations, problems, and from 0.0 to 1.00. tasks in role-play exercises, ●● Assessment validities typically range from .10 to .40. “in-box exercises” that ●● A minimum validity to target is .25 or above. ask them to prioritize and make decisions based Content Validity on incoming information, case exercises, and group ●● Involves providing evidence that an assessment is a direct measure discussion. Video and voice of work behaviors. recordings mirror what ●● Based on expert judgment. candidates see and hear on the job. More sophisticated options, such as branching You may find the array of between scores on an assessment role-plays, can automate possibilities requires you to and important job outcomes. For personal communication seek advice from an assessment example, assume that a sales job by providing adaptive expert. Below we explain some requires exceptional customer responses based on what of the critical information to keep service. If the assessment is a the candidate does or says. in mind. valid predictor of job performance, In some cases, assessment candidates who score higher on Validity centers can also be used customer service will perform for non-managerial job Validity means that an better, with higher sales and higher candidates. “Day in the assessment is job-relevant and customer loyalty ratings, and those life” simulations ask job useful for predicting performance. who score lower will not perform as seekers to handle scenarios Clearly, this is the most important well. To establish criterion validity, in workflows similar to consideration in selecting an a firm would evaluate the strength what happens on the job. assessment. The two most of the relationship between These simulations provide common types of validity human performance on the assessment resource professionals encounter robust information for those and job performance on a scale of are criterion and content validity.35 making hiring decisions 0 to 1.00—although scores would for the candidates. As they Criterion validity involves never reach a perfect 1.00 because become easier, cheaper, demonstrating a useful relationship so many other factors are at play. and more flexible, use of simulations may increase over time, especially in Tip: Look at Validation Studies latter stages of a hiring In considering the use of commercially available assessments, HR process when people professionals should look for how many criterion validity studies tend to be more willing to have been conducted on the assessments and meta-analysis results, invest in a longer, in-depth if available. The more successful validation studies a vendor can commitment. produce from other organizations and the larger the number of individuals included in validity studies, the more confidence you can have in the quality of the assessment.

10 Choosing Effective Talent Assessments to Strengthen Your Organization

Example of Criterion Comparisons of criterion and Looking at the Validity Studies content validity reveal that Numbers: Effect Size ●● Administer the assessment criterion validity is stronger and and Adverse Impact to be validated to a large tends to be viewed as more An effect size can be calculated group of employees (at least credible. However, content to compare the assessment 300). validation is perfectly acceptable scores between two groups. for demonstrating the job Effect sizes range from 0, ●● Collect measures of relevance of work samples, indicating no difference between these same employees’ simulations and other direct groups, to 1.00 or more, performance on the job. measures of work behavior. indicating a large difference. ●● Correlate the assessment Criterion validity can be used to Effect sizes less than .20 are scores with the job validate any type of assessment considered small, and those performance scores, using a and is the only type of validity that near .80 or more are considered correlation coefficient. can be used when an inference large, signaling a likely adverse effect. Understanding effect ●● The correlation is the needs to be made about the strength of the relationship relevance of an assessment sizes is useful because they are between the assessment for predicting job performance often reported in test manuals and job performance. instead of using a direct and research reports evaluating measure of performance. Most assessments. assessments, including cognitive After multiple validation studies ability, personality, bio data and quality that they do not enable have been conducted for a given integrity, require inferences about successful validation. assessment type, summarize the their relevance for predicting results across studies, and make performance, so criterion The bottom line is that criterion a stronger statement about the validation is the most common validity studies are so complex true validity of the assessment. form of validity. that conferring with an assessment In the table above, we analyzed expert is best if you need to do Successful criterion validation validation studies and used them criterion validation work. to identify “low” (.20 or less), relies on having performance or “moderate” (.20 to .40) and “high” other such outcome measures Adverse Impact (.40 to .50) validity. that accurately distinguish employees’ effectiveness on the Adverse impact is a concept Assessments with low-to-moderate job. Performance ratings that defined by the U.S. government validities can add value, especially are available in human resource to establish evidence of when they are used in combination information systems (HRIS) discrimination. It occurs when with other assessments. For or performance management the results of an assessment or example, cognitive ability and systems are often of such poor hiring lead to a disproportionately personality are largely unrelated, but, cumulatively, they combine to produce higher validity than when Adverse Impact used alone. An assessment has adverse impact if it results in hiring proportionately fewer protected group members than majority group Content validity involves members. demonstrating that an assessment provides a direct measure of critical work activities Calculating Adverse Impact or behavior. Comprehensive job The 80 Percent Rule: analysis information and expert ●● 50 females are assessed and 30 females are hired, or 60 percent. judgment are used to develop ●● 50 males are assessed and 45 males are hired, or 90 percent. work simulations, ensuring that ●● The ratio of females to males hired is 67 percent. the assessment closely mirrors ●● A ratio of < 80 percent is evidence of adverse impact. job requirements.

11 SHRM Foundation’s Effective Practice Guidelines Series

small number of protected group evaluated. The U.S. Office of Federal Assessment Utility members (e.g., racial minorities, Contract Compliance Programs females, individuals over age 40) uses an alternative approach, one What are the business being selected. If the proportion to which commercial businesses of outcome benefits of of protected group members a certain size, with federal contracts, using assessments? must comply. With the exception hired is less than 80 percent of Selection Success Gains of South Africa, the concept of the proportion of majority group ●● Number or proportion of adverse impact is not widely members hired—referred to as the successful candidates used outside the United States, 80 percent rule—adverse impact before and after but other countries have specific is suspected. If an assessment implementing a valid laws, regulations and compliance produces an adverse impact, the assessment requirements that govern hiring only way to defend its use is by practices and employee rights. showing that it is a valid predictor Business Outcome Gains of performance and that alternative Adverse impact does not only ●● The financial measures are not feasible. happen in the assessment process. consequences of using a Resume screening, reference valid assessment The 80 percent rule is not the only checks, interviews and recruitment ●● Revenue and profitability way adverse impact is defined or practices can also result in an increases compared to adverse impact against protected assessment costs groups. HR teams should not shy Strategies to Reduce away from using assessments just Adverse Impact to avoid adverse impact, especially strategies include providing test 38 What To Do given the improved performance preparation sessions for candidates and business outcomes that proven and removing test items that have ●● Recruit more qualified larger group differences.39 While candidates from protected assessments provide. some strategies have shown groups. Tradeoffs Between Validity promise, elimination of adverse ●● Assess a wider set of and Adverse Impact impact entirely is still a challenge competencies, such as and is something that organizations All organizations want the best- contextual performance (e.g., should monitor and manage across performing workforce, and most also citizenship, helping others, all talent management processes, want a diverse workforce that does expending extra effort), which systems and decisions. not exclude protected groups. is predicted by measures that produce minimal adverse While evidence of validity can Assessment Utility: impact (e.g., personality and be used to justify and defend Business Benefits bio data). assessments that produce an Assessment utility addresses ●● Use assessments with more adverse impact, most organizations the business benefits of using adverse impact later in want to mitigate adverse impact assessments and is helpful in the process, although this as much as possible. Some explaining the value of assessment approach is likely to help assessments that produce high levels to decision-makers who are outside only when there are many of validity also produce high adverse the human resource team. Two applicants per position. impact. A number of strategies types of utility are apparent in may reduce adverse impact assessments: selection success What Not To Do without reducing validity, including gains and business outcome gains.40 enhanced recruiting of protected ●● Administer test preparation groups, assessing a broader array Selection success gains compare programs. They may increase of competencies beyond the successful candidates before and candidate comfort but do not technical to include motivational and after assessment implementation. reduce adverse impact. behavioral factors,36 and adjusting The greater the selection success ●● Remove test items with large the placement of assessments within gain, the higher the assessment group differences. the hiring process.37 Less effective utility.

12 Choosing Effective Talent Assessments to Strengthen Your Organization

Business outcome gains are more interesting, particularly the Buy Versus Build? revenue and margin gains that result Buying Commercial Assessments from selecting higher-performing Pros: employees compared with the ●● Implementation is fast. assessment costs. The formulas used ●● Assessment quality has been demonstrated. to evaluate business outcome gains are more complicated than simply ●● Vendors provide hosting, maintenance and updating. comparing revenue/profits versus ●● Benchmarks provide talent insights. costs. They consider additional ●● Localized assessments and norms may be available that can be used factors, including how effectively the globally. assessment predicts performance, Cons: the number of applicants and tenure ●● Licensing agreements yield ongoing expense that may be prohibitive. of the selected group.41 ●● Use across situations may raise security or competitive advantage When decision-makers see the concerns. positive impacts of assessment in terms of key business outcome Building Custom Assessments metrics, the value of assessment Pros: usually becomes more apparent and ●● Full customization to the organization’s jobs and strategy is possible. compelling. ●● They present maximum relevance and potential competitive advantage. Costs ●● They have no ongoing licensing fees. Some assessment methods involve Cons: higher costs to develop, maintain ●● Ongoing costs exist to host, upgrade and maintain. and administer than others, although ●● Significant effort and data are needed to localize assessments and technology has significantly reduced develop appropriate norms. the large variability in costs. Mobile ●● Organizations need specialized expertise to build, host and maintain administration should further reduce assessments. and equalize administration costs. In ●● They are time-consuming. terms of development, automated work simulations tend to be the most costly to develop because they assessments in-house? There who are best matched to the typically require test experts to work are issues of time, cost and organization and job. with job experts and IT developers on convenience to consider in each After development, custom design and automation. Work samples case. assessments need to be that are not automated can be even Some organizations prefer custom administered and maintained, which more expensive than automated assessment content that is uniquely requires ongoing investments assessments because they involve designed and validated for their equipment, facilities and training in online hosting, maintaining specific situation and embeds evaluators to observe and score databases of test scores, analyzing the organization’s values, culture candidates. At the other extreme, and reporting data for decision- and context. Using a custom online multiple-choice assessments of making, revising items, and assessment is most important innate characteristics can usually be potentially revalidating in the when the assessment is intended developed independently by testing future. If an organization’s global to serve as part of a larger, highly experts and are highly efficient to assessments have to be adapted branded hiring experience or administer and score. for use in different countries, the as both a job preview and an firm should ensure that the content The buy-versus-build decision is assessment. Some organizations is adjusted and nuanced so key for any organization. Should believe custom assessment candidates in different cultures can you license standard assessment provides a competitive advantage understand it. Effectively managing content or develop and host custom and better identifies candidates a custom assessment program

13 SHRM Foundation’s Effective Practice Guidelines Series

requires specialized expertise ●● An overall great candidate Candidates React among in-house staff or external experience. Positively When: expert consultants. ●● Helpful information and ●● They are able to Organizations may choose to feedback. demonstrate job-relevant use commercially available ●● Assessments relevant to the job. capabilities. assessments, which assessment ●● Assessment procedures providers can implement quickly In general, candidates favor are administered and host, update and maintain. assessments that measure consistently and fairly. Commercial assessments should work behavior directly through always come with evidence methods like work samples and ●● Feedback is provided so of assessment quality across simulations rather than multiple- candidates can determine organizations. Some commercial choice tests of abstract concepts. their standing and decide assessment providers offer Cognitive assessments tend whether to proceed. benchmarking data against which to be viewed more favorably ●● They are given insight into organizations can evaluate how their than personality, integrity and 43 the nature of the job and talent stacks up in their industry or bio data assessments. Job organization. geographical area. Some providers candidates view assessment have also amassed large volumes of items aimed at diagnosing data globally, which allows them to personality or clinical disorders provide localized assessments with much less favorably than items types with excessive words appropriate norms. Some providers that measure job-relevant were viewed less favorably on can also customize products to meet personality characteristics mobile devices compared with a specific organization’s needs. like conscientiousness and graphic and image-based items. agreeableness. As mobile technology expands In setting up assessment programs, and assessments are designed to employers should seek the Candidates want information take advantage of more mobile guidance of assessment experts. about the job, the organization features, such as swiping instead and how they are doing in the In many organizations, assessment of clicking, additional research programs are executed through hiring process, including how will be needed to fully understand a combination of in-house staff they could have performed candidate reactions. and external experts or vendors. better—but this is a tricky SHRM provides comprehensive matter. Employers should and valuable information about provide as much information as LEGAL ISSUES available assessments and related is reasonable, but discussing Human resource professionals topics through the SHRM Testing assessment performance in detail who administer assessments Center (shrm.org/testing/). can divulge information that could should have a basic lead to security breaches. And understanding of the legal Candidate Reactions providing in-depth feedback for issues and requirements that govern their use. In the United As many organizations move all candidates would be extremely toward candidate-centric hiring, expensive. States, two pieces of legislation candidates’ reactions have become are especially helpful: Uniform A final topic relevant to candidate Guidelines on Employee increasingly important in judging reactions concerns views of Selection Procedures and the assessments. Candidates who have mobile assessment. Some studies Americans with Disabilities Act of a positive hiring experience report have found no differences in 1990 (ADA). higher levels of satisfaction and perceived fairness moving from are more likely to recommend a PCs to mobile devices,44 but 42 The Uniform Guidelines specify company to others. others have come to the opposite requirements for selection 45 The most critical factors conclusion. Research has systems covered under the Civil influencing applicants’ reactions shown that satisfaction increases Rights Acts and under Executive boil down to whether they believe as screen size increases, Order 11246 (see https://www. the organization provided: irrespective of the content. Item eeoc.gov/laws/regulations/

14 Choosing Effective Talent Assessments to Strengthen Your Organization

The Importance of Compliance Federal Contractors Some managers and others within your organization may believe and Subcontractors: that using informal selection methods instead of formal assessments Pay Attention to Recent will avoid legal issues and eliminate the need to comply with federal Regulatory Guidelines for regulations. This is not the case. If any aspect of the hiring process Hires with Disabilities clearly produces an adverse impact, an organization will be required Regulatory guidelines to demonstrate validity if challenged or audited. In practice, well- updated in 2013 reflect developed and validated formal assessments actually help mitigate increased emphasis on hiring legal risk and exposure. individuals with disabilities, If members of your firm view legal requirements as burdensome, let and hiring goals are now in them know that the procedures in the Uniform Guidelines are, in fact, place for federal contractors the same procedures an assessment expert would employ if the only and subcontractors, with goal were to identify the best-qualified candidates for a job. Thus, stipulations for actions compliance with legal requirements in this case is synonymous with to be taken in the areas best practices and will yield the best outcomes for any organization. of recruitment, training, record-keeping and policy dissemination. The guidelines for the full text of the Uniform accommodation in assessing that call for voluntary disclosure Guidelines). These requirements skill. However, if there are ways from job candidates to self- address the question of whether a to accommodate disabilities both identify as individuals with procedure is causing an adverse on the job and in the assessment disabilities or protected impact, and if so, which validation process, an organization must veterans. The guidelines also requirements should be met do so. Examples of reasonable require workforce surveys before using the procedure. Two accommodation include: to gather this self-reported documents can assist in applying information. The data are ●● Substituting an oral assessment the Uniform Guidelines: used to create action- for a written one. oriented programs. ●● The Society for Industrial and ●● Providing extra time to Organizational Psychology’s complete an assessment. Principles for the Validation understand and meet their ●● and Use of Personnel Selection Offering an assessment in responsibilities under the ADA. Procedures (see http://siop.org). Braille. One of the most comprehensive

●● ●● Standards for Educational Providing items in larger print. sources of information is the Job Accommodation Network (see and Psychological Testing, ●● Providing an interpreter who http://askjan.org), maintained by developed jointly by the can communicate items in sign the U.S. Department of Labor, American Educational Research language. Association, the American which provides free, expert and confidential guidance on Psychological Association Organizations increasingly are workplace accommodations and and the National Council on held accountable for ensuring disability employment issues. Measurement in Education (see that recruitment and selection http://apa.org/science/programs/ processes are fair and accessible, testing/standards.aspx). and research is slowly accumulating on the fairness of selection tools The Americans with Disabilities and accommodations for people Act created substantial with disabilities.46 For example, requirements for providing research on longer exam times reasonable accommodations to found that they generally improved applicants with disabilities. If a the performance of all students, not job requires a skill that a person just those with disabilities. with a disability is unable to perform, an organization is not Several sources offer guidance required to provide reasonable to help HR professionals

15 Validity means that an assessment is job-relevant and useful for predicting performance. Clearly, this is the most important consideration in selecting an assessment. Choosing Effective Talent Assessments to Strengthen Your Organization

CONCLUSION

In recent years, technological advances, competitive demands for talent and candidate-centric hiring have created powerful commercial forces that are driving changes in assessment methods. Many organizations are demanding shorter, more engaging candidate experiences and less obtrusive assessment methods to help them target talent. Innovative assessment techniques are also being developed. These factors, coupled with continued technological advancement, will almost certainly lead to revolutionary and rapid changes in the assessment field.

Always remember, however, that the primary criterion for evaluating assessments does not change. Managers and human resource professionals must continue to seek effective assessments that will result in talented candidates with strong performance on the job—employees who bring long-term value to the organization.

17 SHRM Foundation’s Effective Practice Guidelines Series

The buy-versus-build decision is key for any organization. There are issues of time, cost and convenience to consider in each case.

Quiderum num faccuptus es ratem res simaion necersperum la simet aut endanihicium fuga. Ut omniam aut omnimil ius doleste nderio ellabo. Ut oditatus et molorio tenis molorepel imusam. - Quotee to come

18 Choosing Effective Talent Assessments to Strengthen Your Organization

SOURCES AND SUGGESTED READINGS

JOB ANALYSIS

Brannick, M. T., & Levine, E. L. (2002). Job analysis. Thousand Oaks, CA: Sage. This is a user-friendly book about how to conduct a job analysis and use the results. Topics covered include the building blocks of job analysis methods, work (task)-oriented methods; worker knowledge, skills and abilities (KSA) methods; hybrid methods that combine both task and KSA job analysis; job analysis for management jobs and teams; job analysis and the law; doing a job analysis study; and uses of job analysis in job descriptions, performance appraisals, job evaluation, job design, staffing and training.

Gael, S. (Ed.). (1988). The job analysis handbook for business, industry, and government. New York, NY: Wiley. The author argues that job analysis is the hub of virtually all human resource administration and management activity and necessary for the successful functioning of organizations. The handbook addresses job analysis topics and issues comprehensively, with material relevant to a wide audience of practitioners working in business, industry, labor unions, universities, the military, and federal, state and local governments.

UNPROCTORED AND MOBILE ASSESSMENT

Beaty, J. C., Dawson, C. R., Fallaw, S. S., & Kantrowitz, T. M. (2009). Recovering the scientist-practitioner model: How IOs should respond to UIT. Industrial and Organizational Psychology: Perspectives on Science and Practice, 2, 58-63. This commentary raises organizational examples and mitigating factors associated with unproctored Internet testing (UIT) in three areas: cheating, ethics and legal concerns. The authors encourage I/O practitioners to execute a research agenda associated with UIT.

19 SHRM Foundation’s Effective Practice Guidelines Series

Beaty, J. C., Nye, C., Borneman, devices or applicant demographic three industries—manufacturing, M., Kantrowitz. T. M., Drasgow, subgroups. customer service/retail and health F., & Grauer, E. (2011). Proctored care. versus unproctored Internet King, D., Ryan, A. M., Kantrowitz, tests: Are unproctored tests as T. M., Grelle, G., & Dainis, A. (in Landers, R. N., Reddock, C. M., predictive of job performance? press). Mobile Internet testing: Cavanaugh, K. J., & Proaps, A. J. International Journal of Selection An analysis of equivalence, (2014). Talent assessment using and Assessment, 19, 1-10. individual differences, and mobile devices. In T. Kantrowitz This paper examines the criterion- reactions. International Journal & C. M. Reddock, Shaping the related validity of unproctored of Selection and Assessment, 4, future of mobile assessment: assessments. Results generally 382-394. Research and practice update. indicate that assessments This paper examines the Symposium presented at the administered in proctored and equivalence of mobile Internet 29th Annual Conference of unproctored settings have similar testing (MIT) compared with the Society for Industrial and validities. testing on personal computers Organizational Psychology, (PCs) and whether attitudes Honolulu, HI. Gutierrez, S. L., & Meyer, J. M. and other individual differences The study consisted of 219 (2014). The mobile revolution: influence responses and individuals who completed five Measurement equivalence and reactions to MIT. Results commercially available and mobile device administration. In demonstrated equivalence for a previously validated assessments T. Kantrowitz & C. M. Reddock, supervisory situational judgment from a large human resource Shaping the future of mobile test across testing modes, consulting firm across two time assessment: Research and but not for a cognitive ability points three weeks apart. The practice update. Symposium test. Respondents reported authors found relatively small presented at the 29th Annual significantly more positive differences in magnitude that Conference of the Society for reactions when tested on a PC were limited to cognitive-type Industrial and Organizational versus a mobile device. measures. Psychology, Honolulu, HI. This study was conducted in Kinney, T. B., Lawrence, Morelli, N. A., Mahan, R. P., partnership with a client organization A., & Change, L. (2014). & Illingworth A. J. (2014). in the customer service field, with Understanding the mobile Establishing the measurement the objective of obtaining empirical experience. Data across device equivalence of online selection support for the organization to and industry. In T. Kantrowitz assessments delivered on deliver its pre-employment tests via & C. M. Reddock, Shaping the mobile versus non-mobile mobile devices. future of mobile assessment: devices. International Journal of Research and practice update. Selection and Assessment, 22, Illingworth, J., Morelli, N., Scott, Symposium presented at the 124-138. S., & Boyd, S. (2014). Internet- 29th Annual Conference of This study examines the based, unproctored assessments the Society for Industrial and measurement equivalence of on mobile and non-mobile Organizational Psychology, selection assessments delivered devices: Usage, measurement Honolulu, HI. on mobile and non-mobile equivalence, and outcomes. This study examined measurement devices. Measurement invariance Journal of Business and equivalence, pass rate differences tests conducted with multigroup Psychology, 30, 25-34. and applicant reactions by device confirmatory factor analysis This study demonstrates that type (smartphone, tablet, PC) suggest that mobile versions of a the use of mobile and non- for several mobile assessments; cognitive ability-type assessment, mobile devices did not produce investigated the factors in mobile two bio data assessments, a any practically significant score users’ test environment; and multimedia work simulation and a differences on assessment across conducted analyses across text-based situational judgment

20 Choosing Effective Talent Assessments to Strengthen Your Organization

test appear to be equivalent to A review of the research literature PERSONALITY ASSESSMENTS non-mobile versions. However, indicates that general cognitive mobile device user latent means ability (GCA) predicts supervisor Barrick, M. R., & Mount, were half a standard deviation ratings and training success M. K. (1991). The big five lower than their non-mobile as well as objective, rigorously personality dimensions and job counterparts for the situational content-valid work sample performance: A meta-analysis. judgment test. performance. The author argues Personnel Psychology, 91, 1-26. that the findings from this study This study examines the validity Tippins, N. T. (2009). Internet support classic learning theory of five personality measures alternatives to traditional over behaviorist theories of for predicting performance proctored testing: Where learning and performance. in five occupational groups are we now? Industrial and (professionals, police, managers, Organizational Psychology: Hunter, J., & Hunter, R. F. (1984). sales, and skilled or semi- Perspectives on Science and Validity and utility of alternative skilled). Three different kinds Practice, 2, 2-10. predictors of job performance. of performance measures were This focal article explores Psychological Bulletin, 96, 72-98. used in this study: job proficiency, Internet alternatives to traditional, An examination of research on training proficiency and personnel proctored testing. It provides a predictors of job performance data. One of the personality comprehensive review of the showed that for entry-level jobs measures—conscientiousness— definition and history of UIT, the cognitive ability was the most was consistently related to all factors that contribute to the rise valid predictor. For selection three performance measures for of UIT and the mitigating factors on the basis of current job all of the occupational groups. that pose risks to organizations performance, the work sample Extraversion was a valid predictor that use UIT. test was slightly better. The for two occupations involving authors suggested that using social interaction—managers and other predictors in conjunction sales. Openness to experience COGNITIVE ABILITY TESTS with ability tests might improve and extraversion were valid validity and reduce adverse predictors of the training Gottredson, L. S. (1986). Societal impact. proficiency criterion. Overall, consequences of the g factor the results illustrate the benefits in employment. Journal of of using the five-factor model Vocational Behavior, 29, 379- Ree, M. J., Earles, J. A., of personality in assessment 410. & Teachout, M. S. (1994). research. This article discusses and rebuts Predicting job performance: Not seven common arguments that much more than g. Journal of Costa, P. T., Jr., & McCrae, R. R. general intelligence (g) is of Applied Psychology, 79, 518-524. (1992). Four ways five factors little or no practical importance The authors investigated the roles are basic. Personality and in employment and illustrates of general cognitive ability (g) and Individual Differences, 13, 653- the effect that differences in specific abilities or knowledge 665. intelligence in a workforce (s) as predictors of work sample A considerable number of may have on the structure and job performance criteria in seven research studies have shown functioning of whole societies. jobs for U.S. Air Force enlistees. Analyses reveal that g was the support for a five-factor model best predictor and that s added of personality traits. These Hunter, J. (1986). Cognitive a statistically significant but include conscientiousness, ability, cognitive aptitudes, practically small amount to this extraversion, agreeableness, job knowledge, and job prediction. openness to experience and performance. Journal of emotional stability. The claim Vocational Behavior, 29, 340- that these factors represent the 362. basic dimensions of personality is

21 SHRM Foundation’s Effective Practice Guidelines Series

based on four lines of reasoning: Raymark, P. H., Schmit, M. J., & An even higher mean validity was Longitudinal and cross-observer Guion, R. M. (1997). Identifying obtained based on studies using studies have demonstrated potentially useful personality job analysis explicitly in selection that all five factors are enduring constructs for employee of personality measures. dispositions that are manifested selection. Personnel Psychology, in patterns of behavior; traits 50, 723-736. related to each of the factors This article describes the BIO DATA TESTS have been found in a variety Personality-Related Position of personality systems and in Requirements Form (PPRF), Hough, L. M. (1984). the natural language of trait a job analysis form used in Development and evaluation description; the factors have been making hypotheses about of the “accomplishment found in different age, sex, race what personality predictors record” methods of selecting and language groups, although will be relevant for predicting and promoting professionals. they may be somewhat differently performance in different jobs. Journal of Applied Psychology, expressed in different cultures; The Big Five personality factors 69, 135-146. and evidence of heritability has provided an organizing framework This study describes suggested that all have some for the PPRF. Subsequent the development of the biological basis. development resulted in “accomplishment record” method, identifying 12 specific sets of which resulted in self-reported descriptions of accomplishments Hough, L. M. (1992). The big five items for facets of each of the Big Five. in areas that were highly relevant personality variables—construct for making selection or promotion confusion: Description versus decisions. The accomplishments prediction. Human Performance, Tett, R. P., Jackson, D. N., & were reliably rated using specially 5, 135-155. Rothstein, M. (1991). Personality prepared rating scales and This article suggests that the five- measures as predictors of job guidelines. To test the validity factor model of personality traits performance: A meta-analytic of the accomplishment record (extraversion, conscientiousness, review. Personnel Psychology, method, 329 attorneys prepared agreeableness, emotional 44, 703-742. accomplishment records according stability and openness to This study examines past research to specified instructions and also experience) is not an adequate to a) assess the overall validity completed a battery of more taxonomy of personality variables of personality measures as typical assessment methods. for predicting important job predictors of job performance; b) Scores on the accomplishment performance measures. The investigate moderating effects record inventory were unrelated author argues that the five of several study characteristics to the traditional assessment factors are too heterogeneous on personality scale validity; and measures (e.g., aptitude tests, and incomplete. This paper first c) investigate the predictability grades and honors), but they discusses the development of of job performance as a function correlated with job performance. personality and job performance of eight personality measures: The accomplishment record had models. Comparisons between neuroticism, extraversion, also been found to correlate with different taxonomies are openness, agreeableness, self-perceptions of success, hard made, and the author presents conscientiousness, locus of work and self-assurance and with correlational evidence control, Type A and miscellaneous. the length of time spent practicing demonstrating the usefulness Consistent with predictions, a profession. of nine personality measures studies using confirmatory (affiliation, potency, achievement, research strategies produced dependability, adjustment, corrected mean personality scale Hough, L. M., Keyes, M. A., agreeableness, intelligence, validity that was more than twice & Dunnette, M. D. (1983). An rugged individualism and locus of as high as that based on studies evaluation of three ‘alternative’ control). adopting exploratory strategies. selection measures. Personnel Psychology, 36, 261-276.

22 Choosing Effective Talent Assessments to Strengthen Your Organization

The authors used a content- the public and private sectors used for pre-employment oriented validation strategy to (including selection of blue-collar selection decisions about the develop three alternative selection and federal government employees trustworthiness of job applicants. inventories that were designed to and work team members), career reflect the job content of positions counseling and development, and Ones, D. S., Viswesvaran, held by attorneys employed job classification. C., & Schmidt, F. L. (1993). with a large federal agency. Comprehensive meta-analysis of These inventories and three INTEGRITY TESTS integrity test validities: Findings traditional assessment methods and implications for personnel were completed by 329 agency selection and theories of job attorneys as part of a concurrent Camara, W. J., & Schneider, D. L. (1994). Integrity tests: Facts and performance [Monograph]. validation study. Criterion-related Journal of Applied Psychology, validities of two traditional unresolved issues. American Psychologist, 49, 112-119. 78, 679-703. inventories (a background inventory The authors conducted a and an interest and opinion This article describes two independent reports, completed comprehensive examination inventory) and one alternative of the validity of integrity tests inventory (an accomplishment by the American Psychological Association (APA) and the U.S. based on 665 validity coefficients. record inventory) were statistically Results indicated that integrity and practically significant. Congress Office of Technology Assessment, that examined test validities were substantial for scientific and measurement issues predicting job performance and Shoenfeldt, L. F. (1999). From concerning integrity testing. Data counterproductive behaviors on dustbowl empiricism to rational were presented on a variety the job, such as theft, disciplinary constructs in bio data. Human of tests collected by surveying problems and absenteeism. Results Resource Management Review, test publishers. The survey data of validity studies conducted 9, 147-167. provided a view of the industry’s on applicants using external This paper presents an attempt to scope (e.g., test audience, user performance measures (i.e., use different types of biographical screening, score reporting) that excluding self-reports) indicated data scales in a concurrent was not available elsewhere. The that integrity tests predicted the validation to predict service article also addresses unresolved broad criterion of organizationally orientation. Several types of issues regarding integrity testing. disruptive behaviors better than scales showed positive validity they predicted employee theft results with the performance alone. The authors concluded that measures. The results were Goldberg, L. R., Grenier, R. M., integrity test validities were positive promising in terms of both the Guion, L. B., Sechrest, L. B., across situations and settings. prediction and understanding of & Wing, H. (1991). Questions customer service orientation. used in the prediction of trustworthiness in pre- Sackett, P. R., & Wanek, J. E. employment selection decisions: (1996). New developments in Stokes, G. S., Mumford, M. An APA Task Force report. the use of measures of honesty, D., & Owens, W. A. (Eds.). Washington, DC: American integrity, conscientiousness, (1994). Biodata handbook: Psychological Association. dependability, trustworthiness, Theory, research, and use of This report presents the findings and reliability for personnel biographical information in of an APA task force that was selection. Personnel Psychology, selection and performance appointed to review available 49, 787-829. prediction. Palo Alto, CA: CPP commercial tests used for the This article reviews journal Books. purpose of assessing the honesty articles, books, book chapters, This book is a comprehensive and integrity of prospective law review articles, convention reference that provides information employees. The report focuses papers and dissertations for understanding and using exclusively on commercially regarding integrity testing biographical data in selection in published instruments for personnel selection.

23 SHRM Foundation’s Effective Practice Guidelines Series

Developments include an 4) have a panel record and rate and decision-making processes. examination of professional and answers; 5) administer the process Next, they discussed more congressional inquiry into this to all candidates; and 6) emphasize recent developments in interview area of testing, rapid growth job relatedness, fairness and research, such as the use of of the validity database, new documentation. When the interview the interview as a means of insight into similarities and was used to hire 149 entry-level assessing person-organization differences between different production employees, it revealed fit and applicant reactions to the tests, and links to the Big Five high interrogator reliability and employment interview. personality dimensions. Inquiries predictive validity as well as into relationships with other evidence for test fairness and utility. constructs are reviewed, as are PHYSICAL FITNESS AND PHYSICAL ABILITIES TESTS applicant reactions to these tests. Eder, R. W., & Ferris, G. R. (Eds.). (1989). The employment Blakley, B. R., Quinones, M. S., STRUCTURED INTERVIEWS interview: Theory, research, and Crawford, M. S., & Jago, I. A. practice. Newbury Park, CA: (1994). The validity of isometric Campion, M. A., Palmer, D. Sage Publishing. strength tests. Personnel K., & Campion, J. E. (1997). This book provides a Psychology, 47, 247-274. A review of structure in the comprehensive treatment of This article examines six studies selection interview. Personnel theory, research and practice in which isometric strength Psychology, 50, 655-702. relevant to the employment tests were used as part of This article reviews the research interview. The topics covered selection procedures. The study literature on the many ways include the interviewer’s focuses on the relationship selection interviews can be decision-making process, between performance on four structured. Fifteen components applicant strategies and isometric strength tests and both of structure were identified that employment interview validity. supervisory ratings of physical may enhance either the content There is also a concluding performance and performance on or the evaluation process of the commentary summarizing the work simulations. Results indicate interview. The authors conclude volume’s implications for theory that isometric strength tests were that interviews could be easily building, research methods and valid predictors of both types of enhanced by using the various effective practice. performance measures across all components of structure. They of the jobs examined. recommend that improvement of Judge, T. A., Higgins, C. A, this popular selection procedure & Cable, D. M. (2000). The Campion, M. A. (1983). Personnel be a high priority for future employment interview: A selection for physically research and practice. review of recent research and demanding jobs: Review and recommendations for future recommendations. Personnel Campion, M. A., Pursell, E. D., & research. Human Resource Psychology, 36, 527-550. Brown, B. K. (1988). Structured Management Review, 10, 383- This review article argues that interviewing: Raising the 406. improvements in personnel psychometric properties of the This comprehensive review selection systems for physically employment interview. Personnel examines research on the demanding jobs are needed due Psychology, 41, 25-42. employment interview process. to EEO considerations, concern This article proposes an employee- The authors began with a for worker physical well-being interviewing technique that review of the traditional areas and the lack of alternative includes the following steps: 1) of interview research: reliability, procedures. develop questions based on validity, structured interviews, a job analysis; 2) ask the same interviewer differences, equal questions of each candidate; 3) employment opportunity (EEO) use examples and illustrations; issues, impression management

24 Choosing Effective Talent Assessments to Strengthen Your Organization

Hogan, J. (1991). Physical and-pencil instrument. The results cognitive ability, job experience abilities. In M. D. Dunnette & L. show that the racial differences and conscientiousness, in the M. Hough (Eds.), Handbook of in situational judgment test prediction of job performance. industrial and organizational performance and face validity The SJI was a valid predictor in psychology (Vol. 2, pp. 753- reactions to the test were three samples and produced 831). Palo Alto, CA: Consulting substantially smaller with the substantial increases in validity in Psychologists Press. video-based method of testing two samples. Relative to the other This chapter identifies factors compared with the paper-and- predictors, SJIs’ correlation with that affect the ability to perform pencil method. performance, controlling for the in the workplace, including other predictors, was superior in physiological responses, training, Motowidlo, S. J., Dunnette, most comparisons. The authors ability and injury, and workload. It M. D., & Carter G. (1990). An concluded that the SJI should reviews procedures and methods alternative selection procedure: prove to be a valuable additional for designing personnel selection A low fidelity simulation. Journal measure in the prediction of systems for physically demanding of Applied Psychology, 75, 640- job performance, but they also jobs and emphasizes the unique 647. suggested several additional physical ability content of job From critical-incident analysis areas of research. analyses, performance measures and judgments by subject-matter and assessments. In addition, experts, a low-fidelity simulation WORK SAMPLE TESTS the chapter argues that the need was developed for selecting to improve development and entry-level managers in the Asher, J. J., & Sciarrino, J. A. implementation of fair physical telecommunications industry. ability selection procedures (1974). Realistic work sample The simulation presented tests: A review. Personnel and the need to understand applicants with descriptions the relationship between Psychology, 27, 519-533. of work situations and five This study reviews the validity physical performance and other alternative responses for each organizational effectiveness evidence for a wide sample of situation. It asked them to select motor and verbal work sample outcomes are important issues for one response they would most future research. tests, which were designed to likely make and one they would measure on-the-job behaviors. least likely make in addressing Motor tests were shown to have SITUATIONAL each situation. The results show higher validities than verbal JUDGMENT TESTS that samples of hypothetical tests when job proficiency was work behavior can predict the performance measure, but Chan, D., & Schmitt, N. (1997). performance, without the props, this pattern was reversed when Video-based versus paper and equipment or role players success in training was the pencil method of assessment often required by high-fidelity performance measure. These in situational judgment tests: simulations, such as work-sample validities were somewhat lower Subgroup differences in test tests or assessment centers. than those for biographical performance and face validity information but higher than those perceptions. Journal of Applied Weichmann, D., Schmitt, N., & for other predictors. Psychology, 82, 143-159. Harvey, V. S. (2001). Incremental This study examines the effects validity of situational judgment Gaugler, B. B., Rosenthal, D. B., of race, reading comprehension, tests. Journal of Applied Thornton, G. C., III, & Bentson, method of assessment, face Psychology, 86, 410-417. C. (1987). Meta-analyses of validity perceptions and Using three different samples, assessment center validity. performance on a situational the authors assess the increases Journal of Applied Psychology, judgment test. The situational in the validity produced by 72, 493-511. judgment test was administered situational judgment inventories This paper examines the validity via a videotape and via a paper- (SJIs), relative to job knowledge, of assessment centers across

25 SHRM Foundation’s Effective Practice Guidelines Series

a large number of studies. Thornton, G. C., III. (1992). ADVERSE IMPACT Higher validities were found Assessment centers in human in studies in which ratings of resources management. Boston, Roth, P. L., Bevier, C. A., Bobko, potential were the performance MA: Addison-Wesley. P., Switzer, F. S., & Tyler, P. (2001). measures, and lower validities While assessment centers Ethnic group differences in were found in studies in which have been used for numerous cognitive ability in employment the assessment center was used human resource management and educational settings: A meta- to make promotion decisions. functions, all assessment centers analysis. Personnel Psychology, The authors also found that the are not alike. This book shows 54, 297-330. validities were higher when a) how assessment for managerial The authors conducted a meta- the percentage of females who selection/promotion, diagnosis analytic review on factors were being assessed was high, b) of managerial training needs underlying the one standard when several evaluation devices and team building among deviation effect size difference were used, (c) when assessors managerial groups are done observed in previous research were psychologists rather differently. The author also shows for Black-White differences for than managers, d) when peer how the assessed dimensions, college application tests (e.g., evaluation was used, and e) when observations and participant SAT) and overall analyses of the study was methodologically feedback all must be tailored to tests of g for job applicants sound. the specific application of the in corporate settings. They assessment center method. determined that the one standard Howard, A. (1983). Work deviation summary of group samples and simulations Thornton, G. C., III, & Byham, W. differences fails to capture many in competency evaluation. C. (1982). Assessment centers of the complexities in estimating Professional Psychology: and managerial performance. ethnic group differences in Research and Practice, 14, 780- New York, NY: Academic Press. employment settings. 796. This book focuses on This article argues that the examining the assessment Sackett, P. R., & Ellingson, J. E. evaluation of professional center experience. It traces the (1997). The effects of forming competence in psychology—by historical development of multiple multi-predictor composites on licensing boards, the American assessment procedures with group differences and adverse Board of Professional Psychology emphasis on those advances impact. Personnel Psychology, or other boards—has not taken relevant to assessment centers; 50, 707-721. full advantage of expertise in critiques all of the published It is commonly assumed that psychological measurement and unpublished research including predictors that within the profession. The on assessment centers; demonstrate smaller group advantages and disadvantages integrates assessment center differences with others that of work samples and simulations procedures into several theories demonstrate larger group for such purposes are discussed of measurement and human differences will help alleviate the and compared with those of judgment; and presents new adverse impact observed. The paper-and-pencil tests. Examples models of job analysis, the purpose of this paper was to from various professions and nature of managerial work, answer the question, “If two or from psychological research work-sampling assessment more predictors that have smaller in personnel selection are methods and the process of and larger group differences presented. A comparison of work human judgment based on the are combined, what will be the samples with simulations shows assessment center experience. magnitude of group differences greater advantages for the latter, and, consequently, of adverse but a combination of exercises in impact?” A number of conclusions an assessment center model is were drawn that clarified the recommended. extent to which combining

26 Choosing Effective Talent Assessments to Strengthen Your Organization

predictors with smaller and group norming fared in terms of reactions more consistently than larger group differences affected tradeoffs between performance procedural justice perceptions. subsequent adverse impact. and minority representation. Procedural justice perceptions The results showed that the explained incremental variance in Sackett, P. R., Schmitt, N., preferred strategy depended on some analyses after the influence Ellingson, J. E., & Kabin, M. B. the relative value the organization of outcome favorability was (2001). High stakes testing in placed on performance versus controlled. employment, credentialing, and minority representation. Results higher education: Prospects in also showed that the effects Gilliland, S. W. (1995). Fairness a post affirmative-action world. of different screen-then-select from the applicant’s perspective: American Psychologist, 56, 302- selection strategies varied as Reactions to employee selection 318. a result of the selection ratios procedures. International Cognitively loaded tests of at the screening and selection Journal of Selection and knowledge, skill and ability often stages, thus precluding simple Assessment, 3, 11-19. contribute to decisions regarding conclusions about the merits of Applicant reactions to selection education, jobs, licensure or each selection strategy. procedures were examined in certification. Users of such terms of the satisfaction and/or tests often face difficult choices APPLICANT REACTIONS violation of 10 procedural justice when trying to maximize both rules. The author first collected the performance and ethnic Bauer, T. N., Maertz, C. P., 237 critical incidents describing diversity of chosen individuals. Dolen, M. R., & Campion, M. A. fair and unfair treatment during The authors describe the nature (1996). Longitudinal assessment selection processes from 31 of this quandary, review research of applicant reactions to individuals who had recently on different strategies to address employment testing and test participated in job search and it and recommend using selection outcome feedback. Journal of hiring. The critical incidents were materials that assess the full Applied Psychology, 83, 892- categorized into 10 procedural range of relevant attributes using 903. justice rules, and the distribution a format that minimizes verbal This study uses a justice of these incidents was examined content as much as is consistent framework to examine reactions for different hiring outcomes and with the outcome one is trying of candidates for an entry-level different selection procedures. to achieve. Regardless of the accounting job at three time The most common procedural strategy adopted, however, they periods: before testing, after concerns reflected selection suggest that it is unreasonable testing but before feedback procedure job relatedness and to expect that one can maximize on whether they passed or the interpersonal treatment both the performance and ethnic failed the test, and after-test applicants had received. diversity of selected individuals. performance feedback. Several procedural justice measures Hausknecht, J. P., Day, D. Sackett, P. R., & Roth, L. (1996). (information known about the V., & Thomas, S. C. (2004). Multi-stage selection strategies. test, chance to perform, treatment Applicant reactions to selection A Monte Carlo investigation at the test site, consistency procedures: An updated model of effects on performance of the test administration and and meta-analysis. Personnel and minority hiring. Personnel job relatedness) predicted Psychology, 57, 639-683. Psychology, 49, 549-562. applicant evaluations regarding An updated theoretical model of This study examines the effects the organization, perceptions of applicant reactions to selection of selection process variables employment testing and applicant procedures is proposed and on employee performance and test-taking self-efficacy. Test tested in this article. Results on minority hiring. The authors outcome favorability (passing indicate that applicants who look at how selection rules that or failing the employment test) held positive perceptions about did and did not include within- predicted outcomes beyond initial selection were more likely to

27 SHRM Foundation’s Effective Practice Guidelines Series

view the organization favorably to candidates and to selection includes a review of research and report stronger intentions to procedure validity and utility. In bearing on the accommodation accept job offers and recommend part one of this two-part study, 110 of tests and testing procedures the employer to others. The newly hired entry-level managers for examinees with disabilities average correlation between and 44 recruiting-employment and presents the experiences applicant perceptions and gender, managers judged simulations, of two large eastern states and age and ethnic background interviews and cognitive tests their programs of accommodation was near zero. Face validity and with relatively concrete item types testing. perceived predictive validity (such as vocabulary, standard were strong predictors of many written English and mathematical Eyde, L. D., Nester, M. A., applicant perceptions, including word problems) as significantly Heaton, S. M., & Nelson, A. V. procedural justice, distributive more job-related than personality, (1994). Guide for administering justice, attitudes toward tests bio data and cognitive tests with written employment and attitudes toward selection. relatively abstract item types examinations to persons Interviews and work samples (such as quantitative comparisons with disabilities. Washington, were perceived more favorably and letter sets). A measure of DC: U.S. Office of Personnel than cognitive ability tests, which new managers’ cognitive abilities Management. were perceived more favorably was positively correlated with This paper provides guidance than personality inventories, the managers’ perceptions of for administering written honesty tests, bio data and the job relatedness of selection examinations to persons with graphology. procedures. In part two, the disabilities. It also includes reactions of 460 applicants to a information on legal requirements Ryan, A. M., Greguras, G. J., & range of entry-level to professional for reasonable accommodations Ployhart, R. E. (1996). Perceived civil service examinations were and offers detailed guidelines job relatedness of physical found to be positively related to for testing applicants who are ability testing for firefighters: procedural and distributive justice deaf or hard of hearing or who Variations in reactions. Human perceptions and willingness to have vision or motor impairments. Performance, 9, 219-240. recommend the employer to The paper includes a checklist This research explores others. for physical accessibility of test perceptions of physical sites and information on effective abilities testing by examining AMERICANS WITH personal interactions with the perceived job relatedness DISABILITIES ACT persons with disabilities. of physical abilities tests. Experience, self-efficacy and Daley, L., Dolland, M., Kraft, Williams, S. K. (2003, January/ consistency of test administration J., Nester, M. A., & Schneider, February). Tips for minimizing were found to be related to R. (1988). Employment testing abuses under the Americans perceptions of job relevance. of persons with disabling with Disabilities Act. Alexandria, conditions. Alexandria, VA: VA: Society for Human Resource Smither, J. W., Reilly, R. R., International Personnel Management. Millsap, R. E., Pearlman, K., & Management Association. This paper provides a discussion Stoffey, R. (1993). Applicant This is a monograph based of the law, its interpretation reactions to selection on a symposium presented and implications for use of procedures. Personnel at both the 1987 International various selection techniques in Psychology, 46, 49-76. Personnel Management relation to the Americans with The premise of this research Association Assessment Council Disabilities Act of 1990. It covers is that applicant reactions to (IPMAAC) Conference and the physical exams, psychological selection procedures may 1987 International Personnel exams, health questionnaires, be of practical importance to Management Association (IMPA) physical agility tests, drug and organizations’ attractiveness International Conference. It alcohol tests, specific oral and

28 Choosing Effective Talent Assessments to Strengthen Your Organization

written questions (e.g., about the Uniform guidelines on employee in two top-tier journals (Journal existence of a disability, about selection procedures. (1978). of Applied Psychology and ability to perform job-related Federal Register, 43, 38295- Personnel Psychology) from 1980 functions), and specific requests 38315. to 2010 to produce empirical (e.g., how the applicant would These guidelines incorporate a effect size benchmarks for 20 perform job-related functions single set of principles designed common research domains. with or without reasonable to assist employers, labor Results indicate that the usual accommodation). organizations, employment interpretation and classification of agencies, and licensing and effect sizes in statistical methods certification boards to comply books as small, medium and large LEGAL GUIDANCE with requirements of federal law bear almost no resemblance to prohibiting employment practices findings in the field. Azar, B. (1994). Could “policing” that discriminate on the grounds test use improve assessments? of race, color, religion, sex and Cohen, J. (1988). Statistical APA Monitor, 16. national origin. This article discusses the power for the behavioral American Psychological sciences (2nd ed.). Hillsdale, NJ: Association’s Standards for GENERAL REFERENCES Erlbaum. Educational and Psychological This book is a guide to power Testing. It discusses issues Borman, W. C., & Motowidlo, analysis in research planning surrounding the enforcement S. J. (1993). Expanding the that provides users of applied of the standards as well as the criterion domain to include statistics with the tools they need possibility of developing more elements of contextual for more effective analysis. specific standards. performance. In N. Schmitt & W. Borman and Associates Finch, D. M., Edwards, B. D., & Society for Industrial and (Eds.), Personnel selection in Wallace, J. C. (2009). Multistage Organizational Psychology. organizations (pp. 71-98). San selection strategies: Simulating (2003). Principles for the Francisco, CA: Jossey-Bass. the effects on adverse impact validation and use of personnel In this chapter, the authors and expected performance for selection procedures (4th ed.). argue that selection criteria various predictor combinations. Bowling Green, OH: Author. should embrace a domain Journal of Applied Psychology, This document outlines of organizational behavior 94, 318-340. principles adopted by the broader than just technical The authors use Monte Carlo Society for Industrial and task performance. Specifically, simulation to explore the trade- Organizational Psychology (SIOP) performance measures should off between expected mean of the American Psychological be expanded to also include performance and minority hiring Association for the validation and contextual activities, such as in multistage selection strategies use of personnel selection and helping co-workers when needed and to identify those strategies assessment procedures, including and putting in extra effort to get most effective in balancing the performance appraisal. The the job done. trade-off. The results show that principles specify SIOP policy on an increase in minority hiring conducting validation research Bosco, F. A., Aguinis, H., Singh, was associated with a decrease using principles of good practice K., Field, J. G., & Pierce, C. A. in predicted performance for in the choice, development and (2014). Correlational effect size many scenarios, although the evaluation of personnel selection benchmarks. Journal of Applied results also show that certain procedures. Psychology, 100, 431-449. multistage strategies are much The authors examine a large more effective than others for number (147,328) of correlations managing the performance and from published research in adverse impact trade-offs. applied psychology reported

29 SHRM Foundation’s Effective Practice Guidelines Series

Heneman, H. G., III, & Judge, provided the best prediction of Ryan, A. M., & Tippins, N. T. T. A. (in press). Staffing job-specific and general technical (2004). Attracting and selecting: organizations (5th ed.). task proficiency, whereas the What psychological research Middleton, WI: Mendota House, temperament/personality tells us. Human Resource and New York, NY: McGraw-Hill. measures best predicted giving Management, 43, 305-318. The book is filled with up-to-date extra effort, supporting peers and This article reviews research research, useful examples and exhibiting personal discipline. on which selection tools work, best business practices in the Scores from the interest inventory which recruitment strategies staffing area. In addition, each correlated more highly with work, how selection-tool use chapter concludes with in-depth technical task proficiency than relates to workforce diversity, applications (cases and exercises) with demonstrating effort and and which staffing and recruiting that enhance skills and provide peer support. processes lead to positive practice in key staffing activities applicant perceptions. The goal and related decision-making. Murphy, K. (1996). Individual is to assist the reader not only in differences and behavior in understanding the present gaps Kehoe, J. (2000). Managing organizations. San Francisco, between research and practice in selection in changing CA: Jossey-Bass. recruitment and selection but also organizations. San Francisco, This book integrates existing in developing skills for employing CA: Jossey-Bass. research and stimulates new ways research in HR practice. This book provides managers of thinking about how individual and HR practitioners with differences affect people’s Rynes, S. L., Colbert, A. E., practical guidance on making behaviors and experiences in & Brown, K. G. (2002). HR decisions about employee organizations. It provides insights professionals’ beliefs about staffing. The chapters are into what influences behavior on effective human resources authored by leading researchers the job. practices: Correspondence and practitioners with extensive between research and practice. experience in the staffing area Potosky, D., & Bobko, P. Human Resource Management, and offer proven strategies for (2004). Selection testing 41, 149-174. the design and management of via the Internet: Practical In this study, 5,000 human resource effective selection processes in considerations and exploratory professionals were surveyed organizations. empirical findings. Personnel regarding the extent to which they Psychology, 57, 1003-1034. agreed with various HR research McHenry, J. J., Hough, L. M., This article presents equivalence findings. Responses suggested Toquam, J. L., Hanson, M. information and practical lessons that large discrepancies existed A., & Ashworth, S. A. (1990). concerning selection testing via between research findings and Project A validation results: The the Internet. The authors identify practitioners’ beliefs in some areas, relationships between predictor several issues associated with especially staffing. In particular, and criterion domains. Personnel measurement and validity, the practitioners placed far less faith Psychology, 43, 335-353. role of individual characteristics, in intelligence and personality In this study, a predictor respondents’ reactions tests as predictors of employee battery of cognitive ability, and behaviors, and other performance than HR research perceptual-psychomotor ability, considerations regarding Internet would recommend. temperament/ personality, interest test administration. They also and job outcome preference report results from an exploratory Schmitt, N., & Borman, W. C. measures was administered to study of the correlation between (1995). Personnel selection in enlisted soldiers in nine Army paper-and-pencil and Internet- organizations. San Francisco, occupations. Scores from the administered cognitively oriented CA: Jossey-Bass. cognitive and perceptual- selection tests (including timed In this book, experts highlight the psychomotor ability tests and untimed, proctored tests). personnel selection issues that

30 Choosing Effective Talent Assessments to Strengthen Your Organization

will receive increasing attention in the years ahead. The book examines specific topics such as recruitment and retention, structured-versus-unstructured interviews, the ethics and effectiveness of computerized psychological testing, perceptions of selection fairness, productivity, turnover, and absenteeism.

Schmidt, F. L., & Hunter, J. E. (1998). The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings. Psychological Bulletin, 124, 262- 274. This article summarizes the practical and theoretical implications of 85 years of research in personnel selection. It presents the validity of 19 selection procedures for predicting job and training performance and the validity of paired combinations of general mental ability (GMA) and the 18 other selection procedures.

31 SHRM Foundation’s Effective Practice Guidelines Series

Technology will continue to spark revolutionary changes in assessment methods and practices. Quiderum num faccuptus es ratem res simaion necersperum la simet aut endanihicium fuga. Ut omniam aut omnimil ius doleste nderio ellabo. Ut oditatus et molorio tenis molorepel imusam. - Quotee to come

32 Choosing Effective Talent Assessments to Strengthen Your Organization

ENDNOTES

1. CEB. (2016). Competing for critical talent [Webinar].

2. CEB. (2012). Q4 global labor market survey.

3. CEB. (2010). Candidate rules of engagement survey.

4. CEB Corporate Leadership Council. Creating enterprise leaders.

5. Bauer, T. N., Maertz, C. P., Dolen, M. R., & Campion, M. A. (1996). Longitudinal assessment of applicant reactions to employment testing and test outcome feedback. Journal of Applied Psychology, 83, 892-903.

6. Kantrowitz, T. M. (2014). Global assessment trends report 2014. Arlington, VA: CEB.

7. Sullivan, J. (2014). The power has shifted to the candidate, so current recruiting practices will stop working. ERE Recruiting Intelligence. Retrieved from http://www.eremedia.com/ere/ the-power-has-shifted-to-the-candidate-so-current-recruiting- practices-will-stop-working/

8. Beaty, J. C., Dawson, C. R., Fallaw, S. S., & Kantrowitz, T. M. (2009). Recovering the scientist-practitioner model: How IOs should respond to UIT. Industrial and Organizational Psychology: Perspectives on Science and Practice, 2, 58-63.

9. Tippins, N. T. (2009). Internet alternatives to traditional proctored testing: Where are we now? Industrial and Organizational Psychology: Perspectives on Science and Practice, 2, 2-10

10. Ibid.

11. Golubovich, J., & Boyce, A. (2013). Hiring tests: Trends in mobile test usage. In N. Morelli, Mobile devices in talent assessment: Where are we now? Symposium presented at the 28th annual conference of the Society for Industrial and Organizational Psychology, Houston, TX.

33 SHRM Foundation’s Effective Practice Guidelines Series

12. Illingworth, J., Morelli, N., (Ed.). (1982). The g factor Development and evaluation Scott, S., & Boyd, S. (2014). in employment. Journal of of the “accomplishment Internet-based, unproctored Vocational Behavior, 29(3). record” methods of selecting assessments on mobile and and promoting professionals. 19. Raymark, M. J., Schmit, M. non-mobile devices: Usage, Journal of Applied Psychology, J., & Guion, R. M. (1997). measurement equivalence, and 69, 135-146. Identifying potentially useful outcomes. Journal of Business personality constructs for 28. Hough, L. M., Keyes, M. A., and Psychology, 30, 25-34. employee selection. Personnel & Dunnette, M. D. (1983). An 13. Morelli N., Mahan, R. P., Psychology, 50, 723-736. evaluation of three ‘alternative’ & Illingworth A. J. (2014). selection measures. Personnel 20. Costa, P. T., Jr., & McCrae, R. R. Establishing the measurement Psychology, 36, 261-276. (1992). Four ways five factors equivalence of online selection are basic. Personality and 29. Hunter, J. (1986). Cognitive assessments delivered on Individual Differences, 13, 653- ability, cognitive aptitudes, mobile versus non-mobile 665. job knowledge, and job devices. International Journal performance. Journal of of Selection and Assessment, 21. Hough, L. M. (1992). The big five Vocational Behavior, 29, 340- 22, 124-138. personality variables-construct 362. confusion: Description 14. Ibid. versus prediction. Human 30. Motowidlo, S. J., Dunnette, 15. Gael, S. (Ed.). (1988). The Performance, 5, 135-155. M. D., & Carter G. (1990). An job analysis handbook for alternative selection procedure: 22. Tett, R. P., & Christiansen, N. business, industry, and A low fidelity simulation. D. (2007). Personality tests at government. New York: Wiley; Journal of Applied Psychology, the crossroads: A response to Brannick, M. T., & Levine, E. L. 75, 640-647. Morgeson, Campion, Dipboye, (2002). Job analysis. Thousand Hollenbeck, Murphy, and 31. Kantrowitz, T. M. (2014). Global Oaks, CA: Sage. Schmitt (2007). Personnel assessment trends report 2014. 16. LoVerde, M. A., & Schmidt, C. Psychology, 60(4), 967-993. Alpharetta, GA: SHL. (2016, April). Overview of the 23. Ibid. 32. Asher, J. J., & Sciarrino, J. A. CEB leadership study (CLS). (1974). Realistic work sample In M. A. LoVerde, Overview 24. Ones, D. S., Viswesvaran, tests: A review. Personnel and selected findings from a C., & Schmidt, F. L. (1993). Psychology, 27, 519-533. multi-organizational, multi-level Comprehensive meta-analysis leadership study. Symposium of integrity test validities: 33. Hogan, J. (1991). Physical conducted at the 31st annual Findings and implications abilities. In M. D. Dunnette & L. conference of the Society for for personnel selection and M. Hough (Eds.), Handbook of Industrial and Organizational theories of job performance. industrial and organizational Psychology, Anaheim, CA. Journal of Applied Psychology, psychology (Vol. 2, pp. 753- 78, 531-537. 831). Palo Alto, CA: Consulting 17. Hunter, J. (1986). Cognitive Psychologists Press. ability, cognitive aptitudes, 25. Stokes, G. S., Mumford, M. D., job knowledge, and job & Owens, W. A. (Eds.). (1994). 34. Hausknecht, J. P., Day, D. performance. Journal of Biodata handbook. Palo Alto, V., & Thomas, S. C. (2004). Vocational Behavior, 29, 340- CA: CPP Books. Applicant reactions to selection 362. procedures: An updated model 26. Shoenfeldt, L. F. (1999). From and meta-analysis. Personnel 18. Ree, M. J., Earles, J. A., dustbowl empiricism to rational Psychology, 57, 639-683; & Teachout, M. S. (1994). constructs in biodata. Human Ryan, A. M., Greguras, G. J., & Predicting job performance: Resource Management Review, Ployhart, R. E. (1996). Perceived Not much more than g. Journal 9, 147-167. job relatedness of physical of Applied Psychology, 79, ability testing for firefighters: 518-524; Gottredson, L. S. 27. Hough, L. M. (1984).

34 Choosing Effective Talent Assessments to Strengthen Your Organization

Variations in reactions. Human 38. Sackett, P. R., Schmitt, N., Organizational Psychology, Performance, 9, 219-240; Ellingson, J. E., & Kabin, M. B. Honolulu, HI. Schmidt, F. L., & Hunter, J. E. (2001). High stakes testing in 45. Gutierrez, S. L., & Meyer, J. M. (1998). The validity and utility of employment, credentialing, and (2014). The mobile revolution: selection methods in personnel higher education: Prospects in Measurement equivalence and psychology: Practical and a post affirmative-action world. mobile device administration. In theoretical implications of 85 American Psychologist, 56, T. Kantrowitz & C. M. Reddock, years of research findings. 302-318. Shaping the future of mobile Psychological Bulletin, 124, 39. Ibid. assessment: Research and 262-274; Roth, P. L., Bevier, practice update. Symposium C. A., Bobko, P., Switzer, F. 40. Heneman, H. G., III, & presented at the 29th Annual S., & Tyler, P. (2001). Ethnic Judge, T. A. (2006). Staffing Conference of the Society for group differences in cognitive organizations (5th ed.). Industrial and Organizational ability in employment and Middleton, WI: Mendota House, Psychology, Honolulu, HI. educational settings: A meta- and New York, NY: McGraw- analysis. Personnel Psychology, Hill. 46. Sireci, S. G., Scarpati, S. 54, 297-330; Barrick, M. R., & E., & Li, S. (2005). Test 41. Ibid. Mount, M. K. (1991). The big five accommodations for students personality dimensions and job 42. Smither, J. W., Reilly, R. R., with disabilities: An analysis performance: A meta-analysis. Millsap, R. E., Pearlman, K., & of the interaction hypothesis. Personnel Psychology, 91, 1-26. Stoffey, R. (1993). Applicant Review of Educational Research, 75(4), 457-490. 35. Heneman, H. G., III, & Judge, T. reactions to selection A. (2006). Staffing organizations procedures. Personnel (5th ed.). Middleton, WI: Psychology, 46, 49-76. Mendota House, and New York, 43. Gilliland, S. W. (1995). NY: McGraw-Hill. Fairness from the applicant’s 36. Borman, W. C., & Motowidlo, perspective: Reactions S. J. (1993). Expanding the to employee selection criterion domain to include procedures. International elements of contextual Journal of Selection and performance. In N. Schmitt, Assessment, 3, 11-19; CEB W. Borman, and Associates Recruiting Leadership Council. (Eds.), Personnel selection (2013). Selecting high-quality in organizations (pp. 71-98). hires for today’s new work San Francisco: Jossey-Bass; environment; CEB Recruiting Hattrup, K., Rock, J., & Scalia, Leadership Council. (2013). C. (1997). The effects of Candidates as customers: various conceptualizations of Optimizing the candidate job performance on adverse experience. impact, minority hiring, and 44. Kinney, T. B., Lawrence, predicted performance. Journal A., & Change, L. (2014). of Applied Psychology, 82, Understanding the mobile 656-664. experience: Data across device 37. Sackett, P. R., & Roth, L. and industry. In T. Kantrowitz (1996). Multi-stage selection & C. M. Reddock, Shaping the strategies. A Monte Carlo future of mobile assessment: investigation of effects on Research and practice update. performance and minority Symposium presented at the hiring. Personnel Psychology, 29th Annual Conference of 49, 549-562. the Society for Industrial and

35

The future of HR is here (and it’s SHRM-certified)

Today’s workplace demands that HR professionals deliver innovative thinking and measurable business results. The SHRM-CP and SHRM-SCP, the new standard in HR certification, uses competencies to test not only what you know, but how you apply that knowledge in the workplace.

Join the more than 90,000 professionals who’ve earned their SHRM Certification, and demonstrate that you are the future of HR.

Get Certified Now:

shrmcertification.org/foundation 16-0179 About the SHRM Foundation At the SHRM Foundation, we are a catalyst for thought leadership. We help predict where the workforce is headed because we’ve been studying its evolution for nearly 50 years. Our mission is to offer unmatched human capital knowledge for the benefit of professional organization leaders with a total focus on studying and reporting the management practices that work. Supporting the Foundation is a chance to contribute to an ongoing study about the direction of human resources in society. The Foundation is governed by a volunteer board of directors, comprising distinguished HR academic and practice leaders. Contributions to the SHRM Foundation are tax- deductible. The SHRM Foundation is a 501(c)(3) nonprofit affiliate of the Society for Human Resource Management (SHRM). For more information, contact the SHRM Foundation at 703.535.6020. Online at shrmfoundation.org.