AHA Reprint

Alternative Solutions assessment of knowledge through a written Health and Human Services System (NHHSS), test, and, for finalists, psychological screening. starting from the essential first step of to the Child Protective After provisional hire, the new recruit often analysis through the development and Services must attend a rigorous training program, validation of competency assessments designed Crisis: including frequent written and performance to predict and retention. testing, to demonstrate required levels of Innovations from knowledge and skill. All of this occurs before The Partnership Industrial/ the officer assumes actual work assignments. None of the activities described herein We believe that the stakes involved in child could be accomplished without a shared vision Organizational protection warrant a similar level of rigor in and supportive agency management. The the selection and training of new staff. University of Nebraska-Lincoln Center on Research conducted across a variety of Children, Families and the Law (CCFL) has By Michelle I. Graef, Ph.D., clearly demonstrates that effective partnered closely with NHHSS since 1988 to and Megan E. Potter, M.A. and selection is essential to provide training, research, and consultation achieve successful training outcomes, high through a variety of projects. We have been Several years ago we conducted an informal levels of staff performance, and increased extremely fortunate to have had extensive telephone survey of 53 child protective services retention (Schmidt & Hunter, 1998). The management and staff involvement in (CPS) agencies, both state and county industrial/organizational (I/O) psychologist’s developing and implementing efforts outlined administered, to obtain a broad snapshot of “toolbox” includes an array of strategies that here, and we wish to acknowledge this what methods agencies across the country can be used to help both public and private visionary support. were using to select new CPS staff. Our goal sector achieve noteworthy The CCFL has been involved in the design was to learn what measures or indicators were improvements in functioning. A recent study of and delivery of training for child protection included in these hiring decisions, with the nearly 1,000 national firms demonstrated that staff (referred to in Nebraska as Protection hope of collaborating across agency lines to the use of what the authors termed high and Safety Workers) for the state of Nebraska develop new approaches. Our results surprised performance work practices, including job since 1988. As part of this work, a us: Many jurisdictions invested minimally in the analysis, selection testing, employee training, comprehensive job analysis was conducted, recruitment and selection of new CPS staff, yet quality of work life programs, and the results of which have guided ongoing they clearly expected new CPS workers to performance appraisals, had an economically revisions and refinements of training achieve high performance after some (varying) and statistically significant impact on employee curricula, the development of unit-based amounts of initial training. With rare exception turnover, productivity, and corporate financial assessments of trainee competence, and a (Bernotavicz & Locke, 2000), we have seen performance (Huselid, 1995). Although we are renewed focus on the importance of selection little evidence of significant change in the skeptical of anyone who professes to have all in retaining a competent, professional, and situation to present. the answers to any problem, we believe I/O productive child welfare workforce. Given the complexity of the work and the psychology has much to offer those seeking high-stake decisions required of child lasting solutions to the CPS staffing crisis. As Job Analysis protection workers on a daily basis, this lack I/O psychologists, our objective is to develop The foundation for Nebraska’s selection of attention to recruitment, selection, and strategies to optimize the fit between the needs and training system is a fundamental human placement of new staff is puzzling. It seems at of job applicants and the ; that is, resource practice known as job analysis. As the odds with what has become the norm for other to identify candidates who most likely will be name implies, job analysis is a process of jobs involving public welfare (e.g., police, satisfactory job performers and who least likely analyzing or dissecting a job and identifying the firefighters). For example, a typical municipal will leave the organization. This article job’s tasks and the associated knowledge, police officer selection process might include a provides an overview of this approach, with skills, and abilities required to successfully situational interview, physical agility test, examples from our work with the Nebraska perform such tasks. Other products of job

Originally Published in Protecting Children, Volume 17 / Number 3, in press Page 1 © 2002 American Humane Association AHA Reprint analysis include information about the The Guidelines were established by the Equal constitutes sound scientific and business organization’s philosophy and structure, Opportunity Commission, the practice. employee authority and accountability, work federal regulatory agency charged with Validity refers to the appropriateness or products and services, tools and equipment, enforcing employment laws. The Guidelines meaningfulness of inferences made from test environmental stressors, safety requirements, specify what is required of employers who use scores. (Keep in mind the broad use of the performance indicators, and job context selection procedures that have been shown to term test.) (AERA, APA, & NCME, 1999). (Brannick & Levine, 2002). Given its myriad disproportionately screen out members of a Therefore, despite frequent misperceptions, a outcomes, job analysis is an essential protected class, resulting in what is known as test is not, in itself, either valid or invalid; prerequisite to the development of a host of adverse impact. The Principles were created by rather, it is the inferences made from test other personnel systems, including the Society for Industrial and Organizational results that are either valid or invalid. The use recruitment, placement, training needs Psychologists, a division of the American of a particular test in one situation could be assessment, , job design Psychological Association. Whereas the valid, whereas use of the same test in a and enrichment, and Guidelines pertain narrowly to employment different situation might not be valid. compensation, job classification, and career law, the Principles outline widely applicable Different types of evidence can be used to development and planning (Gael, 1988). The standards adopted by professionals who demonstrate that an employer has made job analysis conducted on child protection specialize in developing and validating appropriate inferences, and these are more workers in Nebraska has been a foundation for selection and other personnel procedures. The thoroughly discussed in both the Guidelines developing several recruitment, selection, Principles describe what I/O psychologists and the Principles. One way to ensure the training, and performance appraisal tools. believe is good practice in the arena of validity of a selection procedure is to develop In addition, job analysis plays a central . the procedure on the basis of a thorough job role in satisfying legal requirements. Although Both the Guidelines and the Principles analysis. This type of validity evidence is known no law explicitly requires employers to conduct apply to a number of employment decisions, as content validity. A selection tool with content a job analysis, as others have argued (e.g., including hiring, retention, promotion, validity samples knowledge and skills shown to Brannick & Levine, 2002; Sparks, 1988), transfer, demotion, dismissal, referral, or any be necessary for job performance; in other several statutory laws require detailed other actions that affect employment status. words, the content of the test matches the information that cannot be gained without a The selection procedures covered by these content of the job. systematic analysis of the job. For example, documents include paper and pencil tests, An alternative means of demonstrating under Title VII of the Civil Rights Act of 1964, it performance tests, work samples, personality validity is establishing an empirical is illegal for an employer to discriminate inventories, interest inventories, integrity tests, relationship between the selection tool and against a person based race, color, religion, biographical data, application blanks, subsequent job performance. This type of sex, or national origin. Although the Act itself interviews, reference checks, educational evidence is known as criterion-related validity. does not require a job analysis, in claims of requirements, appraisals of job performance, A selection tool with criterion-related validity is discrimination, an employer must show a and many more. statistically related to some criterion, or job relationship between the selection procedure So what do these legal and professional relevant behavior, such as performance or and the employee’s job performance. A resources say about the procedures used to tenure. Although a job analysis is not necessary thorough job analysis will be necessary to make personnel decisions? Both references to establish an empirical relationship between establish this relationship. advise employers to demonstrate the validity of a selection tool and job performance, it is More direct prescriptions to perform job their selection procedures. The Guidelines strongly recommended that job analysis be analyses can be found in legal and professional specifically recommend that the courts make used as the basis for both choosing or guidelines, namely, the Uniform Guidelines on validation a legal requirement for employers developing a selection tool and for developing Employee Selection Procedures (EEOC, 1978) who are faced with charges of discrimination; measures of job performance. To demonstrate and the Principles for the Validation and Use of whereas, the Principles argue more generally validity, employers must show that the selection Personnel Selection Procedures (SIOP, 1987). that using valid selection procedures procedures are related to the job or can

Originally Published in Protecting Children, Volume 17 / Number 3, in press Page 2 © 2002 American Humane Association AHA Reprint reasonably estimate future job performance. classification schemes with predefined and abilities required to accomplish these tasks. Using a thorough job analysis to establish categories. The categories may focus on the We also used a secondary approach known as validity of employment procedures reduces job, the worker, or, in rare cases, both the job the critical incident technique (Flanagan, 1954), reliance on speculation, false assumptions, and the worker. Examples of the dozens of which is a more narrowly focused job analysis anecdotal evidence, and subjective decision predefined categories include decision making, method that identifies specific examples, or making. It provides the foundation for autonomy, communication, environmental incidents, of work behavior. Together, these job scientifically and legally defensible personnel conditions, reaction time, mental processes, analysis efforts required extensive involvement of practices. More simply and most important, it and hearing. Although these published methods the agency field staff and supervisors, who can help employers find employees who will are convenient and readily available, they may contributed a significant amount of time and perform well and stay with the organization. not provide the results employers need to expertise to the project. The tasks; knowledge, The bottom line from the legal system and from adequately accomplish their goals. skills, abilities; and critical incidents were professionals is that employers need to be Alternative methods involve starting with invaluable to developing a variety of tools to conducting thorough and reliable job analyses the job first, rather than a predetermined list of recruit, select, and train CPS workers. Our if they want to be successful in business and in categories, and then deriving a structure based ultimate goal is to help agencies improve court. Despite these recommendations, on the job content. These methods also result employee performance and retention. however, some practitioners still have little or in a survey or inventory, which generally no understanding of why and how job analyses comprise hundreds of job tasks. The tasks are The True Costs of CPS Staff Turnover and validation should be conducted. grouped into logical dimensions, and the survey Much has been written about the So how should an employer conduct a job is administered to job experts to determine difficulties agencies face in recruiting and analysis? The choice of job analysis techniques which tasks are most essential to the job. retaining CPS staff. Rates of attrition for CPS depends on the purpose of the job analysis; Hybrid methods of this sort go behind the task workers appear to vary widely across and different techniques result in different products, inventory and also assess the knowledge, skills, within agencies nationwide, but recent data and different products serve different purposes. and abilities needed to perform the important obtained from one midwestern state suggest an Therefore, the first step is to decide on the tasks identified by the task inventory. Methods average of about 20% turnover in CPS positions purpose of the job analysis. If the purpose is to that start with the job first can be adapted more annually. Although this problem is universally create tools such as work samples or easily to meet an organization’s specific needs, acknowledged, its origins and impact are more performance appraisals, the most desirable but they also can be more time intensive and difficult to pin down. What is the true cost of techniques are those that focus on the tasks can require more on-site job analysis expertise CPS staff turnover? performed in the job. If the purpose is to design than commercially available questionnaires or The cost of CPS turnover can be selection tools or training curricula, other inventories. For a comparison of techniques, considered in a variety of terms: techniques that focus on the knowledge, skills, see Levine, Ash, Hall, & Sistrunk (1983) and and abilities required of workers are especially Gatewood & Feild (1991). • the financial impact associated with staff useful. When multiple purposes have been Every method of job analysis will require replacement, identified, hybrid or combination methods that the availability and input of well-informed job • the added workload for remaining staff assess both the tasks and the worker and personnel experts trained in job analysis. members, requirements likely are most beneficial. In general, job analysis can be time • the emotional and physical toll on staff After a purpose has been clearly identified, consuming; however, it should be viewed as a and supervisors, and a specific method should be chosen. There are sound human resource investment that will • the incalculable impact on client at least 20 published, commercially available yield significant returns. families and children. instruments that vary in their focus and The analysis conducted for the job of CPS outcomes (Peterson & Jeanneret, 1997). They workers in Nebraska followed a hybrid method Any agency invests valuable resources in generally comprise a questionnaire or inventory that allowed us to identify 1) the critical job the recruitment, selection, and training of new designed to assess jobs according to certain tasks and 2) the associated knowledge, skills, staff, but if the individual terminates

Originally Published in Protecting Children, Volume 17 / Number 3, in press Page 3 © 2002 American Humane Association AHA Reprint employment within the first 2 to 3 years, does retention. We conducted interviews with staff theoretical explanations for their efficacy the agency recoup its investment? Evaluating the who had longer than average tenure with the (Breaugh & Starke, 2000; Hom & Griffeth, financial and client impact of staff turnover can agency to assess the factors involved in their 1995). Attention to practical details of the be an enlightening exercise, particularly when retention. Exit survey data were analyzed to design and execution of the RJP appears to be key players are unconvinced of the urgency of determine reasons staff were departing. essential to its success, as is following a the situation. For example, between the cost of In all, our research suggested that, content validation method to ensure that the hiring and retraining new child support workers although some staff exited for unavoidable RJP accurately and completely reflects job and lost collection revenues, the cost of staff reasons (e.g., spouse relocation), many CPS content (Hom & Griffeth, 1995). If it is not turnover to the state of Georgia was more than staff departed because of a poor fit between thoughtfully designed or properly $70 million annually (Brooks, 1999). their individual needs and the demands of the administered, the RJP will have fewer An analysis of turnover costs logically job. It was noted that many new hires were opportunities to affect turnover. For example, leads to discussion of specific strategies to unclear about the true nature of the job and the RJP will be most effective when it is used as manage the controllable aspects of these costs, expressed surprise and dismay during pre- early in the selection process as possible, so such as job redesign, improved recruitment service training when they realized what they that the candidate can self-select out of and selection processes, or changes in would and would not be doing as CPS workers. consideration for a job; it becomes less compensation practices. However, it also helps After considering a variety of robust effective when it is presented to the applicant to have an understanding of the reasons some interventions to address these turnover issues after hire. staff leave and others stay. For example, if the (e.g., McEvoy & Cascio, 1985; Hom & Griffeth, In partnership with the agency, we majority of staff who leave cite pay issues or a 1995), agency management identified several developed a 25-minute RJP videotape that poor work environment, these areas would be innovative approaches they wished us to presents the realistic demands and benefits of potential avenues for future intervention. Thus, pursue for improving person-job fit, focusing Protection and Safety work in Nebraska. performing initial, agency-specific research to on the recruitment and selection of new Through extensive research with CPS workers establish the true nature and impact of the Protection and Safety Workers. and supervisors, we identified a balanced turnover situation is an important first step sample of positive, negative, and neutral before considering potential strategies for Recruitment incidents that frequently occur on the job. improving staff retention. In an effort to manage some of the These incidents are illustrated in the video In Nebraska, we focused on the financial preventable instances and subsequent costs of through unscripted, but guided, interviews impact of turnover. Drawing on methods turnover, an organization can develop a with actual staff and by actors portraying popularized by Cascio (1991), we used the realistic job preview (RJP)(Wanous, 1992). typical CPS activities. We found that watching agency’s database, along with The goal of an RJP is to reduce unnecessary the video results in significant increases in job interviews with personnel directly involved in costs incurred from hiring misinformed knowledge. Moreover, interested people the administrative processes, to calculate the applicants who quit when the reality of the job become more interested in the job and specific cost elements related to CPS turnover hits. In an RJP, applicants are presented the uninterested people become less interested in for 1 year’s time. With this data, we were able realistic demands of the job, most commonly the job. All applicants for CPS positions in to determine the actual annual costs that could via a booklet or a videotape, and can self- Nebraska now are required to view this video be attributed directly to CPS staff turnover select out of the hiring process at an early early in the selection process before the (Graef & Hill, 2000). stage, thereby limiting both applicant and employment interview. The tape has been The findings were compelling and led to agency investment. Implementing an RJP will distributed across the state to workforce much discussion regarding the reasons for this not eliminate all forms of staff turnover, but it development , local job fairs, all turnover and the need for more research. We, will target job candidates who lack accurate, schools of social work, and public libraries, therefore, initiated an investigation into the realistic information about the job and accept and agency staff use the tape when giving personal characteristics, motivations, and a position for which they are ill-suited. community relations talks. Interestingly, the performance levels that related to staff Research on RJPs suggests many potential cost of producing this video was less than the

Originally Published in Protecting Children, Volume 17 / Number 3, in press Page 4 © 2002 American Humane Association AHA Reprint cost of one CPS vacancy, and so it was viewed personality, general cognitive abilities, relation to important job-related tasks, rather as an excellent investment. After viewing the situational judgment, critical thinking skills, than simply documenting candidates’ specific video, applicants completed anonymous and fundamental job knowledge. Two of these degrees or job titles. For example, candidates response cards, which suggested that exposure tools (i.e., an applicant self-assessment provide written descriptions of situations in to this RJP had encouraged several job questionnaire and a structured hiring which they have “explained procedures, candidates to seek employment in other fields. interview) currently are being implemented policies, decisions, or concepts to individuals statewide; the remaining tools are in the or families” in their educational, work, or life Selection process of development or validation. The experience. Applicants who meet established While getting the right people to apply for following describes each of these tools and our scoring guidelines on this assessment may be positions is essential, using effective methods to method for establishing their validity. invited to participate in an interview. guide hiring decisions is even more critical, Interestingly, some applicants self-select out particularly for jobs involving high-stake Evaluating Applicant Training and of the hiring process at this point due to the decisions affecting human safety. The Experience perceived burden of completing this rather implications of CPS hiring “mistakes” go well Applicants come to organizations with a short form. beyond mere organizational inconveniences to variety of educational, work, and life include potential consequences for child experiences that may or may not be pertinent to Structured Hiring Interview permanency, client family functioning, child their ability to perform the job. The challenge One of the most universally used safety, and even child death. Thus, it is prudent lies in determining which of these experiences techniques in personnel selection is the for child protection agencies to devise a have job relevance and are indicators of the interview. Unfortunately, it often is misused in selection system to carefully evaluate potential candidates’ potential. A recent review of the an unstructured, free-flowing format, which CPS candidates to ensure that those hired have literature illustrates the complexity inherent in severely limits its validity and utility (McDaniel, the highest potential to succeed in this stressful, measuring candidates’ work experience and Whetzel, Schmidt, & Maurer, 1994). A properly challenging work. provides suggestions for effectively developed and implemented standardized, In Nebraska, job analysis revealed more operationalizing job experience (Quinones, structured hiring interview can contribute than 50 knowledge, skills, and abilities (KSAs) Ford, & Teachout, 1995). For example, task- reliable, valid information to the selection that subject matter experts thought applicants specific measures of work experience appear to process, as well as enhance the likelihood of should have before being hired. The KSAs be more useful than measures of time in past favorable verdicts for employers who are faced reflect the need for various types of knowledge; positions. Although a variety of methods for with an employment discrimination challenge communication, decision making, organization, developing and validating assessments of (Williamson, Campion, Malos, Roehling, & and interpersonal skills; cognitive ability; and candidate training and experience exist (Ash & Campion, 1997). certain personality traits. A test plan was Levine, 1985; Levine, Maye, Ulm, & Gordon, Two predominant types of structured developed to outline the general and specific 1997), the involvement of subject matter interviews are the behavior description (Janz, tests that could be used to accurately and experts and use of a job analysis are necessary. Hellervik, & Gilmore, 1986) and the situational feasibly assess each KSA dimension. For In Nebraska, we worked in conjunction interview (Maurer, Sue-Chan, & Latham, 1999). example, some of these KSAs are knowledge with agency human resource managers to Behavior description interviews focus on based and thus can be assessed via paper and develop a self-assessment questionnaire for candidates’ past behavior in situations similar pencil or computer-based testing, whereas Protection and Safety job applicants called the to those encountered on the job, based on the others are skill based (e.g., oral “Supplemental Application Form.” This form theory that it is the best predictor of future communication skills, time management skills) is currently used for application screening behavior of a similar type. Situational interviews and are more appropriately targeted by after the initial screening for minimum elicit from candidates how they think they alternative methods. The resulting potential qualifications required by the state. The focus would behave in a specific hypothetical selection test battery includes measures of is on documenting task-specific training and situation. The theory is that the best predictor applicant experience, communication skills, experience in applicants’ backgrounds in of future behavior is candidates’ behavioral

Originally Published in Protecting Children, Volume 17 / Number 3, in press Page 5 © 2002 American Humane Association AHA Reprint intentions. Pulakos and Schmitt (1995) establish a common frame of reference so that Conscientiousness is one of the five well-known compared the validity evidence from both these all interviewers/raters/evaluators approach the personality dimensions included in the Five types of interviews and found some evidence of task from a similar perspective and use the Factor Model (McCrae & Costa, 1994). It has higher validity for experience-based (behavior evaluation criteria in a similar manner. been shown to be a successful predictor of description) interviews over situational types Consistency of ratings across interviewers is both job performance and turnover (Barrick & under tightly controlled conditions. However, necessary to establish a high level of reliability Mount, 1993; Barrick & Mount, 1996). Unlike given proper development and administration, and ensure standardization of the interview other personality measures, which are not including interviewer training, both types of process. With this in mind, we developed and intended to be used for employment decisions, structured interviews have proven to be implemented training for all supervisors in the the experimental measure we used has been effective selection tools. agency. These training programs use a variety designed specifically for applicant screening. It In our case, we collaborated with the of interactive, experiential activities, such as assesses past behavior, known as biographical agency to develop a standardized, structured discussion, viewing and rating videotapes of data or biodata. The scale measures everyday hiring interview for use with applicants for the simulated applicants, and evaluating worker past behaviors that are presumed to reflect Protection and Safety position in Nebraska. performance scenarios. conscientiousness, such as how often a person This interview protocol employs a combination is late for appointments or makes “to do” lists. of the behavior description and situational Investigation of Additional Selection Research on this measure is limited, but there question formats to capitalize on the relative Tools is evidence of its ability to predict job merits of each (Eder & Harris, 1999). In addition to implementing a performance. Questions were developed by a team of subject supplemental application and a structured matter experts. Applicant responses are interview, we also are in the process of Situational Judgment recorded and scored using behavioral rating developing and validating several other highly One increasingly popular selection tool is forms that provide detailed indicators of innovative tools for CPS selection. Recall that a situational judgment test, which is a low- acceptable, marginal, unacceptable, and “red our selection test plan includes measures of fidelity simulation designed to assess judgment flag” responses. Trained interviewing teams personality, situational judgment, cognitive and decision-making skills. It is regarded as comprising Protection and Safety supervisors ability, critical thinking skills, and fundamental low fidelity because the degree of realism is and human resource managers conduct these job knowledge. relatively low, compared to other, high-fidelity interviews. simulations that more closely mirror the job, Results of statewide use of this interview Personality such as work samples (Motowidlo, Hanson, & protocol have been encouraging. In particular, Personality constructs have been the Crafts, 1997). Situational judgment tests candidates report that the questions appear job target of renewed attention during the past 12 present hypothetical problems in a work- relevant and challenging. After their initial years. Several recent reviews have documented related context. The scenarios can be adaptation to this new style of job interviewing, that personality measures can be a valid presented in writing or in a video- or interviewers have noted that they appreciate the predictor of outcomes such as job computer-based format. Respondents are consistency in the rating and scoring process, performance and retention (Barrick & Mount, required to choose or to evaluate various as well as the clear link to the job. 1991; Tett, Jackson, & Rothstein, 1991). Based solutions or courses of action. on this, we assessed the usefulness of Situational judgment tests have been used Interviewer/Evaluator Training personality measures for CPS selection using a since the 1920s (McDaniel, Morgeson, Because human judgment is an integral standardized, commercially available measure Finnegan, Campion, & Braverman, 2001). component of personnel decision making, it is to assess core dimensions of normal Interest in these measures was revived in the critical that all individuals involved in activities personality (e.g., adjustment, sociability) that 1990s when Motowidlo, Hanson & Crafts such as conducting hiring interviews or may predict future work behavior. (1997) developed a situational test to select completing performance evaluations are Another personality construct we entry-level managers in the trained. The goal of such training is to investigated is conscientiousness. telecommunications industry. A recent meta-

Originally Published in Protecting Children, Volume 17 / Number 3, in press Page 6 © 2002 American Humane Association AHA Reprint analysis revealed that situational judgment tests simulate everyday events. The following is an Baughman, & Keil, 1997). Several studies have are significantly related to job performance example of one of the situations used in this shown that cognitive ability tests are predictive and that those that are developed on the basis measure: of future job performance, and some evidence of job analyses are substantially more suggests that they are even better predictors for predictive of performance than those –––––––––– complex jobs (Hunter & Hunter, 1984). One developed without the results of a job analysis You work at a large agency. You personally caveat, however, is that they have the potential (McDaniel, Morgeson, Finnegan, Campion, & have observed a coworker repeatedly rifling to result in adverse impact (Hunter & Hunter, Braverman, 2001). Although situational through papers and files in other workers’ 1984; Schmitt, Clause, & Pulakos, 1996). judgment scores have been shown to be highly offices after those workers have gone home for The long history of intelligence testing related to general cognitive ability, evidence the day. These papers and files have no has resulted in a large number of well- suggests that they can measure something relevance to this worker’s job responsibilities. designed, commercially available tests. In above and beyond what is captured with You have knowledge that this person has used consultation with the agency, we selected a cognitive ability tests (Clevenger, Pereira, this information for personal gain. Please rate written, standardized measure of verbal Wiechmann, Schmitt, & Harvey, 2001; how effective each of the following responses aptitude that can be administered to a group McDaniel, Morgeson, Finnegan, Campion, & would be in stopping this worker’s behavior. and be quickly and objectively scored. The Braverman, 2001). These results are instrument assesses several dimensions of encouraging, considering the relatively • Report to your supervisor that this verbal aptitude, including comprehension, moderate expense required to develop and person is breaching confidentiality. which is defined briefly as the ability to administer these tests. • Don’t say anything about what you have evaluate social situations and recognize The situational judgment test we observed; you shouldn’t interfere with the socially desirable behaviors. Research has developed for CPS selection is a written test situation. shown this instrument to have acceptable that presents realistic situations taken from • Ask the agency to increase levels of reliability. child protection work and relevant everyday security after regular business hours. events. The instrument currently contains 25 • Wait and confront the worker the next Critical Thinking situations, each with an average of 7 solutions. time you observe him rifling through other A related set of cognitive skills that we are Applicants are asked to rate each response in workers’ files. evaluating for CPS selection is critical thinking terms of effectiveness, on a scale from one to • Tell others that you’re concerned about skills. The job analysis revealed the need for four, with one representing “extremely breaches in the office and that they should take skills such as thinking rationally and effective” and four representing “potentially action to secure their files. objectively, making difficult decisions based on harmful.” We began by identifying pre-hire • Set up a time to speak with this worker; accurate information, and sorting relevant from KSAs that we thought would be most amenable give him the opportunity to explain what was irrelevant information. In an attempt to capture to situational testing. Various happening. these types of skills, we chose a written, cognitive/decision-making, organizational, and –––––––––– standardized measure of critical thinking. The interpersonal skills were selected. Subject test assesses various aspects of critical thinking, matter experts were used to generate Cognitive Ability such as inference and deduction. The inference situations and responses and to rate the A fourth construct we are investigating for dimension measures applicants’ ability to effectiveness of each response. It was selection purposes is cognitive ability. Although recognize whether inferences from given data especially challenging to generate situations cognitive ability testing dates back thousands of are true or false, and the deduction dimension and responses that fit the context of CPS work years, theories about intelligence and cognitive measures the ability to determine whether but did not require some job-specific ability are numerous and widely debated even certain conclusions necessarily follow from knowledge, which applicants could not be today. In general, intelligence tests are information given. The instrument has shown to expected to have. The compromise was to designed to capture relatively stable, have good reliability and, although its predictive include some additional types of situations that performance-related capacities (Childs, ability has been examined in one occupational

Originally Published in Protecting Children, Volume 17 / Number 3, in press Page 7 © 2002 American Humane Association AHA Reprint setting only, we are hopeful that it will serve as dimensions, such as arranging services, Families, and the Law. a valid predictor of success for CPS workers. composing reports, and testifying. The CDT Michelle Graef and Megan Potter provide research- also includes a section for supervisory self- based consultation to human services agencies on Knowledge and Organizational Skills assessment of participation in new employee administration issues such as personnel selection, Other measures in early development development, action planning for performance retention, turnover, performance management, and stages include a written measure of general below minimum standards, behaviorally based training design and evaluation. Their work focuses on the knowledge relevant to child protection work, rating anchors, and ratings of prosocial design of organizational interventions to achieve such as knowledge of child maltreatment, organizational behaviors. The evaluations are successful training outcomes, high levels of staff human behavior and family dynamics, and conducted at 3 and 6 months after completion performance, and increased worker retention. For more juvenile delinquency. We also are considering of pre-service training, during each new information, please contact them at 121 S. 13th St, Suite developing an “in-basket” exercise to assess worker’s probationary period. Extensive 302, Lincoln, NE 68588-0227; 402-472-3479. time management and organizational skills, training on the use of the tool is provided to all based on successful experiences in developing supervisors and administrators. References an in-basket competency assessment for the The goals of a 3-year effort to plan and American Educational Research Association, American CPS Intake training unit in Nebraska (Graef, conduct a validation study are close to being Psychological Association, & National Council on Rohde, & Potter, in press). realized. All data for the validation study have Measurement in Education. (1999). Standards for been collected, and we currently are educational and psychological testing. Washington, DC: Test Validation Study conducting preliminary analyses. Assessments American Educational Research Association. To evaluate the usefulness of these shown to be statistically valid predictors of Ash, R. A., & Levine, E. L. (1985). Job applicant instruments, a validation study was initiated. performance and tenure will be recommended training and work experience evaluation: An empirical The goal of the validation study was to assess for inclusion in the CPS selection process in comparison of four methods. Journal of Applied the effectiveness of each tool for predicting CPS Nebraska. Decisions by the agency to Psychology, 70, 572-576. Barrick, M. R., & Mount, M. K. (1991). The big five worker performance and turnover, as incorporate these new measures likely will personality dimensions and job performance: A meta- measured by the length of tenure with the attempt to balance the costs of the various analysis. , 44, 1-26. agency. This study used a predictive criterion- components of the selection system with the Barrick, M. R., & Mount, M. K. (1993). Autonomy as related validity design, which allowed us to usefulness of information each produces. a moderator of the relationships between the Big Five measure the statistical relationship between personality dimensions and job performance. Journal of test performance at the time of hire and later Summary , 78, 111-118. measures of job performance and tenure. Procedures used for the recruitment, Barrick, M. R., & Mount, M. K. (1996). Effects of All new CPS hires during a 2-year period selection, placement, and training of new CPS impression management and self-deception on the were asked to participate and complete the staff can be a critical determinant of staff predictive validity of personality constructs. Journal of battery of tests. These new hires then performance and retention. By using high- Applied Psychology, 81, 261-272. completed an intensive pre-service training quality, scientifically sound, and legally Bernotavicz, F. & Locke, A. (2000). Hiring child program of up to 15 weeks. After completing defensible methods of personnel selection, welfare caseworkers: Using a competency-based approach. training, trainees began a new 6-month agencies can take a positive step toward Public Personnel Management, 29, 33-42. probationary period, during which their creating a high performing, committed Brannick, M. T., & Levine, E. L. (2002). Job analysis. supervisor provided performance ratings using workforce. Thousand Oaks, CA: Sage. the agency’s performance planning and Breaugh, J. A., & Starke, M. (2000). Research on evaluation tool known as the “Competency Michelle Graef, Ph.D., is a Research Assistant employee recruitment: So many studies, so many remaining Development Tool” (CDT). Professor of Psychology at University of Nebraska-Lincoln, questions. Journal of Management, 26, 405-434. The CDT uses supervisory ratings of 43 Center on Children, Families and the Law Brooks, C. (1999, December). The beauty of representative tasks, based on the results of the Megan Potter, M.A., is a Research Specialist with performance measurement. IPMA Assessment Council News job analysis. The tasks are grouped into 17 the University of Nebraska-Lincoln Center on Children, [On-line]. Available:

Originally Published in Protecting Children, Volume 17 / Number 3, in press Page 8 © 2002 American Humane Association AHA Reprint

http://www.ipmaac.org/can/dec99/pracexch.html. Journal, 38, 635-672. Quinones, M. A., Ford, J. K., & Teachout, M. S. Cascio, W. F. (1991). Costing human resources: The Janz, T., Hellervik, L., & Gilmore, D. C. (1986). (1995). The relationship between work experience and job financial impact of behavior in organizations (3rd ed.). Behavior description interviewing: New, accurate, cost performance: A conceptual and meta-analytic review. Boston: PWS-Kent. effective. Englewood Cliffs, NJ: Prentice Hall. Personnel Psychology, 48, 887-910. Childs, R. A., Baughman, W. A., & Keil, Jr., C. T. Levine, E. L., Ash, R. H., Hall, H., & Sistrunk, F. Schmidt, F. L., & Hunter, J. E. (1998). The validity and (1997). Tests of cognitive ability. In D. L.Whetzel and G. R. (1983). Evaluation of job analysis methods by experienced utility of selection methods in personnel psychology: Wheaton (Eds.), Applied measurement methods in job analysts. Academy of Management Journal, 26, 339- Practical and theoretical implications of 85 years of research industrial psychology (pp. 143-184). Palo Alto, CA: Davies- 347. findings. Psychological Bulletin, 124, 262-274. Black Publishing. Levine, E. L., Maye, D. M., Ulm, R. A., & Gordon, T. R. Schmitt, N., Clause, C. S., & Pulakos, E. D. (1996). Clevenger, J., Pereira, G. M., Wiechmann, D., Schmitt, (1997). A methodology for developing and validating Subgroup differences associated with different measures of N., & Harvey, V. S. (2001). Incremental validity of situational minimum qualifications (MQs). Personnel Psychology, 50, some common job relevant constructs. In C. L. Cooper & I. judgment tests. Journal of Applied Psychology, 86, 410-417. 1009-1023. T. Robertson (Eds.), International review of industrial Eder, R. W., & Harris, M. M. (Eds.). (1999). The Maurer, S. D., Sue-Chan, C., & Latham, G. P. (1999). and organizational psychology (pp. 115-140). New York: employment interview handbook. Thousand Oaks, CA: The situational interview. In R. W. Eder & M. M. Harris Wiley. Sage Publications. (Eds.), The employment interview handbook (pp. 159- Society for Industrial and Organizational Psychology Equal Employment Opportunity Commission, Civil 177). Thousand Oaks: Sage Publications. (1987). Principles for the validation and use of personnel Service Commission, Department of Labor, & Department of McCrae, R. R., & Costa, P. T. (1994). The stability of selection procedures. (3rd ed.). College Park: MD: Author. Justice (1978, August). Uniform guidelines on employee personality: Observations and evaluations. Current Sparks, C. P. (1988). Legal basis for job analysis. In S. selection procedures. (1978). Federal Register, 43 (166), Directions in Psychological Science, 3, 173-175. Gael (Ed.), The job analysis handbook for business, 38290-38315. McDaniel, M. A., Morgeson, F. P., Finnegan, E. B., industry, and government (Vol. 1, pp. 37-47). New York: Flanagan, J. C. (1954). The critical incident Campion, M. A., & Braverman, E. P. (2001). Use of John Wiley & Sons. technique. Psychological Bulletin, 51, 327-358. situational judgment tests to predict job performance: A Tett, R. P., Jackson, D. N., & Rothstein, M. (1991). Gael, S. (1988). Job descriptions. In S. Gael (Ed.), clarification of the literature. Journal of Applied Psychology, Personality measures as predictors of job performance: A The job analysis handbook for business, industry, and 86, 730-740. meta-analytic review. Personnel Psychology, 44, 703-742. government (Vol. 1, pp. 71-89). New York: John Wiley & McDaniel, M. A., Whetzel, D. L., Schmidt, F. L., & Wanous, J.P. (1992). Organizational entry: Sons. Maurer, S. D. (1994). The validity of employment interviews: Recruitment, selection, orientation and socialization of Gatewood, R., & Feild, H. S. (1991). Job analysis A comprehensive review and meta-analysis. Journal of newcomers (2nd ed.). Reading, MA: Addison-Wesley methods: A description and comparison of the alternatives. Applied Psychology, 79, 599-616. Publishing. In J. W. Jones, B. D. Steffy, & D. W. Bray (Eds.), Applying McEvoy, G. M., & Cascio, W. F. (1985). Strategies for Williamson, L. G., Campion, J. E., Malos, S. B., psychology in business. New York: Lexington Books. reducing employee turnover: A meta-analysis. Journal of Roehling, M. V., & Campion, M. A. (1997). Employment Graef, M. I. & Hill, E. L. (2000). Costing child Applied Psychology, 70, 342-353. interview on trial: Linking interview structure with litigation protective services staff turnover. Child Welfare, 79, 517- Motowidlo, S. J., Hanson, M. A., & Crafts, J. L. (1997). outcomes. Journal of Applied Psychology, 82, 900-912. 533. Low-fidelity simulations. In D. L. Whetzel and G. R. Wheaton Graef, M.I., Rohde, T. L., & Potter, M. E. (in press). (Eds.), Applied measurement methods in industrial An intake in-basket test for child protective services trainees. psychology (pp. 241-260). Palo Alto, CA: Davies-Black Training and Development in Human Services. Publishing. Hom, P. W., & Griffeth, R. W. (1995). Employee Peterson, N. G., & Jeanneret, P. R. (1997). Job turnover. Cincinnati, OH: Southwestern College Publishing. analysis: Overview and description of deductive methods. In Hunter, J. E., & Hunter, R. F. (1984). Validity and D. L. Whetzel and G. R. Wheaton (Eds.), Applied utility of alternative predictors of job performance. measurement methods in industrial psychology (pp. 13- Psychological Bulletin, 96, 72-95. 50). Palo Alto, CA: Davies-Black Publishing. Huselid, M. A. (1995). The impact of human resource Pulakos, E. D., & Schmitt, N. (1995). Experience- management practices on turnover, productivity, and based and situational interview questions: Studies of validity. corporate financial performance. Academy of Management Personnel Psychology, 48, 289-308.

Originally Published in Protecting Children, Volume 17 / Number 3, in press Page 0 © 2002 American Humane Association