2018 TECHNICAL REPORT The Classic Learning Test

2018 TECHNICAL REPORT The Classic Learning Test

Copyright © 2018 Classic Learning Initiatives, LLC CONTRIBUTORS

Technical Report Lead Noah Tyler

Psychometricians Hong Jiao Liru Zhang

Chapter Leads Stephanie Bucklin Conor McManus Eric Newman Connor Hocking Joseph Loomis Brittany Higdon

Writers/Reviewers Kent Anhari Molly Barnes Mary Trent Ashley Thorne

Leadership Reviewers Jeremy Tate David Wagner

Graphic Designer Meg Prom

Copyright © 2018 Classic Learning Initiatives, LLC. All Rights Reserved. No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form by any means, electronic, mechanical, photocopying, recording or otherwise, without the prior written permission of Classic Learning Initiatives, LLC. Letter from the President I can’t say I've always dreamed of starting a standardized testing company. As a boy, I hoped one day to become a football player, a cowboy, or a police officer. These professions seemed to offer adventure and fulfillment whereas my experience in public school, grades K-12, felt tiresome and stifling. Not once did a teacher or book present any of the questions that could have made school more meaningful. By the time I entered school in the mid 1980s, any question that carried moral or ethical implications, or any question about the purpose of life, sacred responsibilities, or where to find human happiness, had been removed from the classroom. The education I experienced had been designed with purely utilitarian ends in mind. Any transcendent idea had been gutted from the curriculum and as a result, like most of my classmates, I was painfully bored. It wasn't until graduate school that I came to appreciate the holistic education previous Americans had received. The founding fathers of the United States revived my imagination. They were deeply interested in philosophy, human nature, political theory, and the pursuit of happiness. The education they received was aimed, most fundamentally, at making a person more fully human. As I questioned how such a beautiful concept of education had been lost, I came to the conclusion that high-stakes testing, especially the SAT and ACT, were partially to blame. Not once since the launch of CLT has someone refuted the idea that high-stakes testing drives secondary curriculum. David Coleman, CEO of the College Board, has stated publicly that "teachers will teach towards the test. There is no force on this earth strong enough to prevent that." If teaching to the test is an inescapable reality, then shouldn't the most important test engage students with some of the most important ideas, texts, and subjects? CLT was born in response to this question. We hope that by offering a new standard that puts students in front of the thinkers and questions that have most meaningfully shaped our culture for the past two millennia, we can be a catalyst for renewal in education nationwide.

Jeremy Tate Jeremy Tate, CLT President

i 15 17 1 5 ......

Test Scores Test Formats Testing Support Day CLT Test Selecting and Training Item Writers Item Writers Selecting and Training and Uploading Design, Review, Item Review Public vs. In-School Testing Proctors Day Schedules Test Grammar/Writing Test Grammar/Writing Author Bank Quantitative Reasoning Test Calculator Policy Optional Essay Levels Difficulty Improving Students’ Test-Taking Experience Students’ Test-Taking Improving Motivating PositiveEducation Change in Assessment and Report Contents of the Technical Overview Reasoning Test Verbal

4. Test Administration Administration 4. Test 3. Test Development Development 3. Test 2. Standards and Content Coverage Coverage and Content 2. Standards 1. Introduction 1. Introduction Table ofTable Contents ii

TABLE OF CONTENTS 5. Test Security 8. Concordance Relationship 7. Test Results 6. Fairness andAccessibility Summary Results Study Design Overview Features Reporting Forthcoming Secondary SchoolScore Reports ScoreCollege Reports Student Score Reports andRequests Accommodations inTestFairness Accessibility inScoreFairness Interpretations forIntendedUses Testing Duringthe Fairness Process Test DayProcesses andProcedures Incident ManagementandResponse Monitoring andAuditing Access Control Data PrivacyandAcceptableUse Data Security ......

...... 25 21 31 27 iii

TABLE OF CONTENTS 65 77 79 39 45 55 ......

......

Evidence Based on Content Overview of Measurement Error Reliability and Standard Overview Evidence Based on Internal Structure Overview Mode Form Testing and by Test Summary of Scores by Subgroup Scores Summary of CLT Mode and Testing by Subgroup Scores Summary of CLT and Family Income by School Type Scores Summary of CLT Summary of Item Analysis Results for CLT Tests Results of Item Analysis Summary CLT for Introduction to Psychometric Evaluation Evaluation to Psychometric Introduction Quality Control ofControl Quality Analysis Psychometric Appendix 12. Validity 12. Validity 11. Standard Error ofError 11. Standard Measurement 10. Reported Scores Scores 10. Reported 9. Item Analysis 9. Item iv

TABLE OF CONTENTS

to erase. erase. to education. features: ’s score. score. ’s ble points and over 140 colleges and universities have adopted it as an admissions as an admissions it adopted have and universities colleges 140 and over wn devices

1 20 minutes 1. INTRODUCTION ent format day scores E PLATFORM

Online platform, on students’ o Clean, consist Straightforward scoring—120 questions, 120 possi Straightforward Shorter test-taking time—1 Same- » » » » » » » » » » The CLT is a different kind of is a different The CLT improve It aims to dramatically exam. entrance college standardized graders; and the CLT8, an end-of-grade assessment tool designed for 8th-grade students as they prepare for high school. high school. for an end-of-grade prepare 8th-grade assessment tool designed for students as they graders; the CLT8, and The full list of colleges which have adopted the CLT as an admissions exam is provided at https://www.cltexam.com/colleges. https://www.cltexam.com/colleges. at The full list of is provided as an admissions exam adopted the CLT have colleges which The CLT suite of assessments is comprised of the CLT, a college entrance exam; the CLT10, a preparatory exam for the CLT offered to 9th and 10th offered CLT the of suite for exam a preparatory The CLT the CLT10, exam; a college entrance assessments is comprised of the CLT, 2

Students take the test on their own devices. Using an unfamiliar device for a high-stakes test can be frustrating since frustrating can be test a high-stakes for Using an unfamiliar device devices. own on their test the take Students For students, the CLT is refreshingly user-friendly, clear-cut, and modern. and clear-cut, goal of the with was designed It user-friendly, is refreshingly CLT the students, For The CLT is primarily administered online (84% of online administered is primarily The CLT version a paper in 2018), though administration test total The Classic Learning Test (CLT) launched in December 2015 as a long-awaited alternative to the SAT and and SAT the to alternative long-awaited 2015 as a in December launched (CLT) The Classic Learning Test providing the best possible test-taking experience, and it includes the following the includes and it experience, test-taking possible best the providing students’ test-taking experience, and to motivate positive change in assessment and in assessment change positive motivate and to experience, test-taking students’ change their answers with one click without having to fill in Scantron bubbles or take time time take or bubbles Scantron to fill in having click without one with answers their change every desktop, laptop, and tablet has its own subtle differences. Allowing students to use a device they are already already are they to use a device students Allowing differences. subtle own has its and tablet laptop, desktop, every familiar with reduces the possibility that the device itself device the that possibility the reduces familiar with student will impair the from the CLT suite of suite CLT the from assessments, is an option for in-school testing. The online platform is more natural for contemporary students than a pencil contemporary a pencil The than students for platform online natural is more testing. in-school is an option for test. and paper format,and paper of possibility the and it reduces and can select Students confusion and unnecessary mistakes. 2 ONLIN

1 Improving Students’ Test-Taking Experience Test-Taking Students’ Improving ACT. As of an exam taken have States United the across schools at than 15,000 students 2018, more December ACT. 1

CHAPTER ONE Assessment andEducation ACT, available “Multiple choice scores are normally within two weeks after each national test date, but it can Motivating Positive Change in 4 Reasoning andGrammar/Writing section hasexactly four (4)reading passages, andeach passage hasexactly By contrast, wait time for SAT multiple choice scores is13-23 days inthe 2018-2019 academic year. SAME- SHORTER TEST-T STRAIGHTFORWA CLEAN, CONSIST reading questions line upside by side withthe 3 ubro colleges. significantly students’enhances ability to This reach the colleges they care about, andit number of pr yiscnet L ist hnetelnsaeo assessment, and education generally,apart by its content. CLT aimsto change the by landscape of providing a the college application the financial burden of alleviates part of account. For both the SAT andACT, students may send score reports to onlyfour colleges for free, after which academic ability by usingashor there isafee per each additional college. But for students taking the CLT, score sharingisfree to anunlimited tests due toevidence that the scores for many students canbe negatively affected by the mental exhaustion they tes ten (10)questions. Knowing what to expect frees students from anxiety that cancome from aless predictable how close they are to c is 2 hours and45minutesis 2hours long, andthe SAT long. is3hours CLT The wasdesigned to be shorter than those is nopenalty for incorrect answers. 120-point allows scale The the test to be divided into three equally valuable may experience after lengthy testing. students’ CLT to provide aims colleges with amore accurate snapshotof changi standard that isintellectually richer andmore rigorous than other college entran studen sometimes take upto eight weeks.” students andproctor have left the last test site, andstudents may loginto their account to view their results. sections with 40questions each. straightforward This scoringmethod isyet another means by which CLT provides transparency andabetter test e rgesbri rvdda h o fthe page, withapercentage number, A progress barisprovided at the top of to give students avisualsense of h L sdsge ihsmlct n aac.Ec fthe three sections has40questions. Each Verbal CLTThe isdesigned with andbalance. simplicity Each of The CLTThe is120minutes counting the optional30-minute long—2hours—not essay. Incontrast, the ACT font, andthe test aesthetic isclean andfreeThe from distraction. It uses awhite background andareadable serif nteCTteeae10soe usin,ec ot n on,frattlo 120points possible.On the CLT There there are 120scored questions, each worth one point, for atotal of Each browser sectionloadsintoasingle window, that sectionwithout sothat studentscanscroll toany partof Besides offering alogisticallybetter andmore humane test experience, the Classic Learning Test sets itself tdnsmyas hr hi L crswt olgso their choice using their secure, onlineStudents student may share also their CLT scores with colleges of Students canaccess their scores onthe same day they take the test. CLT operations releases scores after the t design. CollegeBoard, GettingScores, WhentoExpectScores, https://collegereadiness.collegeboard.org/sat/scores/getting-scores. ACT Test Scores, “How CanISeeMy Scores?” http://www.act.org/content/act/en/products-and-services/the-act/scores.html. ng pages. t anxiety. DAY SCORES ompletion. AKING TIME ENT FORMAT RD SCORING ter test. 4

Same-day CLT test results eliminate the longwaitandhelp to reduce xperience. passage. process.

ce exams. 3 For the 2 CHAPTER ONE following: o master. o master. tradition. . chievement 5 urriculum. g passages CHIEVEMENT the CLT G PASSAGES education rriculum. Chapter 2. Chapter ducation. ducation. e rigorous

Is mor Provides great readin great Provides good Promotes Assesses both aptitude and a Assesses both aptitude » » » » » » » » CLT vs. SAT vs. ACT. https://www.cltexam.com/comparison. ACT. vs. SAT vs. CLT

CLT also seeks to address the problem of among the problem ACT no longer differentiate also seeks to address and CLT The SAT inflation. test score For two-thirds of two-thirds bank of an author uses CLT For the passages, reading its whose and women 100 men than more Of course, students must draw upon the teaching they have received in order to demonstrate what they have have they what demonstrate to in order Of received have they upon the teaching draw students must course, The CLT, instead of the test, considers students’ intellectual for crammed students have much how evaluating The CLT, The CLT aims to assess not only students’ achievement, but also their aptitude. Thisimportant as an aptitude is also their aptitude. but not only students’ achievement, aims to assess The CLT The CLT is different. Hallmarks of Hallmarks is different. the include content TheCLT’s CLT the The SAT and ACT are designed to reflect public school curricula, specifically the Common Core State State curricula,Core Common the specifically school public reflect to designed are ACT and The SAT An important reason CLT uses older sources is that these texts have stood the test of test the stood have texts these is that sources older uses was Shakespeare time. CLT reason An important draws from works that are grounded and established in and established grounded are that works from draws dependent on a one-size-fits-all c dependent on a one-size-fits-all capacity. The CLT assesses skills students can develop through a variety of a variety through assesses skills students can develop as their ability The CLT such types, education capacity. exams have historically sought to determine not only what students know but how they think. At its inception they how historically sought to determine but students know not only what have exams important thinker, the CLT intentionally acknowledges the particular significance of significance particular the acknowledges intentionally CLT the full author authors. The certain thinker, important in 1926, the SAT’s initials stood for Scholastic Aptitude Test. However, the SAT has since rebranded itself since rebranded has the SAT and However, Aptitude Scholastic Test. for initials stood in 1926, the SAT’s used on the CLT are truly worthy of student’s time and attention. Using older texts also gives the CLT healthy of healthy worthy truly are CLT the also gives texts Using older and attention. CLT time on the used student’s learned. “Achievement” of is a vital and necessary aspect of knowledge learned. “Achievement” the CLT. for assessment, and it is relevant one kind of kind one e tend to tacitly favor certain political opinions. Rather than favoring either side of side either favoring than Rather opinions. political certain favor tacitly to tend CLT the spectrum, political the bank is listed in bank is listed top students. Even a perfect 1600 on the SAT or a perfect 36 on the ACT does not set students apart from others does not set students apart from a perfect 36 on the ACT or a perfect 1600 on the SAT Even top students. SAT upper tail ofat the students differentiate can truly that vs. CLT for The Scoring Comparison the distribution. to communicate clearly, to understand metaphors, to think logically, and to solve puzzles. Thus the test is not puzzles. and to solve to think logically, to understand metaphors, clearly, to communicate test defers to the diversity oftest defers to the diversity definition, has to test, by An achievement curriculum within American education. books and great stories. Their alignment with Common Core also pigeonholes the SAT and ACT to test only only to test and ACT SAT the also pigeonholes Their Core Common with alignment stories. great books and applying for admission to Harvard, MIT, or other top universities. What is needed is a more rigorous standard standard rigorous is needed is a more What or other top universities. MIT, admission to Harvard, applying for assume a particular body of body assume a particular content or set of t standards academic neutrality on contemporary political matters. The SAT and ACT, by drawing primarily from contemporary sources, contemporary sources, primarily from drawing by and ACT, The SAT on contemporaryneutrality matters. political relevant 300 years ago, 50 years ago, and today, and it's likely he will be relevant 100 years from now. The authors now. from 100 years will be relevant he likely and it's and today, ago, 50 years ago, 300 years relevant renounced its former tests students based on their familiarity with the renounced It now identity as an aptitude exam. Standards framework. Those standards rely more on recent writings and “informational texts” than on classic and “informational writings on classic than on recent texts” more rely Those standards framework. Standards 5 MORE RIGOR writings have had a lasting influence on culture and society. While the author bank is not an exhaustive list of list exhaustive an bank is not author the While every and society. on culture a lasting influence had have writings with a strong showing on showing with a strong GREAT READIN GREAT Common Core cu Common Core Colleges are rightly interested in the educational content of content in the educational rightly interested Colleges are But entrance admission. apply for the students who vs. ACT shows that the CLT is more rigorous than the competing tests, and that students can set themselves apart students can set themselves and that than the competing tests, rigorous is more the CLT that shows ACT vs. APTITUDE AND A APTITUDE AND 3

CHAPTER ONE otnso the Technical ReportContents of eiin n 1.% fthe passages isdrawn Religion. One (12.5%)of L ek orietesadr fschooling inAmericaandcontribute to amovementCLT inwhich education is seeks to raise the standard of CLT are soul-enriching. Students preparing for the CLT training increative gain problem solvingandthinking (12.5%) is an excerpt from(12.5%) isanexcerpt Historical/Founding Documents, one (12.5%)isaHistorical Profile, andone (12.5%)is PROMOTING GOOD SAT andACT. the CLT, With the introduction of schools andhomeschooling families have afresh incentive to rather than encouraging a love of learning. In contrast, the authors students read in advance of and during the andduringthe learning. students read Incontrast, the authors inadvance of rather than encouraging alove of reading passages, two (25%)are inPhilosophy/ thereadings camefrom religion. Only6%of are intellectually curious, think deeply, reason well, andlive with integrity. CLT The promotes education that helps thinkers the thinkers the eightavailable SAT practice testsfound that that other standardized Astudy tests neglect. of on how test questions are organized by difficulty level. Chapter 3outlines the steps CLT takes to develop, edit, and eight subject categories. Onevery test, out of of asthosewrittenbyor journalists scientific 48)related to thepassages(1outof only 2%of 6 focus onenduri from Literature, two (25%)are inScience, one measures taken to ensure the test’s security and meaningful an extensive—and expensive—test prep in order to navigate them, and the process lends itself to student anxiety to student anxiety extensive—and expensive—test prep inorder to navigate them, andthe process lends itself thepassagesrelated to ethics, andthat noneof einadamnsrto fthe CLT,design andadministration of 7-12 explain and Chapters andanalyze the test’s drawn from Modern/Influential classic literature. Byfarthelargest SAT reading questions and reported scores, 11and12quantify the test’s andChapters reliability and scores compare to scores onthe SAT andACT. statistics 9and10present onCLT Chapters demographic test students ontheir way to becomingproductive adults, but such qualities are alsoends inth men andwomen education. who secondary Schools instead should aimto form should not be the maingoalof skills that willbenefit them their who prepare each test for administration. 4,5,and6explain Chapters how the CLT isadministered and describe the passages, twiceasmany were writtenby bloggers those passage category wasscience, andof The CLT’sThe reading passages alsocover subjects hstcnclrpr sagieepann h eal fhow the CLT exam works. 1-6 describe the technical Chapters This report isaguide explaining the details of test preparationThe process for the CLT isvastly different than for other tests.The SAT and ACT require hl cleeadcre edns”teojcieo the CommonCore State Standards—is important, it While “college andcareer readiness”—the objective of Ultimately the CLT seeks to enhance the way young people are educated. Students are nolonger limited to the Chapters 7and8provideChapters onhow CLT information scores are reported to students andcolleges andhow these the test itself, including sample questions, the author bank,andinformation Chapter 2presents the content of By elevating excellent writing andincludingstudents inthe Great Conversation that centuries ago, began the By contrast, the CLT hasabalanced distribution “What’s Really ontheSAT?” CLT, 2018. mselves. d lasting. ng ideas. 6

EDUCATION Thinkers. Thinkers. le lives. fairness. fairness. Historical/Foundin Historical Pro 12.5% 12.5% file Modern/Influentia g CLT Readin 12.5% l g Passages metrics. validity. emselves. Science 12.5% Literature 25% 25% Philosophy/Religio n 4 CHAPTER ONE Clark Clark 1 2 concepts. concepts. o think.” ideas. ideas.

. Classical Academic Press, 2013. . Classical Academic Press, 2. STANDARDS AND STANDARDS CONTENT COVERAGE kgrounds. kgrounds. The Liberal Arts Tradition: A Philosophy ofThe Arts Tradition: Liberal Christian Classical Education nowledge. nowledge.

ain ways. ain ways.

Clark, Kevin and Ravi Jain. Jain. and Ravi Kevin Clark, Ibid.

The SAT and ACT are designed around public school standards which change according to educational educational to according change which standards school public around designed are and ACT The SAT Second, the Quantitative Reasoning section assesses students’ ability to solve problems and to think in a think in a and to problems solve to ability assesses students’ section Reasoning Quantitative the Second, First, of instead English two majority of the CLT’s the contemporary sources, coming from passages reading Each CLT exam consists of three mandatory sections, Verbal Reasoning, Grammar/Writing, and and Grammar/Writing, Reasoning, of consists exam CLT Each Verbal sections, mandatory three Whereas other standardized tests rely on what is currently popular and recently legislated in American in American legislated popular and recently is currently on what rely tests standardized other Whereas The Classic Learning Test (CLT) was created in the context of context the in was created foundations the renew to (CLT) movement a national The Classic Learning Test 2 1 trends and legislative actions. In contrast, the CLT is based on enduring concepts accessible to students from a a from students to concepts on enduring is based accessible CLT the contrast, In actions. and legislative trends ofvariety bac educational logical and orderly manner. The test balances assessing mathematical reasoning capacity with testing specific specific testing with capacity reasoning The mathematical assessing balances test manner. logical and orderly skills and k mathematical sections primarily use selections from time-tested authors who have shaped history, literature, and philosophy and philosophy literature, history, shaped have authors time-tested who from selections primarily use sections with interact to students for an opportunity provides thus The CLT centuries. the through ways in foundational of world the in difference a profound made have voices thinkersimportant whose tests in two m in two tests Quantitative Reasoning, as well as an optional Essay. These are similar to the sections in the SAT and are and are SAT in the sections the similar to These are Essay. an optional as as well Reasoning, Quantitative of content the But tests. taking standardized students to recognizable standardized other from is distinct test the education, the CLT focuses on ideas that matter on a much grander scale. These include perennial questions questions These perennial include grander scale. on a much matter that on ideas focuses CLT the education, mathematical history; and universal from lessons world; physical and the human nature about and Jain (2013) write, “Recovering the primacy of the “Recovering is a pivotal arts (2013) write, mathematical arts and the language both the and Jain t in how but think to ofpiece in what just not student the train they paradigm. Together this of than rather ideas true and tried reflects that an assessment simply means “Classic” here education. great in language students trains model, which arts education on a liberal is based CLT contemporary experiments. of acquisition the “to make as a path and mathematics arts and effective.” simple more studies all later Overview Overview 5

CHAPTER TWO 5 od.Tettlms ebten225235 o naeaeo 2,300wor 650 words. total must The be between 2,275-2,325,for anaverage of two presented shorter excerpts together. They always follow aparticu VERBAL REASONING to connect high-level in within apassage andto make connections between concepts and terms ideas students are likely to know already.within apassage terms anduse require analogies These students to be able impossible without knowing to understand the words’analogy definitions, CLT’s analogies refer to concepts analogies were unattached to reading andwere passages criticized for difficult using vocabulary that made the logicalreasoning andsynthesis. Whereas the SAT’s thatthe understanding analogiesrequire ahighorder of SAT questions per set. The eight analogy removed analogies in2005,but the CLTof includes them based on which isalways the second passage of Relationships.” E Analysisquestionsincludethesubdomains“Textual of Analysis”and“Interpretation Comprehension questionsincludethesubdomains“Passage asaWhole,” “Passage Details,” and“Passage VERBAL REASONINGTEST is structured, andwhat couldbe reasonably in inferred based onthe information and words are used incontext, the author’s inaparticular section orinthe passage overall, purpose how atext and synthesize ideas within that text. They must such beable concepts ashow to understand different phrases texts indifferent subject areas andare tested ontheir ability to comprehend the text interact with avariety of Verbal Reasoning Test Tests are calibrated sothat each Verbal 500- Reasoning passage fits narrowly within a word count range of Each Verbal four passages: three fullpassages andone passage composed of Reasoning section consists of Two questions perpassage inthe Verbal Reasoning section test analogies based onthe passage, for atotal Evidence questions always refers to afigure accompanying the Science passage, of the Interpretation One of the 40Verbal ReasoningOf questions, 13fallunder Analysisand27fallunder Comp On theVerbal Reasoning section,questionsare broken down intotwo types:Comprehension andAnalysis. VerbalThe Reasoning section tests astudent’s andanalyze ability to understand atext. Students are asked to » » » » » » » » Historical/Founding Documents (2shorterexcerptspresented Philosoph Science Literature Verbal Reasoning (40 SECTION y/Religion questions) TEXT TYPE the four. BLUEPRINTS Comprehension (27 Analysis (13 S DOMAIN questions) questions) together) lar order: Interpretation o Interpretation ds total. the text. Passage Rel Passage SUBDOMAIN Textua Passa rehension. ge Details as aWhole l Analysis ationships Evidence f vidence.” a passage. 6 CHAPTER TWO question? ly because g section. , 1844. t wi ntury AD. character? , sixth ce entury AD. ic tribes. round him. round le," 1865. olleagues. by Boethius t novelty. . study”) QUESTIONS pport him. ertilize”) Religion and the Rise of Western Culture: The Classic Study of Medieval Culture: Religion and the Rise of Western reality”) tendency”) crocodile. y partial. scribed as Life and Adventures of Martin Chuzzle Life and Adventures s children tic beast. l morality t language The Consolation of Philosophy f Evidence philosophy he unknown ationships phy : soul e : spirit 1950. : body :: : friends as a Whole

ion, l Analysis ge Details

D) Paragraph 6, Sentence 1 (“Consequently . . A) Paragraph 4, Sentence 1 (“And this ...... f B) Paragraph 4, Sentence 2 (“The great C) Paragraph 5, Sentence 2 (“But the . . . C) his allies, Anaxagoras and Zeno, did not su C) his allies, Anaxagoras and Zeno, did to those a D) he lived an upright, ethical life in contrast A) his philosophy was ill-formed and onl filled with barbar B) he traveled to a distant, violent land B) philoso C) politics : D) love A) exercis D) Disinterest in the lives of hi D) Disinterest A) Suspicion of conventiona for eloquen B) Affection C) Fear of t C) a cogent story about an attempt to seek ou C) a cogent story about of the sale of a D) a meandering account A) a subtle exploration of the rivalry between two c A) a subtle exploration of a fantas B) a whimsical tale Below is one sample question for each subdomain in the Verbal Reasonin Verbal in the subdomain each for sample question is one Below Passage adapted from Christopher Dawson's Passage adapted from Civilizat Interpretation o Interpretation thebest evidence in support of the answer to the previous Which lines in the passage provide Passage adapted from Passage adapted from In Passage 1, Philosophy indicates she believes was put to death Socrates primari Passage adapted from Plutarch’s "On Education" in Moralia, first c Plutarch’s Passage adapted from Textua Passage Rel medicine Passage adapted from Charles Dickens' Passage adapted from Passa Pecksniff’s to the what is a hallmark passage, of Mr. According Passage adapted from Fyodor Dostoevsky's "The Crocodi Fyodor Passage adapted from Overall, the de can be best passage Passage VERBAL REASONING SAMPLE REASONING VERBAL 7

CHAPTER TWO Passage adapted from Hilaire Belloc's caring Agreement GRAMMAR/WRITING SAMPLE 565 words. 2,100wor total must The be between 2,075-2,125words, for anaverage of GRAMMAR/WRITING errors. Writing questions test astudent’s ability to improve uponatext’s style, flow, and wo questions include the subdomains“Structure,” “Style,” and“Wor Grammar questions include the subdomains“Agreement” and“Punctuation andSentence Structure.” Writing GRAMMAR/WRITING TEST speech, to match verb tense, well-forme andto make other grammatically correctly, to convey points precisely andconcisely, to make appropriate transitions, to choose the correct partof that text andto improve its readability andflow.The section assesses students on their ability to use punctuation texts indifferent subject areas andare tested ontheir ability to correct errors within interact with avariety of Grammar/Writing Test D) carefully C) careful B) caringly A) Below isone samplequestion for each subdomaininthe Grammar/Writin Tests 460- are calibrated sothat each Grammar/Writing passage fits narrowly within a word count range of four passages. They always follow aparticu Each Grammar/Writing section consists of Grammar questions specifically test astudent’s punctuation, ability to correct agreement, structure, andother the 40Grammar/Writing questions, 20are onGrammarand20are o Of On the Grammar/Writing section, questions are broken down into two types: GrammarandWriting. Grammar/WritingThe section tests astudent’s ability to edit andimprove atext. Students are asked to NOCHANGE » » » » » » » » decisions Modern/Influenti Science Historic Philosoph Gramm decisions decisions (40 decisions SECTION al Profile questions) y/Religion ar/Writing al Thinker

TEXT TYPES The French Revolut BLUEPRINTS QUESTIONS (20 (20 DOMAIN Grammar ion Writing questions) questions) , 1911.

d Choice.” d choices. n Writing. g section. Sentence SUBDOMAIN Punct ds total. W Agreement Structure lar order: ord Choice rd choice. uation and Style Structure

8 CHAPTER TWO

, Booker T. e of Hippo ge Bernard raft, Mary s Aquinas s Aquinas Benedicta r, Francis r, a of Ávila Alexander François François , J. R. R. Jean-Paul s, Dorothy et, Sigrid homas More gh, Evelyn il, Simone the Cross the Cross akespeare akespeare lstoy, Leo lstoy, an-Jacques mith, Adam wain, Mark St. Teres St. Teresa of St. Thoma St. T Ptolemy Pushkin, Rousseau, Je Sartre, Sayer Schaeffe Sh Geor Shaw, S Sophocles St. Augustin Washington, Wau Max Weber, We Wollstonec Thucydides Tolkien To T Unds Virgil Voltaire, von Hildebrand Dietrich

uther, Jr. uther, n Paul II n Paul II u, Charles Michel de i, Niccolò Friedrich Flannery Flannery John Henry edict XVI edict XVI ard, Søren ard, Ratzinger) le, Herman el, Gregor ll, George k, Michael l Wojtyła) son, James on, Thomas ton, Isaac al, Blaise er, Martin er, e Leo XIII cy, Walker cy, lton, John wis, C. S. lanck, Max ocke, John Pope Ben (Joseph Pop Per Plutarch Montaigne, Montesquie Newman, New Nietzsche, Nova O’Connor, Orwe Pasc P Plato Pope St. Joh (Karo Kierkega L King, Martin Le L Luth Machiavell Madi Marx, Karl Melvil Mend Mert Mi hristopher , F. Scott , F. Frederick Alexander ky, Fyodor ky, Jonathan Jonathan er, Martin er, Friedrich , Benjamin , Charles , Charles , W. E. B. , W. , Langston on, Thomas n, Albert n, Albert , Johannes rtes, René , J. Henri i, Galileo es, Thomas s, William , Immanuel el, Georg el, Georg ot, George S. ot, T. yce, James mes, Henry ume, David Kepler H Hughes Ja Jame Jeffers Jo Kant Hamilton, Heg Wilhelm Heidegg Hobb Homer Epicurus Eli Eli Fabre Fitzgerald Franklin Galile Dickens Dostoevs Douglass, Du Bois Edwards, Einstei Dawson, C Dawson, Desca , Nicolaus , Dietrich , Napoleon , Margaret nna Julia nna Julia Simone de l, Winston ton, G. K. am, Jeremy n, Charles eri, Dante , Mortimer hov, Anton hov, c, Hilaire n, Francis ar, Julius ar, dt, Hannah

win, James elou, Maya er, Martin er, le, Robert vin, John vin, John ams, John ams, John

Cooper, A Cooper, Darwi Copernicus Confucius Cicero Chek Churchil Caes Chester Cavendish Bub Cal Bonhoeffer Bonaparte Boy Benth Boethius Bello Bald Beauvoir, Avicenna Baco Averroes Aren Aristotle Ang Ad Adler Alighi

2018-2019 CLT A CLT UTHOR BANK UTHOR

9

CHAPTER TWO figures. areAuthors periodically added to passages are influential thinkers, drawn scientific writings, orhistorical from modern modern one-third of onthe opposite page. authors otherReasoning The andGrammar/Writing passages are drawn from the list of CLTby Verbal time-honored writing from authors c.400B.C. to the present 2018,two-thirds day. of Asof the passages are younger than the students taking the test—the CLT to writings looks recent decades—many of diverse backgrounds andphilosophies. Whereas the SAT andACT of use primarilypassages fromauthors Author Bank Congress the andinthe Government—in National in the Punctuation andSentence Passage adaptedfrom St.Teresa ofÁvila's permeated W Passage adaptedfrom ShirleyChisholm's "FortheEqualRightsAmendme Of course,from prejudice noteliminate heartsofhumanbeings,lawswill the Style Passage adaptedfrom MartinH.Weissman's "Whyprimenumbers stillfascinatemathematicians,2,300yearslat th fitsbestin paragraph.Whichoption endofthis wantstoaddasentencethe The author Structure Passage adaptedfrom JohnF. Kennedy's"Address totheNationonStateofU.S.Econo h L rw nsucsta aehle hp h oreo Western CLTThe intellectual thought, draws including onsources that have of helped shape the course intheCongress,D) intheNationalGovernment, and intheCongressC) intheNationalGovernment, and intheCongress;B) intheNationalGovernment; and A) D) C) B) A) D) Lawsoftheheartshumanbeingsare noteliminatedbyprejudice, C) Ofcourselawswillnoteliminateprejudice from theheartsofhum B) Ofcourse,from human beings’hearts,prejudice willnotbeeliminatedbyhumanlawsth A) D) Pellwouldhavebeenabletocreate twomillionprimeshadhea C) Unfortunately, mostofthesenumberswere B) By1800,independentprojects hadlistedtheprimesupto A) Pellneversolvedtheancientproblems ofDiophantos ord Choice persisted NOCHANGE NOCHANGE NOCHANGE persecuted persevered Structure The Way ofPerfect the list. incorrect. , however. ion Statesto States—to Statesto , 1583. 1 million. States—to computer. an beings. nt," 1970. of course. from them. my," 1962. ey create. e passage? er," 2018. 10

CHAPTER TWO

section. section. Operations pressions pressions of Shapes e Geometry Logic Equations d Problems igonometry Tr and Wor SUBDOMAIN Properties Algebraic Ex Coordinat g section. Arithmetic and

Reasoning QUESTIONS questions) Questions) Questions) Algebra BLUEPRINTS Geometry DOMAIN (10 (14 (16 t domains. t Mathematical

Equations Problems.” Reasoning Operations Questions) SECTION (40

Quantitative

Below is one sample question for each subdomain in the Quantitative Reasonin Quantitative subdomain in the each for sample question is one Below On the Quantitative Reasoning section, questions are broken down into three types: Algebra, Geometry, Geometry, Algebra, types: three into down broken are questions section, Reasoning Quantitative On the The Quantitative Reasoning section tests students’ ability to think logically, use and manipulate symbols, and and symbols, and manipulate use think logically, to ability students’ tests section Reasoning The Quantitative Algebraic Expressions and Algebraic Expressions Arithmetic and QUANTITATIVE REASONING SAMPLE QUANTITATIVE Of the 40 Quantitative Reasoning questions, there are 10 Algebra questions, 14 Geometry questions, and 16 and 16 Of 14 Geometry questions, questions, 10 Algebra are there questions, Reasoning 40 Quantitative the ofas Reasoning questions, Reasoning Quantitative Mathematical each in figures 5 are also There 2018. “Properties of Shapes,” and “Trigonometry.” Mathematical Reasoning questions include the subdomains subdomains the include questions Reasoning Mathematical of“Properties and “Trigonometry.” Shapes,” “Logic” and “Word and Mathematical Reasoning. Algebra questions include the subdomains “Arithmetic and Operations” and and and Operations” subdomains “Arithmetic the include questions Algebra Reasoning. and Mathematical Geometry,” Geometry subdomains “Coordinate the include questions and Equations.” Expressions “Algebraic QUANTITATIVE REASONING TEST QUANTITATIVE their logic and reasoning ability across differen across ability logic and reasoning their understand shapes. Students are asked to complete a variety of a variety complete to asked are Students understand shapes. of questions assess to in order subtypes various Quantitative Reasoning Test Reasoning Quantitative 11

CHAPTER TWO 114.  Wor 112.  Logic Tr Properties Coordinat igonometry d Problems At agiftstore, candlesare soldinpackages of4,chocolates are soldinpackages of10,andthank-you cards are soldinpack A studenthasinvented thefollowing rulefor right t smallest number ofgiftbagsshecanmake suchthatshedoesn't have any items l ages of3.Miranda isputtingtogether giftbags,eachofwhich containsonecandle,chocolate, andonecard. Whatisthe Which ofthefollowing isacounterexample thatdisproves theabove All right triangles have sidelengthsintheratio D) 120 C) 60 B) 30 A) 20 D) Atriangle withsidelengths7, C) Atriangle withsidelengths6, B) Atriangle withsidelengths5,1 A) Atriangle withsidelengths2, e Geometry ofShapes 3,and4. 8, and10. 7, and10. 2, and13. of 3:4:5. riangles:

statement? eft over? - 12

CHAPTER TWO ermitted. ur claims. ur claims. ur claims. ve levels. ve lculator. lculator. y prompts: Describe what you believe a community to be. What defines it? How large is it? What is it? What Describe what you believebe. What defines it? a community to How large deeply concerned The Stoic philosophers were by emotion to and its tendency If so, explain in any situations in which censorship of works is appropriate? there Are Provide examples from history or literature to support yo history or literature examples from Provide If not, explain why not. Use examples to support yo what context and why. are its boundaries, and what determines who is inside and out of it? You can draw on can draw on its boundaries, and what determines who is inside and out of it?are You or literary historical, examples to support yo contemporary, or can it aid reason? to reason, overwhelm. Can emotion be a good thing? Is it a threat progress.

ge of 11.

In Fall 2018, CLT added an optional unscored essay section, in order to enable students to provide colleges colleges provide to students enable to in order section, essay unscored an optional added 2018, CLT In Fall Level 1 questions are the least difficult, and require straightforward reasoning, basic logic, and a minimal basic logic, and a minimal reasoning, straightforward require the least difficult, and 1 questions are Level Difficulty levels levels Difficulty scale on a are scored of questions each section through 5: of 1 of the test contains 8 questions Reading passages in the Verbal Reasoning and Grammar/Writing sections are calibrated to fit narrowly within to fit narrowly calibrated are sections and Grammar/Writing Reasoning passages in the Verbal Reading The CLT is meant to test students’ logical reasoning abilities and their ability to understand and their ability abilities and simplify logical reasoning to test students’ is meant The CLT Calculators are not allowed on the CLT, including on the Quantitative Reasoning section. Questions are Questions are section. Reasoning on the Quantitative including on the CLT, allowed not are Calculators Sample Essay 3:  Sample Essay 2:  The following are sample essa sample are The following Sample Essay 1:  with a sample ofwith The prompt. one answer to 30 minutes have Students limit. a time under ability writing their oftext colleges. to scores their send students when results test their included with be may response written their Optional Essay Optional number of steps to answer. Level 5 questions are the most difficult, and require more complex reasoning, high- reasoning, complex more require of the most difficult, and number 5 questions are Level steps to answer. breakdown The concepts. difficult and the ability to synthesize thinking, level levels is perfectly difficulty of balanced, with 20% of of falling in each questions the fi difficulty as they at each difficulty level, for a total of 24 questions at each difficulty level. On the Verbal Reasoning and Grammar/ the Verbal for a total level, difficulty each at level. On each difficulty at questions of 24 are there which for passage, Each passage. each evenly throughout are distributed levels section, difficulty Writing questions of has 2 in 10 questions, Reasoning section, questions increase On the Quantitative level. difficulty each level ratings, to analyze the difficulty level level the difficulty to analyze ratings, level a range it falls within passage and ensure each of of TE 10-12, with an avera a consistent difficulty level. The test developers use TextEvaluator™, a passage calibration software with software grade a passage calibration TextEvaluator™, use developers The test level. a consistent difficulty Difficulty Levels Difficulty complex topics, rather than testing students’ ability to make complicated calculations. This test policy also secures calculations. students’ ability to make complicated than testing rather topics, complex p are models calculator which monitor and specify to need the avoiding by integrity test the simplifies and designed to be solvable without the use or need of without the to be solvable designed a ca Calculator Policy Calculator 13

CHAPTER TWO

schedule. accurate. us tests. us tests. ure tests. ure AL PROCESS 3.TEST DEVELOPMENT 3.TEST emic year. emic

On the basis ofOn the analysis of this and in conformity past eams, 2, in Chapter laid out blueprints test the with After a test date, once the analytics from a test become available, the Test Development Team meets to to meets Team Development Test the available, become a test analytics from once the date, a test After For each test administered, CLT adheres to a schedule for test development, proofreading, and preparation. and preparation. proofreading, development, test for a schedule to adheres CLT administered, test each For CLT chooses item writers based on their qualifications and demonstrated ability in particular subject areas. areas. subject in particular ability and demonstrated writers item qualifications on their based chooses CLT The CLT writes and edits each test according to a specific set of parameters. The Test Development, of set a specific to according test each and edits writes Development, Test The CLT The parameters. the test goes through two rounds of developmental editing to check for adherence to CLT standards, accuracy, accuracy, of rounds standards, two through goes test the CLT to adherence for check to editing developmental of number a consistent contains test Each fatigue.’ ‘question avoid to rounds between Editors rotate and clarity. Blueprints” 2). The in Chapter sections Editorial “Test the (see domain and subdomain within each questions upcoming acad initial development, After date. test every for key and answer a new test creates Team Development Test the Development Team looks at question performance within each specific difficulty level, and especially any especially and level, performance question looks at difficulty specific each within Team Development Team perform that questions Development of outside Test the of end the At expectations. year, school each the going into guidelines revised and creates tests year’s academic that from all feedback review to meets DEVELOPMENT AND EDITORI performancediscuss the of The Test form, as necessary. test the re-calibrate and to and items sub-sections This schedule is cyclical because each new test takes into account the analytics from previo from analytics account the into takes test new each because is cyclical This schedule Design, Review, and Uploading Review, Design, that each section maintains the high standards of each CLT exam, and is consistent, clear, and of high standards the maintains section each that clear, and is consistent, exam, CLT each Many have experience in fields such as teaching and tutoring. New item writers are supervised by experienced experienced by supervised writers New item are tutoring. and teaching as such in fields experience have Many members of of breakdown on the trained and are team, writing test the and levels, difficulty types, question of of style house rounds multiple through goes then Their work exam. ensure to and editing revision CLT the Selecting and Training Item Writers Writers Item Training and Selecting development, review, and uploading, so that every test undergoes quality control and is ready on and is ready quality control undergoes test every so that and uploading, review, development, of results the analyzes of CLT creation in the data that and uses exam previous each fut Editorial, and Operations Teams work together in the test preparation process. They follow a schedule of a schedule follow They process. preparation test in the together work Teams and Operations Editorial, 15

CHAPTER THREE Post-Upload Review provided inChapter 9for two tests administered inAp this kindis creating anditem creation. afeedback loopbetween Detailed item item analysis of performance content from future test administrations, data are analyzed to identify trends andscreen out certain types of students who answered each answer option. These item onthe test, andthe percentage of subdomain, rankof performance, includingitem difficulty, the passage connected with each item, subject, academic domainand aggregation. CLT analyzes that factors are correlatedfor to item each item, andat different levels of Item Review test data inthe website User Interface are test questions, passages, a components of images through the website User Interface, which includes infrastructure for constituent test management. The document to the Operations Team for uploading. Operations The Team the uploads exam andallpassages and UPLOADI e in advance of answer key andtest content aswell aspro proofreaders simulate andeditors takingthe fulltest duringeach review, the which includes afurther check of ou nfratn,frua,adohrtpso errors which might be introduced with the new focus onformatting, formulas, andother types of the uploadedThe test. paper test will then be reviewed in its entirety final by a of version new editor, with a particular andthe test isconsidered are performed that there are noduplicate answer choices. Once any last changes are madeto the uploaded test, final checks the answers, and to ensure original answer key (that noerrors were introduced), of to finalize the formatting Editorial Team initial andfinal checks tomake sure that performs the answer key onlinecorresponds to the post-upload review,online version.final The reviewer completely is new to the test. In these two therounds of Operations the test. The Team providesreviews of technical support to make any needed changes to the post-upload Editorial Teamand faithfulness to the originaltest document. The two rounds then performs of each question to its associated image orpassage andproofreads the finalupload for completeness, mechanics, POST-UPLOAD REVI each field itexactly is as represented in the test document drafted by the Edito the question type (e.g., “Comprehension—Passage Relationships”). Operations The Team replicates data in of the question (1-5), and answers A,B, C,andD, the correct answer (A,B, C,orD),the difficulty the text of the question itself, the URLassociated with images, the uploaded passage with which the question isassociated, T est Administere fe ahts diitain L opee hruhie nlsso the test. CLTAfter each test administration, examines CLT p-values completes athorough item analysisof editing andproofreading,After the Editorial Team they sendthe test subjects the test to these rounds of Rights andPermissionsThe Team secures for rights any passages under orimages copyright at least 8weeks f ae eso fthe test is required, the editorial team will create the and paper format document using the a paper version of If the passages, images, andtest questions are replicated inthe website, the Operations TeamOnce allof links digital infrastructureThe for test questions includes variable fields for question (1-120), the textnumbers of T est Finalized 2 Roundsof NG PROCESS d ach exam. Analytics Collected EW PROCESS New T T est Uploaded 2 RoundsofEditing complete. est Developed ofreading. ril 2018. the Editor the test’s accuracyandvalidity, under the oversight of additional reviews andproofreaderseditors of perform during the test composition process. Independent Team may, at their discretion, edit passages for length the test through the editorial process. Editorial The the CLT quality standards. help guide In-house editors difficulty levels are well-calibrated, andquestions meet Team that question categories are accurate, confirms spr fthe test development andeditorial process, part of As ial Team. rial Team. nd images. test mode. 16

CHAPTER THREE ’s system. ’s

ibraries, ibraries, Saturdays. HOOL TESTS he spring. he 4.TEST ADMINISTRATION 4.TEST EST SITES EST SITES document. end time. end

ing date. ing date. For in-school testing, instead of individual students registering themselves, CLT creates a school administrator instead of administrator a school creates testing, in-school CLT themselves, For individual students registering In order to register a location as a test site, a point of as a test site, a location to register In order test site complete a must the location representing contact Public CLT test sites are set up at a variety of a variety up at set test sites are l CLT schools, Public be colleges, These sites may locations. In-school tests are offered for specific U.S. secondary schools which contract with CLT to administer the test totest the to administer CLT a with contract which schools secondary U.S. for specific offered are tests In-school The public CLT is open to any registrant who is interested in taking the CLT, whether for personal personal for whether CLT, the in taking is interested who registrant any is open to CLT The public The CLT is offered multiple times per year as a public test, open to all students, and as an in-school test, for for test, and as an in-school to all students, open test, as a public year per times multiple is offered The CLT account in which a school may register students through a bulk import process. This process is outlined under the is outlined under This process import process. a bulk students through register may a school account in which and consists of profile administrator school’s batch uploading a CLT of student names and user IDs into REGISTERING STUDENTS FOR IN-SC contract. Once these schools are in CLT’s system, they may opt in or out of opt may system, they each for as a test site in CLT’s are being registered contract. Once these schools test testing centers, and any private institutions that meet the criteria laid out in CLT’s test site and proctor test site and proctor in CLT’s meet the criteria laid out institutions that private and any testing centers, requirements PUBLIC T selection of their students.This test may be offered online or on paper. In-school test dates are always on Wednesdays Wednesdays on always are dates test ofselection In-school or on paper. online offered be may test students.This their is permitted flexibility slightly more for that except CLT, public to the is identical process The administration and start the able to access the testing site. Public test dates are always on always are dates test Public site. testing the access to able once in the fall and once in t fall and once in the once be student the is that The limitation only intrinsic exam. entrance as a college use or for practice, assessment, students at particular institutions. In the 2018-2019 academic year, CLT will offer a public test date four times, times, four date test a public will offer CLT year, academic 2018-2019 In the institutions. particular at students twice, date an in-school will offer CLT academic year 2018-2019 In the spring. in the fall and twice in the twice Public vs. In-School Testing In-School vs. Public 17

CHAPTER FOUR 1 access tothis may begranted administrator known account. Onlypersons to CLT to work at the school—usually the CLT’s direct contact— Schoolsadvanced see andmetrics analytics through this onhow those students performed. their CLT school the students whose names anduser IDsthey uploaded,aswell as administrators cansee the scores of as many colleges asthey would like, free the which the test istaken. Scores are released after the last test site hassent intheir results, usuallyearly evening of Test Scores for both proctors onallpossible andstudents dependent va Reasonin 30-minute essay. There isa10-minute restroom break between the Grammar/Writing andQuantitative minutes 120minutes, for Quantitative not includingthe optional Reasoning, for atotal test-taking time of minutes are allotted for the proctor to provide instr optional essay, extra andwhether time for ornotthe student has been granted accom Test Day Schedules CLT immediately assuch atesting violation results inavoided te a student opens anew browser orapreviously saved document, proctors are requiredin the room. to notify If not choose to complete the essay are dismissed after they finish the multiple choice sections of optional essay. whether ornot the proctor willstay determines anadditional 30minutes. This Students who do procedures for proctoring amanual. Proctors must become familiarwith the proctor manual, which carefully outlines the of in the form or not there are accom CLTpayment information. proctors are compensated astandard amount, which varies onwhether dependent coach. Once proctors are selected, they must complete aproctor contract inwhich they provide contact and the students they are player proctoringexams, isnotacurrent orformer for provided that that at least one of the students they are proctoring.age orolderandmay not berelated to any Athletic coaches may of proctor proctor selection guidelines. CLT proctors mustcarefully be twenty-one selects proctors years aset using of of Proctors Students taking anin-school test see their scores insame way asthose taking the public CLT. School To see their scores, students simplyloginto their account. From there, they are able to share their scores with CLT’s the online test to students onthe same day on distinctive features isthatOne of it releases scores of Please request the CLTof the different Official possible schedules Test Timecodes foracomplete breakdown Students have 40minutes for the Verbal Reasoning section, 35minutes for Grammar/Writing, and45 Students are required toarrive 15minutes before the scheduled test time. At the scheduled test time, 20 the test location, whether ornot any student istaking the schedule onthe time zoneThe isdependent of While the test isinsession, proctors are instructed to positionthemselves where they canview every screen Before the test begins, the proctor asksthe students to indicate whether ornot they planto complete the time to provide them with basic CLTproctor training willcoordinate staff members with proctors ahead of Proctors are often, but donot have to be, connected with the institution that isserving asatesting site. CLT test day. 2018-2019Public CLT Proctor Manual. g sections modations. the CLT. account. 1

fcharge. of uctions. riables. st score. modations. the test. 18

CHAPTER FOUR t enabled. version. version. ime zone. ime zone. the exam. the bed above. bed dization. dization. aper test. aper accounts. accounts. NISTRATION g changes: ons Team. ons Team. st manual. st

PER OPTION For each test day, CLT has a dedicated team of has a dedicated answer to CLT available are who customer service representatives test day, each For There is currently no essay on the p on the no essay is currently There a .csv into sheets answer their of 30 days within scores their converts receive Students CLT date. test the Students complete a paper answer sheet which proctors are instructed to submit, by mail or by email, to the the to email, or by mail by submit, to instructed are proctors which sheet answer a paper complete Students test before students their register Schools colleges. with scores share to in order accounts CLT need Students The exam booklets and answer sheets are mailed to the school through FedEx or UPS a minimum of a minimum or UPS FedEx through school the to mailed are sheets one and answer booklets The exam The CLT may also be administered on paper upon special request from the schools. This option is available This option is available schools. the from request upon special on paper administered also be may The CLT On occasion certain testing sites are able to provide devices for all of for devices to provide able not testing sites are On occasion certain students are In this case, their students. Questions in the Quantitative Reasoning portion of portion Reasoning Quantitative the in Questions Mathematical notation. mathematical include may exam the Students who take the CLT as a public test do so online on their own devices, which they bring to the testing they which devices, own test do so online on their as a public take the CLT Students who The CLT is administered in two modes: primarily online through http://cltexam.com, and occasionally with and occasionally with http://cltexam.com, through online primarily modes: in two is administered The CLT purpose, CLT uses Zendesk Chat/Zopim to field all questions. This platform allows CLT to respond quickly and to This platform CLT all questions. to field allows uses Zendesk Chat/Zopim CLT purpose, support is test-day tests, CLT and in-school For both public or phone. chat requests via different to many directly test site and t every for leaves until after the proctor arrives the proctor before from available questions from schools, proctors, and parents. This team includes the members of and parents. proctors, schools, questions from the operations/implementation team, members of this customer service team. For as the regular team, as well the technology development answer sheets to ensure that each student’s answers are faithfully represented therein. CLT then uploads these uploads these then CLT therein. represented faithfully answers are student’s each that to ensure sheets answer CLT students’ into the documents Support CLT Day Test spreadsheet using InspiroScan. The Operations Team then cross-references the spreadsheet with the original original the with spreadsheet the cross-references then Team The Operations using InspiroScan. spreadsheet day to allow CLT time to give the schools their students’ user IDs, which is a required field on the answer sheet. sheet. answer the on field is a required which IDs, user students’ their schools the give to time CLT allow to day as descri import, bulk This or through individually done can be CLT Operati CLT at test sites administering the exam on paper receive three (3) additional copies of copies (3) additional three receive on paper exam the administering sites test at (3) extra and three test the in outlined process, a strict follow to expected are proctors CLT, regular the As with as a fail-safe. sheets answer te paper the week ahead of ahead week of attention the to sent are They date. test the primary of point school’s the Proctors contact. for in-school test dates only, and not for public test dates. Parameters are the same as those for in-school tests, tests, in-school for as those same the are Parameters dates. test public for and not only, dates test in-school for followin the with PA required to bring their own devices. This situation is the exception and This exception is the situation is devices. to bring their own discussedrequired individually with the testing of point site standar ensure to and students contact work on most modern devices and browsers. It requires a reliable internet a reliable with Javascrip connection It requires on most modern and browsers. devices work of regardless visible will be and thus in HTML, is scripted notation take to used browser the site. This device may be a desktop computer, a laptop computer (including a chromebook) or a tablet. The test will or a tablet. computer (including a chromebook) a laptop be a desktop computer, This may device site. 2018 academic year, 84% of 84% paper was on the and 16% online, was administration year, 2018 academic CLT total the ONLINE ADMI paper and pencil; the latter only for in-school testing and by special request of request special by and testing in-school only for latter and pencil; the paper 2017- the During school. the Testing Formats Testing 19

CHAPTER FOUR

r profile. ess level. ess ty level. level. ty rocedures. CLT data. CLT PTABLE USE PTABLE d parties. 5.TEST SECURITY s consent. RT ACCESS ACCESS RT SS CONTROL

CLT employees are granted a support role in order to access necessary access to information in order granted customers. support to are role a support employees CLT Web Application Access – The CLT web application security is role-based. By default, all users who register register all users default, By who is role-based. security application web Application Access – The CLT Web CLT data may be accessed either through the web application or through the database directly. All users must All users must directly. database the or through application web the through either accessed be may data CLT Proctors can view limited student data on test day to facilitate the test and verify attendance. Proctors do not do not Proctors attendance. and verify test the facilitate to day test on data student limited can view Proctors Students may opt to share their profile and test results with specific colleges colleges specific with results test and profile their share to opt may Students into and/or opt interest their of CLT considers all student data confidential, including collected identifiable information (email and profile profile information identifiable including collected confidential, (email and considers data all student CLT CLT trains all its employees on the sensitivity levels of levels sensitivity the on of and use access the including employees all its trains data, CLT CLT confidential CLT test security is designed to ensure fairness and equality among test-takers and is broken down into fairness into ensure to down is designed security test among test-takers equality and and is broken CLT Users in a support role can view test registrations and view student data, but they cannot access the test test the access cannot they but data, student and view registrations test can view Users role in a support SUPPO for an account receive the same level of same level the an account receive for minimal acc most the as students, access be authenticated to access CLT data, and authorization is based on securi is based and authorization data, CLT access to authenticated be student information with any thir informationstudent any with ACCE have access to a student’s full profile, test history, or any other data. Proctors are not permitted to share any permitted not are Proctors any data. other to share or any history, test full profile, a student’s to access have CLT’s partnership program in which CLT shares students’ data with partner institutions. Students who opt in in opt who Students institutions. partner with data students’ shares CLT partnership in which program CLT’s of out also opt may thei and editing application web logging CLT the into by time any at program the data) as well as test results. CLT employees may not share any student’s data with a third party without that that without party a third with data student’s any not share may employees CLT results. as test as well data) expres student’s DATA PRIVACY AND ACCE AND PRIVACY DATA data such as personally identifiable information. CLT requires each employee to acknowledge and sign internalto acknowledge employee each requires information. as personally identifiable such CLT data of use acceptable the regarding policies Data Security Data general data security and test day processes and p processes day and test security data general 21

CHAPTER FIVE h olwn ye fapproved the following types of manual instructs proctors to verify students’ identity before admitting them intothe testing room, usingany of ADMITTING STUDENTS INTOTHETE provides them with the CLT procto playerproctoring isnot acurrent orformer for that coach. CLT creates CLT accounts for allproctors and they are proctoring. the students Athletic they coaches may are proctor exams, provided that at least one of PROCTORS CLT test sites are secondar majority of The T Test Day Processes andProcedures will be In the event that user data iscompromised, the issue willbe immediately remediated andthe affected parties INCIDENT MANAGEMENTAN on aregular basisto ensure that have users not been inappropriately acces granted monitored onaregular basisto identify breaches, risks, orunexpected behavior. User roles are alsomonitored MONITORING AN employees. security a third-party The cloudprovider ensures the highest level of change isrequired. Some users, such onlyaread-only asthose onthe Analytics team, may be granted accounts, one read-only andone administrative use their read-only account. Users account unless acritical teams. Network traffic toaccess the database is restrictedby IP address. granted Eachtwo privilegeduser is DATABASE/NETWO Technolog onlybyaccess may be granted the Chief test data. Privileged required andthe acceptable use of to signanadditional policyregarding test integrity answers inexisting tests, change test dates anddeadlines, anddeactivate tests. test dates. includes the ability to addtests, This addandedit questions and modify test data in advance of PRIVILEG the ap management section of EST SITES the students age orolder andmay not be relatedCLT to any of proctors must be twenty-one years of Students must take the exam at anapproved CLT test site that hassubmitted atest site contract with CLT. Students may take the CLT onlyunder secure, supervised conditions. conditions are These a CLTThe Executive Team data. manages allincidents, includingdata breaches and/orunacceptable use of All activities are when logged changes are made inthe software, database, andinfrastructure. is Logging Physical Access –allCLT data isstored inasecure cloudenvironment that isnot accessible by CLT Accessing the database directly fallsunder privileged access andislimited to the development and analytics CLT employees haveA limited number of privileged access that allows them accessto write, review, and On test day, CLT students for their specific test site on their proctors have CLT accounts. the finallist of The contacted. ED ACCESS D AUDITING RK ACCESS plication. photo ID: r manual. y schools. D RESPONSE y Officer. STING ROOM

Privileged access users arePrivileged access users nd access. s to data. s follows: account. 22

CHAPTER FIVE

yet begun. yet been read. been e proctor. e STING ROOM the exam: the THE EXAM THE EXAM provides) ime Zone. Zone. ime ting room: ting d student. ectively. ectively. included) year only) year only) f any kind ilent mode nt ID Form ing device litary ID litary ID

Calculator material o Reference Test ID paper that the proctor utensil (for scratch Writing the ability to communicate with other students, or a calculator, A watch that does not have internet availability, and that is set to s CLT Stude CLT Passport (if photo license or permit Driver’s State ID Mi ID (current High school ID (current HSLDA student » » » » » » » » » » » » » » » » » » » » » » » » » »

In order to take the exam on test day, students must enter the proctor password specific to the exam in exam in the to specific password proctor the enter must students day, on test exam the take to In order Students may bring the following into the tes the into following bring the may Students ELL PHONES Test start times are staggered across US times zones to minimize instances of instances minimize to zones times US across staggered are times start zone time in an earlier students Test One ofOne for lengths time designated the to adhere all students that primary ensure is to duties proctors’ the

These must be turned be in a backpack or purseTheseth with or left must off and stowed Students are not permitted not are Students of any use to during following the

Proctors then assign seats for every admitte every for assign seats then Proctors TART TIMES TART The start time is 10:00 AM in the Eastern and Central Time Zones, 9:00 AM in the Mountain, Pacific, and and The Eastern is 10:00 AM in the time start Pacific, Mountain, 9:00 AM in the Zones, Time and Central T Hawaii and 8:00 AM in the Zones, Alaska Time S not have who zone time in a later students with it about communicating then exam, the taking top-left, center, and right of center, top-left, resp pages, the then begin reading the instructions for the next section. To aid the proctor in determining proctor aid the a glance whether at To section. next the for instructions the reading begin then of section appropriate on the working are students all the A similar aid is is color-coded. section each exam, the of to proctors available of names the exams: paper the at in bold printed are sections first, the and third second, each of each to instructed are proctors has elapsed, section a given for time allotted the Once sections. exam’s the and complied, have all students ensure devices, their of students notify hands from their remove them have this, TIMING TIMING question. The proctor password is displayed on proctors’ CLT accounts on test day—it is never communicated communicated is never day—it on test accounts CLT on proctors’ is displayed password Thequestion. proctor all once password this with students their provide to proctors instructs manual the email—and by them to have instructions preliminary and the and seated admitted been have students authorized PASSWORD C WHAT STUDENTS MAY NOT USE DURING MAY STUDENTS WHAT WHAT STUDENTS MAY BRING INTO THE TE BRING STUDENTS MAY WHAT 23

CHAPTER FIVE 1 proctors to upholdthem, CLT provides anexam that isfair, reliable, and and enforcing specific parameters for test administration. Bycommitting to these rules and training schools and Summary anomalies insection 2.4. are instructed tonote any proctor testing anomalies onthis report. The manual outlines procedures regarding ANOMALIES CLT ensures test security by limiting andcarefully monitoring access levels totest data, andby setting up Proctors must submit the Administration andAnomalyReport toCLT before exiting the testing room. They 2018-2019Public CLT Proctor Manual. » » » » » » » » » » » » » » » » » » » » » » » » » » Disturbances Duri Questions AskedDuri Students WhoBecomeIllDuri Students WhoUseanAdditionalDeviceorOpenAdditi Students WhoLeaveDuri Students WhoArriveLate Students WhoDon’t Arrive Si Devi Wi Pow Emergency E Copying Test te Failure Fi Failure er Failure ce Failure Materials vacuations ng anExam 1 Instructions for potential testing anomalies that are to benoted onthe repor ng anExam ng anExam to anExam to anExam ng anExam onal Page accurate. t include: 24

CHAPTER FIVE setting. setting. design. design. ACCESSIBILITY n college. ize bias. bias. ize 6.FAIRNESS & 6.FAIRNESS

CLT provides testing accommodations to students with documented disabilities to make testing accessible to to accessible testing make to disabilities documented with students to accommodations testing provides CLT The purpose of both an internal exam. are entrance as a college exams as well metric secondary school CLT the All CLT testing takes into account learning differences and disabilities in accordance with the Standards for for learning account into Standards takes the testing with in accordance and disabilities differences All CLT CLT is committed to providing every student with a good and fair test-taking experience. The CLT therefore therefore The CLT experience. a good and fair test-taking with student every providing to is committed CLT

all. CLT’s accommodations apply to those whose disabilities are cited by a licensed professional as hindering as hindering professional a licensed by cited are disabilities whose those apply to accommodations all. CLT’s educational in their used as being documented currently are accommodations and whose education Fairness in Test Accessibility Test in Fairness A given score does not guarantee admission to a give admission to guarantee not does score A given Fairness in Score Interpretations for Intended Uses Intended Interpretations for Score in Fairness Educational and Psychological Testing. The CLT also considers fairness concern, a top in testing The CLT thus works and it Testing. Psychological and Educational universal and ensure accessibility, facilitate bias, minimize to Fairness During the Testing Process Testing During the Fairness CLT seeks to promote diversity and minim diversity promote to seeks CLT them equal opportunity for success, while maintaining the rigor and integrity the maintaining while of success, for opportunity equal them the to adheres The CLT exam. the In compliance (ADA). Act Disabilities with Americans (IDEA) and the Act Education Disabilities Individuals with the test, and in keeping the on potential their fulfill to to all students door the to open efforts with its laws these with Overview offers of a range to give order in physical disabilities learning documented with students for accommodations or 25

CHAPTER SIX 1 of to fulfillthispetition while maintaining theintegrity of their disabilityandmakeaspecial request,andin what CLT CLTwillseek offers, students may submitcertification CLTaccommodations request form. willcontact studentsby emailwithany matters that require clar Accommodations Requ academic year’s accommodations request to CLT at leasteightweeks before thetestdate. submitted for morerequest thanone student willnot be c forms Testing Accommodations Req years who are interested intesting accommodations for additional exams may fillout andsubmit CLT’s Repeat official CLT and/or who have granted aaccommodations on the been CLT8 and/or CLT10 in the past five (5) Form, which isavailable onthe CLT website. Accommodations andRequests L ild t ett oo eussfracmoain fastudent’s desired accommodationCLT isnotyet included willdoitsbesttohonorrequests for accommodation. If thepertinent CLT reception of willnotifythestudent regarding thestudent’s approval status withintwo (2) weeks of Students requesting repeat CLT accommodations are notrequired to re-submit thisdocu documentation must beuploadedasPDForWord followingThe documentstotheCLT two (2)typesof accommodations requests,To requests foraccommodations must besubmitted ensure thetimelyfulfillmentof CLT makes the following accommodations available to students who provide appropriate docum disabilities forThe which CLT provides accommodation individualstudents. Accommodations of mustAll accommodations request be submitted onbehalf forms CLTFirst-time are test-takers required tosubmit fillout and the CLT Testing Accommodations Request CLT Testing Accommodations Request Form, https://www.cltprep.com/accommodations-request-form.html. » » » » » » » » » » » » » » » » » » » » » » » » » » » » » » » » » » » » » » » » » » » » » » Diagnosis byacertified relevant physicianaccompaniedbytest results anddiagn Student’s psychologicalevaluationincluding diagn One (1)ofthe Student’s IEP, 504,orschool-issuedofficialaccommoda Medication th Food breaks (mustbecontained inaclearplasticbagandremain unspoiledinroom temperature foratleast Preferenti Extended time(50%and100%allotmentsinbothuniformselectiv Signed in Written in Reader Large printexam(14-,16-, 18-,or20-p Braille exam(includesallgraphsan MP3 audiot Tic Traumatic Hea Physical Hearing Impairmentan Visual Impairmentand Psychiatric Communication Autism Spectrum Specific Learning ree hours) Disorders Disorders structions al seating structions Disorders est format following: d Injuries Disorders est Form. Disorders Disorders Blindness d Deafness uest Form. d figures) deadlines. oint font) TestingThe Accommodations Request Form lists the current 1 Students who have testing accommodations onthe been granted the exam. ostic code tions plan s include: onsidered. e formats) ostic code mentation. ification. entation. 26

CHAPTER SIX ore Report ore l student. zed tests. zed Image 7.1 Example Student Sc 7.TEST RESULTS colleges.

Students receive same-day test results as part as part results test same-day receive Students CLT provides tailored score reporting to students, colleges, and secondary schools (and home school school (and home and secondary schools colleges, students, to reporting score tailored provides CLT CLT has a unique philosophy behind its score reporting. The growth of The growth reporting. score its behind philosophy has a unique and the education higher CLT text oftext with essay the chooses at no additional cost. If cost. no additional at chooses student the or she he section, essay optional the completes the share to or not whether also choose may CLT website. An example is provided in Image in Image is provided An example website. CLT this 7.1. The share to option has the student or she as he colleges as many with report demographic, school, college preference, and and demographic, preference, college school, students to information.test This is available on the account online student’s the through Student Score Reports Score Student of personal their compiles which report a score organizations). For example, CLT provides secondary schools with advanced analytics on both student and and student on both analytics advanced with secondary schools provides CLT example, For organizations). of areas specific decipher to ability the schools This performance. gives reporting analytic school and strength individua each and for aggregate the level at both school weakness, to communicate scores that are indicative of indicative are is that that scores using content while communicate to and aptitude, achievement both standardi other than rigorous more and richer intellectually metrics pertaining to standardized test scores. Scores are used to inform used to of are Scores a myriad scores. test standardized to pertaining metrics as such decisions levels, ability academic students’ do indicate scores test While awards. and scholarship funding allocation works and thus tension this recognizes CLT should itself testing high-stakes such scrutiny. under always be Overview Overview ofindustrialization of world the led on have testing standardized hyper-focused become to education 27

CHAPTER SEVEN Chart students inthree mainschool types: Home School, Private School, and current CLTThe user group consists of to their us (Image 7.3).CLT percentiles are user-referenced onthe test ascompared andindicate how astudent performed include historical average scores for the school, aswell asscores andCLT percentiles for each student, per test (by section, see Image 7.2),aswell asexamine specific test portal. Inthis portal, school ontests administrators cansee how their school hasperformed their students. administrators have These access to their students’ scores through CLT’s online “School Admin” SchoolSecondary at these colleges receive those students’ CLT aswell score asthe ACT information, andSAT co College Score Reports Test results are available cost, and at two levels: basicandpremium. Basicanalytics are provided free of schools secondary andhome school organizations alsoreceive same-day test results for Administrators of their choice, admissionsstaff When students opt to send their score reports andoptional essays to colleges of er School. er group. Image 7.3BasicTest

Image 7.2Te Analytics st History Score Reports results. over time ncordance.

28

CHAPTER SEVEN

hematical hematical Logic d Problems Reasoning Mat Wor

the page. the of Shapes Geometry Geometry Coordinate ANTITATIVE REASONING Properties QU Equations Algebra Operations and Arithmetic and n Pairings Algebraic Expressions Algebraic Expressions Style Writing ord Choice ord Structure W AR/WRITING Structure Grammar GRAMM Agreement Punctuation and Punctuation Sentence Subdomains

as a whole ge Details prehension ationships Passage

Rel Passa Com Passage Passage f VERBAL and 7.5). REASONING l Analysis

Analysis Evidence Textua Interpretation o

Image 7.4 Four Domain-Subdomai and Bottom Analytics—Top Premium

Domain

Subdomain Table 7.1. Table SUBJECT SECTION Schools with premium analytics can also see any specific student’s test performance by section, domain, and subdomain.by section, performancetest student’s specific any can also see analytics premium with Schools Premium analytics are delivered to schools at two levels: school/home school organization-level, and student-level. and student-level. organization-level, school school/home levels: two at schools to delivered are analytics Premium Premium analytics give schools and home school organizations access to more detailed student and school- student detailed more to access organizations school and home schools analytics give Premium subdomain (Images 7.4 subdomain (Images information identifying of top the at student’s the includes This appears report but 7.5, similar to Image percent correct metric displays the percentage of percentage the displays metric correct percent specified the within correct got school that at students the questions pairingsfour domain-subdomain top and bottom the can see administrators School test. specified the for category, (in terms of of performance), as a breakdown as well performed school the domain, and section, how on each For each level, a percent correct metric is given for each section, domain, and subdomain. At the school-level, this school-level, the At domain, and subdomain. section, each for is given metric correct a percent level, each For level analytics. These form in the come analytics. oflevel performed students how on metrics of areas on certain test, the listed subdomain. These domain and academic academic domains and subdomains are by reported are which in Table 7.1 CLT Sections, Domains, and Sections, Domains, CLT 7.1 Table 29

CHAPTER SEVEN continue to innovate andexpand its test result reporting school administrators, andcolleges. Asmore andmore students take the test andthe userbase grows, CLT will stakeholders—students, educators, school secondary andhome insightful scores andmetrics to awide variety of tha college to more andscores easily download from the information students who have chosen to share scores with the 2018-2019 acad will be available by the end of feature educational similar(anddissimilar) This curricula. schools, organizations/associations, andschools of percentileaggregate metrics, aswell asvisualizations vs. andmetrics onhow the school other performed the ability to compare its results with other schools andassociations. willbe done by This providing school- CLT10, but included for students who take the CL byperformance section, domain,andsubdomain,willbe available for purchase for students who take the also provide to students anequivalent to the premium school analytics. analytics willfeature These student Forthcoming These reporting capabilities,These while still indevelopment, give CLT the opportunity to continue toprovide Furthermore, CLT willbuild into its website acollege data willprovide portal. This the opportunity for a Another feature indevelopment isa“school associations portal.” Onthis portal, CLT willprovide to schools the CLT continues to grow. scoreThe andanalytic reporting of capability ByFebruary 2018,CLT will t college. Premium Analytics—School PerformancebySection,Domain,and Image 7.5

Reporting emic year. Features T orCLT8. features. Subdomain 30

CHAPTER SEVEN - (3rd ed., pp. 221- ed., pp. (3rd (4th ed., pp. 187-220). Westport: Ameri 187-220). Westport: (4th ed., pp. Educational Measurement

ievement. ievement. S AT Educational Measurement Educational Measurement 8. CLT provides a concordance table that links CLT total scores to scores total links CLT that table a concordance provides CLT 2 CONCORDANCE CONCORDANCE RELATIONSHIP 1 CLT TO ACT/ TO CLT abilities. (p. 238). (p. n scores. n scores. ree tests. ree

Petersen, N. S., Kolen, M. J., & Hoover, H. D. (1989). Scaling, norming, and equating. In R. L. Linn (Ed.), and equating. norming, (1989). Scaling, H. D. & Hoover, M. J., Kolen, S., N. Petersen, Macmillan. 262). New York: (Ed.), In R. L. Brennan Linking and Equating. (2006). J. N. & Dorans, W. Holland, P. and Praeger. can Council on Education

These concordance tables enable students, parents, educators, and colleges that meaningfully compares scores scores compares meaningfully that and colleges educators, parents, students, enable tables concordance These In order to give context to CLT scores outside the CLT user group, CLT has developed a set of a set has developed CLT group, user concordance CLT the outside scores CLT to context give to In order Quantitatively, the CLT provides two main measures to give context to its scores: CLT percentiles and ACT and and ACT percentiles CLT scores: its to context give to main measures two provides CLT the Quantitatively, The CLT aims to provide context for its scores in a number of ways, both qualitative and quantitative. The CLT of in a number scores its The for CLT context quantitative. and provide aims to qualitative both The ways, CLT Metrics are meaningless in a vacuum. This is true for any metric, including test scores. However, with the the with However, scores. including test metric, any This meaningless for in a vacuum. is true are Metrics 1 2 between the th the between between the scores” on two different tests (p. 193). (p. tests different on two scores” the between certain to scores section link CLT that tables as concordance as well scores, total and SAT scores Composite ACT sectio and SAT ACT of students” college-bound of examples are Holland and Dorans (2006), “Concordances to According tables. links direct produce scalings that While work remains to be done to establish the national norms for the CLT, the user norm user the as regarded can be norms national the establish to done be to CLT, the remains for work While norms the profile to allow is sufficient stability “this meaningful for that write al (1989) et. be used to Petersen stable. a group to in relation stands or he she where accurately, reasonably assess, to an examinee allow comparisons and to SAT concordance scores. Percentiles are user-normed: that is, they are based on the students who take the test. These test. the take who students on the based are they user-normed: is, are that Percentiles scores. concordance SAT are examinees performed CLT peers. examinee his or her to the how as compared test on the display percentiles of comprised typically students. school and charter students, private-school students, school home main groups: three student’s thinking and writing and writing thinking student’s and college preferences. These indicators can provide context to colleges that are interested in particular groups in particular interested are that colleges to context These can provide indicators preferences. and college a classical with students students, of school home colleges, arts in liberal an interest with students (e.g. students a display can which colleges, to sample writing a genuine provides section essay optional the Furthermore, education). accurate and meaningful insight into a student’s academic aptitude and ach aptitude academic a student’s into insight and meaningful accurate as demographic information, secondary such school data, information examinees, its about qualitative certain collects Overview an and colleges educators, parents, students, provide scores—can CLT scores—specifically, test proper context, 31

CHAPTER EIGHT regressions were then combined to produce the finalconcordan collected ACT andSATgroups andalsoaccounted for scores. anattenuated results The from range of these any difference in the population over an equipercentile served method. linking This to minimize the effect of non-public education. To linear regression waschosen in the world overcome of this obstacle, the method of the test, aswell asthe CLT’s specific appeal to students, isdue to the new nature families,This of andschools ACTand CLTSAT examinees from examinees. differs group the population of date, the population group of shows item countsfor eac section scores were concorded tosections onthe ACT andSAT according to Table 8.1above. table The also students who tookthe CLT. CLT total scores were concorded to ACT Composite andSAT total scores. CLT the study were completed s phase wasconcurrent, while the other parts of and publication. data The collection for each regression, combiningthe data, quality control, parts: data collection, data validation, linear comparable sections onthe AC tables that compared CLT section scores to in August 2018andproduced concordance total scores. secondphase wascompleted The total scores, ACT Composite scores, andSAT produced aconcordance table between CLT phase, first The completed in June 2018, by astudy that wascompleted intwo phases. Study Design One obstacle that hadto be addressed wasthe differencegroups inpopulation between the three tests. To dataThe were collected from June 2016to May 2018by collecting ACT andSAT from score information similar the study consisted of Each phase of CLTThe concordance tables were developed Table 8.3DescriptiveStatisticsoftheSamplesforConcordance Analysis,C Table 8.2DescriptiveStatisticsoftheSamplesforConcordance Analysis,C h section. eparately. T andSAT. Table 8.1 Grammar/Writing Quantitative Verbal Reasoning Total ( (40 Items) (40 Items) (40 Items) soning CLT 120 Items) CLT, AC Respective sectionsofconcordance between ce tables.

Rea T, andSAT - Reading English English Ma ( Composite 215 Items) (60 Items) thematics ACT (75 Items) (40 Items)

LT and SAT LT andACT Reading andWrit Total ( Math Evidence-Based ing (96 Items) SAT 154 Items) (58 Items) the - 32

CHAPTER EIGHT ACT Scores -counts of-counts and test test each between correlation the sample, the n rough 8.2. rough

Figure 8.1 Scatter Plots of CLT Scores vs. Scores 8.1 Scatter Plots of CLT Figure Tables 8.2 and 8.3 below present the the present 8.2 and 8.3 below Tables the correlation between the CLT and its respective sections on the other two tests. The scatter plots indicate indicate plots The scatter tests. two other on the sections respective and its CLT the between correlation the in of each between displayed These are relationships linear another. one to concorded and sections tests the 8.1 th Figures than the correlation between [CLT Quantitative Reasoning] and [SAT Math], which was in turn which higher than Math], and [SAT Reasoning] Quantitative [CLT between correlation the than and Reading Evidence-Based and [SAT Grammar/Writing] + Reasoning Verbal [CLT between correlation the include to and SAT, ACT and the CLT the between high correlation displayed however, sample, Each Writing]. correlation between [CLT Verbal Reasoning + Grammar/Writing] and [ACT English + Reading] was higher was higher English Reading] + and [ACT Grammar/Writing] + Reasoning Verbal [CLT between correlation Mathematics]. and [ACT Reasoning] Quantitative [CLT between concordance and concordance total the than was higher total and SAT total CLT between correlation the sample, concordance SAT the within However, section, and descriptive statistics on each test and test section. Within the CLT-ACT concordance sample, the the sample, concordance CLT-ACT the Within section. and test test on each statistics and descriptive section, Results 33

CHAPTER EIGHT 3 either ACT orS asdisplaying CLTpages) canbe interpreted scores in the context of between students who score perfectly onboth the ACT andSAT. concordance These tables (onthe following SAT scores which concord with top CLT scores. CLT The provides anopportunity for further differentiation ACT andSAT. to ensure consistency bothwith CLT’s correlations andwith the published concordance tables between the Summary Figure 8.2ScatterPlotsof CLT Scores vs. h w eso correlations (CLT andACT, twoThe sets of CLT andSAT) were examined andthen blended with the other  documents/ACT-SAT-Concordance-Information.pdf. Guide tothe2018 ACT® /SAT® Concordance, CollegeBoard, The ACT, Inc.,2018,http://www.act.org/content/dam/act/unsecured/ 3 Top ACT andSAT scores are displayed as“N/A” because there are noequivalent ACT or SAT Scores AT scores. 34

CHAPTER EIGHT

ance Table ance Table 8.4 CLT Total Score Concord Score Total 8.4 CLT Table 35

CHAPTER EIGHT Table 8.4CLT Total Score Concordance Table ( Continued) 36

CHAPTER EIGHT

ance Table Table 8.5 CLT Verbal Reasoning+Grammar/Writing Concord Reasoning+Grammar/Writing Verbal CLT 8.5 Table 37

CHAPTER EIGHT Table 8.6CLT QuantitativeReasoningConcord ance Table 38

CHAPTER EIGHT g equation ial bias. ial bias. , 3,1-18. Journal of Journal Psychology Mathematical , and N is the total number of number total , and N is the item. the answering students I on item on item j (9.1) 9. ITEM ANALYSIS strations. Option or distractor analysis is often conducted to further investigate the discrimination power power discrimination the investigate further to conducted analysis is often or distractor Option RIMINATION 1 is the item score for student student for score item is the

ij DIFFICULTY X

Novick, M.R. (1966) The of axioms and principal results Novick, classical test theory.

Item discrimination is another measure that is often used to quantify the item property with the CTT CTT the with property item the quantify to used is often that measure is another discrimination Item where Item analysis is often conducted to evaluate the psychometric properties of properties psychometric the evaluate to conducted analysis is often Item form. on a test individual items 1 framework. In general, items that can distinguish high ability from low ability students are considered good good considered are students ability low from high ability distinguish can that items In general, framework. ITEM DISC (p-value<0.25). The p-value is group dependent. It may vary as the group ability changes. The p-value may be be may The p-value changes. ability group vary as the dependent. is group The may It p-value (p-value<0.25). comparable is not This index group. ability a low for lower get may it whereas group a high ability for higher admini test different across correctly, and thus the item is judged to be easier. Vice versa, a lower p-value means that fewer students have have students fewer that means p-value a lower versa, Vice easier. be is to judged item the and thus correctly, the falls within p-value the Usually difficult. more be to is judged item the and thus correctly, item the answered ofrange difficult or very (p-value>0.9) easy very could be falls this range outside which 0.9. An item 0.25 to Theoretically, p-values range from 0 to 1. A higher p-value means that more students have answered the item item the answered have students more that means p-value 1. A higher 0 to from range p-values Theoretically, percentage of students who answer the item correctly. It can be computed using the followin using the can be computed It ofpercentage correctly. item the answer who students ITEM in quantified is often termsp-value, item as also known the or score item mean the of difficulty, item In CTT, discrimination. of potent items with to flag examined is functioning item differential Further, item. the Introduction to Psychometric Evaluation Psychometric to Introduction in terms evaluated theory classical test of with are (CTT), items In accordance item and difficulty item 39

CHAPTER NINE Table 9.1OptionAnalysisforanItemwithGoodDiscr is 0.64. this item the low ability students chose p-value Options BandC.The of this item, while the majorityof key of the highandmiddle ability students chose Option D,For which isthe Item 1,itisevident that the majority of point-biserial correlation with the item 0.58for is Item 1inTable 9.1,while that for Item 2inTable 9.2is0.05. highest values. expected It is that the key option shouldhave the highest optionpoint-biserial correlation. The the specific item (the correct answer for thatThe in the option point-biserial are question). numbers the key of misco different types of the correct option while the low abilitystudents would be attracted by which the distractors often represent groups. For anitem with gooddiscriminating power, it isexpected that more highabilitystudents will choose ability levels. total test scores The canbeused to categorize students into low,of middle, andhighability students. Usuallythree subgroups are insuch used ananalysisby dividingstudents into three categories of exercised to scrutinize the item more carefully. percentage This computed is for allstudents andthe subgroups the missingpercentage islarger than 5%,caution shouldbe students who may skipthe item. If percentage of the alsoincluded missinganswers inthe is analysisto get ageneral idea of addition, often the percentage of OPTION/DISTRACTO flagged for further inves between the highandlow achievers. Items with apoint-biserial correlation between 0.1and0.25are often with apoint-biserial correlation smaller than 0.1are often deemed asnot discriminating enough to distinguish item quality. Usuallyitems with point-biserial correlations larger than 0.25are considered acceptable. Items and vice versa. There isnoconsensus related to the cut value for the point-biserial correlation to evaluate point-biserial correlations indicates that students who answer anitem correctly alsohave hightotal test scores the item scores andthe test scores respectively. the standard deviation of High denominator the is product of correlation canbe computed a item-total correlation for dichotomous items. The point-biserial point-biserial correlation isaspecific type of onthe overallthe relationship onanitem andtheir performance between students’ test. performance The the item-total correlation. Thisindex describes of items. Item discrimination canbe quantified in terms Item 1 The followingThe two tables (Tables 9.1and9.2)contrast two items. option highlighted inred The indicates the students choosing each option inadichotomous item. In Option/distractor analysisshows the percentage of numeratorThe inthe equation isthe covariance between the total test scores andthe item scores, while the Option Poin Hi Midd L Total ow Ability gh Ability le Ability t-Biserial nceptions. tigation. (9.2) s follows. R ANALYSIS Option-A -0.17 16% 20% 11% 4% Option-B -0.33 imination. 27% 40% 18% 3% Option-C -0.27 11% 35% 1% 7% Option-D 92% 46% 64% 0.52 5% Missing 0% 0% 0% 0%

40

CHAPTER NINE 1% 1% 0% 1%

Missing lustration an item. an item. Figure 1. 1. Figure DIF il 5% 5% 2% 15% -0.13 Option-D 0.20 51% 33% 49% 62% Option-C 22% 19% 23% 22%

-0.02 T CLT DATA imination. Option-B 21% 31% 23% 13% -0.16 Option-A stigation. UNCTIONING

expected. expected. EXAMPLE - NO le Ability le gh Ability t-Biserial ow Ability ow L Midd Hi Option Total Poin

2 Item Item

To further investigate the item psychometric properties, differential item functioning (DIF) is conducted. DIF DIF (DIF) is conducted. functioning item differential properties, psychometric item the investigate further To Further, the option point-biserial correlation may present more information more of quality the about present may correlation point-biserial option the Further, an item. On the other hand, the point-biserial correlation for Item 2 is low, thus this item has little discrimination. discrimination. has little item this thus 2 is low, Item for correlation point-biserial hand, the other On the assessment while a focal group refers to individuals who are at risk or being disadvantaged by the test, most most test, the by disadvantaged risk or being at are individuals who to refers group a focal while assessment of graph presence the The illustrates following or female. non-white the to related often DIF in matched on their ability measured in the test. This conditional comparison is often conducted between two two This test. in the between conducted comparison is often conditional measured ability on their matched refers group The reference group. focal the is called other the while group reference is called the one groups; in educational or male white the to related often most favor, to was expected test the which individuals for to DIFFERENTIAL ITEM F are they performance after in students’ two differences groups between unexpected on an item the to refers which raises a flag for further inve for further raises a flag which point-biserial correlations. In Table 9.1, the option point-biserial correlations for distractor options A, B, and and A, B, options distractor for correlations point-biserial option 9.1, the In Table correlations. point-biserial point-biserial option 9.2, the 2 in Table Item For as expected. is positive, key the for that while negative C are C is positive, Option distractor for point-biserial option the 0 while to is close option correct the for correlation When computing the option point-biserial correlation, each option would be treated as the correct answer answer correct as the treated be would option each correlation, point-biserial option the computing When option correct the that is expected It of regardless as incorrect be scored would options all other while key. the option negative or distractors options other will have while correlation option point-biserial positive will have positive option point-biserial correlation. This item is a difficult item with a p-value Thisp-value a with item is a difficult item correlation. point-biserial option positive Such occurrences of 0.22. un not but rare are Regardless of the ability levels, all students are attracted by Option C while Option B is the key. This is reflected This reflected is of key. the is B Option while C Option Regardless by attracted are students all levels, ability the highest C has the Option but correlation, point-biserial option has a negative but key B is the Option in that Table 9.2 Option Analysis for an Item with Low Discr with Low an Item for Analysis 9.2 Option Table 41

CHAPTER NINE individual itemscanbefound inAp 0.9. AllothersectionsinForm 1517and1618donothave itemswithp-values larger p-value than0.9. The for Form Form 1618that have 1517and1itemintheGrammar/Writing p-values larger than sectionof section of Writing, andQuantitative Reasoning respectively. Ontheotherhand,there are 4itemsinthe Grammar/Writing Form 1517,while thosefor Formfor each 1618are 0,2,and6for sectionof theVerbal Reasoning, Grammar/ itemswithp-valuestest inForm smallerthan 0.25are 2 1618having thelowest meanp-values. number The of for meanp-values allitemsinthesetwo The rangedfrom forms. 0.44to0.63,withtheQuantitative Reasoning ITEM 1517 including test forms Item Results Analysis for CLT Tests of Summary item content item isunfair. Expert review of item bias. Statistical DIFanalyses provide does not bias implythe aconvenient starting point for the study of of the construct for one subgroup. results The the intended construct orissomehow aless acceptable measure of carefully examined for aparticu potential biasagainst responding to anitem correctly.unequal probability Items incategory of BandCare andshouldbe flagged Table 9.3Cross-Tabulation Table forItemResponsesandGroup Me can be computed using Item p-valuesItem were computed for p-values each Table item in bothforms. 9.4 presents the descriptive statistics of All analyses introduced in the above section were 2018CLT conducted for the April administrations it measures attributes irrelevant to DIF does not necessarily mean that anitem biased. Anitem is isbiased if DIFindicates onlythat the students withequal abilityfrom different subgroups have an presenceThe of Category C:Large DIF, one contrast group isstrongly favored by the stu Category B:Moderate DIF, onecontrast group isslightly favored by the stu Category A:Negligible DIF, nocontrast grou Further, where cell frequencies represented by A,B, CandDinthe above equation canbe related to Mantel-Haenszel statistics andETS categories are two commonlyused DIFindexes. Mantel-Haenszel alpha Group DIFFICULTY and equation and1618. Total Focal Reference (9.3) pendix A1. tm r lsiidit n f the three DIF . Items are classified intoone of is needed. p favored; n1k Ck Ak 1 Item Response Y lar group. mbership. n0k Dk Bk 0 died item. died item; categories Tk nFk nRk Total Table 9.3. 42

CHAPTER NINE 0 4 0 0 1 0 p>0.9 4 7 5 5 2 4 1

The last two columns represent the number of number the columns represent The two last form in each section range the fall in each in items that The last two columns represent the number of number the columns represent The two last form in each section range fall in the in each items that

pendix A2. pendix 1618 1517 The point-biserial correlations are summarized in Table 9.5. The mean point-biserial correlations ranged ranged 9.5. The correlations point-biserial mean in Table summarized are The correlations point-biserial Note: Two types of DIF analyses were conducted, one for gender and the other for race. For gender DIF analyses, DIF analyses, gender of types For Two race. for other and the gender for one conducted, were DIF analyses As illustrated above, option/distractor analysis further demonstrates item performance item demonstrates ability further analysis option/distractor in different above, As illustrated Note: Form 1618 1517 Form Ap 9.5 Descriptive statistics of point-biserial co Table 0.1 to 0.25 raise a warning flag, and those with values less than 0.1 also raise a red flag, such informationis such flag, red a than 0.1 also raise less values with those and a warning 0.25 raise 0.1 to flag, of value lowest the 0.1 with than smaller correlation a point-biserial had Only 4 items as well. summarized 0.01 of section Reasoning in Quantitative in the found individual can be items 1618. The for point-biserial Form from 0.34 to 0.44 with the largest value of value largest the 0.44 with 0.34 to from 1618 and the in Form section Grammar/Writing 0.62 in the ranging from of values with 1618. As items value smallest in Form section Reasoning Quantitative 0.01 in the ITEM DISC of indicat the Table 9.4 Descriptive statistics of statistics 9.4 Descriptive Table the male group was designated as the reference group while the female group as the focal group. For the race race the For group. focal as the group female the while group reference as the was designated group male the confidentiality confidentiality DIFFERENTIAL ITEM F above and provided more detailed information about which option may be the potential cause for the low low the for cause potential the be may option information detailed more which about and provided above flagged. been correlations biserial point analysis and option Option/distractor have that in items discrimination stakeholders of relevant to provided can be individual item of signing the with each upon request for CLT a groups. For test security reasons, this information cannot be summarized in this document. In general, the information this the In general, document. in this summarized be cannot reasons, security test For groups. and summarized observed has been what information cross-validated further analyses in these collected OPTION/DISTRACTO of indicat the 43

CHAPTER NINE 2 Table 9.6Differential ItemF resultsDIF analysis for individualitems canbe found inAppendix items favored the reference groups while other items favored the focal about groups. detailed information The item content. More items were Category CDIFin flagged 1618. Some DIF requireForm further scrutiny of items ineach section were classified with Category ADIF, which isnegligible. Usually, items with Category C the of the CLT issummarized in withDIF items flagged for each section of Table 9.6.In general, the majority package (Magis, Beland, Tuerlinckx, &De Boeck, 2010)wasused with allthe default settings. was treated asthe focal group. Students with missinggroup indicators were excluded from DIFanalyses. difR DIF analysis, the White student group wastreated the as reference group while the Non-White student group functioning. Magis, D., Beland,S., Tuerlinckx, F. &DeBoeck, P. dichotomous different item (2010). A generalframework andanRpackage for the detectionof Form 1517 1618 Behavior Research Methods, Quantitative Quantitative Gramm Gramm Reasoning Reasoning Reasoning Reasoning Writing Writing Subject Verbal Verbal ar/ ar/

unctioning

42,847-862. Groupin variable Gender Gender Gender Gender Gender Gender Race Race Race Race Race Race g 40 40 40 40 40 40 40 40 40 40 40 40 N A3to A8. ubro Items inEach ET Number of 35 34 25 25 21 29 25 24 28 35 29 33 A 10 10 B 3 3 8 8 9 6 7 5 8 2 2 Tenme f number The of S Category 10 C 7 7 9 2 5 5 0 3 5 2 3 44

CHAPTER NINE aw scores. aw ifficulty. reported. 10. CLT REPORTED SCORES REPORTED 10. CLT Non-White.

In this section, descriptive statistics are summarized using adjusted scores by test form, testing mode, form, test mode, testing by scores adjusted using summarized are statistics descriptive section, In this The adjusted scores are reported to examinees on a 0-120 scale, and are determined and are which a method with scale, on a 0-120 examinees to reported are The scores adjusted Each subject test consists of 40 multiple-choice items. For each item, one score point is awarded for a a for is awarded point score one item, of each consists test subject Each For items. 40 multiple-choice The CLT contains three multiple-choice tests, Verbal Reasoning (VR), Grammar/Writing (GW), and (GW), and (VR), Grammar/Writing Reasoning Verbal tests, multiple-choice three contains The CLT White and White 2018, Form 1517 and Form 1618. Identifications of gender, race/ethnicity, and family income are based on based are and family income of 1618. Identifications race/ethnicity, 1517 and Form 2018, Form gender, of small number the to Due examinees. information self-reported from available racial groups in some students categories, general on two based are analyses Indian), the American and Hispanic, African American, (e.g., a scale that is consistent across test administrations and adjusted for test d test for and adjusted administrations test across is consistent that a scale in April administrations test two on the family income and student type, school and race), (gender subgroup are then compared with external anchors (ACT and SAT scores), as well as repeat-person scores, taking into into taking scores, as repeat-person as well scores), and SAT external with compared then (ACT anchors are this shapes of distributional the account From administrations. test previous to as compared scores raw the onto score) and total scores (section score CLT raw the places which information, is produced score an adjusted uses external anchor scores, repeat-person scores, item analysis data, and score distribution data. CLT adjusts adjusts CLT data. distribution and score analysis data, item scores, repeat-person externaluses scores, anchor scores These raw person and item scores. administration test current the examining by section subject each correct responses; the adjusted score is the reported score that takes item/test difficulty into consideration for for consideration into difficulty item/test takes that score reported is the score adjusted the responses; correct of range score same in the forms. test are The scores adjusted across adjustment r the of are tests three the of number is the score The raw response. an incorrect for given points are score zero response; correct Overview of on each scores (CLT) score composite as the (QR). Test as well tests subject three the Reasoning Quantitative 45

CHAPTER TEN Quantitative than malestudents onVerbal Reasoning andGrammar/Writing, better on but male students performed (Mean=66.06; andSD=22.64) inForm slightly better female students performed 1618.Onboth forms, and SD=17.24), while female students scored higher (Mean=69.06 andSD=17.17) than male students males inForm 1517onthe average CLT scores (Females Mean=74.68 andSD=16.29; Males Mean=74.73 CLT Scores by Subgroup of Summary Table 10.1.DescriptiveStatisticsofTest Scores by over the three subtests inForm 1517,but more variations are observed in SD=18.64) than that onthe paper version average (Mean=60.30; SD=20.66). The scores are slightly different 16.72). InForm 1618,the average CLT score ishigher onthe online version with asmaller SD (Mean71.31; higher onthe paper version (Mean=75.83) than that onthe online version (71.92)with asimilarSD (16.49vs. Table 10.2shows the analysisresults by andtesting mode. form InForm 1517,the average CLT score is Form 1618for lower slightly average scores andlarger SD than QR 21.95(SD=5.77-7.31) inForm 1517,andare VR25.10,GW23.05,andQR19.74(SD= 7.07-7.92) in and 67.88(SD=19.92) inForm average 1618.The scores for the three subtests are VR25.58,GW25.99,and Test Form andTesting Mode CLT Scores by of Summary Form 1517 1618 The resultsThe by gender summarized inTable between females and asimilarperformance 10.3suggest wasdeliveredEach test form intwo testing modes, the online andthe paper-and-pencil test (PPT) version. Descriptive statistics inTable that 10.1suggest the average CLT scores are 73.52(SD=16.73) inForm 1517 Quantitative Quantitative Gramm Gramm Verbal Verbal Reasoning. CLT CLT Reasoning Reasoning Test ar/Writing ar/Writing Reasoning Reasoning 276 780 N Range 100 31 87 37 38 37 35 33 Test Form Min.

31 9 2 0 3 8 5 7 Form 1517. Max. 118 108 40 39 38 40 40 40 Form 1618. Mean 25.58 73.52 19.74 23.05 25.10 67.88 21.95 25.99 16.73 19.92 6.32 7.07 7.25 7.92 7.31 5.77 SD 46

CHAPTER TEN SD 6.28 5.85 7.15 6.26 5.55 7.50 7.45 6.68 6.78 8.27 7.55 7.42 16.72 16.49 18.64 20.66 60.30 22.34 19.99 17.98 71.92 24.91 25.43 21.58 75.83 26.55 26.81 22.48 71.31 26.35 24.43 20.53 Mean 40 38 38 40 40 39 40 39 40 38 38 39 107 118 116 108 Max. 8 6 0 2 9 7 5 7 3 5 5 31 32 12 10 21 Min. sting Mode sting 99 34 38 36 87 31 33 34 84 28 29 33 87 35 33 34 Range N 86 461 319 190 Reasoning Reasoning Reasoning Reasoning ar/Writing ar/Writing ar/Writing ar/Writing Test Reasoning Reasoning Reasoning Reasoning CLT CLT CLT CLT Verbal Verbal Verbal Verbal Gramm Gramm Gramm Gramm Quantitative Quantitative Quantitative Quantitative Paper Paper Mode Online Online

1517 1618 Form Table 10.2. Descriptive Statistics of Test Scores by Test Form and Te Form and by Test Scores of Test Statistics 10.2. Descriptive Table 47

CHAPTER TEN Table 10.3.DescriptiveStatisticsofTest Scores byTest Form Form 1517 1618 Gender Female Female Male Male Quantitative Quantitative Quantitative Quantitative Gramm Gramm Gramm Gramm Verbal Verbal Verbal Verbal CLT CLT CLT CLT Reasoning Reasoning Reasoning Reasoning Test ar/Writing ar/Writing ar/Writing ar/Writing Reasoning Reasoning Reasoning Reasoning 270 311 133 82 N Range and Gender 100 29 74 31 33 29 87 31 29 30 82 37 38 37 26 30 Min. 32 11 31 10 10 32 9 8 7 7 2 0 3 8 7 8 Max. 106 118 114 108 38 39 40 40 38 39 40 39 38 40 33 38 Mean 26.18 69.06 23.04 25.84 25.86 74.73 21.57 27.03 26.08 74.68 20.02 22.01 24.04 66.06 18.61 24.27 22.64 17.17 17.24 16.29 8.16 7.83 8.65 5.82 6.39 6.87 7.41 6.08 6.33 7.16 5.44 6.19 SD 48

CHAPTER TEN SD 6.29 7.85 6.06 5.53 7.17 8.07 6.65 6.41 5.99 6.04 7.36 6.66 16.04 18.72 17.35 18.38 20.83 18.97 16.74 77.45 28.77 26.64 22.04 70.85 23.96 24.90 21.99 75.69 26.48 26.88 22.34 56.54 Mean 37 38 37 40 38 39 40 39 39 40 40 39 108 114 118 106 Max. QR 16.74. students. 3 0 2 6 7 7 7 8 Test Form Test 39 12 11 32 11 10 31 10 Min.

34 38 35 69 28 27 33 82 29 29 32 87 30 33 32 98 Range N 100 100 100 107 107 107 107 109 109 109 109 444 444 444 444 100 mple size. mple Reasoning Reasoning Reasoning Reasoning ar/Writing ar/Writing ar/Writing ar/Writing Test Reasoning Reasoning Reasoning Reasoning CLT CLT CLT CLT Verbal Verbal Verbal Verbal Gramm Gramm Gramm Gramm Quantitative Quantitative Quantitative Quantitative Race Non- Non- White White White White

1618 1517 Table 10.6 presents the analysis results by test form, race, and testing mode. In Form 1517, the average average 1517, the In Form mode. and testing form, test race, by results analysis the 10.6 presents Table Descriptive statistics are computed based on adjusted scores by subgroup (e.g., gender and race) and testing and testing and race) gender (e.g., subgroup by scores on adjusted based computed are statistics Descriptive Table 10.4 summarizes the analysis results by race/ethnicity, White and Non-White students. In Form 1517, 1517, In Form students. and Non-White White race/ethnicity, by results analysis the summarizes 10.4 Table Form Table 10.4. Descriptive Statistics of Test Scores by Race/Ethnicity and Scores 10.4. Descriptive Statistics of Test Table (Mean=77.78 and SD=15.56) are higher for White students in the PPT version than the online version. Similar Similar version. online the than version PPT in the students White for higher are and SD=15.56) (Mean=77.78 on the students White for available not are 1618. Statistics in Form students Non-White for found are results sa small a very to due version PPT CLT scores (Mean=71.63 and SD=18.58) and the average scores of scores average the and and SD=18.58) (Mean=71.63 scores Non- for higher CLT are subtests three the scores and subtest CLT average the while version; PPT in the those than version online in the students White The average CLT score and the three subtests for the online version are higher (Mean=72.14 and SD=22.97) and SD=22.97) (Mean=72.14 higher are version online the for subtests three and the score CLT The average male the for SD=20.89) and (Mean=60.25 version PPT on the those than (Mean=73.17 and SD=16.05), as well as on the three subtests, for females; the average CLT scores and the and the scores CLT average the females; for subtests, three the as on as well and SD=16.05), (Mean=73.17 and Mean=75.30 SD=17.45; and (Mean=74.43 males for modes testing between similar are SD corresponding version. PPT females’ for size small sample a very to due available not are 1618, statistics In Form SD=16.91). mode for each test form. The analysis results summarized in Table 10.5 suggest that, in Form 1517, the the 1517, in Form 10.5 suggest that, form. test each in Table for mode summarized The analysis results version online on the that than and SD=16.44) (Mean=76.99 version PPT on the higher is score CLT average Summary of by Scores CLT Mode and Testing Subgroup Form 1618, the mean scores on the subtests were VR 20.83, GW 18.97, and 20.83, GW 18.97, VR were subtests on the scores mean 1618, the Form for example, the average CLT score is 75.69 (SD=16.04) for White students and 70.85 (SD=18.23) for Non- for and 70.85 (SD=18.23) students White for (SD=16.04) is 75.69 score CLT average the example, for 1618, in Form subtests forms. former the test in both students; White three the On latter the than higher scored in students Non-White For 26.64, and QR 20.04. VR 28.77, GW were Students White for scores mean the 49

CHAPTER TEN Table 10.5.DescriptiveStatisticsofTest Scores byTest Form,Gender, Form 1517 1618 G Fem Fe Fem Online Online Online Online Mode Paper Paper Paper Male M Male M ender male ale ale ale ale Quantitative Quantitative Quantitative Quantitative Quantitative Quantitative Quantitative Gramm Gramm Gramm Gramm Gramm Gramm Gramm Verbal Verbal Verbal Verbal Verbal Verbal Verbal CLT CLT CLT CLT CLT CLT CLT Reasoning Reasoning Reasoning Reasoning Reasoning Reasoning Reasoning Test ar/Writing ar/Writing ar/Writing ar/Writing ar/Writing ar/Writing ar/Writing Reasoning Reasoning Reasoning Reasoning Reasoning Reasoning Reasoning 188 176 123 123 123 123 188 188 188 176 176 176 94 94 94 94 79 68 68 68 68 65 65 65 65 79 79 79 N Range 87 31 29 28 79 31 27 30 74 29 28 27 69 31 33 29 74 36 38 34 99 34 31 35 87 26 30 29 andMode Min. 31 10 12 32 11 10 40 10 13 40 11 21 32 7 7 8 8 7 2 0 6 8 5 6 3 7 8 9 Max. 118 111 114 109 106 107 108 38 39 40 38 38 40 37 38 40 39 40 40 38 38 40 39 37 38 33 38 38 Mean 74.43 22.18 27.64 27.17 76.99 21.17 26.63 25.37 73.17 23.32 26.33 25.65 75.30 22.89 25.57 25.97 68.46 18.24 19.69 22.32 60.25 21.88 24.43 25.83 72.14 18.46 23.99 26.01 16.91 17.45 16.44 16.05 20.89 22.97 16.85 7.28 6.09 6.53 7.50 6.08 6.23 7.33 5.52 6.21 7.04 5.37 6.08 7.57 7.61 8.11 8.39 7.35 8.90 5.72 6.27 6.84 SD 50

CHAPTER TEN SD 8.35 6.07 6.09 7.73 7.26 6.92 5.40 5.58 7.22 6.83 6.42 8.00 6.48 6.18 7.69 5.99 5.53 7.15 6.13 5.45 7.18 18.36 19.25 16.30 18.58 18.24 16.22 15.56 78.40 58.00 21.20 54.44 16.59 29.15 27.08 22.17 19.95 16.85 20.29 17.56 26.43 21.99 77.78 27.16 27.66 22.96 71.63 24.08 25.11 22.44 69.78 23.80 24.61 21.37 74.51 26.09 Mean 33 38 38 39 37 35 37 35 38 40 39 40 39 38 40 39 39 37 38 38 40 108 102 106 111 114 111 118 Max. sting Mode sting 2 9 3 6 5 8 6 0 7 8 7 7 7 43 16 13 21 32 12 10 32 11 10 38 13 15 31 10 Min. 31 65 22 25 30 33 31 79 28 29 31 81 34 29 32 98 29 38 82 29 29 32 73 24 23 31 87 30 Range N 41 84 84 84 84 59 59 59 59 41 41 41 63 63 63 63 46 46 46 46 283 283 161 161 161 161 283 283 Reasoning Reasoning Reasoning Reasoning Reasoning Reasoning Reasoning ar/Writing ar/Writing ar/Writing ar/Writing ar/Writing ar/Writing ar/Writing Test Reasoning Reasoning Reasoning Reasoning Reasoning Reasoning Reasoning CLT CLT CLT CLT CLT CLT CLT Verbal Verbal Verbal Verbal Verbal Verbal Verbal Gramm Gramm Gramm Gramm Gramm Gramm Gramm Quantitative Quantitative Quantitative Quantitative Quantitative Quantitative Quantitative ite ite ite ite

hite hite hite hite Paper Paper Paper White White White Online Online Online Online Non-Wh Non-W Non-W Non-Wh Race/Mode

1618 1517 Form Table 10.6. Descriptive Statistics of Test Scores by Test Form, Race/Ethnicity, and Te and Form, Race/Ethnicity, by Test Scores of Test Statistics 10.6. Descriptive Table 51

CHAPTER TEN 1 Form are observed 1517.Similar patterns in subtests (Mean=79.01 andSD=15.27), followed by the familyincome from $50,001to $125,000in scores of categories, the familyincome from $125,001to $225,000orhigher hasthe highest average CLT score and School (Mean=66.29 andSD=20.18), asthe sameistrue with the three 1618, the average CLT score ishigher for Home School (Mean=83.77 andSD=17.47) than that for Private the CLT scores are observed for the three subtests. InForm of (Mean=64.27 andSD=17.54). Similarpatterns than Private School (Mean=73.72 andSD=16.74), but about 10score points higher than Charter School In Form 1517,the average CLT score isslightly higher for Home School (Mean=74.78 andSD=16.57) those statistics, particularly incomparisonbetween oracross of be taken inthe interpretations It isimportant to note that smallsample size may result instatistics with large samplingerror. Caution should reported inthree general categories: $50,000orlower, $50,001—$125,000,and$125,001—$225,000orhigher. Home School, Private School, andCharter School. students taking the exam. Reported school types are schools with areasonable number of only for the type of Quantitative Reasoning, smallsample size, statistics are andthe composite CLT reported scores. Because of for the three each multiple-choice test form based tests, Verbalincome of Reasoning, Grammar/Writing, and School Type andFamily Income CLT Scores by of Summary Descriptive statistics Amongthe are each presented test form. inTable 10.8by income category family of CLTTable scores andscores onthe three subtests by 10.7presents the descriptive school type. statistics of In this section, descriptive statistics are computed based onthe adjusted scores by school type andfamily  28% CharterSchool, 11%Home School. Distributions from each school typevary from testtotest,but examinees onForm 1618, for example, were distributed asfollows: 61%Private School, Form 1618. 1 For the samereason, self-reported familyincomes are

subtests. subgroups. 52

CHAPTER TEN SD 7.71 7.54 7.13 6.07 5.68 7.77 7.75 5.86 6.40 6.03 5.50 7.09 6.33 5.84 7.27 20.18 17.47 16.74 17.54 16.57 29.70 22.27 66.29 24.45 22.12 19.72 22.15 18.82 83.77 31.80 23.30 26.95 27.23 20.59 73.72 25.45 25.91 22.36 64.27 74.78 Mean 38 38 38 38 37 36 39 40 40 40 38 39 38 38 40 108 107 108 118 109 Max. 8 6 0 2 9 5 3 8 9 7 7 16 10 49 16 21 12 13 31 39 Min. Test Form Test 22 28 32 38 35 27 34 58 24 33 33 87 35 27 25 30 87 31 70 100 Range N 30 30 74 74 30 30 74 74 86 86 86 86 164 164 164 164 641 641 641 641

ar/Writing ar/Writing ar/Writing ar/Writing ar/Writing Test Reasoning Reasoning Reasoning Reasoning Reasoning CLT CLT CLT CLT CLT Reasoning Reasoning Reasoning Reasoning Reasoning Quantitative Quantitative Quantitative Quantitative Quantitative Verbal Verbal Verbal Verbal Verbal Gramm Gramm Gramm Gramm Gramm ter ter ate ate ome ome ome ome Type H H School School School School School Priv Priv Char

1618 1517 Form Table 10.7. Descriptive Statistics of Test Scores by School Type and Type by School Scores of Test Statistics Descriptive 10.7. Table 53

CHAPTER TEN Table 10.8.DescriptiveStatisticsofTest Scores byFamilyIncomeand Form 1517 1618 1517 1618 1517 1618 Quantitative Quantitative Quantitative Quantitative Quantitative Quantitative Gramm Gramm Gramm Gramm Gramm Gramm Verbal Verbal Verbal Verbal Verbal Verbal CLT CLT CLT CLT CLT CLT Reasoning Reasoning Reasoning Reasoning Reasoning Reasoning Test ar/Writing ar/Writing ar/Writing ar/Writing ar/Writing ar/Writing Family Incomeof$125,001-$225,000 Reasoning Reasoning Reasoning Reasoning Reasoning Reasoning Family Incomeof$50,001 Family Incomeof$50,00 109 109 109 109 181 181 181 181 181 39 39 39 39 58 58 58 58 22 22 22 22 53 53 53 53 N Range 23 32 30 85 29 26 30 76 24 27 20 66 31 21 27 66 28 23 33 74 31 27 28 76 0 orLower - $125,000 Min. 18 42 15 13 45 33 12 12 38 21 10 10 32 13 10 12 7 6 8 7 9 8 7 7 Test Form orHigher Max. 106 108 108 111 107 114 30 38 38 36 36 40 37 36 38 39 36 40 38 35 40 38 39 40 Mean 16.92 21.41 23.03 61.36 20.31 25.16 24.55 70.02 21.36 26.73 29.27 77.36 24.55 27.56 26.90 79.01 22.74 25.66 28.17 76.57 22.64 26.72 26.85 76.22 16.85 17.21 17.42 15.27 19.53 15.74 5.61 6.29 7.20 6.99 5.67 7.37 6.59 6.17 6.40 7.55 4.89 6.11 7.72 6.57 7.32 7.15 5.70 5.51 SD 54

CHAPTER TEN 11. d scoring. f test the (11.1) (11.2) i e variance e n the test n the of item ERROR OF MEASUREMENT OF ERROR RELIABILITY & STANDARD & STANDARD RELIABILITY ula below: ) = variance ) = scor test total ) = i t ) = Standard deviation o deviation Standard ) = t

variances. ula below:

SD (Y SD var. (Y var. of error Standard of indicator another provides (SEM) measurement accuracy of the which scores, test k = number of number k = o items (Y var. Cronbach alpha (1951) is one of the most widely used estimates to test reliability, which can be computed computed can be which of alpha (1951) is one Cronbach reliability, test to estimates used widely most the Reliability coefficients typically range from zero to one, with the values near one indicating high consistency, high consistency, indicating near one values the with to one, zero from range typically coefficients Reliability Reliability refers to the consistency in test scores. Reliability coefficients quantify the level level the quantify coefficients Reliability scores. in test consistency the to refers of consistencyReliability of where summarizes the amount of amount the summarizes of scores in test or inconsistency errors using the computed can be SEM a test. form where using the formusing the score scale. Reliability coefficients are usually estimated on a single test administration by calculating the inter- the by calculating on a single administration test estimated usually are coefficients Reliability scale. score co item and values near zero indicating low or no consistency. In classical test theory, reliability is defined as the ratio ratio the as is defined reliability theory, In classical test or no consistency. low indicating zero near and values of along the same is the variance error assuming the variance, score observed the to variance score true the through empirical studies so that sound judgments can be made. The reliability of The reliability made. can be judgments sound so that studies empirical through of is a function scores test for procedure the well as as motivation, student and deviation, standard difficulty, item length, test content, test an administration, test development, test test scores. Tests with high reliability coefficients provide stable test scores between test forms across occasions. formstest occasions. between scores test across stable provide coefficients with high reliability Tests scores. test of quality the is a necessary for condition Reliability of reliability to establish is important It a test. scores test Overview 55

CHAPTER ELEVEN between the online andthe PPT modes within accuracy of test scores interms scores for the three subtests. results The alsoindicate the comparability of 2. 0.85-0.87 for Quantitative Reasoning with the SEM of 2.53-2.68; Verbal 2.60-2.84; 0.80-0.89 for Grammar/Writing with the SEM of Reasoning with the SEM of reliability coefficients isin0.80s for the three subtests, 0.80-0.89 for and testing modes. magnitude of The 4.64-4.79 across test forms 0.92-0.95 with the SEM of are reasonably highfor the CLT scores inthe rangeof Table test scores by andtesting mode. reliability test form The coefficients 11.2provides descriptive statistics of highly reliable subte 2.63-2.76 for Form provide 1618.Both forms highlyreliable CLTwith the SEMs of scores andmoderately 2.56-2.84 for Form 1517;while the reliability coefficients are 0.85-0.89 subtests are 0.80-0.86 with the SEMsof 4.67for Form reliability 1618.The coefficientsForm for the three 1517;the reliability is0.95with the SEM of 2017-2018 school year, 4.76for CLT Form for 1517andForm reliability 1618.The is0.92with the SEM of from r minimum andmaximum scores, mean, andstandard deviation (SD) derivedsection with descriptive statistics of Reasoning, Grammar/Writing, andQuantitative Reasoning, andthe composite CLT scores are reported inthis Measurement by Test Formof andTesting Mode Reliability andStandard Error Table 11.1.ReliabilityandStandard Error ofMeasurement ofUnadjustedTest Scores by h eut fanalyses provide evidence to support highlyreliable CLT results scoresThe of andmoderately highlyreliable wasdeliveredEach test form intwo testing modes, the online andthe paper-and-pencil test (PPT) versions. Tableof the 11.1provides the reliability April operated coefficients and SEMs in for the two test forms measurement (SEM) for the three multiple-choice tests, Verbal estimatedThe reliability andstandard error of 1618 1517 Test aw scores. Quantitative Quantitative Gramm Gramm Verbal Verbal Score CLT CLT st scores. Reasoning Reasoning ar/Writing ar/Writing Reasoning Reasoning 276 780 N Min. each form. 26 0 0 3 6 4 6 6 Max. 106 113 37 38 40 40 39 39 74-2.80.

Mean 17.74 23.05 25.10 65.88 20.95 24.99 22.58 68.52 19.92 16.73 7.07 7.25 7.92 7.30 5.76 6.33 SD Test Form R eliability 0.85 0.87 0.89 0.95 0.86 0.80 0.80 0.92 SEM 2.76 2.63 2.64 4.67 2.77 2.56 2.84 4.76 56

CHAPTER ELEVEN 2.61 2.77 4.76 2.71 2.63 2.74 4.76 2.84 2.56 2.77 4.79 2.85 2.58 2.79 4.72 2.82 2.52 2.74 4.67 2.64 2.63 2.76 4.64 2.60 SEM 0.79 0.87 0.95 0.89 0.87 0.85 0.94 0.88 0.85 0.83 0.95 0.89 0.87 0.86 0.92 0.80 0.80 0.86 0.92 0.80 0.81 0.85 0.92 0.80 eliability R SD 5.55 7.51 7.92 7.25 7.07 7.45 6.68 6.78 8.27 7.55 7.42 6.33 5.76 7.30 6.30 5.85 7.14 6.27 19.92 18.64 20.66 16.73 16.72 16.49 sting Mode 25.80 21.49 65.88 25.10 23.05 17.74 69.31 26.35 24.43 18.53 58.30 22.34 19.99 15.98 68.52 22.58 24.99 20.95 66.92 21.92 24.43 20.57 70.83 23.55 Mean 38 40 40 38 37 38 38 37 40 38 36 39 39 40 39 39 38 38 106 106 105 113 113 111 Max. 9 6 6 3 0 0 3 5 3 6 6 0 0 6 6 4 6 6 4 9 19 26 26 27 Min. N 86 319 276 190 780 461 Reasoning Reasoning Reasoning Reasoning Reasoning Reasoning ar/Writing ar/Writing ar/Writing ar/Writing ar/Writing ar/Writing Reasoning Reasoning Reasoning Reasoning Reasoning Reasoning CLT CLT CLT CLT CLT CLT Score Verbal Verbal Verbal Verbal Verbal Verbal Gramm Gramm Gramm Gramm Gramm Gramm Quantitative Quantitative Quantitative Quantitative Quantitative Quantitative

Total Total

Paper Paper Mode Online Online

1517

Test 1618 Table 11.2. Descriptive Statistics of Unadjusted Test Scores by Test Form and Te Form Test by Scores Test of Unadjusted Descriptive Statistics 11.2. Table 57

CHAPTER ELEVEN and moderately highlyreliable subtest scores for both male andfemale students by show analyses highlyreliable CLT scores 2.50-2.84inForm results 1618.The of 0.86 for males withthe SEMs of of 0.80- 2.50-2.82 inForm 1517,while the reliability coefficients are in the range 0.85 for females with the SEMs of of 0.79- of the three subtests are in the range 4.64-4.70 by gender inForm reliability 1618.The coefficients SEMs of of 4.72-4.75 by gender inof the Form 1517,and0.93-0.96 with the CLT scores are 0.92with the SEMs coefficients fromself-reported information examinees. Missingidentifications are excluded from analysis.The reliability variance for some items becomes zero, those items are removed from the that the reliability coefficient may not be stable from case tocase due tosmallsample size (e.g., N<300). When the Reasoning, Grammar/Writing, andQuantitative Reasoning, andthe composite CLT scores. important It is to note Measurement by Subgroup of Reliability andStandard Error 1517 Table 11.3.DescriptiveStatisticsofTest Scores byTest Forma Table 11.3provides the analysisresults by Gender identifications are based on genderthe andtest form. available estimatedThe reliability andSEM are computed by subgroup for the three multiple-choice based tests, Verbal Test 1618 Gender Female Female Male Male Quantitative Quantitative Quantitative Quantitative Gramm Gramm Gramm Gramm Verbal Verbal Verbal Verbal CLT CLT CLT CLT Reasoning Reasoning Reasoning Reasoning Test ar/Writing ar/Writing ar/Writing ar/Writing Reasoning Reasoning Reasoning Reasoning 133 270 311 82 N Min. 27 30 26 0 0 3 6 5 8 9 7 6 8 6 9 7

Max. nd Gender 109 106 104 113 37 38 40 31 38 38 38 39 39 37 38 38 analysis. Mean 69.68 18.02 22.01 24.04 64.06 16.61 24.27 26.18 67.06 22.03 24.83 22.87 69.73 20.57 26.03 23.08 16.29 22.64 17.17 17.24 test form. 8.16 7.83 8.65 5.82 6.39 6.87 7.40 6.08 6.35 7.16 5.44 6.20 SD R eliability 0.92 0.89 0.89 0.91 0.96 0.77 0.84 0.86 0.93 0.86 0.82 0.80 0.92 0.85 0.79 0.79 SEM 2.73 2.64 2.65 4.70 2.78 2.59 2.59 4.64 2.73 2.56 2.84 4.75 2.78 2.50 2.82 4.72 58

CHAPTER ELEVEN 2.77 4.81 2.74 2.72 2.79 4.49 2.52 2.50 2.71 4.79 2.87 2.59 2.75 4.73 2.82 2.51 SEM 0.83 0.86 0.93 0.82 0.83 0.88 0.91 0.78 0.79 0.85 0.93 0.89 0.83 0.81 0.93 0.82 eliability R SD 6.04 7.36 6.73 6.29 7.81 6.06 5.53 7.17 8.07 6.65 6.41 5.99 18.38 16.04 18.72 17.35 26.64 20.04 65.85 20.99 23.90 20.96 70.69 23.48 25.88 21.34 54.54 20.83 18.97 14.74 75.45 28.77 Mean Test Form Test 38 37 39 38 38 38 39 38 37 38 35 40 109 113 104 106 Max. 4 8 9 6 7 6 6 6 3 0 0 11 27 26 37 12 Min. N 107 107 109 109 109 109 444 444 444 444 100 100 100 100 107 107 students. Reasoning Reasoning Reasoning Reasoning ar/Writing ar/Writing ar/Writing ar/Writing Test Reasoning Reasoning Reasoning Reasoning CLT CLT CLT CLT Verbal Verbal Verbal Verbal Gramm Gramm Gramm Gramm Quantitative Quantitative Quantitative Quantitative each form.each Race White White Non-White Non-White

The results of analyses provide evidence to support highly reliable CLT scores and moderately highly reliable highly reliable and moderately ofThe scores results CLT highly reliable support to evidence provide analyses Table 11.4 provides reliability coefficients and SEMs by racial/ethnicity group and test form.test group and by racial/ethnicity Racial/ SEMs and coefficients reliability provides 11.4 Table Test 1618 1517 Table 11.4. Descriptive Statistics of Test Scores by Race/Ethnicity and Scores 11.4. Descriptive Statistics of Test Table for internalfor and accuracy of consistency testing across groups racial/ethnicity two the between scores test within modes 0.89 with SEMs in 2.72-2.79 for Non-White for SEMs in 2.72-2.79 0.89 with of comparability the also indicate The results students. and Non-White White both for scores subtest scores test Non-White students. For the three subtests, the reliability coefficients for Form the with 0.78-0.85 are 1517 for coefficients reliability the subtests, three the For students. Non-White ofSEMs The students. Non-White for in 2.59-2.87 SEMs with and 0.82-0.88 students White for 2.51-2.82 Form SEMs the with 0.82-0.86 are 1618 for coefficients reliability and 0.81- students for White of 2.50-2.77 and Non-White. In Form 1517, the reliability is 0.91 of is 0.91 reliability 1517, the Form In of SEM the with scores and Non-White. CLT students White for 4.73 of reliability and the of SEM the 0.93 with is 0.93 reliability the 1618, In Form students. Non-White 4.79 for of of SEM the with scores CLT of SEM is 0.93 with the reliability and the students White 4.49 for 4.81 for ethnicity identifications are based on the available self-reported information from examinees. Missing Missing examinees. information self-reported from available the on based are identifications ethnicity ofnumber small the to Due analysis. from excluded are identifications as such groups, racial in some students White categories, general two on based are analyses Indian, the and American African American, Hispanic, 59

CHAPTER ELEVEN items are excluded from estimates due to the zero CLT scores is0.95for White students (N=23) onthe paper-and-pencilexample, version. Four the reliability of reliability andSEM based onsmallsample size. For the results inthe comparisonof of in the interpretation test scores between White andNon-White students across testing mode by Caution shouldbe taken test form. (SEMs=2.72-2.79) for Non-White 0.89-0.90 (SEMs=2.45-2.66) for White Students and0.84-0.87 for the three subtests are inthe range of (SEM=4.64) for White students and0.94(SEM=4.72) for Non-White students. reliability The coefficients (SEMs=2.71-2.80) for Non-White students. reliability The coefficient for the paper-and-pencil is0.95 version 0.78-0.86 (SEMs=2.50-2.72) for White Students and0.79-0.90 for the three subtests are inthe range of (SEM=4.49) for White students and0.93(SEM=4.82) for Non-White students. reliability The coefficients (SEMs=2.57-2.91) for Non-White of the three subtests are 0.79-0.86 (SEMs=2.50-2.81) forWhite students and0.80-0.87 reliability coefficients scores is0.91(SEM=4.69) for White students and0.93(SEM=4.83) for Non-White students. Similarly, the CLT 2.61-2.84 for Non-White students. Inthe paper-and-pencil version, the reliability of the SEMs of Writing, andQuantitative Reasoning for White students; while the reliability coefficients are 0.83-0.88 with 2.51-2.83, respectively,three subtests are 0.78-0.85 with the SEM of for Verbal Reasoning, Grammar/ reliability 4.77.The coefficients for the 4.73and0.93for Non-White students with the SEM of SEM of of the CLT scoresForm is0.92 forWhite students with the 1517online version, the reliability coefficient female students between different testing modes within each 2.53-2.84 for 0.79-0.87 with the SEMsof 2.45-2.82 across andtesting modes for genders Form 1517;while the reliability isinthe range of the SEMs of 4.59-4.76 for males across testing modes. 0.78-0.86 with For the three subtests, the reliability isinthe range of 4.59for females onlinetesting and0.95-0.96 with the of SEMs 1618, the reliability is0.92with the SEMof 4.75-4.78. InForm 4.74 for females and0.92-0.93 for males by testing mode with the corresponding SEM of of 4.68- of the scoresCLT are 0.91-0.92 by testing mode with the corresponding SEM reliability coefficients group isbroken down for analysisby previously, test scores across occasionswhen the total smallsample size may result inunstable reliability of Reasoning, Grammar/Writing, andQuantitative Reasoning, andthe composite CLT scores. Asindicated by Subgroup andTesting Mode Measurement Reliability andStandard Error of h eut f nlsspoieeiec ospottesmlrlvlo itra ossec n cuayo consistency andaccuracyof internal analyses provide evidence to support the similarlevel of resultsThe of Similar results are found inForm CLT 1618.For scores is0.92 the online version, the reliability of Table 11.6provides the reliability andSEMs by racial/ethnicity, test form, andtesting mode. Inthe test scores for both male and reliability andaccuracyof analyses the support similarlevel of resultsThe of analysisresultsThe are summarized inTable 11.5by gender, test form, andtesting mode. InForm 1517,the reliabilityThe andSEM are computed by subgroup for the three multiple-choice based tests, Verbal students. students. subgroup. Form 1618. variance. test form.

60

CHAPTER ELEVEN n/a n/a n/a n/a 2.60 2.79 4.59 2.58 2.60 2.72 4.76 2.71 2.68 2.75 4.65 2.61 4.73 2.82 2.52 2.79 4.68 2.81 2.45 2.75 4.75 2.84 2.53 2.75 4.78 2.83 2.58 2.74 SEM n/a n/a n/a n/a 0.95 0.89 0.88 0.87 0.96 0.875 0.895 0.924 0.855 0.828 0.762 0.916 0.87 0.92 0.81 0.82 0.86 0.79 0.78 0.84 0.92 0.80 0.80 0.86 0.93 0.79 0.83 0.91 eliability R 7.35 8.39 8.11 7.61 7.57 6.84 6.27 5.72 7.51 5.77 8.39 8.90 20.89 16.85 21.66 22.97 SD 7.48 6.56 6.08 7.28 6.10 5.37 7.03 6.21 5.52 7.33 6.26 6.08 16.91 16.05 16.44 17.45 sting Mode sting 24.43 19.88 58.25 22.32 19.69 16.24 66.46 26.01 23.99 16.46 83.00 30.67 31.67 20.67 70.14 25.83 21.88 70.30 22.66 25.32 22.32 22.37 25.63 20.16 71.99 24.17 26.64 21.18 69.43 22.98 24.57 68.17 Mean 37 37 40 38 36 38 38 31 96 35 35 26 38 105 104 106 38 38 37 36 38 37 37 37 38 37 39 39 104 106 113 109 Max. 6 3 6 6 0 0 9 8 5 3 30 58 22 25 11 19 7 9 7 7 6 9 9 6 8 6 35 10 10 27 26 35 Min. 3 3 3 3 65 65 68 68 68 68 79 79 79 79 65 65 N 94 94 94 94 176 188 188 188 123 123 123 123 176 176 176 188

Reasoning Reasoning Reasoning Reasoning ar/Writing ar/Writing ar/Writing ar/Writing Reasoning Reasoning Reasoning Reasoning CLT CLT CLT CLT ar/Writing ar/Writing ar/Writing ar/Writing Test Reasoning Reasoning Reasoning Reasoning CLT CLT CLT CLT Reasoning Reasoning Reasoning Reasoning Quantitative Quantitative Quantitative Quantitative Verbal Verbal Verbal Verbal Gramm Gramm Gramm Gramm Verbal Verbal Verbal Verbal Gramm Gramm Gramm Gramm Quantitative Quantitative Quantitative Quantitative ale/Online ale/Online ale/Online ale/Online male/Paper male/Paper ender Mode ender Male/Paper Male/Paper

M M Fe G Fe Fem Fem

Test 1618 1517 Table 11.5. Descriptive Statistics of Test Scores by Test Form, Gender, and Te and Form, Gender, by Test Scores of Test Statistics 11.5. Descriptive Table 61

CHAPTER ELEVEN Table 11.6.DescriptiveStatisticsofTest Scores byTest Form,Race/Ethnicity, andTes 1517 1618 Test Non-W Non-W Wh Wh W W Non-Wh Non-Wh Race/Mode hite/Paper hite/Paper ite/Online Online ite/Online Online hite/Paper hite/Paper ite/ ite/ Quantitative Quantitative Quantitative Quantitative Quantitative Quantitative Quantitative Quantitative Gramm Gramm Gramm Gramm Gramm Gramm Gramm Gramm Verbal Verbal Verbal Verbal Verbal Verbal Verbal Verbal CLT CLT CLT CLT CLT CLT CLT CLT Reasoning Reasoning Reasoning Reasoning Reasoning Reasoning Reasoning Reasoning Test ar/Writing ar/Writing ar/Writing ar/Writing ar/Writing ar/Writing ar/Writing ar/Writing Reasoning Reasoning Reasoning Reasoning Reasoning Reasoning Reasoning Reasoning 283 283 283 283 161 161 161 161 46 46 46 46 63 63 63 63 23 23 23 23 84 84 84 84 41 41 41 41 59 59 59 59 N Min. 26 14 10 33 27 27 11 12 37 13 16 41 19 7 6 7 6 6 9 8 6 9 9 4 7 0 0 6 6 3 6 3 Max. 105 106 104 100 113 106 109 106 36 36 40 37 38 38 31 38 35 35 35 37 38 39 37 37 37 34 38 38 39 37 38 38 Mean 20.99 25.43 23.09 69.51 20.37 23.61 20.80 64.78 21.40 24.11 21.13 66.63 21.96 26.66 24.16 72.78 19.57 25.04 27.35 71.96 20.17 27.08 29.15 76.40 14.59 17.56 20.29 52.44 14.85 19.95 21.20 56.00 ting Mode 16.22 18.24 18.58 15.56 7.15 5.53 5.99 7.69 6.18 6.48 7.93 6.42 6.95 7.18 5.44 6.15 SD 20.76 16.30 19.25 18.36 8.01 7.42 7.74 7.22 5.58 5.40 6.92 7.26 7.73 6.09 6.07 8.35 R eliability 0.85 0.79 0.78 0.92 0.87 0.83 0.80 0.93 0.88 0.83 0.83 0.93 0.86 0.79 0.79 0.91 0.89 0.89 0.90 0.95 0.86 0.80 0.78 0.92 0.84 0.86 0.87 0.94 0.79 0.80 0.90 0.93 SEM 2.66 2.46 2.45 4.64 2.72 2.50 2.52 4.49 2.77 2.72 2.79 4.72 2.80 2.74 2.71 4.82 2.73 2.50 2.81 4.69 2.79 2.51 2.83 4.73 2.79 2.57 2.91 4.83 2.71 2.61 2.84 4.77 62

CHAPTER ELEVEN 2.72 4.64 2.68 2.69 2.79 4.28 2.10 2.34 2.69 4.51 2.67 2.61 2.76 4.69 2.76 2.52 2.84 4.74 2.84 2.54 SEM 0.92 0.93 0.88 0.95 0.88 0.88 0.85 0.79 0.79 0.84 0.92 0.80 0.81 0.86 0.88 0.79 0.81 0.94 0.88 0.83 eliability R ool types. SD 7.77 7.71 7.54 7.13 6.03 5.50 7.09 6.35 5.84 7.26 7.75 5.86 6.40 6.07 5.68 16.57 17.54 20.18 16.74 17.47 Test Form Test 69.78 62.27 20.27 64.29 24.45 22.12 17.72 23.95 26.23 19.59 68.72 22.45 24.91 21.36 23.30 22.15 16.82 81.77 31.80 29.70 Mean 36 38 38 35 36 37 37 39 39 40 38 36 37 40 38 106 104 113 106 105 Max. 8 6 6 0 0 9 7 6 6 6 3 9 3 34 19 12 26 47 16 16 scores. scores. Min. 70 87 28 32 38 35 27 25 30 87 33 33 34 35 27 34 58 24 22 100 Range N 86 74 30 86 86 86 74 74 74 30 30 30 164 164 164 164 641 641 641 641 Reasoning Reasoning Reasoning Reasoning Reasoning ar/Writing ar/Writing ar/Writing ar/Writing ar/Writing Test Reasoning Reasoning Reasoning Reasoning Reasoning CLT CLT CLT CLT CLT pectively. Verbal Verbal Verbal Verbal Verbal Gramm Gramm Gramm Gramm Gramm Quantitative Quantitative Quantitative Quantitative Quantitative ter ter ate ate ome ome ome ome

Type

H H School School School School School Priv Priv Char

1618 1517

The results of analysis by self-reported family income are summarized in Table 11.8 for each test form. test each 11.8 for of The The results in Table summarized are family income self-reported analysis by The results in Table 11.7 show that the reliability coefficients coefficients reliability the that 11.7 show (SEM=4.69-4.74) are 0.92 CLT scores in Table The results of The reliability and SEM, as well as descriptive statistics, are computed based on the raw scores by school type type school by scores raw on the based computed are statistics, as descriptive as well SEM, and The reliability Form Table 11.7. Reliability and Standard Error of Measurement by School Type and by School Type of Measurement Error 11.7. Reliability and Standard Table 0.92-0.95 (SEM=4.37-4.77) in Form 1618. The reliability coefficients for the three subtests are in the range range the in are subtests three the for coefficients 1618. The reliability in Form (SEM=4.37-4.77) 0.92-0.95 of of similar range 1517 and in the in Form (SEM=2.49-2.83) 0.74-0.87 the across (SEM=2.39-2.81) 0.75-0.88 res categories, family income (SEM=2.52-2.84) in Form 1517 and 0.79-0.88 (SEM=2.10-2.79) in Form 1618 across sch 1618 across in Form (SEM=2.10-2.79) 0.79-0.88 1517 and in Form (SEM=2.52-2.84) 1517 and ofreliability in Form categories income family across (SEM=4.58-4.87) 0.91-0.92 are scores CLT for home and private schools in Form 1517 and 0.93-0.96 (SEM=4.28-4.64) for charter, home, and private and private home, charter, for (SEM=4.28-4.64) 1517 and 0.93-0.96 in Form schools and private home for coefficients 1618. The reliability in Form schools range similar the in are subtests three the of 0.79-0.86 of and family income of each test form for the three multiple-choice based tests, Verbal Reasoning, Grammar/ Reasoning, of income and family Verbal tests, based form test multiple-choice each three the for CLT composite and the Reasoning, and Quantitative Writing, School Type and Family Income Income Family and Type School Reliability and Standard Error of Error Standard and Reliability by Measurement 63

CHAPTER ELEVEN Table 11.8.ReliabilityandStandard Error ofMeasurement byFamilyIncomeand Form 1517 1618 1517 1618 1517 1618 Quantitative Quantitative Quantitative Quantitative Quantitative Quantitative Gramm Gramm Gramm Gramm Gramm Gramm Verbal Verbal Verbal Verbal Verbal Verbal CLT CLT CLT CLT CLT CLT Reasoning Reasoning Reasoning Reasoning Reasoning Reasoning Test ar/Writing ar/Writing ar/Writing ar/Writing ar/Writing ar/Writing Reasoning Reasoning Reasoning Reasoning Reasoning Reasoning Family Incomeof$125,001-$225,000 Family Incomeof$50,001 181 181 181 181 109 109 109 109 Family Incomeof$50,00 58 58 58 39 39 39 58 53 53 53 39 53 22 22 22 22 N Range 31 76 32 30 85 29 26 23 33 74 31 27 30 76 23 28 66 21 27 31 20 66 24 27 Min. 27 19 33 31 11 12 40 10 14 18 40 11 7 6 8 6 9 9 5 7 6 8 7 9 Max. 103 104 109 105 106 106 38 38 38 35 35 39 28 35 40 37 38 36 37 38 35 38 36 35 0 orLower - $125,000 Mean 21.57 65.02 23.87 71.22 14.92 21.41 23.03 59.36 19.31 24.14 25.66 28.17 74.57 21.62 25.72 20.74 74.01 23.90 23.55 26.56 29.27 75.36 26.73 19.36 orHigher 17.21 16.85 15.74 19.53 15.27 17.42 7.41 5.55 5.61 6.29 7.20 6.99 5.65 6.57 7.32 7.12 5.70 7.72 6.11 7.55 4.89 6.40 6.17 6.59 SD Test Form R eliability 0.86 0.92 0.74 0.91 0.75 0.82 0.86 0.92 0.84 0.79 0.85 0.88 0.95 0.85 0.81 0.88 0.79 0.91 0.87 0.74 0.93 0.85 0.84 0.83 SEM 2.77 4.87 2.83 4.72 2.81 2.67 2.69 4.77 2.79 2.59 2.54 4.37 2.76 2.48 2.67 2.55 2.80 4.58 2.72 2.49 4.61 2.39 2.56 2.72 64

CHAPTER ELEVEN

2

ntended. fensible. fensible. ed., pp. 13 - 103). New York: American Council on Education and Macmillan. on Education Council American York: 103). New 13 - pp. ed., rd (3

and is related to fairness. While reliability addresses the the addresses fairness. to reliability While and is related cess data. cess 1 , 12.VALIDITY Educational measurement Educational Washington, DC: American Educational Research Association. Research DC: American Educational Washington, , validity evidence is collected from the following aspects: content, response process, process, response content, aspects: following the from is collected evidence , validity Standards validity.

Standards for Educational and Psychological Testing Psychological and Educational for Standards American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). in Education. Council on Measurement & National Association, Psychological American Association, Research American Educational testing. educational and psychological for Standards  In R.L. Linn (Ed.), (1989). Validity. S. Messick,

Response process related evidence of validity can be collected in multiple ways. For example, students taking taking students example, of evidence related process For Response ways. in multiple collected can be validity According to the the to According Validity is another critical aspect that needs to be addressed in test development and evaluation according according and evaluation development in test addressed be to needs that aspect critical is another Validity The ultimate use of test scores is to draw inferences about students’ ability, competence, or behaviors in in or behaviors competence, of use The ultimate ability, students’ about inferences draw is to scores test

1 2 with what the item was intended to be. Further, students’ problem-solving strategies could be investigated by by investigated could be strategies problem-solving students’ Further, be. to intended was item the what with and log files, and time response as item such data analyzing process behaviors, responding students’ observing pro and response responses between relationship the the test can be interviewed about how they respond to the items. Some think-aloud procedure can help item item can help procedure think-aloud Some items. the to respond they how about interviewed can be test the test-takers’ is consistent thinking whether understand better test-takers’ thinking and evaluate developers if purpose the for for of dimensions more relevant reflect and scoring administration test Further, test. the content-related such of provide report Chapters evidence technical 2, 3, 4, and 9 in this content-related validity. of evidence internal structure, relations with other variables, and consequences of testing. Validity evidence related to test test to related evidence of and consequences variables, other with relations Validity internal structure, testing. of alignment specifications, on test based collected can be content curriculum, with content test and instruction measures what it intends to measure. Validity refers to the degree to which evidence collected in the test scores and in scores test in the collected evidence degree the to refers which to Validity measure. to intends it what measures of process the as i scores on test based inferences the supports administration and test development test to the the to consistencytest a whether addresses validity intime, and forms, administrations, different test from obtained scores situations beyond that observed in the testing scenario. The high reliability of The high reliability scenario. testing in the observed that beyond situations in consistency implies scores test and de valid are scores test from drawn inferences the that assure cannot but scores test Overview 65

CHAPTER TWELVE 4 3 Table 12.1TheNumberofItemsMeasuringE domain were similaracross the Tables 12.2and12.3respectively the scores for for Form each 1517andForm distributions 1618.The of remained the same across descriptive the two The forms. statistics for the domainscores are summarized in items assessing each domain assessing each domainacross subjects ranged from number 10to27.The of items items measuring each domainwithin each CLT number subject test. The of summarizes the number of EVALUATION OFTHEINTERNALSTRUCTURECLT USINGE (CFA) are conducted usingthe doma the two Both exploratory forms. factor analysis(EFA) factor analysis andconfirmatory three subjects ineach of of these structure gender and ethnicity on the internal and Form 1618.It further investigates the influences of Verbal Reasoning, Grammar/Writing, andQuantitative Reasoning intwo test administrations, Form 1517 the CLT. the three subjects: evidence structure related of to the internal It evaluates structure of the internal different sources in the test development andadministration process.This chapter focuses oncollecting can be examined to evaluate the intended andunintended consequences of multimethod the study willserve this purpose. Further,CLT oninstruction and school dropout rates the influence of test scores andother variables such asSAT/ACT scores, highschool andcollege GPA, Amultitrait- andgraduation. v evidence of construct-related the student population canprovide evidence of the factorstructure across subgroups of of atest andthe invariance the factorstructure of relevant populations shouldbe addressed. Aninvestigation of to measure. opportunities have been provided to demonstrate to test-takers their abilities onthe constructs atest intends validity evidence,functioning are related Test means that to test fairness. comparable aspart of fairness, the test-taker populations anddifferential item In general, construct equivalence across the subgroups of reliability, anddifferential measurement, item functioning. andtest information) standard of error of (in terms dimensionality, construct equivalence, structure of canbe evaluatedinternal measurement interms precision Structure Evidence onInternal Based h vlaino teitra tutr fthe CLT isconducted usingthe domainscores. Table structure of the internal 12.1 evaluationThe of validity evidence isanongoingprocess. technical This report provides evidence from collection of The validity, the relationship amongthe evidence aninvestigation of To may be conducted of collect another source of atest canprovide important validity evidence. The structure of the internal In addition, the investigation of  AmericanEducational Research Association, etal.,2014. Washington, DC:National Assessment Governing Board andNational Centerfor Education Statistics. Board (NAGB) andthe National Centerfor theNationalAssessment Governing EducationStatistics(NCES), Vol. II setting for assessmentsof large-scale Messick, S. (1995). Standards-based score Establishing interpretation: valid grounds for valid inferences. In Quantitative 3 Gramm Inevaluating such test fairness, questions aswhether the test measures the same construct inall Verbal alidity. Subject Reasoning ar/Writing 4

Reasoning two forms. in scores. ach Domain Mathematical Com Geometry Grammar Domain Analysis Algebra Writing prehension Reasoning testing . rceig fthejointconference onstandard Proceedings of Numbe FA ANDCFA 16 14 10 20 20 27 13 fItems r of (pp. 291-305). 66

CHAPTER TWELVE evidence. 4.04 2.45 2.73 2.97 2.37 4.54 3.17 3.20 2.08 3.18 3.13 2.76 5.53 3.82 Deviation Deviation Std. Std.

9.42 4.63 6.38 6.73 6.74 5.45 7.26 8.23 7.89 15.84 14.82 10.17 17.21 13.63 Mean Mean Form 1517 Form Form 1618 IN SCORES IN SCORES 20.00 10.00 13.00 15.00 13.00 26.00 20.00 20.00 10.00 14.00 16.00 13.00 27.00 20.00 Maximum Maximum .00 .00 .00 .00 0.00 2.00 0.00 0.00 0.00 0.00 0.00 4.00 3.00 2.00 Minimum Minimum ly lower. ly lower. EST SCORES N N 276 276 276 276 276 276 276 780 780 780 780 780 780 780 l_Analysis l_Analysis _Reasoning _Reasoning

ng_Writing ng_Writing al_Compreh al_Compreh Domain Domain n_Geometry n_Geometry an_Algebra an_Algebra ng_Grammar ng_Grammar qu qu

verba verba writi writi qua qua quan quan verb verb writi writi The correlations reported in Tables 12.4 and 12.5 in the box at the right corner are all related to the domain the right corner at all related are 12.4 and 12.5 in the box in Tables reported The correlations The correlations between the domain scores and the subject scores for each test form were computed and form test and computed each for were scores subject and the domain scores the between The correlations multi-traits fall within expectations and provided both convergent and divergent validity and divergent both convergent and provided fall within expectations multi-traits true for the correlations between the domain score between the Verbal Reasoning and Grammar/Writing subjects. subjects. and Grammar/Writing Reasoning Verbal the between domain score the between correlations the for true test and those Reasoning the Quantitative from the domain score between On the other hand, the correlations The across patterns observed all lower. test were test or the Grammar/Writing Reasoning either the Verbal from domain scores were all comparative were domain scores The same is highly correlated. the same subject test tended to be more from domain scores In general, the scores. were higher than those between the Grammar/Writing subject scores and the Quantitative Reasoning domain Reasoning and the Quantitative subject scores the Grammar/Writing higher than those between were and the Grammar/ scores subject Reasoning the Quantitative between the correlations On the other hand, scores. Reasoning and the Verbal subject scores Reasoning the Quantitative and those between domain scores Writing scores. The correlations between the Verbal Reasoning subject scores and the Grammar/Writing domain scores domain scores and the Grammar/Writing subject scores Reasoning the Verbal between The correlations scores. domain Reasoning and the Quantitative subject scores Reasoning the Verbal higher than those between were domain scores Reasoning and the Verbal scores subject Grammar/Writing the between The correlations scores. summarized in Tables 12.4 and 12.5 for Forms 1517 and 1618, respectively. Across forms, similar patterns Across were 1517 and 1618, respectively. Forms 12.4 and 12.5 for summarized in Tables was scores and the Grammar/Writing scores Reasoning the Verbal between the correlation In general, observed. or Grammar/Writing Reasoning and the Verbal score Reasoning the Quantitative higher than those between AND THE SUBJECT T CORRELATIONS BETWEEN THE DOMA CORRELATIONS Table 12.3 Descriptive Statistics for the Domain Scores for the Domain Scores 12.3 Descriptive Statistics for Table Table 12.2 Descriptive Statistics for the Domain Scores for Scores Domain for the Statistics 12.2 Descriptive Table 67

CHAPTER TWELVE Table 12.5Correlations amongtheDomainScores andtheSubjectScores for Table 12.4Correlations amongtheDomainScores andtheSubjectScores for Q Q_Geometry Q_Algebra W_Writing W_Grammar V_Compreh V_Analysis Q_Orig W_Orig V_Orig Reasoning Q Geometry Q_ Q_Algebra W_Writing Grammar W_ V_Compreh V_Analysis Q_Orig W_Orig V_Orig ** Correlation is significant at the 0.01 level ( is significant at the 0.01level ** Correlation _Reasoning _ .504 .451 .443 .688 .661 .958 .837 .538 .746 1 Orig V_ .559 .559 .533 .755 .729 .979 .911 .635 .805 1 Orig V_ ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** .524 .510 .479 .906 .904 .722 .610 .583 1 Orig W_ .601 .560 .562 .926 .917 .792 .726 .663 1 Orig W_ ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** Q_ .890 .892 .812 .535 .520 .520 .442 1 Orig .861 .626 .596 .625 .571 1 Orig Q_ .871 .870 ** ** ** ** ** ** ** ** ** ** ** ** ** ** V_ .427 .364 .353 .562 .541 .646 1 Analysis .503 .510 .470 .682 .655 .806 1 Analysis V_ 2-tailed). ** ** ** ** ** ** ** ** ** ** ** ** .480 .439 .434 .667 .640 1 Compreh V_ .550 .546 .529 .742 .718 1 Compreh V_ ** ** ** ** ** ** ** ** ** ** .458 .462 .432 .638 1 Grammar W_ .543 .491 .516 .700 1 Grammar W_ ** ** ** ** ** ** ** ** .565 .541 .520 1 Writing W_ .491 .461 .436 1 Writing W_ Form1517 Form1618 ** ** ** ** ** ** .622 .657 1 Algebra Q_ .611 .611 1 Algebra Q_ ** ** ** ** .611 1 Geometry Q_ .661 1 Geometry Q_ ** ** 1 Reasoning Q_ 1 Reasoning Q_ 68

CHAPTER TWELVE

atio

able 12.6. able This ratio This ratio 5.688 6 Index Factor Factor , 141-151. If ratio this 7 Difference R Difference 20 nt factor. nt (2), 139-164. , 3.077 0.541 igenvalues E examined. examined. Differences between between Differences Applied Psychological Measurement, 9 Applied Psychological Measurement, Educational and Psychological Measurement % According to Kaiser’s rule, two components were were components two rule, Kaiser’s to According . Ann Arbor, MI: English Language Institute, University of University MI: English Language Institute, . Ann Arbor, Michigan. mulative mulative 5 58.518 73.076 79.903 85.696 90.939 95.602 100.000 Cu ifferences ECPE annual report: 2002 ECPE annual report: 6.826 5.794 5.243 4.663 4.398 f Variance 58.518 14.558 Form 1517 R ANALYSIS % o .478 .406 .367 .326 .308 Total 4.096 1.019

5 6 7 1 2 3 4 Johnson, J. S., Yamashiro, A., & Yu, J. (2003). J. A., & Yu, Yamashiro, S., J. Johnson, Kaiser, H. F. (1960). The application of (1960). The application computers to factor analysis. H. F. electronic Kaiser, Assessing unidimensionality of (1985). Methodology review: J. Hattie, tests and items.

Hattie (1985) suggests using the ratio of (1985) suggests ratio using the Hattie firstthe the to second factor the and factor between difference the The eigenvalues and the eigenvalue differences between factors for Form in T summarized 1517 are for factors between differences eigenvalue and the The eigenvalues Exploratory factor analyses were conducted based on the seven domain scores from the three subject tests for for tests subject three the from domain scores seven on the based conducted analyses were factor Exploratory Component xtracted. xtracted. strong. The scree plot presented in Figure 12.1 for Form 1517 in general supports one domina one supports general 1517 in Form 12.1 for in Figure presented plot The scree strong. Explained for 12.6 Variance Table was dubbed as the Factor Difference Ratio Index (FDRI) in Johnson, Yamashiro, and Yu (2003). and Yamashiro, Johnson, (FDRI) in Index Ratio Difference Factor as the was dubbed than 4, and larger were first the for factor eigenvalues The strong. relatively first 3, the than is factor is larger relatively the this criterion, on around 3. Based was factor first factorstwo firstthe was between difference the difference between the second and the third factor to examine the relative strength of strength relative the examine to factor third the and second the between difference the factor. first The eigenvalues for the first two factors were larger than 1. Kaiser (1960) recommends extracting the the extracting 1. recommends than Kaiser (1960) larger were factors first the two for The eigenvalues 1. than is larger that eigenvalue on the based component e Form 1517, Eigenvalues and Eigenvalue D Form 1517, Eigenvalues both test forms using the software Statistical Package for the Social Sciences (SPSS). Eigenvalues, eigenvalue eigenvalue Eigenvalues, (SPSS). Sciences Social the for forms test both Package Statistical software the using of percentage and the differences were factor each by explained variance EXPLORATORY FACTO EXPLORATORY 7 5 6 69

CHAPTER TWELVE 8 Table 12.7Compon the Grammar/Writing components indicates that the Quantitative Reasoning domainscores are highlyrelated to both the Verbal Reasoning and domain scores loadonboth component 1and2,but with alarger weight oncomponent 1aswell. This component 1isrelated to Verbal Reasoning andGrammar/Writing skills. Onthe other hand,quantitative Grammar/Writing subjects loadpositively oncomponent 1but negatively indicates oncomponent 2.This that the total variance. Ingeneral, one dominant component was 14.6% of Form component 1517,the first explained over 58.5% variance while the secondcomponent explained about the total variance. factor accounts the first for 20%ormore of thatsuggested atest isunidimensional if Percentage ofVariance Figure 12.1.Thescree plotforForm1517basedondoma al 27peet h odn feach domainscores component. The inthe VerbalTable 12.7presents Reasoning the loadingof and variance explained by each factor isalsopresented inTable percentageThe of 12.6.Reckase (1979) Reckase, M.D. (1979).Unifactor latent trait modelsappliedto multifactor tests: Results andimplications. writi verb Extraction Method: Principal Component quan qua writi verba qu ng_Grammar an_Algebra n_Geometry al_Compreh ng_Writing _Reasoning l_Analysis ent Matrix a. 2components Explained extracted. as well. Analysis. .769 .742 .716 .801 .784 .811 .727 1 in scores. Component supported. ora fEducationalStatistics Journal of -.269 -.265 -.341 -.401 .400 .468 .470 2 , 4 (3), 207-230.

8 For For 70

CHAPTER TWELVE 8.592 Factor Factor atio Index atio Difference Difference R ne factor. ne 3.729 0.434 igenvalues E orm 1618. Differences between between Differences % mulative mulative in scores. 65.367 77.465 83.365 88.616 93.297 97.415 100.000 Cu ifferences 5.900 5.252 4.681 4.118 2.585 f Variance 65.367 12.098 % o Form 1618 .413 .368 .328 .288 .181 .847 Explained Total 4.576 5 6 7 1 2 3 4

Component The percentage of variance explained by each component for Form 1618 is presented in Table 12.7. The 12.7. The of in Table The is presented 1618 percentage Form for component each by explained variance The eigenvalues and the eigenvalue differences between factors for Form 1618 are summarized in Table 12.8. 12.8. Table Formin summarized 1618 are for factors between differences eigenvalue and the The eigenvalues first component accounts for 65.4% offirst accounts component 12% about explained second component the while variance total the of F for unidimensionality supported EFA from The results variance. total the Percentage of Variance Percentage Figure 12.2. The scree plot for Form 1618 based on doma 12.2. The scree Figure Table 12.8 Variance Explained for 12.8 Variance Table Ratio Index (FDRI) is larger than 3. The first factor is relatively dominant. Kaiser’s rule (Kaiser, 1960) also also 1960) (Kaiser, rule dominant. Kaiser’s 3. The than relatively first is larger (FDRI) is factor Index Ratio 1618 supports o Form 12.2 for in Figure suggested presented plot The component. one scree Only the eigenvalue for the first factor was larger than 1. The eigenvalue for the first factor were larger than 4, than larger were firstthe for factor than 1. first the eigenvalue The was factor larger for eigenvalue Only the Difference Factor the (1985), Hattie to 3. According two factorsthe first was above between difference and the Form 1618, Eigenvalues and Eigenvalue D and Eigenvalue 1618, Eigenvalues Form 71

CHAPTER TWELVE respectively. three The models are presented inFigures 12.3,12.4,and12.5,resp a one-factor model, atwo-factor model,andathree-factor model were fitted to the domainscores for each form in three content areas: Verbal Reasoning, Grammar/Writing, andQuantitative Reasoning. Thus, three models: while that for Form 1618supported aone-factor model. Intheory, CLT wasdeveloped to assess students’ ability extracted. factors Ingeneral, the empirical data from Form 1517supported atwo-factor model number of CONFIRMATORY FACTO In general, the results from EFA conducted for Forms 1517and1618were not consistent inidentifying the R ANALYSIS Figure 12.3.TheOne-Fac tor Model. ectively. 72

CHAPTER TWELVE

tor Model. tor Model.

Figure 12.4. The Two-Fac 12.4. Figure Figure 12.5. The Three-Fac Figure 73

CHAPTER TWELVE Table 12.9Model Grammar/Writing, andQuantitative CLT with content coverage inVerbal isconsistent with the theoreticalmodel. This design of Reasoning, model. Also, the chi-square test asreported inTable 12.9supported the three-factor model asthe best fitting parameters. Both AICandBICas reported inTable 12.9identified the three-factor model as the best fitting the deviance of to be estimated; one that minimizes fit indexes AICorBIC.The are computed a (AIC; Akaike, 1974) conclusion that both the two-factor andthe three-factor models were the best fittin SRMR), values below 0.10or0.05are usuallyconsidered adequate. Ingeneral, these fit indexes supported the cutoff values couldbejust above 0.90and for fit indices based on residual matrices (suchas RMSEA and practice, arough guideline isthat for absolute fit indices andincremental fit indices (such asCFIandTLI), the cut values for amodel with goodfit are CFI > 0.95,TLI > 0.95,RMSEA < 0.06,and SRMR < 0.08. (RMSEA) andthe standardized root mean square error residual (SRMR). Based onHuandBentler (1999), the comparative fit index (CFI), the approximation Tucker-Lewis Index (TLI), the root mean square error of 11 10 9 and de isconsistent with the theoreticalmodel. This framework andthe theoretical content modelinthe CLT design components to be extracted. CFA empirically identified the best fitting model, the three-factor numbers of the CLT EFA forms. intends to provide validity evidence structure supported different related of to the internal 1517 1618

Form where Further, relative information-based fit indexes were also Theyexamined. are Akaike’s criterion information Several goodness-of-fit indices (GFI) were used to evaluate the model-data These include fit. chi-square (χ2), nsmay hsscinasse h nenlsrcueo tots om fCLT, of two test forms In summary, Form 1517and1618.It structure of this section assessed the internal Shaz .(98.Etmtn h ieso f oe.Anl fStatistics, 6,461-464. a model. Annalsof Schwarz, G. (1978).Estimating thedimensionof Akaike, H. (1974).Anewlookat thestatistical modelidentification.  Structural EquationModeling,6,1-55 Structural Hu, L.,&Bentler, P. M.(1999).Cutoff criteria for fitindexes in covariance structure analysis:Conventional criteria newalternatives. versus velopment. 1-factor 3-factor 2-factor 1-factor 3-factor 2-factor Model ist epseirma f h eine esr ffit; the deviance, ameasure of he posterior mean of N Comparison is the samplesize; and 25229.349 8925.044 8939.704 9044.060 24848.073 24862.271 10 andBayesian criterion (BIC;Schwarz, 1978). information adtedvac fthe posterior model andthe deviance of AIC . 25327.194 24959.896 24964.776 9011.933 9019.353 9120.088 BIC Reasoning. 13 39 2 2 6.828 (11) 9.081 (11) 5.488 (13) 7.279 (13) 1.844 (14) 6.356 (14) c 2 IEEE Transactions onAutomaticControl ( df ) which isthe difference between the posterior mean p s follows. <.001 <.001 0.813 0.020 0.614 0.011 -value bsdo h otro siae fthe based onthe posterior estimates of p RMSEA stenme fmodel parameters is the number of <.001 <.001 0.059 0.175 0.038 0.187 , 19 11 The best fitting modelthe is The , 716-723. g models. 1.000 0.990 0.906 1.000 0.995 0.859 CFI 1.006 0.984 0.858 1.001 0.992 0.789 TLI SRMR 0.013 0.023 0.062 0.009 0.016 0.072

9 In In 74

CHAPTER TWELVE vidence.” vidence.” Parameters for the CLT are are CLT the for Parameters 12

s from it. s from ges value. ges beyond. beyond. d Choice.” iscipline. Problems.” eir ideas. eir

AERA, 2014.

In terms of of Analysis a high level demonstrates exam CLT the and internal content both validity. structure, The breakdown ofThe breakdown of types the mirrors types three into questions Reasoning Quantitative logical reasoning On the Quantitative Reasoning section, questions are broken down into three types: Algebra, Geometry, Geometry, Algebra, types: three into down broken are questions section, Reasoning Quantitative On the Here, Grammar questions serve to evaluate a student’s ability to use English standards and conventions and conventions English use to ability standards student’s a evaluate to serve Grammar questions Here, On the Grammar/Writing section, questions are also broken down into two types: Grammar questions, Grammar questions, types: two into down also broken are questions section, Grammar/Writing On the As a result, students are asked to engage with a text on two essential levels: first, understandingtheir of levels: essential on two engage to asked a text with are students As a result, the On the Verbal Reasoning section, questions are broken down into two types: Comprehension questions, questions, Comprehension types: two into down broken are questions section, Reasoning Verbal On the In addition to being technically valid, the content of content the valid, technically being to In addition to According test. passes also a reasonableness CLT the

12 to key skills in reading, writing, and mathematics. CLT test scores are thus a legitimate measure of measure a legitimate thus are scores test CLT and mathematics. students’ writing, skills in reading, key to and level college the at work academic for preparation and aptitude of the test’s structure found that a three-factor model is a good fit for evaluating its domain scores, as the exam exam the as of domain scores, its evaluating for is a good fit model a three-factor that found structure test’s the ofis composed The types of tests. subject weighted equally three correspond test subject in each questions level mathematics or science coursework in college, but are important indications of indications important are but in college, coursework or science think to ability mathematics level a student’s of a crucial skill regardless and logically, d academic clearly Summary to understand and work with symbols. Geometry questions test students’ spatial abilities and understanding and understanding abilities Geometry spatial students’ test questions symbols. with understandto and work questions of Reasoning Mathematical shapes. and 3-D 2-D and other squares, triangles, as lines, such shapes in pursuing higher- interested students only necessary not These for skills are logical abilities. students’ test “Logic” and “Word “Logic” and “Word ability students’ test questions Algebra and beyond. in college well students will serve and analysis skills that and Mathematical Reasoning. Algebra questions include the subdomains “Arithmetic and Operations” and and and Operations” subdomains “Arithmetic the include questions Algebra Reasoning. and Mathematical Geometry,” Geometry subdomains “Coordinate the include questions and Equations.” Expressions “Algebraic subdomains the include questions Reasoning Mathematical of“Properties and “Trigonometry.” Shapes,” communicate th communicate use language and style to accurately and appropriately convey the tone, argument, and intent of intent and argument, tone, the convey Both and appropriately text. the accurately to and style language use a grasp only have not must students level, college the at succeed to writing: high-level for essential skills are and concisely clearly to ability also the but properly, of arguments their convey to required conventions the which include the subdomains “Structure,” “Style,” and “Wor “Style,” subdomains “Structure,” the include which to ability a student’s evaluate to serve questions and information. ideas Writing convey clearly so as to properly, assess and comprehend a text, but to draw new ideas and conclusion ideas new draw to but a text, and comprehend assess questions, Writing and Structure”; and Sentence and “Punctuation subdomains “Agreement” the include which text’s meaning, the author’s intent, and the information conveyed by the passage, and second, their ability to to ability their and second, passage, the by information and the intent, conveyed author’s the meaning, text’s analysis multi-level the This reflects conclusions. valid information draw to and synthesize text in the analyze only to not asked are students and beyond: college, engage to in during high school, required are students that Analysis questions, which include the subdomains “Textual Analysis” and “Interpretation of “Textual subdomains the include which E Analysis questions, reasonable assessment of assessment reasonable kind of the colle and skills that knowledge and Relationships”; and “Passage Details,” “Passage a Whole,” as subdomains “Passage the include which and theory interpretations the support of of uses proposed for scores test tests.” of indicators appropriate yield results test that ensure to designed thinking higher-level for capacity individuals’ The of range college. for as preparation as well of in each types question a provide sections test three the the Standards for Educational and Psychological Testing (2014), “Validity refers to the degree to which evidence evidence degree the to refers which to “Validity (2014), Testing and Psychological Educational for Standards the Evidence Based on Content Evidence 75

CHAPTER TWELVE 76 tric work. tric CLT exam. CLT y control. screpancy. QUALITY CONTROL OF OF CONTROL QUALITY e quality. e PSYCHOMETRIC ANALYSIS ANALYSIS PSYCHOMETRIC

precision.

Data cleaning is an important step in all psychometric analysis. When CLT delivered data for different different for data delivered CLT When analysis. step is an important cleaning in all psychometric Data This section documents what we did to perform quality control on the psychometric analysis results analysis results perform did to psychometric on the we This control quality what documents section Second, to ensure that the outputs from psychometric analysis for CLT were error-free, our teamour error-free, were CLT analysisfor psychometric from outputs the that ensure to Second, First, CLT identified all psychometric steps, procedures, and analyses at the very and stage initial the at and analyses procedures, steps, all psychometric identified First, CLT The first step in quality control for psychometric work is to develop a process flowchart to standardize standardize to flowchart a process to develop is work psychometric for The first step control in quality Psychometrics is important in the whole process of process whole in the is important Psychometrics in the role an important plays It development. test Previous chapters in this technical report have presented the results of results the presented have report technical in this chapters analysis ofPrevious our psychometric the case laid a fundamental base for quality psychometric analysis. We further cleaned the data when when data the cleaned further We analysis. psychometric quality for base laid a fundamental case of sources included multiple We needed. cleanness the ensure to information control in our quality taken to ensur to taken including the on all variables, statistics, as descriptive such ran basic analyses, forms test we us, to CLT of ran distributions offrequency We identify to take. all variables may or symbols a variable values identify to helped table The frequency record. data student and in each variable outliers in each any check This reasonableness look reasonable. did not or symbols which values ofany each variable and of psychometricians conducted each analysis independently and compared the results afterwards. We ofWe afterwards. results the independentlyanalysis compared and each conducted psychometricians discussed any discrepancy identified inthe comparison of ourresults andexplored furtherto identify of sources the di any have and CLT steps and/or correction actions we and what found, we what document, in this reported ensure that the right and defensible psychometric steps and procedures would be followed in developing developing in followed be would stepsand procedures psychometric and defensible right the that ensure agreement of type as the to reached we discussions, these through psychometric Further, CLT. the who is carried should be analysis out, psychometric a specific how conducted, be to need that analyses psychometric performed should be each control quality from output on the how task, the for responsible qualit for is responsible and who analysis, analysis with different teams. The third key step is to document what has been done, what has been what done, has been what step to document is key third The teams. different analysis with has The CLT steps. correction and take quality ensure to taken has been action and what found, of development in the measures all three taken faithfully the of planning process when entire during the This reinforced further was aspect development. test CLT of rounds had multiple of contents the to staff have CLT the discussions with We report. technical this to item analysis, differential item functioning analysis, scoring, calibration, equating, scaling, standard standard scaling, equating, calibration, scoring, analysis, functioning item differential analysis, item to Thus, tests. quality high ensures work psychometric High quality writing. report and technical setting, of aspects in all these addressed should be control quality psychome the replicate is to control in quality element The key second steps the and follow. to procedure the fairness, and ultimately to confirm that the CLT test measures what it is intended to measure, and with to measure, is intended it what measures test confirm to fairness, and ultimately CLT the that adequate ofprocess then in tests, field and designing specifications test creating with beginning development, test about form test psychometrics from input needs construction Further, data. and analyzing test reviewing greatly contributes of quality psychometrics psychometric overall the forms. test new pulled the Finally, CLT exam at the item level, the subscore level, the subject level, and the overall test level. We performed We level. test overall and the level, subject the level, subscore the level, item the at exam CLT of reliability the assess evidence, validity collect to analyses all these and assure evaluate scores, test the 77

QUALITY CONTROL appendix presents further detailed in report isaccompanied by tables with empirical data aboutthe test items andreported scores, andthe this CLTThe psychometric who signaconfidentiality agreement. 9-12 of analysisinChapters of discrepancy.potential sources Such documents are of available uponrequest for relevant stakeholders documente results, andthe specifics that need to be controlled shouldallbe streamlined, standardized, and theanalysisflowchart for the psychometric analysis, to the steps follow to ensure quality control of and well documented sothat replication by parties isfeasible. independent Further, the action quality control, each for step conducting each psychometric analysisshouldbe standardized of such Automationparallel forms. helped toremove the errors inmanual implementation of the student population across different analyses were conducted for the total andsubgroups of analysis results. Further, some psychometric analyses were automated to avoid humanerrors. Some the psychometric analysisprocedure. We the carefully checked andevaluated the reasonableness of quality quality controlperson policyestablished two layers of process This continued untilthe exactform. match andbetween- within-person wasobtained. This analyses were conducted ondifferent data sets, andany other special issues specific toanindividual dealing with missingvalues, changes that needed to be made when thevariable recoding, ways of for discrepancy. included different These approaches to the exclusion rules applied indata cleaning, analysis. When we hadfollowed standard procedures, we further explored other potential sources verifiedindependently that the same standardized andprocedures steps hadbeen followed in the what implemented was inourown analysis. We screening first of one ormultiple rounds of usmet anddiscussed potential sources for the discrepancy.was found, the two of We each did the results adiscrepancy matched, they were used inthe final reporting.reported values, andif If member ranaquality cont ourown work before the other psychometricwas responsible team for runningquality control of us defense, quality control. each For of layercheck andabetween-person the former level of of was responsible for checking quality control. were Two implemented, awithin-person layers of summarized the results, put them inthe written document, andthen shared it with the other, who discussionsto ensure the quality of psychometric team held multiple rounds of any potential issues andreported them to the CLT staff. Before beginning ouranalysis, we as a data andresults, previous technical reports, andother reports for studies. special We identified the data to be used for psychometric analyses. include previous These analysis of and the integrity Liru Zhang, Hong Jiao, Following these guidelines, we documented quality control results inwriting. helped track This In general, to ensure quality control andto increase the efficiency andaccuracyin the process As elaborated above, we communicated frequently andfollowed closely the standardization of Further, the quality control conducted each party independently analysis. We checked the usled the psychometric task andranthe analysis, During the psychometric analysisstage, one of d aswell. Ph.D., University of Maryland, Co Maryland, Ph.D., University of Ph.D., Independent rol check. Consultant formation. llege Parkllege assurance. the data. analyses. 78

QUALITY CONTROL

ntitative ntitative 0.39 0.15 0.29 0.36 0.38 0.57 0.58 0.31 0.16 0.55 0.39 0.54 0.57 0.46 0.47 0.52 0.35 0.46 0.58 0.43 0.35 0.50 0.39 0.75 0.65 0.72 0.86 0.59 0.78 0.63 0.54 0.35 Reasoning Qua

0.80 0.47 0.72 0.74 0.41 0.87 0.87 0.34 0.68 0.39 0.47 0.86 0.38 0.63 0.36 0.80 0.50 0.87 0.42 0.49 0.68 0.56 0.64 0.16 0.80 0.45 0.85 0.94 0.74 0.63 0.51 0.29 Form 1618 Form Writing Grammar/

0.74 0.30 0.49 0.64 0.76 0.48 0.61 0.67 0.66 0.87 0.67 0.41 0.78 0.42 0.39 0.56 0.81 0.79 0.83 0.50 0.70 0.80 0.78 0.40 0.84 0.58 0.82 0.44 0.77 0.84 0.77 0.75 Verbal Verbal Reasoning

ntitative ntitative 0.41 0.43 0.48 0.52 0.24 0.68 0.55 0.78 0.65 0.70 0.65 0.71 0.48 0.55 0.60 0.41 0.42 0.53 0.49 0.46 0.45 0.33 0.56 0.83 0.73 0.59 0.83 0.65 0.54 0.61 0.85 0.71 Reasoning Qua

0.47 0.81 0.71 0.34 0.41 0.54 0.81 0.95 0.68 0.93 0.36 0.75 0.64 0.65 0.92 0.21 0.37 0.13 0.88 0.87 0.83 0.31 0.83 0.78 0.83 0.50 0.41 0.83 0.82 0.74 0.95 0.31 Writing Form 1517 Form Grammar/

m P-Values

0.39 0.50 0.55 0.85 0.81 0.38 0.60 0.75 0.49 0.64 0.61 0.24 0.49 0.43 0.52 0.59 0.68 0.22 0.80 0.72 0.68 0.55 0.78 0.47 0.62 0.64 0.40 0.86 0.41 0.55 0.70 0.41 Verbal Reasoning

Table A1 Ite A1 Table Appendix 79

APPENDIX Table A2temPoint-BiserialC Table A1ItemP-Values ( With item 0.20 0.29 0.40 0.29 0.36 0.47 0.33 0.34 0.40 0.35 0.35 0.05 0.43 0.27 0.09 0.32 0.29 0.38 0.34 0.21 0.30 0.37 0.38 0.42 Reasoning Verbal 0.33 0.59 0.51 0.62 0.72 0.48 0.42 0.61

-0.02 W/o item 0.21 0.31 0.29 0.13 0.23 0.30 0.31 0.35 0.14 0.22 0.34 0.22 0.29 0.40 0.26 0.27 0.33 0.29 0.30 0.37 0.19 0.01 0.26

With item 0.39 0.45 0.41 0.40 0.42 0.19 0.32 0.33 0.31 0.35 0.32 0.43 0.30 0.11 0.09 0.31 0.34 0.33 0.14 0.39 0.39 0.58 0.41 0.25 Grammar/ Form 1517 Writing Continued) 0.85 0.82 0.53 0.55 0.47 0.60 0.29 0.31 orrelation W/o item 0.01 0.24 0.28 0.25 0.07 0.35 0.31 0.52 0.35 0.17 0.31 0.38 0.35 0.34 0.37 0.11 0.26 0.27 0.26 0.27 0.29 0.36 0.24 0.05

With item 0.32 0.47 0.31 0.45 0.41 0.44 0.35 0.45 0.39 0.53 0.39 0.50 0.38 0.47 0.47 0.44 0.38 0.52 0.52 0.35 0.38 0.24 0.50 0.41 Quantitative Reasoning 0.27 0.25 0.43 0.28 0.27 0.40 0.23 0.40 W/o item 0.36 0.26 0.42 0.25 0.40 0.35 0.38 0.29 0.40 0.33 0.48 0.33 0.44 0.33 0.41 0.42 0.38 0.34 0.47 0.47 0.29 0.33 0.18 0.46

With item 0.22 0.47 0.58 0.56 0.56 0.42 0.29 0.24 0.48 0.26 0.20 0.50 0.46 0.30 0.38 0.52 0.43 0.51 0.46 0.49 0.34 0.46 0.35 0.33 Reasoning 0.71 0.49 0.43 0.64 0.37 0.44 0.56 0.59 Verbal W/o item 0.15 0.41 0.55 0.52 0.52 0.37 0.23 0.18 0.44 0.20 0.15 0.46 0.41 0.25 0.32 0.47 0.38 0.47 0.43 0.45 0.28 0.42 0.29 0.29

With 0.62 0.18 0.51 0.73 0.33 0.29 0.42 0.67 item 0.19 0.36 0.34 0.52 0.46 0.54 0.45 0.44 0.29 0.51 0.36 0.45 0.62 0.52 0.44 0.47 0.21 0.33 0.34 0.34 0.42 0.43 0.49 0.47 Form 1618 Grammar/ Writing W/o item 0.14 0.30 0.28 0.49 0.40 0.50 0.40 0.38 0.22 0.47 0.29 0.39 0.58 0.47 0.41 0.43 0.15 0.27 0.28 0.28 0.39 0.38 0.43 0.43

With item 0.43 0.43 0.36 0.45 0.39 0.41 0.43 0.47 0.42 0.55 0.43 0.47 0.23 0.26 0.44 0.55 0.47 0.56 0.41 0.37 0.42 0.35 0.53 0.57 Qua 0.26 0.22 0.39 0.30 0.22 0.16 0.17 0.37 Reasoning ntitative W/o item 0.37 0.30 0.39 0.33 0.35 0.37 0.41 0.36 0.49 0.38 0.41 0.18 0.20 0.38 0.49 0.41 0.51 0.35 0.32 0.36 0.30 0.48 0.52 0.38

80

APPENDIX 0.46 0.13 0.06 0.31 0.25 0.38 0.24 0.40 0.39 0.12 0.09 0.22 0.39 0.06 0.16 -0.04 0.22 0.51 0.20 0.13 0.37 0.30 0.44 0.31 0.46 0.45 0.17 0.01 0.14 0.28 0.45 0.12 0.44 0.43 0.36 0.40 0.45 0.37 0.52 0.41 0.32 0.28 0.31 0.13 0.21 0.50 0.15 0.22 0.50 0.48 0.42 0.45 0.49 0.43 0.56 0.46 0.38 0.33 0.37 0.19 0.27 0.54 0.22 0.27 0.47 0.55 0.41 0.49 0.55 0.19 0.38 0.53 0.51 0.32 0.40 0.45 0.43 0.49 0.34 0.50 0.51 0.59 0.45 0.53 0.59 0.24 0.43 0.58 0.55 0.38 0.46 0.50 0.48 0.54 0.39 0.55 0.20 0.19 0.32 0.46 0.44 0.33 0.32 0.34 0.34 0.19 0.28 0.28 0.36 0.23 0.04 0.27 Continued) 0.41 0.29 0.11 0.33 0.26 0.52 0.49 0.39 0.38 0.40 0.39 0.25 0.33 0.34 0.25 0.37 0.23 0.31 0.16 0.39 0.35 0.27 0.30 0.25 0.31 0.30 0.46 0.33 0.35 0.17 0.19 0.27 0.30 0.39 0.24 0.46 0.42 0.35 0.36 0.29 0.38 0.34 0.53 0.39 0.40 0.24 0.27 0.35 0.39 0.31 0.34 0.12 0.16 0.37 0.25 0.41 0.22 0.24 0.42 0.38 0.32 0.38 0.35 0.16

0.45 0.38 0.40 0.19 0.24 0.43 0.32 0.46 0.29 0.31 0.48 0.44 0.37 0.44 0.41 0.23 Table A2 tem Point-Biserial Correlation ( Correlation Point-Biserial A2 tem Table 81

APPENDIX Table A3DIFAnalysisResultsforForm1517Verbal Reasoni alphaMH 0.8389 1.0243 0.7476 1.1212 0.7305 0.8579 1.3185 0.9458 0.8349 0.4997 0.4669 1.0072 0.8293 0.7658 1.1798 1.2537 1.9861 1.2677 0.7054 1.3593 1.0412 0.7489 0.7878 0.9629 1.0809 1.1322 1.1698 1.1934 0.9892 1.4305 0.818 0.852 1.059 1.595 1.215 deltaMH -0.0169 -0.3885 -0.5313 -1.6125 -0.5575 -0.7214 -0.0948 -0.1828 -0.1347 -1.0972 -0.2917 -0.3686 -0.4156 -0.4576 -0.8414 -0.0564 -0.2688 -0.6498 0.4129 0.6837 0.3601 0.1308 1.6305 1.7898 0.4722 0.8202 0.6796 0.5606 0.3764 0.0888 0.0255 0.627 0.738 0.424 0.44 Gender ET S Category C C C A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A B alphaMH ng Section 1.1434 0.9017 1.5308 0.8816 0.8283 1.9771 0.9856 0.7572 1.0086 0.9213 1.1389 1.2569 0.6081 1.2218 0.6595 1.1923 0.9971 1.2571 0.9634 0.9576 0.6192 1.1017 1.1692 0.7747 0.9385 0.9211 1.0061 1.2483 1.1613 1.253 0.656 1.339 0.666 1.934 1.042 deltaMH -0.3149 -0.5301 -1.0006 -1.6018 -0.0202 -0.3057 -0.5374 -0.4708 -0.4134 -0.5376 -1.5501 -0.2276 -0.3673 -0.0966 -0.0142 -0.5212 -0.3514 0.2433 0.2961 0.4426 0.0342 0.6537 0.1926 1.1691 0.9909 0.9552 0.9783 0.0068 0.0876 0.1018 1.1264 0.1493 0.1932 -0.686 0.6 Race ET S Category C C A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A B B B 82

APPENDIX B B B B B B A A A A A A A A A A A A A A A A A A A A A A A A A A A C C S Category S ET Race 0.915 0.757 -0.984 0.5985 1.0113 0.2904 0.4328 1.5361 0.1237 0.3268 0.1546 0.2659 1.1824 -0.2948 -0.5987 -1.1001 -0.2729 -0.2463 -0.1273 -0.4517 -0.1513 -0.3571 -1.2718 -0.3213 -1.1741 -1.0713 -0.3538 -0.6459 -1.5446 -0.1011 -0.1818 -0.8717 -0.6481 -0.3814 -0.1204 deltaMH Continued) 1.52 1.597 1.044 0.893 ng Section 1.1336 1.2902 0.7752 1.1231 1.1105 0.6503 1.0557 0.8838 0.8318 1.2119 1.0665 1.1641 1.7181 1.1465 1.6481 0.5201 1.5776 1.1625 0.9487 0.8702 1.3163 1.9295 0.9363 1.0805 1.4491 1.3176 0.6775 0.7246 1.1762 0.6046 1.0526 alphaMH B B B B A A A A A A A A A A A A A A A A A A A A A A A A A A A A A C C S Category S ET Gender 0.6 0.9783 0.9552 1.1264 0.1018 0.0876 0.0068 0.1932 0.1493 0.6537 0.0342 0.4426 0.2961 0.2433 0.1926 1.0709 0.1538 -0.3673 -0.2276 -1.5501 -0.5376 -0.4134 -0.3149 -0.3514 -0.5212 -0.0142 -0.0966 -0.0202 -1.6018 -1.0006 -0.5301 -1.0595 -0.6762 -0.5426 -0.3057 deltaMH

0.666 1.934 1.042 1.253 0.634 0.9971 1.1923 0.6595 0.6192 0.9576 0.9634 1.2571 0.7747 1.1692 1.1017 1.2483 1.0061 0.9211 0.9385 1.5308 0.9017 1.1434 1.1613 0.9856 1.9771 0.8283 0.8816 1.1389 0.9213 1.0086 0.7572 1.2597 0.9366 1.5696 1.3334

alphaMH Table A4 DIF Analysis Results for Form 1517 Grammar/Writi A4 DIF Analysis Results for Table Table A3 DIF Analysis Results for Form 1517 Verbal Reasoning Section ( Reasoning 1517 Verbal for Form Results Analysis A3 DIF Table 83

APPENDIX Table A5DIFAnalysisResultsforForm1517QuantitativeReasoni Table A4DIFAnalysisResultsforForm1517Grammar/Writing Section( alphaMH 0.8455 0.9659 1.1229 0.9289 1.4137 0.7519 0.7443 0.7524 0.8502 1.3395 1.3791 1.3201 0.9324 1.5761 1.2729 1.1958 0.7792 0.9161 0.7595 1.0701 0.5559 1.0728 0.8135 1.0293 1.0027 1.4491 1.0805 0.9363 1.2569 0.6081 1.2218 0.893 1.044 0.656 1.339 deltaMH -0.6526 -1.0692 -0.4201 -0.1592 -0.1652 -0.0679 -0.0063 -0.8717 -0.1818 -0.1011 -0.5374 -0.4708 -0.2724 -0.8135 -0.7553 0.3944 0.0815 0.1733 0.6941 0.6685 0.3815 0.1644 0.5862 0.2059 0.6464 1.3799 0.2659 0.1546 1.1691 0.9909 -0.687 -0.567 -0.686 0.485 0.67 Gender ET S Category A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A B B B alphaMH 0.8922 1.4642 0.4956 0.7715 0.9204 0.8579 0.6252 1.0069 0.7711 1.4328 0.7984 0.9666 1.3372 0.6881 1.3779 1.2312 1.0557 0.9762 0.7444 1.9464 1.5601 0.6952 0.5783 0.7587 1.3925 0.8491 1.5547 2.8763 1.9717 1.4351 0.5632 1.074 1.734 0.62 0.74 ng Section Continued) deltaMH -0.8961 -0.0162 -0.6828 -0.7534 -0.4888 -0.1274 -1.5651 -1.0452 -0.1678 -1.2935 -0.7782 -1.0371 -2.4827 -1.5954 -0.8489 1.1234 1.6496 0.6097 0.3601 1.1037 0.6107 0.5292 0.0798 0.8784 0.7076 0.0565 0.6936 0.8543 1.2871 0.6491 0.3845 1.3494 -0.845 0.268 0.195 Race ET S Category C C C C A A A A A A A A A A A A A A A A A A A A A A A A B B B B B B B 84

APPENDIX B B B B B B B B B A A A A A A A A A A A A A A A A A A A A A C C C C C S Category S ET Race Continued) -0.4 0.89 0.404 4.202 0.9211 1.3074 1.3077 0.2438 1.1177 1.4052 0.5054 1.4525 0.8129 2.8114 0.7972 0.6154 0.2654 0.3992 0.0106 0.6433 -0.0303 -1.4761 -0.1597 -0.9393 -0.5139 -2.1906 -1.1332 -0.0107 -1.6944 -0.4898 -2.2743 -0.3121 -0.0735 -1.2965 -1.0362 deltaMH 2.54 1.013 0.842 0.539 1.142 1.1856 0.6757 1.8741 1.0703 0.5733 0.5732 0.9014 1.4914 1.2444 0.6215 0.5499 0.8065 1.6197 0.7076 0.3023 0.1673 1.0046 0.7123 2.0565 0.7696 1.2317 2.6321 0.6847 1.0318 0.8932 1.7362 0.8438 0.9955 1.5542 0.7605 ng Section alphaMH B B B B B B B B A A A A A A A A A A A A A A A A A A A A A A A C C C C S Category S ET Gender -0.795 0.1341 0.1708 0.7184 1.2257 1.3083 0.7677 1.5199 1.0946 0.1184 1.0021 0.1172 0.5892 0.3245 0.2565 0.1322 0.3556 -1.7605 -0.6268 -0.9381 -0.1207 -0.7149 -1.1735 -0.0073 -1.0261 -2.1642 -1.2477 -2.6029 -0.5865 -1.0649 -0.7414 -0.4881 -0.1911 -0.9252 -0.2107 deltaMH

0.871 0.5731 1.6476 2.1152 1.3057 1.0527 0.9299 1.3556 0.7366 0.5936 0.6276 0.9509 1.4906 0.9445 0.7213 1.4026 0.5237 2.5116 3.0271 1.0031 1.5475 1.7006 0.7782 0.9453 1.4825 0.6528 0.9514 1.5732 1.3709 1.2308 0.8966 1.0847 0.8596 1.0938 1.2835

alphaMH Table A6 DIF Analysis Results for Form 1618 Verbal Reasoni A6 DIF Analysis Results for Form 1618 Verbal Table Table A5 DIF Analysis Results for Form 1517 Quantitative Reasoning Section ( Section Reasoning 1517 Quantitative for Form Results Analysis A5 DIF Table 85

APPENDIX Table A7DIFAnalysisResultsforForm1618Grammar/Writi Table A6DIFAnalysisResultsforForm1618Verbal ReasoningSection( alphaMH 0.8645 1.3675 1.3712 0.8208 0.7589 0.8858 1.7432 1.0382 0.8354 0.9566 0.8005 1.6442 0.4426 0.7182 0.6944 1.0858 0.9935 0.4491 0.8722 1.2059 1.5514 1.0677 2.4245 0.9287 1.4005 0.6859 0.4946 0.9624 2.0925 0.2848 0.831 0.702 1.277 0 deltaMH -1.7351 -1.1685 -0.5745 -0.1935 -1.0321 -0.1539 -2.0812 -0.7915 -0.7355 -0.7419 -1.3059 -0.0881 0.3423 0.6484 0.2848 0.4227 0.1043 0.5228 0.4349 1.9156 0.8316 0.7779 0.0153 1.8813 0.3214 0.1739 0.8859 1.6544 2.9515 0.857 0.464 -0.44 0.09 Inf Gender ET S Category C C C C C C C A A A A A A A A A A A A A A A A A A A A A A A A B B B alphaMH 1.2056 1.0909 0.6921 0.8429 1.5252 1.4931 1.2503 2.7435 1.6725 0.4568 0.8378 1.7242 0.8015 2.7824 1.5467 1.7351 0.9102 2.4539 0.9669 1.3079 1.0469 0.9645 1.4882 0.9036 0.8485 1.7591 0.5765 1.4779 1.4702 0.9167 1.1004 ng Section 0.879 0.531 1.021 Continued) deltaMH -0.4394 -0.2044 -0.9921 -2.3717 -1.2087 -1.2802 -2.4048 -1.0248 -2.1096 -0.6308 -0.1077 -0.9342 -1.3273 -0.9057 -0.2248 -0.0489 0.3032 0.8647 0.4018 1.8412 0.4158 1.4873 0.2211 0.0791 0.2383 1.2944 0.2045 -0.942 -0.525 -1.295 -0.918 0.085 0.386 0.52 Race ET S Category C C C C A A A A A A A A A A A A A A A A A A A A A A A B B B B B B B 86

APPENDIX B B B B B B B B B A A A A A A A A A A A A A A A A C C C C C C C C C S Category S ET Race 1.682 1.0446 0.8514 0.5384 1.7952 0.1694 0.6798 0.3119 1.2819 0.1435 0.8985 0.8002 2.2062 0.4649 1.1388 1.2684 -1.0868 -2.8352 -1.2818 -1.2982 -2.2548 -0.8202 -0.1044 -1.0329 -3.6359 -0.9925 -0.2536 -1.6394 -1.2032 -1.5804 -0.1976 -0.3869 -3.0802 -0.7416 deltaMH Continued) ng Section 1.588 1.552 1.114 2.009 0.616 1.371 0.6411 3.3416 1.7254 0.6961 1.7375 2.6104 1.4177 1.0454 0.7952 0.4658 4.6982 1.5255 0.9305 0.7488 0.8757 0.5795 1.6687 0.4888 0.9407 1.9592 0.6823 0.7114 1.0877 1.1789 0.3911 0.8205 3.7089 0.5829 alphaMH B B B B B B B A A A A A A A A A A A A A A A A A A A A A A A A C C C S Category S ET Gender -0.3 0.76 2.778 1.355 1.709 0.709 0.376 -0.708 0.2332 0.6736 0.2818 0.1821 0.9478 0.0958 0.3599 0.4284 1.2469 0.2891 0.4462 1.4964 0.8454 0.6964 0.2999 -1.6348 -0.7351 -1.3938 -1.4999 -0.8706 -0.5837 -0.3061 -0.6544 -0.0269 -1.0116 -1.1628 deltaMH

2.005 0.887 0.858 1.538 0.529 0.6681 0.3066 0.9055 0.7508 1.3673 0.9254 0.5618 1.8096 0.8334 0.5883 0.8271 1.2819 1.1362 0.9601 0.7396 1.8932 1.4484 0.8521 1.3516 0.4832 0.8842 0.6979 1.3211 1.0115 0.7435 0.8802 1.6402 0.7237 1.1391

alphaMH Table A8 DIF Analysis Results for Form 1618 Quantitative Reasoni Table Table A7 DIF Analysis Results for Form 1618 Grammar/Writing Section ( 1618 Grammar/Writing for Form Results Analysis A7 DIF Table 87

APPENDIX Table A8DIFAnalysisResultsforForm1618QuantitativeReasoningSection( 1.5584 0.6753 0.7317 0.9395 2.2032 1.5788 1.4837 0.5786 0.6938 1.0946 1.1463 0.9008 0.7879 2.1021 0.5706 1.1809 0.6777 1.9803 0.3876 1.1408 2.0012 1.1559 1.2447 1.8205 0.8452 1.6295 0.793 0.786 0.727 0.597 0.811 1.578 -1.1474 -1.8563 -1.0731 -0.9271 -0.2125 -0.3209 -1.7459 -0.3907 -1.6057 -0.3096 -1.6303 -0.3405 -0.5145 -1.0719 -1.4079 -1.0426 0.9227 0.7343 0.1466 0.5449 1.2858 0.8591 0.2456 0.5601 0.7492 1.2122 0.4922 1.3185 0.9142 0.3953 0.566 2.227 C C C C C A A A A A A A A A A A A A A A A A A A B B B B B B B B 1.9794 1.3854 1.0215 1.5522 0.7348 1.2276 1.8889 1.3099 1.0064 1.1529 0.5439 0.9635 0.7345 0.6394 0.3558 1.1084 1.0613 0.7123 1.2692 0.8467 0.7625 1.1114 0.6594 0.2171 1.1905 0.6062 0.8968 0.9616 1.2425 0.312 0.452 1.06 -1.6046 -1.0332 -0.4819 -1.4946 -0.6343 -0.3344 -0.1369 -0.2418 -0.1398 -0.5602 -0.2482 -0.4099 -0.5102 0.7242 2.7373 1.4311 0.0874 0.7252 1.0509 2.4283 1.8659 0.7973 0.3911 0.6373 0.9785 3.5889 1.1761 0.2561 0.0919 -0.766 -0.015 -0.05 Continued) C C C C C A A A A A A A A A A A A A A A A A A A A A A B B B B B 88

APPENDIX