Tennessee Growth Measures Planning Document

TN Growth Measures Planning Document

DEVELOPMENT TEAM: PK–Grade 3 DATE: February 11, 2011

INSTRUMENT SUGGESTION (include only one per document): STAR Early Literacy™ and STAR Early Literacy Enterprise

Each Development Team should consider and answer the following questions for EACH instrument they recommend to the TN Department of Education. Completed planning documents and evidence documentation should be emailed to [email protected] .

**If the team is recommending a composite value-added score and/or a school-wide value-added measure, please go directly to number 19.

INTRODUCTION Based on more than 20 years of experience using classroom assessment, Renaissance Learning™ advocates the responsible use of interim assessment for evaluation and instructional improvement. As a result, Renaissance Learning submits STAR Early Literacy and STAR Early Literacy Enterprise for Tennessee to consider as a measure of growth for students in pre-kindergarten through grade 3.

The STAR Early Literacy and STAR Early Literacy Enterprise assessments are computer-adaptive interim assessments that measure early literacy skills and produce immediate results for use by administrators, teachers, students, and parents.

STAR Early Literacy utilizes a proprietary method for measuring student growth called growth norms; growth norms allow for comparisons of student growth across an academic year to students with similar levels of achievement. In addition, the Web-based platform allows for easy roll-up of data to any level of reporting. Furthermore, the Tennessee Department of Education could use the underlying growth norms utilized by STAR Early Literacy to customize the growth expectations for Tennessee schools.

STAR Early Literacy has been reviewed by the technical review committee of the U.S. Department of Education’s National Center on Response to Intervention (NCRTI) and has been given the highest marks for measuring growth and progress monitoring. STAR Early Literacy can be used three to eight times per year as short-cycle, interim assessments, or up to weekly in intervention settings.

The STAR Early Literacy Enterprise assessment is Renaissance Learning’s newest, most powerful version of the STAR Early Literacy assessment. It takes all the strong psychometric and statistical properties of STAR Early Literacy and enhances those capabilities by providing extended testing and reporting capabilities. Coming in the winter of 2011–2012, interim assessment offers expanded skills-based testing that provides more information than ever before and continues to ensure that highly accurate scores are obtained in less than 15 minutes, with only 25 multiple-choice questions. It adds valuable new features such as broader objectives, enhanced reporting, learning progressions, and more robust item banks. STAR Early Literacy Enterprise also includes innovative reporting capabilities. For example, a new report specifies which 1 students need to learn which materials. In addition, an online search engine puts teachers in contact with instructional resources such as teaching activities, prerequisite information, and sample question items.

QUESTIONS FOR ALL DEVELOPMENT TEAMS: 1. Does the instrument provide a valid and reliable academic score that would measure student growth? Provide documentation as evidence. (e.g. Score Achievement Definition/Level(s), Scale Score Scale definition and example, Technical Manual, Technical Research Studies, etc…)

The STAR Early Literacy score scale was specifically designed to follow students’ achievement growth both across an academic year and across contiguous academic years. The score scale was constructed with measuring student progress in mind.

The basis of the STAR Early Literacy score scale is the large item pool with over 2,000 items ranging in difficulty from pre-kindergarten to 3rd grade. Items were calibrated using modern item response theory, with the Rasch model as the basis for calibrating item difficulty. Items were developed using a rigorous methodology, which can be found in the technical manual chapter in the Content and Item Development chapter. Evidence of the alignment of STAR items to the Tennessee standards is provided in the reference material and is comprehensive across all grades. STAR Early Literacy has a good alignment to the Tennessee Curriculum Standards.

After items were developed, they were field tested and calibrated. Rigorous statistical and psychometric criteria were used for item retention into the operational item pool, which are outlined in the technical manual. Descriptions of the sample that were used in the initial calibration phase are provided in the technical manual. Because STAR Early Literacy was developed through monitoring student progress and growth over time, the initial item calibration methodology also included a design to vertically scale the STAR Early Literacy score scale across grades. This was done to provide a continuous score scale across contiguous grades to track student growth in reading comprehension. The vertical scaling methodology is outlined in the technical manual in the Item and Scale Calibration chapter.

Items are also continually updated throughout the academic year with a dynamic calibration process that allows for the field-testing and calibration of new items throughout the academic year. The same rigorous item retention rules are used when updating the item pool yearly. This ensures that only high quality items make it into the operational item bank.

Because of the large range of student reading achievement that can be encountered in the classroom, items were developed with the intention of having items of sufficient ease and difficulty that students of all levels of reading performance could be challenged by. Items range in difficulty from very easy General Readiness items appropriate for young children who are unable to read all the way to reading comprehension and structural analysis items intended for independent readers.

Renaissance Learning undertook three essential steps to improve and enhance the capabilities of STAR Early Literacy; the result is STAR Early Literacy Enterprise. With this innovative software, educators are able to obtain vital skills information for each student, each time he or she takes an 8- to 15-minute test. Here are the three steps we undertook during the development process:

1. We developed learning progressions for early literacy and early reading development. Creating learning progressions involves placing items in a meaningful sequence that accurately represents the pathway to learning more and more difficult— though interrelated—skills. This approach presumes that each skill is part of a continuum and that each skill has prerequisite skills associated with it, as well as skills for which it, itself, is a prerequisite. Our first step in creating each learning progression was to 2 extensively examine state and national standards and other widely-accepted frameworks. We also reviewed and learned from the work of independent educational organizations such as Achieve. Further, as the Common Core State Standards entered into various stages of completion, we carefully monitored them in draft form and provided public commentary. Next, we used this information to craft tentative learning progressions. Finally, we solicited thorough reviews from recognized content-area experts and adjusted the learning progressions based on their input. 2. We calibrated each STAR test item, pinpointing item difficultly in relation to every other item in the corresponding item bank. This step involved extensive field-testing and an adjustment of item questions if they did not assess students as intended. 3. For each of the STAR assessments, we mapped the scaled scores to the learning progressions by equating the scales. As a result, one piece of information (the student’s scaled score) automatically points to a coinciding, or mapped, piece of information (the ability level of the student, including information on which skills the student is proficient in and which skills the student should strive to acquire next).

STAR Early Literacy has been reviewed by the U.S. Department of Education’s National Center on Response to Intervention (NCRTI) and has been found to have the highest of ratings as both a screening measure, which can be used from 3 to 8 times a year for short-cycle interim testing, and for monthly or weekly progress monitoring for students who are believed to be at risk and need more frequent assessments. The Progress Monitoring review by the NCRTI indicated that STAR Early Literacy obtained the highest rating in 10 out of 10 areas of evaluation (please note that at present the NCRTI website does not reflect the recent change of status on the predictive validity of the slope getting the highest mark, but will be updated at the next ratings change for all measures in the Spring of 2011). For instance, STAR Early Literacy obtained top ratings in reliability of the slope, predictive validity of the slope, sensitivity to student improvement, and end of year benchmarks. Additionally, STAR Early Literacy got the highest rating for disaggregated reliability and validity evidence for progress monitoring with breakouts of student race/ethnicity for which STAR Early Literacy was rated highly for all groups and showed similarly high psychometric qualities with all racial/ethnic demographics.

STAR Early Literacy Enterprise will have the same, if not better, properties of the STAR Early Literacy test today. The STAR Early Literacy Enterprise test will be an enhancement of the STAR Early Literacy test in that it will have more items per test for the student to respond to, thus increasing the reliability of scores. In addition, there will be a wider range of important objectives being assessed at all grade levels to align not only with state standards (see the attached document on the alignment with TN standards) but also with the increasing use of standards from the Common Core State Standards. Therefore, STAR Early Literacy Enterprise will have the same scale as the STAR Early Literacy test, but it will have an increase of items administered per test and a wider range of standards-based objectives in the operational item bank.

2. Is the instrument valid and reliable? Provide documentation as evidence for each. (e.g. Technical Manual, Technical Research Studies, Face validity evidence, etc…)

The STAR Early Literacy technical manual provides a chapter on the evidence of score reliability and a chapter related to validity evidence. Split-half, test-retest, and generic reliability estimates are provided for each grade and for all grades. Evidence indicates that overall generic reliability was .92, split-half was .91, and test-retest was .86. Results by grade show a range of generic reliability from .77 in kindergarten to .85 in pre-kindergarten, 1st, 2nd, and 3rd grades; split-half ranging from .75 in kindergarten to .84 in pre-kindergarten; and, test-retest ranging from .63 in kindergarten to .70 in grade 1.

3 STAR Early Literacy also provides Domain and Skill Set scores, which will be described below, on the 7 main domains and 41 skill sets. Reliability evidence, in the form of Split-half estimates, are provided in Tables 32 and 33. All estimates are high both overall and by each grade.

As STAR Early Literacy is a computer-adaptive test using modern item response theory, a student’s individual test scores have a standard error of measurement (SEM) for each administration. Thus each test score has a conditional SEM. Evidence in the technical manual indicates that the average SEM for all grades is about 38 with a standard deviation of 16.

Evidence of validity was also found (see the technical manual) with numerous correlations to external tests and a skills rating measure. The evidence from the relationship between STAR Early Literacy and the skill rating chart supported the ordinal ranking of important early literacy skills, and a correlation of .69 was found between individual test performance and rating on the skills measure completed by the teachers. Additionally, there are over 60 correlations to external measures of early literacy skills, and a meta-analysis of the correlations is provided in the technical manual. The result of the meta- analysis indicated the overall unadjusted correlation to be .60 with a 95% confidence interval ranging from .57 to .62. Meta-analysis at the grade level indicated that the average unadjusted correlations ranged from .56 in kindergarten to .64 in grade 1.

There are additional post-publication research studies reported in the technical manual. These include research done with the correspondence between scores on STAR Early Literacy and running records, Michigan Literacy Progress Profile, Group Reading Assessment and Diagnostic Evaluation (GRADE), Dynamic Indictors of Basic Early Literacy Skills (DIBELS), Texas Primary Reading Inventory 2004-2006 (TPRI), and a predictive validity study using STAR Early Literacy scores to predict student reading achievement two grades into the future. The study looked at using kindergarten scores to predict 2 nd grade reading, 1st grade scores to predict 3rd grade reading, and 2nd grade scores to predict 4th grade reading achievement. All evidence from these studies provides support for the score validity of STAR Early Literacy.

The NCRTI, as stated in #1 above, has also reviewed STAR Early Literacy and provided the highest ratings on issues related to score reliability and validity for progress monitoring. Additionally, evidence of score reliability and validity for different racial/ethnic groups was found indicating that STAR Early Literacy measures are appropriate for diverse student populations.

STAR Early Literacy Enterprise was developed by content knowledge and state standards experts, in conjunction with highly qualified psychometricians. It will be available in the summer of 2011 and is an enhancement and extension of STAR Early Literacy. The enhancement will include broader reporting capabilities, larger item pools of diverse content standards that have evolved over the past few years, and slightly longer tests, which should enhance the already high score reliability. STAR Early Literacy Enterprise will be an extension of STAR Early Literacy and is expected to have as good, or better, evidence of score reliability, as the tests will be longer, and score validity due to increased test length and expanded coverage of objectives. Attached to this document is a copy of the present alignment of STAR to the Tennessee Curriculum Standards, which shows a strong alignment; this alignment will be enhanced over time. In addition, STAR is aligned with the Common Core State Standards.

3. Could the instrument be used for a statewide standardized administration? Provide documentation as evidence. (e.g. Administration Manual, Administration Criteria, Effective Standardized Practices for Administration, etc…)

STAR Early Literacy Enterprise—as well as STAR Early Literacy Renaissance Place, which, like STAR Enterprise, utilizes Renaissance Learning’s hosting services—is ideal for use as a statewide standardized 4 administration. That’s because the computer-adaptive test is administered through a Web-based platform called Renaissance Place Real Time. Renaissance Place Real Time is deployed in a hosted-only configuration from the Renaissance Learning Enterprise-Class Data Center, which contains network servers operated and maintained by the company. Renaissance-provided Web hosting has a number of advantages over local network servers, three of which are cost, reliability, and security. In addition, the centralization of data makes it possible to aggregate data for analysis, as well as to compare students’ scores to those of students nationwide. And finally, the centralization of data allows educators in leadership roles to compare data both over a geographical range and longitudinally over a number of years.

Due to the power of Renaissance Place, all of the schools within a district can share a central database, which can also contain data on other Renaissance products, such as Accelerated Reader™ and Accelerated Math™. With Renaissance Place, teachers and administrators can manage data and create reports from the student level all the way up to the district level. Teachers and administrators can customize reports to provide results for only a specific sub-group of children (e.g., students receiving free or reduced lunch) to allow for disaggregated progress-monitoring reporting.

Renaissance Place data resides on a centralized server in one single database per district. It is accessible to authorized users, password protected, and tailored to each user. A centralized data management system makes it easy to add and maintain student information because the information is imported only once and updated in one central location. It’s also easy to add and maintain other Renaissance programs as they become available. The software is installed one time, and the most up-to-date version is always available to every computer on the system.

At present, Renaissance Learning hosts tens of thousands of schools nationwide, including in Alaska and Hawaii, along with numerous schools in foreign countries such as Canada and the United Kingdom, and Department of Defense schools in numerous countries. The hosted database, in just the 2010-2011 academic year, has over 1 million individual test records for just STAR Early Literacy that have been taken. This does not count the hundreds of thousands of tests that are taken on all Renaissance Learning products, such as STAR Math, STAR Reading, Accelerated Reader, Accelerated Math, Math Facts in a Flash, etc., every day and year. In Tennessee alone, 1,250 schools are active users of at least one Renaissance Learning assessment program. In the present school year, 2010-2011, approximately 266 schools actively using STAR Early Literacy and over 15,000 students in over 100 schools across the state took at least one STAR Early Literacy test on the hosted platform.

Additionally, historical data is always available and easily retrievable for built-in reports, customizable reporting, and extraction for data analysis. Renaissance Learning has the strategic and technical facilities to expand and scale up this data warehousing capability to meet any level of utilization. The capability of the STAR assessments under a hosting arrangement can very easily accommodate an entire state’s worth of data.

The database warehousing capabilities coupled with the computer-adaptive administration and scoring of student tests allows educators and administrators to evaluate test results as soon as the students have finished. Reporting on the student and larger aggregates like the classroom, grade, school, etc. is virtually instantaneous. In addition to providing uniquely real-time results, this capability also provides critical management functionality for centralized monitoring of compliance with testing occasion rules. For instance, administrators can log on to the system and view the extent to which schools or teachers have followed the intended testing regime or evaluate how many of a set of students have been tested at a specific time in the testing window. This capability allows for evaluating the extent to which testing is proceeding across a district or across a state. This can be invaluable when monitoring the compliance

5 with testing rules at a state level, as travel to inspect compliance across the state can be expensive and time consuming.

There are standardized instructions, practice, and training that are used to prepare the students for their first assessment on STAR. All students will get the same instructions and experience with the mouse/keyboard training and practice items. As the assessment is done from the computer, all students are administered each test in the same fashion. Additionally, all students taking the same item are presented with that item in the same manner and with the same audio and visual information. As STAR Early Literacy is a computer-adaptive test, all students across the state are provided with a standard administration of the actual test, as all aspects of the test are delivered by computer and scored by the computer. There would be none of the usual variability seen with individual administration and scoring of tests that could degrade the measurement properties of the tests.

4. Could the instrument be implemented in all classrooms statewide? If yes, what resources would be required? Provide documentation as evidence. (e.g. Technology required or not, Costs per student for administration, scoring, and reporting, Time considerations for administration, collection, and reporting)

STAR Early Literacy and STAR Early Literacy Enterprise can be implemented in all classrooms statewide. The classrooms must have Internet-enabled computers, and the major Internet browser applications are supported for use. Renaissance Learning already hosts tens of thousands of schools across the country and processes millions of test records for all assessment products, so scaling up to serve an entire state would be well within the technical capabilities of the hosted software platform. Utilizing Renaissance Learning’s technical capabilities would allow for an efficient method to meet the needs of an entire state because administration, scoring, reporting, and data warehousing of critical, confidential data would all be done from the hosted software application.

In addition, Renaissance Learning offers a variety of resources to help schools and districts install and effectively use their technology. These resources include program management services, professional development, the Renaissance Training Center™, and technical support. Trained experts are available to help customers, should any difficulties arise, by either talking over the phone or chatting over the Internet.

STAR Early Literacy costs $0.99 per student, per year. STAR Early Literacy Enterprise costs $2.49 per student, per year. The administration of the test takes less than 15 minutes. The software automatically and instantly scores the tests and generates reports. These features make the STAR products extremely practical for implementation in every classroom.

5. Does the instrument measure content that represents essential instructional objectives? Provide documentation as evidence. (e.g. Instrument to TN Curriculum comparison and alignment, Depth of Knowledge reporting and alignment, etc…)

STAR Early Literacy measures instructional objectives that comprise 41 skills in seven key domains: General Readiness, Graphophonemic Knowledge, Phonemic Awareness, Phonics, Comprehension, Structural Awareness, and Vocabulary. It also provides an accurate estimate of oral reading fluency.

The chart below details the 41 skills measured by STAR Early Literacy.

Domain Skill General Readiness

6 Differentiating letters Differentiating words from letters Comparing word length (written) Differentiating shapes Recognizing position words Matching numbers and objects Differentiating word pairs Completing sequences Identifying word boundaries Graphophonemic Knowledge Naming letters Matching upper- and lowercase letters Recognizing letter sounds Recognizing alphabetic sequence Using alphabetical order Phonemic Awareness Blending word parts Blending phonemes Comparing word length (oral) Discriminating beginning and ending sounds Identifying rhyming words Identifying missing sounds Comprehension Reading and understanding words Reading and completing sentences Reading and understanding paragraphs Phonics Replacing beginning and ending consonants Matching sounds within word families Identifying consonant blends Substituting consonant sounds Identifying beginning consonant sounds Identifying consonant digraphs Replacing vowels Identifying medial short vowels Identifying ending consonant sounds Identifying medial long vowels Matching and recognizing short vowel sounds Matching and recognizing long vowel sounds Vocabulary Matching words and pictures Recognizing antonyms Recognizing synonyms Structural Analysis Finding words Building words Identifying compound words 7 STAR Early Literacy Enterprise measures the same instructional objectives as STAR Early Literacy does. In addition, items in the STAR Early Literacy Enterprise test have been placed on a series of learning progressions for reading. The content for STAR Early Literacy Enterprise is based on analysis of professional standards, curriculum materials, test frameworks, and content-area research, including best practices for reading instruction.

6. What scoring metrics are used for this instrument? Provide documentation as evidence. (e.g. Scale Score, Scale, Raw Score, Rubric Score, etc…)

STAR Early Literacy and STAR Early Literacy Enterprise provide a number of criterion-referenced scores. All scores are well described in the technical manual.

The primary score is generated from the responses of the student to the items, which have all been calibrated using the Rasch model, and is transformed to the Scale Score which spans a 300 to 900 unit scale with unit intervals. The Scale Score range was chosen and scaled in this manner to allow for a score to represent about 100 times the age range of normal student early literacy development. For instance, a score of 500 is stated to represent what could be expected of a child that is five years old and a Scale Score of 800 would represent what is typical for an eight year old student.

STAR Early Literacy also provides 7 Domain and 41 Skill Set scores. These scores are computed using the domain scoring approach appropriate for items calibrated on the same scale. The scores represent an estimate of the percent of items the student would be expected to answer correctly if they took all the items in the particular domain or skill set. For instance, a Domain Score in Phonics of 80% would suggest that if a student were given all of the phonics items in the item bank, we would expect the student to be able to answer 80% of them correctly.

STAR Early Literacy also provides a Literacy Classification metric that suggests a student’s level of reading development. Based on the student’s performance on any STAR Early Literacy assessment, the student can be classified into one of three developmental categories: Emergent Reader, Transitional Reader, and Probable Reader. These classifications represent the growing development of students as they move from learning basic early literacy skills like letter-sound correspondence to simple sight word reading to using phonetic rules to understand short or long vowels to independent reading for comprehension.

Additionally, STAR Early Literacy provides and estimated oral reading fluency (Est. ORF) score for grades 1 to 3. The research study and development of this score is provided in the technical manual. The score provides an estimate of the number of words a student might be expected to read correctly within a minute when given a grade level appropriate passage.

7. Does the scoring take into consideration all cognitive levels? Provide documentation as evidence. (e.g. Blooms Taxonomy, Learning Domains, Knowledge Dimensions, etc…)

STAR Early Literacy is a computer-adaptive test that assesses mastery of key literacy skills for emerging readers in grades pre-K–3, English Language Learners (ELLs), students who are struggling, and those with special needs. The assessment provides diagnostic data to help teachers make instructional decisions and help identify likely gaps in knowledge for students experiencing reading difficulties. The scoring of the test takes into consideration the cognitive levels of emergent readers, transitional readers, and probable readers.

8 The above also applies to STAR Early Literacy Enterprise. In addition, the question items of STAR Early Literacy Enterprise have been aligned to learning progressions for reading. These learning progressions take into consideration the cognitive levels of emergent readers, transitional readers, and probable readers.

8. What performance (achievement) levels have been determined for the instrument? Provide documentation as evidence. (e.g. Advanced, Proficient, Basic, Below Basic, Below Proficient, Mastery, etc…)

STAR Early Literacy assigns each student a grade placement, a scaled score, an estimated oral reading fluency, and scores for each of the seven literacy domains. The test also indicates whether the student is an emergent reader, a transitional reader, or a probable reader. STAR Early Literacy Renaissance Place (STAR Early Literacy that is hosted in Renaissance Learning’s Data Center) has advanced reporting features that contain performance levels. For example, the Screening Report graphically depicts how many students fall into each of the following categories: At/Above Benchmark, On Watch, Intervention, and Urgent Intervention. Renaissance Learning provides default cut-off scores for each of these categories, but customers can adjust these cut-offs, if they desire. The report also details which students fall into which categories. Another STAR Early Literacy Renaissance Place report that gives performance levels is the Annual Progress Report. This report graphically shows how a student—or a class—is performing over time. A trend line shows whether the student or class currently has a low risk, has some risk, or is at risk, as well as whether the progress of the student or class is trending upward and likely to put the student or class into a higher performance level in the near future.

STAR Early Literacy Enterprise has all of the reporting capabilities as described above. Moreover, since Renaissance Learning has linked the STAR Enterprise tests with the Tennessee state test (TCAP), we are able to obtain an estimate as to what STAR scores correspond with the cut-offs for the TCAP. Those estimates allow teachers to gain a sense of what achievement level each student is in, just by having them take a STAR test and viewing the results. Moreover, administrators can gain a sense of what achievement levels groups of students are in, broken down by class or grade.

These results appear on various reports. The Screening Report in STAR Enterprise allows schools and districts to create their own performance levels. If they choose, they can create performance levels to match the performance levels on the TCAP. In addition, STAR Early Literacy Enterprise has a new Flexible Grouping Tool, which groups students by their individual needs. Teachers can use this tool to pinpoint which groups of students need to work on which specific literacy skills. Another new report for Enterprise is the Longitudinal Report. This report helps administrators see how groups of students are progressing; the same categories are employed as in the Screening Report.

9. Could an individual student growth score be calculated from the instrument’s score? Provide documentation as evidence. (e.g. Scale Score, Gain Score, rubric score, etc…)

As the primary score, the scaled score for STAR Early Literacy was developed from an explicit vertical scaling methodology; the score scale was constructed with the measurement of growth in mind. The scaled score that represents student achievement is obtained from the computer-adaptive testing methodology that is the heart of STAR Early Literacy. The final score on the test is presented as a transformation of the underlying IRT model and placed on the score scale ranging from 300 to 900. Thus, the scaled score can be used to represent student growth over time as a simple difference score or more elaborately as a time series representation when multiple measurements have been taken across time. At present, the STAR Early Literacy application will provide a regression-based estimate of student growth when the student has at least five completed tests, which is based on a simple ordinary least squares optimization methodology. Furthermore, STAR Early Literacy provides a unique growth norms 9 methodology to allow for comparisons of students’ growth rates to students across the nation that had a similar initial level of performance. Growth norms will be explained more fully in #10 below.

In addition, numerous reports provide visual representations of student progress over time, from graphical visualizations of a student’s scores over time to tabular representations in table form. Moreover, the underlying data warehouse database structure allows for easy roll-up and aggregation of sets of students. This makes it easy to aggregate a classroom, grade, school, or district that utilizes repeated measures on students over time to track the progress of the aggregate units.

10. What measure of growth could be used based on the instrument? Provide documentation as evidence. (e.g. Growth score, TVAAS, norm population, etc…)

STAR Early Literacy utilizes a proprietary method of growth norms for allowing the comparison of student growth rates to students of a similar grade and achievement level based on our national database. At present there are over one million student records that provide the basis for our growth norms estimates. Growth norms are based on simple difference scores using the scaled scores. Thus, in addition to #9 above, Tennessee could construct their own measure of student growth by utilizing the underlying properties of the STAR Early Literacy IRT-based estimate of student achievement through the scaled score, or Tennessee could also potentially profit from utilizing a nationally normed set of expectations of growth. Additionally, Tennessee could utilize the same methodology underlying the growth norms to develop state-specific growth norms and therefore customize the growth expectations for schools within Tennessee.

The growth norms are based on a student’s present grade and level of achievement, which we operationalize as the student’s present standing with respect to our nationally representative norms. At each grade there are 10 different distributions of growth, one for each decile group. The reason for choosing this set of conditions was based on the fact that growth in reading was not only different across different grades, with more growth apparent in the lower grades than upper grades across an entire academic year, but also because of noticeable differences within each grade. Students at different levels of reading comprehension performance within the same grade will show different rates of reading growth. Therefore, the STAR Early Literacy Growth Norms take into account not only differences in student grade but also their initial level of performance. This allows students, when monitored across time on their reading development and growth, to have their observed growth compared to a similar distribution of growth based on a national sample of students in the same grade having a similar initial level of reading achievement. Therefore, a student’s growth can be referenced against an appropriate normative distribution to make relative comparisons about the student’s observed growth. For instance, this makes it possible to identify students who are making less growth than normally observed with students of a similar grade and performance level.

Additionally, the growth norms can be aggregated across larger units of importance. For instance, the growth of an entire classroom, grade, school, or even district can be computed using the aggregation functions in the reporting features. Furthermore, the growth norms are presently used in the Goal- Setting functionality of the software, which allows for reasonable estimates of growth to be used when setting goals for students across the academic year.

11. Could data be collected at more than one point in time? Provide documentation as evidence. (e.g. Pre-Post test design, prior year administration, multiple administrations, etc…)

The STAR Early Literacy and STAR Early Literacy Enterprise assessments allow for multiple administrations per student during the school year. This is possible both because the assessments are efficient—and thus practical in today’s busy classrooms—and because they were designed to be taken 10 at regular intervals to assess each student’s growth. Teachers can use the STAR assessments three to eight times per year for screening and progress monitoring purposes, or as often as weekly in intervention settings. In addition to being constructed to evaluate students over time, numerous reporting features are available to visually represent student progress over time.

As the STAR assessments are computer-adaptive tests, they tend to provide highly reliable scores in a short amount of time, which facilitates multiple administrations with very little disruption of learning time. Additionally, the item selection procedure can optimally choose items at the appropriate difficulty level for students to allow for an individualized assessment for every student based on their current estimated achievement. STAR also has a large item pool of high quality items to ensure students can take tests on a weekly basis for progress monitoring without depleting the pool of items targeted at the student’s performance level.

All of these three things taken together provide for a strong assessment that is built to be used repeatedly across an academic year. Therefore, multiple assessments can be given during the academic year to help provide educators with some idea of how their classrooms or schools or any specific user- defined group are progressing during the academic year. Educators will not have to wait until the end of the year to see how they have done and what types of growth they have achieved with their students. Educators will be able to administer the tests frequently enough that they can obtain information about students’ growth by the mid-point in the school year at minimum. This can provide an early warning system for educators and administrators to take action to correct the present course before the end of the school year when time has been lost and nothing can be done. A common example used in schools across the nation comes out of the Response to Intervention framework where a series of three benchmarking assessments are used for efficient and reliable universal screening of all students, and then students whose performance is potentially at risk can be progress monitored monthly or bi-weekly for a period of time to evaluate their growth. Furthermore, the proprietary growth norms can be used to provide a relative index to interpret the extent to which the student’s growth over that period of time is below expectation or within normal bounds and can be used proactively to set reasonable goals for students based on the normative distribution of known gains made by students of a similar grade and achievement level.

As stated above, the National Center on Response to Intervention (NCRTI) independently reviewed STAR Early Literacy and verified that the assessment can be administered multiple times throughout the year: 3–5 times per year when used as a screening measure and as often as weekly when used as a progress monitoring tool. STAR Early Literacy earned the highest rating of “convincing evidence” in the “Alternate Forms” category, which required that assessments provide at least nine alternate forms that can be administered throughout a single school year with the same student, in which case the student would not receive the same “form” more than once. In the case of STAR Early Literacy, the number of equivalent “forms” or tests that a student would see is virtually unlimited due to the fact that STAR Early Literacy is an IRT-based computer-adaptive assessment with a large item bank whose content spans grades pre-K–3.

Here is more information about the efficiency, practicality, and psychometric feasibility of retesting with STAR:

 STAR assessments are quick. A student can take a STAR test in less than 15 minutes. And, since all students can be tested at the same time, the majority of class time can be used for instruction and practice, not testing. This ensures that academic learning time is maximized and that short-cycle assessment does not result in an abridged curriculum.

11  STAR assessments are easy. Educators find the STAR assessments extremely easy to administer. The tests are short and all students can be tested at once. Plus, students can’t copy one another’s answers, since each student is given different questions. Educators also find STAR reports extremely easy to view, print, and understand. Administrators, teachers, parents, and students love to view the reports that are tailored to their needs. They feel empowered by their increased knowledge of the student’s progress and their ability to make a change and assess its effectiveness.

 Students won’t see the same question twice. The STAR assessments can be repeated often, due to the large number of items in the item banks. STAR Early Literacy can be administered up to weekly without repeating a test item a student has seen within the past month. Moreover, as students’ skill levels increase, the assessment will provide them harder—and thus, different— question items. In sum, STAR tests can be administered frequently—as often as weekly—for progress-monitoring purposes throughout the school year.

Multiple STAR Early Literacy Renaissance Place and STAR Early Literacy Enterprise scores allow educators to see a student’s progress toward literacy goals. After a student has taken one STAR test, teachers can use the Goal-Setting Wizard to set a moderate or ambitious goal for the student. Then, by viewing the Student Progress Monitoring Report, the teacher can see a graphic depiction of the student’s current ability level and the student’s projected ability level, if the student progresses at the goal rate. After the student takes three more STAR tests, the software automatically draws a trend line that graphically shows the student’s rate of progress. By comparing the trend line to the goal line, educators can assess whether the student is on track to meet the goal. If not, the teacher knows that it’s time to intervene (or try a new intervention). The teacher can then use the software to determine whether the new intervention is effectual in helping the student achieve his or her goal. Multiple STAR scores can also be collected and analyzed at the class level. The Annual Progress Report shows multiple STAR scores of the class as a whole. The software automatically draws a trend line that shows the rate of progress of the class and indicates whether the class has a low risk, has some risk, or is at risk for acquiring the requisite literacy skills by the end of the school year.

Finally, one of the most exciting new reports for STAR Enterprise is the Longitudinal Report. This report compiles data from many test administrations over long periods of time, allowing administrators to judge how well students have been performing during, for example, the past three years. This report is especially useful to administrators who are working to ensure teacher effectiveness through research- based evaluative measures.

12. Is the instrument designed for secure administrations? Provide documentation as evidence. (e.g. secure design, secure delivery, etc…)

STAR Early Literacy and STAR Early Literacy Enterprise are designed for secure administrations, since each student is given different test items and thus can’t copy one another’s answers. Additionally, use of the software is password-protected and different stakeholders—students, parents, teachers, and administrators—receive different levels of access to the software.

Additionally, Renaissance Learning is committed to complying with the regulations of the Family Educational Rights and Privacy Act (FERPA). We do not disclose any student data. Any data we receive for the purpose of a research study is destroyed when the study is complete.

For six years, Renaissance Learning has been storing student data on its servers and working with that data for internal research and development purposes in a manner that protects district, school, teacher, and student privacy. Currently, well over one million student quizzes and assessments are administered

12 and stored on our secure servers every day during the school year. We currently host demographic and performance data for more than 15,000 schools, covering millions of students. We employ strict policies and procedures regarding access to, and handling of, school and student data. Consequently, in its entire 25-year history, Renaissance Learning has never had a breach of security or unauthorized use of data.

Finally, Renaissance Learning has implemented the following security measures:

 Security Implementation for Client Access and Data Encryption: The URL address to the hosted Renaissance Place site uses a secure sockets layer (SSL) protocol for all communications over the Internet. SSL is a cryptographic protocol that provides secure communication across the Internet, preventing eavesdropping and tampering and also guaranteeing that data is only being communicated between two authenticated parties, the customer and Renaissance Learning. We use the same SSL communications that are widely trusted by leading financial institutions for electronic commerce and online asset management. By default, each of our client applications also connects using SSL. Encrypting the data with 128-bit encryption ensures that the Internet traffic cannot be intercepted and read using any type of modern sniffing or filtering software.  Control Measures and Processes to Safeguard Confidential Customer Information: Entry to the Renaissance Learning headquarters building, which houses the Renaissance Learning Enterprise-Class Data Center, is controlled via employee magnetic key entry. Entry into the Data Center is also controlled via magnetic key entry for authorized Renaissance Learning hosted and networking personnel only. In addition, we follow strict protocols in logging and analyzing information about who has had access to the Data Center, both in person and electronically. Additionally, a security/risk analysis is completed once a year at minimum by an external auditing firm.  Method by which User Access Credentials Are Provisioned and Managed: The software contains seven defined user groups, including district administrator, district staff, school administrator, school staff, teachers, parents, and students. Each person using the software is assigned to one of these user groups and a specific position within that group. For example, a school administrator may have an assigned position of principal, assistant principal, or reading coordinator. Each person is then assigned capabilities, which give users the right to perform specific tasks in the software. The settings also allow different levels of access to the software setup and data. For example, district administrators have access to reports for all levels—individual students, classes, teachers, schools, and the district—while a teacher will only have access to view reports for his or her individual classes.

13. How are students with disabilities provided equal access through the instrument? Provide documentation as evidence. (e.g. accommodations, modifications, etc…)

STAR Early Literacy is suitable for use with students with disabilities. Since this assessment uses computer adaptive testing, the difficulty of the test is adjusted automatically to reflect the skill level of the student. Therefore, the assessment is likely to present items that correspond to the broad range of abilities of all students, including those with special needs. STAR Early Literacy minimizes frustration for students because the difficulty of questions is adapted based on a student’s response pattern. For example, if a student misses a question, the next question will be easier. If a student answers a question correctly, the difficulty level will increase.

Here’s more specific information about how students with disabilities can use STAR Early Literacy:

 Students with Limited Vision: For students with limited vision, the introductory screens of the STAR Early Literacy assessment respond to the High Contrast accessibility feature within 13 Windows and the Switch to Black and White accessibility feature in MAC OS 10. In addition, the assessment screens within STAR Early Literacy already provide visual contrast through a light background and black writing. Furthermore, STAR Early Literacy is compatible with Mac OS 10’s Zoom In accessibility feature, which allows users to magnify nearly all STAR screens. If students are blind, however, the STAR Early Literacy assessment may not be appropriate.  Students with Hearing Impairments: STAR Early Literacy items feature an auditory component. This component is critical to the assessment of the skill on which the item is based. The assessment is not appropriate for deaf students and most hearing-impaired students. If you believe that a student has sufficient residual hearing to complete the items successfully, the results should be interpreted cautiously in order not to underestimate the early literacy skills of the student.  Students with Limited Motor Skills: STAR Early Literacy offers accommodations for students with disabilities through the accessibility options built into the computer’s operating system. For instance, students with limited motor skills can execute STAR Early Literacy’s mouse-related functions through the keyboard when mouse keys is selected under Accessibility Options within the Windows operations system. The Mouse Keys option is also available for the Mac under System Preferences > Universal Access > Mouse > Mouse Keys. Generally speaking, the STAR Early Literacy assessment is compatible with the adaptive devices typically used by students with limited motor skills. The devices may be part of the operating system or may be add-on devices such as cursor or keyboard controllers.

Federal IDEA regulations allow the use of Response to Intervention (RTI) processes in special education determinations (although RTI is about more than just special education determinations). In addition, the federally funded National Center on Response to Intervention (NCRTI) has given the STAR assessments some of its highest ratings for screening and progress monitoring—meaning that they are among the best assessment tools for RTI purposes, including use with students in special education programs.

Students taking a STAR Early Literacy test can use either the keyboard or the mouse. To select either the keyboard or mouse in STAR Early Literacy, go to Preferences > Testing Options > Accept Answers Using and then choose either Keyboard or Mouse.

STAR Early Literacy Enterprise has all of the capabilities listed above.

14. How are students exposed to the format of the instrument prior to the administration? Provide documentation as evidence. (e.g. Practice assessments, Item samples, Similar instrument integrated into instructional measurement, etc…)

Pretest Instructions Prior to the test session itself, a brief demonstration video introduces STAR Early Literacy to the student. It presents instructions on what to expect, how to use the mouse or keyboard, and how to answer the multiple-choice test questions.

Hands-On Exercise To ensure that every student understands how to use the mouse or keyboard, a short hands-on exercise precedes the assessment. The tutorial tests one of two abilities:

1. the student’s ability to move the mouse pointer to a target, and to click the mouse pointer on the target or 2. the student’s ability to press the correct key on the keyboard to choose his or her answer, and to remember to press Enter to move on to the next question. 14 Students must demonstrate proficiency in using the mouse or keyboard before the test will proceed. A student must correctly respond to three hands-on exercise questions in a row in order to “test out” of the hands-on exercise. To correctly respond to a question, the student must have no more than one incorrect key press or off-target click (not including the Listen button) and must select the target object within five seconds after the audio instructions are through playing. When software detects that the student is having difficulty using the mouse or keyboard, the student will be instructed to ask the teacher for help.

Practice Session After satisfactory completion of the hands-on exercise, a short practice test precedes the assessment itself. As soon as a student has answered three of five practice questions correctly, the program takes the student into the actual STAR Early Literacy test. Even the youngest students should be able to answer the practice questions correctly. If the student has not successfully answered three questions in the first set of five, a second set of five practice questions is presented. Only after the student has passed the practice test does the actual test begin. Otherwise, STAR Early Literacy will halt the testing session and tell the student to ask the teacher for help.

Test The test items were designed to incorporate text, graphics, and audio, as appropriate, to assess the skills in the most straightforward manner possible. STAR Early Literacy test item instructions were written to be explicit, clear, and consistent from item to item so that students are able to test independently.

The same is true of STAR Early Literacy Enterprise.

15. Are there any noted unintended consequences? Provide documentation as evidence. (e.g. Bias, Misinterpretation of Results, Restricting curriculum, fairness, etc…)

STAR assessments are more than just monitoring devices: Incorporated into instructional planning, they play an active role in improving teaching and learning. Based on more than 20 years of experience using classroom assessments in these ways, we advocate the responsible use of short-cycle assessments for two purposes: (1) to provide additional data points in systems to evaluate educators (both teachers and principals) in a fairer and more timely manner, and (2) as sources of regular feedback to educators so they can fine-tune instruction, provide interventions for struggling students, and identify opportunities to improve their own professional growth.

These two uses of interim data—evaluation and instructional improvement—go together and should be mutually supportive. The process will not only identify effective educators, but also enable virtually all educators to become more effective. Because improving student learning is the ultimate goal, the use of interim data for evaluation should always be accompanied by its broader use as part of the instructional process.

In addition, Renaissance Learning follows strict item-writing specifications including bias and fairness criteria that address stereotypes and characterizations of people or events that could be construed as demeaning, patronizing, or otherwise insensitive. Content-development tools track and report attributes such as gender, age, ethnicity, subject matter, and regional references. Individual attributes, as well as the intersection of multiple attributes, are tracked throughout the development process to ensure that final content is demographically balanced and free of bias. Assessment items must also pass strict quality reviews that check for discipline-specific criteria, accuracy, language appropriateness and readability level, bias and fairness, and technical quality control. 15 16. How does the instrument impact instruction? Provide documentation as evidence. (e.g. Differentiate instruction, improve student achievement, direct student interventions, etc…)

The STAR assessments, including STAR Early Literacy and STAR Early Literacy Enterprise, meet the definition of interim assessments as developed and approved by the CCSSO, as well as by Race to the Top. The STAR tests are short-cycle, interim tests designed to provide feedback to adjust ongoing teaching and learning and improve students’ achievement of intended instructional outcomes. The STAR assessments are meant to assist teachers and administrators as they engage in the formative process that takes place daily and throughout the school year, not merely at its end.

The STAR computer-adaptive assessments take students less than 15 minutes to complete. They measure early literacy skills and produce immediate results for use by administrators, teachers, students, and parents. Educators administer the assessments based on their assessment purposes: screening and progress monitoring, as well as—for Enterprise customers—instructional planning, gauging mastery of state standards, and forecasting future performance with enough time to make adjustments, if necessary. The interim assessments are indispensible to schools interested in assessing their students in a formative way, because they allow educators to assess students’ needs in time to make a difference through instructional planning, instructional monitoring, and intervention.

For instance, educators can use STAR Early Literacy Renaissance Place and STAR Early Literacy Enterprise to help them make screening decisions. With the Screening Report, teachers identify which students are reaching benchmark and which students may need intervention. The Screening Report also provides data to help educators decide how to allocate resources accordingly. The default reporting option generates this information at the grade level to help with grade-level planning via data team meetings. Using winter screening data, this report helps educators evaluate the effectiveness of the Tier 1, core program by keeping track of the students not receiving additional services. Using spring screening data, this report also enables data-driven planning for the following school year. Professional development for STAR is grounded in RTI principles, precisely to help educators make decisions formatively throughout the year.

Another valuable tool included with STAR Early Literacy Renaissance Place and STAR Early Literacy Enterprise is the Goal-Setting Wizard, which allows teachers to set an ambitious or moderate goal for each student, based on growth modeling trends from our large database of STAR users. Progress toward that goal is then automatically tracked in the Student Progress Monitoring Report, so educators can see the effect of interventions over time and make adjustments as needed.

For Enterprise customers, the Instructional Planning Reports help teachers and curriculum facilitators group students based on the skills they are ready to learn next, so they can make sure the students are getting regular practice and instruction at the right level and on the right skills. Another useful report is the longitudinal report, which helps administrators learn whether students are on track to meet literacy goals. Finally, the interactive Learning Progressions tool helps teachers find out which prerequisites students need to have.

17. What are the barriers to using the instrument (statutorily, regulatory, etc.)?

No barriers are known at this time.

18. How has the team reached out to other educators in its representative category (grade/subject) for ideas? Please explain the result of that outreach.

16 IF THE DEVELOPMENT TEAM RECOMMENDS EITHER A COMPOSITE VALUE-ADDED MEASURE (claiming of specific tested students) OR A SCHOOL-WIDE VALUE-ADDED MEASURE FOR THE EDUCATOR CATEGORY, THE TEAM MUST PROVIDE THE FOLLOWING INFORMATION (instead of the questions above):

19. Please provide a brief description of the process the development team took that led to the decision to recommend using either a composite value-added measure and/or a school-wide value-added measure for evaluation purposes. Be sure to include: a. The rationale for this choice and why other instruments are not being recommended. b. How the team reached out to other educators in its representative category (grade/subject) for ideas? Please explain the result of that outreach.

NOT APPLICABLE