Formative Assessment Response to the Mesa Unified School District No

ELECTRONIC PROPOSAL

Assessment Technology, Incorporated Formative Assessment Response to the Mesa Unified School District No. 4 Request for Proposal No. 15-47MP

Submitted February 19, 2015 2:00 p.m. Mountain Standard Time

Submitted by: Assessment Technology, Incorporated 6700 E. Speedway Boulevard TM Tucson, Arizona 85710 Phone: 520.323.9033 Fax: 520.323.9139 Copyright © Assessment Technology, Incorporated 2015 All rights reserved. No part of this document may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording, or any information storage and retrieval system, without permission from the publisher. Assessment Technology, Incorporated, Publishers. Tucson, Arizona, U.S.A. Printed in the United States of America. “Galileo” and the Galileo logos are trademarks or registered trademarks of Assessment Technology Incorporated.

Letter of Transmittal

Creating Technology to Promote Learning

6700 E. Speedway Boulevard Tucson, Arizona 85710

1.800.367.4762 Cindi Hostetler 520.323.9033 Procurement Specialist Supervisor fax 520.323.9139

Mesa Unified School District #4 [email protected] 549 N. Stapley Drive, Building 1 ati-online.com Mesa, Arizona 85203

Dear Ms. Hostetler,

This letter of transmittal accompanies (1) original, seven (7) hard copies, and one (1) electronic copy on CD of the response from Assessment Technology, Incorporated (ATI) to the request for proposal (RFP) for Formative Assessment issued by Mesa Unified School District #4, also referred to in District communications as Mesa Public Schools (MPS). ATI understands that MPS is seeking a student assessment and reporting software package that will provide information regarding student progress toward achieving mastery of Arizona’s College and Career Ready Standards. The solution should provide secure district assessments in grades three through eleven several times during the year and also allow teachers to create formative assessments and activities. Reporting must reflect mastery of state standards and enable schools to target needed areas of improvement. The solution must provide specific feedback in areas where students and schools need help to master the state standards. The system must also allow school management the ability to control the quality and administration of each assessment, monitor student progress, and generate key reports that will enable data-driven decision making.

ATI is Arizona’s leading educational technology corporation providing research-based assessment, curriculum, and reporting services for children from birth through high school in Arizona and nationwide. Founded in Arizona in 1986, ATI currently serves more than 500 clients in 35 states nationwide, including more than 300 districts/charters in Arizona. Over the last 20 plus years, ATI leadership has been awarded large scale district/charter, state, and federal contracts that call for services similar to the current MPS initiative. Moreover, Galileo® technology is already implemented as part of numerous statewide initiatives in Arizona. For example, districts/charters implement Galileo technology and assessments as part of the implementation of their literacy plans related to Arizona’s Move On When Reading initiative and as part of their approaches to measuring teacher performance as consistent with Arizona’s recommended model for educator evaluation. As part of these initiatives and others, ATI staff

ATI is proud to be the first company in the nation to provide a comprehensive research- based instructional improvement and instructional effectiveness system. Our flagship product, the Galileo® K-12 Online Instructional Improvement and Instructional Effectiveness System (IIIES), includes a completely integrated comprehensive assessment system aligned to state standards including Arizona’s College and Career Ready Standards and Arizona State Academic Standards in a variety of content areas. Galileo provides the capability to administer multiple types of reliable, valid pre-built and customized district, school, and classroom assessments (e.g., summative, benchmark, interim, formative) in both state-tested (e.g., English language arts, math, science) and non-state tested (e.g., social studies, art, music) content areas. Galileo also provides access to over 131,000 standards-aligned items created by ATI’s Assessment and Instructional Design staff in math, English language arts, science, and writing including selected-response items, constructed-response items, technology-enhanced items, and performance-based tasks. Participation in the Community Assessment and Item Banking (CAIB) initiative also provides access to over 9,000 items written by districts/charters in additional content areas (e.g., social studies, music). All of the items are certified and aligned to standards of the state in which the items were developed. Participating districts may map items to other state or local standards as desired.

Galileo is supported by an extensive ongoing research program. In particular, the effectiveness of Galileo technology has been documented by independent research funded by a federal grant administered by the Massachusetts Department of Elementary and Secondary Education and conducted by the MAGI Services Company as well as numerous data-verified success stories from ATI partner districts and schools in Arizona and nationwide. Galileo’s reliable, valid district-wide assessments will support the implementation of District instructional improvement and instructional effectiveness initiatives in all content areas by providing an Item Response Theory (IRT) based measure of student ability, the Developmental Level (DL) score, along with empirical information about student growth, achievement, standards mastery, and risk of failing to pass the statewide assessment. This actionable information is presented to teachers on the Galileo Teaching Dashboard along with recommendations for student groupings, targeted standards for differentiated instruction/intervention, and links to standards-based instructional materials (i.e., Instructional Dialogs). Administrators are also provided with the School Performance Dashboard that supports the evaluation of student growth and achievement at the school- and district-level. The Instructional Effectiveness Dashboard also can support the District in implementing initiatives designed to maximize instructional effectiveness and consistent with the Arizona state model for educator evaluation. In addition, the Galileo Digital Curriculum

Letter of Transmittal

Assessment Technology, Incorporated Formative Assessment Response to Mesa Unified School District #4 Request for Information #15-47MP

Transmittal Letter ...... i Table of Contents ...... v Executive Summary ...... vii

Tab 1. Statement of Qualifications and Experience ...... 1 Tab 2. Scope ...... 35 Tab 2.1. General Expectations ...... 35 Tab 2.2. RFP Objectives ...... 53 Tab 2.3. Solution ...... 55 Tab 2.4. Training ...... 71 Tab 2.5. Questionnaire ...... 83 Tab 2.6. Formative Assessment Chart ...... 111 Tab 3. Proposal Cost Form ...... 147 Tab 3.1. Proposal Cost Form ...... 147 Tab 3.2. Proposal Cost Form Attachment: ATI Detailed Cost Proposal 149 Tab 3.3. Acknowledgement of Addenda ...... 153 Tab 4. Additional Costs ...... 155 Tab 5. References ...... 157 Tab 6. Contract ...... 163 Tab 7. Required Forms ...... 165 Tab 7.1. Cost Form ...... 165 Tab 7.2. Offer and Acceptance Form ...... 167 Tab 7.3. Non-Collusion Affidavit ...... 169 Tab 8. Appendix ...... 171 Tab 8.1. ATI Electronic Services Agreement ...... 171 Tab 8.2. Exceptions/Deviations ...... 183 Tab 8.3. Galileo® K-12 Online Technical Manual: Benchmark Assessment Development in the Galileo Educational Management System191 Tab 8.4. Composition of a Comprehensive Assessment System ...... 193 Tab 8.5. Item Development and Assessment Construction Guidelines for Common Core State Standards ...... 195 Tab 8.6. Technology Enhanced Items for Formative Assessments in Dialogic Curriculums ...... 197 Tab 8.7. MAGI Services: Galileo Instructional Data System Pilot Project Evaluation ...... 199

Tab 8.8. Tools for Reform: How a Failing District Used Data to Elevate Student Learning ...... 201 Tab 8.9. Confidential/Proprietary Information ...... 203 Tab 8.10. Text References ...... 205

No part of this document may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording, or any information storage and retrieval system, without permission from the publisher. Assessment Technology, Incorporated, Publishers. Tucson, Arizona, U.S.A. Printed in the United States of America.

“Galileo” and the Galileo logos are trademarks or registered trademarks of Assessment Technology Incorporated.

Executive Summary

Assessment Technology, Incorporated (ATI) is a leading provider of services related to the implementation of an instructional improvement and instructional effectiveness system. We have successfully partnered with a wide variety of school districts and charter schools nationwide and are currently partnering with more than 300 districts/charters in Arizona. We also partner with state departments of education, nonprofit organizations, and partner companies. We have decades of experience assisting our partners to use assessment data to guide differentiated instruction with the goals of elevating student achievement, closing the achievement gap, and supporting all students in achieving their potential.

ATI would welcome the opportunity to partner with Mesa Public Schools to implement a complete Formative Assessment solution as described in the RFP. We are confident that our flagship product, the Galileo® K-12 Online IIIES, can best assist the District in achieving District goals by providing the following: • A research-driven instructional improvement system for grades kindergarten through 12 in both state-tested content areas (e.g., math, reading, science, writing) and non-state- tested content areas (e.g., art, foreign languages, physical education) supporting the development of a comprehensive evaluation of student achievement; • Extensive high-quality secure and community (formative) item banks containing more than 131,000 items for grades K-12 in math, English Language Arts, and science developed and continuously expanded by ATI’s experienced Assessment and Instructional Design staff; • A Community Assessment and Item Banking (CAIB) initiative providing access to more than 9,000 items in non-state-tested content areas (e.g., music, art, physical education, technical careers) developed for shared use by districts/charters who have participated in ATI training related to the development of high-quality items and reliable, valid assessments; • An Instructional Effectiveness System providing support for districts implementing initiatives designed to maximize instructional effectiveness including state of the art statistical analyses and reports supporting the evaluation of student growth; • Pre-built and customized district-wide, school-wide, and classroom assessments aligned to Arizona’s College and Career Ready Standards (AZCCRS), other state standards, and the District’s curriculum and including multiple assessment types such as benchmark assessments, pretests/posttests (including those designed to measure educator effectiveness), and formative assessments; • Valid, reliable pre-built and customized district-wide assessments analyzed using psychometric techniques that accurately forecast statewide test performance; • Efficient, flexible methods of administration (e.g., online testing including Computerized Adaptive Testing (CAT) and via Instructional Dialogs integrating assessment and instruction, offline testing, handheld wireless device testing);

• User friendly access for students to assessment, instructional materials, and reports, and for parents, to student information via the Galileo® K-12 Student Parent Center; • Easy customized access to actionable reports for teachers and administrators via several Galileo Dashboards optimized for different task sets; • Research-based measures of student achievement and growth (e.g., IRT derived Developmental Level [DL] scores) provided at the individual student, group, class, school, and district level in graphical reports; • Empirically based recommendations for grouping students and providing differentiated instruction and intervention based on assessment results along with links to targeted instructional materials (i.e., Instructional Dialogs); • A Digital Curriculum Development Platform supporting the construction and implementation of local online standards-aligned curricula including Instructional Units and Dialogs; • A cost-effective annual student license providing access to all ATI item banks and supporting the administration of multiple types of assessments in various content areas and multiple times per year at a single per student cost; • Comprehensive, customized professional development during initial implementation and on an ongoing basis (both web-based and on-site in a district-provided computer lab); and • Standardized, efficient methods for importation (e.g., assessment content, external assessment data) and exportation (i.e. assessment data), that have been successfully implemented with districts using a variety of student information systems and data warehouses.

ATI looks forward to the opportunity to collaborate with Mesa Public Schools to provide your students and staff with the tools they need to succeed today and in the future.

Tab 1. Statement of Qualifications and Experience

Tab 1.1. Outline your firm's qualifications and experience and the qualifications and experience of the key staff members assigned to this project.

Qualifications and Experience of Assessment Technology Incorporated Established in Tucson, Arizona in 1986, ATI is Arizona’s leading educational technology corporation. ATI has over 25 years of experience in conducting work assisting educational programs to use assessment information in educational management, instructional improvement, and instructional effectiveness. Over the last 20 plus years, ATI has been awarded large-scale school district, charter school, and state contracts that call for services similar to the current MPS initiative.

• Brief History of ATI and Galileo® K-12 Online ATI was among the first companies in the nation to offer online testing to school districts and has successfully provided services to a broad diversity of clients varying in location, technology readiness, and size (ranging up to 60,000 students). As of this writing, ATI serves more than 500 Pre-K and K-12 clients in 35 states nationwide. This includes more than 300 K-12 school districts and charters in Arizona. The vast majority of clients have been using our services for multiple years.

ATI’s flagship application, Galileo Online is comprised of Galileo K-12 Online serving students in grades kindergarten through twelve and Galileo Pre-K Online serving children birth through five. Galileo has evolved in response to client needs, changing federal and state requirements, ongoing research in educational assessment and instruction, and advances in technology. Today, ATI provides clients with a complete, comprehensive system including a broad range of online modules designed to promote learning via rapid and flexible access to assessment, curriculum, reporting, instructional, intervention, and instructional effectiveness tools. Galileo technology assists thousands of educators in Arizona and nationwide in meeting the challenges of federal and state legislation, Common Core implementation initiatives, and state accountability requirements by documenting

In addition to working with multiple district and charter partners, ATI has worked and continues to work with a number of organizations with the goal of enhancing the delivery of services designed to support initiatives to elevate student learning. Prime examples are WestEd, Arizona State University, Maricopa County Education Service Agency, Learning Innovations at WestEd, Turning Technologies, Promethean, SMART, and Scantron. ATI works with the state departments of education in Arizona, Colorado, Florida, Georgia, Massachusetts, North Carolina, and Pennsylvania and is approved as a provider of assessment and/or curriculum services in various capacities in these states. ATI has also attended and led a range of meetings with state educational administrators, project partners, and client districts/charters to help these entities design and implement plans to elevate student achievement and improve instructional effectiveness.

• Unparalleled Experience Supporting Arizona Assessment Initiatives As an Arizona corporation based in Tucson, Arizona and serving more than 300 districts and charters in Arizona, ATI has unparalleled prior experience working with Arizona stakeholders to implement successful initiatives related to assessment in early childhood through high school. The Arizona initiatives described in the following demonstrate ATI’s track record in supporting the successful implementation of Arizona statewide initiatives at the district level, including working with Arizona Department of Education (ADE) and districts to develop solutions to challenges related to data management and other factors. This experience will be invaluable to the MPS initiative.

Illustrations of ATI Qualificiations and Experience

• The Massachusetts Department of Elementary and Secondary Education: Galileo Benchmark Assessment Pilot Project .

• The Colorado Department of Education: Galileo Formative Assessment Initiative.

• The Arizona Department of Education, Effective Teachers and Leaders: Galileo Implementation for Measuring Student Growth in State- and Non-State-Tested Subjects.

• Marylou Fulton Teachers College at Arizona State University: Galileo Implementation to Measure Student Growth and Achievement in College and Career Ready Standards (ELA, Math).

• Center for Student Achievement, Arizona Charter Schools Association: Galileo Implementation to Measure Student Achievement and Facilitate Data-Driven Instruction (ELA, Math, Science).

• Maricopa County Educational Services Agency, Rewarding Excellence in Instruction and Leadership (REIL) and Engineering STEM Identity Project: Galileo Implementation to Measure Student Achievement (ELA, Math, Science, Non-State-Tested Subjects).

o Arizona Move On When Reading Arizona districts/charters implement Galileo® technology and assessments as part of the implementation of their literacy plans related to Arizona’s Move On When Reading initiative. ATI and Galileo support districts/charters in successfully meeting the requirements of the initiative in multiple ways. Districts/charters using Galileo K-12 Online can incorporate comprehensive core reading programs and reading intervention programs into the Galileo Digital Curriculum Platform. Galileo K- 12 Online also can be used to meet district/charter goals related to universal screening, diagnostic assessment, progress monitoring, and summative outcome assessment. Galileo provides districts/charters with reliable, valid comprehensive early literacy assessments and formative assessments with accompanying standards-based diagnostic information about student strengths and weaknesses to guide targeted instruction and intervention.

o ATI Research Informing Arizona Policy Makers and Stakeholders Following discussions with the Arizona Governor’s Office of Education Innovation, ATI conducted research to provide Arizona policy makers, school districts, charter schools, parents, families, and other stakeholders with information about the progress of Arizona students of various abilities towards mastery of the Arizona’s College and Career Ready Standards. The results of this research suggested that Arizona’s College and Career Ready Standards represent skills of varying difficulty. Arizona’s higher-performing students are well on their way to mastering the majority of Arizona’s College and Career Ready Standards. In contrast, Arizona’s lower-performing students are in need of additional instruction and intervention to increase mastery of Arizona’s College and Career Ready Standards. Information about which standards students have mastered can be used to differentiate

o ATI Collaboration with Arizona Department of Education Effective Teachers and Leaders Unit Arizona districts/charters implement Galileo® technology and assessments as part of their approaches to measuring teacher performance as consistent with Arizona’s recommended model for educator evaluation (the “State Model”). ATI is currently collaborating with the ADE Effective Teachers and Leaders Unit to help facilitate the Unit’s efforts to guide Arizona districts and charters in implementing the State Model by using Galileo K-12 Online. As a part of the Unit’s activities in supporting implementation of the state model, the Unit has approved the use of Galileo’s valid and reliable Instructional Effectiveness pretest and posttests in state-tested content areas as partial fulfillment of the student performance requirements under the state model. For non-state-tested and content areas, the state model recommends the use of student learning objectives (SLO) to evaluate student growth and enhance student and teacher performance. As a part of its outreach efforts, the Unit is supporting districts and charters in implementing the SLO approach to evaluating student growth in non-state-tested content areas using district-created pretest and posttests created either independently or as part of the ATI CAIB initiative.

o ATI Collaboration with Arizona State University Marylou Fulton Teachers College Arizona State University Marylou Fulton Teachers College (ASU MLFTC), in collaboration with ARIZONA READY-FOR-RIGOR grant-funded partner districts, and ADE, in partnership with Arizona school districts, are developing and implementing a large-scale data depot system, including a teacher tracking system to link student achievement scores to students’ teachers of record/administrator/school/district. As part of this initiative, information from Galileo K-12 Online collected by ASU identified partner school districts in Arizona is exported to ASU for use in the ASU Data Dashboard. The Galileo K-12 Online data provided for this initiative will be used along with other data being gathered by ASU through other sources to help inform:

• decisions by partner district teachers and administrators on classroom instruction, curriculum and programmatic direction; • methodologies used to calculate partner district “value-added” pay-for-performance salary supplements including approaches using student growth percentiles developed in collaboration with Dr. Damian Betebenner; • evaluation of the effectiveness and impact of ASU MLFTC programs and services on student achievement in partner district schools; • decisions about the nature of ASU MLFTC programs and services to be implemented within partner districts;

• the reporting mandates of the U.S. Department of Education and/or its agents (i.e., Mathematica) who have funded this partnership work; and • potential research publications and presentations to be disseminated about the ARIZONA READY-FOR-RIGOR partnership.

o BUILD Arizona Early Grade Success Workgroup and the Read On Arizona Data Integration and System Linkages Task Force Dr. Jason Feld, ATI’s Vice President for Corporate Projects is a member of the BUILD Arizona Early Grade Success Workgroup and the Read On Arizona Data Integration and System Linkages Task Force. The BUILD Arizona Early Grade Success Workgroup is one of five work groups working under the guidance of the BUILD Arizona Steering Committee. The BUILD Arizona Steering Committee and workgroups are creating work plans focused on supporting early grade success. Their overall goal is to reframe early care and education from birth to age eight (0-8) as a critical component of the overall education system and policy framework. The focus of the Early Grade Success Workgroup includes: 1) providing policy, advocacy and awareness support for READ ON ARIZONA – the statewide literacy initiative geared to improving third grade reading proficiency; and 2) to support related policy as it comes forward to build a bridge between the early care and education and kindergarten to grade 12 (K-12) systems. The Read On Arizona Data Integration and System Linkages Task Force is focusing its efforts on guiding collaborations that create strategies for improvement through data integration, data driven decision‐making, research, and evidence- based solutions in support of the mission of Read On Arizona related to a statewide public/private partnership of agencies, foundations and community stakeholders who are committed to a collaborative approach to improving language and literacy outcomes for Arizona’s children from birth to age eight.

o ATI Collaboration with ADE IT Team Supporting AZDash ATI is currently in the process of working with the Information Technology team at the ADE to explore the ways in which Galileo® data can be displayed to local education agencies (LEA) through AZDash - a component of the Arizona Education Learning and Accountability System (AELAS). One of the major goals of this initiative is to further integrate and present data on student progress toward and mastery of standards in ways that facilitate the effective and efficient use of data to increase student achievement.

• Selected ATI Publications and Presentations ATI provides a timely and continuous stream of white papers, technical manuals, research briefs, and supporting documents to assist efforts to promote learning. ATI also provides seminars and webinars on topics relevant to education today in our home state of Arizona and nationwide. Links to selected examples of ATI publications and presentations follow. Additional publications can be found on the Publications page of the ATI website (www.ati-online.com).

Findings and Implications of Recent ATI Research Related to Arizona’s College and Career Ready Standards: Research Summary http://www.ati-online.com/pdfs/researchK12/AZCCRS-ResearchSummary.pdf

Progress by Arizona Students Toward Mastery of Arizona's College and Career Ready Standards: Research Brief http://www.ati-online.com/pdfs/researchK12/MasteryofAZCCRS.pdf

Comparison of Predictive Validity and Forecasting Accuracy for Assessments Aligned to Arizona's College and Career Ready Standards and Older Arizona Standards: Research Brief http://www.ati-online.com/pdfs/researchK12/CCvsAZPredictiveValidityForecastingAccuracy.pdf

Predictive Validity and Forecasting Accuracy for the 2012-13 School Year: Research Brief http://ati-online.com/pdfs/researchK12/PredictiveValidityandForecastingAccuracy2012-13.pdf

A Case Study Evaluation of a District’s Galileo® Assessments in Kindergarten through Third Grade: Research Brief http://www.ati-online.com/pdfs/researchK12/CorrelationsForecastingGalileoKG3Tests.pdf

Integrating Technology and Teaching Strategies to Develop Common Core Curricula Presented by Scott Cunningham (ATI), February 26, 2013. Webinar recording available on the Publications page at: http://www.ati-online.com/galileoK12/K12Research.html Questions and answers available at: http://ati-online.com/pdfs/CCSSWebinarQA.pdf

Building a Research-Based Approach to Support Local Educational Agency Implementation of Frameworks for Measuring Educator Effectiveness Presented by Christine Burnham (ATI) and Jason Feld (ATI) at the annual meeting of the Arizona Educational Research Organization, October 26, 2012, Phoenix, Arizona. Presentation available at: http://www.ati-online.com/pdfs/AERO2012Web.pdf

Implementing a Comprehensive Instructional Effectiveness Assessment System Presented by Dr. Jason Feld (ATI) via webinar, March 26, 2013. Webinar recording available on the Publications page at: http://www.ati-online.com/galileoK12/K12Research.html Questions and answers available at: http://ati-online.com/pdfs/IEWebinarQA.pdf

Common Core Standards: Integrating Local Accountability and a Comprehensive Assessment System Presented by Joe Sassone (WestED), Bob Rosenfeld (WestEd), Liz Jameyson (WestEd), Lynn Spiller (WestEd), Jason Feld (ATI), and Craig Mayhew (ATI), February 29, 2012. Webinar recording available on the Publications page at: http://www.ati-online.com/galileoK12/K12Research.html

Qualifications and Experience of ATI Key Staff Members Assigned to the MPS Initiative As of this writing, ATI employs 50 professional staff, with 88 percent holding advanced degrees and 46 percent possessing an M.S., an M.A., or a Ph.D. ATI also maintains an external Advisory Board comprised of experts in measurement, educational policy, and research approaches to education. ATI departments and team members that will support Galileo® implementation in MPS include:

John R. Bergan, Ph.D. Vice President and Director of Research and Development Brian Benson, M.A. Senior Software Engineer Christine Burnham, Ph.D. Senior Research Scientist Sarah Estrada, Ph.D. Research Scientist Scott Cunningham, B.S. Assessment and Instructional Design Department Director Kerridan Kawecki, B.A. Professional Development Department Director Craig Mayhew, B.S. Field Services Department Director Lucas Schippers, Ph.D. Content Specialist Brandon Smith, MCSE Information Technology Department Director Karyn White, M.A. Educational Management Services Department Director

The ATI team is comprised of staff with expertise in a variety of disciplines related to education, measurement, evaluation, research, software programming, and web design for education. Over the past 25 years, the team has written books, chapters, The ATI Team research articles, and technical reports on a range of topics in education. Selected key ATI team member professional summaries can be viewed by clicking the icon on the left or at the following link: http://www.ati-online.com/ati/ATICompany.html

ATI has history of excellence in working with school districts and charter schools across the nation, ranging in size from 100 to 60,000 students. Illustrations of these collaborative successes can be viewed by clicking the icon on the left or at the following link: http://www.ati-online.com/GalileoK12/K12WhatWorks_Success.html

As indicated previously, MPS project success will be achieved using Galileo innovations and ATI integrated services related to: Professional Development and Training, Information Technology, User Support, Field Services, Educational Management, Assessment and Instructional Design, and

Research and Development. The responsibilities of all ATI staff are focused, in one way or another, on the various aspects of client-focused services to facilitate the achievement of client goals. The ATI team is comprised of staff with expertise in a variety of disciplines related to education, measurement, evaluation, research, software programming, and web design for education. All ATI staff participating in the MPS initiative have extensive experience and success with projects of similar scope and nature. Brief bios and brief resumes/vitae for key ATI leadership staff are presented in the following pages.

ATI Leadership Staff Brief Bios Jason Kane Feld, Ph.D., Vice President Corporate Projects and Proposed MPS Project Manager Jason Feld received a M.A. in Psychology from New York University and his Ph.D. in Educational Psychology at the University of Arizona. His research and professional activities in Pre- K and K-12 education, assessment, policy and practice span 30 years. In his position as ATI Vice President Corporate Projects, Dr. Feld is responsible for supporting ATI-client partnerships designed to enhance student achievement. Dr. Feld plans and monitors both ATI responses to changes in governmental policies and public needs related to education and ATI projects with K-12 and Pre-K policy implications. Dr. Feld establishes and maintains relationships with state departments of education and with federal and corporate organizations responsible for education policy and practice. Dr. Feld is a published author in books, scholarly journals, technical reports, and early childhood journals and has served on editorial advisory boards, foundations and work groups leading Pre-K and K-12 educational initiatives. He has been a speaker at many state and national Pre-K and K-12 conferences across the nation and has presented the research findings of ATI at the National Press Club in Washington, D.C.

John Richard Bergan, Ph.D., President John Bergan received his Ph.D. in Psychology and Education from the University of Michigan. He has over 30 years of research experience in the area of children’s development. Dr. Bergan is the author of several books and over 70 articles in scientific journals in education and psychology. Dr. Bergan's research contributions include studies of children's cognitive and social development and policy research studies on the development of Head Start children and the transition of Head Start children into the elementary school. He is a Fellow of the American Educational Research Association and the recipient of the Palmer O. Johnson Award from the American Educational Research Association, the Distinguished Psychologist Award from the Arizona State Psychological Association and the Presidential Citation from the National Child Care Association. As founder and president of Assessment Technology, Incorporated (ATI), Dr. Bergan establishes and refines the vision of ATI and, with the Research and Development team, designs new technology to assist educators in public schools, Head Start programs and private childcare settings, to accomplish the goals of enhancing students’ learning and children’s development.

John Robert Bergan, Ph.D., Vice President Research and Development John Bergan received his Ph.D. in Psychology from the University of Arizona. His research experience extends nearly 20 years and covers a wide variety of topics within psychology and education. Since 1996, Dr. Bergan has been directing research and developing technology to create electronic assessment tools for children from birth through the 12th grade. Dr. Bergan is the author of several articles in scientific journals, and has delivered several presentations to educational professionals locally and nationally. He is a member of the American Educational Research Association and National Council on Measurement in Education. As Vice President Research and Development, Dr. Bergan is instrumental in designing and developing Assessment Technology, Incorporated innovations focusing on assessment, accountability, and student learning. He also oversees the design and implementation of psychometric and test scaling analysis procedures and of quality assurance testing procedures relevant to all development efforts and data importation.

Brian Benson, M.A., Senior Software Engineer Brian Benson is a primary architect and developer of Galileo® K-12 Online. He has worked for Assessment Technology, Incorporated (ATI) since 1996, first as a consultant then in 1999 as a full- time employee. He started his career in 1992 as a software engineer for Artisoft, Incorporated and then moved on to hold senior level development positions for IKON and CEO Software, Incorporated. Mr. Benson graduated from the University of Arizona in 1992 with a B.S. in Mathematics and a B.S. in Computer Science. He also completed his M.S. in Management Information Systems in 1999 from the University of Arizona. Mr. Benson is dedicated to providing the best solutions with the highest performance to meet all ATI clients’ needs. Performance of Galileo K- 12 Online including ease of use is always in the forefront of any of his development efforts.

Christine Guerrera Burnham, Ph.D., Senior Research Scientist Christine Burnham received her Ph.D. in Cognitive Psychology at the University of Arizona. Her research has focused on language development and language processing, with a particular emphasis on reading and word recognition. She has presented her research at several national conferences and workshops devoted to cognitive processing. Her B.A. in Psychology and Education was earned at William Smith College, where she also acquired teaching certification in both K-6 and special education. Dr. Burnham joined Assessment Technology, Incorporated (ATI) in 2005. She is a member of the American Educational Research Association and the National Council on Measurement in Education. As a senior research scientist at ATI, Dr. Burnham’s current work focuses on Item Response Theory, risk assessment, test equating procedures and intervention analysis related to ongoing research programs with the goal of increasing learning in students from diverse backgrounds.

Sarah Callahan Estrada, Ph.D., Research Scientist Sarah Callahan Estrada received her Ph.D. in Experimental Psychology at the University of California, San Diego based on research investigating language comprehension in English and

Spanish. During her graduate studies, Dr. Estrada was awarded a National Science Foundation Graduate Research Fellowship as well as two National Institutes of Health pre-doctoral fellowships. Dr. Estrada joined Assessment Technology, Incorporated (ATI) in 2010. As a research scientist with ATI, Dr. Estrada conducts research including statistical analysis of assessment data using principles and techniques based in Item Response Theory. Dr. Estrada also collaborates in the development of new approaches to assessment and instructional intervention.

Scott Cunningham, B.S., Assessment and Instructional Design Director Scott Cunningham received his B.S. in Education from the University of Nebraska, and has held teaching certifications in Nebraska, Minnesota, and Arizona. Mr. Cunningham joined Assessment Technology, Incorporated (ATI) in 2002, and draws on his years of experience in the classroom developing interactive, academically challenging lessons and assessments in his current role of supervising development of item specifications for consistent measurement of standards-based learning standards. Mr. Cunningham also trains and supervises a team of content specialists in the creation of item types and items for the online and offline measurement of standards mastery. Additionally, he is leading the team that is developing ATI’s innovative Instructional Dialogs, a model of interactive, technology-based instruction, which involves educators in developing and sharing proven educational strategies through lessons.

Craig Mayhew, B.S., Field Services Director Craig Mayhew graduated from the University of Arizona in 1996 with a B.S. in Regional Development with a minor in Business and Public Administration/Accounting. He began his career in 1997 with CEO Software, Incorporated as an account manager servicing and assisting clientele within his territory, and eventually moving on to supporting larger strategic enterprise accounts which included first level technical support. A member of the Assessment Technology, Incorporated (ATI) team since 2000, Mr. Mayhew is currently Director of Field Services at ATI. In this capacity, he oversees services to current and prospective clients. He manages the Field Services component of the instructional improvement and instructional effectiveness system to ensure timely and responsive service to client needs. In addition, he plays an important role in assisting large client groups with child outcome initiatives.

Lucas Schippers, Ph.D., Content Specialist Lucas Schippers received his Ph.D. in Teaching and Teacher Education at the University of Arizona and his M.Ed. in Educational Leadership from Northern Arizona University. He is a certified secondary education teacher in the state of Arizona. Before joining Assessment Technology, Incorporated in 2009, Dr. Schippers was an instructor and instructional support specialist at the high school level. In his position as Language Arts and Social Studies Content Specialist, Dr. Schippers writes and edits assessment content, including test items and text for grades K-12; creates and edits instructional content, including lessons and units for grades K-12; and analyzes standards from across the nation to determine and respond to client needs. Dr. Schippers is an author and presenter, and was the recipient of the University of Arizona’s Graduate College Fellowship.

Brandon Smith, MCSE, Information Technology Director Brandon Smith earned a B.A. from the University of Arizona. After graduating from college, he served as an Officer in the United States Air Force and worked as a physical scientist for both the Air Force and the US Army Corps of Engineers. During this time, he worked with large-scale databases, Geographic Information Systems (GIS), and developed an electronic tracking system for chemicals used at a government shipyard. Mr. Smith is a Microsoft Certified Systems Engineer and Microsoft Certified Professional. He has over a decade of experience handling various information technology and multimedia services. Mr. Smith joined Assessment Technology, Incorporated (ATI) in 2001. As Director of the Information Technology Department at ATI, Mr. Smith oversees the development and maintenance of the entire ATI technology infrastructure. His department is responsible for the maintenance of ATI’s internal network as well as the web infrastructure which enables ATI to offer online services. Additionally, he is responsible for designing and supervising the maintenance of ATI databases. Mr. Smith oversees the provision of timely and responsive staff and client technical support services.

Kerridan Kawecki, B.A., Professional Development Director Kerridan Kawecki graduated Magna Cum Laude from the University of Arizona with a B.A. in Secondary Education. She is past Chairman of the Canyon del Oro Technology Subcommittee for the North Central Accreditation Team. A member of the Assessment Technology, Incorporated (ATI) team since 2000, Ms. Kawecki is well versed in educational theory and practice. She is sensitive to the everyday challenges facing educators and includes innovative and practical solutions in the creation of professional development content. Ms. Kawecki is also very sensitive to the importance of using assessment data to inform instruction and provides extensive information for clients on using the information provided in ATI reports to enhance student achievement. She has designed online distance learning courses on Galileo® technology and directs the delivery of on-site and online professional development for both administrators and teachers. Ms. Kawecki has personally delivered hundreds of professional development sessions for ATI clients. In addition, Ms. Kawecki is involved in research focused on evaluating the impact of the Galileo IIIES on learning. Recently, she managed a research project involving ATI’s computer-based literacy curriculum in Head Start programs.

Karyn White, M.A., Educational Management Services Director Karyn White earned her M.A. in Curriculum and Technology from the University of Phoenix and her Post Baccalaureate Certification in Elementary Education and B.A. in Psychology at the University of Arizona. A member of the Assessment Technology, Incorporated (ATI) team since 2005, Ms. White is Director of the Educational Management Services Department and draws from her various experiences in the classroom and in curriculum development when assisting in the development of ATI’s integrated instructional improvement and instructional effectiveness system. Ms. White’s department assists school districts to utilize ATI’s full range of assessments, curriculum, and interventions to meet district educational goals. She serves as a liaison between district

ATI Leadership Staff Brief Resumes/Vitae Brief resumes/vitae for key ATI leadership staff are presented in the following pages.

PROFESSIONAL BACKGROUND B.A. (1977) City University of New York, Queens College: Psychology/Sociology M.A. (1979) New York University Graduate School of Arts and Sciences: Psychology Ph.D. (1986) University of Arizona: Educational Psychology

SELECTED AWARD HIGHLIGHTS Psi Chi National Honor Society in Psychology Elizabeth Kroeber Memorial Award in Biological Research National Child Care Association Presidential Citation of Achievement Navajo Nation Recognition of Achievement in Research and Training Texas Licensed Child Care Association Recognition of Service Arizona Child Care Association Award of Recognition Dean’s Certificate of Recognition College of Education, University of Arizona Navajo Nation Educational Transition Project Award of Recognition National Head Start Association Award of Contribution to Education

HISTORY OF PROFESSIONAL POSITIONS 2012 – present Member. Build Arizona – Helios Education Foundation Early Grade Success Workgroup 2012 – present Member. Virginia G. Piper Charitable Trust - Read on Arizona Data Integration and System Linkages Task Force 2004 – present Member. Editorial Advisory Board for the National Head Start Association Dialogue: A Research-to-Practice Journal for the Early Intervention Field. 2002 – present Member. North Carolina Head Start Association Research Committee 2001 – present Member. Florida Head Start Association Research Committee 2001 – present Adjunct Professor. Pima Community College 1992 - present Co-Developer. National Child Care Association Certified Child Care Professional Credential 1987 - present Vice President and Director. Assessment Technology Inc. 2003 – 2004 Member. Sub-committee on Learning and Development Standards and Assessment for the Assessment for the Texas Program for Society and Health (TPSH) of the James A. Baker III Institute for Public Policy and the Texas Early Childhood Education Coalition. 1998 - 1999 Director. Board of Directors, Southern California Head Start Education Consortium. 1998 – 1999 Advisor. National Head Start Association Reading Readiness Forum 1997 – 2003 ATI Coordinator. Ohio Indicators of Success Project for Measuring Outcomes 1996 - 2006 Director. Board of Directors, Childtime/Tutortime Children’s Centers, Inc (CTIM) 1996 – 1999 Advisory Board Member. Head Start Region IX Southern Cluster Advisory Board. 1994 - 1997 University of Arizona Steering Committee Member. Office of Child Care Initiatives. 1992 - 2002 National Commissioner. The National Early Childhood Program Accreditation Council. 1991 - 1998 Evaluation Coordinator. The Navajo Nation Head Start Transition Demonstration Project. 1991 - 1998 Principal Research Specialist. The University of Arizona, College of Education. 1991 - 1993 Project Director. The NCCA. Institute for Child Care Research and Education. 1989 - 1990 Co-Principal Investigator. The Transition of Head Start Children to Public School. Friends of Children, Jackson Mississippi. 1989 - 1990 Co-Principal Investigator. National Transition Study. Ford Foundation. 1988 - 1991 Adjunct Assistant Professor. University of Arizona, Department of Educational Psychology. 1984 - 1985 Consultant. Southern Arizona Mental Health Center, Tucson Unified School District. 1983 - 1988 Senior Research Assistant. University of Arizona, Department of Educational Psychology. 1981 - 1983 Graduate Research Associate. The University of Arizona, Educational Psychology. 1976 - 1979 Clinical Research Assistant. Downstate Medical Center, Clinical Behavior Research Unit.

Jason Kane Feld, Ph.D. – BRIEF VITA – Page 2

SELECTED PRESENTATIONS 2014 Presenter. Assessment Technology Incorporated and Queen Creek Unified School District with the What Works Panel Arizona Charter Schools Association, Gilbert Public Schools, Queen Creek Unified School District, and Scottsdale Unified School District, Ready for Success in Implementing the Arizona's College and Career Ready Standards Seminar. Queen Creek, AZ 2014 Presenter. Assessment Technology Incorporated Research and Innovations in Technology and Educational Practices: Demonstrating Galileo® Pre-K and K-12 Growth Models to Assess and Enhance School Readiness Webinar. Tucson, AZ 2013 Presenter. Assessment Technology Incorporated and Creighton School District with the What Works Panel, Alhambra Elementary School District, Buckeye Union High School District, Casa Grande Elementary School District, Creighton School District, Maryvale Preparatory Academy, and Queen Creek Unified School District, Building a Successful Local Educator Effectiveness Initiative with Technology and Research Seminar. Phoenix, AZ 2013 Presenter. Assessment Technology Incorporated. Measuring Instructional Efficiency and Effectiveness: the Impact of Research and Technology on Local Initiatives. 26th Annual AERO Conference - The Power of Education Research for Innovation in Practice and Policy. Tempe, AZ. 2013 Presenter. Assessment Technology Incorporated Galileo Innovations for Common Core Implementation: Implementing a Comprehensive Instructional Effectiveness Assessment System Webinar. Tucson, AZ 2012 Presenter. Assessment Technology Incorporated, WestEd: SchoolsMovingUp, and Creighton School District Webinar. Common Core Standards: Integrating Local Accountability and a Comprehensive Assessment System. Tucson, AZ 2010 Presenter. Keynote. Florida Head Start Association 13th Annual Spring Training Institute. Listen When the Data Speak: Head Start Makes a Measurable Difference. Daytona Beach, FL 2010 Presenter. Research Session by Assessment Technology Incorporated. Florida Head Start Association Pre-Conference Research Session. Research as a Reality Check on Intervention Investments for the FHSA Research Group. Daytona Beach, FL 2010 Presenter. Assessment Technology Incorporated Learn More About Implementing an Instructional Improvement System and Providing Professional Development on the Use of Data to Meet Local Needs and RTTT/State Requirements: Demonstrating ATI’s Galileo K-12 Online Instructional Improvement System and WestEd’s Curriculum, Assessment and Alignment Services Webinar. Tucson, AZ 2010 Moderator. Assessment Technology Incorporated Elevating Student Achievement Seminar: Exploring What Works Seminar. Denver, CO 2009 Co-Presenter. Keynote. 22nd Annual Arizona Educational Research Organization Conference. Determining What Works One Day at a Time. Phoenix, AZ 2009 Presenter. Assessment Technology Incorporated Interventions Forum. Dynamic Intervention Systems. Sacramento, CA 2009 Presenter. Assessment Technology Incorporated Interventions Forum. Research as a Reality Check on Intervention Investments. Sacramento, CA 2008 Co-Presenter. Assessment Technology Incorporated and WestEd. California School Board Association’s Annual Education Conference and Trade Show. Innovative Technology Supporting School Interventions. San Diego, CA 2008 Co-Presenter. Assessment Technology Incorporated and WestEd. Texas Assessment Conference. Dynamic Interventions Systems: New Directions Supporting Student Learning. Austin, TX 2008 Co-Presenter. Assessment Technology Incorporated and WestEd. Innovative Learning Conference. Innovative Technology Supporting a School Intervention System Helping to Achieve Success for All Students. San Jose, CA

ACADEMIC BACKGROUND Degree Institution Major A.B. Kalamazoo College Psychology and Music M.A. Wayne State University Clinical Psychology Ph.D. University of Michigan Education and Psychology

SELECTED PROFESSIONAL POSITIONS Position Organization Period President and Board Assessment Technology Inc. 1986-present Chairman Professor Emeritus University of Arizona 1995-present Professor University of Arizona 1971-1995 Associate Professor University of Arizona 1966-1971 Assistant Professor Kansas University 1963-1966

SELECTED ACADEMIC AND PROFESSIONAL AWARDS Organization Award University of Michigan Phi Kappa Phi Honorary Society American Educational Research Association Palmer O. Johnson Memorial Award for Outstanding Research Arizona State Psychological Association Distinguished Psychologist Award American Board of Professional Psychology Diplomat in School Psychology National Child Care Association Presidential Citation

MEMBERSHIPS IN PROFESSIONAL ORGANIZATIONS Fellow, American Educational Research Association Fellow, American Psychological Association, Divisions 15 and 16 Member, Society for Research in Child Development Member, National Council on Measurement in Education Member, Psychometric Society

SELECTED EXAMPLES OF PROFESSIONAL SERVICE TO NATIONAL ORGANIZATIONS AND GOVERNMENTAL AGENCIES Service Organization Period Member, Committee to Review Research United States Department of 2002 Proposals on Early Childhood Curriculum Education Effectiveness Member, Joint Committee on Testing Practices American Educational Research 1985 Association Monitor for Scholarly Knowledge American Psychological 1980-1983 Association: Division 16 Chair: Lightner Witmer Award American Psychological 1979-1980 Association: Division 16

John Richard Bergan, Ph.D. – BRIEF VITA – Page 2

SELECTED PUBLICATIONS

Bergan, J. R., Bergan, J. R., & Guerrera Burnham, C. (2006). Benchmark Assessment in Standards-Based Education: The Galileo® K-12 Online Educational Management System. Tucson: Assessment Technology Incorporated. Bergan, J. R., & Bergan, J. R. (2005). Forecasting Statewide Test Performance and Adequate Yearly Progress from District Assessments. Tucson: Assessment Technology, Incorporated. Bergan, J. R., & Bergan, J. R. (2005). Standards Mastery Determined by Benchmark and Statewide Test Performance. Tucson: Assessment Technology, Incorporated. Bergan, J. R., Bergan, J. R., Rattee, M., Feld, J.K., Smith, K., Cunningham, K., Linne, K. (2002). The Galileo System for the Electronic Management of Learning. Tucson: Assessment Technology Incorporated. Bergan, J.R., Schwarz, R. D., & Reddy, L. A. (1999). Latent structure analysis of classification errors in screening and clinical diagnosis: An alternative to classification analysis. Applied Psychological Measurement. 69-86. Bergan, J. R., Sladeczek, I. E., Schwarz, R. D., & Smith, A. N. (1991). Effects of a measurement and planning system on kindergartner's cognitive development and educational programming. American Educational Research Journal, 28 (3), 683-714. Bergan, J. R., & Kratochwill, T. R. (1990). Behavioral Consultation and Therapy. New York: Plenum Publishing Corporation. Bergan, J. R. (1988). Latent variable techniques for measuring development. In R. Langeheine (Ed.), Latent Trait and Latent Class Models. New York: Plenum Publishing Corp. Bergan, J. R., Stone, C. A., & Feld, J. K. (1985). Path-referenced evaluation of individual differences. In C. R. Reynolds & V. L. Willson (Eds.), Methodological and Statistical Advances in the Study of Individual Differences. New York: Plenum Publishing Corporation. Bergan, J. R. (1983). Latent class models in educational research. In Review of Research in Education, Washington, D.C.: American Educational Research Association. Bergan, J. R., & Stone, C. A. (1985). Latent-class models for knowledge domains. Psychological Bulletin, 98, 166-184. Bergan, J. R., Stone, C. A., & Feld, J. K. (1984). Rule replacement in the development of basic number skills. Journal of Educational Psychology, 76, 289-299. Bergan, J. R., & Henderson, R. W. (1979). Child Development. Columbus, Ohio: Charles E. Merrill Publishing Company.

EDUCATION Bachelor of Science with distinction, University of Arizona, 1987 Master of Arts, University of Arizona, 1992 Ph.D. University of Arizona, 1995

ACADEMIC AND PROFESSIONAL AWARDS University of Arizona Phi Beta Kappa University of Arizona Deans List University of Arizona Honors Program

PROFESSIONAL EXPERIENCE Vice President Research and Development and Director, Assessment Technology, Inc. 1996 to Present Post Doctoral Fellow, Group Health Cooperative, 1993-1995 Clinical Psychology Intern, University of Washington, 1992-1993

MEMBERSHIPS IN PROFESSIONAL ORGANIZATIONS Member, National Council on Measurement in Education Member, American Psychological Association

SELECTED PRESENTATIONS OCCURRING OVER LAST 12 MONTHS 2009 Presenter. Assessment Technology Incorporated’s Interventions Forum. Dynamic Intervention Systems. Arlington, TX. 2009 Presenter. Assessment Technology Incorporated’s Interventions Forum. Designing an Intervention. Arlington, TX.

PUBLICATIONS Bergan, J. R., Bergan, J. R., & Guerrera Burnham, C. (2006). Benchmark Assessment in Standards- Based Education: The Galileo® K-12 Online Educational Management System. Tucson: Assessment Technology Incorporated. Bergan, J. R., & Bergan, J. R. (2005). Forecasting Statewide Test Performance and Adequate Yearly Progress from District Assessments. Tucson: Assessment Technology, Incorporated. Bergan, J. R., & Bergan, J. R. (2005). Standards Mastery Determined by Benchmark and Statewide Test Performance. Tucson: Assessment Technology, Incorporated. Bergan, J. R., Bergan, J. R., Rattee, M., Feld, J.K., Smith, K., Cunningham, K., Linne, K. (2002). The Galileo System for the Electronic Management of Learning. Tucson: Assessment Technology Incorporated. Strosahl, K., Hayes, S., Bergan, J., & Romano P. (1998). Assessing the Field Effectiveness of Acceptance and Commitment Therapy: An Example of the Manipulated Training Research Method Behavior Therapy, 29(1). Bergan, J. (1995). Behavioral Training and the New Mental Health: Are We Learning What We Need to Know? The Behavior Therapist, 18(9). Burgoon, J., Beutler, L., Le Poire, B., Engle, D., Bergan, J., Salvio, M., & Mohr, D. (1993). Nonverbal Indices of Arousal in Group Psychotherapy Psychotherapy, 30(4).

John Robert Bergan, Ph.D. – BRIEF VITA – Page 2 Burgoon, J., Le Poire, B., Beutler, L., Bergan, J., & Engle, D. (1992). Nonverbal Behaviors as Indices of Arousal: Extension to the Psychotherapy Context. Journal of Nonverbal Behavior, 16(3). Beutler, L., & Bergan, J. (1991). Value Change in Counseling and Psychotherapy: A Search for Scientific Credibility. Journal of Counseling Psychology, 38(1). Beutler, L., Engle, D., Mohr, D., Daldrup, R., Bergan, J., Meredith, K., & Merry, W. (1991). Predictors of Differential Response to Cognitive, Experiential, and Self-Directed Psychotherapeutic Procedures. Journal of consulting and Clinical Psychology, 59(2). Mohr, D., Beutler, L., Engle, D., Shoham-Salomon, V., Bergan, J., Kaszniak, A., & Yost, E. (1990). Journal of Consulting and Clinical Psychology, 58(5). Beutler, L., Clarkin, J., Crago, M., & Bergan, J. (1990). Client-therapist matching In Beutler, L. & Clarkin, In Systematic treatment selection: Toward targeted therapeutic interventions, New York: Brunner/Mazel.

EXPERIENCE 09/96 - present Assessment Technology, Incorporated Tucson AZ Senior Software Engineer Product: Galileo® Online This product is a web application for assessing child outcomes. Major responsibilities were choosing the implementation tools and technologies, communicating those decisions to other team members, designing and implementing the application. The application was written using a combination of Visual Basic, ASP, and Visual Studio .NET. The database is SQL Server and is accessed using ADO.

09/97 – 11/99 CEO Software, Incorporated Senior Development Engineer Products: SCHEDULER PLUS 2001CEO Software’s flagship product allows facility managers and meeting planners to schedule, modify and track all aspects of meetings or events. I lead a team of four other engineers to the successful completion of this application. Major responsibilities were choosing the implementation tools and technologies, designing the application, and implementing the design. The application is a standard 3-tier implementation for 32-bit Window platforms. The client and business tiers were written using Visual Basic and the databases the business tier interfaced with through ActiveX Data Objects (ADO) were Access, SQL Server, and Oracle. Reporting was done through Crystal Reports. DAISY (Daily Activity Information System)This product reports on event information from SCHEDULER PLUS 2001 for use with Web browsers by creating HTML files on the fly. Major responsibilities were designing and implementing all aspects of the application.

03/97 – 9/97 CEO Software, Incorporated Software Engineer Product: Net Folders This product is a Microsoft Outlook 97 extension. It allows Outlook clients to share their folders with any other clients independent of the server. Major responsibilities were developing the user interface for publishing a folder and developing the user interface for managing a shared folder. Also integrated in Office Assistant help.

06/92 - 03/97 Artisoft, Incorporated Tucson, AZ. Software Engineer Products: XtraMail This product is a POP3/SMTP mail server for Windows 95 or Windows NT. Major responsibilities were designing and developing the message store and the interface to the message store using Access and Data Access Objects (DAO). LANtastic 7.0 This product is a peer to peer network operating system that connects PCs running Windows 95, Windows 3.1 or DOS. Major responsibilities were designing and developing the user interface for managing print jobs and network resources. Also developed the Windows 95 chat applet under the NETBIOS protocol and the Windows 3.1 modem sharing user interface. Artisoft Exchange This product is a full feature E-mail application for Windows 3.1. Major responsibilities were designing and developing the filtering (rules) engine for automatic management of messages as well as the E-mail editor. Also designed and developed the user interface for the fax, MCI, text pager, MHS, and remote gateways.

SKILLS Extensive commercial experience with Visual Studio .NET, ASP, Visual Basic, ADO, HTML, C, C++, Windows SDK, MFC, Winsock, ODBC, COM, DCOM, MAPI, POP3, SMTP, Crystal Reports.

TOOLS Visual Studio .NET, Visual Basic, Visual C++, Visual Interdev, SQL Server, MS Access, Oracle, Crystal Reports, IIS Server

Brian C. Benson, M.S. – RESUME – Page 2

EDUCATION Master of Science, University of Arizona, May 1999

MAJOR Management Information Systems Thesis: Linear Time Procedure For Two Job Flowshop Foundations of Computing and Decision Sciences. Vol. 23 1998 Brian Benson and Moshe Dror Bachelor of Science, University of Arizona, May 1992

MAJOR Computer Science Bachelor of Science, University of Arizona, May 1992

EDUCATION Ph.D., Psychology, University of Arizona. December, 2004 Dissertation: Flexibility and Constraint in Lexical Access: Explorations in Transposed Letter Priming Advisor: Dr. Kenneth Forster

Master of Arts, Psychology, University of Arizona. June, 2001 Thesis: Re-Examining Lexical Ambiguity Advisor: Dr. Kenneth Forster

Bachelor of Arts, Psychology, Hobart and William Smith Colleges, June, 1987 With a minor in Education.

MEMBERSHIP IN PROFESSIONAL ORGANIZATIONS Member, American Educational Research Association Member, American Psychological Association Member, National Council on Measurement in Education Member, Society for the Scientific Study of Reading

PROFESSIONAL EXPERIENCE Senior Research Scientist, Assessment Technology, Inc.: 2005–Present Research Assistant, University of Arizona: 1998-2005 Teaching Assistant, University of Arizona: 1998-2005 Research Assistant, Hunter College, New York, NY: 1996-1998 Legal Assistant, Schechter, Brucker, & Pavane, P.C., New York, NY: 1990-1998

AREAS OF SPECIALIZATION • Item response theory, risk assessment, test equating and intervention analysis in K-12 education. • Lexical access in reading, including differential effects of within-word letter positions, lexical ambiguity, parafoveal processing, neighborhood density effects and the interactions between orthography and phonology.

TEACHING EXPERIENCE Primary Instructor of Undergraduate Courses, University of Arizona • Cognitive Psychology • Cognitive Development • Research Methods Lab • Research Methods

NYS Teaching Certification in K-6 and Special Education, Hobart and William Smith Colleges, 1987

Christine Guerrera Burnham, Ph.D. – BRIEF VITA – Page 2

PUBLICATIONS Burnham, C.G. (2009). Seeing both the forest and the trees: Measuring progress, forecasting, and intervention. Tucson: Assessment Technology, Incorporated. Forster, K.I., Guerrera, C., & Elliot, L. (2009). The maze task: Measuring forced incremental sentence processing time. Behavior Research Methods, 41, 163-171. Guerrera, C. and Forster, K.I. (2007). Masked form priming with extreme transposition. Language and Cognitive Processes, 23, 117-142. Bergan, J.R., Guerrera, C., and Bergan, J.R.(2007). Assessing student risk of net meeting Massachusetts state standards. Tucson: Assessment Technology Incorporated. Bergan, J.R., Bergan, J.R., & Guerrera-Burnham, C. (2006). Benchmark assessment in standards-based education: The Galileo® K-12 Online Educational Management System. Tucson: Assessment Technology Incorporated.

POSTERS AND PRESENTATIONS Burnham, C.G. (2009). Seeing both the forest and the trees: Measuring progress, forecasting, and intervention. Presentation made at the Assessment Technology Incorporated Interventions Forum, Colorado Springs, CO. Guerrera-Mahoney, C. Forster, K.I., and Nicol, J. (2002). Does semantic status affect the syntactic processing of ambiguous words? Presentation made at the Third Pre-Psychonomics Lexical Access Workshop, Tucson, AZ. Guerrera-Mahoney, C., Forster, K.I., and Nicol, J. (2002). Do semantics affect the syntactic processing of ambiguous words? Poster presented at the 15th annual CUNY Sentence Processing Conference, New York, NY. Guerrera-Mahoney, C. (1999). Priming and access to syntactic information. Presentation made at the Second Pre-Psychonomics Lexical Access Workshop, Tucson, AZ. Laczko-Kerr, I. and Guerrera Burnham, C. (2011). The missing link in school improvement: Evaluating teachers’ formative use of district assessment data. Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA.

CURRENT POSITION: Research Scientist, Assessment Technology, Inc., 2010-present • Performing statistical analysis of assessment data using principles and techniques based in Item Response Theory (IRT) • Collaborating in the development of new approaches to assessment and instructional intervention

EDUCATION University of California, San Diego, CA (UCSD) Ph.D., Experimental Psychology, 2009, GPA: 4.0/4.0 Thesis: A multi-methodological investigation of the processing and interpretation of coordinate sentences involving verb phrase anaphors.

M.A., Experimental Psychology, 2004, GPA: 4.0/4.0 Thesis: All pronouns are not created equal: On-line anaphoric processing in bilingual speakers of Spanish and English

Awards: NSF Graduate Research Fellowship, NIH Pre-doctoral fellowship, Dean’s Scholarship

Cornell University, Ithaca, NY B.A., Psychology, 2001, GPA: 3.9/4.3

Awards: Phi Beta Kappa, National Merit Scholarship, Dean’s Scholarship, Cornell Tradition Fellowship, Dean’s List

RESEARCH EXPERIENCE Post-doctoral Scholar, UCSD, Language and Cognitive Neuroscience Laboratory, April 2009-December 2009 • Expanded my thesis research on referential linguistic forms and extended it to language disordered populations including stroke sufferers and children with atypical language development • Trained and supervised students in experimental design as well as data collection, processing, and analysis

Graduate Student Researcher, UCSD, Language and Cognitive Neuroscience Laboratory, 2003-2009 • Utilized a multi-methodological approach to obtain converging and complementary results regarding the unconscious cognitive and neural processes that underlie the comprehension of referential linguistic forms in English and Spanish • Designed, programmed, and analyzed experiments using behavioral, eye-tracking, and electrophysiological techniques • Collected electrophysiological, eye-movement, reaction time, and questionnaire data • Communicated findings to expert audiences in published peer-reviewed manuscripts and at conferences, as well as to general audiences in colloquia and courses • Expertise in stimuli generation, presentation, and data processing (Neuroscan/EEProbe, Eyelab, Photoshop, Tempo, Excel) • Expertise in statistical methods (correlation, regression, ANOVA, t-test) and analysis programs (SPSS, Statview, JMP)

Sarah Callahan Estrada, Ph.D. – RESUME – Page 2

Neuroscience Research Coordinator, Columbia University, fMRI Research Center, 2001-2003 • Collaborated with senior researchers in the design, programming, and analysis of experiments on language comprehension in Spanish-English bilinguals • Gained expertise in neuroanatomy and the collection, processing, and analysis of functional MRI data (EPRIME, SPM, SPM2) • Coordinated and contributed to fMRI research on various cognitive and neural processes • Wrote successful grant application for funding from National Science Foundation

RELEVANT EXPERIENCE Lecturer, Psychology of Language, UCSD, Dept. of Psychology, 2009 • Provided students with an integrated understanding of a variety of topics related to linguistic and psycholinguistic theory and experimentation • Developed exams and a research paper to evaluate students’ understanding and progress

Lecturer, Introduction to Statistics, UCSD, Dept. of Psychology, 2008 • Provided students with an integrated understanding of statistical concepts and calculations including z-test, t-test, ANOVA, correlation, and linear regression • Developed homework and exams to evaluate students’ understanding and progress • Supervised and directed three graduate and four undergraduate teaching assistants in leading discussion sections and grading exams

Supervisor, Independent Study, UCSD, Dept. of Psychology, 2003-present • Independently mentored and supervised approximately six undergraduate students • Trained students in theory, experimental design, methods, and analysis • Supervised and directed students in data collection and processing

Graduate Teaching Assistant, Various Courses, UCSD, Dept. of Psychology, 2003-2009 • Guided students in mastering a variety of topics in psychology, linguistics, and statistics in discussion sections as well as on a one-on-one basis • Presented guest lectures on topics of expertise including bilingualism, neuroanatomy, and statistics

SELECTED PAPERS Callahan, S.M., Shapiro, L., & Love, T. (2009). Parallelism effects and verb activation: The sustained reactivation hypothesis. Journal of Psycholinguistic Research. DOI: 10.1007/210936-009-9128-0.

Callahan, S.M. (2008). Processing anaphoric constructions: Insights from electrophysiological studies. Journal of Neurolinguistics, 21, 231-266.

Callahan, S.M., Nicol, J., Love, T., & Swinney, D. (in prep). An ERP investigation of the effects of anaphoric form: Processing null and overt verb phrase anaphors.

Callahan, S.M., Nicol, J., Love, T., & Swinney, D. (in prep). The processing and interpretation of null and overt pronouns in Spanish. .

Galileo® K-12 Online Instructional Improvement and Instructional Effectiveness System Prepared for: Mesa Unified School District No.4 RFP #15-47MP Formative Assessment – February 19, 2015 Copyright © Assessment Technology, Incorporated 2015. All rights reserved. 24 Tab 1. Statement of Qualifications and Experience ATI Leadership Staff Brief Resumes/Vita - Confidential Scott Cunningham, B.S. – RESUME EDUCATION CERTIFICATION University of Nebraska, Lincoln, Nebraska Arizona Teacher Licensure 1998-2004 May 1992 Standard Secondary Education, 7-12, Social Studies Bachelor of Science in Education Secondary Social Sciences HONORS AND ACTIVITIES GPA: 3.39 Jessie E. Keller Scholarship (1988-89) Dean's List (Fall 1991) Student Government Candidate (1991) RELEVANT EXPERIENCE Assessment and Instructional Design Director Assessment Technology, Inc. 2002-Present • Organize the development of Item Types and Items for the online measurement of state and district standards. Train Content Specialists in the creation of Item Types, Items, and Instructional Dialog Construction. • Supervise the development of item specifications to generate consistent instruction and measurement of standards-based performance objectives. • Develop instructional materials that reflect best practices in developing student skills and knowledge appropriate to educational standards and student development.

History and Government Teacher Canyon del Oro High School World History (9&10) and U.S./Arizona Government and Economics, 1998- 2001; Amphitheater High School 2001 • Created extensive activities to address varied learning styles. • Developed interactive, academically challenging lessons. • Promoted personal responsibility, high standards of effort and achievement for all my students.

Social Studies Teacher Becker High School, Grade 9 United States History 1 & 2 and World Geography, 1995-98 • Planned and taught in both traditional and block scheduling systems. • Utilized technology as a productive educational resource. • Developed a system of elective projects allowing students to individualize their learning. • Volunteer 8th grade assistant boys basketball coach (1995-96 season).

Special Education Teacher Assistant Discovery School, March 31, 1995-June 7, 1995 • Assisted 5th grade students in special education program. • Aided teachers in supervising special education students.

Substitute Teacher/Assistant Girls Basketball Coach Upsala Area Schools, Grades 8-12 1994-95 Season • Substitute taught in grades 7-12 in all subject areas. • Assisted in coaching the Girls Varsity Basketball team. • Coached the Junior Varsity and "C" teams.

OTHER WORK EXPERIENCE • Bookstore Assistant, Western National Parks Association Bookstore, 2002. • Music Department Supervisor, Barnes and Noble Lincoln, Nebraska, 1994. • Summer Playground Program Activities Specialist, Lincoln Parks and Recreation, Lincoln, Nebraska, 1989-1994. • Security Specialist, FirsTier Bank Lincoln, Nebraska, 1989-1994.

EXPERIENCE Assessment Technology, Inc, - October 2000- Present Field Services Director As Field Services Director at ATI, Mr. Mayhew’s responsibilities include the design and implementation of ATI product and service distribution and promotion plans. He sets department goals, identifies department tasks and helps evaluate department goal attainment. Additionally, his responsibilities include maintaining open communication between the Field Services Department and other departments within ATI to ensure provision of effective educational management offerings to clients. Mr. Mayhew is depended on to lead his staff in providing timely and responsive service to meet client needs. Additionally, he plays an important role in assisting large client groups with outcome research initiatives.

CEO Software Corporation, - August 1997-October 2000 Account Manager Responsibilities included: Management of product sales throughout the US and Canada generating product and consulting leads, prospecting, proposals, and closing of sales. Other responsibilities included assisting in the training of new account managers, first level technical support relating to sales, detection and escalation of strategic enterprise accounts, telephonic training, assisting in the creation, development and deployment of departmental marketing materials and trade shows.

During first full year as account manager, each previous quarter in sales was exceeded. With each additional year, each previous year’s sales revenue was exceeded.

Research Corporation Technologies Tucson, AZ - 1997 Office Clerk Assisted in project research, including the collection and organization of data. Also assisted in the coordination of meetings and client relations.

ArtVac Corporation, Lincoln RI - 1987-1996 Head Shipping Clerk/Shipping Clerk Responsibilities included: Overseeing the shipping and receiving department, restructuring, consolidating and updating the inventory. Proposed and implemented cost effective, efficient programs. Tucson Chamber of Commerce, Tucson, AZ - Spring 1996 Internship Aided in the Economic Development department. Responsibilities covered the compilation of Tucson business information in Excel, and researching Tucson demographics for the publication, "Tucson at a Glance."

EDUCATION University of Arizona, Tucson AZ Bachelor of Science in Regional Development Business and Public Administration Minor

SELECTED PRESENTATIONS 2009 Presenter: National Education Computing Conference. “Intervention Technology Supporting School Interventions.” 2009 Presenter: ATI’s Educational Interventions Forum. “Dynamic Intervention Systems.” 2009 Presenter: ATI’s Educational Interventions Forum. “Designing an Interventions.”

Education 2009 University of Arizona, Tucson, Arizona PhD in Teaching and Teacher Education Major: Teaching and Teacher Education; Minor: Educational Leadership Dissertation title: Standardization of Practice in a High School Social Studies Department: Three Case Studies

2002 Northern Arizona University, Flagstaff, Arizona MEd in Educational Leadership Major: Educational Leadership Graduated with honors

1999 University of Arizona, Tucson, Arizona BA in Education Major: English; Minor: History Graduated with honors, summa cum laude, Regents scholar

Certification 2000–present Arizona Standard Secondary Education, 7–12 Approved areas: English, History, Social Studies Endorsement: Structured English Immersion, K–12

Professional Experience 2009–present Language Arts and Social Studies Content Writer, ATI, Inc., Tucson, Arizona Write and edit assessment content, including test items and texts, for grades K–12; create and edit instructional content, including lessons and units, for grades K–12; analyze state standards from across the nation to determine and respond to client needs. 2008–2009 Freshman House Coordinator/Instructional Support Specialist, Tucson High Magnet School, Tucson, Arizona Created calendar of activities for the Freshman House program; developed and presented professional development activities for Freshman House faculty; coordinated activities of Freshman House teachers; administered standardized tests, including quarterly benchmark assessments; collected and analyzed student achievement data; advocated and provided instructional support for teachers.

2007–2008 Instructor, Tucson High Magnet School, Tucson, Arizona Taught American history (grade 11) and world history (grade 10).

2006–2007 Instructor, Cholla High Magnet School, Tucson, Arizona Taught GATE English (grade 10), GATE non-Western civilization (grade 10), regular English (grade 10), and yearbook (all grades).

1999–2006 Instructor, Vail High School, Tucson, Arizona Taught Advanced Placement English (grade 12), modern literature (grade 11), world history (grade 9), regular English (grade 9), drama (all grades), and yearbook and media productions (all grades).

Lucas J. Schippers Continued

Professional Activities 2009 Presenter, Reading Apprenticeship Inservices, Tucson Magnet High School Provided training in Reading Apprenticeship instructional methods to Tucson High Magnet School teachers.

2008 Cooperating Teacher, Tucson Magnet High School Served as cooperating teacher for a student teacher through the University of Arizona.

2007 Cooperating Teacher, Cholla Magnet High School Served as cooperating teacher for a student teacher through the University of Phoenix.

2004–2005 Cooperating Teacher, Vail High School Served as cooperating teacher for two student teachers through the University of Arizona and the University of Phoenix.

2003–2005 Yearbook Advisor, Vail High School Produced two digital yearbooks on DVD and one traditional print yearbook.

2001–2005 Testing Coordinator, Vail High School Administered and proctored the Stanford 9 test, AIMS test, Northwest Evaluation Association Level Test, school district writing assessment, and benchmark tests.

Professional Growth Activities 2008–2009 Leadership in Reading Apprenticeship Training, Oakland, California 2007 Skillful Teacher Training, Tucson, Arizona 2006 Gifted and Talented Teacher Training, Tucson, Arizona 2004 Teacher Observation Protocol Training, Tucson, Arizona 2001 Blue Ribbon Schools Conference, Phoenix, Arizona 2000 First Days of School Conference by Harry Wong, Tucson, Arizona

Awards and Scholarships 2008–2009 Graduate Tuition Award, University of Arizona 2008–2009 Graduate College Fellowship, University of Arizona 2005–2006 Kate Farness Scholarship, University of Arizona 2004–2005 Graduate Tuition Award, University of Arizona 1996–1999 Regents Scholarship, University of Arizona

Papers Schippers, L. J. (2009). Standardization of practice in a high school social studies department: Three case studies (Doctoral dissertation). Available from Dissertations and Theses database. (UMI No. 1934691641)

EDUCATION University of Phoenix Microsoft Certified Systems Engineer (MCSE) Certificate Program Completed: March 2001, Certified MCSE January 2002 University of Arizona, Tucson AZ, graduated December 1990 Major: Chemistry, Minor: Math/Physics

WORK EXPERIENCE 1 Starting - ending dates: 2005 – Present Position title: Information Technology Director Employer: Assessment Technology, Inc., 6700 E. Speedway Blvd., Tucson AZ 85710

Starting - ending dates: March 2001 – 2005 Position title: Network/Database Administrator Employer: Assessment Technology, Inc., 6700 E. Speedway Blvd., Tucson AZ 85710 Supervisor: Mr. David Bergan, VP, Media and Technical Services Duties and accomplishments: Windows 2000 Network Administrator. Perform administration and maintenance of 8 mission critical servers and 40+ clients in a Windows 2000 Wide Area Network (WAN) environment with minimal supervision. Responsible for all system availability, performance and security. Maintain MS Active Directory database, utilizing group policy to meet performance and security needs of corporate network. Ensure maximum availability and performance for multiple-server web farm using MS Internet Information Services (IIS) 5.0, ensuring successful delivery of Internet-based application for over 4,500 clients. Monitor all known security and performance issues, applying updates, service packs and system changes when deemed necessary. Responsible for all network architecture and system upgrades to maintain scalability of entire network as dictated by internal and external growth. Plan domain controller and file server backup program to ensure recoverability in the event of hardware failure. Plan for all future hardware and software upgrades to continue operation of all systems during physical plant and infrastructure changes. Devise security plan to ensure minimal risk from exposure of mission critical systems to Internet. Maintain availability and security of corporate Virtual Private Network (VPN) for external access by company employees. Perform client technical support on an as-needed basis. Database Administrator. Perform maintenance and backup of mission critical application (back-end) database using MS SQL Server 2000 on high- availability hardware platform. Monitor performance of database to determine necessary upgrades for full scalability in conjunction with client needs. Plan and execute all physical system upgrades. Maintain and execute regular backup plan to ensure full database recoverability in the event of catastrophic hardware failure. Maintain internal client database in support of field services staff. Maintain internal financial database in support of operations staff. Design/implement additional databases using MS Access in support of operations staff to meet pre-determined needs.

Brandon J. Smith Continued WORK EXPERIENCE 2 Starting - ending dates: August 1998 – March 2000 Position title: Environmental Scientist Employer: US Air Force, Grand Forks Air Force Base, 525 6th Ave, Grand Forks AFB, ND 58205 Supervisor: Mr. Wayne Koop, Environmental Management Flight Chief Duties and accomplishments: Windows NT Work Group Administrator. Perform administration of environmental sub-domain, including security updates, NT patches, and user administration. Assist with logon difficulties, operating system errors, and software support. Design curriculum and execute training sessions for multiple users with network operating system and MS Office products. Environmental Webmaster. Gather information from all program managers, coordinate editing and draft intranet/Internet sites for flight using MS Front Page/HTML, Corel Draw, Adobe Photoshop, MS Photo Editor. Follow initial design through to implementation. Assist squadron webmaster in repair & upgrade of unit intranet sites using HTML, Java, and JavaScript. Environmental Geographic Information System (GIS) manager. Work with squadron GIS Coordinator to ensure all flight needs are considered with during squadron-wide implementation of this cutting edge technology. Ensure all data is converted to proper format (MS Access/Excel/dBase II-IV) for inclusion. Import necessary information and merge with CAD files/shape files using ESRI ArcView/ArcInfo GIS software for graphical display. Customize graphical user interfaces (GUI) using Avenue Script, specific to individual users. Perform training to ensure successful unit-wide implementation. Environmental Programming/ Funds Manager. Responsible for justification and awarding of over $2.4M in environmental compliance- related projects, and over $150K annual environmental operations and services program at an Air Force base with over 7,000 employees. Environmental Audit Program Manager. Supervise 20+-member team performing audit of Air Force installation. Assess compliance with environmental- and safety-related laws and regulations such as the Oil Pollution Act of 1990; the Resource Conservation and Recovery Act (RCRA); the Comprehensive Environmental Response, Compensation, and Liability Act (CERCLA); the Emergency Planning and Community Right-to-Know Act (EPCRA); the Clean Air Act; the Clean Water Act. Suggest, approve and monitor progress of all corrective actions and finding closures. Conduct introductory environmental awareness training for first-term military members, including basic responsibilities under all aforementioned laws and Air Force environmental regulations. PROFESSIONAL Microsoft Certified Systems Engineer (MCSE), January 2002, registration REGISTRATION #2180283 Microsoft Certified Professional (MCP), December 2000 • Windows 2000 Server • Windows 2000 Professional • Windows 2000 Directory Services Infrastructure (Design and Implementation)

Brandon J. Smith Continued

COMPUTER - Windows 2000 Network Security RELATED SKILLS - Windows 2000 Network Infrastructure (Design and Implementation) Novell & MS Windows (95/98/NT4.0/2000) OS/Network OS, MS SQL Server database administration, MS Office (95/97/2000), Front Range Goldmine 5.5, HTML 4.0 (site design with/without frames), JavaScript, MS Front Page, MS Internet Information Server, Macromedia Fireworks, Macromedia Dreamweaver, Macromedia Fireworks, Corel Draw, Adobe Photoshop, ESRI ArcView/ArcInfo Geographic Information System (GIS) Software & Avenue script

EDUCATIONAL BACKGROUND University of Arizona, Tucson, AZ., Bachelor of Arts, Secondary Education. December 1998. Major in History with a minor in English. 3.9 GPA Manchester Community College, Manchester, CT. Associate Degree, General Studies. May 1995. 3.9 GPA

PROFESSIONAL EXPERIENCE Assessment Technology Incorporated Instructional Development and Professional Development Director (1/2000-Present) • Designed and create a menu of on-site trainings and seminars. • Instruct teachers and school administrators on how to use the Galileo® system. • Administered hundreds of trainings/workshops • Designed and created four online distance learning courses. • Manage online courses. • Managed a research project evaluating the impact of ATI's Preschool curriculum on Head Start children. Amphitheater School District-Canyon Del Oro High School, Tucson, AZ. U.S. / AZ History Teacher (8/99-1/00). • Provide instruction of material to college bound high school students. • Design and create lesson plans. • Align class curriculum with current state and district standards. • Manage six classes of 25-30 students. Arizona Historical Society, Tucson, AZ. Education Assistant (1/99-7/99). • Provided support and assistance to outreach programs. • Consulted and reviewed portions of the Studies in Arizona History Teacher’s Manual. • Consulted and reviewed portions of the Social Studies State Standards. • Designed teacher activities and educational supplement for the Gold Rush exhibit. Museum Discovery Teacher (5/98-8/98). • Designed and created lesson plans. • Taught lessons on southwestern history and culture. Project S.O.A.R., University of Arizona, Tucson, AZ. Supervisor, Flowing Wells High School, Teen Parent Program (9/97-5-98). • Oversaw and managed mentor-student relationships. • Designed and implemented counseling sessions, academic workshops, social activities and a leadership retreat. • Met and interacted with administrators, teachers, and counselors to provide support for students. Mentor, Flowing Wells High School(9/96-5/97). • Provided three at-risk, pregnant teenagers with academic support by offering tutoring and working closely with their teachers.

Kerridan Kawecki, B.A. - RESUME– Page 2

PROFESSIONAL ORGANIZATIONS & ACTIVITIES • Chair of Technology Subcommittee North Central Accreditation Team for Canyon del Oro High School (8/00-Present). • Member of Technology Subcommittee North Central Accreditation Team for Canyon del Oro High School (9/99-5/00). • Co-Advisor to the National Honor Society at Canyon Del Oro High School (8/00-Present). • Member of the Arizona Education Association (8/99-Present). • Member of the Arizona Council for the Social Studies (1/99-12/99).

AWARDS • Graduated Magna Cum Laude. • Certificate of Award: Class of December 1998. Outstanding Academic Achievement in History. • Certificate of Award: Class of December 1997. Outstanding Undergraduate Research Paper. • Golden Key National Honor Society.

Karyn White, M.A. – RESUME EDUCATION University of Arizona Tucson, AZ 1987-1992 Bachelor of Arts in Psychology, GPA 3.04 Post Baccalaureate Certification in Elementary Education, GPA 4.00 University of Phoenix Online 2004-2006 Masters of Arts in Curriculum and Technology, GPA 3.98

ENDORSEMENTS AND HONORS AEPA Mathematics, Psi Chi Honor Society, Deans Honors List

PROFESSIONAL EXPERIENCE Assessment Technology Incorporated Educational Management Services Director (2008 – Present) • Working with district assessment leaders to aid in assessment design • Working with district staff in developing assessment strategies and translating those strategies into a district educational benchmark plan • Working directly with district leaders to develop Instructional Dialogs and to use Instructional Dialogs to meet goals and monitor interventions • Consulting with district staff responsible for completion and use of the Dialog Manager • Working as consultant with district staff subsequent to ATI Assessment and Instruction Content Development Department actions related to district suggestion obtained during district test review Senior Assessment Content Specialist (2005 – 2008) • Consulted with management relative to new products and refinements in ATI’s integrated learning system • Liaison and for Research Development and Assessment Department • Wrote, edited, and presented curriculum and assessment items • Trained other employees and created training manuals for new system developments Teaching and Curriculum Specialist Positions Classroom Teacher and Curriculum Development (1993 – 1997, 2004 - 2005) • Safford Middle Magnet School (2004 - 2005) Tucson, AZ o Taught Arizona state reading, writing, and math standards to seventh-graders and eighth-graders o Completed SIOP training. • Global Education Foundation (2003 - 2004) Tucson, AZ o Taught cross-curriculum Arizona state standards to Kindergarten and 1st graders o Developed school curriculum based on the Arizona state standards (K-8) • Wood Dale Jr. High School (1993-1997) Wood Dale, IL o Taught reading and writing to seventh-graders and all subjects to third-graders o Pom-pom coach and school newspaper and yearbook sponsor • Brichta Elementary (1992) Tucson, AZ o Student taught fifth-graders Primerica Financial Services Financial Services Agent Trainer/Owner/Office Manager (1997 – 2003) • Created training classes and curriculum for personal financial analysts • Developed systems for licensing agents and helping them fulfill all requirements for continuing educational requirements • Presented training courses and motivational seminars for financial analysts

Tab 2. Scope

Describe, in detail, your plan to meet the scope of this project. Include detailed timeframe for project including estimated start date (or days from contract award) through completion. Tab 2.3.4 and Tab 2.3.5 provide ATI’s detailed project plan including a suggested Implementation Plan to meet the scope of the project. Initial implementation discussions will begin immediately following contract award. Galileo® rollout is scheduled to be complete within 30 days following contract award except for the importation of existing assessment content which will be completed within a timeline mutually agreed to by the District and ATI.

Tab 2.1. General Expectations ATI understands that MPS desires a student assessment and reporting software package that will provide information regarding student progress toward achieving mastery of Arizona’s College and Career Ready Standards. ATI understands that MPS would like to choose a solution that meets the criteria specified below. This section provides a brief overview of how Galileo technology and ATI services can meet these criteria and enable District staff to efficiently and effectively elevate student achievement.

2.1.1. Is easily used and mastered so that teachers and administrators are able to focus on teaching and on meeting the needs of each and every student Galileo has been successfully implemented by hundreds of school districts and used by thousands of educators nationwide. Designed in keeping with widely accepted standards for usability for web-based applications, Galileo includes intuitive menu-driven, drop-down, and “clickable” link-based features that will be familiar to anyone accustomed to using the Internet. Galileo’s integrated Dashboards and easy-to-understand applications ensure that Galileo is accessible to all users and requires no prerequisite skills. For example, Galileo Dashboards support teachers in assessing students, evaluating student performance, and using assessment data to guide instruction from one convenient location. Similarly, Galileo Dashboards minimize the number of clicks necessary to obtain actionable information to support curriculum

In addition to the usability and design features of the Galileo® system, ATI also offers initial and ongoing professional development that ensures that MPS staff can use the most current version of Galileo effectively. Ongoing professional development helps ensure that MPS educators stay abreast of the rapid changes in technology and practice that characterize the current educational environment. ATI professional development options include on-site large-group professional development sessions at District-provided sites (typically equipped with an Internet-connected computer lab), web-based small-group live and recorded webinars including the Learning On- Demand series, and pre-recorded video tutorials, quick reference guides, manual, and help files accessed through the Forum and the Help interface. Opportunities for flexible and timely professional development are enhanced for MPS by the fact that the ATI home office is located in Tucson, Arizona.

ATI typically employs a Train-the-Trainer model which prepares select staff to transfer skills by serving as trainers for additional staff. On-site professional development sessions walk users through accessing the website, and logging into the application with time set aside to ensure that users know how to navigate the system. A District trainer can typically be trained on the full capabilities of the system in twelve hours. Twelve hours of professional development will also prepare trainers to train other staff, and support other staff members in their Galileo use. In the Tips for District Professional Development Instructors session provided by ATI, it is recommended that teachers be trained in short, 30 minute – one hour sessions and that these sessions be presented strategically throughout the implementation process. For example, an initial training on how to administer assessments using Galileo is typically provided prior to the first district-wide assessment administration. This sets the stage for a later, follow-up training after the first administration is complete that focuses on running reports and using the results to meet the needs of all students by providing differentiated instruction and implementing targeted interventions. Most districts find that dedicating between four and six one hour sessions is adequate to train teachers to be independent, effective users of the Galileo K-12 Online system.

2.1.2. Enables teachers to assess mastery of specific skills and objectives in a manner of their choosing Galileo is designed to be truly flexible, enabling districts and teachers to assess mastery of specific skills and objectives in a manner of their choosing. Using Galileo, teachers and administrators can easily design and create a large variety of district-wide, school-wide and classroom assessments including benchmark and formative assessments as well as pretests, posttests, placement tests, end-of-course examinations and others. All aspects of the design and creation of assessments can be accomplished entirely using web-based interfaces included within Galileo K-12 Online.

Using the Assessment Planner, district staff participates in the design of district-wide assessments aligned to standards and sequenced according to district curriculum and pacing guides. The Assessment Planner allows the district to: 1) determine the number and timing of assessments, 2) indicate the specific standards covered in instruction and targeted for inclusion on each assessment, and 3) indicate the number of items desired for each standard covered on the assessments. After tests are assembled by ATI, they may be reviewed by the district if desired. Within Galileo®, the Test Review utility provides the capability to review each item in a test including the item parameters associated with each item as well as to make comments regarding the overall test. If desired, district reviewers can replace items by selecting an item from the ATI secure item banks that is more closely aligned to their expectations or instructional focus for the assessment.

Using the Test Builder interface, teachers can also create formative assessments including items from ATI’s extensive existing community item banks as well as, if desired, any District-created item banks. Within the Test Builder interface, users may add items from existing banks in two ways. The Generate Questions tool allows users to quickly create a test by adding a set of randomly generated items aligned to a desired standard or set of standards while the Search Item Bank tool allows users to create a test by selecting an item or set of items from all the available items aligned to a desired standard.

Galileo Search Item Bank results – Confidential Screen Shot

Galileo Test Builder also allows teachers and other district personnel to create, modify, and edit test items and place them in their own libraries. All items created within Galileo can be aligned to standards. Teachers can construct items that are multiple-choice, true/false, yes/no, short answer, essay, and submission of work samples. Assessment items may also be accompanied by images, text, and links to a variety of visual media (e.g., graphs, pictures, short stories,

Galileo® K-12 Online Instructional Improvement and Instructional Effectiveness System Prepared for: Mesa Unified School District No.4 RFP #15-47MP Formative Assessment – February 19, 2015 Copyright © Assessment Technology, Incorporated 2015. All rights reserved. 37 Tab 2. Scope Tab 2.1. General Expectations informational readings). Test Builder uses a rich-text editor and includes spell check capabilities compatible with Microsoft Word. In addition, Test Builder includes an Equation Editor feature that makes it possible for teachers to easily construct mathematics and science items containing equations, scientific notation, formulae, and symbols in ways that create consistency across items. Any user at the various school sites has access to the Test Builder and all of its components and can create items and assessments, which are stored in a system of libraries. Access to these libraries is controlled to maintain each library with the required degree of security, but flexible enough to permit teachers to share items with colleagues and to contribute items to school- or district-level libraries if the school or district elects to provide such access.

Another option for building and implementing formative assessments is Dialog Builder. Dialogs are online communications between one or more teachers and one or more students that integrate assessment with instruction. Dialogs include question slides that can accommodate the full range of available item types including technology-enhanced (TE) items. Dialogs also include instructional slides designed to promote learning. Dialogs can include multimedia content. As a result, they can function as instructional resources. ATI provides Instructional Dialogs in grades one through 12 in English language arts, mathematics, and science. Districts can also create their own Dialogs in any content area or access Dialogs created by districts and shared within Galileo®.

Another option for the implementation of existing formative assessments is Automated Scoring Key (ASK) technology. ASK technology enables teachers to import, store, schedule, deliver, administer (online, offline, or via handheld wireless devices), automatically score (where applicable), and report on existing assessments, as well as publisher-purchased assessments with appropriate permissions.

2.1.3. Enables the district to have secure periodic benchmark assessments Galileo includes a comprehensive standards-based assessment system that can provide MPS with secure periodic benchmark assessments as well as a wide variety of other types of assessments. Galileo contains a variety of built-in, integrated security measures that are designed to ensure the confidentiality, integrity, and viability of online data. Galileo also includes a number of tools enabling districts to implement the desired level of assessment security by controlling access privileges for students and staff.

Galileo provides access to ATI pre-built student assessments aligned to standards including Arizona’s College and Career Ready Standards and Arizona State Academic Standards: • Comprehensive Benchmark Assessment Series (CBAS) providing interim assessments • Early Literacy Benchmark Assessment Series (EL-BAS) providing TE items designed for young children

• Early Math Benchmark Assessment Series (EM-BAS) providing TE items designed for young children • Comprehensive Pretests/Posttests • Instructional Effectiveness (IE) Pretests/Posttests specifically designed to measure educator effectiveness • Computerized Adaptive Tests (CAT) • Comprehensive Screening Assessments supporting universal screening goals by identifying students at-risk of failing to show mastery on the statewide test • Comprehensive Placement Assessments indicating student readiness for advanced placement • Pre-built Writing Prompt assessments • Formative Assessment Series with short five-item quizzes for each standard

Galileo® also provides the capability to create customized standards-aligned assessments to serve a variety of purposes: • Customized district-wide assessments aligned to a district’s curriculum and pacing guide (e.g., benchmark assessments, pretests, posttests, placement, end- of-course exams) • Customized school-wide and classroom assessments (e.g., formative assessments, quizzes, end-of-course exams) • Customized IE Pretests/Posttests in any content area (e.g., music, art, physical education), with psychometric analyses (with sufficient sample size) for districts participating in the CAIB initiative • Customized educator rating-scale assessments that can be used to assess teacher and administrator proficiency

2.1.4. Seamlessly aggregates, analyzes, disaggregates and reports assessment information immediately to all appropriate people (students, teachers, principals, district administrators) Galileo’s reporting functionality displays actionable, real-time, reliable, and valid standards- aligned data on student learning and includes Dashboards for teachers and administrators as well as an overview page for students and parents in the K-12 Student-Parent Center. A broad set of ready-made standards-aligned reports are available to Galileo users. In addition, with the use of Form Builder, Report Builder, and Custom Test Reports, users have the option of designing their own customized reports.

Data within Galileo are organized in a hierarchical structure that allows for rapid aggregation at the student, class, school, and district levels using real-time data. Moreover, customizable student filters based on information uploaded into Galileo (e.g., demographic information,

For assessments taken online or using handheld wireless devices, the Test Monitoring Report presents student responses in real-time as they take the assessment. All other Galileo reports that are based on raw scores are available immediately as soon as the student’s responses (for computer-scored items) or teacher scores (for teacher-graded items) have been recorded in Galileo. If the assessment is administered online, responses are recorded in real-time as the student takes the assessment, and the data are available for reports based on raw scores as soon as the student hits the “Exit Test” button signaling that he or she is finished taking the assessment. If the assessment is taken offline, then reports that draw on raw scores are available as soon as the bubble sheet is scanned into the system.

Test Monitoring Report presenting results in real-time – Confidential Screen Shot

ATI conducts an IRT analysis of every Galileo® K-12 Online district-wide assessment (e.g., benchmark, pretest, posttest) that is administered by the district. In addition, the DL scores on all such assessments are placed on a common scale so that student progress can be monitored, and each is aligned with student scores on the relevant statewide assessment so that the cut scores will serve as good indicators of the student’s likely performance on the statewide assessment. These analyses require that the distribution of student scores be as complete as possible and so cannot be conducted until the district has finished administering the

2.1.5. Provides teachers with reports, intervention activities and/or tools that help guide instruction and improves their effectiveness in the classroom Galileo helps guide instruction and improve the effectiveness of teachers in the classroom in a number of ways. Galileo’s reporting functionality provides teachers with actionable, real-time, reliable, and valid standards-aligned data on student learning and provides standards-based recommendations for intervention/enrichment as well as links to follow-up intervention activities (i.e., Instructional Dialogs, brief quizzes). In particular, the Teaching Dashboard enables teachers to access classroom-level instructional tools and assessment results in one convenient location. The Teaching Dashboard supports teachers in working with both classes and intervention groups and is the go-to page for planning instruction, assessing students, and analyzing student progress. The Class Calendar and scheduling tools for lessons, assignments, and tests are accessed here. Dashboard information includes: Recent Events, Upcoming Events, Class Roster, Class Risk Level Summary, and Student Growth and Achievement by Class. Links within this view connect to Test Monitoring, Item Analysis, and Intervention Alert reports.

The Teaching Dashboard enables teachers to implement curriculum as well as track individual student performance, class performance, and the performance of students in intervention groups. The Teaching Dashboard not only provides access to assessment results, but also supports instruction related to those results. It is the go-to page for planning instruction, assessing students, and analyzing student progress at the class or intervention-group levels. The classroom calendar and scheduling tools for lessons, assignments, and tests are accessed here. Dashboard information includes: Digital Curriculum Explorer (scheduled for release in spring, 2015), Recent Events, Upcoming Events, Class Roster, Class Risk Level Summary, and Student Growth and Achievement by Class. Links within this Dashboard connect to Test Monitoring, Item Analysis, and Intervention Alert reports.

The Teaching Dashboard enables teachers to easily implement curriculum as well as differentiated instruction and intervention based on assessment results. The reports and tools included on this Dashboard support the grouping of students based on a variety of factors including overall student achievement, student growth, student standards mastery, student risk of failing the statewide assessment, and student performance on specific items. The following screen shot illustrates the Teaching Dashboard.

Teaching Dashboard – Confidential Screen Shot

The top portion of the Teaching Dashboard provides teachers with a quick view of Recent Events and Upcoming Events such as tests, assignments, and lessons. The Teaching Dashboard also provides links to reports and to interfaces widely used by teachers such as the Benchmark Results page and the Class Calendar. Each recent event entry links to useful reports that display data that is actionable at the classroom level. Each upcoming event is linked to the scheduling page for that event.

The Digital Curriculum Explorer widget (scheduled for release in spring, 2015) supports the dissemination of a district’s digital curriculum providing teachers with access to the curricular, instructional, and related assessment materials from a single location. From this widget, these materials can easily be scheduled as part of classroom instruction or as assignments for students or groups of students. Implementing a Galileo® digital curriculum helps districts determine what works by measuring what has been taught as well as what has been learned. Educators are able to easily determine what instruction has taken place by viewing scheduled Dialogs in the Digital Curriculum Explorer widget. Educators can also determine the effectiveness of instruction by evaluating student performance on brief formative assessments administered as part of scheduled Dialogs. This information can then be used to make further instructional decisions and to revise curricular materials as needed to optimize student learning.

Digital Curriculum Explorer widget on the Teaching Dashboard scheduled for release in spring 2015 – Confidential Screen Shot

At the bottom of the Teaching Dashboard, the teacher’s class roster is displayed with links to add students, enroll students, and manage student passwords. For each student in the class, a link is also provided to their Assessment History Report displaying the results of all the assessments the student has taken as well as their student information page.

Next to the class roster, two additional reporting widgets provide access to actionable results that can be used to guide differentiated instruction and intervention. The Class Risk Level Summary provides, for the teacher’s class, a count of the students at each level of risk for failing the statewide assessment. Students are classified into risk levels (e.g., On Course, Low Risk, Moderate Risk, High Risk) based on their performance on district-wide assessments. Via drill downs from this widget, the teacher can access additional reports summarizing student performance, access the Instruction Performance Tracker providing instructional recommendations for students in each risk level, and schedule follow-up assignments and quizzes appropriate to the needs of each risk group.

The Student Growth and Achievement by Class widget provides a graphical cross-classification of students based on their growth between two tests and their achievement on the second test. This graphical summary is designed to facilitate the identification of student groups for

2.1.6. Provides principals with indicators of where students are, and are not, on track for mastery of grade level expectations and student and teacher growth/improvement over time The Galileo® School Performance Dashboard provides principals and other school- and district- level staff with indicators of where students are, and are not, on track for mastery of grade level expectations. The School Performance Dashboard also enables principals to track student and teacher growth/improvement over time. Widgets include: Categorical Growth Summary, Student Growth and Achievement by School, and Risk Level Summary. When a value-added modeling analysis has been conducted, an additional widget is included, Value-Added Model. In addition to summarizing data for one or more schools, these widgets provide the results of state-of-the-art statistical analyses forecasting student performance on the statewide test and evaluating the effects of teachers and administrators on student growth. The Categorical Growth Summary widget presents a summary of the Categorical Growth Analyses for a selected school. Categorical Growth Analysis provides information about whether a teacher’s students have exceeded, maintained, or not maintained expected growth between two district-wide assessments. The Student Growth and Achievement widget presents a graphical cross- classification of students based on their growth between two tests and their achievement on the second test. This graphical summary is designed to facilitate the identification of student groups for intervention and enrichment (e.g., high performing students showing inadequate growth, low performing students showing exceptional growth). The Risk Level Summary widget provides a school- or district-level aggregated count of the students at each level of risk for failing to master grade level expectations as measured by the statewide assessment. Students are classified into risk levels (e.g., On Course, Low Risk, Moderate Risk, High Risk) based on their performance on district-wide assessments. Via drill downs from this widget, a principal can access additional reports summarizing student performance, access the Instruction Performance Tracker providing instructional recommendations for students in each risk level, and schedule follow-up assignments and quizzes appropriate to the needs of each risk group.

School Performance Dashboard – Confidential Screen Shot

Value-Added Model widget on the School Performance Dashboard – Confidential Screen Shot

2.1.7. Provides a measure of student growth/improvement and group progress Techniques based in IRT are used to provide DL scores for district-wide Galileo® assessments and to place those scores on a common scale to support the measurement of growth across different assessments within a grade and content area. Note that this is not possible with raw scores such as number correct or percent correct.

The DL scores for each assessment are also aligned with student scores on the relevant statewide assessment (e.g., in the future, the assessment provided by Arizona’s Measurement of Educational Readiness to Inform Teaching [AzMERIT]). This helps ensure that Galileo Risk Levels will serve as good indicators of the student’s likely performance on the statewide assessment. ATI’s ongoing research routinely demonstrates that Galileo assessments demonstrate high reliability (typically stabilizing at >0.80 for assessments with 50 items or more), predictive validity (recent mean correlation between scores on Galileo assessments and scores on the statewide test = 0.78), and accuracy in forecasting student performance on the statewide test (recent overall accuracy = 85 percent).

Statewide Test Performance by Galileo Risk Level

100 90 80 70 60 50 94 40 69 30 20 44 10 17 0

Percent of Students Passing Percent of Passing Test Statewide Students On Course Low Risk Moderate Risk High Risk

Figure 1 Mean percentage of students passing the statewide assessment for each risk level.

Numerous Galileo® reports provide DL scores supporting the tracking of student progress over time, the forecasting of likely statewide test performance, and the provision of empirically-based recommendations for instruction and intervention. In particular, Galileo provides a Student Growth and Achievement Report that categorizes observed student growth relative to research- based growth expectations. Galileo also provides the results of state-of-the-art statistical analyses (i.e., Categorical Growth Analysis, value-added modeling) designed to evaluate educator impact on student growth.

ATI is also engaged in a continuous research program related to the measurement of growth. For example, ATI is currently working with Arizona State University and Dr. Damian Betebenner

2.1.8. Helps students, parents, teachers, and administrators focus on knowledge and skills required for promotion, course completion and success on tests of state standards mastery (e.g. AzMERIT) Galileo is a fully standards-based system that integrates curriculum, instruction, and remediation/enrichment activities to help students, parents, teachers, and administrators focus on the knowledge and skills required for promotion, course completion, and success on tests of state standards mastery (e.g., AzMERIT).

Integrating Instruction with Assessment through the Galileo Digital Curriculum Platform The Galileo Digital Curriculum Platform supports rapid dynamic curriculum development integrating instruction with assessment to promote learning. Districts can develop new curriculums from scratch, integrate existing curriculums, and easily modify curriculums as needed. Once district curriculum information has been entered into Galileo®, school and classroom level users may access the information in order to guide the planning of instructional and assessment activities.

Within Galileo, a digital curriculum consists of two components: a series of online Instructional Units defining the scope and sequence of instruction and offering a variety of resources to guide instruction, and within each Unit a series of online Instructional Dialogs enabling the online delivery of assessment and instruction. Digital curriculums may include multimedia content, technology-enhanced assessments, and tools supporting online synchronous and asynchronous communication between students and teachers. A Digital Curriculum Explorer widget on the Teaching Dashboard (scheduled for release in spring 2015) will enable teachers to explore, customize, and take advantage of all the digital curricula content available to them from one easily accessible location.

Digital Curriculum Explorer widget on the Teaching Dashboard scheduled for release in spring 2015 – Confidential Screen Shot

Instructional Units outline what will be taught in a designated period of time. Instructional Units typically contain the standards to be addressed, instructional content, resources, and Instructional Dialogs. Instructional resources may include multimedia recordings, links to online resources, and links to other Dialogs. A series of Units can form the complete curriculum for a particular subject and grade. Units may be bulk scheduled for rapid, broad dissemination. Once scheduled, Units are easily accessed from the Class Calendar and the Teaching Dashboard.

Instructional Dialogs consist of online interactions between teachers and students which integrate assessment into the instructional process to promote learning. ATI offers a variety of Dialogs which are customizable, sharable, and may be linked together to achieve complex instructional goals. They are designed to provide adjustable instruction to target individual students, groups, or even an entire class. TE items may be included in the Dialog to provide valuable experience with the item types and to assess what has been learned so far. The student's response to the item prompts real-time feedback to guide learning which evidence shows is more effective than delayed feedback.

Identifying and Implementing Targeted Remediation and Enrichment Activities Galileo® provides standards-based recommendations for remediation and enrichment for students at all academic levels and for any content area for which an assessment has been administered. Galileo also provides teachers and students with access to remediation and enrichment activities in the form of Instructional Dialogs. ATI provides Instructional Dialogs in grades one through 12 in English language arts, mathematics, and science. Districts can also create their own Dialogs in any content area or access Dialogs created by districts and shared within Galileo.

A primary way of accessing enrichment and remediation activities is via recommendations provided in Galileo reports. The Instruction Performance Tracker provides recommendations for remediation and enrichment activities based on student performance on district-wide assessments for which an IRT analysis has been performed. A similar report, the Intervention Alert, provides recommendations for remediation and enrichment activities based on student performance on any assessment, test, or quiz administered via Galileo K-12 Online. This report does not rely on the IRT analysis applied to district-wide assessments so it can be used to access intervention plans based on student performance on any assessment, test, or quiz, no matter how informal.

Accessing a customized intervention plan within the Intervention Alert – Confidential Screen Shot

Both the Instruction Performance Tracker and the Intervention Alert contain links enabling teachers to access remediation and enrichment activities in the form of online interactive Instructional Dialogs, schedule these activities for one student or a group of students, and assign a follow-up quiz (if desired) to evaluate the impact of the activity. For remediation purposes teachers may schedule Instructional Dialogs that address standards on which the students’

2.1.9. Reduces paperwork and saves time for teachers and administrators Galileo®’s flexible web-based technology is specifically designed to reduce paperwork and save time for teachers and administrators by minimizing the amount of work required by District staff to develop, administer, score, and aggregate results for district-wide assessments.

Assessment development can potentially be one of the most time-consuming aspects of a district-wide assessment regime. Galileo provides MPS with access to a variety of reliable and valid pre-built assessments designed by ATI that can be rapidly delivered. ATI is also currently in the process of developing an assessment order form interface that will enable MPS to order an assessment with the click of a mouse.

Similarly, Galileo’s Assessment Planner and Test Review interfaces help the District efficiently manage the development of customized assessments aligned to the District curriculum. The Assessment Planner is an easy-to-use, point-and-click interface that enables district staff to: 1) determine the number and timing of assessments; 2) indicate the specific standards covered in instruction and targeted for inclusion on each assessment, and 3) indicate the number of items desired for each standard covered on the assessments. The Test Review interface provides District staff with the capability to review each item in a test including the IRT item parameters as well as to make comments regarding the overall test. If desired, district reviewers can replace items by selecting from the ATI secure item banks an item that is more closely aligned to their expectations or instructional focus for the assessment.

Final Review interface – Confidential Screen Shot

Another potentially time-consuming aspect of a district-wide assessment regime is assessment administration and scoring. Galileo® provides a variety of easy methods for administering assessments in the system. These include offline administration using scannable plain-paper and high speed answer sheets and printed test booklets, secure online administration including CAT and Instructional Dialogs integrating instruction and assessment, and handheld wireless device (i.e., “clicker”) administration. The Galileo Testing Activity Report enables MPS administrators to ensure that assessment administration is progressing and has been completed district-wide as well as in each school and class.

Regardless of the method of administration, Galileo® minimizes the effort required for the scoring of assessments and aggregation of data. Galileo provides automatic scoring for multiple types of items including dichotomously scored items (e.g., multiple choice, true/false, TE where applicable) as well as offline worksheets and online interfaces supporting teacher scoring of items including the viewing of scanned-in student work samples and scoring rubrics, where applicable.

Tab 2.2. RFP Objectives The Mesa Public Schools staff has identified required components and optional/desired components of a comprehensive Formative and Benchmark Assessment System solution. These requirements are listed and categorized on the following pages, according to overall and specific Formative and Benchmark Assessment System components.

Please see Tab 2.6 for ATI responses for each identified required and optional/desired component of the comprehensive Formative and Benchmark Assessment System solution.

Tab 2.3. Solution 2.3.1. The solution proposed must address the features and functions listed on the previous pages. Tab 2.1 provides an overview of how ATI and Galileo® can provide the features and functions expressed in the MPS scope of work and general expectations on page 24 of the RFP. Tab 2.5 describes how ATI and Galileo meet the technical specifications described on page 24 of the RFP. Tab 2.6 provides ATI responses for each identified required and optional/desired component of the comprehensive Formative and Benchmark Assessment System solution as well as further details were requested.

2.3.2. The proposal should also include available features and functions that were not specified above, but that are available as part of the solution. Only currently available systems and the latest releases of software should be proposed. Systems and/or software under development or in the planning phase will not be considered. Features that will be available only on a future release of software will be considered unavailable. The Galileo K-12 Online Instructional Improvement and Instructional Effectiveness system is a complete, comprehensive system including a broad range of online modules designed to promote learning via rapid and flexible access to assessment, curriculum, reporting, instructional, intervention, and instructional effectiveness tools. These tools represent the currently available Galileo system and are described throughout this proposal. In addition, ATI routinely enhances Galileo in response to advances in research and technology as well as changes in educator needs and federal and state requirements. All enhancements to Galileo are provided at no additional charge and, following release, are immediately available to all users upon their next log in to Galileo K-12 Online. Enhancements are typically released approximately monthly.

2.3.3. Please specify whether the software can be hosted at Mesa Schools or if it is an ASP, hosted by the vendor. Galileo is provided as a web-based application hosted on ATI servers. A distinct advantage of a vendor hosted, web-based solution is that no hardware costs are incurred by the District.

2.3.3.i. If the product is available as an ASP, all features of the Formative and Benchmark Assessment System must be operational on any Mesa Schools end-user workstation without modification. The system must have an architecture that supports scaling to 5,000+ simultaneous users or more. The proposal must describe the infrastructure that supports the solution, including open standards and platform independence.

Please provide a copy of your Application Service Provider (ASP) Web Hosting Service Level Agreement.

Galileo® is provided as a web-based application hosted on ATI servers (i.e., ASP).

Compatibility of Galileo with MPS Workstations As a web-based application, Galileo system requirements are based on the supported web browsers. All Galileo features are anticipated to be operational on any MPS end-user workstation using a supported web browser and including the required plugins. Information about Galileo supported web browsers is provided in Tab 2.5.7.T along with information about the computer operating systems supported by each browser. For the convenience of the District, links to a full list of manufacturer-published operating system requirements for each compatible browser are also provided in Tab 2.5.7.T as well as on the Galileo K-12 Online system requirements page at: http://www.ati-online.com/galileoK12/K12TrainingSystem.html. Some reporting functions may require the Adobe Acrobat Reader plugin and certain media functions may require use of the Adobe Flash Player plugin. Bank, Dialog and Test Builder functions may also require the Java Runtime Environment. All required plugins are available as free downloads from their respective vendors.

Galileo Scalability, Architecture, and Supporting Infrastructure The Galileo K-12 Online system has been designed for maximum scalability and already accommodates many more than 5,000 simultaneous users. The architecture and supporting infrastructure for Galileo, illustrated in the following screen shot, includes the use of clustered application servers in order to maximize scalability and minimize down time. If work needs to be done on a server, there is no need for the application to be taken offline. If traffic dictates that additional application servers are needed, more may be easily added, again with no down time. Data storage scalability is also built to accommodate future needs without service disruption. Additionally, the IT Technical Services team ensures the scalability of all systems to support increasing data needs and users.

Figure 2 Galileo® K-12 Online Logical Network Architecture

Information about ATI Service Levels ATI is committed to providing the highest level of service related to application service provider (ASP) web hosting of the Galileo system. The following paragraphs describe ATI standards and practices related to system availability, system performance, catastrophic system failure recovery time, phone help desk response times, email help desk response times, and data export request response times.

• System Availability All Galileo planned system maintenance is conducted outside core operating hours and restricted to late night weekends. Maintenance typically reflected here includes, but is not limited to, server hardware and operating systems upgrades, networking hardware changes, and database maintenance. Maintenance windows are posted on the application login pages one

ATI has maintained uptime of 99.5% for Galileo® K-12 Online over the past 12 months excluding planned maintenance windows typically lasting less than 1 hour per month. All maintenance is conducted outside core operating hours and restricted to late night weekends, with advance notice provided to clients via site announcement and/or email. Metrics are calculated using statistics provided by a dual-site, external third party monitoring service. Galileo software “builds” as referenced throughout the response are performed without downtime.

• System Performance System performance is closely monitored through database trace results, used to identify tuning opportunities and query optimization requirements. The Galileo Pre-K Online application server farm is monitored externally for simple page result response time with Information Technology personnel alerted in the event predetermined thresholds are exceeded. Average response time across the Galileo application is well under 300 ms, with most pages returning results in under 50 ms. Metrics are calculated by monitoring measured request response times logged by all application servers. Logs are retained for a minimum of 30 days for performance analysis.

• Catastrophic System Failure Recovery Time ATI has a catastrophic event recovery contingency plan that enables recovery quickly from a multitude of possible disaster scenarios within four hours. This plan includes a secondary data server located in a secure data center housed elsewhere in the region, data- and transaction-log backups, as well as a storage architecture including primary data files, secondary data files, and transaction logs.

ATI operates multiple data centers to enable failover to alternate site in the event of catastrophic system failure at the primary site. Recovery Time Objective for catastrophic event system failover is 30 minutes, with a 15 minute Recovery Point Objective. These standards are calculated based on the frequency of database log shipping and restore to the failover site. Logs are shipped every fifteen minutes, with a maximum restore time of 10 minutes.

• Help Desk- Phone Help desk telephone support lines are covered during all hours of service, 8:00 a.m.-5:00 p.m. MST Monday through Friday. All calls to the main ATI phone line are answered immediately by ATI front desk staff. Calls to phone lines for individual ATI staff members are typically also answered immediately unless the staff member is away from their desk or on the phone with another client. The standard for response to phone messages is set for same day, or next business day for requests received after hours. All support cases are logged in the ATI Customer Relationship Management (CRM) software database.

• Help Desk- Email Help desk email is monitored at all times by ATI personnel, with a standard of response set for same day, or next business day for requests received after hours. This standard of response pertains to the initial resolution of a case involving making contact with the client and collecting information related to the help request. Help requests take a variety of forms ranging from a simple question related to the use of Galileo® to a request for assistance in creating a custom form to a report of a “bug” in Galileo functionality. The time required to resolve the support case varies based on the type of request. All support cases are logged in the ATI CRM software database.

2.3.4. The proposal must address how the project will be conducted and managed. A narrative description of the approach and a detailed project plan are required. This section addresses the ways in which Galileo implementation will help facilitate MPS goals and addresses how the project will be conducted and managed, including a narrative description of the approach. A detailed suggested project plan including a description of implementation milestones, a schedule for implementation, and information about the proposed roles of MPS and ATI staff is provided in the section following this one.

Context for Implementation and Management of the MPS Project ATI understands that MPS wishes to procure a student assessment and reporting system that will facilitate the achievement of several integrated and overarching MPS goals. The first is to provide MPS stakeholders actionable information MPS Vision: "Unprecedented regarding student progress toward achieving mastery Excellence in Education" of the Arizona’s College and Career Ready Standards. The second goal is to ensure effective and efficient utilization of the Attainment of MPS Charting Excellence Strategic Goals system by all elementary, junior high, senior high and focus schools for grades Attainment of MPS three through eleven. The third goal is to Formative Assessment System Goals provide MPS with secure district assessments several times during the MPS - ATI Partnership Team year, as well as to provide teachers with Collaboration tools to create formative assessments

Galileo K-12 and activities. The fourth goal is ensure Online timely access to an array of user-friendly Implementation reports that provide actionable information about student mastery of standards and which enable schools to target needed areas of improvement. In addition to MPS overarching goals, the

ATI has a wealth of expertise and a track record of success in working in partnership with large school districts, state departments of education, and other educational organizations in implementing Galileo K-12 Online. At the heart of ATI’s implementation philosophy is the conviction that, when a school district enters into a contract for Galileo K-12 Online, ATI is not simply selling them a product. Awarding a contract to ATI means that ATI is engaging in a collaborative partnership with the district with a focus on achieving the mutual goal of improving instruction and elevating student learning. In this regard, ATI’s highly successful Partnership Model approach to implementation of Galileo K-12 Online will readily accommodate MPS needs for transitioning to a student assessment and reporting system.

Implementation through the MPS-ATI Partnership Teams Galileo implementation will be facilitated through a team effort involving a dedicated ATI project director as well as the full multi-disciplinary, hands-on ATI team representing staff from all the departments as described in Tab 1.1 of this proposal. Bringing the entire company “to the table” ensures optimization of continuous, integrated services provided to MPS. With ATI as a partner, MPS needs are addressed directly and in a timely fashion. Moreover, ATI ongoing enhancements to item banks, instructional content, user features, service, and technical support are accessible to MPS in a timely fashion and at no additional cost. The composition of the MPS team will be determined by MPS. The teams will work collaboratively to complete a detailed MPS Implementation Plan.

Dr. Jason K. Feld will serve as the ATI Project Director for the MPS Initiative. Dr. Feld has decades of experience in project management. He has successfully managed large-scale federally funded projects, state-funded projects and many large-scale initiatives in school districts. His knowledge and skill in project management and in educational research and practice will make a significant contribution to the success of the MPS initiative. Dr. Feld will coordinate ATI partnership team activities with MPS as well as work directly with MPS throughout all phases of the Galileo K-12 Online implementation planning, roll-out and implementation, technical and service support, and professional development. Dr. Feld received a M.A. in Psychology from New York University and his Ph.D. in Educational Psychology at the University of Arizona. His research and professional activities in Pre-K and K-12 education, assessment, policy and practice span 30 years. As ATI Vice President Corporate Projects, Dr. Feld is responsible for supporting ATI-client partnerships designed to enhance student achievement as well as coordinating ATI activities with state departments of education, universities, and ATI educational partners such as WestEd. Dr. Feld is a published author in

Project success will be achieved using Galileo® innovations and ATI integrated services related to: Professional Development and Training, Information Technology, User Support, Field Services, Educational Management, Assessment and Instructional Design, and Research and Development. Moreover, providing Galileo as an online service allows ATI staff to take care of all application management and monitoring tasks. These include, but are not limited to, system use, bandwidth use, application server performance, and database efficiency. Galileo also includes an array of online Administrative Reports for use by ATI and MPS in managing implementation. The reports facilitate scheduling, tracking student completion of assessments and online Instructional Dialogs, real-time monitoring of student progress during assessment administration, and user activity. Finally, because Galileo is web-based, MPS benefits both from what is available now and from enhancements made to Galileo, generally on a monthly basis and at no additional charge. Enhancements are immediately available to all users upon their next log in to Galileo K-12 Online.

Implementation Support Resources Throughout MPS implementation of Galileo, ATI’s Field Services and Technical Support staff are easily accessible and provide exceptional proactive service and technical support Monday through Friday between 8:00 a.m. to 5:00 p.m. MST, at no additional cost. Toll-free phone support is staffed, not auto attendant-based. A support session may include file review, desktop sharing, or remote control of the user’s computer. ATI maintains round-the-clock emergency assistance including 24-hour server response capability in the event of service outage. ATI’s Comprehensive Suite of Resources to support implementation includes:

• Galileo Education Publications and ATI Research Resources http://www.ati-online.com/galileoK12/K12Research.html Composition of a Comprehensive Assessment System http://www.ati-online.com/pdfs/researchK12/CompositionComprehensiveAssessmentSystem.pdf Forecasting Statewide Test Performance and Adequate Yearly Progress from District Assessments http://www.ati-online.com/pdfs/researchK12/ForecastingStatewideTestPerformance.pdf Item Development and Assessment Construction Guidelines for Common Core State Standards http://www.ati-online.com/pdfs/researchK12/ItemDevelopmentAssessmentConstructionGuidelines- CCSS.pdf Technology Enhanced Items for Formative Assessments in Dialogic Curriculums http://www.ati-online.com/pdfs/researchK12/TEItemsforFormativeAssessmentsinDialogicCurriculum.pdf • Online Support Request Form: http://www.ati-online.com/forms/contactustech.asp • Email Support: [email protected] • Best Practices Resources

Videos http://www.ati-online.com/galileoK12/K12VideoLibrary.html#Videos Webisodes http://www.ati-online.com/galileoK12/K12VideoLibrary.html#Webisodes Webinars & Dialogs http://www.ati-online.com/galileoK12/K12VideoLibrary.html#WebinarsDialogs Learning On-Demand http://www.ati- online.com/galileoK12/K12VideoLibrary.html#LearningOnDemand • Professional Development Resources http://www.ati-online.com/galileoK12/K12ProfessionalDevelopment.html • System Requirements and Technical Support Resources http://www.ati-online.com/galileoK12/K12TrainingSystem.html • Galileo® Features and Implementation Resources http://www.ati-online.com/galileoK12/indexK12.html • Links to Galileo News and the Galileo Educator Community http://ati- https://www.facebook.com/AssessmentTechnologyIncorporat online.com/ati/ATIWebinars.asp ed

http://townhallblog.ati- https://www.youtube.com/AssessmentTechnology online.com/

Use of the ATI Partnership Model to Support MPS Implementation The ATI Partnership Model and its focus on the Instructional Improvement Cycle figure prominently in the initial and ongoing implementation of Galileo K-12 Online in any school district. For this reason, the Partnership Model and Instructional Improvement Cycle are described in some detail in this section. The ATI Partnership Model engages ATI staff in collaboration with administrators and staff from a district. This model is at the very core of ATI’s commitment to build a collaborative, responsive, and enduring relationship with each school district partner. ATI has found this approach to be highly effective in supporting district-wide implementation of a standards-based, research-supported online assessment system for the purpose of improving student achievement. Within the context of the Partnership Model, ATI staff members engage each client district in responsive collaboration to address current district needs and plan for future district needs. This makes it possible for ATI to continually adjust services and expand technology capabilities in response to changing needs within and across districts.

ATI Instructional Improvement Cycle The ATI Instructional Improvement Cycle is a key approach that ATI implements in partnership with districts to help ensure successful implementation of Galileo K-12 Online. The cycle links goal-setting and planning with implementation and evaluation in a continuous iterative cycle for the purpose of delivering, managing, evaluating and refining standards-based education for the purpose of elevating student achievement. The first phase of the Instructional Improvement Cycle involves collaborative goal-setting, planning and start-up activities between MPS and ATI

Galileo® K-12 Online Instructional Improvement and Instructional Effectiveness System Prepared for: Mesa Unified School District No.4 RFP #15-47MP Formative Assessment – February 19, 2015 Copyright © Assessment Technology, Incorporated 2015. All rights reserved. 62 Tab 2. Scope Tab 2.3. Solution in preparation for implementation of Galileo®K-12 Online. These include, for example, design and delivery of professional development to identified groups, initial importation of the district student database, existing assessments, and prior assessment data into Galileo, scheduling import updates, and the collaborative development of a customized comprehensive assessment strategy. The second phase is the actual implementation of Galileo K-12 Online and associated MPS-ATI monitoring and management activities to support implementation. The third phase Figure 5 is evaluating implementation ATI Instructional Improvement Cycle impact on learning. Within the Instructional Improvement Cycle, the evaluation phase provides MPS with critical information to inform planning in the next cycle of implementation including goal-setting and planning of instruction and intervention.

ATI’s experience in working with numerous school districts suggests that use of the Instructional Improvement Cycle not only helps to facilitate and sustain implementation, but also provides a vehicle for building consensus and user adoption within the district. The cycle supports consensus building activities at the very start of the initiative by encouraging a common understanding of the importance of school and district efforts to improve student learning and the important contributions that can be made to this effort by each stakeholder in the district.

2.3.5. At a minimum, the project plan must include a milestone chart including tasks (for all phases of the project) to be performed, the timeframe and proposed staff member(s) designated for the completion of each task. This section presents a detailed suggested project plan including a description of implementation milestones, a schedule for implementation, and information about the proposed roles of MPS and ATI staff. The suggested MPS Implementation Plan that follows has been effectively deployed in school districts throughout Arizona and in other states and can effectively facilitate MPS goals as described in the RFP. As is standard practice, final details regarding implementation goals, procedures, timelines and responsible staff will be guided by planning discussions between MPS and ATI. It is anticipated that the implementation of Galileo K-12 Online by MPS will not differ substantially from other large scale implementations and will be

• Professional Development and Training Services • Information Technology Services (e.g., importation of external test data and SIS data) • User Support, Field Services, Educational Management Services (e.g., district- wide assessment design and delivery) • Assessment and Instructional Design Services (e.g., Item banks and Instructional Dialogs) • Research and Development Services (e.g., system enhancements and psychometric analysis of MPS assessments and research initiatives in collaboration with MPS)

Under typical circumstances, most districts that are just beginning to implement Galileo® K-12 Online work with ATI to create a balanced and achievable Implementation Plan aligned to district goals and that optimizes the effective and efficient use of Galileo technology to enhance student learning. In this regard, ATI will work closely with MPS staff to facilitate implementation of Galileo.

Implementation Milestones ATI has a great deal of experience in successfully implementing Galileo K-12 Online in school districts, and over time has developed a prototype Implementation Plan. The plan provides a blueprint that reflects the sequence of professional development and other support activities that most frequently support a successful implementation. The ATI approach to year one implementation of Galileo K-12 Online involves achieving three milestones that will contribute to a successful rollout and sustained implementation over the short- and longer term.

• Year One Implementation Milestones The first step of implementation is the identification of district resources, goals and challenges, and the customization of an Implementation Plan that best serves the District’s needs. Consequently Milestone One focuses on goal-setting and planning. Objectives to be accomplished in Figure 6 Implementation Milestones

Milestone Two encompasses the rollout and initial implementation of Galileo K-12 Online. Objectives to be accomplished in reaching this Milestone include: 1) finalizing and implementing procedures related to Galileo K-12 Online agreed upon during Milestone One; 2) conducting initial Galileo professional development including on-site Phase One professional development; and 3) creating and administering the initial set of district-wide assessments. In reaching Milestone Two, ATI and MPS continue to optimize and fine-tune the Implementation Plan and strategies as necessary. All Galileo initial rollout activities are scheduled to be completed within thirty days after contract award except migration of assessment content and historical assessment data from the District’s student information system which will be completed efficiently and in a mutually agreed to timely fashion.

Milestone Three represents ongoing implementation of Galileo® K-12 Online. In reaching this milestone, ATI and MPS continue to support stakeholders in maximizing the potential of Galileo K-12 Online for assessment, reporting, and instructional planning. ATI staff continues to provide ongoing support including ATI on-site Phase Two professional development and web-based ongoing professional development if desired by the District. At the end of year one, ATI and MPS conduct a collaborative review of implementation and finalize the Implementation Plan for the next year. The details of implementation can easily be documented in the Galileo K-12 Online Implementation Plan. Use of the online Implementation Plan helps to ensure that information about the District initiative including goals, tasks, timelines, roles, and responsibilities is easily recorded and accessible to District stakeholders. Moreover, in those instances where adjustments are made to the plan, the online Implementation Plan provides a mechanism for rapid dissemination of these changes. The ATI Implementation Team will work collaboratively with the District to complete the Implementation Plan at the beginning of the term of the contract between the District and ATI. Collaboration is essential for determining project goals, planning professional development offerings, establishing timelines, responsibilities and expected outcomes.

Suggested Implementation Plan for Year One The following outline will be collaboratively developed into a formal Implementation Plan.

1. GOAL SETTING & PLANNING: [FIRST FEW WEEKS OF CONTRACT] • START-UP and PROJECT MANAGEMENT TASKS (PM, FS, IT; MPS) •Identify ATI and MPS Implementation Teams and contact information •Create administrative login/password for MPS •Review Scope of Services with MPS and set goals & timeliimes •Determine needs related to integration with other systems (e.g.,Canvas, Versifit)

•PROFESSIONAL DEVELOPMENT TASKS (PD, FS, PM, MPS) •Finalize Phase 1 and Phase 2 professional development plans as outlined in the RFP by MPS including participants, topics for trainings •Identify District needs for Community Assessment and Item Banking initiative training to develop high-quality items and tests in non-state-tested areas

•ASSESSMENT, REPORTING, INSTRUCTION, CURRICULUM TASKS (EMS, FS, PD, PM, MPS) •Confirm strategy for new Galileo® and current MPS assessments •Identify subjects/dates for online, plain paper, clicker administration •Confirm Galileo reporting, curriculum & instructional tools to be used

2. ROLLOUT •PROJECT MANAGEMENT TASKS [ONGOING] (PM, FS, MPS) •Evaluate implementation status on an ongoing basis

•PROFESSIONAL DEVELOPMENT TASKS [MPS DETERMINED] (PD, FS, MPS) •ATI-District planning for MPS Phase 1 on-site Train-the-Trainer training •Plan, schedule and implement systemic districtwide Phase 1 training

•DATA MANAGEMENT TASKS [WITHIN 30 DAYS OF START-UP] (FS, IT, MPS) •Complete initial data importation (e.g., Synergy, state test scores) •Complete initial setup of staff and student logins and passwords •Import district created tests [MUTUALLY AGREED TO TIMELINE] •import/export activities with Versifit, Canvas through GDI and GDE [MUTUALLY AGGREED TO]

•ASSESSMENT TASKS - ATI Comprehensive and Customized [VARIES AS PER FOLLOWING] (FS, EMS, AID, RD, MPS) •Prepare initial District-wide assessments [6 WEEKS PRIOR TO TEST DATE] •Schedule/administer initial assessments [MPS DETERMINED] •Create/administer teacher and school level tests [MPS DETERMINED]

3. IMPLEMENTATION, MONITORING, MANAGEMENT •PROJECT MANAGEMENT TASKS [ONGOING] (PM, FS, MPS) •Monitor and optimize implementation through all ATI Departments •Conduct end-of-year review and develop plans for year 2

• TRAINING/PROFESSIONAL DEVELOPMENT TASKS [MPS/ATI] (PD, FS, MPS) •Conduct Phase 2 Train-the-Trainer professional development (reports and instruction) •Plan, schedule, and implement systemic district-wide Phase 2 training •Ongoing professional development via ATI web-based resources

• TECHNICAL AND DATA MANAGEMENT TASKS (FS, IT, MPS) •Update student information data as needed [ONGOING]

•ASSESSMENT, REPORTING, INSTRUCTION, CURRICULUM TASKS (FS, EMS, AID, RD, MPS) •Continue to create and administer comprehensive assessments and use reporting, instruction and curriculum tools to achieve MPS Goals [ONGOING]

Key: ATI Project Manager (PM), ATI Field Services, (FS), ATI Information Technology (IT), ATI Professional Development (PD), Research and Development (RD), Educational Management Services (EMS), Assessment and Instructional Design (AID); MPS Team (MPS)

Ongoing Implementation Monitoring Once implementation begins, a number of Galileo® K-12 Online reports will be available to MPS at any time to assist in easily managing and monitoring implementation of Galileo. For example, data uploads into Galileo, such as uploads of data from the District student information system and external test data such as AIMS and AzMERIT assessment scores (when available), are coordinated between the District and ATI IT staff via the online Data Upload Planner. In addition, an email is automatically sent to designated district staff when a data upload is completed.

Galileo® K-12 Online Data Upload Planner – Confidential Screen Shot

MPS personnel may verify the content of data imported into Galileo at any time by using the online Data Checker – Form Information Report. The Data Checker – Form Information Report enables the user to select a specific data set and see how many records contain the pertinent information in each field in the dataset.

Galileo K-12 Online Data Checker Form Information Report – Confidential Screen Shot

Once implementation has proceeded to the point where assessments are being administered via Galileo® K-12 Online, the Testing Activity Report will enable District and school administrators to monitor the administration of assessments within Galileo. The Testing Activity Report indicates the percent of students who have taken a set of assessments they were scheduled to take. The data are reported at the school and classroom levels so that administrators may follow-up with teachers that are falling behind in test administration.

Galileo® K-12 Online Testing Activity Report – Confidential Screen Shot

In order to further manage the implementation of Galileo K-12 Online within MPS schools, the Galileo User Activity Report enables administrators to monitor how frequently Galileo K-12 Online is being accessed. The User Activity Report is illustrated in the following screen shot. Please note that this is a report of real user activity, and so a great deal of image blurring was necessary to protect the privacy of the teachers and administrators captured in the screen shot.

Galileo® K-12 Online User Activity Report – Confidential Screen Shot

Tab 2.4. Training Training shall be conducted in two phases for Mesa Schools’ staff: Provide a full description of all training phases and identify all costs associated with each. 2.4.1. Phase One: On-site training during installation and implementation of the product. On-site training for five (5) Technology Services staff for skills transfer should include system operation, maintenance trouble shooting and software usage. On-site training for 25 testing and curriculum staff for skill transfer should include district setup, assessment design and implementation and data analysis training.

ATI provides Galileo users with a variety of professional development and training opportunities. Professional development options include on-site large-group professional development sessions at district-provided sites equipped with an Internet-connected computer lab. Additional professional development options include web-based small-group live and recorded webinars as well as live and recorded sessions on popular topics as part of the Learning On-Demand series. District staff can also access pre-recorded video tutorials and quick reference guides through the Professional Development Forum (discussion board). The Galileo interface also includes a direct link to help files designed to guide users through a variety of tasks. ATI offers on-site professional development sessions at $2,000 per six-hour session including trainer travel expenses. ATI also provides small-group webinars with mutually agreed frequency throughout the contract period at no additional cost. All Galileo users also receive access to all of the other web-based professional development options described in this section at no additional cost. Please see the ATI Detailed Cost Proposal in Tab 3.2 for further details.

This section begins with an overview of ATI’s on-site and web-based professional development offerings including the components included in the Galileo K-12 Online professional development program. This section concludes with a proposed professional development and training plan for MPS which includes two phases of training providing training for the identified staff on the desired topics.

ATI On-site Professional Development Offerings On-site professional development sessions are intended to prepare staff including administrators, lead staff, district trainers, teaching staff, IT department staff, and other personnel to utilize Galileo® K-12 Online to its fullest potential. Each on-site professional development session typically accommodates up to 45 participants in the district’s computer facilities. ATI typically provides professional development and training within a Train-the-Trainer model. Consistent with this model, ATI will instruct designated school contacts and district-level trainers

Galileo® K-12 Online Instructional Improvement and Instructional Effectiveness System Prepared for: Mesa Unified School District No.4 RFP #15-47MP Formative Assessment – February 19, 2015 Copyright © Assessment Technology, Incorporated 2015. All rights reserved. 71 Tab 2. Scope Tab 2.4. Training along with lead staff on how to present Galileo K-12 Online to teachers and other district users. Galileo’s user-friendly features and easy-to-understand applications facilitate the successful utilization of a Train-the-Trainer model. During on-site professional development sessions, each participant also receives a detailed professional development session-specific manual covering the delivered topics. ATI has professional development manuals that are not only specific to the agenda topic, but also materials that are specific to the level user (i.e., class level, school level, district level). Additionally, district trainers are provided electronic copies of all the materials and receive permission to copy the full set of ATI copyrighted professional development materials to distribute in the trainings that they provide for the district. As the Galileo application is enhanced, and as the training materials are updated, trainers are provided with the revised versions of materials.

The Galileo professional development program includes a variety of topics designed to prepare testing staff, curriculum staff, and lead staff (e.g., trainers, teachers, administrators) to take advantage of the full capabilities of Galileo K-12 Online. These topics provide training on all aspects of Galileo implementation including district setup, assessment design and implementation, and using the data and reports to improve instruction at the class, school, and district levels. There are eleven different professional development components that form the core Galileo® K-12 Online professional development program. Please note that the components are customized to meet the needs of the participants being trained.

TABLE 1 Professional Development Components

Galileo® K-12 Online Professional Development Components

Component 1: Assessment Planner and Test Review Learn how to use the Galileo Assessment Planner to develop customized district benchmark assessments aligned to local standards, Common Core State Standards and individual state versions, and district pacing guides. Learn how to use Galileo tools to review customized district-wide assessments. Component 2: Online, Response Pad, and Offline Testing Learn and practice the steps involved in both monitoring student rosters and in scheduling, administering, taking, and scoring tests. Learn to administer assessments online or through the use of response pads. Learn to use Scanline for offline testing with online test scoring and data aggregation. Component 3: Building Formative Assessments Construct and auto-generate classroom assessments aligned to instruction and local standards or Common Core State Standards and individual state versions with Quiz Builder and Test Builder tools. Use ATI’s formative item banks and/or build a district’s library of formative test items for sharing across schools, grades, and classes. Component 4: Community Assessment and Item Banking (CAIB) Initiative Participate in three professional development modules supporting local educational communities in the development and sharing of assessment items and assessments for non-state-tested subject areas. Develop and share both assessment items and assessments for use in evaluating student capabilities in subject areas not tested on the statewide assessment (e.g., art, music, foreign languages). Access a continually growing repository of shared, district-written, high-quality items and customized assessments in areas not currently addressed on statewide tests. Use Galileo Bank Builder technology to develop assessment questions based on specifications to guide the construction of standards-aligned selected-response, constructed-response, and essay items. Learn how to create district review teams to vet district-written items and how to use the Galileo Bank Builder tools to facilitate item review. Receive item review feedback on district-created items, from the experts at ATI. Component 5: Generating Reports and Using Data Get actionable data through easy-to-use Galileo Dashboards and reports providing rapid access to student achievement information. Learn how to analyze student, class, school, and district data. Learn how to use assessment data to aid decision-making to inform instruction. Monitor the implementation of district, school, and classroom assessments and instruction. Component 6: Curriculum Learn how to build your local Common-Core-aligned curricula using ATI Instructional Unit and Instructional Dialog technology. Use Galileo tools to create assignments, interactive Instructional Dialogs, lesson plans, and unit plans which link assessment of state and Common Core State Standards to instruction. Learn to deliver Instructional Dialogs as independent activities and teacher-managed lessons. Learn how to monitor the implementation of your local Common-Core-aligned curricula using Galileo's curriculum monitoring tools.

TABLE 1-Continued Professional Development Components – Continued

Galileo® K-12 Online Professional Development Components

Component 7: Implementing Interventions Gain an understanding of how Galileo may be used to implement interventions at the class-, school-, and district-level, including Response to Intervention initiatives. Become familiar with the ways in which Galileo automatically identifies intervention groups. Learn how to create targeted intervention groups. Use Galileo tools to manage and evaluate the effectiveness of interventions and curricula. Component 8: Instructional Effectiveness Learn how to complete the Proficiency Rating Scales when conducting class and site observations. Become familiar with the various categories and elements that make up your district’s compiled staff evaluation. Learn how to access and analyze staff instructional effectiveness results including the Score Compiler Summary Report, Proficiency Rating Scale Results Report, and Student Growth and Achievement Report data. Component 9: Tips for District Professional Development Instructors Review of all components of the Galileo system with special emphasis on professional development instruction and supporting others. Component 10: Lead Staff Summarize how to quickly get started with Galileo. Complete the Implementation Checklist, begin Galileo rollout, and review the district's instructional plan for Galileo. Component 11: Grading: Gradebook and/or Report Cards Gain an understanding of the web-based Gradebook's features. Become familiar with printing reports for parent conferences. Learn how to record grades and comments for midterm and end-of-period reports.

The basic topics of the professional development program are broadly organized into two phases of on-site professional development. Phase One professional development offerings are focused on supporting the initial implementation of Galileo and providing users throughout the district with the capability to design and administer district-wide (e.g., benchmark) assessments using the methods that best fit the district’s current goals and technological capabilities. Other suggested topics for Phase One include online testing, offline testing, and testing with response pads (i.e., “clickers”) as well as tips for trainers and lead staff to ensure successful district-wide implementation and training.

Sample Phase One objectives include:

• Learn how to design and create district-wide assessments (e.g., using ATI’s Assessment Planner and Test Review tools) • Learn how to administer district-wide assessments online, offline, and using handheld responders. • Prepare and plan for district-wide training of staff. • Evaluate current state of implementation.

Phase Two professional development offerings are focused on providing administrators and teachers with the tools to maximize the potential of Galileo® K-12 Online as an instructional improvement and instructional effectiveness system. Suggested topics include generating reports to inform instructional decisions, scheduling Instructional Dialogs, scheduling and administering formative classroom assessments, and using Intervention Groups to plan targeted and differentiated instruction.

2.4.2. Phase Two: Four days of On-site training on how to use the data and reports to improve instruction. It is anticipated that this training will not occur until after the first wave of district assessments.

Sample Phase Two objectives include: • Learn how to use assessment data to improve instruction and aid decision-making. • Learn how to implement Instructional Dialogs. • Learn how to utilize intervention groups for instructional and assessment purposes. • Learn how to create additional assessments utilizing district-created and community items.

If the District desires, the District may also choose to participate in the CAIB initiative to support district assessment of students in non-state-tested areas. In this session, ATI would invite the Galileo district leads and content specialists to attend the training. The professional development session covers the development of item specifications to guide the construction of standards- aligned items (e.g., selected-response, constructed-response), as well as use of Bank Builder technology to support peer review and certification of items for use in assessment. Participants in the CAIB initiative gain access to the repository of more than 9,000 CAIB items in a broad range of subject areas that have been created by educators across districts that have received the same in-depth training. All of the items are certified and aligned to standards of the state in which the items were developed. Participating districts may map items to other state or local standards as desired.

ATI Web-based Professional Development Offerings ATI provides ongoing web-based professional development in a variety of formats including webinars, the Learning On-Demand series, the Professional Development Forum including access to the Resource Library, and the Galileo Help interface.

• Webinars Webinars are web-based seminars and tutorials. Both live and recorded small-group webinars are provided to district staff members by ATI and are complimentary. The district, ATI Field Services, and other ATI departments mutually determine webinar topics and frequency throughout the school year. As consistent with all of ATI’s professional development offerings, the content of the webinars focuses on the specific and unique needs of the audience members.

Webinars are an ideal way to offer short, online training sessions to staff during the professional day. Webinars provide guided tours of basic and advanced Galileo tools. Participants are encouraged to ask questions and explore the real-life implications of the tools. Prior to an online webinar session, ATI Field Services Coordinators will provide complete setup instructions to the district to ensure that all technical requirements are in place for a smooth professional development program.

• Learning On-Demand Series The ATI Professional Development Department offers complimentary online Learning On- Demand live and recorded sessions to assist educators in becoming proficient users of Galileo K-12 Online technology. Sessions cover multiple topics of interest to a variety of audiences - ranging from information services staff to curriculum coaches. Sessions are offered on multiple dates and times, and recorded for “on-demand” access by district staff. Examples of session topics include: Data Importation for New Users, Implementing Assessments, Student Enrollment, and Accessing Professional Development Resources.

• Professional Development Forum and Resource Library All Galileo® end-users have access to the online Professional Development Forum. This Forum is a discussion board that allows teachers, curriculum specialists, site administrators, district administrators and all other Galileo end-users, to post questions, share experiences and expertise, connect with other educators, and make suggestions to ATI for product development. The Forum is divided into a number of categories, such as Assessments, Data Management, Dialogs, and Reports, and is moderated by ATI staff members who respond to messages and post helpful information.

Within the Professional Development Forum, Galileo users can also access the Resource Library containing quick reference guides and recorded video tutorials to assist users in performing a wide variety of tasks within Galileo. Quick reference guides are brief documents

Sample video tutorial from the Resource Library – Confidential Screen Shot

• Galileo® Help Interface All Galileo users have direct and immediate access to the help interface within Galileo K-12 Online by simply clicking on the Help link at the top of the screen. Once the help interface has been accessed, users can either select the desired help file or search by keyword. A sample help file is illustrated in the following screen shot.

A sample Galileo® K-12 Online help file – Confidential Screen Shot

Proposed Professional Development and Training Plan for MPS ATI proposes a plan for on-site and web-based professional development and training that will enable MPS to successfully meet the goals expressed in the RFP. As described previously and consistent with the approach desired by MPS, ATI proposes to provide professional development and training in two phases. During initial implementation planning, ATI Professional Development and Field Services staff members will work with MPS administrators to plan the timing of training, confirm appropriate trainees (e.g., Technology Services staff, testing staff, curriculum staff, trainers, administrators), and confirm the topics to be covered during training. In this way a training schedule will be designed that best supports the rollout of Galileo throughout the District as the year unfolds. Once the desired professional development components are identified, an on-site and webinar professional development schedule and specific agendas will be determined mutually by ATI and MPS. Prior to either an on-site or online session, ATI Field Services Coordinators will provide complete setup instructions to MPS to ensure that all technical requirements are in place for a smooth professional development program.

• Phase One Training (Year One) Training MPS Training Training Training Topics Phase Participants Format Phase One Technology Services Phone, email, Development of importation/exportation staff small-group strategies, installation of Scanline and Mercury webinar, on-site software (if desired), troubleshooting, software visit (if desired) usage.

Please note: As a vendor-hosted system, all upkeep and maintenance of Galileo are conducted by ATI. Phase One Testing staff, curriculum Six-hour on-site Customized Component 1 and 2 training staff, two lead staff from sessions focused on district setup, assessment design each site and implementation, online/offline/response pad administration, data analysis training.

As part of Phase One training, ATI IT and Field Services staffs will provide guidance and training for District Technology Services staff. Proposed training topics include the development of strategies for the importation of data (e.g., student enrollment information, student demographic information) into Galileo from District systems (e.g., Synergy student information system) as well as the development of strategies for the exportation of data from Galileo to District systems (e.g., District Versifit data warehouse, Canvas learning management system). Other topics may include installation of Scanline and Mercury software (if desired), trouble shooting, and software usage. Since Galileo is a vendor-hosted system, all upkeep and maintenance of the software is performed entirely by the ATI Development team without interruption of service to members of the district’s educational community. In ATI’s experience, training for District Technology Services staff is typically easily accomplished via phone, email, and/or live webinar; however, if desired, ATI staff can also conduct an on-site visit.

In year one, ATI also proposes to provide MPS with four six-hour Phase One on-site professional development sessions. Proposed Phase One training will include customized training related to Components 1 and 2 of the Galileo professional development program. This training will focus on topics related to district setup, assessment design and implementation, online/offline/response pad administration, and data analysis. The proposed sessions will provide six hours of Phase One training for MPS testing and curriculum staff. Based on ATI’s experience with Districts implementing initiatives similar to the MPS Formative Assessment initiative, ATI also proposes to provide Phase One training to approximately two lead staff per site (e.g., administrators, lead staff designated as trainers). This will ensure that MPS staff members at each site have the skills needed to administer the first set of district-wide assessments. If desired, District Technology Services staff can also attend these on-site professional development sessions. Consistent with the Train-the-Trainer model, this approach will ensure that an adequate number of MPS staff members including staff members at each site have the skills needed to administer the first set of district-wide assessments and provide additional training and support to teachers and other end- users as needed.

• Phase Two Training (Year One) Training MPS Training Training Training Topics Phase Participants Format Phase Two Testing staff, curriculum Six-hour on-site Customized Component 5 training focused on staff, two lead staff from sessions using Galileo data and reports to improve each site instruction at the class, school, and district level, implementing interventions.

In year one, ATI proposes to provide MPS with four six-hour Phase Two on-site professional development sessions. These sessions will be conducted after the first wave of district-wide assessment administration. Proposed Phase Two training will include customized training related to Component 5 of the Galileo professional development program. This training will focus on topics related to using Galileo data and reports to improve instruction at the class, school, and district level as well as implementing interventions. The proposed sessions will provide six hours of Phase Two training for MPS testing and curriculum staff as well as two lead staff per site (e.g., administrators, lead staff designated as trainers). If desired, District Technology Services staff can also attend these on-site professional development sessions. Consistent with the Train-the- Trainer model, this approach will ensure that an adequate number of MPS staff members including staff members at each site can provide training and support to teachers and other end- users related to the use of Galileo data and reports to improve instruction.

• Additional Training Throughout the Contract Period In addition to year one training, one six-hour on-site professional development session will be provided in each subsequent year to accommodate new staff and/or extend the skills of returning staff. Throughout the life of the contract, ATI will also provide ongoing web-based professional development in a variety of formats (i.e., small-group webinars, the Learning On-Demand series, the Professional Development Forum including access to the Resource Library, and the Galileo® Help interface). District staff members will also receive ongoing support from ATI’s Field Services Coordinators, Education Management Services staff, Information Technology staff, and Technical Support staff.

Tab 2.5. Questionnaire 2.5.1. If the proposed solution is “district hosted” the Offeror must specify all hardware requirements necessary to run the software being proposed. Although hardware cost may not be a part of this response, the Offeror must define, in detail, the hardware platform and system architecture required to robustly implement its solution. The Offeror must define its overall approach to hardware implementation and scalability. If the proposed solution is “vendor hosted” (ASP) hardware specifications will not be a required part of your response. Galileo® is offered as a web-based application hosted on ATI servers. A distinct advantage of a vendor hosted, web-based solution is that no hosting hardware costs are incurred by the district. In addition, ATI IT staff members handle all issues related to hosting hardware implementation and scalability.

2.5.2. The Offeror must define the recommended, not minimum, desktop requirements to support its proposed solution. The proposed software must be scalable to provide expandability and growth to the District as a whole. The proposed software must function in the District’s existing environment. Workstations will be windows based, operating over a high speed Ethernet switched VLAN routed wide area network. This section provides recommended system requirements related to Galileo including supported browsers, supported mobile platforms, requirements for online assessment administration, and requirements for assessment administration using handheld wireless devices. The architecture of Galileo includes the use of clustered application servers in order to maximize scalability and minimize down time. Data storage scalability is also built to accommodate future needs without service disruption. Additionally, the IT Technical Services team ensures the scalability of all systems to support increasing data needs and users. Based on the description of the District’s existing environment in the RFP, Galileo is anticipated to function in the existing environment.

Galileo K-12 Online Supported Browsers As a web-based application, Galileo system requirements are based on the supported web browsers. Information about Galileo supported web browsers is provided in the following table along with information about the computer operating systems supported by each browser. For the convenience of the District, links to a full list of manufacturer-published operating system requirements for each compatible browser are provided in the electronic version of the proposal

TABLE 2 Galileo® K-12 Online System Supported Web Browsers Supported Web Supported Manufacturer-Published Operating Browsers Operating System System Requirements Link MS Internet Explorer http://windows.microsoft.com/en-US/windows- Windows 8

11 8/system-requirements http://windows.microsoft.com/en- Windows 7 us/windows7/products/system-requirements http://windows.microsoft.com/en-US/windows- MS Internet Windows 8 8/system-requirements Explorer 10 http://windows.microsoft.com/en- Windows 7 us/windows7/products/system-requirements http://windows.microsoft.com/en- MS Internet Windows 7 us/windows7/products/system-requirements Explorer 9

Windows Vista SP2 http://support.microsoft.com/kb/919183

http://windows.microsoft.com/en-US/windows- Mozilla Firefox Windows 8 8/system-requirements (current version) http://windows.microsoft.com/en- Windows 7 us/windows7/products/system-requirements

Windows Vista SP2 http://support.microsoft.com/kb/919183

Windows XP SP3 http://support.microsoft.com/kb/314865

Mac OS X 10.8 http://support.apple.com/kb/SP654

Mac OS X 10.7 http://support.apple.com/kb/SP629

Mac OS X 10.6 http://support.apple.com/kb/SP575 NOTE: Extended service release (ESR) version is also supported - current ESR is 17.0esr

Apple Safari 6 Mac OS X 10.8 http://support.apple.com/kb/SP654

Mac OS X 10.7 http://support.apple.com/kb/SP629

Mac OS X 10.6 http://support.apple.com/kb/SP575

Apple Safari 5 Mac OS X 10.7 http://support.apple.com/kb/SP629

Mac OS X 10.6 http://support.apple.com/kb/SP575 http://windows.microsoft.com/en-US/windows- Google Chrome Windows 8 8/system-requirements (current version) http://windows.microsoft.com/en- Windows 7 us/windows7/products/system-requirements

Windows Vista SP2 http://support.microsoft.com/kb/919183

Windows XP SP3 http://support.microsoft.com/kb/314865

Mac OS X 10.8 http://support.apple.com/kb/SP654

Mac OS X 10.7 http://support.apple.com/kb/SP629

Mac OS X 10.6 http://support.apple.com/kb/SP575 Notes: Reporting functions may require Adobe Acrobat Reader plugin, Media functions may require Adobe Flash Player plugin, and Bank, Dialog and Test Builder tools may require Java Runtime Environment

Supported Mobile Platforms Galileo® K-12 Online can also be used on supported mobile devices running the default browser as indicated in the following.

TABLE 3 Supported Mobile Platforms Galileo K-12 Area MS Apple iPad Android Google Surface RT Tablet Chromebook and Pro Galileo K-12 Online Core Yes Yes Yes Yes Functions Student Testing Yes Yes Yes Yes Multimedia Student Test Yes No Yes No Content (Includes Early Literacy Benchmark Assessment Series) Instructional Dialogs Yes No Yes No Bank Builder Tools Yes No Yes No Test Builder Tools Yes No Yes No Dialog Builder Tools Yes No Yes No

Supported Student Response Devices Mercury is an ATI software application used with Galileo K-12 Online technology that lets students instantly send assessment and/or instructional responses from a wireless response pad to the online system. Galileo’s Mercury component allows assessments to be administered to students using a variety of student response devices. The system requirements for Mercury including a complete list of currently supported student response devices follow.

TABLE 4 Additional Requirements for Mercury Operating System Version Required Windows Vista or 7 Intel-based Mac OS 10.4 (Tiger), 10.5 (Leopard), or 10.6 (Snow Leopard)

Hardware and Software Requirements OS Version: Windows Vista, XP, 7 Mac OS 10.4, 10.5, 10.6 Hardware Required Input Device CPS IR, Pulse, Gen2 or CPS Pulse, Gen2, IR or Spark Response Pad, Spark Response Pad‡ Promethean Activote PRM- AV2-01 Response Pad w/ Receiver PRM-AH1-01, or Promethean Activote PRM- AV3-01 Response Pad w/ Receiver PRM-AH2-01 SMART Response Pads¥ Promethean ActivExpressions Response Pads † Projector Projector or shared display Software Required † Testing Application Mercury software Support Software Microsoft .Net Framework, version 2 or higher† (freeware) Support Software eInstruction CPS Software ATI-Provided CPS Device v6.70†, Promethean Activote Driver† Drivers†, or Promethean ActivExpression Drivers† SMART Response connector for Galileo¥ † Instructor computer only ‡ OS X 10.6 only ¥ More information and support provided by SMART Technologies

2.5.3. Is the proposed solution web delivered via the intranet from a centralized “district hosted” location or is the proposed solution “vendor hosted” and web delivered via the internet as an ASP?

Galileo® K-12 Online is vendor hosted and delivered over the Internet.

2.5.4. Submit a detailed implementation plan including delivery, installation, and training to be completed within first 90 days after notice of award or receipt of order. The plan should include detailed information concerning the number of personnel that will be used on this project and their resumes, along with the time table in which each section of work is to be completed. Tab 2.3.4 and Tab 2.3.5 provide ATI’s detailed project plan including a detailed suggested Implementation Plan including a proposed time table for implementation. Initial implementation discussions will begin immediately following receipt of a purchase order verifying contract award. Galileo® rollout is scheduled to be complete within 30 days following contract award except for the importation of existing assessment content which will be completed within a timeline mutually agreed to by the District and ATI.

ATI brings the full expertise and experience of the entire company to each client district. As of this writing, ATI employs 50 professional staff, with 88 percent holding advanced degrees and 46 percent possessing an M.S., an M.A., or a Ph.D. Resumes for key ATI leadership staff are provided in Tab 1.1.

2.5.5. Explain in detail the implementation process. ATI’s approach to implementation of Galileo and the details of the implementation process are described in detail in Tab 2.3.4 and Tab 2.3.5.

2.5.6. Show the name of the software package being proposed and state whether it is an ASP solution or a district hosted solution: ATI proposes the Galileo® K-12 Online Instructional Improvement and Instructional Effectiveness System as a solution for the MPS Formative Assessment initiative. Galileo is provided as a web-based application hosted on ATI servers (i.e., an ASP solution).

2.5.7. Application Service Provider (ASP-Vendor Hosted) Questions 2.5.7.A. Where is the ASP’s data center and equipment located? The primary data center is located in-house in a secured environment at ATI’s corporate offices in Tucson, Arizona and is managed by ATI IT personnel. The backup data center is hosted in another west coast city, allowing for failover to the alternate location in the event of catastrophic system failure at the primary data center or southern Arizona regional disaster. This failover capability is one strong advantage of an Application Service Provider, allowing for increased uptime without any hardware or facilities cost to the district.

2.5.7.B. What are the bandwidth requirements needed to access the ASP’s service? The Galileo® K-12 Online testing environment has been designed to ensure that the demands on a district’s bandwidth are very low. In those cases where a test question has no images, a typical page is within the range of 10-20 K. In those instances where images are included, a typical page is approximately 50 K. In those cases where images are included, page size is kept minimal by strictly controlling the size of images that are saved in the database and transmitting them in a flattened form. Because the pages are small, a single T1 connection should be able to support a typical school’s use of online testing with minimal impact on latency.

2.5.7.C. Will Mesa Schools be allowed to import key business data into the hosted application thereby providing access to it immediately? Will Mesa Schools be allowed to export key business data into District assets on a routine basis? Galileo provides importation and exportation capabilities via three modules: the Galileo Data Import (GDI) module, the Galileo Data Extraction (GDE) module, and the Student Interoperability Framework (SIF) module. ATI is experienced in assisting districts in setting up automated routines. Imports and exports can be accomplished as frequently as nightly, if desired.

Galileo Data Import (GDI) Module The vast majority of Galileo® K-12 Online clients import key business data using the GDI module, due to its flexibility and the ease with which it may be implemented. This module can be used to import student information data, external assessment data such as statewide assessment data, or any other type of data. In this process, data is extracted from the relevant system (e.g., the district student information system) by district technical personnel. The extract file contains all the information desired for upload. For example, for an upload from a student information system, the file would typically contain student enrollment and any demographic information, as well as teacher (class-level user) and class information. Data may be provided in any delimited text file. Files of this type can be readily produced by most data management

Following extraction, the file is transmitted to ATI through FTP-S and uploaded to a test database for quality assurance review prior to being loaded to the Galileo® K-12 Online database. During the import process, communication between ATI and the district ensures all desired data will be uploaded as intended – data points within the file are mapped to class, student, and teacher information within the Galileo K-12 Online database for importation. The GDI module can accommodate the addition of any desired field, either during the initial or any subsequent upload. All ATI requires to process custom form information is a file mapping the appropriate Galileo ID (StudentID, TeacherID or SectionID) to the desired new data point(s). Using the GDI module, Student enrollment and other student information system data importation and updating can be scheduled as frequently as nightly. Users can plan and monitor the importation of data directly within Galileo using the Data Upload Planner illustrated in the following screen shot.

Galileo® Data Upload Planner – Confidential Screen Shot

Galileo Data Extraction (GDE) Module Galileo K-12 Online provides the capacity for regularly scheduled exports of data to the District data system via the GDE module. The GDE module is the Galileo K-12 Online exportation process used by most clients due to its ease and flexibility. This module can be used to export data to a data management system such as a data warehouse system, or to produce a delimited file that can be stored and/or used as desired. Prior to data extraction, ATI staff work with district staff to determine the desired data for inclusion and to develop a standardized file format to ensure that all fields are mapped appropriately during importation into the desired system. The

GDE module can be used to accommodate a one-time export of data or more frequent exports such as might be desired for a district maintaining their own data warehouse. Using the GDE module, data exportation can be accomplished as frequently as nightly. Many districts utilizing data warehouses find it useful to automate the required routines underlying the data transfer process. ATI is experienced in working with districts to set up such automated routines so that the data transfer is as efficient and accurate as possible.

Student Interoperability Framework (SIF) Module ATI has developed the SIF module to support districts who desire a solution that involves the rapid import, export, and/or updating of data between Galileo® and another data management system. The School Interoperability Framework (SIF) initiative represents a significant effort within the educational software industry to address the issue of transferring information among databases. Participants in the SIF initiative have written a precise and extensive set of protocols and specifications for data transfer. These protocols can be thought of as a common language governing how data should be moved between participating software programs. Educational software businesses are also writing software modules to support and utilize this data transfer language. ATI has been SIF-compliant since 2009.

2.5.7.D. What rights does Mesa Schools have to move or copy the data housed at the ASP’s data center? Will the data given to the ASP by Mesa Schools, as well as any data produced by the ASP’s applications, remain the property of Mesa Schools? MPS users with appropriate permissions can modify, move, or copy data within Galileo databases using online interfaces. For, example, a user with appropriate permissions can move an assessment from one library to another. As described in Tab 2.5.7.C, MPS can also move data within Galileo databases or access copies of data within Galileo databases as part of routine imports and exports conducted via the GDI, GDE, and/or SIF modules. For example, a routine import from the MPS student information system can move a student from one class or school to another.

Through the ATI Electronic Services Agreement provided in Tab 8.1 and incorporated by reference herein, ATI grants Mesa Public Schools (the User) limited permission, renewable annually, to access ATI Intellectual Property. In connection with the licensing of ATI Intellectual Property, ATI may receive confidential information from User. Upon cancellation, expiration or termination of this Agreement and at the request of the User, ATI shall return to the User or destroy any confidential information belonging to the User and still in ATI’s possession. ATI has the right to use information that the User enters into the system to conduct educational research and publish educational research findings for such purposes as establishing the validity and reliability of scales provided through ATI’s Electronic Services. If User develops new tests, test items, Instructional Dialogs and/or other instructional materials that are developed by User and

2.5.7.E. Describe the ways student data will be kept secure and compliant with FERPA, COPPA and other privacy related regulations. As described throughout this response, Galileo® K-12 Online contains a variety of built-in, integrated security measures that are designed to ensure the confidentiality, integrity, and viability of online data, to meet state standards for data security and records storage, and to store, transmit, and manage data to meet Family Educational Rights and Privacy Act (FERPA) protection laws and regulations.

Key security measures and safeguards include 1) a high-security secondary data server housed elsewhere in the region, 2) multiple firewalls, 3) the implementation of SSL, 4) user authentication using globally unique identifiers (GUID), and 5) program control of staff access privileges. ATI also enforces corporate policies stressing the protection of privacy of clients and particularly children served by clients. The FERPA regulations are posted with ATI Personnel Policies that specifically state that violation of a child’s right to privacy is a violation of federal law and may result in employee termination. These security measures and policies help protect against data loss, protect against nonuser access to data, allow districts to control staff access to data, and ensure the privacy of clients and children.

Firewalls Extensive external security is in place to ensure the protection of all client data. Student information including test information is protected by encryption during transmission and stored in a database system that is not directly accessible from the Internet. Transactions with the database must be performed through the Galileo K-12 Online application servers. All client traffic to and from the database system must pass through one of two corporate gateway firewalls and one additional screened subnet firewall.

Secure Socket Layers and Data Encryption Another security measure is the implementation of Secure Sockets Layers (SSL). When a user clicks on the link to log in at ati-online.com, the entire session from this point forward (including entry of the username and password) is conducted using SSL. Every time data is transmitted to Galileo K-12 Online, the entire transaction is performed with 128-bit encryption. Only the Galileo K-12 Online web servers hold the private key to decode the encrypted data sent by users. All traffic for Galileo® K-12 Online is transferred using SSL, typically transferred through a network (including the Internet) using TCP Port 443. Galileo K-12 Online utilizes the highest level of encryption possible while still maintaining a focus on performance. Our security is verified with a top-level site certificate hosted by VeriSign, an industry leader in computer security certification.

User Authentication Utilizing GUIDS All user accounts in the Galileo K-12 Online system are assigned a GUID, as are all transactions placed in the system. This ensures the highest level of accuracy when performing data entry/retrieval within the Galileo K-12 Online system. When users access Galileo K-12 Online, their password is authenticated against their record in the system using the GUID, and entry is granted only in the event of a perfect match. No ‘fuzzy logic’ is used in authenticating passwords.

District Control of Staff Access Privileges Galileo K-12 Online users are authenticated with the standard username and password construct. All usernames are entered and passwords set by the individual clients. This can only be done once an identified district administrator has been entered into the system by ATI. This ensures greater control and accuracy for clients. When the agency administrator sets up access for a user, they will determine the level of access allowed that user. This user will then only be allowed access to their individual school-, class-, or student-level information. Similarly, each online assessment scheduled for a particular timeframe is always assigned its own password.

Enforced In-House Privacy Policies ATI has and enforces corporate policies stressing the protection of privacy of clients and particularly students served by clients. FERPA regulations are posted with ATI Personnel Policies. The personnel policies themselves specifically state that violation of a student’s right to privacy is a violation of Federal law and may result in employee termination.

2.5.7.F. What is the ASP’s proposal concerning upgrades and program improvements, concerning costs, training, implementation? Upgrades and Program Improvements ATI continually updates Galileo to address advances in educational research, online technology, educational legislation, and evolving client needs. Because Galileo technology is web-based, clients benefit not only from what is available now, but also from the enhancements made routinely available to all users as they are completed. All enhancements to Galileo are provided at no additional charge and, following release, are immediately available to all users upon their next log in to Galileo K-12 Online. Galileo K-12 Online is updated, on average, twice a month and Galileo architecture makes it possible to provide enhancements with no system down time or installation steps by the user. Updates to the system are communicated to ATI clients in a number of ways including directly on the Galileo login page as Galileo Enhancements, which are comprised of documentation of improvements and updates and guidance on how to use them.

Upgrades to Scanline and Mercury are installed by district staff. Upgrades to Scanline and Mercury are made available as free downloads from the Internet so installation is an easy process.

Costs, Training and Implementation Tab 2.3.4 and Tab 2.3.5 provide a description of ATI’s approach to implementation as well as a detailed suggested Implementation Plan. The ATI Detailed Cost Proposal in Tab 3.2 also provides detailed information about the costs related to implementation of Galileo® K-12 Online.

Galileo K-12 Online is a cost-effective solution to meet the District’s needs related to a formative assessment system. Galileo is offered as a renewable annual license at $8 per student. The annual license includes services related to setup, assessment (including ATI secure and community item banks), curriculum, grades, student/family, staff, and reports. There are no charges related to district/school staff use. All users receive at no charge, project support including implementation/project management and consultation as well as application-related support. Also provided at no charge are data analysis services (e.g., psychometric and test scaling analyses), start-up and data importation/exportation services, and application upgrades.

ATI provides on-site professional development at $2,000 per six-hour session including trainer travel expenses. ATI also offers Scanline, an electronic service facilitating offline testing and online aggregation as a one-time purchase at $460 per installation. Please note Scanline carries an 8.1% transaction privilege tax. At no charge with license of Galileo, ATI also offers Mercury, a software application that lets students instantly send assessment and/or instructional responses from a wireless response pad to Galileo. District is responsible for the purchase of all required hardware. Check with ATI Field Services staff for ATI supported scanning units and wireless response pad systems.

2.5.7.G. Describe the security plan for the ASP. What measures are in place to prevent un-authorized personnel from seeing or hacking into the system/data? There are a number of safeguards that are in place to make sure that the system is secure and the privacy of student records is protected. Elements of the security plan include methods to prevent unauthorized data access, monitoring and auditing procedures. Areas of coverage include:

Enforced In-House Privacy Policies ATI has and enforces corporate policies stressing the protection of privacy of clients and particularly students served by clients. The FERPA regulations are posted with ATI Personnel Policies. The personnel policies themselves specifically state that violation of a student’s right to privacy is a violation of Federal law and may result in employee termination. ATI Personnel Policies as they relate to privacy will be available to MPS upon request.

Security Designed to Protect Against Non-User Access to Data Extensive external security is also in place to ensure the protection of all client data. Student information including test information is protected by encryption during transmission and stored in a database system that is not directly accessible from the Internet. Transactions with the database must be performed through the Galileo® K-12 Online application servers. All client traffic to and from the database system must pass through one of two corporate gateway firewalls and one additional screened subnet firewall. Filtering rules on the back-end firewall prevent access to the data server LAN subnet by all unauthorized computers.

Multi-Level Passwords Galileo is comprised of a variety of built-in, integrated security measures that are designed to ensure data confidentially, integrity, and viability of online data. Galileo users, for example, are authenticated with the standard username and password construct. All usernames are entered and passwords set by the individual clients. This can only be done once an identified district administrator has been entered into the system by ATI. This ensures greater control and accuracy for our clients. When the agency administrator sets up access for a user, they will determine the level of access allowed that user. This user will then only be allowed access to their individual school, class, or student level information. For instance, each online assessment scheduled for a particular timeframe is always assigned its own password.

User Authentication Utilizing GUIDS All user accounts in the Galileo K-12 Online system are assigned a GUID, as are all transactions placed in the system. This ensures the highest level of accuracy when performing data entry/retrieval within the Galileo K-12 Online system. When users access Galileo K- 2 Online, their password is authenticated against their record in the system using the GUID, and entry is granted only in the event of a perfect match. No ‘fuzzy logic’ is used in authenticating passwords.

Secure Socket Layers and Data Encryption Another security measure is the implementation of SSL. When a user clicks on the link to log in at http://www.ati-online.com, the entire session from this point forward (including entry of the username and password) is conducted using SSL. Every time data is transmitted to Galileo K-12 Online, the entire transaction is performed with 128-bit encryption. Only the Galileo K-12 Online web servers hold the private key to decode the encrypted data sent by users. All traffic for Galileo K-12 Online is transferred using SSL, typically transferred through a network (including the Internet) using TCP Port 443. Galileo K-12 Online utilizes the highest level of encryption possible while still maintaining a focus on performance. Our security is verified with a top-level site certificate hosted by VeriSign, an industry leader in computer security certification.

Security Logging and Auditing Logging of security- and access-related events is recorded in several places – external and back-end firewalls, the Galileo® K-12 Online application servers, database servers, domain controllers, load balancers, email and FTP servers. All logs are reviewed regularly by ATI IT staff, with email and/or pager alerts set to alert staff immediately for high-priority events.

Physical Security ATI primary and backup data centers are protected by multifactor authentication including biometric locks, personnel security at all entrances, 24-hour security cameras, and monitored security alarms.

2.5.7.H. What is the physical security of the data center, the security of the network and the servers? Physical Security The primary data center is located at ATI corporate offices in Tucson, Arizona. Physical security in place includes single point-of-entry and biometric/combination/key locks to prevent unauthorized access. The facility is monitored by 24-hour video surveillance with security camera footage retained for a minimum of two weeks.

The failover data center is housed in a security with full-time armed security, multifactor authentication including biometric, combination and RFID key entry. ATI equipment housed at this facility is accessible only by or at the direction of select ATI personnel.

Network and Server Security All communication with the Galileo K-12 Online application is transferred via HTTPS (SSL) through fiber optic lines. Traffic is routed through redundant firewalls, with all web requests filtered through a redundant array of hardware load balancers presented to the Internet. Multiple web- and application servers are connected to the load balancers, with Galileo application server requests filtered through a backend screened subnet (DMZ) firewall to query the Galileo K-12 Online data servers.

All private connections, including offsite employee access and the backup data center, are performed through firewall-terminated encrypted VPN tunnel.

2.5.7.I. Describe the ASP’s disaster recovery plan. ATI has a catastrophic event recovery contingency plan that enables recovery within four hours. ATI utilizes a secondary data server located in a secure data center housed elsewhere in the region. The backup architecture uses automated transactional database replication. The failover server is housed in a facility with full-time security, multi-factor authentication including biometric, combination and RFID key entry and full data provider and power redundancy. ATI equipment

ATI performs regular backup of the Galileo® K-12 Online database, with multiple disk copies of the backup file stored internally on separate servers. Fifteen minute transaction log backup files are also stored on redundant internal servers. The ATI Disaster Recovery Plan (DRP) mandates online transfer of the Galileo K-12 Online transaction log backup files to the secondary data center facility described previously, updated every fifteen minutes.

In addition, the DRP includes weekly tape archival of the full Galileo K-12 Online database, stored in a secure facility separate from the location of the online transfer server. Personnel and physical security protect this facility, with magnetic media stored in a fireproof safe. With the secondary data server, data and transaction log backups and storage architecture of the primary/secondary data files and transaction logs within the Galileo data servers, ATI has planned for rapid recovery from a multitude of possible disaster scenarios.

2.5.7.J. What is the service availability of the solution? Describe downtime expectations. ATI is committed to continuing to optimize the Galileo environment to ensure high levels of service availability. The Galileo environment has been designed with multiple layers of redundancy to maximize uptime. This includes redundant storage array hardware, redundant power and redundant data server hardware in our primary data center. ATI also maintains a complete Galileo K-12 Online system at a hosting facility in another west coast metropolitan area to provide service in the event of catastrophic system failure at the primary ATI data center or southern Arizona regional disaster. ATI has maintained uptime of 99.5% for Galileo K-12 Online over the past 12 months excluding planned maintenance windows typically lasting less than 1 hour per month. Planned maintenance is typically conducted late nights on the weekend, with notice provided seven days in advance. ATI IT maintains internal- and external monitoring systems for all Galileo K-12 Online servers and receives text alerts within 3 minutes of any system incident with a maximum 1-hour response at all times.

2.5.7.K. What are the terms of the termination clause included in the Service Level Agreement? The following provides the Termination Policy as indicated in the ATI Electronic Services Agreement provided in Tab 8.1 and incorporated herein by reference.

Termination Policy • Scope: The User may terminate the Contract for ATI Electronic Services for cause in the event ATI is in default as defined in the following paragraph, by giving ATI a written

“Notice of Default.” ATI shall have thirty (30) calendar days from ATI’s receipt of a written Notice of Default to correct any default. In the event ATI fails to remedy the default within that time period, the User may send ATI a written “Notice of Termination.” Upon receipt of such Notice of Termination, ATI shall cease all work immediately or on the date, if any specified in said Notice of Termination, whichever is later, and thereafter take all reasonable steps to not incur additional costs associated with this Contract, except those costs necessary to closeout business on this Contract. Additional costs shall be reviewed by the User and must be mutually deemed allowable before incurred and payment is made. • Termination for Default: The User may terminate this Contract for reason of ATI’s default if: (a) a federal or state proceeding for the relief of debtors is undertaken by or against ATI, or (b) a receiver or trustee is appointed for ATI, or (c) ATI makes an assignment for the benefit of its creditors, or (d) ATI becomes insolvent, which shall be deemed to have occurred if ATI has ceased to pay its debts in the ordinary course of business or cannot pay its debts as they become due, whether or not ATI has committed an act of bankruptcy or is considered insolvent within the meaning of the federal bankruptcy law, or (e) ATI breaches any material term or violates any material provision of the written proposal/bid response to User’s RFP and does not cure such breach or violation within thirty (30) calendar days from ATI’s receipt of a written “Notice of Default.” ATI may terminate this Contract for any breach of the Contract by the User, including, but not limited to, the User’s failure to make payment, the User’s unauthorized use of account access information, the User’s breach of this Electronic Services Agreement including confidentiality breach, the User’s improper use of ATI’s copyrighted materials, or the User’s misuse of the interactive components of ATI’s electronic services. Prior to termination, ATI shall give the User a written “Notice of Default.” The User shall have thirty (30) calendar days from receipt of Notice of Default to correct any breach. In the event the User fails to remedy the breach within that given time period, ATI may send the User a written “Notice of Termination.” Termination of the Contract shall be effective as of the date of the Notice of Termination. ATI shall have no liability to the User for terminating the User’s access to ATI’s Electronic Services for cause.

• Please Note: o The breach by User in exceeding the scope of the licenses granted in this Agreement or by breach of ATI confidential information shall result in irreparable harm to ATI for which no adequate remedy at law exists. Accordingly, upon any such breach, ATI shall be entitled to seek injunctive or other appropriate extraordinary relief, such relief being in addition to, and not in lieu of, any other rights and remedies, including the award of damages available at law or in equity.

o The termination of this Agreement shall terminate this Agreement as a whole, including all aspects of ATI’s written proposal to User’s RFP or other requests.

• Return of Confidential Material: Upon cancellation, expiration or termination of this Agreement and at the request of the User, ATI shall return to the User or destroy any confidential information belonging to the User and still in ATI’s possession. Confidential information is considered to be any information that identifies a particular person or particular institution.

• Proprietary Rights: ATI has the right to use information that the User enters into the system to conduct educational research and publish educational research findings for such purposes as establishing the validity and reliability of scales provided through ATI’s Electronic Services.

2.5.7.L. How many dedicated employees does the ASP have? ATI brings the full expertise and experience of the entire company to each client district. As of this writing, ATI employs 50 professional staff, with 88 percent holding advanced degrees and 46 percent possessing an M.S., an M.A., or a Ph.D. The staff turnover rate is very low at ATI. At this time, 50 percent of ATI staff members have been with the company for three years or more and 36 percent have been with the company for more than 10 years. The responsibilities of all ATI staff members are focused on the various aspects of providing continuous instructional improvement services to clients.

Tab 2.5.7. District-Hosted Questions Galileo® K-12 Online is provided as a web-based system hosted on ATI servers. Since Galileo does not support a district-hosted solution, Questions M through R are not applicable.

2.5.7.M. Does your system architecture address necessary backup and recovery technology? 2.5.7.N. Does the proposed solution operate on a centralized Microsoft SQL database? 2.5.7.O. Does the proposed solution operate on an industry standard Relational Database Management System (RDBMS)? 2.5.7.P. If you are proposing a solution, with vendor provided hardware, does your proposed solution allow for remote hardware diagnostics? 2.5.7.Q. Does the proposed solution operate in a Windows 2008 or most current stable Windows Server Operating system? 2.5.7.R. Describe how software updates are made available to customers.

Tab 2.5.7. General Questions 2.5.7.S. Explain in detail the methods by which Mesa Public Schools student data will be extracted for population into your proposed solution. As described previously in Tab 2.5.7.C, Galileo® supports the extraction of MPS student data from District systems for population into Galileo. This section provides a brief overview of how most districts use the GDI module to accomplish this task; however, the Student Interoperability Framework (SIF) module can also be used, if desired. Please see Tab 2.5.7.C for further information on the SIF module.

The GDI module is the Galileo K-12 Online importation process used by most clients due to its flexibility and the ease with which it may be implemented. This module is used by ATI staff to import and update nearly five million student records annually. This module can be used to import student information data, external assessment data such as statewide assessment data (e.g., AIMS, DIBELS, AzMERIT assessment data when available), or any other type of data. In this process, data is extracted from the relevant system by district technical personnel. The extract file contains all the information desired for upload. For example, for an upload from a student information system, the file would typically contain student enrollment and any demographic information, as well as teacher (class-level user) and class information. Data may be provided in any delimited text file. Files of this type can be readily produced by most data management systems. As an example, details for specific file contents for imports from student information systems can be found in ATI’s Data Importation Instructions at http://www.ati- online.com/pdfs/ImportInstructionsK-12.pdf.

Following extraction, the file is transmitted to ATI through FTP-S and uploaded to a test database for quality assurance review prior to being loaded to the Galileo K-12 Online database. During the import process, communication between ATI and the district ensures all desired data will be uploaded as intended – data points within the file are mapped to class, student, and teacher information within the Galileo K-12 Online database for importation. The GDI module can accommodate the addition of any desired field, either during the initial or any subsequent upload. All ATI requires to process custom form information is a file mapping the appropriate Galileo ID (StudentID, TeacherID, or SectionID) to the desired new data point(s). Using the GDI module, data importation and updating can be accomplished as frequently as nightly. Users can plan and monitor the importation of data directly within Galileo using the Data Upload Planner illustrated in the following screen shot.

Galileo® Data Upload Planner – Confidential Screen Shot

2.5.7.T. Is the proposed solution completely Web based and browser driven requiring no remote clients? Galileo K-12 Online is a web-based, browser-delivered application. For this reason, in order to access Galileo K-12 Online, the District only needs to ensure client computers are running a supported browser. Upkeep and maintenance of Galileo K-12 Online software is performed entirely by ATI staff without interruption of service to members of the District’s educational community. Upgrades to Galileo K-12 Online are tested by ATI staff and accomplished through “builds” posted to the application servers by ATI staff and are immediately available to users upon their next login to Galileo K-12 Online.

As a web-based application, Galileo system requirements are based on the supported web browsers. Information about Galileo supported web browsers is provided in the following table along with information about the computer operating systems supported by each browser. For the convenience of the District, links to a full list of manufacturer-published operating system requirements for each compatible browser are provided in the electronic version of the proposal as well as on the Galileo K-12 Online system requirements page at: http://www.ati- online.com/galileoK12/K12TrainingSystem.html.

TABLE 5 Galileo® K-12 Online System Supported Web Browsers Supported Web Supported Manufacturer-Published Operating Browsers Operating System System Requirements Link MS Internet Explorer http://windows.microsoft.com/en-US/windows- Windows 8

Windows Vista SP2 http://support.microsoft.com/kb/919183

http://windows.microsoft.com/en-US/windows- Mozilla Firefox Windows 8 8/system-requirements (current version) http://windows.microsoft.com/en- Windows 7 us/windows7/products/system-requirements

Windows Vista SP2 http://support.microsoft.com/kb/919183

Windows XP SP3 http://support.microsoft.com/kb/314865

Mac OS X 10.8 http://support.apple.com/kb/SP654

Mac OS X 10.7 http://support.apple.com/kb/SP629

Mac OS X 10.6 http://support.apple.com/kb/SP575 NOTE: Extended service release (ESR) version is also supported - current ESR is 17.0esr

Apple Safari 6 Mac OS X 10.8 http://support.apple.com/kb/SP654

Mac OS X 10.7 http://support.apple.com/kb/SP629

Mac OS X 10.6 http://support.apple.com/kb/SP575

Apple Safari 5 Mac OS X 10.7 http://support.apple.com/kb/SP629

Windows Vista SP2 http://support.microsoft.com/kb/919183

Windows XP SP3 http://support.microsoft.com/kb/314865

Mac OS X 10.8 http://support.apple.com/kb/SP654

Mac OS X 10.7 http://support.apple.com/kb/SP629

If offline testing is desired, the District will also need to install Scanline software to support scanning from proprietary hi-speed scanners and multipurpose scanning units. Similarly, if testing using handheld response pads is desired, the District will need to install Mercury software to support the sending of assessment and/or instructional responses from a wireless response pad to the online system.

Due to the web-based delivery model, all server licensing is handled by ATI, with the exception of Scanline installations to accommodate offline testing and software licensing to support scanning from web/document cameras. Scanline installations require a license for installation on each machine with which it will be used. The purchasing of the appropriate licenses for Scanline software is coordinated by the ATI Field Services Coordinator assigned to the District as a standard element in the implementation of Galileo® K-12 Online.

2.5.7.U. Are plug ins required to support any portion of your proposed solution? As a browser-based web application, Galileo K-12 Online supports multiple web browsers running in desktop/notebook and tablet environments. In most cases, plugins are not required for use of Galileo K-12 Online for delivering and reporting on student online assessments. Exceptions are some reporting functions, which may require the Adobe Acrobat Reader plugin and certain media functions that may require use of the Adobe Flash Player plugin. Bank, Dialog and Test Builder functions, discussed elsewhere in this proposal, may also require the Java Runtime Environment. All required plugins are available as free downloads from their respective vendors.

2.5.7.V. If plug ins are required provide a list of all required plug ins Some reporting functions may require the Adobe Acrobat Reader plugin and certain media functions may require use of the Adobe Flash Player plugin. Bank, Dialog and Test Builder functions, discussed elsewhere in this proposal, may also require the Java Runtime Environment. All required plugins are available as free downloads from their respective vendors.

2.5.7.W. Specify ‘recommended’ (NOT MINIMUM) desktop hardware requirements. As a web-based application, Galileo system requirements are based on the supported web browsers. Information about Galileo supported web browsers is provided in Tab 2.5.7.T along with information about the computer operating systems supported by each browser. For the convenience of the District, links to a full list of manufacturer-published operating system requirements for each compatible browser are also provided in Tab 2.5.7.T as well as on the Galileo K-12 Online system requirements page at: http://www.ati- online.com/galileoK12/K12TrainingSystem.html.

2.5.7.X. Is the proposed solution compatible with Microsoft’s Internet Explorer (IE) and Google Chrome using only default installation settings? Yes. Supported browsers include Microsoft IE (version 10+), Google Chrome (Stable Release, version 35.0.1916.153 at the time of writing this proposal), Mozilla Firefox (Extended Service Release, version 30.0esr at the time of writing this proposal) and Apple Safari (version 5+). No custom browser settings are required for any supported browser.

2.5.7.Y. Indicate minimum browser level supported. Supported browsers include Microsoft IE (version 10+), Google Chrome (Stable Release, version 35.0.1916.153 at the time of writing this proposal), Mozilla Firefox (Extended Service Release, version 30.0esr at the time of writing this proposal) and Apple Safari (version 5+).

2.5.7.Z. Is the proposed solution Open Database Connectivity (ODBC) compliant? All student data within Galileo® K-12 Online is housed in a Microsoft SQL Server database. Microsoft SQL Server is ODBC compliant, however all maintenance and enhancement of Galileo K-12 Online is performed by ATI staff or directly within the web-based solution. This benefits districts by ensuring student data remains secure as a result of limited access to the data server LAN subnet.

2.5.7.AA. Does the proposed solution support all students and school sites within a single district wide database that requires no data replication for updating? Yes. All MPS students within Galileo will be stored centrally to enable district-wide student testing and reporting at the individual student, class, school and district levels. Storage of all MPS students in one database obviates the need for any data replication or updating to view reports.

2.5.7.BB. Does the proposed solution provide for the capability to transfer students’ data from one school to another based on mobility within the district? Yes. The Galileo K-12 Online enrollment functions support enrollment/disenrollment of a student from classes, schools, or the district. To ensure inter-school or class transfer records are not overwritten during data importation, it is recommended that districts effect the transfer in their Student Information System (SIS) and update the Galileo record during the upload of data.

2.5.7.CC. Does the proposed solution provide consolidated district level reports from a centralized database? Yes. Galileo K-12 Online provides a wide variety of reports that can be printed or viewed online. These reports are available as part of the Dashboard, in the reports section of Galileo K-12 Online, in the K-12 Student-Parent Center, and in other selected locations designed to provide

In addition, with the use of Form Builder, Report Builder, and Custom Test Reports, users have the option of designing their own customized reports. Various formats are available for reports including PDF, Microsoft Excel, RTF, and comma-separated/delimited files. Comma-separated or other delimited files are particularly useful for exportation into other reporting, analysis, and graphing tools such as SPSS. An overview of the major types of Galileo® reports is provided in the following table.

TABLE 6 Galileo® K-12 Online Reports Overview: Galileo K-12 Online Reports

Curriculum Monitoring Provide the ability to monitor both the progress of instruction and Reports evidence of student learning for each standard in a district curriculum.

Present highlights of data from multiple sources in one convenient location with 'At-A-Glance' Overviews direct links to more detailed reports. Provide counts and percentages regarding student performance on each of the Development Profiles standards assessed on a test. Ideal for monitoring progress in mastering specific standards. Present a series of standardized scores that summarize development at the Development Summaries individual student level or at various levels of aggregation. Ideal for monitoring progress in terms of the relative standing of a student, class, or school. Provide Developmental Level Scores for multiple tests presented on a common scale used to monitor progress and estimate mastery. These reports include two important Aggregate Multi-Test Reports forecasting tools for district-wide assessments: the Benchmark Performance Levels and Risk Levels displays.

Aggregate Multi-Test Provides counts and percentages of students at each of the statewide assessment performance levels. The user can drill down to identify the students at each of these levels Displaying Benchmark for each benchmark assessment. Presented simultaneously for multiple assessments in Performance Levels order to monitor progress. Uses patterns of student performance across benchmark assessments to identify and Aggregate Multi-Test monitor groups of students at various levels of risk with regard to passing statewide Displaying Risk Levels assessments. The user can drill down to identify groups of students at each level of risk for intervention. The report identifies standards for which intervention is required. Provide various types of information about tests, including item analysis Testing Reports (the pattern of student answers for each question), test scores aggregated at various levels, and IRT parameters for each item on the test. Facilitate the monitoring of the system activity including tracking administration of district- Administrative Reports wide assessments, real-time monitoring of student progress during assessment administration and, user activity.

2.5.7.DD. Is the proposed solution scalable for increased school sites? Yes. Due to the varying resource requirements for different uses of the application, the Galileo® K-12 Online system has been designed for maximum scalability. The architecture of Galileo includes the use of clustered application servers in order to maximize scalability and minimize down time. If work needs to be done on a server, there is no need for the application to be taken offline. If traffic dictates that additional application servers are needed, more may be easily added, again with no down time. Data storage scalability is also built to accommodate future needs without service disruption. Additionally, the IT Technical Services team ensures the scalability of all systems to support increasing data needs and users.

2.5.7.EE. Does the proposed solution provide online documentation and help for end users? ATI and Galileo provide end users with a wide variety of types of ongoing professional development including access to online documentation and help. ATI web-based professional development options were discussed in detail in Tab 2.4 and include webinars, the Learning On- Demand series, the Professional Development Forum including access to the Resource Library, and the Galileo Help interface. These services are included with license of Galileo K-12 Online at no additional charge.

2.5.7.FF. Describe the availability of technical support services. Since its inception in 1986, ATI has provided exceptional customer service and technical support to clients all across the country. ATI utilizes a proactive approach to provide technical support for client school districts. This support, delivered by ATI Field Services and Technical Support staffs, is available Monday through Friday between the hours of 8:00 a.m. MST to 5:00 p.m. MST through our toll-free phone and email support systems. ATI also maintains round-the-clock emergency assistance including 24-hour server response capability and automated notification of any service outage to ensure maximum availability of Galileo K-12 Online.

The overall goal of the ATI Field Services and Technical Support staffs is to work in partnership with districts to help achieve district goals in utilizing Galileo K-12 Online. To ensure a successful partnership between ATI and a district, all users of Galileo K-12 Online are provided with full technical support of ATI services. One of the advantages of Galileo K-12 Online being a web- based application is that the need for on-site technical support is eliminated. ATI staff can address all server hardware issues within the ATI data center. Any client-side software issues can be addressed by telephone support, using remote control of the client computer on occasions where the support technician requires direct access to the user interface.

ATI also provides several additional forms of technical support. An online support request form may be accessed at http://www.ati-online.com/forms/contactustech.asp or by following the

“Contact Us” link on the ATI corporate site at http://www.ati-online.com and clicking the “Technical Support Contact Form” link. The ATI Technical Support team can also be contacted at [email protected]. Technical support documents are also conveniently located within the Galileo® K-12 Online application and provide information on topics ranging from data importation to Scanline installation and use. In addition, utilities needed for system operation (such as the latest version of Internet Explorer or Adobe Reader) can be easily downloaded from the Internet via links in the application.

Tab 2.6. Formative Assessment Chart Tab 2.6.A. General Requirements Required/ Score A. General Requirements Desired

1. Vendor has a thorough understanding of and system compliance with federal D 5 legislation (e.g., NCLB, IDEA, FERPA, COPPA, FISMA). 2. The Student Assessment and Data Analysis components may be two entirely R 5 standalone systems. However, if components are sold separately, a complete and seamless integration of the Student Assessment System and/or Data Analysis Tool must be assured and provided upon delivery. 3. Provides encrypted and highly secure access from home via the internet D 5

4. Interfaces with student information system (Synergy). Please provide details. R 5 (details follow Chart A) 5. Ability to import student course enrollment information from student information R 5 system. (Synergy) 6. Ability to import student demographic information from student information system. R 5 (Synergy) 7. Transfer of MPS content and data to the district, including any locally developed R 5 items, in the event of termination of services by the district or vendor. Please (details provide details. follow Chart A) 8. Vendor provides a robust plan and specific timeline for implementation, with built-in R 5 flexibility for the district. Please provide details. (details follow Chart A) 9. Describe client requirements and server/infrastructure requirements. Please R 5 provide details. (details follow Chart A) 10. System interfaces and shares results with other district-utilized enterprise R 5 information systems (e.g., Versifit data warehouse). Please provide details. (details follow Chart A) 11. Vendor agrees to district-determined periodically scheduled downloads of content R 5 and data to the district throughout the contract period. 12. Interfaces with other student learning management systems (e.g., Canvas). R 5 Please provide details. (details follow Chart A) 13. Exports data to major statistical packages such as SPSS. Please provide details. R 5 (details follow Chart A) 14. Proven ability to manage assessment data for a school district with up to 100,000+ R 5 students. Please provide details. (details follow Chart A) 15. Ability to tailor the terminology used to customize the name of the levels in the D 5 structure (state or district terminology) 16. Ability to manage multiple curriculum structure profiles within the same system D 5 (e.g. General Education, Special Education, Vocational Education, ELL, etc.) 17. Meets Schools Interoperability Framework (SIF) conformance requirements. D 5

18. Has been certified as compliant through the SIF certification program. D 5

19. Employs a simple, easy-to-learn web interface. R 5

20. Provides secure access to needed information for all users including, but not R 5 limited to, students, parents, teachers, counselors, school psychologists, principals, curriculum directors, assessment directors, Title I directors, superintendents, etc. 21. System operates with a reasonable response time. Please provide details. R 5 (details follow Chart A) 22. Provides simple, easy-to-learn web portals for district staff, principals and teachers R 5 to access appropriate information, based on individual login and password. 23. Provides simple, easy-to-learn web portals or integrates into the existing parent D 5 portal for parents and students to access appropriate information, based on individual login and password. 24. Provides electronic discussion groups within the system (e.g., sixth grade teachers D 5 can discuss student performance and interventions). 25. Ability to create custom classes or groups of students that can be shared across R 5 teachers. 2.6.A.4. Interfaces with student information system (Synergy). Galileo® supports the integration of data from a District’s student information system via the GDI module and the Student Interoperability Framework (SIF) module. The GDI module is used by most Galileo clients due to its ease and flexibility. ATI has successfully used this module to accomplish the importation of student data from Synergy in many districts. ATI is experienced in working with districts to set up such automated routines so that the data transfer is as efficient and accurate as possible. The GDI and SIF modules are described in detail in Tab 2.5.7.C.

2.6.A.7. Transfer of MPS content and data to the district, including any locally developed items, in the event of termination of services by the district or vendor. The GDE module can be used to export data to a data management system such as a data warehouse system, or to produce a delimited file that can be stored and/or used as desired. In the event of termination of services, ATI could make use of GDE module capabilities to provide MPS with content and data including locally developed items. This module is described in detail in Tab 2.5.7.C.

With respect to MPS content developed within Galileo including any locally developed items, as described in the ATI Electronic Services Agreement provided in Tab 8.1 and incorporated herein by reference, new tests, test items, Instructional Dialogs and/or other instructional materials that are developed by MPS and are not an adaptation of ATI materials or information including ATI Intellectual Property, may be exported from Galileo for use by MPS.

2.6.A.8. Vendor provides a robust plan and specific timeline for implementation, with built-in flexibility for the district.

Tab 2.3.4 and Tab 2.3.5 provide an overview of ATI’s approach to implementation as well as a suggested Implementation Plan including a specific timeline for implementation. The suggested Implementation Plan is designed with built-in flexibility for the district. As part of initial implementation planning and within the context of the ATI Partnership Model, this suggested plan will be collaboratively developed into a formal Implementation Plan that best meets the District’s needs.

2.6.A.9. Describe client requirements and server/infrastructure requirements. Galileo® K-12 Online is a web-based application, hosted entirely within ATI data centers. All ATI application and web servers run Windows Server 2008 R2, Standard edition with Internet Information Services (IIS) 7.5. ATI data servers also run Windows Server 2012 R2, with Microsoft SQL Server 2012.

In order to access Galileo K-12 Online, the District only needs to ensure District computers are running a supported browser and have the appropriate plugins. Information about Galileo supported web browsers is provided in Tab 2.5.7.T along with information about the computer operating systems supported by each browser. This information can also be found on the Galileo K-12 Online system requirements page at: http://www.ati- online.com/galileoK12/K12TrainingSystem.html. Galileo reporting functions may require Adobe Acrobat Reader plugin, Galileo media functions may require Adobe Flash Player plugin, and Galileo Bank, Dialog and Test Builder tools may require Java Runtime Environment.

2.6.A.10. System interfaces and shares results with other district- utilized enterprise information systems (e.g., Versifit data warehouse). Galileo is designed to support the sharing of results with other district-utilized enterprise information systems (e.g., Versifit data warehouse). Users can export data directly from Galileo using the Custom Test Reports and Report Builder interfaces. These interfaces generate comma-delimited files. ATI can also provide customized exports via the GDE module. This module can be used to export data to a data management system such as a data warehouse system, or to produce a delimited file that can be stored and/or used as desired. Prior to data extraction, ATI staff work with district staff to determine the desired data for inclusion and to develop a standardized file format to ensure that all fields are mapped appropriately during

2.6.A.12. Interfaces with other student learning management systems (e.g., Canvas). The GDI, GDE, and SIF modules described in Tab 2.5.7.C support data importation/exportation for purposes of interfacing with other District student learning management systems (e.g., Canvas).

2.6.A.13. Exports data to major statistical packages such as SPSS. As described in Tab 2.6.A.10, Galileo® supports the exportation of comma- and other-delimited files via the Custom Test Reports and Report Builder interfaces as well as via ATI-customized exports provided via the GDE module. Comma- and other-delimited files can easily be imported into major statistical packages such as SPSS.

2.6.A.14. Proven ability to manage assessment data for a school district with up to 100,000+ students ATI currently serves more than 500 clients nationwide including school districts, charter schools, and early childhood programs in 35 states as well as universities and state departments of education. Galileo® K-12 Online currently serves more than 750,000 students nationwide. Client student enrollments range from 200 to 60,000. Since Galileo has been designed for maximum scalability, the system is easily able to manage assessment data for a school district with up to 100,000+ students.

2.6.A.21. System operates with a reasonable response time. With respect to application response time, Galileo® K-12 Online application servers are tuned for performance and scalability. Internal and external monitoring systems also alert IT staff if server performance metrics fall outside pre-determined parameters for disk or CPU utilization or application server “web request queuing.” Because Galileo K-12 online is a hosted application that relies on a database back-end, query response times may impact system response times as well. To maximize performance, larger reports and some functions requiring large-scale data transfer are offloaded to Microsoft Message Queuing (MSMQ). When functions use MSMQ, the end-user is returned to active application use as soon as the request is generated so they may continue with other activities within the system and can retrieve the report results once processing is finished. ATI staff actively monitors database response times and trace results to

Tab 2.6.B. Assessment System Requirements Required/ Score B. Assessment System Requirements Desired

1. A valid, secure periodic district assessment system is required in reading and R 5 math 2. A valid, secure periodic district assessment system in desired in science and D 5 writing 3. Accommodates online assessment with automated scoring. R 5

4. Accommodates local (at school site) scanning of student answer sheets to R 5 input assessment data. 5. Accommodates online keying of student answers by the student to input R 5 assessment data. 6. Accommodates online keying of student answers by the teacher to input R 5 assessment data. 7. Accommodates online keying of student assessment scores (e.g. rubric R 5 scored, observational/checklist, numeric score, etc.) by the teacher. 8. Accommodates delimited text file batch import to input assessment data. R 5

9. Accommodates handheld wireless device (e.g., clickers, phones, etc.) to input D 5 assessment data. Please provide details. (details follow Chart B) 10. Accommodates paper-based assessments in multiple formats including R 4 selected response (multiple-choice, true/false, matching, etc.), constructed- response (open-ended short answer), rubric-scored, observational/checklist, grid-able, etc. 11. Accommodates computer-based/online assessments in multiple formats such R 5 as equation response, graphic response, multiple select, proposition response, matching items, table items, multiple choice, Evidence-based selected response , editing task (type in), editing task with choice, graphic response, hot text and other item formats that may be used in the AzMERIT testing 12. Enables teachers to create pre-identified answer sheets for all types of D 5 assessments and print at the school. 13. Store and updates test statistics. Please provide details. R 5 (details follow Chart B) 14. Availability of computer adaptive testing or ability to interface with such a D 5 system. Please provide details. (details follow Chart B) 15. Ability to document who authored and edited a test. R 4

16. Process to author, edit and approve tests. Please provide details. R 5 (details follow Chart B) 17. Solution allows teachers to assign instructional activities for students to D 5 complete in class or as homework. 18. Solution provides grades to assignments and assessments with limited need D 5 for teacher scoring 19. Solution allows students to become familiar with online assessment and R 5 technology enhanced items in preparation for the AzMERIT test. 20. Solution incorporates universal design principals to maximize student access R 5 to the tests/activities

2.6.B.9. Accommodates handheld wireless device (e.g., clickers, phones, etc.) to input assessment data. As discussed in Tab 2.5.2, Galileo® is web-based and supports students in inputting assessment data via student response devices (i.e., “clickers”). Mercury is an ATI software application used with Galileo K-12 Online technology that lets students instantly send assessment and/or instructional responses from a wireless response pad to the online system. Galileo’s Mercury component allows assessments to be administered to students using a variety of student response devices. The system requirements for Mercury including a complete list of currently supported student response devices are provided in Tab 2.5.2.

As a web-based system, Galileo K-12 Online can also be used on supported mobile devices (i.e., MS Surface RT and Pro, Apple iPad, Android Tablet, Google Chromebook) running compatible browsers. Certain Galileo interfaces have been specially constructed to support the unique display requirements of smaller devices for use by district staff. Further information about Galileo functions on supported mobile platforms is provided in Tab 2.5.2.

TABLE 7 Supported Mobile Platforms Galileo K-12 Area MS Apple iPad Android Google Surface RT Tablet Chromebook and Pro Galileo K-12 Online Core Yes Yes Yes Yes Functions Student Testing Yes Yes Yes Yes Multimedia Student Test Yes No Yes No Content (Includes Early Literacy Benchmark Assessment Series) Instructional Dialogs Yes No Yes No Bank Builder Tools Yes No Yes No Test Builder Tools Yes No Yes No Dialog Builder Tools Yes No Yes No

2.6.B.13. Store and updates test statistics. ATI stores and updates test statistics for a wide variety of types of assessments. These test statistics are produced as part of ATI ongoing psychometric analyses of items and assessments and as part of ATI statistical evaluation of educator impact on student growth. ATI also conducts ongoing statistical research on the reliability, validity, and forecasting accuracy of Galileo district- wide assessments (e.g., summative, interim, benchmark, pretest, posttest). ATI research

routinely produces statistics indicating that Galileo® assessments are reliable, are valid, and are effective forecasters of student performance on statewide assessments.

Psychometric Analysis of Items ATI uses procedures based in IRT to conduct statistical evaluations of items in the secure item banks at a minimum once a year, and typically two or three times a year. When district- created items are administered to a sufficiently large sample of students, ATI can also conduct IRT analyses to evaluate the characteristics of these items. ATI IRT analyses produce statistics in the form of IRT-based item parameters for each item (i.e., discrimination parameter, difficulty parameter, and guessing parameter). Item parameters for the items on each district-wide assessment are available as part of assessment development during the optional district Test Review phase. Once an assessment has been administered and analyzed, item parameters are also reported within the Item Parameters Report illustrated in the following screen shot.

Galileo® Item Parameters Report – Confidential Screen Shot

Psychometric Analysis of Assessments ATI conducts ongoing psychometric analyses of district-wide assessments including pre-built ATI assessments, customized district-curriculum aligned assessments, and district assessments created as part of the CAIB initiative to support educator evaluation in non- state-tested content areas. These psychometric analyses use procedures based in IRT to evaluate item characteristics (where applicable), provide estimates of student ability (i.e., DL scores), place scores on a common scale across tests to support the measurement of growth, and evaluate reliability. When existing district-created assessments imported using ASK technology are administered to a sufficiently large sample of students, ATI can also conduct IRT analyses for these assessments.

Setting Cut Scores to Forecast Statewide Test Performance Once the psychometric analysis is complete, ATI sets cut scores for each district-wide assessment using well-established statistical procedures including equipercentile equating and the application of research-based growth expectations. These cut scores divide students into performance levels in a similar way as the relevant statewide test and support the classification of students into Galileo® Risk Levels that accurately predict student risk of not passing the statewide test.

Statistical Evaluations of Educator Effects on Student Growth ATI conducts state-of-the-art statistical analyses (e.g., Categorical Growth Analysis, value- added modeling) that evaluate educator effects on student growth (i.e., Expected Growth Exceeded, Expected Growth Maintained, Expected Growth Not Maintained). Categorical Growth Analysis uses a repeated-measures t-test to compare the observed growth for a group of students (i.e., average change in DL score between two tests) to a growth expectation for the time period between the tests. Growth expectations in math, English language arts, writing, and science (grades kindergarten through 12) are based on annual research conducted by ATI using regression techniques. Growth expectations for other content areas are based on the average observed growth for the district. When sample sizes are sufficient, value-added modeling can also be conducted. This analysis uses hierarchical linear modeling techniques to evaluate the effect of teachers and schools on the observed growth of students.

Research on Reliability Galileo provides access to a variety of pre-built assessments (e.g., benchmark, pretest, posttest) designed and created by ATI. ATI pre-built assessments are designed to maximize the overall reliability of the assessment while providing information about ability and standards mastery for students of a wide range of abilities. Table 8 illustrates the reliabilities for a set of ATI district- wide assessments in math, reading, and writing aligned to Common Core State Standards.

These pre-built assessments were recently developed by ATI as part of ATI’s CBAS. For each assessment in the chart, the reliability was estimated for a normally distributed student population where student ability ranges from three standard deviations below the mean to three standard deviations above the mean.

TABLE 8 Reliabilities for a representative set of ATI pre-built assessments in math Reliabilities for a Representative Set of ATI Pre-Built District-Wide Assessments aligned to Arizona’s College and Career Ready Standards Grade Math English Language Arts KG 0.94 0.93 1 0.89 0.88 2 0.90 0.88 3 0.89 0.90 4 0.90 0.89 5 0.89 0.89 6 0.87 0.90 7 0.86 0.89 8 0.86 0.86 9 0.86 0.82 (Algebra I) 10 0.83 0.85 (Geometry) 11 0.88 0.81 (Algebra II) 12 0.85

Please note that this chart is intended to provide reliabilities for a representative set rather than a complete array of ATI assessments. ATI offers several series of pre-built assessments aligned to Common Core State Standards, Arizona’s College and Career Ready Standards, and other state standards in a wide variety of grades (i.e., kindergarten through 12) and content areas (i.e., math, reading, science, writing) supporting administration multiple times a year. In addition to access to pre-built assessments, Galileo® also provides the capability for districts to create customized assessments according to the district’s specifications and aligned to the district’s curriculum and pacing guide. ATI provides districts designing customized assessments with guidance and recommendations to help ensure that tests will produce adequate levels of reliability. For example, reliability is related to test length, with longer tests typically associated with higher levels of reliability. For district-wide assessments (e.g., benchmark assessments), ATI recommends that districts include approximately 35 to 50 items. Assessments of this length are typically observed to display adequate reliabilities (i.e., 0.80 or higher).

As soon as each Galileo® pre-built or district-customized district-wide assessment has been administered by a district, ATI also conducts an IRT analysis that estimates the reliability of the assessment based on the population of students who took the test within the district. This enables ATI to evaluate the functioning of the assessment and the individual items for the district student population. The reliability statistic is provided to districts as part of the Item Parameters Report illustrated previously.

Research on Validity Investigations of predictive validity are carried out by ATI on an annual basis once districts/charters have uploaded their statewide assessment data for individual students. This ongoing predictive validity research consistently indicates that student performance on Galileo district/charter-wide assessments is correlated with student performance on the statewide assessment. In a recent investigation, correlation analyses were conducted for the first 43 districts to provide ATI with their statewide assessment data for individual students in grades two through high school in math, reading/English language arts, and science. The 43 districts/charters included in the sample represented students in four different states and administered 1,848 district/charter-wide assessments. As Table 9 shows, the mean correlations range from 0.65 to 0.81 across grades and content areas with an overall mean of 0.78. A correlation between 0.7 and 0.9 indicates a high correlation between the two measures, while a correlation between 0.5 and 0.7 indicates a moderate correlation. Thus, the observed correlations suggest that student scores on the Galileo assessments were strongly related to student scores on the statewide assessment.

TABLE 9 Mean correlations for the Galileo assessments and the statewide assessment for each grade/content area. Correlation of Galileo Assessments with the Statewide Assessment Content Area English Math Language Science Overall Arts 2 0.77 0.80 N/A 0.78 3 0.76 0.78 N/A 0.77 4 0.77 0.78 0.74 0.77

5 0.78 0.78 0.72 0.78 6 0.79 0.77 N/A 0.78

Grade 7 0.81 0.76 N/A 0.78 8 0.79 0.75 0.73 0.76 HS 0.65 0.68 0.70 0.67 Overall 0.77 0.76 0.73 0.78 N/A= Data not available

Research on Forecasting Accuracy Studies of predictive validity provide the foundation for investigations aimed at determining student risk of not meeting the standards on the statewide assessment. ATI research on risk forecasting accuracy examines patterns of standards mastery across multiple district/charter- wide assessments (e.g., benchmark assessments) and the statewide assessment. An initial risk assessment can be made based on a student’s performance on a single benchmark test. As additional data become available for subsequent benchmark tests, risk assessments can be adjusted and refined.

ATI monitors the accuracy of the risk level forecasts for each district/charter on an annual basis. Like the correlation analysis discussed earlier, the forecasting accuracy analysis can only be conducted once the school district has uploaded the student scores on the end-of-year statewide assessment. The results of these investigations are presented as part of the Galileo® Forecast Report. The Forecast Report illustrates the accuracy of Galileo risk forecasts for a district/charter with respect to the observed student performance on the statewide assessment at the end of the year. Correlation coefficients indicating the relationship between student scores on each administered district/charter-wide assessment and student scores on the statewide assessment are also presented in this report. A sample Galileo Forecast Report is presented in the following.

Forecast Report – Confidential Screen Shot

In a recent investigation, forecasting accuracy analyses were conducted for the first 43 districts to provide ATI with their statewide assessment data for individual students in grades three through high school in math, reading/English language arts, and science. The study included 544 forecasting opportunities, where a forecasting opportunity is the student-level predictions made for the students within a given grade level in a specific district/charter with regard to their performance on the statewide assessment in a given content area (i.e., one forecasting opportunity is for the third grade students in a given district with regard to the statewide assessment in math).

Figure 3 illustrates, across districts/charters, the mean percentage of students in each risk level who passed the statewide assessment. Figure 4 illustrates, across districts/charters, the mean overall forecasting accuracy as well as the mean forecasting accuracy for each risk level. There are two important aspects of the forecasting accuracy analysis to evaluate. First, as student risk level increases the likelihood of failure on the statewide assessment should increase. This is a prerequisite for accurate forecasting. As Figure 3 shows, the majority of students who were classified as being “On Course” based on their performance on the Galileo® district/charter-wide assessments did in fact pass the statewide assessment, while the majority of those who had been classified as being at “High Risk” of not demonstrating mastery on the statewide assessment did in fact fail. The other two risk level groups performed as expected as well.

Statewide Test Performance by Galileo Risk Level

100 90 80 70 60 50 94 40 69 30 20 44 10 17 0

Percent of Students Passing Percent of Passing Test Statewide Students On Course Low Risk Moderate Risk High Risk

Figure 3 Mean percentage of students passing the statewide assessment for each risk level.

Second, overall forecasting accuracy should be adequately high. It should be noted that teachers and administrators are using the data provided by Galileo® district/charter-wide assessments to implement effective instructional interventions, therefore many students who have been classified as being at some risk of failing the statewide assessment end up passing it instead, reducing the accuracy of risk assessment forecasts for those student groups. ATI considers forecasting accuracy to be adequate if a student’s risk level accurately predicted performance on the statewide assessment for at least 75 percent of students. As Figure 4 shows, the overall forecasting accuracy was quite high, with statewide test performance accurately forecast for 85 percent of students. This research is consistent with similar other research investigations performed by ATI and suggests that Galileo assessments and risk levels continue to demonstrate adequate levels of forecasting accuracy, supporting the use of information in Galileo reports to inform differentiated instruction and elevate student achievement.

Forecasting Accuracy of Risk Levels

100 90 80 70 60 50 94 40 83 85 69 30 56 20 10

Percent of Students Accurately Percent of Accurately Forecast Students 0 On Course Low Risk Moderate Risk High Risk Overall Forecast to Pass Not Forecast to Pass

Figure 4 Mean overall forecasting accuracy and mean forecasting accuracy for each risk level.

2.6.B.14. Availability of computer adaptive testing or ability to interface with such a system. ATI offers multi-stage CAT. An ATI CAT includes three stages with three item bundles per stage. The item bundles vary in difficulty to support the goal of increasing measurement precision by selecting items with difficulty levels matching the ability of the examinee. The ATI approach creates 27 different assessment paths. The prescribed path for a given examinee is selected based on examinee bundle responses.

A multi-stage approach has advantages over a traditional item-by-item approach. For example, a multi-stage approach provides greater control over item content than is the case for the traditional item-by-item approach. The traditional item-by-item approach can lead to the same set of items being chosen over and over again when examinees of similar abilities take the test. With a multi-stage approach, it is possible to control the extent to which content is repeated from one assessment to another. Similarly, the multi-stage approach is particularly useful in implementing standards-based education because of the control over content that it provides.

2.6.B.16. Process to author, edit and approve tests. Galileo® supports districts in creating a wide variety of types of customized assessments. For example, MPS can author, edit, and approve district-wide assessments aligned to the district’s curriculum and pacing guide. Within Galileo, the process for creating a customized test begins with the Assessment Planner. Using the Assessment Planner, district staff participates in the design of district assessments aligned to standards and sequenced according to district curriculum and pacing guides. The following screen shot shows the interface for the Assessment Planner.

Sample Assessment Planner – Confidential Screen Shot

When used in the design of customized assessments, the Assessment Planner allows the school district to determine how many assessments there will be during the year, when testing will occur, which standards will be covered in instruction and assessment, and how many items will be included for each standard on each assessment.

ATI Educational Management Services staff works closely with district personnel during the design phase to ensure that the resulting assessments are of the highest possible quality. For example, a district may request assessments that are too short to provide reliable data. In such cases, an ATI staff member explains the importance of reliability and encourages the district to modify the plan so that each assessment contains at least 35 to 40 items. Similarly, on occasion a district may request an assessment that is of sufficient length but which addresses only a few standards, with seven or 10 questions addressing each standard. Under these circumstances ATI staff may encourage the selection of a broader range of standards.

Test Assembly Once the design of a district’s assessments has been finalized in the Assessment Planner, the district user responsible for finalizing the plan presses the Plan Complete button in the Planner. This generates an automatic e-mail advising ATI Educational Management Services staff that the planning process has been completed. The Educational Management Services staff then generates the tests to match the submitted plan using ATI’s extensive banks of secure assessment items. The goal in test generation is to create a test that represents the goals to be measured at a variety of ability levels to give teachers the ability to measure student achievement and to identify areas that should be the focus of additional instruction. In assembling an English Language Arts test, the focus is to provide items in conjunction with a reasonable number of text sources to ensure students are able to complete the assessments in a timely manner.

Test Review After tests are assembled, they may be reviewed by the district. During the review phase, tests are moved to a secure library accessible to district staff and those individuals who are designated by the district as reviewers. District staff participating in the review process can use item replacement tools to select items that are more closely aligned to their expectations or instructional focus for the assessment. For example, some math objectives include a number of operations in the wording of the standard, but the district curriculum may have emphasized one of these operations in the particular period leading up to the assessment so reviewers may choose to redistribute the selection of items for that objective to reflect this emphasis.

Test review is conducted using the online Test Review utility. This utility provides the capability to review each item contained in a test. Comment boxes in the utility afford the ability to make comments regarding the overall test. The use of the Test Review utility is controlled with appropriate permissions. Two levels of permissions are provided. One allows for a series of initial reviewers. The second designates a final reviewer responsible for accepting a final version of the test under review. The Final Review interface is illustrated in the following screen shot.

Final Review interface – Confidential Screen Shot

Test Instructions Each assessment includes a set of specific instructions to teachers detailing how to administer the assessments. These directions are crafted to ensure that all students being evaluated on a particular item are provided with the same level of information and support in attempting the item. For example, objectives for early elementary grades have instructions specifying which items require teacher involvement and to what degree each teacher is to assist the students in

Test Publication Following the completion of assessment review, the district-designated final reviewer clicks on the “Review Complete” button. At this time the tests are moved back to the ATI production areas to archive the comments and to address any requests or comments that were not addressed by district users via item replacement. Once all of the comments/requests have been addressed, the test will be published to a secure library accessible only to district level staff until such time as the assessment is scheduled to be implemented.

Tab 2.6.B2. Requirements for the items that make up the tests

B2 Requirements for the items that make up the tests Required/ Score Desired

1. Allows for straightforward entry of district-developed items and assessments. This R 5 includes the use of graphics and specialized mathematics symbols (i.e. equation editor, Word Draw, etc). 2. Allows for straightforward entry of school-developed items and assessments. D 5

3. Allows for straightforward entry of teacher-developed items and assessments. D 5

4. Vendor provided test items are valid, of high quality and are consistent with the R 5 format of AzMERIT test item format and style. 5. Ability to search for test items based on the Arizona College and Career Ready R 5 Standards down to the cluster level. 6. Ability to search for test items based on the Arizona College and Career Ready R 5 Standards down to the standard level. 7. Ability to search for test items based on level of difficulty/understanding. R 4

8. Ability to search for test items based on keywords. D 5

9. Ability to include source & permission documentation for test items & their prompts. D 5

10. Allows for separation of district benchmark or end-of-year test items from other R 5 district-developed test items. 11. Allows for separation of district-developed test items from school and/or teacher- R 5 developed test items. 12. Accommodates a large expandable bank of juried test questions (vendor-provided, R 5 district-developed and district purchased) for AzMERIT (e.g. math, reading, writing, science) content areas. 13. Accommodates a large expandable bank of juried test questions (vendor-provided, D 5 district-developed and district purchased) for non-state assessment test content areas (e.g., social studies). 14. Process to author, edit and approve items. Please provide details. R 5 (details follow Chart B2) 31. Store and update the following information for each item, where appropriate: R 4 Relationship to Arizona College and Career Ready Standards at the cluster and (details standard level. follow Chart B2) Local district curriculum Short description of the item The complete item Scoring rules (correct answer, points for each response option, points for item) Item Resources (things needed to administer the item-graphics, tools, reference materials, etc.) Item Statistics (difficulty level, popularity of each distracter,) * Item History (desired feature) * Rubric (desired feature) * Approval Cycle (desired feature) Subject areas Units of Study * Keyword (desired feature) Grade Level * Other. Please provide details. 32. Ability to document who authored and who edited an item. R 5

33. Allows for easy collaboration and electronic sharing of items among building staff D 5 (test questions, assessments, lesson plans, activities, achievement data, etc.) with (details appropriate permissions and safeguards. Please provide details. follow Chart B2) 34. Allows for easy collaboration and electronic sharing of items between district and D 5 building staff that includes peer review (test questions, assessments, lesson plans, (details activities, achievement data, etc.) with appropriate permissions and safeguards. follow Chart B2) Please provide details.

15. Provides valid district tests that can be given in no more than 45 minutes. D 5

16. Provides an update of item maps and item coding to standards when state R 5 standards change

2.6.B2.14. Process to author, edit and approve items. Galileo® includes Bank Builder and Test Builder technology that enables the authoring (i.e., creation) of items and the placing of these items in shared libraries for editing and approval, if desired. In particular, Galileo Bank Builder technology is specifically designed to manage the process of developing a high-quality item bank and includes interfaces specifically designed to supports the authoring, editing, and approval of items. Bank Builder is used by ATI’s expert in- house Assessment and Instructional Design staff to develop items for the ATI secure and community (formative) item banks. ATI’s item banks are among the largest item banks in the nation. As of this writing, the Galileo K-12 Online item banks contain more than 131,000 items and writing prompts created by ATI and aligned to standards in math, reading/ELA, science, and writing including state standards and Arizona’s College and Career Ready Standards. ATI’s team of dedicated item writers continues to add several hundred items to the banks each month.

Districts and schools participating in the CAIB initiative also receive access to Bank Builder technology and professional development related to the ATI item development process that enables district and school staff to produce high-quality standards-aligned items. The CAIB initiative enables districts to collaborate and share high-quality, district-developed items representing a broad range of subjects not currently addressed on statewide assessments. The CAIB community item banks currently contain over 9,000 district-created items representing a broad range of subjects including social studies, art, music, and foreign language. All of the items are certified and aligned to standards of the state in which the items were developed. Participating districts may map items to other state or local standards as desired.

The Galileo Test Builder interface also enables teachers and other district personnel to create, modify, and edit test items and place them in their own libraries. Once such items have been created, they can be imported into future assessments by selecting the library that contains them during test creation.

Item Development The discussion that follows will describe ATI’s approach to item development for both the secure and community (formative) item banks including the authoring, editing, and review of items. A similar approach is also used by districts participating in the CAIB initiative and desiring to build their own item banks.

• Item Specifications The construction of an item specification is based on a review of the standards which are to be assessed. Each standard is broken down into the skills that make up the standard. These skills are the starting point for developing an online list of item specifications defining the characteristics of the classes of items to be written. Item specifications indicate the defining characteristics of the item class, the rationale for the class, and the required characteristics for each item component. Specifications address such factors as the cognitive complexity intended for items included in the specification class, the appropriateness of vocabulary, and Lexile® measure requirements related to readability levels. Creating item specifications to guide item development is recognized as a critical part of a process documenting that assessments are reliable and valid indicators of the ability they are intended to measure (Haladyna, 2004). Their structure and specificity also afford many advantages for ensuring that assessments may be readily adapted as district needs and or state/federal requirements change.

Item specifications may be designed to align closely to guidelines and released items from statewide assessments and multi-state assessment consortia (e.g., AzMERIT, Partnership for Assessment of Readiness for College and Careers [PARCC], Smarter Balanced Assessment Consortium [SBAC]). The careful design of item specifications is especially critical to the development of assessment items similar to those students are likely to encounter on future statewide assessments.

Sample Item Specification – Confidential Screen Shot

• Item Construction Items can be constructed in accordance with the relevant item specification using the online Bank Builder utility. Galileo Bank Builder includes features that promote item quality (e.g., size, font, and resolution standards) and yield items that are visually suited to online and offline administration. Bank Builder also allows for the use of item families, or groups of items that refer to the same contextual material. For example, several language arts items may refer to the same reading passage. The item construction process should also include attention to important principles of item writing. For example, when writing selected-response items, incorrect alternatives must be plausible answers that indicate how the student conceptualizes the problem. Distracters should resemble the correct choice in grammatical form, style and length (Haladyna, 2004). Care should also be taken to ensure that items are written to accommodate students from diverse backgrounds. When writing items including contextual material such as text passages, care should be taken to ensure that the passages are not ones that the student is likely to have read previously. In many instances, original texts are utilized. When previous passage exposure is not controlled, student performance may reflect differential exposure to content rather than comprehension of the material presented (Haladyna, 2004). Careful attention should also be paid to ensure that the material is at the appropriate reading level. For example, a measure of readability in the form of Lexile® measures is used by ATI, but the reading passages are also analyzed more subjectively to ensure that reading ability does not, for example, prevent students from correctly answering a math question.

• Review of Items and Related Materials The review of items and associated materials is typically a multi-stage process. For example, ATI’s initial review process for item specifications, items, and item families occurs in-house and is composed of three steps. ATI also encourages districts building their own item banks to follow this process. Step one is an independent review using the online bank review tool in Galileo® K- 12 Online. Reviewers may accept the material, reject it, or accept it with modifications. If the material is accepted with modifications, the reviewer should provide the writer with a list of suggested modifications, which are recorded in the online review. The modifications guide revision. Following the initial review, the material can be subjected to a final review. When consensus is reached between two reviewers and the writer, the material under review is accepted for use. In the case of item review, the item can then be certified. Certified items are locked so that they cannot be changed by Galileo users.

The item review process can continue once a certified item is made available for use in an assessment. For example, if the district desires, items placed in district-designed assessments can be reviewed by each member of the district’s review team during the test review process. Test review provides the district with the option to accept an item or replace it with an item that more closely meets their needs. Reviewers may also suggest item changes.

2.6.B2.31. Store and update the following information for each item, where appropriate: Relationship to Arizona College and Career Ready Standards at the cluster and standard level. Local district curriculum Short description of the item The complete item Scoring rules (correct answer, points for each response option, points for item) Item Resources (things needed to administer the item-graphics, tools, reference materials, etc.) Item Statistics (difficulty level, popularity of each distracter,) * Item History (desired feature) * Rubric (desired feature) * Approval Cycle (desired feature) Subject areas Units of Study * Keyword (desired feature) Grade Level * Other As described in the following, ATI stores and updates multiple types of item information where appropriate including:

• Each item is aligned to Arizona’s College and Career Ready Standards at the domain, cluster, and standard level. • Items are aligned to the district curriculum either by placing them in online Instructional Dialogs, which are aligned to curriculum units in the district curriculum or by including them in assessments aligned to the district pacing calendar. • Item descriptions are included in the item specification to which the item is built. • The complete item is stored and made available for selection, review and test administration. • Scoring rules are stored with the items and made available in item review interfaces within Bank Builder as well as within interfaces to support teacher scoring, where applicable. Correct answers are available in multiple test construction and test reporting interfaces. • Item resources may be identified as part of the item specification and included as part of an item (e.g., graphics), the assessment (e.g., reference materials such as formula sheets), or, for technology-enhanced items, as part of the online item display (e.g., tools such as calculators). • Item statistics (difficulty level, popularity of each distracter) are calculated and made available in interfaces and reports. Item parameter estimates (i.e., discrimination, difficulty, guessing) can be viewed as part of Test Review and in the Item Parameters Report. Information about the popularity of each distracter can be viewed in the Detailed Item Analysis Report. • A history of item parameter estimates is stored and updated over time. • Rubrics are stored along with an item and made available within interfaces to support teacher scoring, where applicable. • Bank Builder technology supports the implementation of an approval cycle for items involving review and certification. • Items are typically organized in libraries reflecting subject areas. Items can be aligned to District units of study either by placing them in online Instructional Dialogs, which are aligned to units in the district curriculum or by including them in assessments aligned to the district pacing calendar. • Bank Builder enables the searching for items with a particular key word in the stem. • Grade level is stored and updated via the grade level of the standard to which the item is aligned. • Other: Multiple psychometric properties of items are stored. Detailed discussion of these psychometric properties follows.

The psychometric properties of items deserve special attention because those properties affect the interpretation of test scores. For example, item parameter estimates affect the ability to

Galileo® K-12 Online Instructional Improvement and Instructional Effectiveness System Prepared for: Mesa Unified School District No.4 RFP #15-47MP Formative Assessment – February 19, 2015 Copyright © Assessment Technology, Incorporated 2015. All rights reserved. 136 Tab 2. Scope Tab 2.6. Formative Assessment Chart: B2. Requirements for the items that make up the tests measure academic progress. ATI uses IRT to estimate item parameters for items included on ATI assessments. In the case of multiple-choice items, ATI uses a three-parameter logistic response model. In this model, three parameters are estimated for each item: difficulty, discrimination, and pseudo guessing. The estimation of the guessing parameter is important because failure to estimate guessing produces underestimates of item difficulty, which can lead to misleading recommendations regarding next instructional steps. In the case of constructed response items and various TE items where the probability of guessing the correct answer to an item is negligible, ATI uses the Samejima graded model.

Parameter estimation is conducted using the Maximum Marginal Likelihood (MML) estimation procedures developed by Daryl Bock and his colleagues. The MML approach has the advantage of producing maximum likelihood estimates that meet the standard of being Best Linear Unbiased Estimators (BLUE). Parameter estimates are updated between one and four times a year. In the updating process, estimates are added for new items administered for the first time to a sufficient number of students during the updating period. In addition, at the conclusion of each school year, updated estimates are provided for existing items based on student performance during that school year. Parameter estimates for a set of widely administered existing items are held constant. The parameters for the remaining items are freshly estimated.

ATI’s parameter estimation approach is derived from a strategy used in parameter estimation for the National Assessment of Educational Progress (NAEP). It is also similar to approaches widely used in statewide assessments. The intent is to ensure that test scores can be compared across years while at the same time recognizing that parameters may change through the influence of factors such as changing demographics and changes in educational practice. The combination of fixed parameter estimates and new parameter estimates make it possible to place scores across years on a common scale. This ensures the link to the past. At the same time, new items and updated parameters are continually added to the bank. This ensures that assessments will reflect current standards and advances in item construction occurring as the nation moves rapidly toward online assessment with TE items.

The ATI parameter estimation process is unique in that parameters are estimated based on all of the student responses to a given item. All of the items included in assessments at a given subject and grade are assumed to be indicators of an underlying latent variable under the model. The number of students responding to a given item tends to be large. Thus the standard errors for the parameter estimates tend to be small. An additional advantage to the approach is that the model makes it possible to place scores from different assessments administered by different districts/charters on a common scale. This is particularly useful in supporting the use of statistical procedures requiring large samples. For example, ATI is currently working with Arizona State University and Damion Betebenner to implement student growth percentiles. The calculations necessary to produce student growth percentiles require large samples. The ATI model makes it

2.6.B2.33. Allows for easy collaboration and electronic sharing of items among building staff (test questions, assessments, lesson plans, activities, achievement data, etc.) with appropriate permissions and safeguards. Galileo® content and data including test questions, assessments, lesson plans, activities, and achievement data, are placed in libraries that allow easy collaboration and electronic sharing among building staff. Library permissions control access and provide safeguards where necessary. Libraries are created using Library Builder. Library Builder enables the user to assign hierarchical permissions at the class, school, and district levels. In addition, permissions may be assigned to specific users with special responsibilities. Permissions may include the ability to read information assigned to a library or to read and write to documents contained in a library. Write permissions provide support for collaborative projects conducted by district staff.

2.6.B2.34. Allows for easy collaboration and electronic sharing of items between district and building staff that includes peer review (test questions, assessments, lesson plans, activities, achievement data, etc.) with appropriate permissions and safeguards.

Collaboration and electronic sharing of items between district and building staff is provided through the library technology described previously. Peer review is also provided through library technology and the assessment review included in the system. For example, customized assessments may be reviewed using the Test Review utility described in Tab 2.6.16. In the initial review period, library technology is used to assign as many reviewers for an assessment as the district chooses. In the second stage of the review, library access is provided to a final reviewer responsible for the review used to guide test development and publication.

Tab 2.6.C. Data Analysis Requirements

Required/ Score C. Data Analysis Requirements Desired

1. Availability of comprehensive query tools for Building Administrators, District R 4 Central Office 2. Includes flexible drill-down capability from any form of data aggregation. R 5

3. Provides encrypted and highly secure access from home. D 5

4. Interfaces with other district-utilized enterprise information systems (Synergy, R 5 Versifit data warehouse, Microsoft suite of programs, etc.). Please provide details. (details follow Chart C) 5. Provides varied methods of representing information (e.g., tables, graphs). R 5

6. Interface provides immediate access to relevant information. R 5

7. Provides clear, varied, relevant, and comprehensive pre-formatted reports available to every level of user (e.g., Parent, Student, Teacher, Counselor, Building R 5 Administrators, District Central Office, etc.). 8. Ability to create a student report that has similar content to the state report so that parents become familiar with the test scores and content. [The format of the state report has not been finalized, this item relates to the flexibility to customize a report D 3 with data if it is available with the formative assessments) 9. Ability to present longitudinal (K –12) data available at every level of user (e.g. Parent, Student, Teacher, Counselor, Building Administrators, District Central R 4 Office, etc.) and at every appropriate level of aggregation. 10. Provides capacity to perform data cleansing. R 4

11. Allows for expansion past initial implementation. R 5

12. Provides encrypted and highly secure access for data transmission. R 5

13. Integration of different areas of information is seamless to the user. R 5

14. Accepts numerous common data formats. R 5

15. Exports into common data formats. R 5

16. Provides links to state domains, clusters, and standards. R 5

17. Provides links to remediation activities and instructional resources based on D 5 individual student performance. 18. Ability to analyze student test data based on teacher for the teacher at the time of R 5 the assessment and for the following year’s teacher. 19. Ability to analyze student test data based on demographic, performance and R 5 program information at the time of the assessment. 20. Ability to predict performance on state AzMERIT tests based on student D 5 performance on district level assessments.

2.6.C.4. Interfaces with other district-utilized enterprise information systems (Synergy, Versifit data warehouse, Microsoft suite of programs, etc.).

Galileo® K-12 Online is designed to maximize interoperability with other data management systems (e.g., student information systems, data warehouse systems). ATI and Galileo K-12 Online flexibly support districts in accomplishing a variety of importation, exportation, and updating tasks via three Galileo modules: the Galileo Data Import (GDI) Module, the Galileo Data Export (GDE) Module, and the School Interoperability Framework (SIF) module. These modules are described in detail in Tab 2.5.7.C.

Tab 2.6.D. Reporting Requirements Required/ Score D. Reporting Requirements Desired

1. Provides learner profile and longitudinal assessment information from the time a D 5 student enters school (will begin with archived data from state assessments from two-three years back). 2. Provides pre-formatted reports, based on system data, in easy to understand R 5 formats (e.g. color coding, bar graphs, line graphs, pie charts, tables, etc.). 3. Provides longitudinal item analysis for individual students and groups of students R 5 (schools, teacher, programs, etc.). 4. Provides longitudinal reporting of assessment data for individual students and R 5 groups of students (schools, teacher, programs, etc.). 5. System enables teachers to generate individualized reviews and homework D 5 assignments, based on student performance data and individual needs. 6. Supports design and propagation of additional reports by District personnel. R 5

7. Provides students/parents with reports or other feedback that guides student D 5 learning in a student/parent friendly format. 8. Solution will provide reports of student growth over one year and over multiple R 5 years if available. 9. Ability to create a student report that has similar content to the state report so that D 3 parents become familiar with the test scores and content. [The format of the state report has not been finalized, this item relates to the flexibility to customize a report with data if it is available with the formative assessments)

Tab 2.6.E. Conversion Requirements Required/ Score E. Conversion Requirements Desired

1. Vendor conversion of district-developed or textbook publisher-provided items and R 4 assessments. 2. Vendor entry of district-developed items and assessments. R 5

Tab 2.6.F. Training and Support Requirements Required/ Score F. Training and Support Requirements Desired

1. Vendor provides robust staff development for all user levels on the system (trainer- R 5 of-trainer model, printed training materials, online help, etc.). 2. Vendor provides telephone support and face-to-face support as needed. Please R 5 provide details. (details follow Chart F) 3. Vendor provides training on using the data and reports to improve instruction at the R 5 school level (train the trainer model). 4. Vendor provides training for Information Systems and Educational Technology staff R 5 on system installation and maintenance

2.6.F.2. Vendor provides telephone support and face-to-face support as needed. Tab 2.4 describes the on-site and web-based professional development and training provided by ATI during the initial implementation of Galileo® as well as throughout the contract period. In addition to these professional development opportunities, ATI also provides extensive technical support to all Galileo users. ATI utilizes a proactive approach to provide technical support for client school districts. This support, delivered by ATI Field Services and Technical Support staffs, is available Monday through Friday between the hours of 8:00 a.m. MST to 5:00 p.m. MST through our toll-free phone and email support systems. ATI also maintains round-the-clock emergency assistance including 24-hour server response capability and automated notification of any service outage to ensure maximum availability of Galileo K-12 Online.

ATI also provides several additional forms of technical support. An online support request form may be accessed at ati-online.com/forms/contactustech.asp or by following the “Contact Us” link on the ATI corporate site at ati-online.com and clicking the “Technical Support Contact Form” link. The ATI Technical Support team can also be contacted at [email protected]. Technical support documents are also conveniently located within the Galileo K-12 Online application and provide information on topics ranging from data importation to Scanline installation and use. In addition, utilities needed for system operation (such as the latest version

Tab 3. Proposal Cost Form

Include all costs including any warranty costs for the life of the project. Maintenance and Support costs must be included if requested in the Scope section of the RFP.

This section provides the Proposal Cost Form along with the attached ATI Detailed Cost Proposal. The ATI Detailed Cost Proposal provides information about all costs.

Tab 3.1. Proposal Cost Form The Proposal Cost Form follows.

1. Cost Sheets and Summary for Required Items and Services

Present all itemized costs and total cost associated with the purchase, installation, and implementation of the proposed Formative and Benchmark Assessment System for Mesa Schools hosted remotely on the Vendor’s servers. Any costs not identified in this section will not be reimbursed. Your cost proposal should include, but is not limited to the following required components: a. Software Licensing Costs $ 337,664 See Note 1____ (per student, per site, per module, etc., as appropriate) b. System Maintenance and Upgrades $_Complimentary______c. Hardware, including required server specifications and quantities $_Not applicable- See Note 2 d. Data Import/Export (e.g., student information, external test data, $_Complimentary______ASK technology, assessment scores) e. Software Development and Customizations $ 0- Estimated (Programming Services) See Note 3___ f. Consultant Services (w/ mutually agreed frequency) $ Complimentary______g. Project Management $ Complimentary______h. Federal and/or state mandated modifications $ Not applicable______i. End-user Training $ 16,000 See Note 4_____ j. System Management / Technical training $ Complimentary______k. Post-implementation Support $ Complimentary______l. Installation $ Not applicable (Galileo- ATI-hosted; Scanline/Mercury installed by District m. Other costs not covered above $ 36,800 (Scanline) See Note 5 (supply back-up documentation) $ Complimentary (Mercury) See Note 6 $ 0- Estimated (Data Entry) See Note 4 TOTAL PROPOSED COST: $ 390,464 plus applicable state and local taxes Note 1: Annual User Fees for Galileo® K-12 Online based on 42,208 students in grades 3-11 @ $8.00 per student. Note 2: No District server hardware required- Galileo is hosted on ATI servers. District will need to purchase sufficient computers/tablets to meet District needs, scanning units to support offline testing with online aggregation (if desired), and wireless response pads to transfer student responses into Galileo (if desired). Note 3: If services beyond those detailed in the ATI Detailed Cost Proposal or programming services unrelated to system upgrades are requested and authorized by the District and agreed to by ATI, compensation will be at the following hourly rates: Software Engineer Hourly Rate $ 175.00; Data Entry Specialist Hourly Rate $ 30.00 Note 4: On-site PD/Training (Four on-site Phase One and four on-site Phase Two sessions supporting initial implementation). All on-site PD sessions include: PD/training in District-provided computer lab(s) accommodating up to 45 participants; attendee- focused content; visuals in presentations; attendee-specific session materials; trainer travel expenses. Sessions are led by a trainer with educational background; session length is generally six hours. Per six-hour on-site session: $ 2,000. Note 5: Scanline v.3.5 – electronic service facilitating offline (paper/pencil) testing and online data aggregation. One-time purchase – user installation on user purchased scanning units. One Scanline installation required for each scanning location. $460 per installation. Cost of $36,800 assumes 80 installations. Check with ATI for currently supported scanning units. Scanline carries an 8.1% Transaction Privilege Tax. Estimated Tax Year 1- $2,980.80. Note 6: Mercury is a software application used with Galileo® K-12 Online technology that lets students instantly send assessment and/or instructional responses from a wireless response pad to the online system. The response pad system hardware is purchased independently by District. Check with ATI for currently supported response pad systems. **If the cost of your solutions is on a per pupil basis. How will the number of pupils will be assessed? Please identify the process. Is there a mutually agreed to point in time for a grade level or course enrollment? Does your formula allow for excluding students not accessing the regular curriculum? Annual per student user fees are based on MPS reported number of enrolled students who will be accessing ATI electronic services over the contract period.

Tab 3. Proposal Cost Form Tab 3.2. Proposal Cost Form Attachment: ATI Detailed Cost Proposal - Confidential

Tab 3.2. Proposal Cost Form Attachment: ATI Detailed Cost Proposal

The ATI Detailed Cost Proposal attachment to the Proposal Cost Form follows.

ATI Detailed Cost Proposal Prepared For: Mesa Unified School District No. 4 (also known as Mesa Public Schools) Cindi Hostetler, Procurement Specialist Supervisor 549 North Stapley Drive, Building 1, Mesa, Arizona 85203 Prepared By: Assessment Technology, Incorporated (ATI), 6700 E. Speedway Boulevard, Tucson, Arizona 85710 Response To: RFP #15-47MP Formative Assessment COST TOTAL COST DELIVERABLES EACH COST YEAR 1 YEAR 2-5 YEARS 1-5 1. ELECTRONIC SERVICES USER FEES Annual User Fees for Galileo® K-12 Online based on 42,208 students @ $8.00 per student including the following services as described in the response: $ 337,664 $ 337,664 $ 1,688,320 (for 42,208 (assuming • • • Assessment Curriculum Dashboard students in no change Comprehensive/Customizable Planners: Unit, Lesson, Teaching, School Performance, grades 3-11) in student Assessments Assignment and Instructional body or Item Banks- Selected- and Instructional Dialogs Effectiveness Views desired • Constructed-response, (Searchable Bank for Reports Informing grade Performance-based, Class Instruction, Decisions levels) Technology-enhanced Items Tutorials) Test Scores (Individual, Class, Construction of Test Banks Builders: Dialog, Resource, School, District) Builders: Test/Item Book Standards Mastery (Intervention Ordering/Planning Tests Curriculum Scheduling Alert, Profile– Individual, Test Scheduling Reports (Monitoring, Class, School, District) Test Administration (Online Portfolio) Test Sets (Benchmark, including Computerized • Grades Formative, Multi-Test, Adaptive Testing, Offline, via Schedule Assignments Intervention, History) Student Response Devices) Enter Grades Growth (Categorical Growth, Test Scoring Parent Notification Value-Added Modeling) Test Results Grade Reports Monitoring (Testing, Dialog, & Printing Capability • Setup User Activity) • Student/Family School Information Test Properties (Test Blueprints, Records: Enrollment, Builders: Form/Report Item Analysis, Item Class Schedule, Family Importation: Student Parameters, Forecasting) Student-Parent Center Data/External Test Student, Family and Staff • Staff Data Setup Information Demographics Assigning/Building: External Tests Staff Assignment Standards, Libraries Custom Reports Forum User Accounts/Security

2. PROJECT SUPPORT

Implementation/Project Management and Consultation: On-Site (mutually agreed frequency) and Remote (e.g., telephone, email, small-group webinars - mutually agreed frequency). Complimentary Application-Related Support: Technical, Client Use including Assessment and Instructional Design and Report Use provided by ATI staff at no additional charge. 3. DATA ANALYSIS SERVICES Psychometric analyses supporting instructional improvement and instructional effectiveness by establishing reliability, validity, and forecasts of statewide test performance, evaluating growth Complimentary (Categorical Growth Analysis, Value-Added Modeling) and measuring progress by placing test scores on a common scale. 4. START-UP AND DATA IMPORTATION/EXPORTATION SERVICES • Implementation Planning Services • Assessment Planning and Instructional Dialog Planning Services Complimentary • ATI Data Importation Services (e.g., Student Information, External Test Data, ASK Technology) and ATI Exportation Services (i.e., Assessment Scores) 5. APPLICATION UPGRADES Application Upgrades: Provided at no additional charge. No maintenance fees. Complimentary ELECTRONIC SERVICES USER FEES, PROJECT SUPPORT, DATA ANALYSIS $ 337,664 (for 42,208 SERVICES, START-UP AND DATA IMPORTATION/EXPORTATION SERVICES, and $ 337,664 $ 1,688,320 students in APPLICATION UPGRADES FOR APPROX. 42,208 STUDENTS: TOTAL USER FEES grades 3-11)

ATI Detailed Cost Proposal - Continued COST TOTAL COST DELIVERABLES EACH COST YEAR 1 YEAR 2-5 YEARS 1-5 6. SCANLINE* 6.1. Training on use of Scanline will be provided via phone contact or webinar by ATI at no Complimentary charge for as many as two clerical staff at each Scanline station location. -0- 6.2. Scanline v.3.5 – electronic service facilitating offline (paper/pencil) testing and online data $ 36,800 (assuming aggregation. One-time purchase – user installation on user purchased scanning units. One (assuming 80 $ 36,800 no additional Scanline installation required for each scanning location. $460 per installation. Check with ATI installations) for currently supported scanning units. Scanline carries a 8.1% Transaction Privilege Tax. installations) Estimated: Estimated: Estimated: SCANLINE SOFTWARE: TOTAL $ 36,800* $ 0 $ 36,800* 7. MERCURY Mercury is a software application used with Galileo® K-12 Online technology that lets students Complimentary instantly send assessment and/or instructional responses from a wireless response pad to the with Galileo online system. The response pad system hardware is purchased independently by District. K-12 Online License Check with ATI for currently supported response pad systems. 8. ON-SITE/ONLINE PROFESSIONAL DEVELOPMENT** Based on the goals expressed in the RFP, ATI proposes to provide MPS with four Phase One on-site professional development sessions in year one as well as four Phase Two on-site professional development sessions in year one. This approach will provide two days of training for MPS testing and curriculum staff as well as two days of training for approximately two lead staff per site (e.g., administrators, lead staff designated as trainers). As part of initial implementation planning, ATI IT and Field Services staffs will also provide training for District Technology Services staff via phone, email, live webinar, and on-site visit (if desired). In addition, one on-site professional development session will be provided in each subsequent year to accommodate new staff and/or extend the skills of returning staff. A customized PD/training plan with specific agendas will be developed in partnership with MPS. Community Assessment and Item Banking training also available if desired. On-site PD/Training (PD/training supporting initial and ongoing implementation. All on-site PD $ 16,000 $ 2,000 sessions include: PD/training in District-provided computer lab(s) accommodating up to 45 (assuming four (assuming $ 24,000 participants; attendee-focused content; visuals in presentations; attendee-specific session Phase One and one on-site materials; trainer travel expenses. Sessions are led by a trainer with educational background; four Phase Two PD session) session length is generally six hours. Per six-hour on-site session: $ 2,000. on-site sessions) Online PD/Training (small group incl. webinars with mutually agreed frequency, Learning On Complimentary Demand, Professional Development Forum, Quick Reference Guides, Videos, Help files) Estimated: Estimated: Estimated: ON-SITE AND ONLINE PROFESSIONAL DEVELOPMENT: TOTAL $ 16,000 $ 2,000 $ 24,000 If services beyond those detailed in the costs above or programming services unrelated to system upgrades are requested and authorized by the District and agreed to by ATI, Estimated: compensation will be at the hourly rates for the following: -0- - Software Engineer Hourly Rate $175.00 - Data Entry Specialist Hourly Rate $ 30.00 SPECIAL SERVICES: TOTAL $0 $0 $0 ELECTRONIC SERVICES USER FEES 42,208 STUDENTS ANNUALLY, PROJECT SUPPORT, DATA ANALYSIS SERVICES, START-UP AND DATA IMPORTATION/ $ 337,664 $ 337,664 $ 1,688,320 EXPORTATION SERVICES, and APPLICATION UPGRADES: TOTAL ANNUAL USER FEES SCANLINE SOFTWARE – ONE-TIME PURCHASE: TOTAL $ 36,800* $ 0 $ 36,800* MERCURY: TOTAL $ 0 $ 0 $ 0 ON-SITE AND ONLINE PROFESSIONAL DEVELOPMENT: TOTAL $ 16,000 $ 2,000 $ 24,000 SPECIAL SERVICES: TOTAL $ 0 $ 0 $ 0 GRAND TOTAL $ 390,464 $ 339,664 $ 1,749,120 * Scanline (one-time purchase) carries an 8.1% transaction privilege tax. Estimated Tax Year 1- $2,980.80; Years 2-5: $0 unless District adds additional scanning installations. **Offered in addition to project management, consultation, technical, application-use and educational management, instructional, and assessment support provided by ATI staff at no charge. Price Clauses: 1. Prices as stated are complete for the services, software, and materials offered in proposal and include all associated costs with the exception of consumables (e.g., paper, toner). There are no charges related to district/school staff use. 2. If the District elects to use Scanline, the District will also need to purchase independently scanning units in sufficient quantity to meet the District’s scanning needs. If OpScan units are elected by User, Scantron OMR scan forms will also need to be purchased separately by User. Check with ATI Field Services staff for information on ATI supported scanning units and OMR forms. 3. If the District elects to use wireless response pads to transfer student assessment and/or instructional responses directly into Galileo for immediate teacher use, the District will need to purchase independently those units in sufficient quantity to meet District goals. Check with ATI Field Services staff for information on ATI supported wireless response pad systems. Payment Terms: Net 30 days following activation of user fees for Galileo K-12 Online services. State and local taxes not included.

Tab 3. Proposal Cost Form Tab 3.3. Acknowledgement of Addenda

Tab 4. Additional Costs

A listing of any items such as letters, phone calls or other types of services generating a cost to the District and not included in the fees shown on the proposal are to be included, plus a formula or explanation of how these additional costs will be determined and billed to the District. All costs associated with the licensing and implementation of Galileo K-12 Online are contained in the ATI Detailed Cost Proposal and reflected in the MPS Proposal Cost Form. Both can be found in Tab 3 of this proposal.

Tab 5. References

List at least three (3) references. Include the name and phone number of the contact person and some info regarding how this reference (size and project) compares to the scope of this RFP.

This section provides reference information for three ATI clients as well as information regarding the size and scope of the project. Selected letters of reference are also provided.

ATI References Cartwright School District Contact Name: Dr. Alfredo Barrantes, Director of Assessment & Data Management Phone: 623.691.4000 Description: In 2005, Cartwright School District issued an RFP for Curriculum Assessment Software and ATI was awarded the contract. In 2010, at the conclusion of that contract, Cartwright School District issued an RFP for Curriculum Assessment Software and ATI was awarded the contract. Since 2005, ATI has provided the Galileo® K-12 Online Instructional Improvement System, including services related to assessment, curriculum, and reporting. In addition, ATI has provided professional development, technical support, software-use support and psychometric services. The District has purchased Scanline for offline testing and online data aggregation. In Cartwright School District, Galileo K-12 Online is currently serving 20,375 students.

Gilbert Public Schools Contact Name: Lynn Emerson, Testing Coordinator Curriculum & Assessment Educational Services Phone Number: 480.497.3392 Description: In 2004, Gilbert Public Schools issued an RFP for Online Benchmark Testing for its middle and high school students. ATI was awarded the contract and has provided to the District the Galileo K-12 Online Instructional Improvement System including services related to assessment, curriculum, and reporting since October, 2004. ATI has provided professional

Scottsdale Unified School District Contact Name: Dr. Andi Fourlis, Assistant Superintendent Phone: 480.484.6272 Description: In 2004, Scottsdale Unified School District issued an RFP for Data Technology Tools and ATI was awarded the contract. In 2010, at the conclusion of that contract, Scottsdale Unified School District issued an RFP for Assessment and Instructional Improvement Systems and ATI was awarded the contract. Since 2004 ATI has provided the Galileo® K-12 Online Instructional Improvement System, including services related to assessment, curriculum, and reporting. In addition, ATI has provided professional development, technical support, software-use support and psychometric services. The District has purchased Scanline for offline testing and online data aggregation. In Scottsdale, Galileo K-12 Online is currently serving 23,252 students.

Selected ATI Letters of Reference

Tab 6. Contract

The form of contract for any award made as a result of this proposal will be a district purchase order (issued annually), referencing this RFP, which shall be considered a part of the contract. The amount will be based upon the fees shown in the proposal, and will take into consideration previous and anticipated expenses for the forthcoming year. If your firm will require the District to sign an additional or separate contract, a copy of the proposal contract must be included with the proposal.

ATI will not require MPS to sign an additional or separate contract.

Tab 7. Required Forms

Tab 7.1. Cost Form The Cost Form follows.

Tab 7. Required Forms Tab 7.2. Offer and Acceptance Form

Tab 7.2. Offer and Acceptance Form The Offer and Acceptance Form follows.

Tab 7. Required Forms Tab 7.3. Non-Collusion Affidavit

Tab 7.3. Non-Collusion Affidavit The Non-Collusion Affidavit follows.

Tab 8. Appendix Tab 8.1. ATI Electronic Services Agreement

Tab 8. Appendix

Tab 8.1. ATI Electronic Services Agreement The ATI Electronic Services Agreement follows.

Tab 8.2. Exceptions/Deviations

Exceptions to Uniform General Terms and Conditions Mesa Public Schools RFP 15-47MP – Formative Assessment

Item 1: Page 10, Paragraph 2.D. Inspection and Testing: The Contractor agrees to permit access to its facilities, Subcontractor facilities and the Contractor’s processes for producing the materials, at reasonable time for inspection of the materials and services covered under this Contract. The School District/ public entity shall also have the right to test at its own cost the materials to be supplied under this Contract. Neither inspection at the Contractor’s facilities nor testing shall constitute final acceptance of the materials. If the School District/ public entity determines non- compliance of the materials, the Contractor shall be responsible for the payment of all costs incurred by the School District/ public entity for testing and inspection.

Page 10, Paragraph 2.D should be deleted in its entirety. See Contractor’s Termination Policy set forth below in Item #15.

Item 2: Page 11, Paragraph 2.G. Property of the School District/ Public Entity: Any materials, including reports, computer programs and other deliverables, created under this Contract are the sole property of the School District/ public entity. The Contractor is not entitled to a patent or copyright on those materials and may not transfer the patent or copyright to anyone else. The Contractor shall not use or release these materials without the prior written consent of the School District/ public entity.

Page 11, Paragraph 2.G. should be deleted in its entirety and the following inserted in lieu thereof: Intellectual Property: Contractor’s Electronic Service Agreement, incorporated herein by reference, is the governing document with regard to Contractor’s Intellectual Property licensed to Users on an annual and renewable basis. Contractor currently owns copyrights and patents on Contractor’s intellectual property. Contractor’s Intellectual Property includes patent rights (U.S. Patents 6,322,366, 6,468,085 B1, 7,065,516, 8,529,270, 8,630,577, and 8,784,114 with others pending), copyrights and trade secrets covering software, display screens, layouts, forms, item banks, learning activities, Instructional Dialogs, and professional development materials used to provide Contractor’s Electronic Services. Contractor’s Intellectual Property further includes source and object code, text, audio visual files, and images. The Contractor’s intellectual property continually undergoes development supported by funds other than from this Contract; these developments are Intellectual Property

belonging to the Contractor and will contain the Contractor’s copyright and where applicable the Contractor’s patents. Specifically note that Contractor shall continue to expand, as an ongoing activity, benchmark assessment and classroom assessment item banks, comprehensive prepared and customized assessments, quizzes, Instructional Dialogs and other content. All content developed by the Contractor constitutes part of the Intellectual Property owned by the Contractor and will be copyrighted/patented by the Contractor.

Item 3: Page 12, Paragraph 4.A. Amendments: This Contract is issued under the authority of the Procurement Officer who signed this Contract. The Contract may be modified only through a Contract Amendment within the scope of the Contract. Changes to the Contract, including the addition of work or materials, the revision of payment terms, or the substitution of work or materials, must be mutually agreed to and signed by the Procurement Officer and Contractor’s authorized representative. Changes directed by an unauthorized employee or made unilaterally by the Contractor are violations of the Contract and of applicable law. Such changes, including unauthorized written Contract Amendments, shall be void and without effect, and the Contractor shall not be entitled to any claim and this Contract based on those changes.

Page 12, Paragraph 4.A. should be kept and amended as indicated in italics.

Item 4: Page 12, Paragraph 5.C. Indemnification – Patent and Copyright: To the extent permitted by law, the Contractor shall defend and hold harmless the School District/ public entity against any liability, including costs and expenses, for infringement of any patent, trademark, or copyright arising out of Contract performance or use by the School District/ public entity of materials furnished or work performed under this Contract. The School District/ public entity shall reasonably notify the Contractor of any claim for which it may be liable under this paragraph.

Page 12, Paragraph 5.C. should be kept and the following added: This clause is subject to Warranties, Limited Warranty/Limitations of Liability and Indemnification sections of the Contractor’s Electronic Services Agreement incorporated herein by reference.

Item 5: Page 13, Paragraph 6.B. Warranties - Quality: Unless otherwise modified elsewhere in these terms and conditions, the Contractor warrants that, for one year after acceptance by the School District/ public entity of the materials or services, they shall be: 1. of a quality to pass without objection in the trade under the Contract description; 2. fit for the intended purposes for which the materials or services are used; 3. within the variations permitted by the Contract and are of even kind, quality, and quality within each unit and among all units; 4. adequately contained, packaged and marked as the Contract may require; and 5. conform to the written promises or affirmations of fact made by the Contractor.

Page 13, Paragraph 6.B. should be deleted in its entirety and the following inserted thereof: Quality: Unless otherwise modified elsewhere in these terms and conditions, the Contractor warrants that for one year after acceptance by the School District/ public entity of the materials or services, they shall be: 1. Fit for uses specified in the Contractor’s Proposal;

2. Conform to the written promises made by the Contractor.

Item 6: Page 14, Paragraph 6.C. Warranties - Fitness: The Contractor warrants that any material or service supplied to the School District/ public entity shall fully conform to all requirements of the Solicitation and all representations of the Contractor, and shall be fit for all purposes and uses required by the Contract.

Page 14, Paragraph 6.C. should be deleted in its entirety and the following inserted in lieu thereof: Fitness: Unless otherwise modified elsewhere in these terms and conditions, the Contractor warrants that for one year after acceptance by the School District/ public entity of the materials or services, they shall be: 1. Fit for uses specified in the Contractor’s Proposal; 2. Conform to the written promises made by the Contractor.

Item 7: Page 14, Paragraph 6.E. Warranties - Exclusions: Except as otherwise set forth in this Contract, including Contractor’s Electronic Services Agreement incorporated herein by reference, there are no express or implied warranties or merchant ability fitness.

Page 14, Paragraph 6.E. should be kept and amended as indicated in italics

Item 8: Page 14, Paragraph 6.G.1. Survival of Rights and Obligations after Contract Expiration or Termination: 1. Contractor’s Representations and Warranties. All representations and warranties made by the Contractor under this Contract shall survive expire upon the expiration or termination hereof. In addition, the parties hereto acknowledge that pursuant to A.R.S. 12-510, except as provided in A.R.S. 12- 529, the School District/ public entity is not subject to or barred by any limitations of actions prescribed in A.R.S. Title 12, Chapter 5.

Page 14, Paragraph 6.G.1 should be amended as indicated in strikethrough and italics.

Item 9: Page 14 Paragraph 6.G.2. Survival of Rights and Obligations after Contract Expiration or Termination: 2. Purchase Orders. The Contractor shall, in accordance with all terms and conditions of the Contract, fully perform and shall be obligated to comply with all purchase orders received by the Contractor prior to the expiration or termination hereof, unless otherwise directed in writing by the Procurement Office, including, without limitation, all purchase orders received prior to but not fully performed and satisfied at the expiration or termination of this Contract. Any directions by the school district/ public entity in writing or otherwise that seek to change the terms and conditions of Contractor’s performance must be submitted in a Contract Amendment that is agreed to in writing by the Contractor.

Page 14, Paragraph 6.G.2. should be amended as indicated in strikethrough and italics.

Item 10: Page 14, Paragraph 7.A. Right to Assurance: If the School District/ public entity in good faith has reason to believe that the Contractor does not intend to, or is unable to perform or continue performing the Contract, the Procurement Officer may

demand in writing that the Contractor give a written assurance of intent or ability to perform. Failure by the Contractor to provide written assurance within the number of days specified in the demand may, at the School District/ public entity’s option, be the basis for terminating the Contract under the Uniform General Terms and Conditions.

Page 14, Paragraph 7.A. should be deleted in its entirety. See Contractor’s Termination Policy set forth below in Item #15.

Item 11: Page 14, Paragraph 7.B. Stop Work Order: 1. The School District/ public entity may, at any time, by written order to the Contractor require the Contractor to stop all or any part, of the work called for by this Contract for a period of up to of ninety (90) days after the order is delivered to the Contractor, and for any further period to which the parties may agree. The order shall be specifically identified as a stop work order issued under this clause. Upon receipt of the order, the Contractor shall immediately comply with its terms and take all reasonable steps to minimize the incurrence of costs allocable to the work covered by the order during the period of work stoppage.

2. If a stop work order issued under this clause is canceled or the period of the order or any extension expires, the Contractor shall resume work. The Procurement Officer shall make an equitable adjustment in the delivery schedule or Contract price, or both, and the Contract shall be amended in writing accordingly.

Page 14, Paragraph 7.B. should be deleted in its entirety. See Contractor’s Termination Policy set forth below in Item #15.

Item 12: Page 15, Paragraph 7.D. Nonconforming Tender: Materials supplied under this Contract shall fully comply with the Contract. The delivery of materials or a portion of the materials in an installment that do not fully comply constitutes a breach of Contract. On delivery of nonconforming materials, the School District/ public entity may terminate the Contract for default under applicable termination clauses in the Contract, exercise any of its remedies under the Uniform Commercial Code, or pursue any other right or remedy available to it.

Page 15, Paragraph 7.D. should be deleted in its entirety. See Contractor’s Termination Policy set forth below in Item #15.

Item 13: Page 15, Paragraph 7.E. Right to Offset: The School District/ public entity shall be entitled to offset against any sums due the Contractor, any expenses or costs incurred by the School District/ public entity or damages assessed by the School District/ public entity concerning the Contractor’s nonconforming performance or failure to perform the Contract, including expenses, costs and damages described in the Uniform General Terms and Conditions.

Page 15, Paragraph 7.E. should be deleted in its entirety. See Contractor’s Termination Policy set forth below in Item #15.

Item 14: Page 15, Paragraph 8.D. Termination for Convenience: The School District/ public entity reserves the right to terminate the Contract in whole or in part at any time,

when in the best interests of the School District/ public entity without penalty recourse. Upon receipt of the written notice, the Contractor shall immediately stop all work, as directed in the notice, notify all Subcontractors of the effective date of the termination and minimize all further costs to the School District/ public entity. In the event of termination under this paragraph, all documents, data and reports prepared by the Contractor under the Contract shall become the property of and be delivered to the School District/ public entity. The Contractor shall be entitled to receive just and equitable compensation for work in progress, work completed, and materials accepted before the effective date of the termination. The cost principles and procedures provided in A.A.C. R7-2-1125 shall apply.

Page 15, Paragraph 8.D. should be deleted in its entirety. See Contractor’s Termination Policy set forth below in Item #15.

Item 15: Page 16, Paragraph 8.E. Termination for Default: 1. In addition to the rights reserved in the Uniform Terms and Conditions, the School District/ public entity reserves the right to terminate the Contract in whole or in part due to the failure of the Contractor to comply with any term or condition of the Contract, to acquire and maintain all required insurance policies, bonds, licenses and permits, or to make satisfactory progress in performing the Contract. The Procurement Officer shall provide written notice of the termination and the reasons for it to the Contractor. 2. Upon termination under this paragraph, all goods, materials, documents, data and reports prepared by the Contractor under the Contract shall become the property of and be delivered to the School District/ public entity. 3. The School District/ public entity may, upon termination of this Contract, procure, on terms and in the manner that it deems appropriate, materials and services to replace those under this Contract. The Contractor shall be liable to the School District/ public entity for any excess costs incurred by the School District/ public entity in re-procuring the materials or services.

Page 16 Paragraph 8.E. should be deleted in its entirety and the following inserted in lieu thereof:

Termination Policy: . Scope: The School District/ Public Entity (“User”) may terminate the Contract for ATI Electronic Services for cause in the event the Contractor (“ATI”) is in default as defined in the following paragraph, by giving ATI a written “Notice of Default.” ATI shall have thirty (30) calendar days from ATI’s receipt of a written Notice of Default to correct any default. In the event ATI fails to remedy the default within that time period, the User may send ATI a written “Notice of Termination.” Upon receipt of such Notice of Termination, ATI shall cease all work immediately or on the date, if any specified in said Notice of Termination, whichever is later, and thereafter take all reasonable steps to not incur additional costs associated with this Contract, except those costs necessary to closeout business on this Contract. Additional costs shall be reviewed by the User and must be mutually deemed allowable before incurred and payment is made.

. Termination for Default: The User may terminate this Contract for reason of ATI’s default if: (a) a federal or state proceeding for the relief of debtors is undertaken by or against ATI, or (b) a receiver or trustee is appointed for ATI, or (c) ATI makes an

assignment for the benefit of its creditors, or (d) ATI becomes insolvent, which shall be deemed to have occurred if ATI has ceased to pay its debts in the ordinary course of business or cannot pay its debts as they become due, whether or not ATI has committed an act of bankruptcy or is considered insolvent within the meaning of the federal bankruptcy law, or (e) ATI breaches any material term or violates any material provision of the written proposal/bid response to User’s RFP and does not cure such breach or violation within thirty (30) calendar days from ATI’s receipt of a written “Notice of Default.” ATI may terminate this Contract for any breach of the Contract by the User, including, but not limited to, the User’s failure to make payment, the User’s unauthorized use of account access information, the User’s breach of this Electronic Services Agreement including confidentiality breach, the User’s improper use of ATI’s copyrighted materials, or the User’s misuse of the interactive components of ATI’s electronic services. Prior to termination, ATI shall give the User a written “Notice of Default.” The User shall have thirty (30) calendar days from receipt of Notice of Default to correct any breach. In the event the User fails to remedy the breach within that given time period, ATI may send the User a written “Notice of Termination.” Termination of the Contract shall be effective as of the date of the Notice of Termination. ATI shall have no liability to the User for terminating the User’s access to ATI’s Electronic Services for cause.

Please Note: . The breach by User in exceeding the scope of the licenses granted in this Agreement or by breach of ATI confidential information shall result in irreparable harm to ATI for which no adequate remedy at law exists. Accordingly, upon any such breach, ATI shall be entitled to seek injunctive or other appropriate extraordinary relief, such relief being in addition to, and not in lieu of, any other rights and remedies, including the award of damages available at law or in equity. . The termination of this Agreement shall terminate this Agreement as a whole, including all aspects of ATI’s written proposal to User’s RFP or other requests.

Return of Confidential Material: Upon cancellation, expiration or termination of this Agreement and at the request of the User, ATI shall return to the User or destroy any confidential information belonging to the User and still in ATI’s possession. Confidential information is considered to be any information that identifies a particular person or particular institution.

Proprietary Rights: ATI has the right to use information that the User enters into the system to conduct educational research and publish educational research findings for such purposes as establishing the validity and reliability of scales provided through ATI’s Electronic Services.

Item 16: Page 17, Paragraph 13. Contractor’s Employment Eligibility: By entering into the contract, The Contractor warrants compliance with A.R.S. 41-4401, A.R.S. 23-214, The Federal Immigration and Nationality Act (FINA), and all other federal immigration laws and regulations.

The District/ public entity may request verification of compliance from any Contractor or Subcontractor performing work under the Contract. The District reserves the right to confirm compliance in accordance with applicable laws.

Should the District/ public entity suspect or find that the Contractor or any of its Subcontractors are not in compliance, the District/ public entity may pursue any and all remedies allowed by law, including, but not limited to: suspension of work,

Tab 8. Appendix Tab 8.2. Exceptions/Deviations

Tab 8.3. Galileo K-12 Technical Manual: Benchmark Assessment Development in the Galileo Educational Management System The Galileo K-12 Online Technical Manual: Benchmark Assessment Development in the Galileo Educational Management System follows.

Tab 8.4. Composition of a Comprehensive Assessment System The Composition of a Comprehensive Assessment System follows.

Tab 8.5. Item Development and Assessment Construction Guidelines for Common Core State Standards The Item Development and Assessment Construction Guidelines for Common Core State Standards follows.

Tab 8.6. Technology Enhanced Items for Formative Assessments in Dialogic Curriculums The Technology Enhanced Items for Formative Assessments in Dialogic Curriculums follows.

Tab 8.7. MAGI Services: Galileo® Instructional Data System Pilot Project Evaluation The MAGI Services: Galileo Instructional Data System Pilot Project Evaluation follows.

Tab 8.8. Tools for Reform: How a Failing District Used Data to Elevate Student Learning The Tools for Reform: How a Failing District Used Data to Elevate Student Learning follows.

ati-online.com How a Failing District Used Data to Elevate Student Learning

How can teachers use student assessment data to improve student learning and target their individual needs? At the Creighton School District in Phoenix, Arizona, educators wanted an assessment system that could guide teachers to make the best instructional decisions for each child. There was urgency for the task: In 2008, the Arizona Department of Education had designated Creighton as a failing district, slated for state takeover. Six of its nine schools had been labeled “Underperforming” and one as “Failing to Meet to Academic Standards.” As a high poverty, inner city, elementary and middle school district, Creighton faced an enormous challenge. So in 2008, it launched a reform initiative that led to a remarkable turnaround: Today, eight of its schools have been relabeled “Performing Plus” and one is “Highly Performing,” based on Ari- zona Learns achievement profiles. Creighton is no longer a failing district. How did Creighton achieve this dramatic improvement? A key ingredient for their success, say district leaders, was changing how they assessed students and, more importantly, how they analyzed results to fine-tune instruction. “This district,” says Dr. Lynne Spiller, Creighton’s Director of Research and Evaluation, “believes profoundly that there is no reason to assess a child if you are not going to use the data to determine the best instructional decisions for that child.” Integrating assessment with instruction and curriculum was a cornerstone of the district’s reform plan. Creighton wanted to build a system that gave classroom teachers immediate data—not just a test score but assessments that were diagnostic, showing student misconceptions about learning objectives and how to address them. The system was developed in partnership with WestEd, a nonprofit research and service agency, and Assessment Technology Incorporated (ATI), whose Galileo K-12 Online Instructional Improvement System (IIS) provided a powerful and innovative technological component for the reform effort.1 Dr. Jason Feld, Vice President of Corporate Projects at ATI, describes Gali- leo as a comprehensive set of assessment, reporting, instructional, and intervention tools “designed to support educator goals to elevate student learning.” These tools, he adds, are research-based, reliable, and aligned to both state standards and the new Common Core State Standards. For its part, WestEd provided district site and school site staff with a full menu of ongoing professional development and technical assistance focused on improving instruction, curriculum, and assessment systems in Creighton.2 WestEd and ATI drew on their shared expertise and experience as partners, working with other districts and schools on successful reform efforts.3 The marriage of intensive, high-level professional development and sophisticated but user-friendly assessment tools has helped Creighton sustain their reform effort (see table 1 below).

1 ATI began developing Galileo assessment technology in Arizona and currently provides Galileo to pre-K programs, K-12 districts, and charter schools in 35 states. In Arizona, they currently partner with 182 K-12 districts and charter schools. 2 The multi-year initiative is funded by the Ellis Center for Educational Excellence, a Phoenix-based philanthropy focused on improving inner city, high poverty districts such as Creighton. 3 Their collaborations include work with the Mesquite Elementary School, one of three schools recognized by the state of Arizona for the Best Practices as part of the Spotlight on School Success for 2004-2005. It was designated an A+ school by the Arizona Educational Foundation in 2005. Its principal, Connie Erickson, was named a National Distinguished Principal of the Year, and student achievement on AIMS (Arizona’s Instrument to Measure 1 Standards) increased substantially in a two-year period in three testing areas (Math, Reading, and Writing). See Overcoming Growing Pains: Arizona Elementary School Achieves Excellence in Record Time (ATI, n.d.) Table I Creighton AZ Learns Performance 2008-2011 Data

Prior to the reform initiative, says Spiller, the district had relied solely on blueprint assessments: “They gave us an idea each quarter how students were progressing on the grade level standards and allowed us to forecast whether they would master them by the end of the year.” While predictive of students’ risk levels on a long-cycle for their performance on state tests, the blueprint assessment data was not timely or sufficient enough for teachers to adapt their teaching to be more proactive in the classroom. As Paul Bambrick-Santoyo, managing director of Uncommon Schools, argues without regular assessment, there is “no way to track student progress through the year and no way to identify problems in time to correct them.”4 So WestEd and ATI helped the district add mid-cycle or common formative assessments (CFAs) to refine the system. Now teachers monitor student progress towards mastery of curriculum objectives with weekly CFAs and quarterly benchmarks so that they can target instruction and plan immediate interventions geared to each student’s specific learning needs. These interventions, a form of differentiated or targeted instruction called “Reteach and Enrich,” are based on student responses to the CFAs developed by an assessment committee at Creighton using ATI Test Builder technology and formative item content. They focus on particular learning objectives, aligned to state standards and adopted as the curriculum for all of the district’s schools. So the CFAs can measure in real time the actual learning that is taking place in grade level classrooms at each school. Principals, their coaching staff, and teachers work hard to analyze this data in weekly collaborative meetings.

4 Bambrick-Santoyo, P. Driven by Data: A Practical Guide to Improve Instruction, (Jossey-Bass, 2010) p. xxx

2 Reteach and Enrich: Using Common Formative Assessments (CFAs) to Inform Instruction At Creighton’s Loma Linda School, students spend a week focusing on a specific standard or performance objective, for example, understanding an author’s purpose or the main idea of an assigned text they’ve read. At the end of that initial instruction, they take an assessment, a CFA that generally involves 5-10 multiple-choice test items focused on the performance objective. This computer-scored test is usually a bubble sheet that also identifies the students, the grade, the school, and teacher. Using ATI’s Galileo, results are then analyzed and discussed at weekly data meetings involving grade-level teachers, student achievement coaches, and principals to target instructional responses to the CFA data. These collaborative teams identify which students did well enough to benefit from “Enrichment” instruction, for example, something around author’s purpose that will be more challenging for these students and engage higher order thinking based on Depth of Knowledge measures. For students who did not do so well, the idea is to redesign a “Reteach” lesson based around those students’ misconceptions about the author’s purpose or main idea. An important feature of Galileo is that test item responses include “distracters” that reveal the specific misconceptions students may have that led them to select a particular response on the CFA. The Reteach class can then address them. For example, a student may have grasped the general notion of “main idea” but have trouble with understanding “supporting details.” So the work of the team meetings is to analyze the data, determine which students have which misconceptions, and then split them into small groups to receive targeted instruction.

Data meetings operationalize Creighton’s assessment system at grade-level in each school. “They are used every single week by teacher collaborative teams,” says WestEd’s Joe Sassone,5 Director of Inte- grated Services and former assistant superintendent of the Vail Unified School District. “The power of ATI’s Galileo,” he adds, “is that it gives teachers some information as to what students are struggling with, and they can use that information to plan lessons.” At the weekly meetings, the team analyzes student misconceptions based on the CFA data, and then discusses “ideas for how to reteach this and differentiate instruction so we can help them reach the next level.” A distinctive feature of Galileo, says ATI’s Feld, is that “it gives a true measure of progress. It’s not using the traditional classical test theory where you simply look at percent scores or how many items the student responded to correctly; it allows you to measure student progress along a path of learning as

5 In 2005, Sassone was awarded the Golden Bell Award from the Arizona School Board Association for District Academic Achievement and the Spot Light Success Award for Using Data to Drive Instruction from the Arizona Department of Education.

“ATI not only helped my leadership team but also our teachers to have a more data-driven focus.” 3 — Dr. Stephanie DeMars, Principal, Loma Linda opposed to isolated test scores.” But he emphasizes that it’s not just about the technology; making the system work “requires collaborative development of an implementation plan that involves all the stakeholders—a collaborative problem solving model. There’s also management and monitoring of the plan so that it moves forward successfully. And we’ve built tools into Galileo to help districts and schools do that.” The 30-minute weekly data meetings are a component of a comprehensive reform toolkit that aims to integrate assessment with the technical assistance that WestEd provides for instruction and curriculum. The former involves intensive training on an array of research-based instructional strategies known in Creighton as Teach for Success or CT4S. The CSD diagram below has been adapted to show how the district plans and integrates instruction with assessment on a monthly cycle.

4 “We’ve done a lot of training with Creighton’s principals, their instructional and student achievement coaches on the entire cycle,” says Sassone. “So when they sit down with teachers and do planning, they have a step-by-step approach. Depending on the standard, the cycle represents about a month, including two weeks of planning and initial instruction—steps one through three—and then another two weeks of assess and reteach as shown in steps four, five, and six, which is where ATI comes into play with the CFAs.” WestEd and ATI are also helping the district move to assessments based on the Common Core State Standards. ATI’s Assessment and Instructional Design staff is currently writing new item content, including constructed response items and other innovative item types that are more appropriate than multiple-choice items for assessing certain skills and addressing depth of knowledge concerns that are a focus of Common Core State Standards. Sustaining Creighton’s reform effort, says Lynne Spiller, depends upon training all teachers in how to analyze data. She credits the new system with changing the mindset around assessment: “It helped change the paradigm from guessing, to knowing what kids need. And to do that, you need data that gives a better idea of what kids need for an intervention. It’s the biggest evidence that we have actu- ally turned the corner.”

“ATI’s Galileo provides immediate feedback to staff and has changed our approach to instruction.” — Dr. Charlotte Boyle, Superintendent, Creighton School District

5 Tab 8. Appendix Tab 8.9. Confidential/Proprietary Information

Tab 8.9. Confidential/Proprietary Information As requested in the RFP #15-47MP for Formative Assessment, ATI has indicated throughout this response the following information that shall be withheld from disclosure to others, as it is confidential and proprietary in nature:

• Federal Employer Identification Number found in the Offer and Acceptance: Contains confidential company information • Resumes: Contain confidential information • ATI Detailed Cost Proposal: Contains confidential information • References: Provide confidential client information • Letters of References: Provide confidential client information • Screen shots throughout bid: Contain proprietary content

In addition, this response bears the 2015 copyright of Assessment Technology, Incorporated. All rights to the content are reserved. With the exception of distribution within MPS, no part of this document may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording, or any information storage and retrieval system, without permission from the publisher. Assessment Technology, Incorporated, Publishers. Tucson, Arizona, U.S.A.

Tab 8.10. Text References Haladyna, T.M. (2004). Developing and validating multiple-choice test items. Mahwah, NJ. Lawrence Erlbaum Associates, Inc. .