Next-Gen MCAS Communications Workgroup Massachusetts Department of Elementary and Secondary Education 75 Pleasant St., Malden, MA 02148 -- Room 303 Tuesday, June 14, 2016, 11:00 am to 1:00 pm

WORKGROUP MEMBERS AND MEETING PARTICIPANTS

New England Association of Teachers of English Beth Herman-Davis Board Member (NEATE) Beverly Miyares Professional Development Specialist Massachusetts Teachers Association (MTA) Massachusetts Association of School Bill Lupini Superintendent Superintendents (MASS) Massachusetts Association of Science Teachers Bob Gilmore President (MAST) MCAS Chief Analyst and PARCC Massachusetts Department of Elementary and Bob Lee Coordinator Secondary Education (ESE) American Federation of Teachers–Massachusetts Dan Murphy Director of Education Policy (AFT-MA) Assistant Director of Literacy and Massachusetts Department of Elementary and Dave Buchanan Humanities Secondary Education (ESE) Massachusetts Charter Public Schools Association Erica Brown Chief of Policy and Practice (MCPSA) Gary Gilardi State Coach Massachusetts Teachers Association (MTA) Massachusetts Council for the Social Studies Gorman Lee President (MCSS) Massachusetts Association of School Committees Heather Peske Sr. Associate Commissioner (MASC) Massachusetts Association of School Committees Jake Oliveira President (MASC) Massachusetts Department of Elementary and Jass Stewart Special Assistant to the Chief of Staff Secondary Education (ESE) Massachusetts Department of Elementary and Jeff Wulfson Deputy Commissioner Secondary Education (ESE) Massachusetts Secondary School Administrators’ Martin Geoghegan Principal Association (MSSAA) Massachusetts Department of Elementary and Matt Deninger Coordinator, Delivery Secondary Education (ESE) Director of School Improvement Grant Massachusetts Department of Elementary and Matt Pakos Programs Secondary Education (ESE) Special Assistant to the Deputy Massachusetts Department of Elementary and Maureen LaCroix Commissioner Secondary Education (ESE) Massachusetts Department of Elementary and Michol Stapel Associate Commissioner Secondary Education (ESE) Massachusetts Department of Elementary and Nate Tashjy Communications Intern Secondary Education (ESE) Massachusetts Department of Elementary and Rob O’Donnell Sr. Policy Analyst Secondary Education (ESE) Massachusetts Parent Teacher Association Stephanie Gray President (Massachusetts PTA) Association of Teachers of Mathematics in Steven Rattendi Past President Massachusetts (ATMIM) Senior Advisor to the Commissioner for Massachusetts Department of Higher Education Sue Lane P-16 Access and Alignment (DHE) Massachusetts Organization of Educational Theresa Craig Board Member Collaboratives (MOEC)

MEETING NOTES 1 Review of Spring 2016 testing

Stapel: We are wrapping up with testing. 70% of students are taking PARCC; 60% on paper. We are now moving into reporting. Reports will be released to parents in late fall. The PARCC service center is much improved, and Student Assessment fielded lots of calls. State reporting will not be possible because there’s no representative sample. But we will report out district- and school-level numbers. This is a one-time occurrence because so many students took PARCC this year, and all students will be back on a single testing system in 2017.

Wulfson: Absentee rate was very low – around 2% – on computer-based testing. This is the same as previous years.

Q&A

Does the lack of a representative sample this year affect equipercentile linking?

Will preliminary results for MCAS and PARCC be released at the same time?

Do absentees include refusals?

Lee: Equipercentile linking will not be impacted. It was set last year, and we don’t recalculate it every year.

Stapel: Full public reports will come out at the same time. Our policy is to farm out data to districts as it’s scored. This helps with reviewing of data and the discrepancy resolution process. Then we have it cleaned up for the summer. Official reports and parent reports are released in September.

Lee: The paper-based test scores are released a little later. But my understanding is that many of them have been scored. Our policy is to alert each school when their data is available.

Wulfson: Absences include refusals. There is no opt-out option in our state. There were small pockets of opt outs. But we’re hoping this is part of the transition and things will go back to normal when there’s one MCAS.

Procurement Workgroup update

O’Donnell: We are in the middle of the review process. Vendors were in for demonstrations and the review team has scored the applications. The Commissioner has been provided recommendations. Once the Commissioner makes a selection, we will go into contract negotiations. We are looking to make an announcement this month. Both proposers are solid firms with lots of credentials.

Q&A

Did Measured Progress suggest Pearson?

Wulfson: Yes, they proposed using the Pearson platform. Our contracts with vendors go beyond the technology platform, however. We work with them on planning, management, psychometrics, etc. We’ve used both vendors under various contexts. The team was asked to look at the quality of proposals. There’s no extra credit for doing work with the Department.

Review of standard setting design process

Lee: The request for responses called for the winning bidder to form a standards setting/policy committee. The board said to use the best of both worlds. For example, PARCC uses 5 levels, MCAS uses 4. A lot of the early work of the committee will be identifying key policy decisions and benchmarking. Is the test signaling that a student is ready to do college level work? If not, then maybe it’s only two levels, or maybe you want three levels.

Q&A

With MCAS, what does advance signal? That doesn’t signal college readiness?

Lee: It’s sophisticated; beyond strong. However, it wasn’t designed with the college readiness notion we have now.

2 Computer-based testing (CBT) transition plan

Wulfson: The consensus by the Test Administration Workgroup is that we don’t want to leave it open and have some districts attempt to transition to computer-based testing in 2019 – which is the target date to have all testing on the computer. We learned from PARCC that the learning curve is easier if you do it in stages. Our early thinking is for next spring to have all students in grades 5 and 8 take the next-generation MCAS on the computer. For those districts that indicate they are not ready, we will work with them to find solutions. We will continue to give paper tests, and paper will be an option if you can’t do CBT for 5 and 8 next year. It’s important for districts to try to test on the computer next year, as well, because hold harmless is in place.

Q&A

I thought the whole school had to take the same test format?

60% of the PARCC districts did it on paper, correct?

Do we know how many districts implemented technology at the upper grades? The elementary buildings are normally the older buildings.

There is this bias that kids with more technology exposure have a better chance of doing well on tests.

Have you looked at what devices students are using compared to their scores? Screen format was an issue, for example, on some devices.

Wulfson: Having the entire school administer the same test mode is a PARCC legacy. This is not an issue for the next- generation MCAS. In addition, we were aware that folks were concerned with mode effect. From our standpoint, one advantage of moving toward CBT in 2019 is that there will be primarily one mode.

Lee: There is no data to support that only low-income schools were doing worse with online testing. It was also the case for high-income schools. So student’s access to technology didn’t appear to bias the results in a significant way.

Wulfson: We want to make sure that the platform we have can be used across devises, and for the next testing season, we are only focused on the lower grades. The legacy MCAS remains in place for the class 2019. We have thoughts from the High School Testing Workgroup that we are mulling over. We want to present our latest thinking to the Assessment Oversight Committee over the summer.

Q&A

When will the next-generation MCAS be administered?

How are timed/untimed decisions to be made?

Other accommodations. How will those decisions be made?

When does item selection and development begin?

Stapel: We released the testing schedule last week. It’s a single, longer window. We heard from the Test Administration Workgroup that people were not interested in maintaining two testing windows. High school testing is still outside of this decision until testing options are finalized. We’re still making decisions around timed vs. untimed, the number of sessions, etc. The testing window looks long, but it’s up to the district to create its own local window. The Accessibility Workgroup looked at accommodations. Concerning test item development, ESE will work with the vendor to create an item bank. This work will start this fall. We’ll manage PARCC and MCAS items, then we’ll see where there are gaps, and new items will be developed. Educators get to review all the items and we have a special bias committee. The test content committees bring a practitioner’s view to each item.

Update on technology/digital readiness

3 Deninger: The Massachusetts School Building Authority is making $10 million in zero-interest loans available to districts. There will be applications for the state bond bill grant. If a district is unsuccessful in securing a grant, it will have access to the no-interest loan.

Wulfson: A lot of the need is simply in devices, and the prices of these keep coming down. We are encouraging districts to place these costs in their operating budgets. The larger expenses are servers, etc. A small number of districts have very little technology, and this is where we want to leverage e-rate. We’ll sit down with districts and provide guidance on how to access funding. On a different note, some districts have been piloting PARCC’s Partnership Resource Center. Going forward, we’re not electing to do that in MA. If districts want to do it, they can do it for a fee. We had about 1,500 users trying it out, but we don’t have a sense of its usefulness. But it’s now a local decision with local costs.

Update on project communications

Stewart: The board has expressed a strong desire for the Department to aggressively promote the work of the assessment workgroups and the standards review panel. Some of the things we are doing include:  Tele Town Hall. On June 8, the Commissioner participated in a Tele Town Hall meeting with former Massachusetts Teacher of the Year Audrey Jackson. This statewide conference call focused on student testing, the next-gen MCAS assessment, and the state’s learning standards. Organized by Stand for Children, more than 12,000 residents participated.  Media outreach. Our ongoing, targeted media outreach to statewide and local media includes press releases, opinion pieces, and an upcoming press event where reporters will have the opportunity to interview educators onsite at the July ELA/math standards review panel.  Conference presentations. The next-generation MCAS team has scheduled 13 presentations with statewide education associations. Starting in May, five presentations have already occurred and the sessions will continue into November. These sessions provide hundreds of educations with the opportunity to get the most up-to-date and accurate answers to their questions.  Monthly e-newsletter. Later today, we will distribute the first edition of a newly designed, monthly e-newsletter. Originally targeted to state legislatures (in circulation for the last two months), the new version will also include outreach to 56 statewide education and parent associations, both teachers’ unions, superintendents, charter school leaders, principals, the hundreds of educators who applied to the various next-generation MCAS workgroups, and all ESE staff.  Standards review website. Also this month, we will launch the new standards review webpage, featuring updates on the ELA and math standards review, as well as the revisions to the HSS standards and the development of the associated assessment.  Additional outreach activities. In addition to the items above, we are exploring local cable television opportunities, planning a back-to-school packet for superintendents and principals, and a media outreach campaign under MassIT’s new marketing division, DigitalMass.

Wulfson: We are looking for feedback. We have a history of communicating to the field, but not to the general public.

ELA/Math standards review; HSS standards revision and assessment development

Wulfson: (Introduced Heather Peske, who is now leading the Curriculum and Instruction unit.)

Buchanan: Concerning the review of the ELA/math standards that started in February, we’ve had three meetings of a combined group of ELA/math educators. The first meeting was an orientation and the second meeting was reviewing themes from the survey, which remains open to this day. We had our third meeting on June 3 to review edits we had made during the previous meeting. Our plan is to have a fourth meeting in July to pull the remaining threads together, and to clarify changes that will go to the board in late September. We had 700 people provide comments online. Regarding the history/social science standards, we are in the process of finalizing the panel. The panel members and the online survey will be posted this month. We’ll use the survey responses as guidance for the panel’s discussion and suggested revisions. We expect the panel to meet into next year and the following year.

Wulfson: The board has signaled that it will add HSS to the MCAS testing program. However, this is years away, and will follow the revisions to the frameworks.

Accountability system transition update

4 Wulfson: Accountability and the next-generation MCAS are not the same project, but there is overlap. And I thought it would be great to have Matt Pakos bring us up to speed. The Every Student Succeeds Act (ESSA) takes place next year, and dovetails nicely with our new testing system.

Pakos: ESSA was signed by President Obama on December 10, 2015. This reauthorizes the federal Elementary and Secondary Education Act (ESEA), which replaced No Child Left Behind (NCLB). It maintains certain accountability requirements for schools, which take effect in SY 2017-18. The Department’s ESEA/NCLB flexibility waiver expires on August 1, 2016. ESE is now gathering input from a wide variety of stakeholders to plan for the 2017-18 transition. From April to June 2016, the Department is hearing from stakeholders about possible accountability options. So far, we’ve held 6 conference calls and 14 meetings are scheduled through mid-July. In addition, stakeholders have sent over 1,800 comments using our online feedback form (final number of respondents was over 2,200). The online survey is open until June 17. From June to September 2016, ESE staff will be modeling different options. And finally, from September to December 2016, we will be developing a final set of proposals for what the new accountability system should look like.

Q&A

Next spring will be the first administration of new test. It’s a hold harmless year. The second year would be the second administration of the test. So you would have to set up rankings, etc. on one test year instead of multiple testing years?

Can the hold-harmless data be used later in aggregate years to push a district down a level?

Wulfson: There are many issues Matt’s team is trying to deal with – equating different tests, using tests from years that are held harmless, etc.

Pakos: It’s complicated. We can use multiple years of data, but there are some technical issues. One possibility could be a policy decision that we cut the cord and start building up to multiple years of data from the new assessments. Hold harmless in this context means a district or school's level can’t go down based on the data for that year. At the moment, there is no policy that prevents the state from using the data from the hold-harmless years to assign future levels.

Process to get to final recommendations

Stewart: We’ve developed a process to get us from the workgroup recommendations to final recommendations that need to either be approved by the Commissioner or the board. From January to May of this year, most of the workgroups have met and published their recommendations. Notable exceptions would be any work dealing with test item development because this work can’t begin until the vendor is selected. By the end of this month, ESE will have completed an analysis of the recommendations – pros, cons, considerations, etc. We’ll then put this document out for public comment over the summer and into the early fall. We’ll then review and integrate the public comments, get final approval from the Commissioner, then present our recommendations to the board by the end of the calendar year. In some cases, because of logistical reasons, decisions have to be made on a faster timeline. In those cases – like the CBT transition plan – staff will work with the Commissioner for approval so that information can be published sooner.

5