Data Driven Decision-Making in South Dakota

DATA DRIVEN DECISION-MAKING IN SOUTH DAKOTA

EFFECTIVE USE OF STATE DATA SYSTEMS

JOHN ARMSTRONG

SD This paper is based on research funded in part by the Bill & Melinda Gates Foundation. The findings and conclusions contained within are those of the author(s) and do not necessarily reflect positions or policies of the Bill & Melinda Gates Foundation.

INTRODUCTION...... 3

THE IMPORTANCE OF DATA FOR THE EXECUTIVE...... 4

THE IMPETUS FOR SDBOR DASHBOARDS...... 6

REACTIONS TO SDBOR DASHBOARDS...... 11

DISPELLING MYTHS WITH DATA...... 12

CONCLUSION...... 13

Analysis of student-level data to inform policy and promote student success is a core function of executive higher education agencies. Postsecondary data systems have expanded their collection of data elements for use by policymakers, institutional staff, and the general public. State coordinating and governing boards use these data systems for strategic planning, to allocate funding, establish performance metrics, evaluate academic programs, and inform students and their families. The State Higher Education Executive Officers association (SHEEO), as part of a project funded by the Bill & Melinda Gates Foundation (BMGF), surveyed state coordinating and governing boards on their collection and use of postsecondary student-level data. Following this, SHEEO identified seven states whose survey responses indicated an exemplary use of data in specific subject areas. In-person interviews were conducted by SHEEO agency staff in seven states selected for follow-up.

In 2015, SHEEO visited the South Dakota Board of Regents (SDBOR) to discuss the presentation of data to external stakeholders, its unique focus on Native American students, and the relationship between SDBOR leadership and its research staff. The Board of Regents governs the system of public higher education in the state. It has constitutional authority over South Dakota’s six public universities as well as two special schools 1 (see Figure 1). While each university in South Dakota has its own institutional research department, SDBOR staff members serve as administrators for campus data systems and have access to all of the underlying data. Regents Information Systems (RIS) contain unit record information on college readiness, enrollment, completions, degree trends, transfer outcomes, employment outcomes of graduates, and a variety of course- level data, among many other data elements.2

FIGURE 1: LOCATION OF SOUTH DAKOTA BOARD OF REGENTS CAMPUSES AND ADMINISTRATION

SDBOR has generated an annual Fact Book using the information institutions have submitted to RIS since 1985.3 In recent years, SDBOR has also generated a series of publicly available dashboards using RIS data and nationally available data sets. Many of South Dakota’s responses to SHEEO’s survey on data systems referenced the dashboards, which are continually updated to reflect the most recent RIS data. The dashboards produce a number of positive outcomes for SDBOR and serve a wide variety of higher education audiences. They result from close collaboration between agency leadership and research staff.

1. Special schools include the South Dakota School for the Deaf and the South Dakota School for the Blind and Visually Impaired. 2. John Armstrong and Christina Whitfield, The State of State Postsecondary Data Systems: Strong Foundations 2016. Boulder CO. State Higher Education Executive Officers association. 51 (May 2016) Available at http://www.sheeo.org/resources/publications/ strong-foundations-2016 3. Available here: https://www.sdbor.edu/mediapubs/factbook/Pages/default.aspx

As higher education agencies collect data for planning, program review and finance, the need for agency leadership to thoroughly understand the data and its implications is paramount. Business intelligence methods are an increasingly important tool for higher education executives. Higher education executives can look to public management information system (PMIS) theory from the business community for ideas on improving the use of data to drive decisions. PMIS best practices include collaboration among government agencies, and the intentional recruitment and retention of information system staff.4 SDBOR’s executive leadership has shared data linkages with other agencies, used publicly available data where available, and prioritized the hiring of talented research analysts to inform its decision-making process. In this respect, they have incorporated PMIS theory recommendations, which is a public-sector application of business intelligence techniques. Business intelligence has an increasingly important role in executive higher education agencies.

Throughout the course of the interview with SDBOR staff, it was clear that communication among the research staff of the Board of Regents is closely aligned with the vision of the senior leadership team. At the time of the visit, SDBOR was led by Dr. Jack Warner, currently an associate professor of higher education at Johnson Wales University. Warner served as the state higher education executive officer (SHEEO) of the Rhode Island Board of Governors for Higher Education and as vice chancellor of the Massachusetts Board of Higher Education before leading the SDBOR as executive director and chief executive officer.

In Warner’s remarks, he noted that he has invested in institutional research throughout his leadership tenures. He thinks it is vital for agency leadership to understand, as deeply as they can, the data on students attending higher education institutions in their state. This desire for understanding prompted his investment in research and analysis staff for the agency. After assembling the necessary information, leaders can choose which key metrics to focus on in order to move the needle and improve student success. This practice is consistent with literature on effective policy analysis.5

All coordinating and governing boards across the country have access to basic enrollment statistics and, in most cases, demographic information about their students. Warner advocates for starting with these commonly available measures to paint a picture of the overall higher education landscape in the state before deciding which metrics and areas of inquiry will require further research and what questions are needed to generate new and critical information. “It’s up to the leader to craft the questions.” From there, the agency must translate the questions into usable data that informs policy, guides decision-makers and influences an agenda that improves student outcomes across the state.

To that end, the staff who researched the questions Warner asked worked closely with him and other key stakeholders throughout their research process. Members of the research team are on SDBOR’s senior staff and regularly meet with agency leadership to report on progress, adjust lines of inquiry, and translate information into decision points.

4. Barry Bozeman and Stuart Bretschneider, “Public Management Information Systems: Theory and Prescription,” Public Administration Review 46. pp 475-487 (1986). 5. Eugene Bardach, A Practical Guide for Policy Analysis: The Eightfold Path to More Effective Problem Solving. Third Edition. CQ Press, Washington, D.C. (2009).

© 2016 by the State Higher Education Executive Officers Association (SHEEO) 5 One example of a report that followed this data to question policy is SDBOR’s report on Native American students attending South Dakota universities, entitled, Like two different worlds.6 Based on the basic demographic data that was available to Warner and SDBOR when he started, he developed the question of how to improve outcomes for these students. More information was needed to ascertain the nature of the gap in access between Native American and white students. He directed the research team to explore the issue further before coming up with a series of policy recommendations aimed at improving the student experience for Native Americans.

The report identified a series of barriers that Native American students face when pursuing a postsecondary degree or certificate. These barriers include stark poverty, a lack of mentorship, fear of culture shock and leaving family, and a lack of quality information about postsecondary education, among others. To address these barriers, the report recommended improved outreach to Native American high school students— particularly those living on reservations—that is family-centered and contains a mentoring approach in the admission and scholarship process for postsecondary education. It also recommends expanding scholarship and financial aid opportunities for these students and improving the cultural footprint on South Dakota campuses to improve retention.

Warner described the report as providing SDBOR with a deeper understanding of the problem and influencing SDBOR’s strategy for addressing it. As a result of this report and the grant that funded it, SDBOR was able to put two staff in the field who were specifically focused on these students and placed an academic advisor on each of the campuses that work with low-income students and provide summer bridge programs for Native American and other students. The continued collaboration between Warner and the research team influenced the report’s process, its recommendations, and the actions implemented to address the problem after the report’s release.

The timeline for using data to inform research, which then informs policies and agendas, is a model for state coordinating and governing boards. Additionally, it is critical for research staff to understand agency leadership’s vision for higher education, and which questions are of critical importance. Likewise, agency leadership benefits from investing in talented research staff who are able to provide critical information on state higher education. Regular communication between these two groups is essential for successfully implementing a data driven decision-making agenda.

6. Available here: https://www.sdbor.edu/mediapubs/documents/americanindiancollegegoingstudybor1213.pdf

The method of using information to articulate strategic priorities and drive institutional change, enabled by collaboration among leadership and research staff, is further assisted by having transparent, detailed information for a wide variety of audiences. In addition to interviewing Warner, SHEEO staff spoke with Dr. Paul Turman, the system vice president for academic affairs at SDBOR and Dr. Daniel Palmer, the assistant vice president for institutional research and planning. Palmer first had the idea for the dashboards after continually releasing reports and white papers only to be asked additional questions and given requests for analysis by specific subpopulations. By putting the data in the hands of the stakeholders, structured in a way they can easily access the information and explore drill-down topics, many potential questions by Board of Regents members can be answered on demand.

For Turman, this is contrasted with previous reporting processes at SDBOR. Previous reports submitted to regents were static tables, charts and figures that were continually updated, with additional detail being processed by follow-up reports in response to requests from the regents. The dashboards allow users to manipulate elements based on their specific questions, and explore the information at the high school district-level (for data related to college readiness) and the postsecondary institution level to see differences not only for subpopulations but across secondary and postsecondary schools.

Both Turman and Palmer described the process of having many anticipated questions and follow-ups available to be explored in the dashboards as an efficiency gain for their research department. When, in the past, Board of Regents members requested additional breakouts or district-level data, it often meant that research staff would have to gather additional information and create a new report in order to address their questions. Many of these inquiries are now able to be answered in a centralized location.

Palmer described the process of taking the unified data from all SDBOR campuses and putting it into an online dashboard as a much faster process than anticipated. The amount of time spent synthesizing the RIS information and developing layouts for dashboards has offset the time that was previously spent doing additional follow-ups based on regents’ or legislators’ questions. SDBOR has also had a commitment to make the RIS data transparent and useable for a wide variety of audiences. There was a need for institutional stakeholders, state policymakers and the general public to have better access to information on postsecondary enrollment, completions, course outcomes and information about South Dakota higher education. The dashboards enable this access and transparency.

© 2016 by the State Higher Education Executive Officers Association (SHEEO) 7 EXAMPLES OF SDBOR DASHBOARDS Each of the sixteen dashboards available on SDBOR’s website7 has a research question that informs the user about what the underlying data can answer. Examples of lead-in questions to the dashboards include: “How well do first-year students from South Dakota high schools perform during their first year of college; what are the college-going rates of South Dakota’s high school graduates, and where do they enroll; and how does South Dakota stack up against other states with respect to the cost of a public university education?” The high school transition dashboard, for example, contains school district-level enrollment figures for each South Dakota university as well as ACT score averages and remedial course enrollment rates (see Figures 2 and 3).

FIGURE 2: HIGH SCHOOL TRANSITION DASHBOARD

Aberdeen Central High School HS / County Students SDBOR High School Transition Report 2014 Grad Year 92 Aberdeen Central High School University All

Enrolled by University Admission Reqs Met ACT Mean Scores and Benchmark Attainment x l l

Aberdeen 7.6% o o (ACT Mean Scores) Central High o o h h U U School (7) c c S S S S

H H Composite English Reading Math Science h h B B

All Other High 9.0% i g i g 23.0 23.3 23.4 23.2 Schools H H 22.5 22.9 22.5 (242) 22.1 21.9 l l 21.2 a a 97.8% Aberdeen 2.2% t r t r 20 Central High Did Meet (2) C e n C e n U U School

D S D S All Other High 6.1% Schools (164) d e n d e n 10 b e r b e r A A Aberdeen 58.7% Central High (54) 0

U U School s s l l N S N S All Other High 5.7% o o o o

Schools (153) h h c c (ACT Benchmark Attainment) S S

Aberdeen 2.2% h h Central High i g i g 98.9% T T (English) (Reading) (Math) (Science)

(2) H H

School

M M Did Meet 90% e r e r 80.4% 82.6% D S D S

All Other High 9.0% t h t h S S 80%

O O 72.8%

Schools

(242) l l 69.6% l l 70%

A A 62.9% Aberdeen 17.4% 60% Central High 50.0%

U U (16) School 50% 45.7% 44.8% D S D S

S S All Other High 44.5% 40% Schools (1,202) Academic Distinctions 30% Aberdeen 12.0% All Other 20% Central High (11) HS High Schools 10% D D School

U S U S All Other High 25.8% SDOS Recipients 26.1% 33.7% Schools (698) Regents Scholars 29.3% 39.6% Aberdeen Central High School 0% 20% 40% 60% 80% Took AP Course(s) 19.6% 17.5% All Other High Schools

7. www.sdbor.edu/dashboards

Aberdeen Central High School HS / County Students SDBOR High School Transition Report 2014 Grad Year 92 Aberdeen Central High School University All

Placement into Remedial Coursework First Year Outcomes (GPA, Credits) by Remediation Group x(Circle size indicates number of students) No Rem. Placement 57.6% 72.0% None Rem. English 15.2% 15.4% 3.0 Rem. Math 39.1% 23.1% None Rem. Eng. AND Math 12.0% 10.5% 2.5 0% 20% 40% 60% 80% Math or English Math and English Aberdeen Central High School A P

G Math or English All Other High Schools 2.0 r Y e a s t 1 e

Retention into 2nd Semester in SDBOR System g

a 1.5 Math and English e r v A

85.9% 14.1% 1.0 Aberdeen Central High School Retained

0.5 Aberdeen Central High School

89.6% All Other High Schools All Other High Schools Retained 0.0 0 5 10 15 20 25 Average 1st Year Completed Credit

The High School Transition report uses institutional and secondary district-level data that analyze unique students in RIS. However, other dashboards present South Dakota in context with other states across the country using nationally available data sets. To answer a question about whether South Dakota is a net importer of first-time college students, Palmer pulled data from the Integrated Postsecondary Education Data System (IPEDS). While this data is available publicly, SDBOR is one of the few examples of state-level postsecondary agencies presenting national information alongside other state-specific data elements. The net migration dashboard contains information on the student exchange rates of South Dakota and all other states. While the dashboard defaults to view information about South Dakota, it contains similar information for all other states. State agency personnel from other states could find the information useful (see Figure 4).

Select a Measure SDBOR Freshmen Migration Dashboard Net Migration Balance Click map to select a state Year (for map) Sector 2014 All

-2136 2629 1 5

1223 1 2 3 1 2

-4992 1181 1 0

2786 618 3 8 0 0 7 9 1225 2618 5 708 4 3759 9 2 609 1310 5988 1 8278 7871 8 0 882 - 6 5787 - 1 15060 -5219 5304 2 4 -17438 7652 -27505 0 0 1 0 1 -1273 4989 0 0 2 0 4 0 6 0 8 0 1 2 2 -26 2 2 2 2 2 2 4190 1717 3151 3459 4097 4217 -12435 -7607 Annual Trends for Selected State 4602 1729 6278 7202 109 3313 6513 State-by-State Exchange Data for Selected State 4148 8088 -1522

-9601 2014 1718 817 MN 750 IA 270 1226 Foreign 84 Sector Trends 957 1191 for Selected State CA 70 583 CO 66 Private 2-yr 670 WY 46 Private 4-yr 238 176 97 389 WI 41 21 11 Public 2-yr -68 -51 -29 -3 -34 -5 IL 40 Public 4-yr -36 -7 -73-58 -190 -123 -59 -56 -97 TX 26 2000 2002 2004 2006 2008 2010 2012 2014 FL 18

Palmer also uses nationally available data sets to provide users with context for workforce indicators in the dashboards, including labor force participation and projected demand for specific occupations. In response to Board of Regent inquiries about graduate salaries, Palmer built a dashboard that displays state comparisons of wages by occupation. The dashboard uses publicly available data from the Bureau of Labor Statistics (see Figure 5).

Select a Measure SDBOR Occupational Wages Dashboard Average Annual Pay

Average Annual Pay in 2015 Show Bars as: States District of Columbia $80,150 Massachusetts $59,010 New York $57,030 Show Occupations by: Connecticut $56,280 By occupational category Alaska $55,760 California $55,260 By specific occupation New Jersey $54,950 Maryland $54,630 Select an Occupation/State Please ensure all filters havWea sahi nsgetolnection. $54,010 All Occupations Virginia $51,670 Architecture & Engineering $51,180 Colorado Arts, Design, Entert., Sports, & Media Rhode Island $50,780 If they do, but you are still seDeeilanwgare this message, you may $50,300 Building & Grounds Cleaning & Maint. need to select a new geographIylli nooirs a different combination $49,970 Business & Financial Operations Minnesota $49,740 Community & Social Service of filters, since the BureaNue wo Hfa Lmpashbiroe r Statistics does not $48,710 US Total $48,320 Computer & Mathematical provide all estimates for all geOoregornaphies. $48,100 Const. & Extraction Hawaii $47,740 Education, Training, & Library Texas $46,560 Farming, Fishing, & Forestry Pennsylvania $46,550 Michigan $46,310 Food Preparation & Serving Related Vermont $46,060 Healthcare Practitioners & Tech. Wyoming $45,850 Healthcare Support North Dakota $45,660 Georgia $45,420 Installation, Maint., & Repair Arizona $45,310 Legal $44,750 Ohio Life, Physical, & Social Science North Carolina $44,170 Utah $44,130 Management Wisconsin $43,930 Office & Administrative Support Missouri $43,640 Maine $43,260 New Mexico $43,170 Kansas $42,930 Florida $42,860 Nevada $42,800 - Chart Options - Nebraska $42,630 Indiana $42,070 Year Alabama $41,920 2015 Iowa $41,840 Oklahoma $41,820 Tennessee $41,300 Show/Hide US Total Line Kentucky $40,880 Show Idaho $40,810 Louisiana $40,810 Montana $40,620 Show/Hide All Occupations Line South Carolina $40,580 Show West Virginia $39,100 South Dakota $38,820 Arkansas $38,540 Split by Typical Education Mississippi $37,620 (use with "Show Bars as Occupations-Specific") 0K 10K 20K 30K 40K 50K 60K 70K 80K 90K No

By framing each dashboard with a research question and using all relevant data sources available to the Board of Regents, the research team has constructed a transparent, detailed and user-friendly interface that is available to a wide variety of audiences.

The primary audience for the SDBOR dashboards is the Board of Regents. In regular reports to the regents, SDBOR research staff present information about student success and finance indicators. After presenting topics of importance to the regents, including content from the dashboards, this information becomes available to SDBOR for interacting with the state legislature.

Legislators and regents have reacted positively to these new tools, according to Warner. In particular, when policymakers were presented with national-level data—even data that was unflattering for South Dakota—the information gave them confidence in the strategic plan outlined by the Board of Regents to improve student outcomes. "Legislators will invest in confidence…That’s what we try to do—use data to build credibility."

In addition, the dashboards have resulted in greater internal confidence in the data submitted to the RIS. If policymakers are the primary audience for the dashboards, then institutional staff are a secondary audience. According to Palmer and Turman, institutions have reacted positively to seeing information from their campuses helpfully presented alongside their peer institutions, and they have confidence that the information is being correctly presented.

Finally, the information in the dashboards is available for use by the general public, including prospective students, parents and educational researchers. Palmer has presented his dashboards to national audiences and invited his peers in other states to use the information presented by SDBOR.

The dashboards have been a way to easily combat a persistent and false narrative about higher education in South Dakota. According to SDBOR staff, there was a perception among state legislators that South Dakota was experiencing “brain drain,” the departure of high school graduates who seek their college education out of state and don’t return as adults. One of the key takeaways from the nationally available data sets displayed in the dashboard was that South Dakota is actually a net importer of students. Displaying this information alongside information collected from the institutions allowed regents and other policymakers to see data that dispelled the myth of South Dakota’s “brain drain.”

Warner credits the dashboards with changing the perceptions of regents and legislators about student migration. According to Warner, the issue has been present in the minds of policymakers for some time. In the late 2000s, South Dakota lowered its out-of-state tuition rates in an effort to reduce the number of students who migrated. Local media has linked to the migration dashboards with the phrase, "Actually, we're in the black,” implying surprise that South Dakota is currently a net importer of students.8

Alongside the information on student migration, the dashboards also present labor market outcomes of South Dakota college graduates. “Legislators seem to be most interested in students who start in South Dakota or get a degree from us. Are they taking jobs in South Dakota?” Using data from RIS and the National Student Clearinghouse, SDBOR determined that 60.2% of graduates from South Dakota institutions were present in the South Dakota labor force one year after graduation.9 As of June 2016, South Dakota has the lowest unemployment rate in the nation.

8. http://madvilletimes.com/2014/11/south-dakota-enjoys-decade-long-trend-of-increased-college-freshman-imports 9. This figure does not include those who are enrolled in graduate school, employed by the federal government or military, or are self-employed.

South Dakota’s response to SHEEO’s survey on state data systems served as the impetus for gathering more information on the Board of Regents’ use of postsecondary student unit record data. In response to the question, “How has the student unit record system provided the greatest value to your state?” they responded, “It would be difficult to understate the advantages of a unified data system such as ours. It provides us the ability to quickly and reliably extract comparable data for all of our campuses, which in turn leads to high internal and external confidence in our analyses.” By publicly displaying much of the available information in the unified RIS database, SDBOR allows the wealth of information collected to be put to use by policymakers, institutional staff, researchers and the general public.

South Dakota is not the only state to publish interactive tools that allow for the general public to drill down into specific student success indicators. However, their dashboards, coupled with the close relationship between research staff and agency leadership, serve as an example to the higher education research community of the value of data transparency. Transparent and easily understandable figures and charts enable internal and external confidence in the data (allowing for data to be updated if mistakes are discovered) and reduce burden on research staff by anticipating questions that could be answered with summary-level data.

We encourage the higher education research community to adopt the “question first” model of conducting research, to invest in quality, trained research staff who closely collaborate with agency leadership to articulate agendas, and to make information publicly accessible to a broad audience.

© 2016 by the State Higher Education Executive Officers Association (SHEEO) 14 STATE HIGHER EDUCATION EXECUTIVE OFFICERS 3035 CENTER GREEN DRIVE, SUITE 100, BOULDER, COLORADO, 80301 303.541.1600 • SHEEO.org