The Academic Program Review Guide

DUTCHESS COMMUNITY COLLEGE The Academic Program Review Guide

APR revised 6/21/2017 Table of Contents

APR revised 6/21/2017 INTRODUCTION

PURPOSE

The Academic Program Review (APR) at Dutchess Community College enables faculty and administrators to evaluate the success of the academic programs at the College. Transfer APRs examine how the programs meet the institutional objective to provide courses and programs to prepare students for transfer to four-year colleges and universities. Occupational and Certificate APRs examine how the programs succeed in meeting the institutional objective to provide courses and programs that prepare students for employment.

The process assures the college community that each program is part of a concerted effort to provide coursework that will enable students to meet their academic and career goals. The process also addresses the appropriate use of college resources to support academic programs.

OVERVIEW

SUNY requires the assessment of every college program every six years. The APR is multi-step evaluation process that can normally be completed in two-semesters. Program Chairs begin the APR with an in-depth self-study which includes analysis of the curriculum, faculty and staff, student enrollment, program facilities, changes made since the last APR, impact of the changes, and Program Learning Outcome Assessment and Action Plans.

The Self-Study should be completed with input from the Program Review Committee and should be shared with appropriate departments before completion.

In the second step of the process, the self-study is distributed to members of an external review team. The External Review Team consists of two to three faculty from similar SUNY 2-Year and 4-year programs or from private or non-SUNY transfer programs. They are invited to campus for a full day of meetings including class visits and facility tours.

In step three, the external reviewers submit their report or reports. Then, the Program Chair meets with the Program Review Committee to draft an Action Plan. Finally, the Program Chair meets with OAA and OAA sends all necessary documents to SUNY.

OPERATING ASSUMPTIONS

The Academic Program Review is an integral part of the Annual Strategic Planning process. By addressing the Action Plans of each program, the College can ensure institutional improvement and fulfill its mission as a comprehensive college committed to delivering the best in post-secondary education at an affordable cost. The APR process provides an opportunity for programs to re-align with the College mission, vision, values and goals.

The external evaluator reports clarify both the definitions of quality and the meaning of success in the program being reviewed. There should be an open and honest sharing of data among all contributors to allow conclusions that are supported by the evidence. Dutchess Community College (DCC) demonstrates accountability to internal and external stakeholders by evaluating the effectiveness and efficiency of academic programs and using the results as a basis for planning and improvements.

3 BENEFITS OF PROGRAM EVALUATION1

The evaluation of academic programs has direct benefits for faculty and students. When conducted effectively, evaluation provides meaningful information that can be used to strengthen programs, fosters dialogue among stakeholders, and results in the continuous improvement of institutions and programs, allowing them to remain responsive and relevant in today’s environment of exponential change.

CHARACTERISTICS OF EFFECTIVE EVALUATIONS OF ACADEMIC PROGRAMS2

Effective academic program evaluations have the following characteristics.

Goal-oriented - The program has a clear statement of goals and student learning outcomes.

Useful - The program evaluation helps faculty and staff make appropriate decisions for improving the program, developing new goals and plans and/or making resource allocations.

Cost-Effective - The program evaluation yields benefits that justify the investment made in it, particularly in terms of faculty and staff time.

Reasonably accurate and truthful - The evaluation is based on reasonably credible and multiple sources of evidence rather than anecdotal information.

Planned, organized, systematic and sustained - The evaluation is part of an ongoing cycle of information gathering, analysis and action, rather than a once-and-done event.

1 SUNY University Faculty Senate Undergraduate Committee (2012). Guide for the Evaluation of Undergraduate Programs. Retrieved from http://system.suny.edu/media/suny/content-assets/documents/academic- affairs/assessment/GuideForTheEvaluationOfUndergraduatePrograms_FINAL-ONLINE.pdf

2 SUNY University Faculty Senate Undergraduate Committee (2012). Guide for the Evaluation of Undergraduate Programs. Retrieved from http://system.suny.edu/media/suny/content-assets/documents/academic- affairs/assessment/GuideForTheEvaluationOfUndergraduatePrograms_FINAL-ONLINE.pdf

4 TIMELINE FOR A SUCCESSFUL ACADEMIC PROGRAM REVIEW The Academic Program Review process is designed to be a two-semester process that can commence in either fall or spring based on the preference of the program chair. The time to dedicate to the process in those two semesters is significant and should not be underestimated. Program chairs should seriously consider using their full reassigned time for the program review.

Prior to the start of a program review, OAA requests data from Office of Institutional Research, Planning and Assessment.

In the first semester, the Program Chair will: ● Organize the Academic Program Review committee and plan the review in detail. ● Collect and analyze evidence of student learning outcomes. ● Gather any program-specific ancillary data of interest. ● Review the data collected; examine elements of the program to prepare the Self Study. ● Forward CVs of 2 – 3 prospective external evaluators to OAA for approval. ● Transfer programs will: o Review SUNY Seamless Transfer Path requirements and/or articulations agreements to ensure alignment. o Request and review data from the three top baccalaureate institutions to which our students generally transfer. ● Occupational programs will: o Meet with Program Advisory Committee. ● Write the Self Study draft, review with Department Chair and Academic Program Review committee, and send to Office of Academic Affairs.

● Coordinate evaluator visit with OAA.

In the second semester, the Program Chair will:

● Conduct the external review of the program. ● Discuss the External Evaluator’s report, survey results, and program review with Department Chair, Academic Program Review committee, and OAA for input on Action Plan. ● Compile recommendations and provide draft of the Academic Program Review final report including Action Plan to Department Chair, Academic Program Review committee, and OAA for comment. ● Submit the Academic Program Review final report to OAA.

Subsequently, the Program Chair will meet with the Department Chair and OAA regarding the implementation of the Action Plan. They will include status of Action Plan items in annual program reports. OAA will submit the final APR with all supporting documents to SUNY and post internally.

5 CONTENTS OF PROGRAM REVIEW The basic elements of the program review include an examination of the program’s objectives, curriculum, faculty and staff, and students. The outline below is based largely on the SUNY Guidelines for the Implementation of Campus-based Assessment of the Major and the SUNY Faculty Senate’s Guide for the Evaluation of Undergraduate Academic Programs (2012), and MSCHE requirements (2016).

Programs and certificates in the process of a review for reaccreditation purposes, should enfold this program review process within that of the accrediting body.

1. Table of Contents a. Pages numbered. 2. Program Information a. Program description and Program Student Learning Outcomes b. Program four-semester course sequence (as provided in college catalog) c. Curriculum Map as provided in TracDat d. Required program-specific course names and course descriptions (include courses projected to be of particular interest and value to external reviewers and SUNY) e. Description of how the curriculum is reviewed for currency and relevancy to institutional, community, and student needs, including recent deletions, additions, and revisions. f. Summary of actions taken to address Action Plan from previous APR. g. Program’s advisement procedures. h. Plan to ensure that required courses and electives are offered on a schedule to meet student needs, if applicable. i. Program-specific policies, if applicable. 3. Program Resources a. Report on the full- and part-time faculty and staff assigned to the program, including a.i. Credentials and experience. a.ii. Recent professional development activities of the program faculty that are important to the program goals. a.iii. Faculty involvement in activities that contribute to the fulfillment of the program’s mission in relation to the college and the community. a.iv. Distribution of teaching loads among faculty by rank, full-time, part-time. a.v. College grant support including Improvement of Instruction and Assessment grants. a.vi. Instructional support staff, field supervisors or laboratory support staff. b. Identify facilities, dedicated space, equipment, etc. c. Identify additional resources and facilities needed to accommodate present and anticipated program changes, developments, and enrollment growth. d. Identify resources available that provide supplemental instruction and peer tutoring. 4. Occupational Programs: Program Articulation with Workforce Needs a. Curriculum Advisory Committee a.i. List of Curriculum Advisory Committee members and affiliation. a.ii. Description of how the program identifies and recruits members for its Curriculum Advisory Committee. a.iii. Summary of the process for the program faculty to work with the Curriculum Advisory Committee in order to understand the workforce trends and to review courses for relevancy to community and workforce needs. Summary of recent input received from this review process. 5. Transfer Programs: Program Articulation a. Review of SUNY Seamless Transfer Path requirements to ensure alignment.

6 b. Review of SUNY Gen. Ed. and NYSED LAS requirements. c. Articulation c.i. Review of articulation agreements in place. c.ii. Overview of communications with transfer institutions regarding articulation. c.iii. Summary of the number of program credits that transfer schools accept in the major. c.iv. Summary of the number of program credits that transfer schools accept only as free electives. Exploration of the content of these courses for currency and applicability to the program. 6. Assessment of Student Learning Outcomes a. Summarize key points from Program Assessment Overview (PAO) Report from TracDat including how assessment results have been used to enhance student learning and inform planning and resource allocation. (Directions for generating this report are included in Appendix A.) b. Other topics that may be addressed in this section depending on the program include: b.i. Methods used to ensure comparable learning outcomes among multiple sections of a course. b.ii. Summary of how well the program’s outcomes are supported by required courses from outside the program. b.iii. Rationale for course prerequisites. b.iv. Applied learning opportunities and rationale for assigning credit where appropriate.

b.v. Program-specific accreditation requirements.

7. Recruitment and Orientation

a. Strategies to recruit students.

b. Program’s minimum requirements for admitting students and explanation of any academic placement procedures, if applicable.

c. Orientation activities, including Introductory Seminar.

8. Data Analysis

a. Analysis of the program-specific data provided by the Office of Institutional Research, Planning and Assessment.

b. Analysis of course data retrieved from Argos reports (directions provided in Appendix B).

c. Summary of implications and trends

d. Summary of results of any surveys or focus groups, including student satisfaction with the program’s offering and operations and employers (if applicable). Only reference surveys that are included in your appendix.

9. Appendices

a. Program Assessment Overview (PAO) Report from TracDat.

b. Data received from Institutional Research.

i. Enrollment data (new first-time, new transfer, total), last 5 years

ii. Persistence of first-time full-time students, first fall to spring, last 4 cohorts

iii. Degrees awarded, last 5 years

iv. Time to completion (among students who started in the program full-time), last 5 years

7 v. Two-year graduation rates, last 3 cohorts

vi. Three-year graduation rates, last 2 cohorts

vii. Transfer data from SUNY, last 3 years

viii. Transfer data from National Student Clearinghouse, most recent report available

ix. Post-graduation data from Survey of Graduated Students, most recent report available

x. Diversity (male/female, IPEDS race/ethnicity, FT/PT) in program AND college-wide c. Data retrieved from Argos reports. d. Surveys and/or focus group questions. e. CVs of external evaluators. f. Reports from external evaluators g. 6-Year Action Plan

8 SIX-YEAR ACTION PLAN The 6- Year Action Plan is compiled from information from the program review, external evaluator reports, advisory committee members (where applicable), and the results of any surveys distributed. The Program Chair also discusses this information with the Department Chair, the Academic Program Review committee, and OAA for input.

The action plan includes 1) the action item 2) person responsible 3) which ISLO, PSLO, and/or college goal is being addressed by that action item 4) the data to support the action item 5) resources needed 6) estimated cost and 7) a timeline for implementation.

The Program Chair then compiles recommendations and provides a draft of the Academic Program Review final report including Action Plan to Department Chair, Academic Program Review committee, and OAA for comment. Comments are incorporated in to the final report submitted to OAA.

Subsequently, the Program Chair meets with the Department Chair and OAA regarding the implementation of the Action Plan. They will include status of Action Plan items in annual program reports.

9 CERTIFICATE PROGRAM REVIEW The Certificate Program Review is an abridged version of the Degree Program Review. Most certificate programs at Dutchess Community College are closely tied to academic programs and should be reviewed in the same year as the degree-granting programs. The final report for the certificate program need not duplicate information that is provided in the degree-granting program’s final report, the certificate program report may simply refer to that information.

While the process for the Certificate Program Review process is similar to that of the degree-granting programs, the requirements in several stages may vary slightly. Certificate programs may not require external evaluators unless mandated by outside accreditation requirements and other program-specific assessment measures.

Students that earn certificates are expected to demonstrate their proficiency in the program’s student learning outcomes, but they are not required to meet all the SUNY general education standards. Since the institutional goals refer to expectations for those completing associate degrees, those completing certificates designed to meet specific needs are not required to meet institutional objectives or student learning outcomes.

10 EXTERNAL REVIEW GUIDELINES The External Evaluator serves as an independent consultant to review the program in comparison with other transfer, two-year or certificate programs. The Guidelines for External Evaluators are sent to all external evaluators with their appointment letter from the Office of Academic Affairs (OAA); the Guidelines describes the topics and questions to be addressed by each member of the review team.

According to SUNY Guidelines, “External review teams should consist of not less than two (2) persons who have no academic, professional or other significant relationship to full-time faculty in the program/department, no previous significant or formal affiliation with the institution, and who come from academic or professional institutions.” For Transfer Programs, at least one of the evaluators must be selected from a baccalaureate institution to which a large percentage of the program graduates transfer. For Occupational Programs, one of the external evaluators should be drawn from the professional community. SUNY Guidelines state that “[t]wo year career programs may include one Advisory Board member as part of each review team.”

For programs in the process of re-accreditation, the review from the accreditation body supplants this external review process.

The steps for organizing the review team are:  In the first semester of the review, the Program Chair compiles CVs of prospective evaluators and forwards them to OAA for approval. OAA sends the appointment letters and coordinates the visit, contracts, and stipends.  OAA sends a copy of the self-study, background materials regarding the program and curriculum, and the Guidelines for External Evaluators to review team members.  The Program Chair, in conjunction with OAA, arranges the dates and sets the agenda, for example: o .5 hr Meet with OAA o 1 hr Tour of program and campus facilities o 1 hr Meet with students o 1 hr Lunch with faculty o 1 hr Meet with Department Head and Program Chair o 1 hr Exit interview with Office Academic Affairs o In addition, there may be 1 – 2 classroom visits, a meeting with Curriculum Advisory Committee, and adjunct faculty.  The external review team conducts the site visit.  Evaluators submit Final Report to the Office of Academic Affairs, who forwards copies to the Department Chair and Program Chair.

EXTERNAL EVALUATOR GUIDELINES

The duties of the External Evaluator include: 1. Preparation for Site Visit. a. Review the program review and the Guidelines for External Evaluators, sent by the Office of Academic Affairs. b. Review of Self-Study Report sent by the Office of Academic Affairs, including relevant pages from the most recent DCC College catalog and any other information stipulating by the agreed- upon honoraria; and the schedule of events.

11 2. Site Visit. 3. Participate in on-site campus visit during which Evaluator meets faculty, students, and administration. 4. Visit/observe class sessions. 5. Tour campus facilities. 6. Meet and brief OAA on preliminary recommendations. 7. Each evaluator must submit an independent report to OAA within a month of the Site Visit.

The External Evaluator Final Report should include the following sections and topics: 1. Program Information a. Are the program degrees, descriptions, goals, and outcomes, clearly articulated? b. Is the curriculum up-to-date? c. Are program-specific policies clearly stated, well understood, and available in writing? 2. Program Resources a. Are faculty credentials, activities, scholarship and creative contributions appropriate to the program? b. Are teaching loads appropriate/adequately distributed among faculty by rank, full-time/part- time? c. How does the college support faculty development activities? d. Is dedicated space and equipment adequate for program needs? 3. Transfer and/or Employment a. Are articulation and transfer agreements current? b. Are the number of advisory committee members and their affiliation appropriate? (if applicable) c. Is information from advisory committee meetings incorporated into the program review process? d. Is there a clear method to assess student and employer satisfaction? Are those results used for program development? 4. Assessment of Student Learning Outcomes a. Are institutional and program student learning outcomes clearly mapped? b. Is there evidence of how assessment results are used to enhance student learning and inform planning and resource allocation? c. Are extended course outlines periodically reviewed for currency and relevancy? d. Since the last program review, what initiatives have been implemented to improve student success, course completion, and/or career preparation? e. Where applicable, are applied learning opportunities identified? f. Are required courses and electives offered on a schedule to meet student needs? 5. Recruitment and Orientation a. Comment on recruitment and orientation activities described in the self-study and offer recommendations, if appropriate. 6. Data Analysis a. Comment on data analysis described in the self-study. 7. Conclusions and Recommendations a. What are the major strengths of the program? b. What are the major weaknesses of the program? c. What are your overall recommendations concerning faculty, resources, curriculum, and action plan?

12 SURVEYS AND FOCUS GROUPS A program review may include information derived from surveys and focus groups. These tools can be used to collect quantitative and qualitative information from alumni, current students, faculty, and community professionals on topics such as job placement, student satisfaction, and skills desired by potential employers.

Currently, IR administers a survey every fall semester to DCC graduates. Program Chairs can leverage off this work by submitting three custom questions for inclusion in the survey. Custom questions must be sent to IR by October 1.

It is important that the content of surveys and focus groups complement the other materials incorporated in the program review. To that end, Program Chairs planning to conduct their own surveys or focus groups are encouraged to discuss the wording of questions with both OAA and IR. Note that for surveys/focus groups targeting current or former students, appropriate contact lists (e.g., email addresses, phone numbers, and mailing addresses) can be obtained from IR; please send requests at least two weeks in advance.

13 APPENDIX A – TRACDAT DIRECTIONS FOR GENERATING PAO Program Assessment Overview (PAO) reports are generated directly from TracDat. To generate the report, navigate to your Program Unit using the pull-down menu in the top bar. Click on “Reports” in the Unit Navigation Menu and click on “Standard Reports”. Refer to Figure 1 for a screen shot of what you should see.

Figure 1 Generating Program Assessment Overview Reports

All assessment information that is recorded in a course related to a PSLO can be pulled in to the report called “Relationships and Assessment: Course Links by Program Outcomes” under Course Reports (Standard Reports). Figure 2 below shows what the page will look like once you click on the Relationships report link.

14 Figure 2 Creating PAO report from course assessments

Note that under format you can select a Word document allowing you to copy and paste the report in to other documents such as the appendix of the APR.

Also note that if you click on the “All(#)” link next to “Courses Selected”, you can readily select a smaller list of courses rather than all the courses assigned to the program. When selecting courses, you can hold down the Control key to select multiple courses that are not adjacent to each other on the list at the same time before clicking on the arrow to place them in the filtered list.

Also, when you click in the other fields under “Filter”, you will be given options in pull down menus. This is a little misleading as they look like text boxes but they are actually really easy to use to filter out information not desired for the report.

Note that if you choose not to show details by unclicking the “Show Details” check box at the bottom, you will essentially get a curriculum map. It will show each PSLO and then list the courses that are linked to that PSLO.

When done making selections on the page shown in Figure 2, click on “Open Report”. The PDF report will open in another browser tab. The Word report will download.

Depending on your program, some assessment information may not be recorded directly in courses. This is the case whenever an assessment cuts across multiple courses as is the case in an ISLO assessment. Whenever possible, assessment information is included in the course where the assessment took place but for those instances that this is not practical, feasible, or even logical, another report needs to be generated to gather that program-level assessment information.

To generate the assessment report that presents assessment information entered only at the program, rather than course, level, click on “Assessment: Assessment Unit Four Column” under the heading “Assessment Unit Reports” as indicated in Figure 1 above. This will lead you to a page similar to the one shown in Figure 2 above except the filter choices are a little different and do not include filtering courses since this report is not presenting assessment at the course level. Again, you can choose to generate the report in PDF or DOC formats. Once you have selected the filtering desired, click on “Open Report”.

15 16 APPENDIX B – DIRECTIONS FOR ACCESSING ARGOS REPORTS

1. Argos should be accessed using DCC’s Virtual Machine, using the Internet Explorer browser. 2. URL: http://is-report01.sunydutchess.edu/argos/ 3. There may be a pop-up about Java being blocked – select “run”, “run this time” or “allow” (depending on the specific error message). 4. Enter your DCC username and password. 5. In the left-hand explorer window, click on the Banner folder, then ST0325.

Instructions to Run 1. Select a term. 2. Select a course. 3. Pick and run one of the reports. (Report descriptions below.)

Final Grade Distribution by Student Type - Banded  This report pulls a simple final grade distribution report by course, and creates a printable copy of it.  The pass rate standards are a C or better.  It then sums all of the grades into a seat counter, sums all of the passing grades, and calculates a passing percentage.  Grade tallies are separated by student type only. Mid-Term Grade Distribution by Student Type - Banded  This report pulls a simple mid-term grade distribution report by course, and creates a printable copy of it.  The pass rate standards are a C or better.  It then sums all of the grades into a seat counter, sums all of the passing grades, and then creates a passing percentage.  Grade tallies are separated by student type only.  This also has a column for "No Grade" meaning either that a midterm grade hasn't been entered for the student, or the grade letter wasn't A through F or a W.

17 Grade Distribution by Student Type - CSV  This report is the same data as the banded reports however, this is in a downloadable CSV that puts both Mid-term and Final grade counts together.  The pass rate standards are a C or better.  This is for users who want to make graphs and charts off of the data.