Preliminary Report on Graduate Education at UWW Fall 2013-Spring 2014

Preliminary Report on Graduate Education at UWW – Fall 2013-Spring 2014

Project Goals: The main purpose of this project is to identify and gather data needed to respond to feedback from the last Higher Learning Commission (HLC) review in 2006 in preparation for the fall 2015 HLC review. Visiting accreditation teams from the Higher Learning Commission in 1995-96, and again in 2006, have either stated directly or strongly encouraged the campus to further explore what differentiates the “undergraduate experience” from the “graduate experience.” The feedback provided by the visiting team in 2006 concluded that “The Graduate Council needs to have a discussion about what really constitutes the difference between a graduate level program and a corresponding undergraduate program and how it should ensure that such a difference exists and is maintained.” In spring 2004, the Graduate Council concluded that graduate education at UWW differs from undergraduate education in terms of content, intensity, and self-direction; however, empirical data supporting this conclusion has been limited. This project was designed to gather useful empirical data that could be used to further discussion of this issue.

A second goal of this project is to provide empirical data that may be useful to the Graduate Council as it re-examines the essential learning outcomes of graduate education at UWW. In 1997, the Graduate Council approved a set of twelve global learning outcomes for UW-W programs. These learning outcomes provided the basis for the exit survey administered to degree completers from all UW-W graduate programs since 2000. The university’s adoption of LEAP essential learning outcomes in 2010, and the involvement of graduate faculty and programs in various LEAP workshops, illuminate the need to revisit the graduate education essential learning outcomes and determine if they still accurately portray the broader educational purposes of master’s level education at UW-Whitewater.

To meet these goals, the following data have been collected:

Content Analyses of Course Syllabi

Three content analyses of a random sample of available course syllabi from UWW graduate programs were conducted. (Note: Business Ed and School Business Admin were subsumed within the MBA program and PD was subsumed within Special Ed; however, separate analyses could be performed if needed.) One analysis (see Table 1) involved examining the extent to which the stated learning goals of graduate courses were consistent with the Specialized Knowledge outcomes from the Degree Qualification Profile 1.0. The DQP was designed ostensibly to distinguish learning outcomes at the associates’, baccalaureate, and master’s level. The extent to which syllabi content is consistent with the identified master’s level learning goals can be used as one indirect method of distinguishing graduate education from undergraduate education.

A second content analysis examined the extent to which course syllabi included the five components of the graduate school mission statement as central foci (subjectively discerned) of the course (see Table 2). The five components include the following: 2

1. Comprehend and discuss advanced theoretical questions and current issues; 2. Collect, analyze, and interpret data applicable to complex questions and problems; 3. Conceptualize, evaluate, and implement solutions to complex problems; 4. Use appropriate technologies as needed; 5. Synthesize and articulate multiple concepts in a clear, concise, and persuasive manner.

A third content analysis of course syllabi focused on the materials required of students, including texts, authored or edited books, manuals, primary journals, websites, materials on e- reserves and D2L, and magazines. From this data, syllabi were categorized as requiring textbook and additional materials, textbook only, and no textbook (see Table 3).

Preliminary Summation of Findings from Content Analyses of Syllabi

Although specific programs vary in the degree to which their course syllabi are consistent with DQP standards, the data show a pattern in which the majority of our graduate programs’ syllabi are consistent with DQP 1.0 standards for master’s level training. There is variability in the extent to which graduate course syllabi reflect the five components of the Graduate School Mission statement. The vast majority of graduate courses require at least one textbook and the majority of 600- and 700-level courses require both textbooks and other educational materials.

Face-to-face Interviews with Program Coordinators

Information and perspectives from face-to-face interviews with the eleven current graduate program coordinators were obtained. Brief interviews of between 20 and 40 minutes were conducted in which coordinators were asked two broad questions and multiple follow-up questions. All coordinators were asked “how does your graduate program differ from undergraduate education here at UWW (or undergraduate education with which you are familiar) and “how would you respond to criticisms that your graduate program is not as rigorous as traditional master’s (MS) programs?”

Preliminary Summation of Interviews

The majority of coordinators stated emphatically that their programs are at least as rigorous as traditional master’s programs, many noting that their programs require students to demonstrate a high level of skill at independently finding, analyzing, and integrating complex data relevant to real-world professional applications and making professional decisions. All coordinators described greater expectations for student independence and direction, responsibility, and complexity of thinking in graduate programs as compared to undergraduate education. Graduate programs were also described as more intense, requiring more reading, writing, and need to adapt to a faster pace. All coordinators spoke of rigorous “capstone” experiences (e.g., professional reports and presentations, intensive written exams, observation hours, practicum and internship requirements, state and national licensure and certification requirements) and/or embedded verticality in their curricula requiring students to demonstrate mastery of fundamentals of the discipline prior to taking 600- and 700-level courses needed to 3 complete the program. Several coordinators indicated that their programs had unique admissions requirements in addition to the requirements of the Graduate School. Examples of such admission requirements included certain types of prior employment experiences and Bachelor’s level course work of a certain quality in a specific area. Several coordinators acknowledged that external constituents might perceive certain emphases within their programs as less rigorous, but that faculty and students within the program recognized the unique requirements of the emphases that made the overall level of rigor and intensity of the different emphases mostly comparable. Several coordinators reported that their programs had successfully reduced the number of dual-listed courses that tend to reduce student satisfaction and increase reviewers’ concerns about the rigor of the program.

Exit survey data

Quantitative data and written comments from exit satisfaction surveys of graduating students from fall 2007-spring 2013 were obtained. One part of the exit survey contains 12 items assessing the extent to which graduating students perceive the importance of the broader educational objectives of UWW graduate education and the extent to which they believed their master’s degree contributed to their attainment of these objectives. Students’ written comments were examined and coded into categories including the following: Opportunities for hands-on experience and job preparation, use of modern technology, high expectations of faculty for students, current research based information received, lack of challenge and differentiation from undergraduate work, too much repetition between courses, desire for more opportunities to work with technology, and desire for more preparation and real-world experience (see Table 4).

Preliminary Summation

In general, the results suggest that graduating students perceive the value of the educational objectives of UWW graduate education and believe that their programs have improved their intellectual skills in a number of related domains. Mean scores were generally above 4.0 using a 1-5 Likert scale in which higher numbers indicated greater valuing and perceived improvement. Several exceptions occurred in regard to both program and intellectual skill. For example, means were lower for “improved my ability to use technology effectively” and “a better understanding of the current and potential use of technology in my discipline” across programs. Accounting students reported relatively lower scores on “improved my ability to express my thoughts and/or positions in writing.” Both accounting and business administration students reported relatively lower scores on “I’ve improved my ability to orally express my thoughts and/or positions.”

Distinguishing UG and Graduate Education Survey

Quantitative data and written comments from an online survey administered to graduate faculty (n = 155) and graduate students (n = 303) in spring 2014 were obtained. This survey was specifically designed to elicit comparisons between undergraduate and graduate education at 4

UWW by asking participants to identify the extent to which descriptor items were more characteristic of undergraduate education, graduate education, equally descriptive of both, or not applicable to their education. Survey items were obtained from a variety of sources including the DQP, UWW graduate school mission statement and comprehensive objectives, and published articles on master’s level education and programming. An open-ended request for additional information for perceived differences between undergraduate and graduate education was also included.

Preliminary Summation

Initial examination of the data suggests quite a bit of similarity in faculty and student perceptions of graduate and undergraduate education and some clear patterns in regard to how UWW graduate and undergraduate education is both similar and different. A summary table of the results should be available soon.

Data from UWW Graduate School to be Collected and Summarized

Existing data on graduate students (e.g., undergraduate GPA, UG institution attended, gender, age, race/ethnicity, and data on critical thinking and general knowledge from the four graduate programs that participated in the university DQP project) and specific programs (e.g., numbers of students matriculated by year, type of program – cohort, online, retention and graduation from programs, numbers of dual-listed courses, faculty credentials) will be gathered and used to augment our understanding of our graduate programs. The most recent self-study from each graduate program has already been reviewed.

Initial Reflections:

1. Results of syllabi assessments don’t necessarily mean that certain programs are not requiring adequate or rigorous graduate-level work. Rather, the results may suggest the need to be more explicit in how program goals and objectives are communicated in syllabi. The more non-traditional a graduate program is (i.e., the more distinct it is from a traditional MS program), the greater the need to explicitly articulate its goals and requirements in written program documents.

2. Dual-listed courses should specify the unique expectations of graduate students as required by the Graduate School (see document titled “Graduate Level Requirements in Dual-Listed Courses)

3. Graduate faculty may want to reflect more on the kinds of learning materials required of their students. Several program coordinators commented on the amount of reading and independent learning their students are required to do. One question to ask is whether students are engaging with the kinds of materials that will promote deep, comprehensive, and critical thinking in their fields and how can this be documented. 5

4. Several coordinators indicated that having additional admissions requirements might reduce the number of prospective students who seem to simply “want a Master’s degree” without understanding the nature of a specific program. One coordinator commented that some prospective students seem to be “shopping” for an “easy” program they believe they can complete (regardless of the content). One coordinator mused that the fact that most of our programs have few identified screening tools may increase the misperception of our programs as “easy.” A more rigorous admissions process might reduce perceptions that UWW graduate programs aren’t rigorous. The admissions process could involve additional unique requirements rather than a certain test score or higher undergraduate GPA. For example, in addition to writing a brief personal statement of their interest and background, students could be required to submit a written description of their understanding of the program they wish to pursue and the specific profession they hope to enter upon graduation.

5. There is a need to establish clarity regarding the rigor of “non-traditional” graduate requirements (e.g., portfolios, professional presentations). Questions such as “what makes a capstone portfolio as rigorous as a thesis?” need to be addressed.

6. Graduate programs with corresponding undergraduate programs need to make the differences between their UG and G programs clear. A lack of articulated distinction here is simply too easy a target.

7. In many cases, the face-to-face interviews with coordinators were more descriptive and positive in regard to the quality and structure of the programs than the written program self-studies. If these self-studies are a part of the HLC review, attention needs to be paid to their completeness, accuracy, and optimal presentation of unique program qualities.

8. The face-to-face interviews highlighted unique elements of many programs that make them rigorous in ways distinct from both undergraduate experiences and traditional MS programs (e.g., “intense and sustained self-reflection and faculty evaluation,” “professional presentation to a panel of Wisconsin professionals in the field”). These uniquenesses need to be articulated and shared.

9. Graduate faculty could work together to create a clearer description of how research is an integrated element of their programs. Interviews with coordinators indicated that research is essential to graduate study in their programs; however, it often differs in form from the traditional MS thesis option. These research experiences are also likely to differ from many research experiences at the undergraduate level. In general, the research element is described like this: research is applied, theory is incorporated, looking to solve practical problems using critical thinking and existing data. The applied nature of our programs is perceived as their greatest strength by both coordinators and graduates. Graduates’ comments from the exit surveys indicate that they want more opportunities to apply their new skills and knowledge. 6

10. It might be helpful for graduate faculty to discuss the skill sets their graduates have compared to undergraduates. Explain what it means to be “applied.”

11. Many coordinators emphasized the importance placed on the ability to integrate information from a variety of sources to arrive at effective interventions/solutions that can be implemented. In addition, many coordinators described the importance of understanding and effectively communicating with a wide variety of constituents (families, businesses, schools, government, etc.). The ability to deal effectively with real- world complexity was an emerging theme that likely distinguishes our graduate programs from both UG and traditional MS programs that place more emphasis on understanding complexity within a specific content area.

12. Information from program advisory boards, external constituents, and alumni working as professionals in related fields could be very useful to share. 7

Table 1

Summary of Major DQP Specialized Knowledge Foci in Eight Graduate Programs as Identified in Examined Syllabi

DQP Foci Busin Account Communica Comm Counse Safe School Special ess ing tions Sci & lor ty Psychol Educati Admi Disord Educati ogy on n ers on 1 - Describes 81.8% 75% 75% 77.8% 100% 60% 60% 87.5% the scope & principal theories of field of study, citing at least some of its core theories & practices, & offers a similar explication of at least 1 related field. 2 – Illustrates 100% 100% 100% 77.8% 100% 80% 73.3% 100% contemporar y terminology used in the field. 3 – Generates substantially error-free products, data, etc. as appropriate to field. 4 – Defines & 80% 80% 50% explains the boundaries & major sub- fields & practices of 8 the field. 5 – Defines & properly uses the principal specialized terms used in the field, both historical & contemporan eous.

DQP Foci Busin Account Communica Comm Counse Safe School Special ess ing tions Sci & lor Ed ty Psychol Educati Admi Disord ogy on n ers 6 – 75% 75% 66.7% 100% 60% 53.3% 75% Demonstrate s fluency in the use of tools, technologies, & methods common to the field.

7 – Evaluates, clarifies, and frames a complex question or challenge, using perspectives & scholarship drawn from the student’s major field & at least 1 other field. 8 – 66.7% 80% 50% Constructs a project 9 related to a familiar but complex problem in the field by independentl y assembling, arranging, & reformulating ideas, concepts, designs, or techniques. 9 – 54.5% 80% 53.3% 62% Constructs a summative project, paper, or practice- based presentation that draws on current research, scholarship, or techniques in the field.

DQP Foci

10 – 63.6% 50% 50% 66.7% 100% 60% 60% 87.5% Elucidates the major theories, methods, & schools of practice in the field and articulates their sources 11 - 72.7% 50% 88.9% 80% 80% 93.3% 100% Illustrates applications 10 of the major theories, methods, & schools of practice of the field 12 – 75% 75% 74% 75% Illustrates relationships between the major theories, methods, & schools of practice in the field and their relationships to allied fields. 13 – Assesses 50% the contributions of major figures (and organizations, if applicable) in the field. 14 – 81.8% 75% 75% 66.7% 100% 73.3% 100% Describes the major methodologie s & practices in the field

15 – 54.5% 66.7% 80% 73.3% 62.5% Implements at least 2 of 11 the major methodologie s & practices (above) through projects, papers, or presentations .

16 – 50% 50% 55.6% 80% Articulates a full range of challenges involved in practicing the field. 17 – Elucidates the leading edges of the field. 18 – Delineates the current limits of theory, knowledge, or practice in the field by independentl y initiating, assembling, arranging, and reformulating concepts, designs, or techniques in carrying out a project directed at a challenge in the field that lies outside 12 conventional boundaries.

Table 2

Syllabi Analysis: Percent of Syllabi Consistent with the Five Components of School of Graduate Studies Mission Statement

Graduate Comprehend Collect, Conceptualize, Use Synthesize Program and discuss analyze, evaluate, and appropriate and advanced and implement technologies articulate theoretical interpret solutions to as needed multiple questions and data complex concepts in current issues applicable problems a clear, to complex concise, questions and and persuasive problems manner Communication 100% 0% 75% 75% 0% Accounting 100% 0% 75% 75% 0% Business 75% 27% 55% 55% 55% Administration Safety 20% 0% 0% 0% 20% Communication 67% 44% 44% 90% 89% Science & Disorders Counselor 100% 20% 80% 40% 60% Education Special 75% 38% 38% 38% 38% Education School 87% 60% 73% 60% 93% Psychology Note. “Consistency” was defined as a score of 3 or higher on the following 5-point Likert scale:

1 2 3 4 5 Not at all a goal Only slight a goal Quite a bit A great deal a CENTRAL goal of this course of this course 13

Table 3

Textbook Usage in Graduate Programs Department Courses Using Textbooks Courses Using Only Courses Not Using and Additional Materials Textbooks (%) Textbooks (%) (%) Communication 500-Level: 42.9% 500-Level: 42.9% 500-Level: 14.3% 600-Level: 33.3% 600-Level: 55.5% 600-Level: 11.1% 700-Level: 83.3% 700-Level: 16.7% 700-Level: 0% Accounting 500-Level: --- 500-Level: --- 500-Level: --- 600-Level: 100% 600-Level: 0% 600-Level: 0% 700-Level: 38.5% 700-Level: 30.8% 700-Level: 30.8% Business Administration 500-Level: --- 500-Level: --- 500-Level: --- 600-Level: --- 600-Level: --- 600-Level: --- 700-Level: 58.5% 700-Level: 26.8% 700-Level:14.6% Safety 500-Level: 0% 500-Level: 75% 500-Level: 25% 600-Level: 83.3% 600-Level: 16.7% 600-Level: 0% 700-Level: 83.3% 700-Level: 16.7% 700-Level: 0% Communication Sciences 500-Level: 75% 500-Level: 0% 500-Level: 25% and Disorders 600-Level: 66.7% 600-Level: 0% 600-Level: 33.3% 700-Level: 55.5% 700-Level: 44.4% 700-Level: 0% Counselor Education 500-Level: --- 500-Level: --- 500-Level: --- 600-Level: --- 600-Level: --- 600-Level: --- 700-Level: 70.4% 700-Level: 29.6% 700-Level: 0% Special Education 500-Level: 71.4% 500-Level: 0% 500-Level: 28.6% 600-Level: 80% 600-Level: 10% 600-Level: 10% 700-Level: 100% 700-Level: 0% 700-Level: 0% School Psychology 500-Level: --- 500-Level: --- 500-Level: --- 600-Level: 100% 600-Level: 0% 600-Level: 0% 700-Level: 92.3% 700-Level: 0% 700-Level: 7.7% 14

Table 4

Top Three Categories of Student Comments from Each Graduate Program

Student Comment Categories Program Opportunities Used High faculty Based in Lack of Too much More More for hands on Modern expectation current challenge or repetition opp for preparation or real life Technolog s research differentiation in working for/real experience y from UG courses w/ tech world opportunities Accounting 1 2 3 Bus Ed 1 Bus Admin 3 2 3 1 Comm 2 2 1 Comm Sci- 3 3 2 Dis 1 Counseling 1 3 2 Prof Dev 1 3 2 Safety 2 3 1 3 3 School Bus 1 3 2 3 Spec Ed 2 3 1 School 1 2 3 Psych