Douglass & Thomson Presentation

Adrift or Engaged? Exploring a Multi-Engagement Model of the Student Experience Using SERU Data

BEAR Seminar – Graduate School of Education – UC Berkeley – March 21 2017

Gregg Thomson (CSHE/Saint Mary’s College) - John Douglass (CSHE)

Center for Studies in Higher Education – Goldman School of Public Policy – UC Berkeley

SERU RUA Project Colleagues: Igor Chirikov (CSHE/HSE-Moscow) Ron Huesman (UMN), Tongshan Chang (UCOP), and Steve Brint (UC Riverside) SERU - Turning to the Student Side of the Research University’s Mission

Research/Know ledge Production

Teaching and Public/ Learning Community Service

Seeking a Holistic Understanding of the Student Experience SERU - Turning to the Student Side of the Research University’s Mission

UG Survey – 2002 Graduate Student Survey – 2014

Research/Know Census – Online - Customized ledge Production

Teaching and Public/ Learning Community Service

Seeking a Holistic Understanding of the Student Experience SERU-AAU Consortium Members University of California – Berkeley+ all the UC UG Campuses Rutgers University University of Florida University of Michigan University of Minnesota University of Oregon University of Pittsburgh University of Texas University of Southern California University of North Carolina University of Virginia Texas A&M University Purdue University University of Iowa University of Washington Michigan State University University of Toronto University of Connecticut** University of Delaware SERU-AAU Consortium Members SERU-I Consortium Members University of California – Berkeley+ all the UC UG Campuses Universidade Estadual de Campinas (Unicamp) Rutgers University Hunan University University of Florida Nanjing University University of Michigan Xi’an Jiaotong University University of Minnesota Amsterdam University College (AUC) University of Oregon University of Cape Town* University of Pittsburgh University of Bristol* University of Texas University of Oxford* University of Southern California Higher School of Economics – Moscow University of North Carolina Osaka University University of Virginia Tongji University Texas A&M University Hiroshima University Purdue University Lancaster University University of Iowa Utrecht University University of Washington Lund University Michigan State University Universität Bremen University of Toronto Georg-August-Universität Göttingen University of Connecticut** Humboldt-Universität zu Berlin University of Delaware University of Freiburg National University of Singapore SERU-AAU Consortium Members SERU-I Consortium Members University of California – Berkeley+ all the UC UG Campuses Universidade Estadual de Campinas (Unicamp) Rutgers University Hunan University University of Florida Nanjing University University of Michigan Xi’an Jiaotong University University of Minnesota Amsterdam University College (AUC) University of Oregon University of Cape Town* University of Pittsburgh University of Bristol* University of Texas University of Oxford* University of Southern California Higher School of Economics – Moscow University of North Carolina Osaka University University of Virginia Tongji University Texas A&M University Hiroshima University Purdue University Lancaster University University of Iowa Utrecht University University of Washington Lund University Michigan State University Universität Bremen University of Toronto Georg-August-Universität Göttingen University of Connecticut** Humboldt-Universität zu Berlin University of Delaware University of Freiburg National University of Singapore SERU Undergraduate Survey Instrument

Core Module (36 questions) Academic and research engagement, time use, learning outcomes, personal development, plans and aspirations, satisfaction and background characteristics

Module 1 Module 2 Module 3 Module 4 Module 5 (11 questions) (23 questions) (11 questions) (12 questions) (6 questions)

Academic Community Student Life Uses of International Experience and Civic and Technology Students and Engagement Development Globalization

Campus Wildcard: a set of questions specific to the campus Uses of SERU Data • Catalyst for Campus Discussion on Strengths and Weaknesses • Program Review • Campus Internal Studies/Departmental Use • Common Source of Info – what does SERU tell us?

• Campus accreditation SERU • National Accountability External Regimes – in US VSA DATA • Media and public relations

• Research and Policy Analysis: • RUA Project • International Students Policy • Research Engagement • Global Skills . . . • In Turn: • Leads to New HEI’s Policy • Informs the SERU Survey SERU - Research University Advantage Project (RUA) UG Education - The Narrow View

• Academically Adrift (2011) induced critique of American higher education:

o the curriculum is not rigorous. o students do not spend enough time in class and studying. o students do not learn enough and most are drifting through their academic careers – at all types of HE institutions. “Its official,” stated a one news article, “college student learn next to nothing.” UG Education - The Narrow View

• Academically Adrift (2011) induced critique of American higher education:

o the curriculum is not rigorous. o students do not spend enough time in class and studying. o students do not learn enough and most are drifting through their academic careers – at all types of HE institutions. “It’s official,” stated one news article, “college student learn next to nothing.” Is this True? UG Education - The Narrow View

• Academically Adrift (2011) induced critique of American higher education:

o the curriculum is not rigorous. o students do not spend enough time in class and studying. o students do not learn enough and most are drifting through their academic careers – at all types of HE institutions. “It’s official,” stated a one news article, “college student learn next to nothing.” Is this True?

• Perception (reality?) that Research Universities are not concerned enough about the quality of Undergraduate Education – a la Boyer Report UG Education - The Narrow View

• Academically Adrift (2011) induced critique of American higher education:

o the curriculum is not rigorous. o students do not spend enough time in class and studying. o students do not learn enough and are drifting through their academic careers – at all types of HE institutions. “It’s official,” stated a one news article, “college student learn next to nothing.” Is this True?

• Perception (reality?) that Research Universities are not concerned enough about the quality of Undergraduate Education – a la Boyer Report

• The primary [only?] function of higher education is the production of a skilled labor force and promise of future personal income. UG Education - Our Broader SERU View

UG Student • Undergraduates at Research Intensive Experience Universities have opportunities for multiple forms of engagement and learning that are perhaps unique in their range and depth.

• The purpose of higher education should be to produce graduates equipped for both careers and democratic citizenship. What Are Employers Looking For? Template for Recent Graduates’ Resume from Yale University

Curricular Engagement

Study Abroad

Research Engagement

Extra-Curricular Engagement

Civic Engagement Students and the Multiversity Metaphor

The modern comprehensive university is like a city. It has many subcultures, experiences, and responsibilities that reflect its growing role in UG Student society. Experience “Some get lost in the city,” Clark Kerr once explained, “while others, both faculty and students, find communities within the larger institution.” The “city,” continued Kerr, “is more like the totality of civilization.” Students and the Multiversity Metaphor

• Are our AAU universities helping students navigate and take advantage of the opportunities for multiple forms of Engagement and learning?

• Are they innovating in an era of reduce public investment and often rising student to faculty ratios? Our SERU RUA Conceptual Lens The Four RUA Spheres of Multi-Engagement

Curricular Research Engagement Engagement

UG Student Experience

Public/Community Extracurricular Service Activities

Seeking a Holistic Understanding of the Student Experience Popular Sense of the Average?

Curricular Engagement

Research Engagement

UG Student Experience

Extracurricular Activities Public/Com munity Service Closer to AAU Reality Average?

Curricular Engagement Research Engagement

UG Student Experience

Public/Community Extracurricular Service Activities Four RUA Spheres – Most Students are Multi-Engaged

Curricular Engagement Research Engagement UG Student Experience Extracurricular Public/Community Activities Service Most Students are Multi-Engaged – Disciplines Matter

The Average for a Third Year Engineering Major?

Curricular Engagement Research Engagement

UG Student Experience

Public/Comm unity Service Extracurricular Activities RUA Research Questions

• Do students appear to be engaged or adrift when various modes of engagement (curricular, research, extracurricular, and civic) are examined? Curricular Research Engagement Engagement

UG Student Experience

Public/Communit Extracurricular y Service Activities RUA Research Questions

• Do students appear to be engaged or adrift when various modes of engagement (curricular, research, extracurricular, and civic) are examined? Curricular Research Engagement Engagement • To what extent are various modes of engagement mutually exclusive, UG Student Experience independent, or correlated? • How do the patterns of engagement vary by Public/Communit Extracurricular y Service Activities student background, GPA, level of study, field of study? RUA Research Questions

SERU/UCUES 2014 Data – Total of 130,125 Responses

UC-Berkeley 9785 University of Minnesota 8332 UC-Davis 10456 Rutgers University 7186 UC-Irvine 10193 University of Pittsburgh 5733 UC-Los Angeles 6807 University of Southern California 1960 UC-Merced 1947 Texas A&M University 11806 UC-San Diego 8064 University of Virginia 4763 UC-Santa Barbara 6822 Indiana University 3205 UC-Santa Cruz 6089 Purdue University 2944 UC-Riverside 6160 University of Iowa 4814 University of Michigan 4930 University of Washington 8129 RUA Data Sources

• SERU/UCUES 2014 Data – Total of 130,125 Responses

• RUA Data source: SERU-2014 Data (UCUES data is not included) • 11 AAU SERU Campuses with >63,000 Student Responses Construction of Indices

Curricular, research, civic, and extracurricular engagement are latent variables.

How to measure them? Construction of Indices

Curricular, research, civic, and extracurricular engagement are latent variables.

How to measure them?

Equal weights (Stuart and Thomson, 2013; NSSE)

Pros: Easy to interpret, Easy to Calculate

Cons: Not all forms of engagement are equal in terms of efforts, time, etc. Construction of Indices

Curricular, research, civic, and extracurricular engagement are latent variables.

How to measure them?

Equal weights (Stuart Weights are and Thomson, 2013; determined based on NSSE) experts opinion

Pros: Easy to Pros: Inequality interpret, Easy to between forms of Calculate engagement is taken into account

Cons: Not all forms of Cons: How to engagement are equal organize experts in terms of efforts, nomination? time, etc. How to resolve disagreement? Construction of Indices

Curricular, research, civic, and extracurricular engagement are latent variables.

How to measure them?

Equal weights (Stuart Weights are Weights are and Thomson, 2013; determined based on determined based on NSSE) experts opinion Factor Analysis (PCA, CatPCA)

Pros: Easy to Pros: Inequality Pros: The most interpret, Easy to between forms of accurate method Calculate engagement is taken into account

Cons: Not all forms of Cons: How to Cons: Harder to engagement are equal organize experts interpret, Multiple in terms of efforts, nomination? solutions, Data- time, etc. How to resolve sensitive disagreement? Construction of Indices

During this year, how often How frequently have you - a* have you- Contributed to a + b* Communicated with a faculty class discussion member by e-mail or in person: - Never - Never

- Rarely Equal distance? - Rarely

- Occasionally - Occasionally

- Somewhat Often - Somewhat Often

- Often - Often

- Very often - Very often

Categorical Principal Component Analysis (CatPCA) Major Initial SERU RUA Findings Correlations Between Indices Curricular Research Civic Engagement Extracurricular Engagement Index Engagement Index Index Engagement Index Curricular Pearson Correlation 1 .212 .296 .259 Engagement Control variables: Class level Index & University & Field of study 1 .168 .271 .232 Control variables: Gender & Social class & Race 1 .206 .321 .259 Control variables: GPA 1 .192 .310 .259 Sig. (2-tailed) .000 .000 .000 Research Pearson Correlation .212 1 .181 .167 Engagement Control variables: Class level Index & University & Field of study .168 1 .123 .119 Control variables: Gender & Social class & Race .206 1 .202 .220 Control variables: GPA .192 1 .183 .214 Sig. (2-tailed) .000 .000 .000 Civic Pearson Correlation .296 .181 1 .333 Engagement Control variables: Class level Index & University & Field of study .271 .123 1 .285 Control variables: Gender & Social class & Race .321 .202 1 .345 Control variables: GPA .310 .183 1 .334 Sig. (2-tailed) .000 .000 .000 Extracurricular Pearson Correlation .259 .167 .333 1 Engagement Control variables: Class level Index & University & Field of study .232 .119 .285 1 Control variables: Gender & Social class & Race .259 .220 .345 1 Control variables: GPA .259 .214 .334 1 Sig. (2-tailed) .000 .000 .000 Are Students Engaged or Adrift?

0.69% of students are completely disengaged.

They belong to the lowest quartile in all 4 areas (curricular, research, civic, extracurricular engagement) Are Students Engaged or Adrift?

8% of students have below median engagement in all 4 areas (they belong to the lowest 2 quartiles)

Which means according to this initial analysis … Are Students Engaged or Adrift?

92% of students have above average level of engagement in at least one area (curricular, research, civic or extracurricular)

But a more narrow analysis looks at only the top quartile - those most engaged in one or more of the Four Spheres of Engagement … Are Students Engaged or Adrift?

67% of students have high level of engagement (upper 25%) in at least one area (curricular, research, civic or extracurricular) Are Students Engaged or Adrift?

67%

• What are the Patterns of Engagement?

• How about Learning Outcomes? Curricular, Research and Extracurricular Engagement – Above Average

Research Curricular 14% Engagement Engagement

12% 12%

24%

14%

12%

Extracurricular 12% Engagement Multi-Engagement: Above Average Engagement in Two or More Areas

Research Curricular Engagement Engagement

73% of students at research universities are multi-engaged

Civic Extracurricular Engagement Engagement Multi-Engagement: High Engagement in Two or More Areas

Curricular Engagement Research Engagement

37% of students at research universities have high (upper 25%) engagement in two or more areas

Civic Extracurricular Engagement Engagement NEW RESEARCH

• 2012 SERU/UCUES DATA - Includes 9 UG UC Campuses • Has extensive co-curricular and civic engagement module • Expanded framework for the four types of engagement • Defines highly engaged in absolute terms rather than distribution of scores across quartiles NEW RESEARCH USING 2012 SERU

Includes UC data Has extensive co-curricular and civic engagement module Expanded framework for the four types of engagement Defines highly engaged in absolute terms rather than distribution of scores across quartiles CURRICULAR ENGAGEMENT

Curricular Engagement 1 Time spent studying: 26 hours/week or more

Curricular Engagement 2 Active engagement with instructor and in classroom: Average of “Often” or higher on five-item scale The two measures overlap very slightly. RESEARCH ENGAGEMENT

Index of research activity– two or more of the following Research seminar (often) Assist faculty research for course credit Assist faculty research for pay Assist faculty research as volunteer CIVIC ENGAGEMENT

Two factor analysis of reasons for community service: Belief in the cause, change conditions Opportunity for self-development Civic Engagement 1 and Civic Engagement 2 defined by positive factor score AND either 10 hours/week or total of more than 50 hours service Two measures overlap substantially C0-CURRICULAR ENGAGEMENT

Co-Curricular Engagement 1

Officer or leader in at least one the following: Student government, Fraternity/sorority, campus political group, advocacy group, service group Co-Curricular Engagement 2 Officer or leader in at least one the following: media, performing, recreational group or sports team OR at least 10 hours/week in student club or organization The two measures have little overlap Research and Co-Curricular Engagement

Research Co-Curricular engagement engagement

45% 45%

10% Civic and Co-Curricular Engagement

Civic Co-Curricular engagement engagement

43% 45%

12% Curricular and Research Engagement

Curricular Research engagement engagement

53% 31%

16% Curricular and Co-Curricula Engagement

Curricular Co-Curricular engagement engagement

55% 11% 34% Civic and Research Engagement

Civic Research engagement engagement

43% 44%

13% Curricular and Civic Engagement

Curricular Civic engagement engagement

55% 32%

13% HOW MANY STUDENTS ARE HIGHLY ENGAGED?

No Engagement 52 One Form of Engagement 33 Two Forms of Engagement 12 Three Forms of Engagement 3 Four Forms of Engagement 0.3 ENGAGEMENT BY YEAR IN SCHOOL

Frosh Soph Junior Senior Curricular1 11 12 11 9 Curricular2 11 13 15 17 Research 7 10 20 26 Civic1 9 11 14 16 Civic2 6 10 11 12 Co-Curricular1 6 10 12 13 Co-Curricular2 4 7 10 11 No Engagement 63 51 44 39 Any Engagement 37 49 56 61 Multiple Engagement 8 14 21 24 GPA, GENDER, SENSE OF BELONGING, AND SLEEP

GPA Female Belong Sleep <7 hrs Curricular1 3.22 60 74 60

Curricular2 3.28 56 70 52 Research 3.29 59 70 50 Civic1 3.22 71 72 58 Civic2 3.21 71 70 56 Co-Curricular1 3.17 55 76 50 Co-Curricular2 3.22 52 71 48 No Engagement 3.10 59 59 47 Any Engagement 3.22 60 70 52 Multiple 3.29 60 73 55 Engagement ENGAGEMENT BY FIELD OF STUDY (%)

Curr1 Curr2 Reas Civic1 Civic2 Co-Curr1 Co-Curr2 Arts 10 23 9 8 5 8 12 English 7 25 8 11 7 9 10 Psychology 8 12 26 15 12 9 7 Economics 9 12 10 8 6 14 8 Political Scie 7 22 12 14 8 18 8 Sociology 8 16 14 17 11 10 7 Biology 13 9 26 16 16 7 7 Business 7 16 10 11 8 13 8 Phys Science 14 13 26 6 6 6 6 Engineering 17 10 16 7 4 10 8 Mathematics 12 11 11 8 6 9 10 Top-5 Areas with Most Significant Self-Assessed Learning Gains for Each Area of Engagement

Curricular Research Civic Extra-Curricular Engagement Engagement Engagement Engagement

Ability to prepare and Ability to appreciate cultural Research skills Leadership skills make a presentation and global diversity Analytical and critical thinking Understanding your field of Research skills Library research skills skills study Analytical and critical Understanding your field Ability to understand Research skills thinking skills of study international perspectives Ability to appreciate and Understanding your field Ability to prepare and Ability to appreciate the fine understand racial and ethnic of study make a presentation arts diversity Ability to read and Ability to be clear and effective Library research skills comprehend academic Interpersonal (social) skills when writing material

Upper-division students, self-reported learning gains, 2 or more steps increase Areas Where Self-Assessed Learning Gains Significantly Exceeded Gains for Non-Engaged – Upper-Division Students

Curricular Curricular Curricular Research Engagement ! Engagement 2 Engagement 2 Engagement (continued) Ability to prepare and Ability to be clear and Research skills make a presentation effective when writing Analytical and critical Ability to prepare and make a Quantitative skills thinking skills presentation Ability to prepare and Ability to read and comprehend Research skills make a presentation academic material Ability to read and Understanding your field of comprehend Research skills Leadership skills study academic material Ability to read and Understanding your Understanding your field of comprehend academic field of study study material Areas Where Self-Assessed Learning Gains Significantly Exceeded Gains for Non-Engaged – Upper-Division Students

Civic Civic Engagement 1 Civic Engagement 1 (continued) Engagement 2

Leadership skills Leadership skills

Research skills Research skills Ability to prepare and Ability to prepare and make a Interpersonal (social) skills make a presentation presentation Ability to appreciate and Understanding the Self-awareness and understand racial and importance of personal social understanding ethnic diversity responsibility Understanding the Ability to appreciate and importance of personal understand racial and ethnic social responsibility diversity Areas Where Self-Assessed Learning Gains Significantly Exceeded Gains for Non-Engaged – Upper-Division Students

Co-Curricular Co-Curricular Engagement 1 Engagement 2

Leadership skills Leadership skills

Self-awareness and Interpersonal (social) skills understanding

Ability to prepare and make a presentation Self-awareness and understanding Impact of Changing Just One Criterion for Engaged

26 per week or more 21 per week or more No Engagement 52 38 One Form of Engagement 33 41 Two Forms of Engagement 12 17 Three Forms of 3 4 Engagement Four Forms of Engagement 0.3 1 Conclusions

• New analysis expands our four forms of engagement framework

• Results illustrate the multiple pathways to engagement at research universities

• The number of students who will be identified as “engaged” is highly sensitive to the methodology and criteria for engagement used PLANNED RUA RESEARCH AGENDA

• Explore Further Disciplinary and Subpopulations • More Fully Explore Engagement with Academic Performance and Graduation • Link SERU AAU Data with Job and Income Outcomes • Seek Selected Comparative Data from Other Non- Research Intensive Colleges and Universities • Utilize the SERU RUA Analysis for Diagnostic Purposes and to Further Improve the UG Experience

We Seek Your Ideas and Input! Thank you!