The NSSE Challenge

Toward Strengthening Agenda General Education: Beyond Curricular Change - 1. Context for focus on improving Educational Environments educational environments Designed for Learning 2. What we’ve learned from NSSE about ACA Summit, 2005 effective educational practice 3. Using data to evaluate and improve Jillian Kinzie educational environments Associate Director 4. Stimulating action around effective NSSE Institute and Indiana educational practice University Center for 5. Discussion Postsecondary Research

What we know about student Some Assertions about learning… Undergraduate Education : Assertion 1: Too few students are getting the most out Educators must be concerned with the of their college education. design of the learning environments – Assertion 2: The total learning environment must be in and outside the classroom, socio- examined to improve undergraduate education. cultural aspects and physical settings Assertion 3: We value what we measure. – in which students interact with How do we get more students to take greater advantage of educational opportunities? peers, the content, educators and To what extent are you willing to increase your others, and the implementation of investment in student success initiatives? strategies that help guide the student Are you willing to consider the effectiveness of toward the intended outcomes current practice in teaching and learning? How do you know that what you’re doing enhances student learning and success?

Promising Signs The NSSE Challenge

“Learning that lasts” requires How might we more effective educational processes and effectively use data about outcomes -- how do we create environments that facilitate student quality in undergraduate learning?? education to: 9provide evidence of Institutions must take a deep, student learning comprehensive look at themselves to 9motivate and inspire assess the quality and effectiveness of the undergraduate program. effective educational practice Assessment and improvement 9strengthen the learning activities like NSSE can help! environment?

1 ACA schools and Teagle NSSE and ACA schools Foundation Alice Lloyd College | Berea College | Bryan Goal to strengthen general education College | Campbellsville University | Davis & Elkins by supporting colleges in the collection College | Emory & Henry College | Lee University and use of student outcomes | Lees-McRae College | Lincoln Memorial information provided by ACT CAAP University | Lindsey Wilson College | Mars Hill (writing, reading, math, science) College | Maryville College | Milligan College | Union College | University of Charleston | Institutions will study gains scores to University of the South | Virginia Intermont identify areas in general education College | Warren Wilson College | West Virginia curriculum to revise and enhance Wesleyan College | Wheeling Jesuit University ACA schools will collaborate to develop Consortium – Work college, Christian colleges new approaches to structuring content and pedagogy

Integrating data points around Lessons from the Research on What general education Matters to Student Learning Comprehensive review of gen ed requires Examination of learning outcomes data and The greatest impact appears course taking patterns to pinpoint where work to stem from students’ total needs to be done in the curriculum level of campus engagement, Consideration of the total learning particularly when academic, environment to broadly assess quality of the interpersonal, and undergraduate program and identify educational practices that need attention or extracurricular involvements can help further efforts to strengthen general are mutually reinforcing… education HOW will understanding outcomes data Pascarella & Terenzini, How College Affects change things on campus??? Students, 2005, p. 647

Lessons from the Research on What Path Model for Assessing Change in Matters to Student Learning Student Learning (Pascarella)

Structural/Organization al Characteristics of Interactions with agents What students do during college counts institutions e.g., of socialization e.g., more in terms of desired outcomes than •Enrollment •Faculty and Staff who they are or even where they go to •Faculty-student ratio •Peers •Selectivity college. •% Residential Learning and The voluminous research on college cognitive development student development shows that the time Student Institutional background/Pre- Environment and energy students devote to College traits educationally purposeful activities is the •Aptitude •Achievement Quality of single best predictor of their learning and student effort personal development. •Personality •Aspiration •Ethnicity

2 Good Practices in What Really Matters in College: Undergraduate Education (Chickering & Gamson, 1987; Pascarella & Terenzini, 2005) Student Engagement 9 Student-faculty contact Because individual effort and involvement are the critical 9 Active learning determinants of impact, 9 Prompt feedback institutions should focus on the ways they can shape their 9 Time on task academic, interpersonal, and extracurricular offerings to 9 High expectations encourage student engagement. 9 Experiences with diversity Pascarella & Terenzini, How College Affects 9 Cooperation among students Students, 2005, p. 602

National Survey of Student Engagement The Student Engagement Triad (pronounced “nessie”)

Community College ¾ What students do -- time and energy Survey of Student devoted to educationally purposeful Engagement activities (pronounced “sessie”) ¾ What institutions do -- using effective College student surveys that educational practices to induce assess the extent to which students to do the right things students engage in educational practices associated with high ¾ Educationally effective institutions levels of learning and channel student energy toward the development right activities

NSSE Benchmarks of Effective Academic Challenge by Institutional Educational Practice Type 75 75 First-Year Students Seniors 70 70 Level of Active & 65 65 Academic Collaborative 60 60 Challenge Learning 55 55 Student 50 50 Faculty 45 45 Interaction 40 40 Benchmark Scores Enriching Supportive 35 35 Educational Campus 30 30 Environment 25 25 Experiences Doc Doc Bac Bac Nation Doc Doc Bac Bac MA MA Nation Ext Int LA Gen Ext Int LA Gen

3 Academic Challenge at Two Public Universities What have we learned so far?

Grades, persistence, student satisfaction, and engagement go hand in hand

Student engagement varies more within than between institutions.

What have we learned so far? Outcome Measures - NSSE & NSSE & Graduation Rates Educational and Personal Gains (% “very much” or “quite a bit”) Self-Reported Educational and Personal Gains from First-year First-Year Students Seniors Seniors* College students* Thinking critically and analytically 81% 87%

Acquiring a broad general education 82% 86% Academic Challenge .60 .46 Working effectively with others 66% 78%

Writing clearly and effectively 72% 77% Active & Collaborative Learning .23 .09 Learning effectively on your own 70% 77% Student Faculty Interaction .28 .37 Using computing and information technology 65% 76% Acquiring job or work-related knowledge and skills 57% 72% Enriching Educational Experiences .53 .48 Speaking clearly and effectively 60% 72% Understanding yourself 60% 66%

Analyzing quantitative problems 55% 65% Supportive Campus Environment .38 .26 Solving complex real-world problems 49% 58%

Understanding people of other racial and ethnic backgrounds 50% 52% *All correlations are significant at p<.01 Voting in local, state, or national elections 24% 23%

What have we learned so far? Diversity Experiences positively What have we learned so far? related to engagement First-year students Seniors Divers ity Divers ity Divers ity Divers ity Spirituality, Liberal learning & Student Density Climate In Course- Diversity Density Climate In Course- Diversity Engagement Dependent variable Index Diversity Work Press Index Diversity Work Press Student Engagement • More students report participating in spirituality Academic challenge +++enhancing activities Higher order thinking +++ +++ Active and Collaborative +++ +++ • No dampening effect found between participation Diversity-related activities ++++++++ and liberal learning experience and outcomes Supportive Campus Env. Supportive Campus Env. ++ • Students who report spiritual practices also Interpersonal -+ participate in broad cross section of collegiate Support for learning +++ +++ activities Satisfaction -+ + Gains-Interpers. Dev. • Students at faith-based colleges engage in spiritual Gains - Personal/social +++++++ practice more, but participate less in having Gains-Social Awareness conversations with peers with different beliefs, Gains-Understanding div. ++++++++ engage less in deep learning activities Gains -Cont. to community +++ +++ Gains-Understand self +++ +++

4 What have we learned so far? FSSE-NSSE Gap Analysis

Promising findings for First Year Students: ¾ About two-thirds (65%) of faculty expect students to 62% “often” or “very often” ask questions in spend more than 25 hours class preparing for class 58% “often” or “very often” discussed ideas with others outside of class (students, ¾ Only about one-fifth family members) (20%) think that students spend this amount of time - Is this adequate? ¾ Only about one in ten - How often do students report these (12%) students actually activities on your campus? spends this much time - What benchmarks for effective practice do you use?

NSSE – FSSE comparison data Prompt Feedback

Lower Upper Division Division

FACULTY gave prompt feedback often or very often 93% | 93%

1st yr. Seniors Students

STUDENTS received prompt feedback often or very often 64% | 76%

Faculty Priorities and Student Course Emphasis Engagement Lower Upper Division Division AVG STUDENT FACULTY report very much or Academic Active- Diversity Student- quite a bit of emphasis on 14% | 29% AVG FACULTY challenge collab experiences faculty memorizing Academic challenge emphasis 9 9 9 Active-collab 1st yr. 999 9 Seniors practices Students Emphasis on diversity STUDENTS report very much or experiences 9 9 9 quite a bit of emphasis on 65% | 63% Emphasis on higher memorizing order thinking 999 Importance enriching educ experiences 9 9 9

5 Faculty Priorities and Selected Student Outcomes What to make of this?

AVG STUDENT 1. When faculty members emphasize Integrative General Personal/ Practical AVG FACULTY learning education social comp certain educational practices, Academic challenge 9 9 students engage in them to a emphasis greater extent than their peers Active-collab elsewhere. practices 9 9 9 9 elsewhere. Emphasis on diversity experiences 9 9 9 Emphasis on higher 2. Good things go together order thinking 9 9 Importance enriching educ experiences 9 9 9 9

What have we learned so far? Outcome Measures - Deep Deep learning: Learning Activities Clusters ¾ Attend to the underlying meaning of Higher-Order Learning— activities that require information as well as content students to utilize higher levels of mental activity than those required for rote memorization (2b,c,d,e) ¾ Integrate and synthesize different ideas, sources of information Integrative Learning— activities that require integrating acquired knowledge, skills, and ¾ Discern patterns in evidence or phenomena competencies into a meaningful whole (1d,e,i,p,t) ¾ Apply knowledge in different Reflective Learning — activities that ask students to situations explore their experiences of learning to better understand how they learn ¾ View issues from multiple perspectives

Deep Learning Associated with Purposeful Use of Time

“Deep learning is learning that takes root in our apparatus of understanding, in the embedded meanings that define us and that we use to define the world.”

J. Tagg (2003). The learning paradigm college (p. 70). Bolton, MA: Anker

6 Deep Learning Items: Deep Learning Items: Higher-Order Learning Integrative Learning

Students indicate how much (1 = “very little” to 4 = “very Students indicate how often (1 = “never” to 4 = “very often”) much”) their coursework emphasizes: they did the following during the current school year: ¾ Analyzing the basic elements of an idea, experience, ¾ Worked on a paper or project that required integrating or theory, such as examining a particular case or ideas or information from various sources situation in depth and considering its components ¾ Included diverse perspectives (different races, ¾ Synthesizing and organizing ideas, information, or religions, genders, political beliefs, etc.) in class experiences into new, more complex interpretations discussions or writing assignments and relationships ¾ Put together ideas or concepts from different courses ¾ Making judgments about the value of information, when completing assignments or during class arguments, or methods, such as examining how discussions others gathered and interpreted data and assessing the soundness of their conclusions ¾ Discussed ideas from your readings or classes with faculty members outside of class ¾ Applying theories or concepts to practical problems or in new situations ¾ Discussed ideas from your readings or classes with others outside of class (students, family members, co- workers, etc.)

Deep Learning Items: Reflective Learning* DEEP Learning Findings

Students indicate how often (1 = “never” to 4 = “very often”) Seniors in the social sciences, arts and humanities, they did the following during the current school year: professional fields, and education score above ¾ Learned something from discussing questions that have no clear answers biology, while seniors in business, physical sciences, and engineering score below. ¾ Examined the strengths and weaknesses of your own views on a topic or issue For higher-order learning, however, seniors in ¾ Tried to better understand someone else's views by engineering and physical science score higher than imagining how an issue looks from his or her on other scales. perspective ¾ Learned something that changed the way you This advantage is particularly dramatic for understand an issue or concept engineering students. ¾ Applied what you learned in a course to your personal life or work Data show that no fields are essentially void of such activities, while at the same time every field has ¾ Enjoyed completing a task that required a lot of thinking and mental effort room forimprovement.

Partial Correlations Deep learning Implications

¾ Strong relationship between gains in Encouraging deep approaches to learning is important to student personal and intellectual development and learning and development deep learning (.58 to .63 across Student satisfaction is not all about social life and easy academics disciplines) If grades are to reflect the quality of student learning, then assignments ¾ Moderate relationship between and activities that contribute to grades should require students to satisfaction and deep learning (.28 to .37 employ higher-order, reflective, and integrative thinking skills across disciplines) To foster more deep learning, faculty can: ¾ Relatively weak relationship between Ask students to identify and solve unstructured problems that require grades and deep learning (.09 to .20 the use of multiple data sources. across disciplines) Encourage autonomous, experiential learning by taking students into the field and challenging them to deal with real world complexities. ¾ Patterns hold across subscales Progressively increase the intellectual challenge of students’ learning experiences sequentially across courses and throughout their degree.

7 Best Practices in Using NSSE Best Practices in Using NSSE Results: Institutional Examples Results: Institutional Examples Concerned about FY-SP retention. Lees-McRae College used NSSE results to Used NSSE to identify conflict identify areas in the general education between UC image and student curriculum where more structure was needed, experience. Shared with stakeholders to focusing on basic skills in math, writing, brainstorm around retention. reading, and computing before moving on to discipline specific courses. Only 50% of FY students reported that they participated in a “learning community,” University of the South was concerned about though all were required. UC made LC goals, low levels of active and collaborative learning purpose more explicit, saw rise to 75% in among its first year students. NSSE data next NSSE. reinforced faculty members concerns so institution created First Year Program – UC mission focuses on writing, yet NSSE data interactive, stimulating first year seminar did not stand out. Used data to propose new courses. In and outside the classroom strategies around developmental writing. learning activities. Saw 5% retention boost, over 2 years.

Inquiring Minds Want to Know… Project DEEP

To discover, document and describe what high What does an performing institutions educationally do and how they effective college achieved this level of look like in the effectiveness. 21st century?

*Selection criteria: Higher-than- Project DEEP Project DEEP Schools* predicted graduation rates & Higher-than- predicted NSSE Doctoral Extensives Liberal Arts scores To discover, University of Kansas California State, Monterey Bay University of Michigan Macalester College document, and Sweet Briar College describe what Doctoral Intensives The Evergreen State College George Mason University Sewanee: University of the South high performing Miami University (Ohio) Ursinus College Wabash College University of Texas El Paso institutions do Wheaton College (MA) to achieve their Master’s Granting Wofford College Fayetteville State University Baccalaureate General notable level of Gonzaga University Alverno College University of Maine at Farmington effectiveness. Longwood University Winston-Salem State University

8 DEEP - Six Shared Research Approach Conditions Case study method 1. “Living” Mission and “Lived” Educational Philosophy Team of 24 researchers review institutional Educational Philosophy documents and conduct multiple-day site 2. Unshakeable Focus on Student visits, 2 visits to each school Learning 3. Environments Adapted for Educational Observe individuals, classes, events, offices, Enrichment public space 4. Clearly Marked Pathways to Student Individual and group meetings Success 2,700+ people, 60 classes, 30 events, 5. Improvement-Oriented Ethos - Discover and describe effective practices and “Positive Restlessness” programs, campus culture 6. Shared Responsibility for Educational Quality Interim report, debriefings, final report

Engaging ALL Students In Learning: Lessons from DEEP CSUMB anchored in an “assets philosophy” – students prior knowledge is used to foster learning Inventory to Enhance UTEP learning communities emphasize active and Educational collaborative learning, structured group Effectiveness presentations, peer evaluation, and meetings outside of class Sewanee emphasizes importance of prompt, detailed, meaningful feedback George Mason, Wheaton focus on upper division experiential learning in the form of internships, field studies, service learning, capstone courses

Discussion and Comments

Jillian Kinzie, PhD. NSSE Institute Indiana University Center for Postsecondary Research 1900 East 10th Street Eigenmann Hall, Suite 419 Bloomington, IN 47406 Ph: 812-856-5824 Fax: 812-856-5150 www.iub.edu/~nsse [email protected]