What Is Cooperative Education?

All rights reserved. No portion of this book may be reproduced in any fashion, print, facsimile, or electronic, or by any method yet to be developed, without express written permission of the copyright holder.

For further information contact the publisher at:

Center for Cooperative Education Research and Innovation University of Cincinnati P.O. Box 210115 Cincinnati, Ohio 45221

ISBN: 978-0-615-25136-3

The contents of this publication were developed under a grant from the Fund for the Improvement of Postsecondary Education (FIPSE), U.S. Department of Education. However, these contents do not necessarily represent the policy of the Department of Education, and you should not assume endorsement by the Federal Government. Leveraging Cooperative Education to Guide The Development of a Corporate Feedback System for Continuous Improvement Curricular Innovation

Editors: Cheryl Cates and Kettil Cedercreutz

Authors: Kettil Cedercreutz, Cheryl Cates, Anton Harfmann, Marianne Lewis, Richard Miller, Michael Zaretsky, Alexander Christoforidis, Vasso Apostolides, Anita Todd, Zach Osborne, Louis Von Eye, T. Michael Baseheart, Ann Keeling, Darnice Langford, Catherine Maltbie, Thomas Newbold, Jennifer Wiswell Table of Contents

Preface page 1

Abbreviations page 3

Acknowledgements page 5

Biographies page 7

Overview page 11

Chapter 1: Getting Started page 21 What is Cooperative Education? Cooperative Education at the University of Cincinnati The UC Corporate Feedback System Is There a Need for a Corporate Feedback System? Opportunity for Transformational Change via Corporate Feedback Considerations in the Research Team Design Considerations in the Creation of the Corporate Feedback System

Chapter 2: Managing Multiple Levels of Assessment page 34 Overview of Assessment Structure Professional Assessment and Learning System, PAL Assessment Instruments and Evolution of Coop Assessment Closing the Loop

Chapter 3: Generating Effective Reports page 41 Reporting Tools Frequent Questions Results of Practical Implementation Reflections Chapter 4: Case Study - Architecture page 67 Motivations for Joining the Project Description of the School, Architecture Program and Curriculum The Curricular Reform Effort and Results Results and Impact of the Project Future Directions

Chapter 5: Case Study - Business page 86 Motivations for Joining the Project Description of the School, Architecture Program and Curriculum The Curricular Reform Effort and Results Results and Impact of the Project Future Directions

Chapter 6: Case Study – Civil and Environmental Engineering page 96 Motivations for Joining the Project Description of the School, Architecture Program and Curriculum The Curricular Reform Effort and Results Results and Impact of the Project Future Directions

Chapter 7: Lessons Learned page 105 Systems Oriented Lessons Learned Organizational Lessons Learned Future Opportunities

Index page 113

Appendix page 115

Bibliography page 119

Preface

Leveraging Cooperative Education to Guide Curricular Innovation, the Development of a Corporate Feedback Loop for Curricular Improvement is written as a handbook, to guide those interested in implementing a similar system in an experiential learning environment. The book has been co-funded by the Fund for Improvement for Postsecondary Education (FIPSE) and the University of Cincinnati. It covers the organization of cooperative education, information system principles, data analysis, and includes a number of cases describing the implementation of the feed back system for use in curricular innovation within the environments of design, business, and engineering. The goal of the editors has been to produce a handbook that is readable and easy to use. To promote readability, the editors decided to keep literature references in the text to a minimum however the book does contain a large bibliography covering 184 sources.

Wishing you a productive reading experience,

Cincinnati, November 27, 2008

CC and KC

Preface 1

Abbreviations

ABET Accreditation Board of Engineering and Technology ACCE Accreditation Council for Cooperative Education ACT American College Testing Program AI I Assessment Instrument I (General) AI II Assessment Instrument II (Situation specific) AI III Assessment Instrument III (Focus group) ASEE American Society of Engineering Education BEM Building Information Modelling CAFCE Canadian Association for Cooperative Education CAD Computer Aided Design CAS University of Cincinnati, OMI College of Applied Science CEE Civil and Environmental Engineering CEIA Cooperative Education and Internship Association CHEA Council for Higher Education Accreditation CoB University of Cincinnati, College of Business CoE University of Cincinnati, College of Engineering Co-op Cooperative Education DAAP College of Design, Architecture, Art and Planning EAC Evaluation and Assessment Center for Mathematics and Science Education at Miami University ESC Evaluation Services Center FIPSE Fund for the Improvement of Postsecondary Education ET1–ET3 Environmental Technology one, two, and three FYE First Year Experience GPA Grade Point Average IDS Integrated Design Sequence IRB Institutional Review Board HTML Hyper Text Markup Language LC Learning Communities MSM Mean standard deviation matrix MSP Management Skills Practicum NAAB National Architecture Accreditation Board NCCE National Commission for Cooperative Education NCAA North Central Accreditation Agency PAL Professional Assessment and Learning System PP Division of Professional Practice SAID School of Architecture and Interior Design SAT Formerly Scholastics Aptitude Test, today SAT STDEU Standard Deviation TQM Total Quality Management UC University of Cincinnati WACE World Association for Cooperative Education WIL Work Integrated Learning ∆MSM Delta Standard Deviation Matrix * Mathematical and statistical abbreviations covered in chapter 3, table 3.2.

Abbreviations 3

Acknowledgements

This publication is a product of the research cinnati. Grant finances were managed by project “Developing a Corporate Feedback Professional Practice Business Administra- System for Curricular Reform” jointly tor Edward Basista. Professors Vasso Apos- funded by the U.S. Department of Educa- tolides, Alexander Christoforidis, Bruce tion’s Fund for Postsecondary Education Evans, Ann Keeling, Darnice Langford, (FIPSE) and the University of Cincinnati Tom Newbold, Zach Osborne, Anita Todd (UC). Without the ample support of these and Lou Trent from the Division of Profes- two institutions this research would not sional Practice were instrumental members have materialized. The duration of the proj- of the research team. Administrators Nina ect was four years during which the Univer- Thomas and Jennifer Wiswell served in key sity received a total of $555,133 (57 percent). roles for the College of Business. Professors The matching contribution of the University Michael Zaretsky, T. Michael Baseheart, of Cincinnati was $421,396 (43 percent), Bipin Brabhakar, Robert Rost, and Margaret making the grand total $976,529. The grant Reed were key college level representatives was filed by Cheryl Cates as PI and Kettil to the research team. Recording secretaries Cedercreutz as co-PI in a joint effort with Mary Davis and Martha Minges enabled the Accreditation Council for Cooperative the research team to keep an ongoing Education. record of activities. Finally, Applications The project was pursued jointly by UC Analyst Louis Von Eye and a team of co-op Professional Practice faculty and faculty students were pivotal in the development members and team leaders at UC colleges. of an online assessment database without The objective of the grant was to build a which the research could not possibly have closed loop system that measures student been pursued. performance while on co-op and directs The project allows schools accredited by this feedback into curricular development. the Accreditation Council for Cooperative The project involved the Departments of Education (ACCE) including Case Western Architecture (Prof. Harfmann, College of Reserve University; Georgia Institute of Design, Art, Architecture and Planning), Technology; Mississippi State University; Civil and Environmental Engineering (Prof. North Carolina State University; Univer- Miller, College of Engineering), Construc- sity of Central Florida; University of North tion Management (Prof. Uwakweh, College Texas; and others to directly tap into the of Applied Science) and the College of Busi- findings of the research. The letters of sup- ness (Prof. Lewis, College of Business). The port from these ACCE universities were grant was internally assisted by College of crucial in attracting the grant and we would Education-Evaluation Services Center (Dr. like to acknowledge their assistance in the Catherine Maltbie) and externally evaluated development of a highly transferable system by the Miami University Evaluation Cen- for the betterment of cooperative education. ter (Dr. Amy Cassedy and Dr. Jane Butler As ACCE liaisons, the executive director Kahle). The grant laid the foundation for the for the Division of Professional Practice at Center for Cooperative Education Research Georgia Institute of Technology, Thomas and Innovation at the University of Cin- M. Akins, MSIE, and the director of Coop-

Acknowledgements 5 erative Education and Career Services at gram officer for this project, Donald Fischer, Mississippi State University, Luther B. Ept- has been an invaluable member of the team ing, deserve special recognition for work in through his willingness to offer his wisdom connecting the research agenda to the needs during all stages of project evolution. of the field of co-op practitioners. Doctors William McKelvie, Mehmet The research builds on the tradition Karamanoglu, J. Joseph Hoey IV, Tim established by Dean Herman Schneider Mott, William Williams, Ralph Katerberg, who in 1906 initiated Cooperative Educa- Emmanuel Contomanolis have in various tion at the University of Cincinnati. Dur- capacities served as project consultants. ing the past 100 years the methodology has Their support has been pivotal in making spread to over 900 institutions world wide. the grant a success. The research pursued in this publication A very special kudos goes to Beth Hess, relies directly on the three party assessment whose eye for graphic design has made this procedure developed by Division of Pro- publication come to life. fessional Practice (PP) faculty. The faculty is also very much to be credited with the development of an assessment system that today gathers in excess of 400,000 data points annually. The internal development of the Professional Assessment and Learn- ing System (PAL) would never have been possible without the input of the entire faculty. The FIPSE project has been strongly supported by the four deans of the UC co-op colleges. Without support on a deca- nal, department head, and faculty level the building of a feedback process for curricular reform would obviously not be possible. The support of the University of Cincinnati administration, including President Nancy L. Zimpher, Senor Vice President and Pro- vost Anthony Perzigian, Senior Vice Provost Kristi Nelson and Director Emeritus E. Sam Sovilla has been pivotal. The UC administration has strongly backed the funding of the initiative, as well as supported the presentation of research results at external and internal forums. In addition to the financial support from the U.S. Department of Education’s Fund for the Improvement of Postsecondary Education, the advice and counsel of their program officers has been invaluable part of the project’s success. In particular, the pro-

6 Preface Biographies

Vasso Apostolides, MS in urban and regional planning and library and information science, and registered architect (Greece) is an associate professor of professional practice at the University of Cincinnati and lead faculty responsible for the professional practice education of students in the undergraduate Architecture program. Her research interests are on topics related to professional learning outcomes and developing curricula for optimizing student benefits from cooperative education. Her findings have been published in the Journal of Cooperative Education.

T. Michael Baseheart, PhD is an associate professor at the University of Cincinnati Department of Civil and Environmental Engineering. Baseh- eart has over the years been awarded numerous awards for excellence in teaching by college faculty and students. Professor Baseheart teaching interests range from structural dynamics to reinforced concrete design. He has been the program director for the Civil Engineering program since 1995.

Cheryl Cates, MBA, ABD is an associate director and associate professor at the University of Cincinnati Division of Professional Practice. She holds a master of business administration degree and a bachelor of arts degree and has co-authored “Learning Outcomes, the Educational Value of Coop- erative Education,” as well as chapters for the “Handbook for Research in Cooperative Education and Internships and the International Handbook for Cooperative Education.” Professor Cates has worked with cooperative education for 18 years and has served as the director of the UC Center for Cooperative Education Research and Innovation since 2007.

Kettil Cedercreutz, PhD serves as associate provost and director of the Division of Professional Practice at the University of Cincinnati, were he oversees the integrity of the cooperative education program originally developed by Dean Herman Schneider. He is an associate professor and former department head of mechanical engineering technology. Ceder- creutz has served as a department head and senior lecturer at the Swed- ish Institute of Technology, Helsinki today a part of Arcada Polytechnic, Finland.

Biographies 7 Alexander D. Christoforidis, AIA AICP is a practicing licensed architect, and a certified planner. He is an assistant professor at the University of Cincinnati’s Division of Professional Practice and teaches graduate students of architecture. He holds a bachelor of science in architecture, a master of architecture, and a master of city and regional planning from The Ohio State University. He designs residential, commercial and eccle- siastical projects, and has led teaching initiatives which bring practitioners into the education of architecture students, including work with the AIA Practice Academy.

Anton C. Harfmann, MArch and registered architect is an associate professor in the School of Architecture and Interior Design, at the University of Cincinnati College of Design, Architecture, Art and Planning. He also serves as the associate dean for academic technology and facilities for the college. Professor Harfmann has produced “Voices of Practice” a documentary film capturing the life histories of practicing architects and “Imagine Building,” a documentary on the design and construction of the Lois and Richard Rosenthal Contemporary Arts Center.

Darnice R. Langford, MBA is an associate professor of Professional Practice where she advises and places business and engineering students. She chairs the Professional Practice Curriculum Committee, and has co-written and led the effort to develop the co-op textbook titled “Professional Practice: A Student Text/Workbook Designed to Enhance the Cooperative Education Experience.” She has published on various co-op topics including diversity and learning outcomes in the Journal of Cooperative Education.

Marianne W. Lewis, PhD is an associate professor of management at the University of Cincinnati and the director of Kolodzik Business Scholars. Her research explores tensions, conflicts, and paradoxes that both impede and enable innovation. In particular, her work addresses the challenges of developing new products, implementing technological and organizational change, and building organization theory. She has published in the Acad- emy of Management Review, Academy of Management Journal, Organiza- tion Science, Journal of Operations Management, Human Relations and Journal of Management Education.

Ann E. Keeling, MS is a professor in the University of Cincinnati, Division of Professional Practice. Her academic work focuses on cooperative education including multiethnic students. She has authored articles on market- able resumes and cover letters published in The Black Collegian (2002) and co-authored a chapter on College-to-Career Transition Programs for Multiethnic Students published in The Senior Year Experience - Facilitating Integration, Reflection, Closure & Transition (Gardner, et.al., 1998). Profes- sor Keeling has 37 years of experience in the field of cooperative education.

8 Biographies Catherine V. Maltbie, EdD is a research associate in the Evaluation Ser- vices Center for the University of Cincinnati College of Education, Crimi- nal Justice and Human Services. She has a BS in chemical engineering and an EdD in educational foundations related to the cognitive and social effects of integrating technology into instruction. Dr. Maltbie has extensive experience in outcomes based assessment and evaluation, as it pertains to experiential learning. She is also a trusted focus group facilitator.

Richard Miller, PhD, PE, FPCI is a professor of civil and environmental engineering at the University of Cincinnati. His areas of expertise are prestressed concrete design, concrete bridges and concrete materials. Dr. Miller is one of the initiators of the PCI Big Beam Contest for engineering students and he serves as head of the rules and judging committee. He is the chair of the Civil and Environmental Engineering Curriculum Committee and a member of the College of Engineering Curriculum Committee.

Tom Newbold, MEd is an associate professor at the University of Cincin- nati, Division of Professional Practice, where he is in charge of the co-op instruction of the Civil and Environmental Engineering, Operations Man- agement, and Industrial Management programs. Professor Newbold holds 23 years of co-op experience earned in both a five year, as well as a community college environment. He has been named by the Ohio Cooperative Education Association (OCEA) Educator of the Year.

Zach Osborne, MEd is assistant professor and assistant director at the University of Cincinnati, Division of Professional Practice. He is the 2008 president-elect of the Midwest Cooperative Education and Internship Association. Osborne completed his undergraduate work in Operations Management and graduate work in education at the University of Cincin- nati. He has been strongly engaged in the development of the Professional Assessment System PAL.

Anita M. Todd, MEd is an associate professor at the Division of Profes- sional Practice. She is the 2007/08 president of the Cooperative Educa- tion and Internship Association (CEIA) and has held board positions in the American Society of Engineering Education Cooperative Education Division (ASEE/CED) and the Pennsylvania Association for Colleges and Employers. She is a faculty advisor for the Society of Women Engineers (SWE) section at UC. Professor Todd was awarded the 2004 Dean Herman Schneider Award. She is actively involved in the development of the co-op assessment system PAL.

Biographies 9 Louis Von Eye, BSIT is an applications analyst at the University of Cincin- nati, Division of Professional Practice. He received his bachelor’s degree in information technology from the University of Cincinnati in 2007. He has co-oped at General Electric, Seapine Software, and the University of Cin- cinnati, where he worked closely with coding the PAL assessment system. Since 2007, Louis has been heading the development of the UC PAL system on a full time basis as his first post graduate position.

Jennifer Wiswell is associate director of undergraduate programs at the University of Cincinnati, College of Business with particular responsibili- ties for advising, engagement and retention. Jennifer has a career of successive promotions from academic advisor, senior academic advisor, director of student engagement, to associate director of undergraduate programs at the university. She earned a BA in English literature and language from Ohio University’s Honors Tutorial College in 1997.

Michael L. Zaretsky, MArch is a licensed architect, LEED AP and an assistant professor of architecture at the University of Cincinnati. He has been involved in the practice, research, consulting and teaching of architecture that emanates from sustainable principles. He is a faculty member in the School of Architecture and Interior Design teaching undergraduate and graduate students of architecture and interior design in the studio and lecture contexts. He is a board member of the Cincinnati chapter of the US Green Building Council and the secretary/treasurer of the Society of Build- ing Science Educators.

10 Biographies Project Overview By Kettil Cedercreutz and Cheryl Cates

This publication is based on the body of The data used in the project was knowledge developed through a project aggregated from 5,000 annual evalua- Developing a Corporate Feedback Loop tions of co-op students completed by their for Curricular Reform which was funded work supervisors. The main objective is to by the U.S. Department of Education’s build a recurring process that allows the Fund for Improvement of Postsecondary institution to react to a variety of changes Education (FIPSE). The project is set in a in its environment and make appropriate cooperative education environment. The adjustments to the curriculum. Project objective is to develop a methodology by collaborators included the four University which student cooperative education work of Cincinnati co-op colleges together with assessment data can be used to enhance the UC College of Education Evaluation curricular efficacy. The project included Services Center (ESC), the Division of Pro- participants from four University of Cin- fessional Practice and representatives for cinnati (UC) colleges; the Colleges of Engi- the Accreditation Council for Cooperative neering (CoE); Design, Architecture, Art, Education (ACCE). The partnership with and Planning (DAAP); Applied Science the Accreditation Council for Coopera- (CAS); and Business (CoB), and involved tive Education enabled the University of individual programs as follows: Civil and Cincinnati to build a system that is highly Environmental Engineering, Architecture, transferable to other ACCE accredited Construction Management, Accounting, institutions. The Evaluation and Assess- and Information Systems. ment Center for Mathematics and Science From the standpoint of the Division of Education at Miami University (EAC) was Professional Practice (PP), the centralized responsible for the external evaluation of unit responsible for cooperative educa- the project. tion at UC, a part of the motivation was to In order to penetrate all four co-op develop a methodology to assist faculty in colleges in parallel, the project was initially assessing the efficacy of the curriculum. An designed to include one department from important outcome of the project is that each college. The PIs invited departments the Division of Professional Practice can who were assumed to be very receptive to now partner with degree granting depart- the idea of a corporate feedback system, ments not only in their curricular devel- based on an established track record of opment efforts but also in their efforts to external orientation and a strong history of demonstrate to accreditation bodies that partnerships with industry. The prestige of their programs meet employment needs. the program and the individual status of the The project demonstrates the potential of team member within the college was also a co-op and work integrated learning to form consideration in forming the UC team. a foundation for the formative assessment The participating academic depart- pursued by an institution. The data can help ments share three common characteris- faculty reflect upon their own teaching, and tics: they are highly ranked by their peers; in doing so find ways to enhance student their strategic aspirations are explicitly learning and program efficacy. industry orientated; and they have a strong

Project Overview 11 track record of engagement in cooperative lege level. The College of Business has education. Their academic focus is similar a long tradition of offering programs enough to allow a coherent agenda, but based on optional cooperative educa- different enough to promote a variation tion. As segregation into majors takes in their problem solving approach. The place late in the curriculum the College characteristics of the programs are pre- of Business team eventually decided sented below: to participate in the project without departmental differentiation. Architecture: Hosted at the College of Design, Architecture, Art, and Plan- Professional Practice: The Division ning, the baccalaureate level archi- of Professional Practice is responsible tecture program has been ranked in for the field-based academic instruc- the top 10 in the country by Design tion of over 4,000 students annually Intelligence for the past seven years. at the University of Cincinnati. Its The department has a long tradition of unmatched, professionally-oriented industry orientation offering coopera- faculty base, its strong ties to more tive education on both a baccalaureate than 1,500 employers and its 100-year and a graduate level. track record of co-op, makes the division one of the leading academic units Civil and Environmental Engineer- engaged in cooperative education in ing: Hosted at the College of Engi- the world. neering, the Civil and Environmental Engineering Department has done The programs offered by the above groundbreaking work in the devel- departments in cooperation with the opment of cooperative education on Division of Professional Practice cater both a baccalaureate and a graduate to shared groups of employers. The pro- level. The recently introduced five- grams share curricular elements, but the year combined MS/BS programs are focus of specific curricula is very differ- already setting a new standard for ent ranging from the perspective of art cooperative education. (architecture), analysis (civil and environmental), management and technology Construction Management: Hosted (construction management) and economy at the College of Applied Science, the (college of business). The programs gradu- Construction Management program ate individuals on associate, baccalaureate focuses on the education of man- and masters levels through both manda- agers for construction sites from a tory and optional cooperative education technologically-oriented approach. programs. The fact that each depart- The department offers both baccalau- ment approaches problem solving from reate and associate level programs in a unique perspective provided a strong construction management and archi- foundation for finding best practices tecture technology. The department through cross fertilization. Each leader of did however opt to leave the project a program-specific team also served as a during its second year. liaison between the project and individual Business Administration: The College departments and colleges. of Business participated in the process The project addressed the curricular on both a departmental (accounting reform challenge by applying a three level and information systems) and a col- assessment methodology relying on three

12 Project Overview different assessment instruments; Assess- piled for four programs (Civil Engi- ment Instrument I (standard faculty neering, Accounting, Architecture and developed student work performance Information Systems). The initial data assessment instrument), Assessment reporting was tracked on a quarter-by- Instrument II (tailored assessment instru- quarter basis, but the low n values (total ment to be used for a variety of specific number of responses per sample group) measurements), and Assessment Instru- did not produce a reliable base for deci- ment III (focus group interviews poten- sion making. The research team worked tially involving students, employers, and diligently to find the best way to analyze faculty) throughout two cycles of curricu- all 50,758 data points (74.7 percent return lar revisions. rate, 1,656 surveys, and 41 questions per survey). Six Sigma type process stabil- Assessment Instrument I (AI I) ity analysis methodologies were used to produce a picture of the stability and This instrument was the primary instru- effectiveness of the academic programs ment used in all programs and all cycles. under scrutiny. During three consecu- Assessment Instrument I was developed tive years, all four programs showed in the mid 1990s and was at that point remarkable stability. The annual aver- validated for use by co-op faculty, co-op age of all measured parameters varied students, and co-op employers as a tool between ±0.03, ±0.04, and ±0.05 for three to assess individual students. The instru- most stable programs. The most unstable ment is reliable having a Cronbach Coef- program had an annual mean fluctua- ficient Alpha of .80 and above for all tion of ±0.09. The high program stabil- constructs. The validation was enhanced ity (uniformity of results independent through a web-based survey mapping the of variations in job market and student importance of a variety of skills at differ- enrollment) allowed the project to apply ent curricular levels. This survey yielded process stability analysis to the measured 505 responses out of which 115 pertained parameters. Analyzing the difference in to the five programs participating in the performance between rolling three-year grant. The survey was sent to all 1,500 UC averages of sophomores, pre-juniors and co-op employers. The return rate of the juniors resulted in statistically signifi- survey was 33 percent. The report gave cant differences (95 percent confidence relevant information useful for further level) between student performance on refining and filtering of measurement the different academic levels. The project results. The data shows that the skill pal- further developed visual tools for com- let required by employers varies slightly municating measurement results. By between programs and as a function of plotting the measurement outcomes in the students’ progression in the program. two different matrixes, Mean Standard Survey results were further discussed Deviation Matrixes [MSM], Figure Ia, and in employer focus groups. Assessment Delta Mean Standard Deviation Matrixes Instrument I used in this research is [∆MSM], Figure Ib, the team was able to effective in the measurement of soft skills, graphically illustrate the entry skill level, as these typically can be observed by the the exit skill level, and the development supervisor. of the skill level within each program. Three years of quantitative employer Chapter 3 of this publication examines evaluation assessment data was com- the methodology in depth.

Project Overview 13 Standard Deviation se be used to compare different programs

Low Medium High to one another; 4) the instrument shows remarkable differences between program profiles; and 5) the matrixes can be High developed into a program fingerprint. The MSM and ∆MSM methodology have been presented at a number of conferences

Mean and meetings, and feedback indicates Medium that the project has found a novel way of communicating assessment data in an

educational context. Companies, as well Low established as UPS, have shown interest in using the methodology in the assessment of their employees. Figure I a) Mean Standard Deviation Matrix. Assessment Instrument II (AI II) Program-specific assessment instruments (AI II) were developed by each program-level team based upon data from Assessment Instrument I and Assessment Instrument III (focus groups, details presented below). The instrument was implemented in accordance with the needs of individual programs. During the first cycle of curricular reform, the project chose to implement Figure I b) Delta Mean Standard Assessment Instrument II for civil and Deviation Matrix. environmental engineering and architecture during fall quarter 2005 in order A quality educational process pro- to gather both pre and post data. The duces a high outcome level at a low level other programs chose to implement AI of variation. When a process is in control II in the spring of 2006 to collect forma- student averages increase, and the vari- tive data used to plan for their curricular ability [measured through the standard changes. The Architecture AI II focused deviation] decreases as a function of the on mapping performance with regard to student’s progress in the program. The construction detailing, whereas the Civil MSMs give a snapshot of a program’s per- Engineering AI II focused on understand- formance at a specific stage in the curric- ing of structural analysis during the first ulum, whereas the ∆MSM describes the cycle. Both of these instruments were progress of a student population over a quantitative in nature. specific period of time. Initial data shows During the second cycle of curricular that: 1) no findings were counter intuitive; reform, the architecture program added 2) the instrument is relative and should a second AI II that focused on sustain- be used for continuous improvement able design. Civil engineering chose to purposes; 3) the instrument cannot per focus on structural design and therefore

14 Project Overview chose to use the same AI II questions for Accounting: Employers expect differ- its second curricular change. During the ent skills based upon the number of second cycle, the business team opted co-op terms the student has worked. to use a subset of AI I parameters as Conceptual and analytical ability, their AI II, focusing on students’ abili- professional qualities and teamwork ties in leadership, project management, are particularly important of upper communication skills and teamwork. class students. In creating the custom assessment tool Architecture: UC’s co-op students are known as Assessment Instrument II, highly creative and innovative, they content knowledge from faculty was a excel at design, and they have a strong key requirement. appreciation of a company’s work cul- When institutionalized, AI II will ture. Areas of concern include the stu- provide a custom methodology with dents’ ability in construction detail- which individual departments can assess ing, the fact that students only stay for critical curricular questions. AI IIs are two quarters with any employer, and planned to have a short life cycle, which the length of time spent on the assess- puts high requirements on the testing of ment of students. the validity of the instrument. In order to be successful AI IIs must measure issues Civil and Environmental Engineering: that the supervisors have the capacity to Students need to be well grounded in assess. The methodology lends itself well theory. The employers want to retain to measuring behavioral issues in student quality co-op students for their entire performance, but less well to measuring co-op experience (six of six quarters). technicalities related to student knowl- The assessment of students is also edge. This is especially true when students considered to be too time consuming. perform specialty tasks, the details of Construction Management: Employ- which supervisors lack understanding. ers want to retain quality students The use of AI II also requires that one is for their entire co-op experience (six measuring behavior in situations to which of six quarters). Students need to be a co-op student is frequently exposed. The well-prepared in the core curriculum. project showed that AI II will only gather Specific skills that are important for reliable data if used to measure issues that upper class students include sched- are transparent to a supervisor. uling, software proficiency, and the exhibition of line and grade skills. Assessment Instrument III (AI III) Information Systems: Employers want to retain quality students for their Focus Groups entire co-op experience (six of six In the first cycle of curricular reform quarters). There is little concern for a total of five discipline-specific focus specific skills in entry level co-op jobs groups were conducted to determine as employers are convinced that these strengths and weaknesses of co-op stu- skills can be gained via on-the-job dent performance skills and to develop a training if students have the intelli- more specific concentration for curricu- gence and appropriate work ethic. The lar reform. The focus groups produced ability to work independently as well program-specific results as follows: as on teams is essential.

Project Overview 15 In the second cycle of curricular reform, with as many skills as possible to the project opted to use individual tele- increase their employment opportu- phone interviews with employers as the nities and productivity during their geographic separation of the key employ- first coop experience. Consequently, ers was too large to hold a focus group the curriculum includes a computer interview. The goal of the individual inter- skills class during the winter quarter to views was to focus on the outcomes of the give students critical computer aided first cycle of reform as formative input for design (CAD) skills. The CAD class the second cycle. The results of the indi- focuses primarily on skill develop- vidual interviews were inconclusive as no ment and does not offer any exposure clear patterns of opinions regarding the to the use of the software to produce results of the curricular change in observ- construction documents. Parallel to able student performance emerged. From the CAD class, students take their these two cycles of interviews the research first comprehensive construction class team learned that focus groups/individual which exposes them to various types of interviews are more effective when map- building construction. The construc- ping general issues whereas the use of sta- tion class and CAD class have evolved tistics is more likely to prove differences as independent courses over the years in before and after studies. with no interaction between them. Faculty identified these two classes as Curriculum Reform Efforts and Drivers excellent candidates for an intervention and developed an additional, more- Changes in curricula have wide conse- specific instrument (AI II) asking quences, and departments tend not to employers to verify the effectiveness of pursue them without a significant consid- the curricular change in these courses. eration. The curricular changes precipi- A single shared project that bridged the tated by this research project are described two classes was developed by faculty in further detail in chapters four, five and and implemented during the winter six. Two departments (civil engineering and architecture) found an opportunity to quarter of 2006 and repeated during make a curricular change during the first the second cycle of curricular reform and second cycles of the grant. The College in the winter of 2007. of Business implemented change, in the A second curricular change included second cycle only. the introduction of a completely Architecture: In the first cycle of revised Environmental Technology reform, initial evaluation of the feed- I (ETI) course taught to architec- back from employers revealed concern ture students in the fall of 2006. The about students’ limited ability to navi- department hired a new tenure track gate the complexity of technical aspects faculty member with a specialization of building. The review also revealed a in sustainable design. His charge was curricular shortcoming in the winter to redesign the ETI course, therefore, quarter of the sophomore year, the last AI II questions focused on the changes quarter of the undergraduate cur- he introduced and compared student riculum before students begin their understanding of sustainable design mandatory co-op rotation. Within this concepts from the previous course to quarter the goal is to provide students the redesigned one.

16 Project Overview Civil end Environmental Engineering: a sequence of two courses, Management While the architecture program focused Skills Practicum I and II offered during on employer evaluation data to deter- the fall and winter quarters of the fresh- mine the area in which to make a men year. The sequence was designed change, the Department of Civil and to enhance students’ academic and Environmental Engineering (CEE) was professional preparation. The sequence primarily prompted to make curricular was drastically revised from a lecture changes as a result of a recent accredi- based structure to a course with three tation review by the Accreditation interwoven components: lectures and Board for Engineering and Technol- discussions on critical business func- ogy (ABET) and utilized the feedback tions; team based research projects each system to verify the impact of the sug- focusing on a different organization; gested change. The department required and executive panels in which manag- undergraduate students to take three ers from leading local businesses share required courses in the emphasis area their experiences and expertise. of structures. In this area students took The research team focused not only on the Basic Strength of Materials, Structures specific curricular changes that were intro- I, and Structures II. The ABET evalua- duced during the project but also on map- tor indicated that this sequence did not ping the motivations for curricular change expose students to design. To remedy in general within the various colleges and this situation, the Faculty changed the departments. Drivers for any curriculum sequence to Basic Strength of Materi- change at the College of Business included als, Structural Analysis and an elective the need for transfer and articulation agree- course in Structural Design. As a result, ments, co-op employer satisfaction, national less time is spent on some material reputation, sustainability of the change, and other material is dropped com- and leveraging feedback to build stronger pletely as Structures I and II become relations. Drivers for 95 percent of the Civil one Structural Analysis course. AI II and Environmental Engineering cur- asked employers about the value of ricular changes could be attributed to the specific topics in Structural Analysis requirements of either co-op or ABET. The as well as to rank their co-op student’s curriculum is developed around the idea understanding of the topics using the that the program must give students spe- same scale as used on AI I. The required cific knowledge and skills to use on co-op. Structural Analysis course became The department is also strongly motivated the focus of study in the first cycle of to keep its ABET accreditation. Drivers curricular reform whereas the elective for Architecture curricular change, also Structural Design course became the included the accreditation of the program focus of study in the second cycle of by the National Accreditation for Architec- curricular reform. ture Board. The changes are further driven Business: The College of Business used by the National Council for Architecture the first cycle as a planning cycle and registration board that oversees licensure. gathered specialized data regarding the UC’s general education requirements con- knowledge and skills that employers stitute a minor driver. A more significant want students to possess before their driver is the pressure to be leading the field first co-op work term. The College of of architecture and evolving the theory of Business curricular reform occurred in the profession given the UC Architecture

Project Overview 17 program’s national ranking. Architecture started are described in chapter one as faculty members are focused on looking well as some rationales that can be used by 10 to 15 years ahead which differs from potential adaptors to secure a strong foun- the focus of most supervisors. This allows dation on their own campuses. the curriculum to provide skills that the The research team discovered that companies do not yet possess. For exam- the implementation of the methodology ple, smaller firms have limited abilities requires a user-friendly system that is con- to create 3D animated walk throughs of venient and easy to use for the supervisor, their building designs. The co-op students the student and the faculty member as well can, with their laptops, quickly model the as the person statistically analyzing the building making them more valuable. student performance. The system needs to be able to create a strong historical baseline to allow for effective assessment of student Opportunities, Challenges performance. An effective system needs to and Lessons Learned also have mechanisms for eliminating data The research was pursued in an environ- pertaining to students who stop following ment of strong institutional support. The a specific schedule or transfer to, from, or University of Cincinnati is the founding between programs. It is vitally important institution of Cooperative Education, that the system is able to measure both committed to retaining the position as the the means and the uniformity of student premier institution engaged in Co-op. On performance throughout the curriculum. an individual student basis, assessment of Chapter two describes the online assess- each cooperative education work experiment system and instruments used within ence was a part of the standard operating the project in further detail. procedure long before this project was At the outset the availability of quality initiated, which provided a strong founda- data was considered highly probable, as the tion for further development. primary assessment instrument in use (AI A series of multifaceted challenges I) records over 200,000 data points annu- faced the project despite its strong founda- ally. During the project the research team tion. The project required the creation of learned that employer data placed into a research team with a variety of perspec- Mean Standard Deviation and Delta Mean tives and a variety of institutional roles. Standard Deviation Matrixes, borrowed An important challenge for the project from the six sigma literature, are effective was to develop a climate of trust and form measures to convey performance levels and a cohesive team. The internal culture of uniformity of various student cohorts. Due trust was necessary given the fact that a to the nature of the feedback, employer transfer of assessment power from faculty data is effective when pursuing continuous to employers could be seen as threatening improvement of individual programs. The in an academic setting. The research team data is however not applicable to compar- needed to develop a vision for the project ing two programs with one another but and a familiarity with the characteristics can be used to establish benchmarks. If of the various programs. A key challenge one program has a higher probability of the project faced was to address different developing a skill than the others that statistical methodologies in order to deter- program can be analyzed and others can mine the most appropriate method for the try to meet this benchmark by following in analysis. Some of the challenges of getting its footsteps.

18 Project Overview The developed methodology measures to behavioral diversity. Initially the second only issues that are visible to supervisors section to enter co-op does slightly better and measures student performance from a than the first section, however, perfor- supervisor perspective. From this perspec- mance levels switch when the first section tive, for example, the development of team- hits junior level. Preliminary findings work skills can lead to lower perceived goal indicate that having more practical experi- orientation as the student takes the lead ence seems to support the understanding from the group rather than the supervisor. of junior courses better. It is important to consider the perspective As the project evolved the research of the assessor when creating meaning team discovered that when interacting from the data. The primary assessment with departments it is easy to deliver a instrument (AI I) measures behavior in positive message but more challenging to analytical ability and management skills deliver results that are less than stellar. very well, however, it is less effective at With positive results departments tend measuring creative ability limiting its use to have faith in the data and the analysis. in some areas. Negative results unleash criticisms of the Effective analysis can be pursued by methodology; therefore, the establishment analyzing data based on snap shots; rolling of a solid research protocol is essential. averages, as well as longitudinal studies. Interpretation of the data is best left in the The project showed that mature programs hands of the academic departments as this exhibit stable student behavior when com- will contribute to an atmosphere of trust, paring one academic year to another. The which can sometimes be difficult to build research shows that the diversity of asses- within departments and is even more diffi- sor bias (lean and tough raters) becomes cult to build across departments. Programs manageable whenever n-values exceed that fail to develop trust in the team and 20 in each group subject to comparison, the project tend to become disengaged in and whenever the amount of supervisors the research. Trust, therefore, is a criti- is of the same order of magnitude as the cal program component. The researchers amount of students in each cohort. When also discovered that the issue of training compared to the typical size of any depart- versus education is a debate that the proj- mental faculty, the amount of industrial ect can not resolve, yet bringing the issue assessors is often orders of magnitude to the forefront of faculty attention was higher which provides a relatively stable extremely helpful. Faculty will always face reference of student performance. Consult the challenge of maintaining the right bal- chapter three for more in depth discussion ance between preparing students for their on effective reporting of quantitative data. immediate as well as long term future. Preliminary findings support the The developed assessment method- notion that programs that teach to external ologies demonstrated that learning can tests (such as CPA licensure exams) exhibit be verified through the aggregation of remarkably high performance uniformity, employer feedback which creates a pro- as compared to programs that focus on gram fingerprint revealing the progression the creation of problem solving abilities. of student skill development. The data Preliminary findings further support that identifies areas in which students are sys- adding more rigor to the instructional pro- tematically as well as serendipitously learn- cess adds to the uniformity of student out- ing thus enabling departments to focus on comes whereas reduction of rigor increases the quality of teaching and learning in a

Project Overview 19 new way. Combining this analysis with the decidions. This allows faculty to come to introduction of targeted questions and the their own conclusions by looking at the opportunity to bring together focus groups data generated through the corporate feed- of employers allows departments to target back system. areas for improvement, implement changes, and evaluate the results of those changes as part of a systematic feedback loop. The project achieved its objectives, but as is so often the case, with some variation from the original plan. A more comprehensive discussion of the lessons learned from the project can be found in chapter seven. The primary activities in the original plan included analysis of co-op employer data and gathering additional survey data on student performance to determine if changes were needed in the curriculum to better educate the students. In addition to those original activities, the project gave faculty a better understanding of the skills students have, or don’t have, and how those classroom skills translate to job skills. The faculty also developed a better understanding of how what they teach impacts student performance. Individual subjects (e.g. a specific method of analyzing a structure) have almost no impact on student ratings because of the tremendous variety exhibited by employers in their projects and work situations. Individual courses have some impact as to whether or not a student has the skills to get a specific job. In general, the biggest impacts are seen in broader skills such as communication, analytical ability, problem solving, etc. Faculty learned that employability is largely based upon the skills provided through the student’s curriculum. Per- haps most importantly, the research team learned that the framing of the corporate feedback system is critical and that it should never be presented as ‘business dictating the curriculum.’ Instead, faculty members and faculty groups must view the corporate feedback system as a tool that can be used to help make curricular

20 Project Overview Chapter One

Getting Started By Cheryl Cates and Kettil Cedercreutz

“To be successful in the increasingly complex, demanding, and competitive setting…the academy must recognize the leadership challenges it faces and devote increasing attention to expectations and concerns articulated by the external constituencies that provide the moral and financial support necessary to our functioning. As with other service providers, changes in marketplace needs and expectations create intensifying demands and emerging opportunities. Organizations must adapt or risk obsolescence and atrophy.” — Brent Ruben, 2004

The project described in this publication The results of the project indicate that developed methodologies to use assessment the methodology of using assessment data data of co-op student work term perfor- generated through co-op student work term mance in curricular development, thereby performance evaluations can inform faculty continuously aligning cooperative-education decisions regarding curricular reform based curricula with industrial needs. It assuming that the data is gathered and pre- was funded by a grant by the U.S. Depart- sented in an effective manner. This chapter ment of Education’s Fund for the Improve- describes the foundations upon which the ment of Postsecondary Education (FIPSE). University of Cincinnati FIPSE project was The ultimate objective was to move schools built as guidance for others who may wish engaged in cooperative education to a new to pursue a similar endeavor. era of market alignment by building feedback structures that keep schools abreast with a rapidly-changing environment. What is Cooperative Education? This would allow these schools to receive Cooperative education was pioneered by enhanced benefits from cooperative educa- Dean Herman Schneider in 1906 at the tion as well as support an efficient allocation University of Cincinnati and since then the of educational resources. The idea behind concept has become an educational initia- the University of Cincinnati FIPSE project tive that has transcended time, disciplines, “Developing a Corporate Feedback System and programs. Schneider’s cooperative for Use in Curricular Reform” was to develop system of education introduced the concept a system that would allow corporate feedback of linking theory with practice through to guide decision making and to alert faculty the alternation of time spent in classroom to important issues. The objective was not to instruction with time spent in work based make an overly sensitive system as false sig- practical experience in the students’ chosen nals, asking faculty to react to issues of little fields. Even though students were exposed importance, would soon discredit the system. to engineering theory through pictures in a Avoiding the boy who cried wolf syndrome text or demonstrations in a lab, these would was considered very important. The reports only be abstract concepts until the students generated through the system are to be con- actually began to apply them in a work sidered more as a standard physical, than a setting. As students operated various types deep analysis of a specific syndrome. of machinery, the latent educational value

Chapter 1 Getting Started 21 of the machines emerged and the students’ • understanding by the university of the motivation, curiosity, and thinking had students’ aptitudes, academic prepara- been directed along productive lines. The tion and the relationship between class- first cooperative education program was room theory and practical applications; established on the University of Cincin- • development of methods in which the nati campus in September, 1906 with 27 student is led to observe and think for engineering students and 12 participating himself/herself at the work site; companies. The students were paired up • grading of the student’s cooperative and alternated between work and classroom education experiences; on a weekly basis as part of a well structured program with the intent of preparing • and the recognition of cooperative students to become strong professionally education work experiences as part of a oriented engineers. student’s overall education. A century later, the design of the corpo- Additional pedagogic principles must also rate feedback system sought to re-examine guide an institution in the creation of its and build upon the original core principles cooperative education model including the of the cooperative system of education, belief that: founded by Dean Herman Schneider. These • critical selection and systematic analy- guiding principles are the key foundational sis of work experiences maximizes the concepts under which the cooperative edu- education that students gain; cation model was established and include: • selecting positions on the basis of their • company selection of co-op students as educational value is critical and bringing employees (not placement in positions out a position’s educational value requires by the university); a knowledge of student aptitudes and an • a sequential training environment in understanding of the curriculum; which students come to understand • an ongoing examination of types of theory through its practical application; work experiences as well as their place • the critical evaluation of co-op positions in a sequential development advancing by the university and inclusion of posi- from entry-level to more responsibility tions with the greatest possible amount is necessary; of educational content; • and the belief that the combination • conditions of actual employment (not of theoretical instruction carried out artificial conditions imposed upon simultaneously with practical training employers which could reduce the expe- provides students with a foundation in rience to a laboratory experience in a basic principles and the ability to use campus environment); those principles in practice. • co-op positions as paid positions Program administration based upon these because this is the primary component principles requires: of the work environment; • the creation of a co-op schedule/co-op • curriculum planning by the university calendar which determines the start and to provide the opportunity for sequen- stop date for each co-op work term; tial training in a work environment; • establishment of policies regarding the • ongoing evaluation of curricular design student’s ability to change from one both on campus and off; work environment to another;

22 Chapter 1 Getting Started • creation of an administrative structure learning through the systematic introduc- to handle business questions that are tion of practical work in a real world setting not of an educational nature; to complement the traditional classroom. • determination of workload issues For the purposes of this publication, we use regarding how students and employers cooperative education as our only meth- will be serviced and by whom; odology but recognize that many other experiential models will be able to duplicate • development of a curriculum and defi- similar results. The more the model used nition of the co-op mission in relation resembles cooperative education at the Uni- to that curriculum; versity of Cincinnati, the more similar the • establishment of a methodology to results. With some modifications the cor- gather relevant positions from industry porate feedback system developed through and make those positions available to this project could be used in a variety of students; experiential learning environments pro- • creation of a process by which student vided there is an opportunity for multiple learning is documented and assessed; measures of student development. programmatic flexibility to the changing conditions in the job market without compromise to the fundamental guiding Cooperative Education principles of cooperative education; at the University of Cincinnati • preparation of students for industry The University of Cincinnati is, in a global through coursework and other activi- perspective, one of few institutions to ties and institutional policies regarding combine high impact research with a strong supervision by the employing organi- professional profile. With a research budget zation as well as systems to document that exceeds $330 million and the largest student performance by the co-op and most diverse population of cooperative supervisor. education students at any public institution Since the introduction of the coop- in the United States, UC was in a unique erative education model a century ago, position to produce an innovative corpo- other forms of experiential education have rate feedback system for use in continuous emerged which encompass many of the improvement of curricula. The structure same principles. Often the debate over of the UC co-op program is based upon model can cloud the real issue of student full-time, alternating quarters of study and

Section 1 2 3 4 5 6 2

Section 1 2 3 4 5 6 1

F W S S F W S S F W S S F W S S F W S

Freshmen Sophomore Pre-Junior Junior Senior

n School Work Figure 1.1 Typical UC co-op curriculum.

Chapter 1 Getting Started 23 co-op work experience beginning in the adviser both prior to and after each work sophomore year and extending over three assignment in order to prepare for each years. This format makes the achievement assignment and reflect upon the learning of a bachelor’s degree at UC a five-year from the previous assignment. The Pro- endeavor. Students must complete a mini- fessional Practice faculty adviser plays an mum of four co-op quarters, although the important role as a liaison of the university majority of students complete six. Stu- with industry and monitors changes that dents are also required to remain with an seem to be of significant interest to faculty employer for a minimum of two quarters in the colleges. The close relationship with in order to provide a reasonable depth of the adviser helps the student to prepare for experience. the work term, and gives the institution Figure 1.1 shows a typical alternating quality feedback about the match between University of Cincinnati co-op curriculum. student preparation, aptitudes and aspira- The curriculum is based on two alternating tions, and employer needs. sections; sections 1 and section 2. When Prior to the FIPSE project, Professional section 1 is on work assignment, section Practice faculty advisers have been 2 is at school, and vice versa. This alterna- involved with the college curriculum tion paces the development of the student development by providing anecdotal evi- frame of reference with the progression of dence of the impact of the curriculum on the curriculum. Employers are able to hire student employability such as: co-op students for meaningful positions • In the architecture and interior design due to the alternating structure, as positions co-op program, the co-op advisers can be filled by students on a year-round basis. Every co-op work quarter is evaluated were unable to secure positions with through a three-party assessment process: technical experiences for students in by the student, by the employer, and by the their first co-op quarter because the faculty member. The Division of Profes- students had not mastered AutoCad. sional Practice (PP) is the centralized unit The school reacted to this informa- for the administration of all programs of tion by moving AutoCad earlier in the cooperative education offered by the Uni- curriculum. These students no longer versity of Cincinnati. Professional Practice spend their first co-op work term gath- faculty and staff work with approximately ering samples and doing research, but 4,000 co-op students annually and are are involved in actual projects as func- responsible for programs of cooperative tioning team members illustrating the education in four colleges: Applied Science strong connection between curriculum (a mandatory co-op program); Business (an and employability. optional co-op program); Design, Architec- • In the materials science engineering ture, Art, and Planning (a mandatory co-op curriculum, the course Introduction program); and Engineering (a mandatory to Metallurgy, was historically a part co-op program). of the sophomore curriculum and Co-op students are assigned to a Profes- co-op employers came to rely upon the sional Practice faculty adviser by discipline. understanding that students will gain This faculty adviser is responsible for all from that course. When the department aspects of the cooperative education pro- moved the course to the third year, sev- gram for their assigned disciplines. Students eral co-op employers complained that are required to meet with their faculty they were no longer able to use entry

24 Chapter 1 Getting Started level students therefore a few students interests, reflection exercises designed to lost their positions literally within the focus the students’ attention on the educa- first term the change took place illus- tional process of cooperative education, and trating the rapid reactions of industrial the evaluation of the students’ progression partners to changes in the curriculum. and growth in relation to the educational goals of the program. Another critical • In the mechanical engineering technol- aspect of teaching is the evaluation of the ogy program, supervisor feedback in the educational opportunities themselves. The mid 1990s indicated that approximately three party assessment process that the 20 percent of the employers requested division executes provides ample opportu- that students be trained in traditional nity for employers and students to evaluate drafting techniques. A simple analy- the educational value of the co-op positions. sis of this data indicated a curriculum In documenting teaching effectiveness it is change might be warranted however understood that the faculty member must a closer inspection revealed that this also demonstrate the validity of the learning employer group represented companies opportunities as part of the students’ overall that were behind the times, using anti- curriculum. Another important aspect quated methodologies with respect to of Professional Practice instruction is the design technology illustrating the need faculty members’ methodology in work- to filter the data appropriately. ing with students and employers which are These examples show the traditional judged against standards of good teaching. approach of cooperative education to moni- Faculty members in the degree grant- tor corporate feedback and make adjust- ing departments are further pivotal to the ments while the students are still enrolled success of cooperative education. They in the program. The downside of relying provide the classroom training that enables on anecdotal evidence is that it makes the students to effectively work in industry adaptation reactive rather than proactive. prior to graduation. While the shear vol- The corporate feedback system developed ume of placements makes it much more through the FIPSE project can be used to efficient for Professional Practice faculty identify discrepancies between curricular members to work exclusively on securing offerings and stakeholder needs before they positions within industry, those positions have produced obvious negative impact are based upon the overall goals set by the on an operational level. The recurring curriculum. On a regular basis, Profes- statistically-valid feedback process that this sional Practice faculty members attend project developed will be an invaluable asset faculty meetings of degree granting depart- for any institution that wants to develop its ments, and meet with department heads to operation in harmony with both academic provide input about the way students are criteria and employer demand. being received by industry. Over the years a The work of Professional Practice fac- partnership between these two faculty bod- ulty members at UC is evaluated through ies has developed in which each facilitates a the reappointment, promotion, and tenure portion of the cooperative education experi- process. To achieve tenure in the Division ence for the student. of Professional Practice, a faculty member While cooperative education has much must demonstrate excellence in teaching, to offer, the practical and applied nature of including individual meetings held with the educational methodology means that students to determine their aptitudes and co-op has been often overlooked by the

Chapter 1 Getting Started 25 Figure 1.2 Corporate Feedback Loop principle.

traditional academe. Cooperative education evolution also requires a sincere effort on requires an approach in which students are the part of faculty to systematically examine prepared for immediate corporate pur- student skill development throughout the suit, which can challenge a basic faculty educational process. The corporate feedback belief that colleges and universities provide system described in this publication is an education rather than training. Yet today’s effort to address this complex issue. world with its rapid changes make the issue even more complex in that neither education nor training has the longevity it once The UC Corporate Feedback System had. If higher education produces graduates In 2004 the University of Cincinnati who meet the demands of industry and are launched a project designed to expose therefore gainfully employed without com- faculty to outcomes based assessment data promising on the integrity of the academic of student work performance generated process and the imperative to educate for by co-op employers. The objective of the a lifetime, then we could achieve several admirable goals. We would be able to project was to build a program for continu- increase the immediate productivity of indi- ous improvement of curricula. The univer- viduals which would help to enhance the sity received a total of $555,133 (57 percent) competitives of industry. We could reduce from the U.S. Department of Education. inefficiencies in higher education thereby The matching contribution of the University alleviating some of the financial hard- of Cincinnati was $421,396 (43 percent), ships that are currently plaguing academia. making the grand total $976,529. Finally, we could reassure critics of higher The grant was filed by Associate Profes- education that the programs we offer match sors Cheryl Cates as PI and Kettil Ceder- industrial needs. Demands for greater creutz as Co-PI in a joint effort with the accountability require new ways of think- Accreditation Council for Cooperative ing about student development. Program Education (ACCE). Liaisons for the project

26 Chapter 1 Getting Started were Executive Director Tom Akins of the establishment of a student learning oriented Georgia Institute of Technology and Dr. partnership between the Division of Profes- Luther Epting of Mississippi State Univer- sional Practice and the UC co-op colleges; sity. The project was a collaborative effort by 3) the development of an enhanced research UC Professional Practice Faculty and faculty mindset within the Division of Professional members and team leaders at the UC col- Practice; and 4) the development of a new leges who pursue cooperative education. The mindset within the degree granting units objective of the grant was to build a closed in which they view the Division of Profes- loop system that measures student co-op sional Practice not only as a service unit work performance and directs this feedback that places students and evaluates learning, into curricular development. The project but also as an unit capable of partnering involved the Departments of Architecture with the departments on research related to (Team Leader: Prof. Anton Harfmann, Col- student learning. lege of Design, Art, Architecture and Plan- A first crucial step to the develop- ning), Civil Engineering (Team Leader: Prof. ment of a corporate feedback system was Richard Miller, College of Engineering), the creation of a vigorous online database Construction Management (Team Leader: that would capture work term assessment Prof. Benjamin Uwakweh, College of data. Historically, data had been captured Applied Science) and the College of Business in paper format which made any further (Team Leader: Prof. Marianne Lewis, Col- analysis cost prohibitive. The online data- lege of Business). The grant was internally base allowed employers to directly input assisted by the College of Education-Evalua- assessment data into a database that could tion Services Center (Dr. Catherine Maltbie) provide subsets of data based upon specified and externally assessed by the Evaluation parameters. These data subsets would then and Assessment Center for Mathematics and be used to conduct further analysis related Science Education at Miami University (Dr. to student development. Chapter two fur- Amy Cassedy). Including matching funds ther describes the online assessment system the University of Cincinnati project invested and the instruments used in the project. $0.976M in industry-integrated curriculum Next, the development of a statistically development. sound methodology to view co-op employer The project relied on methodologies as evaluations within the context of student follows: 1) the creation of a process to sys- learning would be critical to the success of tematically examine employer input gath- the project. Multiple cycles of curricular ered through individual evaluations of co-op reform would maximize opportunities for students as an aggregate; 2) the development success. In the final stages of the project of additional assessment instruments that the creation of a reporting system that would be used for a specific period of time would allow reports on student learning and targeted toward a particular issue of to be accessible to the departments would importance; and 3) fine tuning of data gath- become the focus. This would not only ered through the online assessment database institutionalize the project but also demon- through employer focus groups. strate to a broad audience the potential for To implement the above methodolo- cooperative education/experiential learning gies: 1) the creation of a steering commit- to bring industrial input to college cam- tee of committed individuals who would puses in a way that had not been previously serve as champions for the project within considered. Chapter three further discusses their own departments and colleges; 2) the the generation of effective reports.

Chapter 1 Getting Started 27 Is There a Need for a is further set on building an inclusive, Corporate Feedback System? convincing, energizing and value-adding process. Despite a vast number of publi- Continuous improvement has grown to cations, the literature is relatively coher- become the standard operating procedure ent. Good assessment can be defined as in the production of goods and services being goal oriented and stemming from th during the latter half of the 20 century. the educational values of the institution. Since the mid 1990s the academe has Assessment should be focused not only to moved towards implementing continu- outcomes but also to the experiences that ous improvement programs to enhance lead to those outcomes Achievements need the development and delivery of curricula. to be compared to goals. Measurements The call for increased productivity and should be verifiable on a repeatable scale. accountability by legislators and accredita- Assessment information that gives answers tion agencies has been heard, and the focus to questions people really care about will of individual institutions has moved from be valued and therefore used. Given the monitoring instructional content, to mea- current economic climate only a cost effec- suring learning outcomes. The academe tive process can withstand the competition is in short moving from a content driven, for resources. Good assessment involves to an outcomes driven paradigm. Using a direct evidence of student learning so vocabulary coined in the manufacturing exhibited through student performance. industry, higher education is moving from Direct evidence focuses on the actual out- push to pull control. comes of student performance. The stron- Learning outcomes can be assessed gest type of direct evidence is also blind using a number of methodologies. As the and contextual in that it is pursued in a mission of individual institutions differ so professional rather than campus environ- does the methodology to most effectively ment and under the independent guidance assess student learning. The methodol- of those external to the teaching process. ogy described in this publication is par- In other words, analyzing the result of ticularly applicable to institutions with a actual work rather than analyzing student strong focus on Work Integrated Learning scores on an exam would be direct, blind (WIL). The presented methodology has and contextural assessment. Good assess- been developed in a cooperative education ment begins with the institutional values, environment, but can be transferred to has a strong conceptual framework and other WIL-environments that rely on the institutional support, provides cost effec- assessment of student work performance tive feedback and does ultimately lead to by onsite supervisors. improvement. Academia is challenged with develop- Good assessment is performed in rela- ing a set of best practices with regard to tion to the goals an institution sets for its outcomes assessment. Towards that end, performance. Furthermore, setting goals is central publications in the field of con- an art, not a science. It is important to rec- tinuous improvement in higher education ognize that ambiguity in the goal setting show a number of general trends. Authors discussion is a sign of a healthy academic tend to describe continuous improve- environment. Educational institutions, ment as a process that involves a cycle of having a remarkably different history and assessment, feedback, adjustment, imple- culture than their industrial counterparts, mentation, and reassessments. The focus are faced with significant challenges when

28 Chapter 1 Getting Started embarking on the road of continuous needs. In addition to the UC FIPSE project, improvement. These institutions are pres- the most extensive research in the field of ently working to come to grips with how to co-op student work performance has been implement continuous improvement in a pursued at Georgia Institute of Technology culture characterized by academic freedom and Iowa State University. Both institutions and joint governance. have incorporated some level of statistical analysis of co-op student work performance in their standard operating procedures. The Opportunity for Transformational research pursued at Iowa State University Change via Corporate Feedback includes an extensive mapping of the rela- Since the early 1990s, outcomes-based tionship between co-op outcomes and cur- accreditation has become the central para- ricular input. The research shows that these digm in accreditation criteria. As an exam- curricular maps are very effective when ple, the Accreditation Board for Engineer- trying to find causalities between curricular ing and Technology (ABET) developed their structure and student work performance. Criteria 2000 to reflect a market-driven The experience at Iowa State further shows paradigm as the consensus had been grow- the value of using web-based software for ing among engineering educators regarding the collection of a large amount of infor- the value of deriving curricular offerings mation. Cooperative Education is further from industrial needs. The same transfor- seen as a unique contextual environment mation to an outcomes-based paradigm is for the assessment of student performance further reflected in the revised criteria of a in a professional setting. Research at the large number of regional accrediting bod- Georgia Institute of Technology demon- ies recognized by the Council for Higher strates a strong, at times almost linear Education Accreditation (CHEA). Regional correlation between work performance accreditation bodies report that they are and student curricular progression. The increasingly starting to collect direct evi- research further shows the importance of a dence of student learning outcomes as part positive trusting institutional atmosphere of their judgments about quality or accredi- when trying to implement employment tation status. All six regional accrediting based curricular improvement. At the bodies and most professional accrediting University of Cincinnati research shows bodies now require comprehensive assess- that a cooperative education environment ment in both public and private institutions. per se already constitutes an educational Most accrediting bodies have in essence system that gears curricular content instructed institutions to define program- towards meeting industrial needs. The matic outcomes for instruction and verify research demonstrates both conceptually that students achieve these outcomes. and practically that parameter averages Accrediting bodies have moved from a and standard deviations of cooperative prescriptive to an outcomes-based approach education assessment data can be used to allowing academic institutions to develop get an appreciation of which educational assessment systems that put their academic processes are stable, and which require offerings in relation to stakeholder needs. further improvement. The situation has opened an opportunity for The collection of such information, institutions that offer large-scale coopera- along with subsequent analysis and use of tive education programs to analyze student information is critical to the introduction work performance and link it to stakeholder of transformational change and can also

Chapter 1 Getting Started 29 focus faculty on the concept of continuous tinuous improvement are likely to find improvement in a new way. Seeing educa- themselves left behind. Societal pressures tion as a value adding process is a model for accountability, combined with emerg- under which the concept of continuous ing assessment policies and practices, pro- improvement in higher education can ulti- vide and opportunity for transformational mately be successful. Faculty members are change. This publication gives an in depth adept at collecting and synthesizing infor- look at one transformational change effort mation and while it may be that faculty as it strives to provide a foundation for members are continuously improving their replication at other institutions of higher classroom performance most have not had education with similar infrastructure. access to the type of feedback produced by corporate feedback system. This feedback could become a significant new source of Research Team Design information for faculty members to use An important consideration that any insti- in the continuous improvement of their tution must place at the top of the list is the teaching. creation of the team that has the potential While much of common wisdom to enact transformational change on its and understanding related to continu- own campus. ous improvement in an industrial setting The Corporate Feedback System clearly is directly applicable to higher education addresses society’s need for higher edu- there are several aspects that are unique to cation to become more responsive to its academic organizations. The academe con- needs in terms of addressing student learn- tributes in elemental, all encompassing and ing and workforce preparation. The proj- permanent ways to the personal and pro- ect’s principle investigators recognized that fessional lives of its students and graduates. faculty members are the key constituents The academe also contributes to society as it is only through behavioral change and to the community through contribut- in faculty and administration that these ing a unique cultural and scholarly vital- needs can be addressed. Departments ity. On an economic level, the academe is invited to participate in the project exhib- becoming increasingly significant as both ited characteristics as follows: a large employer and an economic driver by capitalizing on its potential for innova- • The departments were actively engaged tion and discovery. In a capitalistic society with their employer base outside of the the idea that an organization exists to classroom. create a profit for its shareholders is clearly • The departments, as a part of their cul- understood. The mission of institutions ture, had both a desire to enhance their of higher education to expand and dis- students’ education and a track record seminate knowledge, does however make it of effort in this area. difficult to directly implement paradigms developed in the corporate world in an • The departments, as were looked at academic environment. within the colleges as leaders such that a successful result would likely be rep- Higher education institutions today, licated within other departments. big or small, private or public, do however need to adopt a continuous improvement • Key faculty in the department had strategy to deal with the rapid rate of previously been involved with efforts to change. Those that do not embrace con- enhance teaching and learning.

30 Chapter 1 Getting Started • The culture within the department at hand and engage in seasoned discussion was seen as supportive of the project. about team goals and individual roles on the team. The performing stage occurs as • The departments were strong and well goals and structures are securely in place respected, and would therefore not be and the work of the group proceeds full intimidated by a thorough examination steam ahead. The final stage of adjourning of their teaching efforts, nor by system- occurs as the group evaluates their work, atic employer input into their teaching. gives each other feedback and, if the group • The departments had high placement is ongoing, determines the frequency of rates and a positive attitude towards such reviews. cooperative education. Within the project organization a cross Once the PIs had a clear idea of the functional team design was used to allow type of department that would likely maximum learning to take place. Research be a successful participant of the proj- team members were selected based upon ect departments were invited to join the their knowledge and specialized skills group. Each department was further asked related to the project objectives with the to select a faculty member to serve as a goal of creating cross functional syner- team leader with regard to feedback and gies. The research team was composed so curricular revisions. Only through strong that members had complementary skills faculty leadership could the goals of the that promote learning from one another. corporate feedback system have a realistic Mutual accountability within the team probability of success. itself evolved from trust, understanding, and shared vision. Team members were As with any team, the University of equally as important as project leaders. Cincinnati research team dedicated to Project leaders were expected to get the the creation of a corporate feedback sys- team members involved in problem solv- tem went through the same type of team ing, to help the team develop boundaries development as described in the litera- and outcomes, to resolve conflicts and ture. Team development is a predictable manage interpersonal aspects of the team, sequence of transitions based upon group and to clarify roles, build trust and estab- member characteristics, group charac- lish expectations for the team members. teristics, and situational characteristics. Research team members were expected As group members become integrated as to contribute their unique understanding a team they follow behavioral patterns of the various aspects of the educational including forming, storming, norm- system that envelops students in higher ing, performing and adjourning. In the education. As with all team formation, the first stage of forming members may feel effective formation of the cross-functional anxious but are usually polite and atten- research team required: tive and focus on the group’s leaders. The second stage of storming occurs as the • A clear problem or issue that can focus group begins to feel frustrated by depen- the team and define its boundaries. dency on their leader and tries to clarify • A champion for the formation of the roles as independent of the group’s leader. cross functional team to provide senior In the norming stage the group begins to level expertise and guide the team develop cohesiveness through improved through the political path of the communications as they focus on the task institution.

Chapter 1 Getting Started 31 • Team members who can represent the continuous improvement concept. Yet views of all stakeholders and are directly despite its hesitancy to adopt the philoso- involved with the issue or problem. phy, higher education has tremendous • Meeting space, time, location and team capability to become a highly effective size that is conducive for the mission of learning organization by putting into the group. practice many of its unique strengths in research and reflection, and turning the • A project time line that allows team lens inward that has been so effective at members to examine the situation and understanding other parts of society. discuss the issues in a collegial man- Successful transformational change ner in order to develop a plan that will requires, besides an investment in time be achievable within the institution’s and resources, a core group of people culture. who are committed to the cause. Suc- It is typically wise to provide the team cessful academic change initiatives are with a written charter that defines the always connected with real goals, com- scope of the project. A charter provides mitted to improving performance, involve the team with legitimacy on campus and people with power to take action, balance the ability to collect data, conduct survey’s inquiry and experimentation, provide or take other action required to address the opportunity to reflect without pres- the various problems. Within the context sure to make decisions, and focus on of the University of Cincinnati team this learning about learning. Those involved was not in question given the significant in the change must see direct personal legitimacy the FIPSE grant. However, for benefit to reinforce sustained change. others who undertake similar activities Their subsequent enthusiasm leads to an without the benefit of external funding, investment in the change and a desire to this type of written charter will certainly further capabilities. Beyond individual be beneficial. Even with the scope of the results, a team of committed people must project well defined by the grant proposal, move a change forward through their it is a struggle not to allow mission creep interactions with others outside the team to influence the development of the system. itself. Informal networks play a key role in organizational change as they diffuse innovations throughout the organization. Considerations in Creation of These informal networks are superior to the Corporate Feedback System organized channels in that they already The creation of a Corporate Feedback exist, they are essential for daily work and System challenges the status quo on cam- the information that passes through them pus. The role of power within a transfor- has instant credibility. This grass roots mational change such as this should not approach to change is the second critical be ignored. Power and politics exert a component in generating transforma- profound influence over organizational tional change. The final component is that change. Both cooperation and competi- of business results. As new business prac- tion can play an important role as the tices lead to better results, more people needs of the organization or groups are become willing to invest their time, balanced with individual needs and energy and resources into creating similar interests. Today higher education is one of practices. All three, personal results, busi- few industries that has not fully embraced ness results, and networks of committed

32 Chapter 1 Getting Started people are the necessary ingredients for generating transformational change. For all of the positive benefits that a corporate feedback system can create on a university campus with a strong experiential learning component and culture, it is important to recognize it’s potential pitfalls as well. While the data provided by the system is value neutral the impact that the data could have will not be. Chapter three examines the potential challenges the data could receive and provides strong advice on how to address those challenges. Still for any group embarking on this journey, it is wise to take great care in forming the team and in enlisting the support of university champions as a first strategic step in the process. Creating a pilot group to test the concept is perhaps the most important step. Success within that pilot group can lead to incremental advances within the larger campus community with potentially less conflict than any large scale implementation. Multiple simultaneous projects within the pilot group can also significantly increase the chances for success by providing an opportunity for a mix of methodologies and enhanced abilities for experimentation. The University of Cincinnati pilot hosted three distinct college specific projects. Each project was a variation on the concept of the corporate feedback system. The individual projects are covered in detail in chapters four, five, and six of this publication. With all the human capital in place and the planning stage completed, the next critical step that the project pursued was the collection of the data. The next chapter describes the online assessment database that the University of Cincinnati developed in conjunction with the FIPSE project as well as the three assessment instruments utilized in the development of the corporate feedback system.

Chapter 1 Getting Started 33 Chapter Two Managing Multiple Levels of Assessment By Cheryl Cates, Kettil Cedercreutz, Anita Todd, Zach Osborne, and Louis Von Eye

“Technology is not about tools, it deals with how Man works.” Peter Drucker, 1970

Overview of Assessment Structure • Assessment Instrument II is primarily quantitative in nature, focusing on spe- The assessment principle of the FIPSE project “Developing a Corporate Feed Back cific problem areas indicated by Assess- Loop for Curricular Reform” was designed ment Instrument I. Assessment Instru- to rely on three layers of assessment as pre- ment II will allow the research team to sented in Figure 2.1. examine details of curricular performance. Being a tool that allows tailored The assessment instruments corre- programmatically-oriented questions, sponding to the different assessment levels have characteristics as follows: the instrument supports detailed assessment of specific curricular aspects sub- • The standard instrument used in the ject to scrutiny. Assessment Instrument three-party assessments by employers to II is an integral but flexible appendix to evaluate students is defined as Assess- Assessment Instrument I. ment Instrument I. The instrument is well-established, and has a long history • Assessment Instrument III is specified of application that has resulted in a as a qualitative tool designed to refine wealth of longitudinal data. It is primar- and clarify questions raised through ily quantitative in nature. Assessment Instruments I and II.

Figure 2.1 Levels of assessment in the UC FIPSE Project Developing a Corporate Feedback Loop for Curricular Reform.

34 Chapter 2 Managing Multiple Levels of Assessment The focus group approach will help the The Employer Component regulates research team propose solutions based employer interaction with the system. on underlying needs. The employer component has two parts, Assessment Instruments I and II are one for human resources and one for conveniently delivered in a web-based for- supervisor level contact information. mat, whereas Assessment Instrument III, The employer component maintains a for obvious reasons relies on face to face historical record of all students that have contacts between faculty, students, and worked directly for the particular super- employers. visor. The system allows the supervisor to view and revisit a student’s completed learning assignments and learning Professional Assessment objectives, and complete all related and Learning System (PAL) employer assessments. As the university The Professional Assessment and Learn- encourages honest student feedback ing System (PAL), was developed at UC in supervisors and HR personnel are not parallel with the FIPSE project. The PAL able to view students’ confidential com- system consists of five interconnected ments related to the co-op work assign- components: a student component, an ment. The HR level contact can view employer component, a division compo- the assessment provided by the supervi- nent, a college component, and an admin- sor, and view their placement history, istration component. These five compo- update job descriptions, and alert UC of nents feed data into and retrieve data from impending employment needs. one central assessment database. The Division Component regulates The Student Component regulates co-op faculty advisor interaction with student interaction with the system. the system. This component allows Each quarter students are required to co-op faculty to download assess- register their placement in PAL. The ment data, and to view data submitted registration serves the purpose of col- by both students and employers. The lecting important student placement component provides communication data, registering the student with the functionality for faculty and staff to university for their co-op quarter. The contact specific groups of students and registration further enables the stu- employers. Faculty and staff can utilize dent to download placement related system functionality for reminders, information and receive placement progress updates, et cetera. The system specific instructions. The system gives has a strong reporting capability that the student the ability to review, save, enables faculty members to view reports edit, and submit a complete assessment of aggregate student learning data. of the work term. The student compo- The Administration Component of the nent keeps a historical record, allowing online assessment system allows a sys- students to revisit and modify (within tem administrator to manipulate system specified time limits) their co-op place- data and add or modify student data, ment record. The application for entry placement data, employer data, learning into the University of Cincinnati co-op module and question data, support data, program is further managed by the and to send and edit emails automati- student component. cally managed by the system.

Chapter 2 Managing Multiple Levels of Assessment 35 Figure 2.2 Professional Assessment and Learning system block chart.

The College Component allows college Assessment Instruments and administrators and faculty members to Evolution of Co-op Assessment interact with the system. This component enables downloads of aggregate Cooperative education and internship pro- assessment data for further analysis. grams have historically collected employer The system also has a strong reporting and student performance evaluation data capability that enables faculty members each term. These evaluations have been to view reports of aggregate student used for reflection and to improve stu- assessment data. dent or employer performance. The data has however not typically been used in a The PAL Database relies on building formal way to provide feedback to depart- blocks as follows: ments and colleges relative to educational • Programming language & platform: goals, nor were the assessment tools Visual Basic Script & Active Server designed to do so. Pages (ASP) Quality co-op and internship programs • Database platform: Microsoft SQL work closely with employers soliciting Server 2000 their evaluations on a term-by-term basis throughout the student’s academic career. • Reporting platform: Microsoft SQL The University of Cincinnati, along with a Server 2005 Reporting Services number of institutions engaged in experi- The system is being developed for off- ential learning, developed new assessment site implementation. Figure 2.2 provides an tools focusing on learning outcomes in the overview of the PAL system structure. late 1990’s. Developing feasible assessment

36 Chapter 2 Managing Multiple Levels of Assessment instruments was, however, only a first step back from students, employers, and Pro- in closing the loop. Until 2003, these evalu- fessional Practice faculty advisors forms a ations had been gathered on paper, making comprehensive feedback loop. Many ques- the task of analyzing and summarizing tions that in Assessment Instrument I are feedback for the departments and colleges completed by the employer are mirrored in very time consuming. The conversion from the student and faculty instruments allow- paper to a web-based online data collection ing the university to capture feedback from system has greatly improved the ability multiple perspectives. to quickly collect, compile, analyze, and PAL is a comprehensive system that distribute data. not only incorporates Assessment Instru- At the University of Cincinnati stu- ment I and II but also learning modules, dents are required to complete a set of communication functions etc. The FIPSE assessment instruments that include project focused only on Assessment learning objectives, a learning module, Instruments I and II which generated the and an evaluation for each work assign- data used in the developed feedback loop. ment. Employers are asked to evaluate student achievement as well as the student’s understanding of a specifically assigned Assessment Instrument I: learning module. Employers assess student The Standard UC Co-op Work skill development based upon the work Performance Assessment Instrument performed during the quarter. Profes- The primary assessment instrument used sional Practice faculty advisors evaluate in the FIPSE study was Assessment Instru- student learning as demonstrated by the ment I, focusing on employer assessment learning module and review and discuss of Professional Practice students. The ques- the objectives and evaluations with the tions on the instrument, presented in Table students in individual student meetings. 2.1 (on the following page), were developed Students receive a passing grade for coop- by Division of Professional Practice fac- erative education if the work experience ulty. In the development of this instru- and the required assignments are satisfac- ment, faculty consulted with the relevant torily completed. This grade gets recorded research on the types of skills that students on the student’s transcript. require to be successful in the 21st century; The co-op related assessment instru- the various criteria of the accrediting bod- ments used by the University of Cincin- ies under which the degree programs oper- nati were developed using the criteria of ate; and the general education goals of the accreditation bodies to which colleges institution. In addition faculty consulted must adhere as well as the university’s with employers regarding their company general education goals. The main instru- specific requirements for student skill ment for the FIPSE project is Assessment development. As the instrument evolved, Instrument I, which is described in more a series of focus groups involving of both detail below. Assessment Instrument I was students and employers were used to refine designed to be used in co-op programs the final layout. In chapter three, you will offered by the College of Engineering, find a discussion regarding the reliability College of Applied Science, College of of the instrument. A more comprehensive Business and the College of Design, Archi- layout of the instrument, together with all tecture, Art and Planning. The three-party qualitative parameters, can be found in assessment system which includes feed- Appendix 1.

Chapter 2 Managing Multiple Levels of Assessment 37 Table 2.1 Assessment Instrument I. Grading Scale; 1=Unsatisfactory; 2=Poor; 3=Satisfactory; 4=Good; and 5=Excellent

Category Question Coding A COMMUNICATION Speaks with clarity and confidence A1 Speaking Writes clearly and concisely A2 Writing Makes effective presentations A3 Presenting Exhibits good listening and questioning skills A4 Listening B CONCEPTUAL AND ALYTICAL ABILITY Evaluates situations effectively B1 Evaluates Situations Solves problems/makes decisions B2 Problem Solving Demonstrates original and creative thinking B3 Creative Thinking Identifies and suggests new ideas B4 Idea Generation C LEARNING/THEORY AND PRACTICE Learns new material quickly C1 Learning Accesses and applies specialized knowledge C2 Appl. Spec. Knwldg Applies classroom learning to work situations C3 Appl. Classrm Lrnng D PROFESSIONAL QUALITIES Assumes responsibility/accountable for actions D1 Accountability Exhibits self-confidence D2 Self Confidence Possesses honesty/integrity/personal ethics D3 Integrity Shows initiative/is self-motivated D4 Self Motivation Demonstrates a positive attitude toward change D5 Positive Attitude E TEAM WORK Works effectively with others E1 Works with Others Understands and contributes to the organization’s goals E2 Goal orientation Demonstrates flexibility/adaptability E3 Flexibility Functions well on multidisciplinary team E4 Multidiscipl. Team Fct F LEADERSHIP Gives direction, guidance and training F1 Gives Direction Motivates others to succeed F2 Motivates Others Manages conflict effectively F3 Conflict Management G TECHNOLOGY Uses technology, tools, instruments and information G1 Use of Technology Understands complex systems and their interrelationships G2 Systems Understanding Understands the technology of the discipline G3 Underst. Of Tchnlgy H DESIGN & EXPERIMENTAL SKILLS Displays ability to design a component, system or process H1 Comp. Design Ability Demonstrates ability to design and conduct experiments H2 Exprmnt Design Ability Analyzes and interprets data efficiently H3 Data Analysis Ability I WORK CULTURE Understands and works within the culture of the group I1 Work Culture Und. Respects diversity I2 Respects Diversity Recognizes political and social implications of actions I3 Rec. Political Impl. J ORGANIZATION PLANNING Manages projects and/or other resources effectively J1 Project Management Sets goals and prioritizes J2 Goal Setting Manages several tasks at once J3 Task Management Allocates time to meet deadlines J4 Time Allocation K EVALUATION OF WORK HABITS Professional attitude toward work assigned K1 Professional Attitude Quality of work produced K2 Work Quality Volume of work produced K3 Work Volume Attendance K4 Attendance Punctuality K5 Punctuality

38 Chapter 2 Managing Multiple Levels of Assessment Assessment Instrument II: 3. Understands the complexity of detailing. A Tailored Assessment Instrument 4. Understands how modeling and In conjunction with Assessment Instru- drawing translate into a structure. ment I, the primary base instrument for the identification of program strengths and weaknesses, the project provided the AI II; CIVIL AND opportunity for program specific teams ENVIRONMENTAL ENGINEERING to hone in on specific skills that were not Questions related to registered by this standard instrument. Fundamental Principles of Structures Assessment instrument II was developed Is it important for co-op students at your as a second assessment instrument for company to understand the following fun- this purpose. The introduction of the damental principles of structure? online assessment system allowed for a flexible introduction of a second layer of 1. Bending stresses Yes No performance measurement to address the 2. Shear stresses Yes No needs of specific programs in curriculum 3. Combined stress Yes No revision situations. The questions used in Assessment Instrument II were developed 4. Eccentric loading Yes No by the various departmental teams to elicit 5. Column buckling Yes No information that would be particularly 6. Determinate structures Yes No effective in their curricular reform efforts. These questions were primarily quantita- 7. Indeterminate structures Yes No tive in nature; however, a one program 8. Influence line Yes No opted in addition to use qualitative mea- In case of an affirmative response, the sures. The research team limited Assess- system further prompted co-op employers ment Instrument II to the measurement of to rank (using a five demarcation Likert no more than 10 parameters to minimize scale) the performance of their current the chances that employers would discon- co-op student’s knowledge of the specific tinue participation in assessment activities principle. due to a significant increase in workload. The different program teams elected to introduce a variety of assessment Instru- AI II; COLLEGE OF BUSINESS ment IIs presented below as examples: Questions related to Program Subject Matter Development AI II; ARCHITECTURE 1. What specific knowledge or skills would Assessment Instrument II measures you want any Information Systems four constructs on a five demarcation co-op student to possess before their first Likert scale; 1=Unsatisfactory; 2=Poor; co-op work term with your company? 3=Satisfactory; 4=Good; and 5=Excellent 2. What specific knowledge or skills would Building Construction you want any Accounting co-op student to possess before their first co-op work 1. Effectively uses digital technologies to term with your company? represent building construction. The three different Assessments Instru- 2. Understands the complexity of ment IIs reflect the diversity of the aims of building construction. the investigators. In the example of archi-

Chapter 2 Managing Multiple Levels of Assessment 39 tecture, the investigators simply monitor understand when curricular changes are student performance levels. In the case of made that may directly influence assess- civil and environmental engineering, the ment data. investigators are interested in both the Success in closing the loop is equally relevance of the subject matter, as well as dependent on the underlying technology the proficiency of the students. In the case as it is dependent on human factors. As of the College of Business, investigators are assessment data will reveal both positive asking employers for guidance with regard and negative issues, it is important that the to adding subject matter to their curricula. messenger be well prepared to respond to the critique often associated with negative Assessment Instrument III; Focus Groups messages. The assessment feedback must be presented in an atmosphere of dialogue Received assessment data is only as good and trust. Co-op practitioners should be as the measured parameters. Whenever involved in the loop, as the relevance of investigators omit to ask the right ques- any data is closely associated with the rel- tions, the data will be a poor guide for cur- evance of the co-op positions themselves. ricular revisions. Qualitative and quantita- All parties need to have a positive attitude tive data need always to be accompanied with regard to critique. The system mea- with, various focus groups. The UC FIPSE sures a fit between employment and educa- grant used focus groups to validate instru- tion. The co-op practitioners must be ready ments, and to develop questions for Assess- to engage in a discussion that questions ment Instrument IIs. The focus groups will the relevance of offered job opportunities. be discussed in conjunction with program Academic faculty must be ready to engage specific cases. in a discussion of student skill development as a function of the curriculum. The assessment data will inevitably lead to Closing the Loop upgrades of all functions associated with With the development of the new assess- the feedback loop. ment systems, it is possible to provide Once trust and dialogue are in place, more comprehensive assessment data in the organization can focus on designing a user-friendly format to the academic reports that are appropriate under a vari- departments. Since departments must ety of circumstances. The structure of the report to accrediting bodies they will reports, as well as a thorough understand- benefit greatly from information on their ing of the uncertainty of the data, will be students’ participation in co-op and discussed in chapter three. An important internship courses, and information on finding of the research team is that a feed- students’ and employers’ perceptions of back system can only be effective when- students’ academic preparedness. Poten- ever it becomes integrated in the standard tially, academic departments could be operating procedures of the university. provided with real time data not only on Once the administration becomes aware an individual student’s skill development of the power of the data, this is typically but also on the skill development of an not a difficult sell. There will be a pull to entire class of students. A closer partner- institutionalize the system, however until ship and knowledge sharing relation- such time; the development of a corporate ship between co-op offices and academic feedback loop must be spearheaded by a departments could help both parties core group of academic enthusiasts.

40 Chapter 2 Managing Multiple Levels of Assessment Chapter Three Generating Effective Reports By Kettil Cedercreutz, Cheryl Cates, and Catherine Maltbie

“Gutta cavat lapidem, non vi, sed saepe cadendo.” The drop cavitates the stone, not by force but by frequent dropping. — Latin Proverb

Let us assume that we are administering a A Mean/Standard Deviation Matrix mature co-op program. We have an assess- is a plot of all scores of a specific popula- ment instrument that has been developed tion in a mean standard deviation plane. A over a number of years that we are reason- low standard deviation of the performance ably happy with. We have implemented a score, implies a low variation in student web based system that gathers quantitative performance which suggests a high stabil- data on student performance, and we have ity of the educational process. Conversely, run it for a couple of years. Where do we go a high standard deviation of a particular from here? How do we close the loop in an score indicates a high variation in student encouraging manner that motivates faculty performance suggesting that the educa- responsible for classroom instruction to use tional process is unstable. In the latter this data to the fullest? case students seem to be left to their own devices; some learn and some do not. High means are typically preferable, and vice Reporting Tools versa. The matrix is visual and simple to It is obvious that all reporting requires read. Scores in the upper left hand corner effective tools. How do we present mea- tend to indicate a quality process whereas surement results of tools such as Assess- scores in the lower right hand corner need, ment Instrument I, presented in Table in general, to be avoided. 3.1? The six sigma literature provides an The MSM gives a good picture of where ample roster of methods that can be used the educational process is stable and where to report statistical data effectively. After further improvements are needed. Six an extensive search of the literature, the Sigma and Lean Manufacturing literature research team found Mean/Standard Devi- typically encourage process developers to ation Matrixes (MSM) and Delta Mean/ initially focus on limiting the standard Standard Deviation Matrixes (∆MSM) to deviation of a particular process outcome. be a very effective way to communicate Only after the control of a process has been outcome levels and process stability with obtained can the means of process out- various stakeholders. The Principle of a comes be enhanced. Mean/Standard Deviation Matrix (MSM) MSM’s describe a process in relation to is schematically presented in figure 3.1. It the assessment of a specific group of asses- gives a snapshot of the level and diversity sors. They show a picture of a process at a of the performance of a student cohort specific state of development. ∆MSM’s are parameter by parameter. used to compare the development of a spe-

Chapter 3 Generating Effective Reports 41 Standard Deviation

Low Medium High High Mean Medium Low

Figure 3.1 Mean/Standard Deviation Matrix (MSM).

cific student cohort over the course of the reasonable to hypothesize that the learn- curriculum, and are valuable instruments ing in this case is not well-supported by especially in before and after studies. Fig- the curriculum. The curriculum might ure 3.2 shows the principles of a ∆MSM. provide opportunity for learning, but all A ∆MSM gives valuable information students seem not to be able to tap into the about the effectiveness of an educational opportunity. Quadrants three and four process. The effectiveness of a ∆MSM can are increasingly more problematic. When- be demonstrated in an example compar- ever the mean decreases and the standard ing the performance of sophomores and deviation increases (quadrant three), seniors. In an ideal case the results are students are surreptitiously exhibiting a plotted in quadrant one of the ∆MSM. deterioration in performance. Whenever Whenever the mean of student perfor- both the mean and the standard deviation mance increases and the standard devia- decrease, the program is systematically tion decreases as a function of curricular contributing to the deterioration of student progression, the educational program performance. Fortunately very few pro- is doing what it is supposed to do. The grams exhibit this kind of behavior. average of student performance increases Besides MSM’s and ∆MSM’s conven- and the students exhibit this behavior tional tables obviously have their niche uniformly as a group. When both the to fill in data reporting, and the graphi- mean and the standard deviation increase cal representations should in most cases (results plotted in quadrant two), learning be complemented with tables detaining is occurring, but some students exhibit a sample sizes, assessor numbers, means and better performance than others. It seems standard deviations.

42 Chapter 3 Generating Effective Reports Figure 3.2 Delta means standard daviation matrix (∆MSM).

Frequent Questions 7. What statistical methodology do you use to compare cohort means? Life would be very easy, if all we had to do was to produce MSM’s ∆MSM’s and 8. How do you compare the results in the corresponding tables. The UC FIPSE differences in standard deviations? project however shows that this is just the 9. Does the Likert scale of 1 to 5 have beginning. The experience of the research enough resolution? team was that a positive message is very easy to deliver, whereas a negative mas- 10. What compromises do you have to sage instantly causes an (often emotionally make when using rolling averages? driven) attack on the validity of the data. 11. Do you have sample sizes large enough Before you engage in systematically giving to be able to demonstrate statistically feedback to departments make sure that significant results? you have done your homework. The most 12. What are the pros and cons with common questions from doubtful listeners longitudinal data? that we experienced are listed below: 13. How do you account for transfer 1. Is the instrument ambiguous? Are we students, students switching majors sure that all assessors understand the and dropouts? questions the same way? 14. How do you make the connection 2. Whose perspective does the data reflect? between the curriculum and work 3. Can you compare majors to one performance? another based on the data? 15. Do you have the right employers? 4. What traits can be measured? 16. Does your measurement instrument 5. How do you account for some assessors demonstrate sufficient internal being lenient in their grading, and consistency? others being tough? 17. Can you group your measurement 6. What kind of models do you use to results more effectively? incorporate assessor bias into measure- 18. Are there any general trends with ment results? regard to cause and effect?

Chapter 3 Generating Effective Reports 43 The reminder of this chapter holds Is the instrument ambiguous? answers to the questions above. Once you Are we sure that all assessors understand have answers to these questions, you will the students the same way? be able to successfully address most audi- In many cases assessment instruments have ences. The definitions and abbreviations been developed for one-on-one interaction found in Table 3.2 will be used in answer- with students. In this environment they ing these 18 questions. have been tested and found effective. Can we from here draw the conclusion that the Table 3.2 Central parameter definitions questions will work equally well for mea- and abbreviations. suring the performance of an entire group? Reading The actual measurement recorded by one The obvious answer is that we can not. If assessor. a question is worded “Speaks with clarity Signal The value of the actual property being and confidence,” an advisor may be able measured; the true performance of the to know, after a one-on-one session with a student before the aggregation of assessor student whether a low rating is caused by uncertainty. low confidence or low clarity. In a statistical Noise Random error / type A error. setting we will not be able to see that level of differentiation. In a case like this we still Bias Systematic error/type B error. have to make clear for ourselves whether we S Standard deviation. can be happy with only knowing the aggre-

SREAD Standard deviation of reading. gate, or whether we need to split the ques- S Standard deviation of signal. tion in two, measuring both dimensions. SIGN The validation of the instrument can typi- S Standard deviation of systematic error. SYST cally be done by experts or focus groups. In SRAND Standard deviation of random error. the case of the FIPSE project we subjected 1 and 2 Under indexes for group differentiation. all instruments to expert group validation, U Uncertainty. and to focus groups involving students and employers. The objective was to analyze U Statistical Uncertainty STAT to what extent the questions measured a (Other indexes same as for S). commonly agreeable behavior pattern. A Average reading. We further subjected the instrument to a ∆A Difference between average readings. statistical analysis of relevance which was k Amount of assessors. tested by the UC Evaluation Services Center using a web-based survey that ran for one V, V , V ,… Assessors filing 1,2,3,… assessments each. I II III quarter in conjunction with the regular kEFFECT Effective assessor amount. co-op assessment cycle. Five hundred and n Amount of returned assessments. four employers returned the survey, which N Total amount of students. corresponds to an approximate return rate of 30 percent. Table 3.3 shows the results m Amount of measurements. of the survey for the population at large, as t t-factor for specific confidence level and specific well as for one specific program, Civil and degree of freedom. Environmental Engineering (CEE). The F F-factor for specific confidence level and specific CEE data is based on 26 returns. The table degree of freedom. shows that 97.5 percent of the parameters v Percentage of unique assessors not shared by are deemed important by the majority of two samples. supervisors at large. The corresponding

44 Chapter 3 Generating Effective Reports Table 3.3 Percentage of supervisor population that consider a specific parameter to be relevant. Parameters are split into A, B and C categories of relevance based on CEE returns.

Code Abbreviation CEE All PROGRAMS A1 Speaks with clarity and confidence 100 percent 99.6 percent A2 Writes clearly and concisely 92 percent 96.5 percent A4 Makes effective presentations 100 percent 100.0 percent B1 Exhibits good listening and questioning skills 100 percent 99.0 percent B2 Evaluates situations effectively 100 percent 99.4 percent B3 Solves problems/makes decisions 96 percent 97.6 percent B4 Demonstrates original and creative thinking 92 percent 97.4 percent C1 Identifies and suggests new ideas 100 percent 100.0 percent C2 Learns new material quickly 92 percent 96.1 percent D1 Accesses and applies specialized knowledge 100 percent 99.0 percent D2 Applies classroom learning to work situations 100 percent 98.2 percent D3 Assumes responsibility/accountable for actions 100 percent 99.8 percent D4 Exhibits self-confidence 96 percent 99.4 percent D5 Possesses honesty/integrity/personal ethics 96 percent 95.9 percent E1 Shows initiative/is self-motivated 100 percent 100.0 percent OR Y A

G E2 Demonstrates a positive attitude toward change 96 percent 93.6 percent TE

CA E3 Works effectively with others 100 percent 99.4 percent E4 Understands and contributes to the organization’s goals 100 percent 95.2 percent G1 Demonstrates flexibility/adaptability 100 percent 99.6 percent G3 Functions well on multidisciplinary team 96 percent 96.1 percent I1 Gives direction, guidance and training 100 percent 98.4 percent I2 Motivates others to succeed 100 percent 98.8 percent I3 Manages conflict effectively 96 percent 92.3 percent J2 Uses technology, tools, instruments and information 92 percent 97.5 percent J3 Understands complex systems and their interrelationships 100 percent 98.0 percent J4 Understands the technology of the discipline 100 percent 99.2 percent K1 Displays ability to design a component, system or process 100 percent 100.0 percent K2 Demonstrates ability to design and conduct experiments 100 percent 100.0 percent K3 Analyzes and interprets data efficiently 100 percent 90.7 percent K4 Understands and works within the culture of the group 100 percent 100.0 percent K5 Respects diversity 100 percent 99.2 percent C3 Recognizes political and social implications of actions 85 percent 93.4 percent F3 Manages projects and/or other resources effectively 73 percent 83.2 percent OR Y B

G G2 Sets goals and prioritizes 85 percent 92.0 percent TE H3 Manages several tasks at once 77 percent 93.4 percent CA J1 Allocates time to meet deadlines 81 percent 88.4 percent A3 Attendance 50 percent 84.1 percent F1 Quality of work produced 58 percent 59.6 percent

OR Y C F2 Volume of work produced 58 percent 47.6 percent G

TE H1 Professional attitude toward work assigned 62 percent 70.3 percent CA H2 Punctuality 27 percent 58.2 percent

Chapter 3 Generating Effective Reports 45 ratio is 95 percent for civil and environmen- It is further important to understand tal engineering supervisors. The quality of that the data only reflects the observa- work produced, volume of work produced, tions of a set of supervisors, and does not, attendance, and punctuality are considered per se, give an indication of the value of less important by 60 percent of either popu- a specific competence. A dimension like lation. The observations support a hypoth- “demonstrates original and creative think- esis that employers see co-op as an invest- ing” can be seen as a positive characteristic ment in the future rather than as a way to in aerospace engineering, whereas it might attract low cost labor. not have the same value in an accounting environment. Whether a certain performance is a positive or a negative, is ulti- Whose perspective does the data reflect? mately a value judgment. The data reflects an aggregate of the supervisor’s opinion of a specific student cohort. It is thus no more or no less value neutral Can we compare majors to one than an opinion of a specific supervisor, of another based on the data? the work of a specific student. A supervisor The parameters measured using a variety is certainly using a grading opportunity, of quantitative instruments, is always valid both to encourage the student, as well as to only in the context of the environment in send a message to the faculty of a specific which it is measured. Terminology used institution. As the data reflects perfor- in Assessment Instrument I is very envi- mance in a work environment, the data ronment specific, and inferences can not can certainly be seen as very productivity be transposed from one environment to oriented when observed from the vantage another. Just reflect over the parameter point of a supervisor. In some instances manages projects and/or other resources this can lead to some very surprising effectively, and apply it to such different results. Adding team work to a freshmen environments as fashion design, account- orientation course at a business college ing, and aerospace engineering. It is obvious as an example had a positive measurable that the parameter refers to very different (conf. 90 percent) difference on param- kinds of project management skills in each eters as follows: Exhibits self-confidence; context. The tool can thus be effectively Functions well on multidisciplinary team; used for benchmarking, but not necessar- Manages projects and/or other resources ily for comparing different programs to one effectively, Motivates others to succeed, another. It would not make sense to say that and Gives direction, guidance and train- one program produces better aerospace ing. The parameter exhibits good listening engineers than what another produces and questioning skills; shows initiative/ accountants. In such a case we would truly is self-motivated; demonstrates a posi- be comparing apples and oranges. tive attitude toward change; understands/ contribute to the organization’s goals did on the other hand show (conf. 90 percent) What traits can be measured? a diminishing quality. This supports the As work based measurement of co-op is hypothesis that teamwork makes students based on aggregating data from supervi- more confident in the group, but, from the sor evaluations, it seems obvious that the supervisors perspective, less likely to follow system produces its most reliable data, when directions and contribute to the goal of the the feature measured is a) a frequent part of organization. the duties of a student population, b) when

46 Chapter 3 Generating Effective Reports the performance related to the measured ing environment, so few, that it might be parameter can be observed by the supervi- difficult to obtain statistically significant sor, and c) when the supervisor has a suf- amount of observations even assuming ficient understanding of the skills required that it meets criteria a, b and c. to produce the outcome. Let us look at It is easy to see, that anything measured these boundary conditions in light of a few must happen frequently, be observable, examples: and fall into the realm of supervisor • Speaks clearly and concisely is a param- competency, in order to yield significant eter that we can assume meets criteria measurement results. a, b and c. In a majority of jobs speaking is required frequently, it is as a rule easy to observe by supervisors and they How do you account for some assessors typically have the competency to rate its being lenient in their grading, and others effectiveness. being tough? We all know that individual assessors can • Makes effective presentations is likely either be lenient, tough or somewhere in to produce less reliable data, per criteria between. In order to be able to detect dif- a, as all jobs do not require students to ferences in performance between various make presentations. The performance is student cohorts we could go three ways: however easily observable by the supervisor and most supervisors would be I. We could hold assessors to an exter- competent in judging whether a presen- nal assessment standard, in order to tation is effective or not. Conclusively obtain comparable results. the parameter meets criteria b and c. II. We could test an assessor in relation • Ability to apply recursive HTML to a group of students, and issue a programming in web design. As many handicap based on the lenience in students producing web pages are grading. supervised by people having relatively broad competency, it will be very hard III. We could observe the distribution for the supervisor to give any kind of of the systematic error (or bias) of a judgment with regard to the effective cohort of assessors and incorporate use of specific programming commands the uncertainty caused by the distri- in web page design. This type of ques- butions to our calculations. tion would not seem to meet the critical The selection of the uncertainty aggrega- criteria c. tion principle of co-op assessors is dictated • Effectively designs beams against by the environment. As an example the buckling using Euler’s theorem. As University of Cincinnati annually com- beam dimensioning is very critical pletes more than 5,000 placements of 4,000 with regard to the safety of various students which are assessed by 2,500 asses- designs, supervisors responsible for sors. The sheer volume of assessors makes beam dimensioning tend to have a good a systematic employer training unfeasible. understanding of the theory of strength Testing assessors with regard to their grad- of materials. The skills are also easy to ing leniency would require the development observe by a specialist. Jobs incorporat- of a specific test, the administration of ing the dimensioning of beams are, even which would be very time consuming. This in a civil and environmental engineer- holds true for almost any co-op program.

Chapter 3 Generating Effective Reports 47 There are typically 10 to 20 times more into account when assessing measurement work supervisors as compared to classroom significance. Focus groups show architects faculty. Knowing how difficult it is to get to have a very diverse value base. It is thus faculty to agree upon a common grading fair to assume that the value of one-third scale, it is easy to understand that training of a unit forms a conservative estimate that 10 to 20 times more supervisors would be a can be used for a variety of majors. The completely daunting task. principles used to assess the level of the bias We are thus left with option III, approx- are presented below. imating the distribution of assessor bias and The data was analyzed with the objective aggregating that to our uncertainty calcula- of getting an understanding of the order of tions. During the FIPSE project the assessor magnitude of assessment uncertainty. The bias was approximated to be one-third of analysis of the systematic uncertainty was a unit of the five demarcation Likert scale pursued by organizing the data according to used in Assessment Instrument I. The value the matrix presented in Figure 3.3. for the standard deviation of assessor bias Seven of the students in the above study was derived based on set of 25 architecture were enrolled in the Civil and Environ- and civil and environmental students that mental Engineering Program, whereas the during a single work term were assessed by reminder of the students pursued an MS or two different assessors. The study returned BS level architecture degree. The filtering of a standard deviation of assessor bias of data controlled for a number of factors. The one-third of a unit, combined with an study took into consideration 33 of the 41 interrater reliability of 63 percent. As the questions of Assessment Instrument I. Eight interrater reliability is less than 70 percent, questions were excluded as a majority of the it is advised that the assessor bias is taken assessors had rated them as non applicable.

Questions

QA QB QC QD ... QN

A B C D N Assessor I a a a a a … a Student I Assessor I b A b Bb Cb Db . Nb

Q1 A B C D N Assessor II a a a a a … a Student II Assessor II b A b Bb Cb Db . Nb

A B C D N Assessor III a a a a a … a Student III Assessor III b A b Bb Cb Db . Nb ……………………………………

Assessor m a A a Ba Ca Da … Na A B C D N Student m Assessor m b b b b b . b

Figure 3.3 Setup of study examining both the systematic and the random assessor uncertainty aggregation. Each student represented in the matrix is assessed by a pair of assessors.

48 Chapter 3 Generating Effective Reports Bias As a first step, the difference in averages (∆A) was calculated between the two asses- 14 sors analyzing the same student. The aver- Architecture 12 Civil & Envir. Eng ages were calculated on mutually assessed parameters, excluding questions having 10 only a single response by one assessor. (The exclusion brought the effective return rate 8 down from approximately 95 to 90 percent). Cases Thereafter the differences of the averages, 6 the ∆A’s, were plotted in a histogram which is presented in Figure 3.4. 4

The data show that 92 percent of all ∆A 2 values fall between ± 0.63 units from the origin. This corresponds to a t-factor of 1.83. 0 - 1.1 - 0.9 - 0.7 - 0.5 - 0.3 - 0.1 0.1 0.3 0.5 0.7 0.9 1.1

Here SSYST can easily be solved for which Difference in Averages comes out as being ± 0.346 units [= ±0.63 units divided by 1.83], which is an approxi- Figure 3.4 The distribution of differences mation of the standard deviation of the ∆A in averages of 25 cases of two assessors distribution sample. In order to get the stan- assessing a single student. [Note that dard deviation of the assessor distribution, the bell curve has been constructed by the value actually has to be divided by the mirroring the distribution of absolute square root of two. Figure 3.5 illustrates the situation in detail. The ∆A distribution ∆A’s over the y-axis.]

Observed Dis

Assessor B tribution S SYST

S∆

S SYST = S

Assessor A 2

Figure 3.5 The histogram of systematic error couples describes a set of lines plotted in the Assessor A Assessor B plane. Dimensions derived in the ∆A domain need to be scaled down by a factor of the square root of two in order to have relevance in the assessor distribution dimension.

Chapter 3 Generating Effective Reports 49 plotted in the histogram can be interpreted Asessor Consistency as a multitude of straight lines the general 70%

form of which is presented in Expression 3.1. 60% (3.1) AASESSOR1 A ASESSOR2 $A 50%

40% The histogram in Figure 3.4 actually presents the distances between these paral- 30% lel lines. As already covered in Figure 3.5, uncertainties aggregate according to the 20% Pythagorean Theorem. Hence, in order to 10% get applicable values, the standard deviation in the delta domain must be divided 0% by the square root of two. 4 3 2 1 0 1 2 3 4 A chi-square estimate further returns Discrepancy [∆ Grade ] a 95 percent confidence level upper boundary for the assessor systematic error Figure 3.6 Aggregated measurement noise distribution as 0.33 units. embedded in statistical signal. (The bell curve has been constructed by mirroring the distri- The data was further used to measure bution of absolute deviations over the y-axis.) interrater reliability. The bivariate data presented in Table 3.4 has been plotted to implies that 174 times out of 747 the asses- convey a picture of how the discrepancy sors agreed on giving the student the grade in grading between two assessors 5. A score of 91 in cell (3, 4) implies that 91 manifests itself in a real life situation. times out of 747 one assessor graded the Each cell in Table 3.4 contains the same parameter with a three as the other record of the number of times the two assessor considered it worth a four. The assessors gave a specific combination of color code helps illustrate the required diag- scores when assessing the same student for onal tallying. The discrepancy tally shows the same trait. The score 174, in cell (5, 5) an agreement score of 473, a disagreement

Assessor A Diagonal

5 4 3 2 1 Totals

5 174 119 30 1 2 n ∆

B 4 227 91 5 0 2 4

3 59 25 0 1 3

s s e s s o r 2 [n values] 13 1 35 2 A

1 0 236 1

Total: 747 473 0

Table 3.4 Bivariate matrix of 50 assessors assessing 25 students in pairs. All deviations between individual scores are accounted for in absolute values.

50 Chapter 3 Generating Effective Reports by one unit score of 236, and so forth. This assessed by approximately half as many offset curve forms a normal distribution, assessors. The aggregation of the systematic and is presented in Figure 3.6. uncertainty of assessor behavior to the ulti- Figure 3.6 shows a relatively high con- mate measurement result requires a reliable sistency between assessors. The histogram model of the actual measurement process. of this bivariate data has an approximate The assessment process has its parallels to standard deviation of 0.86 units. Divid- the naïve truck height measurement system ing the standard deviation with the square presented in Figure 3.7. The figure presents root of two, and subjecting the result to the schematic setup of a system established a chi square analysis establishes that the to measure the average height of a series of underlying standard deviation with a con- trucks passing over a bridge. The heights of fidence level of 99 percent ranges between trucks are measured by assessors who have 0.57 and 0.65 units. The interrater reliabil- rigged their measurement devices on indi- ity, also defined as the percentage of agree- vidual rafts floating on a river. Their -mea ment between assessors, is approximately surement tripods are somewhat different in 63 percent. height. The difference in height combined with the waves of the sea, constitute sources of systematic and random uncertainty. The What kind of models do you use fact that the tripod height-differences fol- to incorporate assessor bias low a normal distribution makes the error into measurement results? aggregation relatively simple. In the model, The Cooperative Education assessment the wave distribution is equally normally process has its unique and robust character- distributed. Once the standard deviation istics. Annually, a typical program enrolls of the height of the tripods is known one hundreds of students pursuing a multitude could, over a series of measurements, easily of work terms. Individual students get get estimates of the average height of the

M u l t i p l e M e a s u r a n d s

M e a s u r e m e n t s Systematic Uncertainty Random Uncertainty

M u l t i p l e A s e s s o r s

Figure 3.7 Mechanical model of the assessment system at hand.

Chapter 3 Generating Effective Reports 51 trucks belonging to a specific population. the assessment always relies on a personal One could further decide not to account relationship between the student and the for the height of the waves, as their random supervisor. Co-op grades given by differ- impact would be canceled when distributed ent assessors should never be compared. In over a large number of measurements. As the measurement of aggregate values the long as the water level in the river was stable system is however relatively accurate, as a the system could be used to measure the larger number of n values tend to reduce the difference in the average height of trucks uncertainty of the mean. of various truck cohorts. The fact that the actual height of the water in the river is What statistical methodology unknown would not impact the result of the do you use to compare cohort means? measurements as long as the objective is to detect differences. The mechanical system As the bias of assessors follows a bell curve is na ve, but it has its parallels to the co-op we need a simple methodology to take this work performance measurement system bias into account when reporting confi- under scrutiny. The trucks correspond to dence levels of the differences in means. The the students; the rafts with their systematic simplest way to do this is to develop a t-test and random uncertainty to the assessors. that takes into account the assessor bias In a co-op situation the observer does not distribution (having a standard deviation of necessarily know the systematic error of the approximately a third of a unit). We will call assessor. Still, if the standard deviation of this t-test an expanded t-test. the systematic uncertainty component of Before diving into the intricacies of an individual assessor populations is known, expanded t-test, let us first look into the calculations can return an estimate for the details of the traditional t-test. Comparing uncertainty level of the measurement result, two means can in the statistical domain be when comparing one set of measurements pursued using a simple t-test. The formula to another. for a t-test which can be found in any basic Applying the na ve model presented statistics handbook, is presented below in in Figure 3.7 to a complex reality requires Expression 3.2: a quantification of the systematic component of assessor uncertainty distribution, as well as the development of a quantitative t b expression that helps levy the estimated 2 2 sREAD1 sREAD2 (3.2) uncertainty on specific sets of measured means. The beauty of both the model and n1 n2 the co-op assessment system in reality is that they carry a lot of redundancy. The Here ∆A represents the distance system is built around an abundance of between means SREAD the standard devia- assessors, making the aggregate less vul- tion of a specific reading, and n the amount nerable to discontinuities. of readings in a specific sample. The under For the measurement of the perfor- indexes 1 and 2 refer to reading one and mance of an individual student the system reading 2. Graphically the t-test measures is relatively inaccurate. Due to differences in whether the arithmetic difference between the systematic error of assessors the grades the means is larger than the aggregated of two individual students can never be uncertainty of two bell shaped readings as fully comparable. On an individual level, shown in Figure 3.8.

52 Chapter 3 Generating Effective Reports A is dependent on both n-values (amount of readings) and k-values (amount of individual assessors). The expression requires that the degrees of freedom relevant for the t-test (technically dependent on both k and 2 t S 2 t S READ 1 READ 2 n values) be selected conservatively. This n 1/2 n 1/2 1 2 typically entails using the smallest value

of n1-1, n2-1, k1+ k2-1. In most cases these Figure 3.8 A t-test compares the difference numbers are of the same order of magni- in means to the square root of the sum of tude, in which case an approximation does the squared uncertainties. The comparison, not present much of a loss in confidence. using the t-function, helps the observer The formula can further be expanded to understand with what probability the Expression 3.4, which takes into account the means differ from one another. amount of mutual assessors in two independent samples. Whenever we add the standard devia- The impact of the assessor bias can tion of assessor bias into the picture, the theoretically be alleviated in cases where uncertainty of the measurement becomes two populations share assessors as somewhat larger than the statistical presented in Figure 3.10 (on next page). uncertainty presented in Figure 3.8. Figure 3.9 illustrates the situation. In this case we can use the Expression 3.4 to calculate statistical uncertainty. A

2 2 2 2 Aggreg ated sREAD 1 1 v1 sSYST sREAD 2 1 v2 sSYST Uncerta inty $A r t (3.4) n1 k1 n2 k2

The factors 1v and v2 account for the Statistical percentage of assessors within each group Uncerta inty that is mutual between the groups. As an Figure 3.9 Comparison of means when ag- example, a situation having a k1 of 30 and k of 40 and a k amount of 20 mutual gregating statistical and systemic uncertain- 2 m assessors would result in a v value of 0.66 ty. Incorporating the systematic uncertainty 1 and a v value of 0.50. If the amount of to a t-test simply widens the skirt of the bell 2 curve, requiring a larger ∆A to demonstrate mutual assessors is 100 percent, the factors [v -1] and [v -1] are both reduced to statistically significant differences. 1 2 zero. This totally eliminates the impact Mathematically the t-test can be on the end result of the systematic error expanded to a formulae presented in of assessor behavior. A situation with Expression 3.3. mutual assessors obviously cancels any bias between the two assessments. The b beauty of the setup is that the impact of (3.3) t 2 2 2 2 READ SYST READ SYST systematic error is inversely proportion- s 1 s 1 s 2 s 2 ate to the percentage of mutual assessors n1 k1 n2 k2 in either group. In essence this means that What Expression 3.3 shows us is that if the assessor population changes with the confidence level of a difference in means a rate of 50 percent a year, the effect of

Chapter 3 Generating Effective Reports 53

kII Uncertainty

Quantification Preprocessing Interpretation & Interpretation

km Population

Comparison k

kI Uncertainty Preprocessing Quantification Interpretation & Interpretation

Report Generat ion

Figure 3.10 Assessment information flow block chart. k = Total assessor population, kI = Sampled assessors group I; kII = Sampled assessors group II; km = Mutual assessors. Assessor populations kI and kII constitute a sub group of the larger assessor population k. The overlap of populations kI and kII constitutes the group of mutual assessors km. the systematic error gets reduced to half assessments. In the expression the letter k when comparing the development between indexed with a roman numeral stands for two consecutive years. The above reason- the number of assessors contributing to ing assumes time-invariance in assessor the process 1, 2, 3, … n times. systematic uncertainty. While responsible Table 3.5 further applies Expression assessors undoubtedly maintain rela- 3.5 to the assessor data of 2005 presented tively strict assessment practices, further in Figure 7.5. Table 3.5 spells out that the research is needed to determine to what 94 assessors contributing to the assessment extent assessor systematic behavior is process between one and six times each strictly time-invariant. had an equal effect as 71 assessors filing an Expression 3.4 works when the asses- equal amount of assessments. The finding sors in each group contribute with an supports spreading the workload as evenly equal amount of assessments to the pro- as possible between assessors. cess. When ever this is not the case expression 3.5 can be used to calculate a parame- 2

I 2 3 4 … n ter defined as the effective assessor amount k kII kIII kIV kn kEFFECT 2 2 2 2 (3.5) or kEFFCT. This factor accounts for the loss kI 2 kII 3 kIII 4 kIV … n kn of certainty in cases where the individual assessors perform different numbers of

54 Chapter 3 Generating Effective Reports 2 index Assmnt/ Amount k a k a kEFFECT Assessor (a) of Assessors (k) I 1 43 43 43 II 2 33 66 132 III 3 10 30 90 71 Table 3.5 Aggregation IV 4 5 20 80 of the effective assessor V 5 2 10 50 amount for Civil and VI 6 1 6 36 Environmental 94 175 431 Engineering in 2005.

How do you compare the results in Does the Likert scale of 1 to 5 the differences in standard deviations? have enough resolution? The standard-deviation-mean-matrixes There is a common misperception that one presented earlier in this chapter rely on could add significantly more confidence to determining the uncertainty of both the ones measurements by shifting from a five mean as well as the standard deviation. to a let us say nine demarcation Likert The mean is typically considered the more scale. Could we not get higher confidence important measure of student perfor- levels by adding one decimal of accuracy? mance, and the calibration efforts have Unfortunately merely adding resolution to thus largely focused on quantifying the a scale is not the answer, as far as behavior uncertainty of the mean. This implies that measurement is concerned. Research done for the sake of the uncertainty require- in conjunction with the FIPSE project ments, a standard F-test will return suffi- shows that the total uncertainty (bias and cient results. The elegance of the F-test lies random) combined have a total standard in its simplicity. The F-test can be pursued deviation of approximately 0.55 units. Let using Expression 3.6. us examine a fictive case where a student 2 would perform at exactly (if there was such ¥ ´ ¦ s1 µ (3.6) a thing) on the level of 3.5. Figure 3.11 F ¦ µ § s2 ¶

5 5 The value of the function F is depen- 5 dent on the statistical significance 4 4 4 required, as well as degrees of freedom of 3 3 3 both standard deviation S1 and S2. The test essentially ensures that when demonstrat- 2 ing the statistical significance of the dif- 2 2 ference between two standard deviations, 1 1 the lower boundary of the larger standard deviation is higher than the upper boundary of the smaller standard deviation. Whenever the values are close, care must be given in choosing one or two tailed Figure 3.11 Relationship between scale reso- approaches. The CEE data shows that it lution and human measurement uncertainty. is difficult to get a higher than 80 percent Whenever the standard deviation is of the statistical significance in the differentiation same order of magnitude as the rating scale, of the standard deviations covered by the individuals will use a wide range of ratings to assessment process. communicate identical performance levels.

Chapter 3 Generating Effective Reports 55 shows how a typical assessor population would assess the situation. A small fraction of assessors would give 2s and 5s, but most would give 3s and 4s. The human factor holds a much larger uncertainty than our measurement scale. With a sample size of 50 measurements the result would with a confidence level of 90 percent have a value of 3.5 ±0.16. This uncertainty (M ± t S / √n) would not be significantly impacted by the resolution. The diagram on the left shows the rating distribution on a scale having a resolution of one. The diagram on the right shows how the ratings could vary if we move to a rating scale of 0.1. Switch- Figure 3.12 Photo of water in motion. ing rating scales is a very risky proposition, a) [Left] Long exposure time. as doing so will jeopardize the validity of a b) [Right] Short Exposure time. [Source: longitudinal study. As there is no short cut Public Domain Photo; www.pdphoto.org, to an added resolution, staying conserva- retrieved October 17, 2006.] tive is typically the best bet. Figure 3.12 a) shows a photo taken of a waterfall using a long exposure time. The Do you have sample sizes large enough extended exposure causes a loss of informa- to be able to demonstrate statistically tion of the position of individual droplets significant results? at a given point in time resulting in a shady The problem with assessing student’s work picture of water flowing through space. performance statistically in a cooperative This picture, however, has strong statistical education environment is that programs qualities. The lighter areas bear evidence of tend to operate close to the lower boundary a high water density, and vice versa. A long of acceptable confidence levels with regard exposure time gives the observer a good to both n and k values. Programs barely picture of the waterfall as a process, reveal- have sufficient enrollment to allow the plot- ing little about the position of each indi- ting of program-specific bell curves. The vidual droplet at a specific point in time. temptation is to increase n and k values by Figure 3.12 b) on the contrary, was taken combining data pertaining to different sec- using a short exposure time. The picture tions, adjacent co-op quarters, or successive conveys information of where each indi- academic years. This can be done when- vidual droplet is at the particular point in ever the limitations of this operation are time when the photo was taken. It conveys fully understood. Whenever n-values are an abundance of in-formation of a specific artificially boosted the measurement tends case, but filters away information on how to loose some accuracy pertaining to the the process acts statistically. Metaphorically, subject of the measurement. The academic Figure 3.12 a) corresponds to high n-values process under scrutiny is not time invari- whereas Figure 3.12 b) is based on observing ant as the process slightly changes over very few measurands. The advantage of the time. Figure 3.12 illustrates this dichotomy long exposure time is that the photographer through a metaphor. with relatively little cost can give the photo

56 Chapter 3 Generating Effective Reports statistical qualities by simply extending exceed 15, research results tend to be lost the exposure time. Unfortunately coopera- to measurement uncertainty. Low student tive education, as a process, is considerably numbers and assessor variability combined less time-invariant than a waterfall built with small changes in behavior would make on solid rocks. An educational organiza- the results unreliable and excessively diffi- tion changes from one year to the next. cult to interpret. The significance of the change obviously The uncertainty caused by small sec- varies case by case. The waterfall allegory tion sizes can be alleviated whenever an reveals that the observer might gain statisti- educational process exhibits relatively sta- cal information of a process by combining ble characteristics. The education a student information from different time intervals. receives is far more than a set of syllabi Simultaneously, one must be prepared for a arranged to form a curriculum. Program loss in certainty caused by the evolution of strength is impacted by admission and the process. graduation requirements, extracurricular The section below illustrates the princi- options, faculty credentials, university ple with a practical example from the co-op organization, student commute, student domain. As covered above, small cohort housing, socioeconomic circumstances, sizes form a challenge when assessing the and so forth. A single curricular change relationship between curricular content does not typically cause a substantial and job performance. If one would want to change in a student’s work performance. know what impact a particular curricular The development of a program is paced by change has on work performance one could the learning curve of the entire organiza- approach the problem by measuring work tion. However, the inertia of an institution performance before and after the co-op can be used to one’s advantage when striv- quarter and compare it to a control group ing to capture a picture of the quality of still pursuing the old curriculum. The prob- the educational processes that it offers. lem with this approach (illustrated in Figure We can look at the stability in light of a 3.13) is evident. As section sizes seldom few observations. The analysis showed that

Figure 3.13 Comparison of job performance between two consecutive co-op quarters. Observations on a section level typically get lost in noise.

Chapter 3 Generating Effective Reports 57 the averages of data collected under con- cess has changed one should refrain from trolled conditions during two consecutive doing so. In reality processes under scrutiny years are close to identical. Table 3.6 com- evolve relatively slowly. Thus, whenever the pares the means and standard deviations of boundary conditions affecting a specific civil and environmental engineering co-op outcome have not changed, one can use ratings returned during academic years rolling averages. 2001/02, 2002/03, and 2003/04 (collected using a paper based instrument). How do you account for transfer students, The data presented in table 3.6 shows students switching majors and dropouts? no significant differences between the three years. In fact an ANOVA (not considering Structural uncertainty considerations fluctuations in assessor systematic uncer- form the foundation for any inferences tainty) shows that there is an 81 percent the observer wants to make with regard probability that the three samples are taken the measurement results. Earlier in this from the same population. This observation chapter we pursued a relatively extensive supports the use of rolling averages when discussion of mathematical uncertainty trying to alleviate the effect of noise. aggregation. By implementing this mathematical process, one can determine whether there is a significant difference in Academic Year 2001/02 2002/03 2003/04 performance between students pursuing A 4.16 ± 0.15 4.12 ± 0.14 4.19 ± 0.14 the program at different levels. In order to N 213 201 198 draw inferences on the extent to which dif- n 160 175 162 ferences in measured student performance s 0.75 0.77 0.78 reflect instructional quality, one needs to S (approx.) 0.33 0.33 0.33 SYST have an understanding of the structural k (approx.) 70 70 70 EFFECTIVE integrity of the program. When examining Table 3.6 Means and standard deviations the system more closely, one realizes that of co-op GPA’s of three consecutive years. the biggest sources of uncertainty stem (Civil and Environmental Engineering). from various program-specific boundary conditions. Students entering or leaving *Statistical confidence level: 95 percent. Infinite population. a program mid stream cause leakage that significantly impacts both the trends of the What compromises do you have to assessment data, and the conclusions that make when using rolling averages? can be drawn based on it. Low n-values When using rolling averages as presented form another inherent problem with in Figure 3.14 one is able to demonstrate regard to the assessment process. When significant progression between different the size of a typical graduating class is stages of the curriculum. approximately 40 students, quarter specific The variable nature of the process n-values are typically no larger than 20. under scrutiny poses a challenge for the This section of the chapter will cover researcher. In what cases can one combine structural program specific uncertainty data, and in what cases should one refrain sources and discuss possible methodolo- from doing so? There is no simple answer to gies that can be used to alleviate their this question. The simplistic answer is that impact. Leakage, illustrated in Figures whenever the process has not changed, one 3.15 and 3.16, implies that students leave, can combine data, and whenever the pro- enter, or transfer between programs mid-

58 Chapter 3 Generating Effective Reports Mean 4.28 4.31 N: 612 n: 497 4.03 Ret: 81% 3.97 Uncert : ≈ 0.10

88 191 148 70 = Filed Returns

Sophm . PreJr . Jr. Sr. 01/02 Figure 3.14 The principle of rolling 2005 2004 2003 2002

U F W S U F W S U F W S U F W S averages. Co-op GPA data manually collected for students enrolled in the Civil 02/03 and Environmental Engineering program 2006 2005 2004 2003

U F W S U F W S U F W S U F W S has not been subject to filtering. Statisti- cal and measurement uncertainty ± 0.10 03/04

2007 2006 2005 2004 units, assuming 95 percent confidence U F W S U F W S U F W S U F W S level (N = 612, n = 497 fi r = 81 percent).

stream. Whenever leakage occurs ran- drop out of the program before graduation. domly, it impacts the uncertainty through Transfer students, entering the program changes in n and k values only. In reality, mid stream with considerable life experi- however, leakage tends to be systematic, ence might also skew the statistics toward thus having a severe impact on the mea- showing substantial progress. Leakage surement results. constitutes a considerable challenge when University records show that the co-op using the assessment information for cur- program of the College of Business (CoB) ricular reform. Extensive research that leaks two ways; students leave and enter goes beyond the scope of this project is the program mid stream. The program required to map to what extent the apti- is optional which allows low perform- tude of the leaking populations differ from ing students to drop out of the program, the one of the main stream population. significantly skewing the statistics. Leak- Whenever there is no significant differ- age can skew the statistics in any direction. ence, co-op work performance data stem- In some cases, over-performing students ming from the entire population can be receive lucrative job offers causing them to used to drive curriculum development. As

Figure 3.15 Leakage implies that students Figure 3.16 Students transfer from one enter or leave programs mid-streams. program to another midstream.

Chapter 3 Generating Effective Reports 59 this still most likely is not the situation, age between the two programs, which is outcomes-based assessment and curricular reflected in the program-specific progress reforms addressing these specific popula- reports. The situation is complicated by the tions will have to be developed. fact that the formal transfer application can be filed by the student long after the transfer actually took place. What are the pros and cons with longitudinal data? When trying to control for situations How do you make the connection between where programs leak, one faces multiple the curriculum and work performance? challenges. Longitudinal studies require A robust assessment system that codes the elimination all individuals that have student performance with unique student, leaked into or leaked out of the system. and work term identifiers, is a key for This might cut the amount of applicable successful data analysis. In order to make n-values by a significant factor. The elimi- sense of the information, all data has to nation causes an observer to take the mea- be coded in relation to the classroom cur- surement signal from a very limited popu- riculum by tracking the evaluation to a lation, which might not be equitable when specific term and then placing that term considering the greater whole. Leakage can into the context of the curriculum. FIPSE unfortunately also retroactively affect the project data shows that there is a high validity of the data. If a student drops out correlation between stellar work perfor- before the last year of a three-year co-op mance, and following the curriculum on program, one will have to eliminate him or schedule. her from the longitudinal reports covering all three years. This can lead to a very frus- Do you have the right employers? trating situation necessitating a retroactive revision of data, just because a student One question that often gets asked is drops out of the program before the end of whether the present co-op employers are the last quarter. the right employers for the students of a specific co-op program. There is obviously Leakage between programs, illustrated no right answer to this question, but the in Figure 3.16, is evidently an equally fact that the question gets asked shows important error source as compared to signs of an emerging dialogue between the leakage in and out of the program. co-op practitioners and classroom faculty. The College of Applied Science (CAS) The typical counter question tends to be, Construction Science department can whether the students have the right skills serve as an example of a case demonstrat- to get the right job. The debate that ensues ing leakage between programs. The depart- is a very healthy one, and is likely to bring ment offers two baccalaureate programs, a lot of value to the program. one in construction management and one in architecture engineering technology. Freshmen are drawn strongly to the Archi- Is the instrument well developed to tecture Engineering Technology program. measure what you claim it does? While in the program they notice that Cronbach’s alpha procedure is typically the employment situation is more robust used for the measurement of internal for construction managers. As a result consistency of the instrument. Cronbach’s there is a considerable, but selective, leak- alpha tells the researcher how consistent

60 Chapter 3 Generating Effective Reports the underlying parameters are in measur- Could we group the underlying ing a specific construct. We can, as an parameters more effectively? example use Cronbach’s alpha to give a Finding more effective ways to group value for how well speaks with clarity and underlying parameters can be done using confidence, writes clearly and concisely, factor analysis. Factor analysis is a meth- makes effective presentations, exhibits odology that is used to regroup constructs good listening and questioning skills as an into more meaningful clusters. The theory aggregate measures communication skills. of Factor analysis is based on rotating the It is good practice to run the internal con- coordinate system representing different sistency every time we use an instrument factors in the hyperspace defined by the to measure something in a new context. different measured parameters. The tables Cronbach’s alpha gives a value between below give an overview of a puzzling prob- 0 and 1 as a measure of how well differ- lem solved during the project using factor ent arbitrary subsets of a construct, give analysis. consistent results to when compared to other arbitrary subset of the construct. As A before and after study showed an an example internal consistency of Assess- unusual development in a comparative stu- ment Instrument I was measured using dent cohort study. Tables 3.8 and 3.9 reflect the Cronbach Alpha algorithm. The mea- the data before and after factor analysis. surement was based on the 390 electronic The analysis based on predetermined returns presented in Table 3.7. The findings constructs shows a mixed bag of results. are very much in line with the findings Even though the internal consistency is pursued by UC Evaluation Services Center consistently above 0.80, the positive and presented earlier in this chapter. All con- negative parameters are distributed over structs show a coefficient alpha larger than a number of constructs. Regrouping the 0.80. Table 3.7 gives a detailed overview of parameters using factor analysis brings the Cronbach alpha values calculated for the internal consistency of all constructs individual constructs. but one above 0.90. All parameters having

Abbrev. Category Cronbach’s Alpha No. of DATA SETS Alpha,ALL FACTORS A1 – A4 Communication 0.81 160 B1 – B4 Conceptual And Analytical Ability 0.90 342 C1 – C3 Learning/Theory And Practice 0.84 320 D1 – D5 Professional Qualities 0.88 367 0.983 E1 – E4 Team Work 0.87 325 [based on F1 – F3 Leadership 0.88 150 41 data sets] G1 – G3 Technology 0.81 372 H1 – H3 Design & Experimental Skills 0.88 94 I1 – I3 Work Culture 0.80 335 J1 – J5 Organization Planning 0.94 308 K1 – K5 Evaluation of Work Habits 0.86 383

Assessment Instrument I consistently throughout the FIPSE project returned values of 0.8 or larger. Table 3.7 Cronbach’s Coefficient Alpha and corresponding return values for all underlying constructs of Assessment Instrument I, calculated using SPSS 14.0 for Windows.

Chapter 3 Generating Effective Reports 61 Table 3.8 Results based on pre determined constructs. A COMMUNICATION [Cronbach Alpha = 0.80] A1 Writes clearly and concisely POS A2 Speaks with clarity and confidence POS A3 Makes effective presentations POS A4 Exhibits good listening and questioning skills NEG d PROFESSIONAL QUALITIES [Cronbach Alpha = 0.86] D1 Assumes responsibility/accountable for actions NEG D2 Exhibits self-confidence POS D3 Possesses honesty/integrity/personal ethics POS D4 Shows initiative/is self-motivated NEG D5 Demonstrates a positive attitude toward change NEG E T teAM WORK [Cronbach Alpha = 0.85] E1 Works effectively with others POS E2 Understands/contributes to the organization’s goals NEG E3 Demonstrates flexibility/adaptability POS E4 Functions well on multidisciplinary team POS F LEADERSHIP [Cronbach Alpha = 0.89] F1 Gives direction, guidance and training POS F2 Motivates others to succeed POS F3 Manages conflict effectively POS J ORGANIZATION PLANNING [Cronbach Alpha = 0.85] J1 Manages projects and/or other resources effectively POS J2 Sets goals and prioritizes POS J3 Manages several tasks at once - J4 Allocates time to meet deadlines POS

Table 3.9 Regrouped parameters using factor analysis. A PROFESSIONALISM [Cronbach Alpha = 0.94] R1 Exhibits good listening and questioning skills NEG R2 Assumes responsibility/accountable for actions NEG R3 Possesses honesty/integrity/personal ethics POS R4 Shows initiative/is self-motivated NEG R5 Demonstrates a positive attitude toward change NEG R6 Understands/contributes to the organization’s goals NEG R7 Demonstrates flexibility/adaptability NEG D E exPRESSIVE COMMUNICATION [Cronbach Alpha = 0.93] EC1 Writes clearly and concisely POS EC2 Speaks with clarity and confidence POS EC3 Makes effective presentations POS EC4 Exhibits self-confidence POS J ORGANIZATIONAL PLANNING [Cronbach Alpha = 0.93] J1 Manages projects and/or other resources effectively POS J2 Sets goals and prioritizes POS J3 Manages several tasks at once - J4 Allocates time to meet deadlines POS F LEADERSHIP [Cronbach Alpha = 0.90] F1 Gives direction, guidance and training POS F2 Motivates others to succeed POS F3 Manages conflict effectively POS T teAMWORK [Cronbach Alpha = 0.78] T1 Works effectively with others POS T2 Functions well on multidisciplinary team POS

62 Chapter 3 Generating Effective Reports a negative development are now clustered The data is presented on the next four under one relatively independent con- pages in a Table form (Table 3.10) and struct. The new grouping certainly helps graphically as an entry, exit and delta understand the development of the stu- MSMs (Figures 3.17, 3.18 and 3.19). How dents more effectively. can we use these reporting tools to communicate the status of the program? Are there any general trends Table 3.10 contains data having an with regard to cause and effect? excellent data measurement significance, even after considering the bias uncer- Preliminary project data shows that there tainty of the assessor population. The are certain general trends with regard to table shows that the program, with a sig- process enhancement and learning out- nificance level of 95 percent (two tailed) comes. Whenever care is given to the teach- has achieved a change in mean in more ing process, the learning outcomes tend to than 80 percent of the measured param- be more homogeneous. Whenever teaching eters. The standard deviation that reflects resources are diminished the opposite is the quality of the educational process, the case. Means do not however seem to decrease with a confidence level of 95 be affected as much. In a less supportive percent in more than 50 percent of the environment, the differences between stellar measured cases. and failing students become obvious. The entry profile shows that the students strongest work performance (upper Results of Practical Implementation left hand corner) is exhibited with regards to K4 Attendance, D3 Integrity, E1 Work- We will here apply the principles covered ing Effectively with Others, and I2 Respects in this chapter using a real world example. Diversity. The fact that these parameters The Mechanical Engineering Program are on top, tell a story about a student at the University of Cincinnati is a well cohort that is serious about its commit- established program that has relied on ment, its values, and interaction with its cooperative education for over a century. environment. D4 Self motivation and D2 The analysis is based on comparing the Self confidence are however problems. The co-op performance of second and fourth standard deviation tells a story of a rela- year students enrolled in the program over tively diverse preparedness in this area. academic years 2005/06 and 07/08. The The lowest performance is exhibited with comparison is based on a snapshot study, regard to F1 Gives Direction, F2 Motivates which means that we are comparing the Others, and F3 Conflict Management. entry and exit performance of two different sets of individuals. (A longitudinal The exit profile reviles a significant study was not possible as assessment data difference when compared to the entry at the time was only available for two con- profile. The level and uniformity of stu- secutive years.) As the admission require- dent performance, do both increase as a ments to the program have not changed result of the program. significantly, there is no reason to believe The Delta profile tells the story of how that the differences in means and standard student growth. B2 Problem Solving is deviations could be attributed to a dif- the star exhibiting a 0.30 unit rise as ference in the credentials and aptitude of well as a significant reduction in stan- enrolled students. dard deviation. A4 Listening, A2 Writing,

Chapter 3 Generating Effective Reports 63 Table 3.10 Mechanical Engineering, Co-op Performance Measurement Results.

entrY exit t-test (incl. Bias)* F-test Topic n A S N A S ∆ A CONF** ∆ S CONF*** A1 Speaking 359 3.96 0.80 205 4.16 0.66 0.20 99.3% -0.14 99.9% A2 Writing 341 4.00 0.74 193 4.09 0.68 0.09 75.4% -0.06 92.0% A3 Presenting 241 3.93 0.77 136 4.15 0.69 0.22 98.3% -0.07 89.6% A4 Listening 361 4.23 0.77 204 4.41 0.62 0.18 98.7% -0.15 100.0% B1 Evaluates Situations 351 4.03 0.70 203 4.17 0.63 0.14 95.3% -0.07 95.6% B2 Problem solving 350 3.90 0.79 203 4.21 0.67 0.31 100.0% -0.11 99.3% B3 Creative Thinking 350 3.99 0.78 202 4.23 0.62 0.23 99.8% -0.15 100.0% B4 Idea Generation 345 3.97 0.79 201 4.23 0.72 0.27 99.9% -0.07 91.7% C1 Learning 361 4.40 0.64 205 4.55 0.56 0.14 96.7% -0.08 98.2% C2 Appl.Spec. Knowledge 332 4.05 0.71 194 4.26 0.64 0.21 99.5% -0.07 93.9% C3 Appl. Classrm Lrnng 318 3.88 0.72 192 4.08 0.67 0.19 98.9% -0.05 86.5% D1 Accountability 356 4.22 0.71 205 4.45 0.59 0.23 99.9% -0.12 99.8% D2 Self Confidence 361 3.99 0.82 205 4.23 0.72 0.25 99.8% -0.10 98.2% D3 Integrity 360 4.51 0.62 204 4.61 0.52 0.10 86.9% -0.10 99.7% D4 Self Motivation 361 4.26 0.83 205 4.41 0.72 0.15 94.3% -0.11 98.5% D5 Positive Attitude 345 4.35 0.66 203 4.48 0.57 0.13 95.2% -0.09 98.6% E1 Works with others 360 4.42 0.62 205 4.57 0.55 0.15 97.7% -0.07 96.6% E2 Goal Orientation 353 4.21 0.71 201 4.36 0.62 0.15 95.9% -0.09 98.5% E3 Flexibility 358 4.35 0.66 205 4.49 0.57 0.14 96.1% -0.09 99.0% E4 Mutidiscip. Team W 307 4.29 0.67 188 4.44 0.60 0.14 95.1% -0.08 96.3% F1 Gives Direction 187 3.74 0.75 145 4.01 0.78 0.27 99.3% 0.04 72.3% F2 Motivates Others 197 3.80 0.73 147 3.88 0.75 0.09 63.6% 0.02 63.3% F3 Conflict Management 191 3.80 0.75 153 3.99 0.73 0.19 95.6% -0.02 62.5% G1 Use of Technology 355 4.22 0.67 203 4.36 0.63 0.14 96.0% -0.04 81.1% G2 Syst. Understanding 342 3.91 0.71 197 4.11 0.68 0.21 99.3% -0.03 73.8% G3 Underst. of Tchnlgy 345 3.94 0.72 196 4.16 0.63 0.22 99.7% -0.09 97.8% H1 Comp. Design Ability 251 3.96 0.74 166 4.16 0.72 0.20 98.0% -0.02 65.0% H2 Exp. Design Ability 224 3.97 0.72 142 4.17 0.66 0.20 97.1% -0.06 86.0% H3 Data Analysis 299 3.99 0.72 185 4.17 0.64 0.18 98.2% -0.08 95.2% I1 Work Culture Und. 357 4.28 0.68 203 4.41 0.62 0.14 94.8% -0.06 93.9% I2 Respects Diversity 341 4.44 0.64 195 4.49 0.60 0.05 55.1% -0.05 87.7% I3 Rec. Political Impl. 320 4.20 0.76 190 4.28 0.69 0.08 69.7% -0.07 92.9% J1 Project Management 324 4.01 0.75 194 4.21 0.71 0.19 98.6% -0.04 79.1% J2 Goal Setting 335 3.96 0.75 195 4.15 0.72 0.19 98.4% -0.03 70.9% J3 Task Management 345 4.11 0.77 200 4.35 0.68 0.23 99.8% -0.09 97.5% J4 Time Allocation 345 4.03 0.75 199 4.30 0.68 0.27 100.0% -0.07 94.7% K1 Professional Attitude 357 4.40 0.69 205 4.54 0.56 0.14 95.9% -0.14 100.0% K2 Work Quality 357 4.25 0.73 204 4.42 0.63 0.16 97.8% -0.11 99.3% K3 Work Volume 357 4.13 0.76 204 4.28 0.69 0.16 96.4% -0.07 92.6% K4 Attendance 358 4.61 0.67 204 4.63 0.59 0.03 29.8% -0.08 97.6% K5 Punctuality 358 4.54 0.74 203 4.54 0.69 0.00 0.0% -0.04 83.6% *See page 53. ** Two Tailed. *** One Tailed.

64 Chapter 3 Generating Effective Reports 4.70

4.60 K4

4.50 D3

I2 E1 4.40 C1 K1

E3D5

4.30 E4 I1 K2 D4 G1 D1 A4 4.20 I3 E2 MEAN

K3 J3 4.10

C2 B1 J4 J1 4.00 H3 B3 D2 H2 H1 J2 B4 A1A2 G3 A3 G2 3.90 B2 C3

3.80 F2 F3

F1 3.70 0.50 0.55 0.60 0.65 0.70 0.75 0.80 0.85 STDEV Table 3.17 Entry profile of students enrolled in the University of Cincinnati Mechanical Engineering Program Data Collected 2005/06 -07/08. For parameter definitions see Table 3.10.

4.70

K4 4.60 D3 E1 C1

4.50 K1 I2 D5 E3 I1 E4 K2 D4 4.40 D1 A4

E2 G1 J3

4.30 J4 I3 C2 D2 K3 B3 B4 B2 J1 4.20 H3 MEAN B1 A1 H2 A3 J2 G3 A2 H1 G2 4.10 C3

F1 4.00 F3

3.90 F2

3.80

3.70 0.50 0.55 0.60 0.65 0.70 0.75 0.80 0.85 STDEV Table 3.18 Exit profile of students enrolled in the University of Cincinnati Mechanical Engineering Program Data Collected 2005/06 -07/08. For parameter definitions see Table 3.10.

Chapter 3 Generating Effective Reports 65 0.35

B2 0.30 J4

B4 F1

D2 0.25 B3 D1 J3 G3 A3 G2 C2 A2 H1 0.20 H2 C3 J1 J2 F3 A4 H3 MEAN K2 B1 E2 C1 K3 D4 E1 0.15 G1 K1 I1 E3 E4 D5

D3 0.10 F2 I3

I2 0.05

0.00

-0.18 -0.15 -0.13 -0.10 -0.08 -0.05 -0.03 0.00 0.03 0.05 STDEV

Table 3.19 Delta MSM profile of students enrolled in the University of Cincinnati Mechanical Engineering Program. Data Collected 2005/06 -07/08. For parameter definitions see Table 3.10. D1 Accountability, D2 Self Confidence, Reflections J4 Time Allocation and B4 Idea Genera- The principles of the feedback process tion, all exhibit a healthy combination of are relatively straight forward, and can increased cohort uniformity and performance level. F1 Motivates Others and F2 be mechanized with relative ease. The Gives Direction exhibit an increasing stan- challenge for future development will dard deviation. The data suggest a creative be to present the information in a non- technology oriented rather than manage- threatening way using existing adminis- ment oriented program. On I3 Under- trative processes. Preliminary data show stands Political Implications, I2 Respect stark differences between programs, as Diversity and K4 Attendance the program well as a strong consistency from year to does not seem to have much of an impact. year. The future of feedback driven higher It is noteworthy that two of these three education, will be to point by point, move parameters are exceptionally high already student performance from one quadrant to on the entry level profile. another.

66 Chapter 3 Generating Effective Reports Chapter Four Case Study: Architecture By Anton Harfmann, Michael Zarestsky, Catherine Maltbie, Alex Christoforidis, Vasso Apostolides

“In theory, there is no difference between theory and practice… but in practice there is.” — Marvin Malecca, FAIA 2008

Motivations for Description of the School, Joining the Project Architecture Program and Curriculum At first glance, the premise of using co-op The Design Futures Council and the employer evaluations to drive curricular journal, DesignIntelligence, conduct an change is entirely offensive to academics annual employer poll to gauge the schools who see their role as teaching students to that best prepare students for profes- become employers themselves rather than sional practice. Consistently over the years remaining employees. The concept of hav- the UC Architecture program has been ing employers indirectly drive curriculum ranked at or near the top when it comes also seems self-serving, as their concerns to preparing graduates for practice along and evaluations are assumed to be driven by with the likes of Harvard, Yale, Princ- practice as it currently is, while faculty are eton, Columbia and Cornell. The School compelled to prepare students for a practice of Architecture and Interior Design at environment that will be. This dichotomy the University of Cincinnati advances the creates an interesting challenge—to search profession of architecture while it prepares for an overlap between an academic area students for critical engagement with within the curriculum that could benefit practice. In light of constantly chang- from modifications while simultaneously ing conditions for practice, the program addressing a concern uncovered by pars- seeks to increase insights and abilities in ing co-op employer feedback. Ideally, any every student. Students learn creative and adjustment made in response to feedback technical skills in the studio environment, should be mutually beneficial to the stu- taught by a passionate faculty whose goal dents’ overall architectural education and is to guide students as they grow both to their co-op employer. Suspending skep- intellectually and professionally. ticism about the premise of using co-op The University of Cincinnati’s College employer feedback as a force to drive cur- of Design, Architecture, Art, and Planning ricular change we set out to identify an area (DAAP) requires co-op participation all its where intervention would satisfy the aca- professionally oriented degree programs. demic concerns as well as address an area Specifically, in the School of Architecture of apprehension expressed by employers. In and Interior Design (SAID), co-op is a doing so, we offer a description of the cur- curricular requirement for students in the rent architectural academic and cooperative bachelor of interior design (six alternating education curricula. co-op quarters at three different compa-

Chapter 4 Case Study: Architecture 67 nies), the bachelor of science in archi- planner. About three fourths of all stu- tecture (four alternating co-op quarters dents are employed by architecture, inte- at two different companies), and master rior design, and urban design companies, of architecture (two alternating and two the rest being employed at design-related sequential co-op quarters). Working at a multidisciplinary firms, design build firms, variety of companies within the profes- construction and development organiza- sions of architecture and interior design, tions, public agencies, and commercial students have the opportunity to explore a companies. The PP database hosts in excess multitude of career paths, as well as work of 1,125 architecture co-op employers. Four- environments and locations to be well hundred thirty-three employers are active equipped as they take the first steps in having hired co-op students within the last their professional careers. three years. In Cincinnati alone PP has 166 The co-op program is managed by companies, with 60 active employers that the Division of Profession Practice (PP) have hired co-op students within the last which comprises of 22 full time faculty three years. For Ohio these figures are 351 members responsible for preparing and versus 109; for New York City, 85 vs. 55; for advising students as they begin their pro- Boston, 14 vs. 6; for Chicago, 48 vs. 19; for fessional careers. The co-op faculty also Washington DC/Bethesda, MD area, 31 vs. directs all communications between the 17; for Seattle, 17 vs. 12; for San Francisco, Division of Professional Practice, employ- 40 vs. 15. While over the last three years ers participating in the program, and the students were placed in the U.K. 5, France respective academic programs with which 1; Denmark 1; The Netherlands 2; Germany they are associated. Cooperative education 5; and Spain 2. During the year 07-08 PP for students in the bachelor of science in had 226 undergraduate architecture place- architecture and master of architecture is ments of which only 39 were in companies the responsibility of three faculty members, located in Cincinnati. The PP salary survey including two who are exclusively dedicated shows that undergraduate architecture to these majors. The graduate program is students earn on average between $14 and under the sole responsibility of one full time $18 per hour which is enough to cover their faculty who is a registered architect and expenses during the co-op quarter, poten- certified planner, while the undergraduate tially allowing the student to save for the program is led by a full time faculty who is following quarter. Graduate students earn also a registered architect and a professional between $15 and $20 per hour.

Figure 4.1. Structure of the Master of Architecture curriculum in SAID.

68 Chapter 4 Case Study: Architecture In 2000, the School of Architecture and The fourth year serves as a capstone Interior Design at the University of Cincin- during which students complete an nati used the conversion from a five-year intensely integrated Structures/Construc- bachelor of architecture degree to a five- tion/Environmental Technology studio year master of architecture degree as an that spans two quarters and integrates opportunity to reevaluate and reinvent the coursework in all three technical areas that curricula. The reformulation challenged comprise its title. Students receive a non- old habits and methods of delivering design professional bachelor of science degree at skills and technical knowledge. The result- the end of year four and thereafter apply to ing curriculum, presented in Figure 4.1, a professionally accredited master of archi- suspends conventional building design tecture graduate program to continue education for the first year, focusing instead their studies. on intensive skill building and developing design vocabulary and compositional skills. Students emerge from the first year The Curricular Reform Effort and Results with general drawing, computer modeling/ The Architecture FIPSE team conducted not rendering, physical modeling and composi- one but two curricular reform efforts. The tional abilities. first effort occurred in the winter of 2006 Based on a concept introduced by past within two sophomore level courses, Con- program director Daniel Friedman, PhD struction and Introduction to Computer the first quarter of the sophomore year Aided Design. The second effort began immediately casts students into an “immer- in the fall of 2006 in the Environmental sion quarter”—an experience in which Technology I class and continued on a more students are engrossed in a building design limited scale in Environmental Technology for the entire academic term. The experi- II taught in the spring and summer quar- ence is coupled with an intense coordina- ters of 2007. Both cases are described in the tion of all other course work. By suspending order they occurred. the design of a building during the students’ first year the SAID is able to capitalize on the high level of student energy and antici- Case 1: Instruction of Computer pation, introducing them to the choreo- Aided Design and Construction graphic and integrative nature of design. We begin by setting the pedagogical and Immediately following the intensity and curricular stage of the architecture program holistic approach of the immersion studio, in the School of Architecture and Interior the winter quarter of the sophomore year Design at the University of Cincinnati. The returns to a more conventional pattern of review of the program is followed by an course work with students enrolled in five analysis of the winter quarter sophomore discreet courses; Design Studio, Construc- curriculum as an area where modifications tion Technology, Architectural History, might enhance the overall pedagogy as Structures and Computer Skills. well as address employer concerns focusing During the middle years of the pro- primarily on two specific courses—namely, gram, students begin the sequence of tra- the sophomore Construction class and the versing between academic quarters which introduction to Computer Aided Design include design studios and elective courses, (CAD) class. Analysis of co-op employer followed by quarter-long cooperative educa- evaluations were used to verify that this tion work assignments. area of the curriculum did indeed warrant

Chapter 4 Case Study: Architecture 69 some type of intervention and that adjust- ranging from an on-line self-taught course ments made to these two courses would to a structured course with its own assign- directly benefit the students in both aca- ments. Furthermore, out of necessity, the demic and practice realms. The curricular CAD class focuses primarily on skill devel- corrections identified through this process opment and does not offer any exposure to are outlined, comparing examples of work the use of the software to produce construc- from both pre and post altered courses. tion documents. Finally, comments from the graduate Parallel to the CAD class in the winter assistants for the class who took the courses quarter, students take their first comprehen- prior to the adjustments are used as initial sive construction class. This course focuses feedback while we wait for more formal on exposing students to various types of responses from employers to verify whether building construction, from wood frame, the changes have any effect in the practice to concrete, to steel frame, recognizing that environment. this is also a critical area of knowledge that The early sophomore year focus on spe- can increase a student’s success in co-op. cific architectural knowledge and technical The Construction class and the CAD class skills prepares students for their first co-op have evolved as independent courses over position. As such, there is considerable the years having no interaction between pressure to provide students with as many them. This is in large part results from a skills as possible in order to increase their lack of a standard curriculum, a regular employment opportunities, productivity, faculty member, or a particular method and learning potential. In particular, co-op of teaching this CAD class for years. With employers benefit the most from students a move to offer the class regularly and to who exhibit technical proficiency in con- hire a faculty member to teach it, the SAID struction technology and have the ability to could explore an adjustment in the cur- use computer-aided drafting programs such riculum. An intervention that addresses as AutoCAD. Consequently, the curriculum the isolated AutoCAD class by relating it includes a computer skills class during the more directly with the construction class is winter quarter to give our students critical viewed as a potentially beneficial modifica- CAD skills prior to their first co-op experi- tion to the curriculum. It is also anticipated ence. This course and skill development has that a closer relationship between the two always been viewed as somewhat of a misfit classes might address co-op employer con- in its position in the curriculum since the cerns. Initial evaluation of the statistical and specific skill of using AutoCAD to produce anecdotal feedback from employers revealed conventional 2-D drawings is pedagogically a clear, albeit minor, concern about stu- opposite the intentional and concentrated dents’ limited ability to navigate the com- focus on designing and communicating in plexity of technical aspects of building. Also three-dimensions. However, proficiency in implied was a concern about limitations in the use of AutoCAD is perhaps the single the students’ ability to communicate this most important skill that ensures that complexity in drawing and model form. students can be productively involved in While the data is not abundantly clear, the work based learning during their first co-op overlap between our own academic con- experience. The course has been some- cerns warranted a closer investigation. This what ostracized as the “necessary evil” of led to the development of additional and the sophomore year, and has experienced much more specific questions about the several iterations for delivering instruction technical proficiency of co-op students in

70 Chapter 4 Case Study: Architecture Figure 4.2 Survey form of supplemental questions. their early years. Four detailed questions Instrument I) form received relatively high were developed and added to the stan- marks, employers were much more critical dard “employer evaluation” as a means to when answering the more detailed ques- confirm our suspicions that the student’s tions. The responses verified our suspicions knowledge of building technology and that students struggle in specific areas of their ability to represent complex construc- building construction and detailing as tion assemblies are indeed areas of mutual well as developing proficiency in technical concern. (in the context of the FIPSE project drawing and representation using digital Developing a Feedback Loop for Curricular technology such as AutoCAD. The sum- Reform, the instrument was characterized mary analysis of the four detailed questions to belong to the Assessment Instrument is shown in the chart in Table 4.1. II category.) The questions were posed The supplemental questions confirmed to co-op employers employing first time that both construction technology and sophomore students. The supplemental web- CAD skills are areas where laying a stron- based questionnaire is shown in Figure 4.2. ger foundation in school could return While the general category of technol- enormous short and long-term benefits in ogy in the overall evaluation (Assessment practice. More importantly, the overlap with

Table 4.1 Analysis of supplemental questions.

Chapter 4 Case Study: Architecture 71 curricular concerns makes any modification to these courses mutually beneficial. It is important to note that all supervisors did not answer all questions. Consequently, the “n,” at the bottom of the chart is the number of respondents that answered all questions. It is also worth noting that the relatively small “n” results from the fact that we sent the new questions to employers with first time sophomore co-op students only. In a cohort of 75 undergraduate architecture students, approximately half of which are on co-op term at any given time, the maximum n-value could only be about 38. In response to the academic and co-op employer concerns, it appeared obvious to Figure 4.3 Typical wall section from the construction and CAD faculty that an construction class before modification. intervention that results in a better integra- The projects in the CAD class focused tion of course work between the construc- on producing traditional 2-D drawings with tion and CAD classes would be pedagogi- dimensions and notes. Students were asked cally advantageous. Both classes have well to use their own house as their subject and defined course outlines and a clear, logical were expected to produce plans, elevations, sequence of introducing material. The con- interior elevations and sections, and finally struction class follows a sequence begin- plot them out by the end of the quarter. ning with light wood frame construction With limited time in the class, and the need and culminates in steel frame structures to offer sufficient exposure to the depth and interior assemblies. The CAD class fol- of AutoCAD, the class has been unable lows a sequence that gradually introduces to focus on using the software to produce students to more features and complex construction documents. Consequently, the operations in AutoCAD beginning with drawings produced exhibit technical profi- basic concepts and menus, culminating in ciency in the use of CAD with little atten- relating drawings to each other and plot- tion paid to construction. An example of a ting. While there is little to no overlap in typical plan drawing produced in the class the specific lecture material, the projects in is shown in Figure 4.4. both classes do follow a similar path. The construction class has focused heavily on the production of wall sections to exercise the students’ ability to choreograph materials into a functional and poetic assembly. The students produced a series of these wall sections incorporating various materials during the course of the quarter. All sections were hand-drawn and included a three-dimensional sketch overlay of the assembly. An example of one of the four Figure 4.4 Typical drawing produced in wall section projects is shown in Figure 4.3. CAD class before modification.

72 Chapter 4 Case Study: Architecture What appears clearly when comparing 2. The graduate assistants are able to use the independent work from the two courses the CAD lab time to reinforce lecture is that merging the two assignments into a topics and to answer both construc- single project for both classes would bring tion and CAD questions. This essen- greater meaning to the CAD project and tially provides a lab for the construc- bring a much needed skill development tion class without compromising aspect to the construction class. Rather than the CAD requirements and brings a discrete, unrelated exercises, a holistic single construction aspect to the CAD class, project could also enhance the students’ which more closely emulates the prac- ability to grasp construction assemblies tice environment. and improve their ability to represent them. Combining projects allows the structure 3. With better integration, access to and schedule of the courses to remain graduate assistants and common project largely unchanged with construction lecture requirements, the construction and meeting twice a week and the CAD class CAD classes are able to add depth and meeting once a week for a one-hour lecture, content to their existing course outlines. and once a week for a two-hour lab. Since In particular, the construction class is students are required to take both classes able to introduce the concept of Build- they can be divided equally between four ing Information Modeling (BIM) to graduate assistants who serve both the the curriculum and include a strong construction and CAD classes. This union 3-D component to the project and the of the two courses around a single shared CAD class is able to introduce students project has resulted in three distinct and to alternative software for producing immediate benefits; construction documents. 1. The focus of the CAD class has shifted The building chosen for the combined toward a more meaningful project that project is the Half-House designed by John directly relates to another class the stu- Hejduk. The house consists of three pri- dents are taking. This also reduces the mary shapes, a triangle, semi-circle and amount of “busy work” as students are rectangle connected by a common circula- fulfilling requirements for two classes tion spine. The plan and axonometric view with one effort. of the house is shown in Figure 4.5.

Figure 4.5 Schematic plan and axon of Hejduk’s half-house.

Chapter 4 Case Study: Architecture 73 The shared project requires students to offers an unprecedented opportunity to develop a set of documents describing the offer feedback to students during the proj- construction of the building using both ect. Under the separate class structure there 2-D and 3-D electronic representation. To simply was insufficient time to incorporate ensure that students have unique and broad this valuable teaching method. exposure to various construction types each For the final submission, students are portion of the building is assigned vari- required to correct their drawings and ables for the structural system, materials, include a detailed 3-D digital model of the dimensions and foundation. For example, steel frame portion of their building. The one student may be assigned a steel frame 3-D model is a new requirement and intro- over a slab on grade for the semi circle while duces students to the concept of Build- another may be assigned a wood frame over ing Information Modeling (BIM). While a full basement for the same portion of the the vast majority of practice continues to building. The project begins with students utilize 2-D drawing for conveying con- producing a simple 3-D model of the form struction intent, the emerging 3-D Build- and a 2-D plan of the first floor according to ing Information Modeling strategy will their unique parameters. This is followed by radically transform practice by replacing the production of two wall sections through drawings with digital models. By consoli- the wood frame and masonry bearing dating projects, the course can fulfill its portions of their building. By mid-term, obligation to prepare students for practice students submit plots of the foundation in the future while simultaneously giving plan, first floor plan, elevations and two wall them the skills they need to practice in the sections. The midterm plots are “red-lined” present. Students are introduced to the BIM by the four graduate assistants and returned concept in lecture where the same Half- to students for corrections. An example of a House building is used as a case study to portion of a red-lined submission of a sec- illustrate construction principles of wood tion and partial red-lined plan is shown in frame and masonry bearing walls. The Figure 4.6. While the drawings in the figure “lead by example” of modeling during class may be confusing to most non-architects, sets students up to model the steel section the intent of including these images is to of their own project. Furthermore, having point out that the new, combined course students navigate between 2-D and 3-D will hopefully increase their understanding of complex assemblies as well as their ability to represent them in 2-D or 3-D. An example of part of the 3-D class model of the Half- House is shown in Figure 4.7. Evaluating the final projects from the first offering of the modified classes reveals very interesting academic benefits. While AutoCAD skills may have improved only slightly, the difference in the work performed for the construction class resulting from the merger is astounding. The quality of drawings and depth of construction understanding exhibited in them far Figure 4.6 “Red-lined” mid term submission. exceeds the level of work completed before

74 Chapter 4 Case Study: Architecture tion reveals that the student is beginning to understand the complexity of his building. His acknowledgement of the sloped site is evident in the stepping foundation wall. Furthermore, the consistency between the plan representation and the elevation with respect to the various foundations types indicates a three dimensional understanding of the complexity of construction and design. While the wall section in Figure 4.10 is technically on-par with sections produced Figure 4.7 In-class detailed half house model. in previous years, it stands apart as it is not a stand-alone project but rather a section describing a particular portion of the Half- the merger of the classes. The inclusion of House project. Furthermore, it successfully the 3-D component modeling of the steel integrates the use of AutoCAD as part of frame portion of the building yields even the process giving the student more oppor- greater benefits. From the 3-D model it tunity to practice their skills while solving is abundantly and immediately apparent construction and design related issues. whether a student understands the basic principles covered in class since the accurate 3-D modeling process reveals issues that cannot be illuminated in a 2-D drawing. Figures 4.8, 4.9, 4.10, and 4.11 illustrate the outcomes of the pilot shared comprehensive project for the courses and show the greater level of detail of the work. All images are taken from Mitch Duyser, a sophomore student who was enrolled in both classes. Figure 4.9 Elevation. Figure 4.8 shows Mitch’s foundation plan of the Half-House. The project shows its depth and holistic nature in the elevation and wall section in Figures 4.9 and 4.10 respectively. The eleva-

Figure 4.8 Foundation Plan. Figure 4.10 Wall section.

Chapter 4 Case Study: Architecture 75 Figure 4.11 3-d half house assembly.

Perhaps the most compelling evidence - GA’s do reviews the week before of enhanced learning is the three-dimen- midterms – 1-on-1 critique; this is the sional model describing a portion of the first time these students are exposed Half-House project. Figure 4.11 includes to such a process preparing students two images of a series of 3-D images for the most typical way they receive extracted from the final project showing feedback on the job. the complexity of the assembly recogniz- - The students can get immediate ing the construction sequence of building feedback systems and components. Once again, the focus on these complicated images - The GA’s can be used for instruction should be less on the actual information • GA’s stay with the students throughout contained in the model but on the fact that the lab time. sophomore students were able to produce • The students also have each other to use this level of complexity in the short time of as resources – “develop a learning one quarter. This would not have been possible in the previous split course scenario. community” First-hand accounts from the four • Students seem to be better prepared graduate assistants also confirm that the • Students are working on more than one course corrections significantly improve drawing at a time the experience in both classes. They also report that the sophomore students’ depth of understanding of construction princi- Case 1: Results - Computer Aided Design ples is superior when compared with their and Construction Reform own sophomore experience. Comments The two methodologies were compared recorded by an external evaluator further during the academic year of 2004/05 and confirm the initial success achieved by the 2005/06. Group I followed the traditional course adjustments, a few of which follow: curriculum in which the Construction and the CAD course were taught separately, • GA’s are more accessible: and Group II followed a curriculum hav- - Available four hours per week to talk ing interaction between the two classes. The about AutoCAD and construction results of the employer evaluations, follow-

76 Chapter 4 Case Study: Architecture Group I Group II Param: A b C d A b c d Mean 4.27 3.78 3.52 4.08 4.26 3.81 3.56 3.91 STDEV 0.72 0.85 0.82 0.86 0.66 0.76 0.76 0.79 n 26 27 25 25 58 53 50 57 Table 4.2 Performance data of groups I and II.

ing the course are presented in Table 4.2 and systematic difference in the standard devia- Figure 4.13. Group I has an n value of 27, tions of all ratings. Merging the delivery whereas group II is more than twice as large. of the classes results in decreased standard The reason for the discrepancy is that the deviation across the board (increased uni- treatment group is comprised of two sec- formity of student performance) for all four tions whereas the control group only holds measured parameters. The confidence level one section. that the change is not attributed by chance Figure 4.13 illustrates the story of the is approximately 70 percent. The low confi- impact of the changes in teaching method- dence level can be attributed to low n-values. ology. It is clear that the changes have had Still the results are certainly not coun- only a minor impact on means. Parameters ter intuitive. They support a very simple a, b, and c exhibit statistically negligible hypothesis. We have focused on addressing changes between 0.01 and 0.04 units. Even the educational process, and received a more the changes in parameter d, - 0.17 units is uniform result. The change in the pedagogic not statistically significant. It is fair to say process indicates that we have reduced the that the curriculum revision resulted in no level of outliers. The histogram of co-op detectable difference in student performance rating distributions presented in Figure 4.14 level. There is however a remarkable, and illustrates the situation.

Architecture; CAD & Construction Before & After Curricular Change 4.4 a 4.2 4.0 d

3.8 b

3.6 c MEAN 3.4 Group I 3.2 Group II 3.0 0.60 0.70 0.80 0.90 STDEV

Figure 4.13 MeanStandard Deviation Matrix of Groups I and II. a, w, c, and d refer to measured parameters.

Chapter 4 Case Study: Architecture 77 In the first curricular change, the tunately comes at the expense of five’s, courses initially identified were disjointed which is compensated by a slight reduction computer and construction classes. Evalu- in both two’s and three’s. The arithmetic ations from the employers lead us to recon- average between the aggregate of the given sider how both courses were taught. A new ratings shows remarkable stability as it curriculum that combined them around a moves by only 0.01 units (from 3.91 for single project was proposed to address the Group I, to 3.90 for Group II). shortcomings identified by the employers. One can only speculate why the change The project addressed the complaint that in standard deviation, combined with a students entering their first coop position stable mean, takes place. It is natural to could be better prepared in terms of com- hypothesize, that applying a more student puter skills as well as overall construction centered pedagogy leads to a reduction in understanding. Implementing the change low performers. In a less supportive envi- meant convincing fellow faculty members ronment, however, control oriented stu- that using coop employer feedback to drive dents might tend to over perform, in order curriculum changes did not represent sell- to secure their GPA. ing out. Also, constructing the curriculum The increase in the process level, seems to satisfy employer concerns as well as the to have a payback in outcome uniformity. larger pedagogical goals of the curriculum The results promote moving the discussion was challenging. to a philosophical level. Should an institu- Figure 4.14 shows a sharp increase in tion promote uniformity at the cost of a the percentage of fours as a result of the loss in top student performance, or is there introduced pedagogy. The increase unfor- value in applying a survival of the fittest

50% Group I 45% Group II 40%

35%

30%

25%

20%

15% Percent of Grades Given

10%

0% 1 2 3 4 5 Work Performance Rating

Figure 4.14 A histogram of all given grades in percent. Group I totals 103 filed grades, whereas group II totals 218.

78 Chapter 4 Case Study: Architecture culture that benefits the few? What is the Sustainability is primarily addressed in institution’s role as a provider of services to the Environmental Technology sequence, both students and employers? What is the with is part of the core curriculum in every social consequence of diverse performance NAAB-accredited architecture program. levels? This course content includes specific data on climate, energy, thermodynamics and heat flow, mechanical systems, lighting and Case 2: Instruction of Sustainable Design acoustics. In recent years, NAAB has added Based on qualitative responses from earlier required content on sustainability through- co-op employer feedback, one area in which out the architecture core curriculum and co-op students needed improvement was in many schools have added some of this their understanding of environmental issues content in the ET courses. and specifically in the burgeoning subject All students in the undergraduate archi- area generally referred to as “sustainabil- tecture and interior design programs take ity.” Sustainability is used in many different two required courses that introduce them to contexts, but the most commonly under- environmental design principles: Environ- stood definition comes from the Brundtland mental Technology 1 (ET1) and Environ- Report of 1987 also known as Our Com- mental Technology 2 (ET2). The sequence mon Future. In this document, sustainable introduces the students to the topic of development is defined as “development sustainability. The course is delivered in a that meets the needs of the present without lecture format to a mixed cohort of students compromising the ability of future genera- enrolled in the undergraduate architecture, tions to meet their own needs.” graduate architecture and undergraduate It is also understood that the built interior design programs. These courses world, specifically architecture, is respon- include exams and projects that are com- sible for as much as half of the energy used pleted both individually and in groups. in the United States. The majority of this In the SAID design curriculum, the energy usage in the U.S. is through the content of ET1, ET2 and ET3 has been in use of non-renewable energy sources and flux in recent years, but most recently, ET2 is, therefore, inherently not sustainable. has included specific conceptual and techni- As a result, every architecture school in cal content on energy, heat, thermodynam- the United States of America is presently ics, and electric lighting. Sustainability was addressing the need to integrate issues of not specifically addressed in ET courses at sustainability throughout the architecture UC because of the amount of required tech- curriculum. At the University of Cincinnati, nical content deemed critical by the existing comments in the co-op employer feedback ET faculty. showed recognition of that growing trend in architecture and questioned whether The recent changes in the curriculum students were amply prepared for their that were intended to focus to a greater role in sustainable design. Coincidentally, extent on sustainability precipitated a need the SAID made a significant commitment for change in ET2. This, however, has not to increase the integration of sustainable been possible without either adding addi- principles within the curriculum in autumn tional credit hours or removing some exist- 2006 with a faculty retreat that focused on ing content. this issue. Additionally there was a faculty The ET2 v2 course taught in summer hired specifically within this subject area. 2007 included approximately 80 percent of

Chapter 4 Case Study: Architecture 79 Autumn 06 Winter 07 Spring 07 Summer 07 Autumn 07 Sect A: Co-op (Q1) ET2 v2 Co-op (Q2) ET 1 Sect B: ET2 v1 Co-op (Q1)

Table 4.3 Graphic representation of course delivery and measurement (Q1, Q2, and Q3 refer to co-op quarters 1, 2, and 3.).

the technical content from the previous with architecture employers to develop a course. There was some change in lecture set of questions on sustainability to assess content, though the greatest change was employer perception of student knowledge in the assigned projects. The integration of on sustainability. The resulting questions sustainable principles was largely achieved were as follows: through individual and group projects that 1. Understands how climate and location focused on application of the technical affect design. content within design projects. The first project in the new ET2 cur- 2. Understands how sustainable principles riculum was a simple on-line assessment inform design. of each student’s “ecological footprint 3. Understands how to apply the quiz” that used inputs from the student principles of embodied energy and to provide a numerical description of the indoor air quality to material selection. number of earths that would be needed to provide our resources if everyone on the 4. Understands principles of day-lighting earth used the same number of resources and heat gain. (www.myfootprint.org). The students were 5. Understands principles of energy effi- then asked to write a paragraph describing ciency and water conservation. what they would change if they intended to These questions were included in the decrease their impact. The average number employer feedback surveys that are com- of earths used was 4.6 earths per student. pleted following each co-op students’ The second assignment involved an employment at an architecture company. assessment of the total annual energy The questions are assessed on a five demar- usage of their favorite appliance and an cation Likert scale, as presented earlier in investigation of the financial and physical this chapter. cost of that energy usage. These types of ET1 is delivered in the autumn of the assignments provided a new context for sophomore year (the students’ fourth quar- the discussions of mechanical systems and ter at DAAP). The students are randomly energy usage in architecture. The technical split into two sections in spring quarter. content was decreased, but the application Half of the cohort embark on their first of the content was increased through this co-op assignment in spring quarter and curricular change. take ET2 when they return summer quar- All SAID students share the same ter (Section A). The other half takes the course load through their first year at second ET 2 course in the spring and has DAAP. The course schedule for the second their first co-op in the summer (Section B). year is presented in detail in Table 4.3. The table uses the encoding as follows to In autumn of 2006, Professors Anton reflect how sustainability was integrated in Harfmann and Michael Zaretsky consulted the curriculum:

80 Chapter 4 Case Study: Architecture 1. ET1 = Environmental Technology 1 As we can see the two sections are with sustainability integrated into the never fully comparable, as Section A in curriculum a co-op situation is ahead with regard to 2. ET2 v1 = Environmental Technology 2 practical experience, whereas section B is with traditional curriculum ahead with regard to coursework. 3. ET2 v2 = Environmental Technology 2 with sustainability curriculum Case 2: Results - Sustainable Design Reform The data is plotted parameter by parameter The objective of the study is however to in Figure 4.15. measure to what extent the different delivery of ET 2 affects the behavior students At an initial glance the data seems to exhibit during co-op performance. The support that the students demonstrate the study is based on two test groups that are best results after ET1. The results obtained measured three times as presented below: via ET2v2 seem to be somewhat stronger than ET2v1. When subjected to a one way • The work performance of Section A ANOVA the data returns close to statistical was measured twice: significance for question; 5 Understands – Once after they attended ET 1 principles of energy efficiency and water – Once after they attended ET2 v2 conservation (Conf. 86.3 percent) and 2 Understands how sustainable principles • The work performance of Section B inform design (Conf. 80.3 percent). With was measured: regard to these two parameters we can be – Once after having both ET1 & ET2 v1. in excess of 80 percent confident that the

4.80

4.60

4.40

4.20

4.00

MEANS 3.80

3.60

3.40 A / ET1 3.20 B / ET1 ET2v1 A / ET1 ET2v2 3.00 1 2 3 4 5 MEASURED PARAMETERS

Figure 4.15 Means pertaining to questions one through five presented graphically.

Chapter 4 Case Study: Architecture 81 MEAN STDEV n Aver. Stdv exp t-test STD Quest. A ET1 b ET1 ET2v1 A ET1 b ET1 ET2v1 A ET1 b ET1 ET2v1 ∆ ∆ 1 tl 2 tl c90% 1 4.53 4.31 0.61 0.60 19 16 -0.21 -0.01 81% 63% n 2 4.30 4.06 0.73 0.80 20 18 -0.24 0.07 81% 62% n 3 4.50 4.08 0.67 0.76 12 13 -0.42 0.09 90% 79% n 4 4.47 4.00 0.62 0.87 17 17 -0.47 0.24 94% 88% y 5 4.64 4.06 0.50 0.80 14 18 -0.59 0.31 98% 96% y Table 4.4 Results of expanded t-tests and F-tests when comparing groups A/ET1 to B ET1 ET2v1.

MEAN STDEV n Aver. Stdv exp t-test STD Quest. A ET1 b ET1 ET2v1 A ET1 b ET1 ET2v1 A ET1 b ET1 ET2v1 ∆ ∆ 1 tl 2 tl c90% 1 4.31 4.47 0.60 0.72 16 17 0.16 0.12 73% 45% n 2 4.06 4.35 0.80 0.79 18 17 0.30 -0.02 84% 68% n 3 4.08 4.33 0.76 0.78 13 12 0.26 0.02 77% 54% n 4 4.00 4.36 0.87 0.74 17 14 0.36 -0.12 86% 72% n 5 4.06 4.33 0.80 0.78 18 12 0.28 -0.02 80% 60% n Table 4.5. Results of expanded t-tests and F-tests when comparing groups B/ET1 and ETv1 to A ET1 ET2v2.

MEAN STDEV n Aver. Stdv exp t-test STD Quest. A ET1 b ET1 ET2v1 A ET1 b ET1 ET2v1 A ET1 b ET1 ET2v1 ∆ ∆ 1 tl 2 tl c90% 1 4.53 4.47 0.61 0.72 19 17 -0.06 0.11 59% 17% n 2 4.30 4.35 0.73 0.79 20 17 0.05 0.05 58% 15% n 3 4.50 4.33 0.67 0.78 12 12 -0.17 0.10 69% 38% n 4 4.47 4.36 0.62 0.74 17 14 -0.11 0.12 66% 31% n 5 4.64 4.33 0.50 0.78 14 12 -0.31 0.28 84% 69% y Table 4.6. Results of expanded t-tests and F-tests when comparing group A at two different points in time.

difference in results was not random. The extremely high. The employers perceive remainder of the parameters shows confi- that students have a very high under- dence levels between 70 and 25 percent. standing of these principles of sustainable The co-op employee questions from the design. However the “n” values are quite architecture and interior design employees low, which suggests that many employers tend to be quite high in comparison with chose not to answer these questions. We other schools. For the sustainability ques- can assume it is either because they were tions, there was no question in any of the not interested or they were not knowledge- groups tested that was below 4.0. Tables able on issues concerning sustainability 4.4–4.6, illustrate the differences in light of themselves. expanded t-tests (assuming an assessor sys- The next significant result is that all of tematic uncertainty of 0.33 units) and F-tests. the scores actually dropped from ET1 to In assessing the data, what is most ET2. In ET1, students were presented with noticeable is that all of the scores are concepts of sustainability in a broad sense.

82 Chapter 4 Case Study: Architecture In ET2, students are learning principles Impact of the Project and applications of thermodynamics, day- Both of the presented cases focusing on lighting and electric lighting. In the new such different topics as the instruction of version of ET2 v2, there is an increased computer aided design and construction as focus on passive design and application of well as sustainability show a strong poten- these principles in practice. There is less tial for the use of work data in the continu- focus on the scientific computation behind ous improvement of curricula. The projects the principles, but more focus on their involved using coop employer feed back to relationship to design and sustainability. identify areas of the curriculum that could In assessing why these values dropped, we benefit from a curricular overhaul. may be informed by the student evaluations from these courses. In the original It is clear that the use of the principle version of ET2, the comments suggest that is more straightforward, whenever we are students are overwhelmed by the amount dealing with an area in which the concepts of complex data that they are expected to are familiar to co-op supervisors. Whenever learn. this is not the case, the results will suffer from low n-values, making it hard to dem- We can see in the third comparison onstrate statistical significance. The results between ET1 and ET2 v2 that they are also support that whenever care is given to nearly statistically equivalent with the instructional delivery, the performance of exception of the fifth question on energy students become more uniform. efficiency and water conservation. This data leads us to believe that there is some Based on the results of this initial improvement in the development of the effort it seems that listening to the voice of ET2 course in regards to sustainabil- employers for curricular advice can indeed ity although there are still issues to be impact the students’ academic and profes- addressed. We can see that according to sional proficiency. What is not evident from employers, students in ET2 that incor- this chapter is the difficulty in convincing porate sustainability into the curriculum faculty to participate. The suspicion that are showing a greater understanding of faculty would be wholly offended by any sustainable principles than those taking process that allowed employers to affect the previous version of ET2. The measure- curriculum was absolutely true. This hurdle ments are however, not strictly comparable, was actually overcome while working on as some students enrolled in ET2v2, had the project. Temporarily suspending skepti- one more quarter of co-op under their belt, cism allowed the work to progress and it as compared to the control group enrolled soon became apparent that employers were in ET2v1, However, we need to address much more interested in larger educational the fact that employers’ perceptions of a issues than in making specific self-serving student’s understanding of sustainability is demands. Based on this effort several sug- not increasing from ET1 to ET2. gestions can be made to help other academics considering a similar strategy: We are using this data to assess the successes of ET1 and consider how to better • Hold roundtable discussions with incorporate this teaching methodology into employers early on to get initial feed- ET2. This is still being discussed within the back on curricular issues. It was quite SAID curriculum committees but this data a surprise to learn that employers are has been extremely valuable for us in this far more interested broad educational process. questions instead of satisfying their

Chapter 4 Case Study: Architecture 83 specific needs. This became clear dur- Getting a course change approved ing roundtable discussions with sev- by the curriculum committee is always a eral employers. Their enthusiasm and burden. In one case to overcome this, we dedication to education is immediately only changed the sequence of material apparent and will serve to alleviate being covered in two courses and the two most faculty apprehensions about faculty worked together to develop a single including their feedback as part of cur- shared project that satisfied the educa- ricular discussions. tional objectives of both classes. In another case the revision rested in the hands of • Ask broad level questions instead of only one faculty member. This allowed us specific questions. Faculty members to avoid a formal curricular review pro- are most concerned that employers will make specific curricular demands such cess. When the curricular revision process as “students need to know how to use rests with the individual faculty member XYZ software.” While some employers or a partnership between faculty members will make such specific requests most it allows them to listen to minor com- employers reported that they can eas- plaints about the curriculum from outside ily teach software XYZ if the student sources. Minor adjustments in a course or understood principles ABC. Conse- courses can have major impacts. Faculty quently, a question such as “should must approach this type of effort with students be familiar with principles of an extremely open mind and thick skin. ABC?” is considerably less threatening Reading evaluations and listening to criti- to a faculty member and gets at the root cism is hard work and even more difficult academic question as well. to not take them personally. One must be willing to reflect seriously on old habits • Include students in the development of and assumptions in order to see alternative survey instruments and course modifi- strategies to teaching. cations. While students may not be able Curricular changes brought up as a to appreciate larger pedagogical issues result of the project were consequently they are the ones regularly travers- positively reflected in student skill devel- ing the theory-practice chasm. Their opment and performance. Student prepa- insight during the process can stimulate ration is linked to performance on co-op, fresh ideas and help faculty develop an and since employer expectations affect how appreciation for practice by serving as student performance is evaluated, feedback a common link between the academic from employer expectations and evalu- and practice realms. In particular, ations led to curricular changes. Close involving graduate students who are a collaboration between assigned faculty product of the curriculum is extremely members from the School of Architecture helpful since they have a mature and and Interior Design, Professional Practice experienced point of view that can yield faculty and statisticians taught us that tremendous insight and are closer to student preparation prior to the first co-op faculty than employers. assignment and employer assessment of • Change is good. While there is consid- performance are correlated. Having (1) a erable investment in altering curricu- strong and dedicated multidisciplinary lum or coursework, the lessons learned team, (2) readily available data, (3) the far outweigh the effort to institute the time to bring up curricular changes and changes. test these changes during the course of the

84 Chapter 4 Case Study: Architecture research, and (4) the right mechanism to to modify the revised courses and plan to effectively engage the industry were the intensify the relationship between the two. key. This was an invaluable study substan- As the curriculum continues to develop, tiating the impact of cooperative education we are able to quantify the effects of those on a student’s overall education. By includ- changes through the study of co-op evalu- ing a diverse group of skills, from techni- ations. Our access to coop employers and cal to professional to behavioral, the study their evaluations of student performance brought to the fore information that can to help make curricular decisions is some- only strengthen the relationship between thing that very few institutions have. We academic and practice based learning. should be utilizing this access and pursue data much more intensively than we currently are. Future Directions As we continue to collect responses from the sophomore employment experience we will monitor comments and consider additional changes in response to the feedback. We are also addressing some shortcomings with the current modifications; in particular, we are searching for a way to focus more on detailing and developing a better holistic understanding of the relationship between the various technical systems in a building. From various conversations with local architectural firms we are witness- ing a strong desire to incorporate Building Information Modeling techniques into practice. As more firms adopt this paradigm and expect students to be prepared for practice in this environment we will use employer feedback to better prepare our coursework in this area. We are also expanding our review of the curriculum in the area of sustainable design and are using the methods established through this pilot effort to evaluate new curricular adjustments. Base questions on sustainable design are currently being collected from employers and the curriculum has been modified with significant new focus on sustainability. Once students who have completed sustainability-focused courses enter their co-op work assignments we will analyze the effects of the new course material relative to those students under the previous curriculum. We have continued

Chapter 4 Case Study: Architecture 85 Chapter Five Case Study: Business By Marianne W. Lewis, Jenn Wiswell, Darnice Langford, Ann Keeling

“…changes in the context of business, and the unprecedented pace of change, place added pressures on business schools to continuously experiment with their curricula so as to stay breast of these changes…Collected evidence from business school alumni suggests that the most important predictor of business success is management effectiveness. Alumni rate interpersonal, leadership, and communication skills as highly important in the business world, yet they often rate these skills as among the least effective components of business school curricula.” — Report of the Management Education Task Force to the AACSB International Board of Directors (2002)

Motivations for Joining the Project deemed critical to setting students’ expectations, building their institutional commit- At the time the FIPSE project was awarded ment, and developing essential curricular to the University of Cincinnati, the College foundations. Yet the UC business program, of Business was in the midst of a major curriculum revision to meet requirements like most at that time, included only a from the State of Ohio to aid in transfer single business course. Freshmen typically and articulation agreements. Immediately focused on completing their general educa- following the state mandated change, tion requirements across campus. Second, the college embarked on a curricular the accrediting body of business schools change based upon pedagogic enhance- confirmed what many had long known, ments including a drive to increase co-op that personal and managerial effectiveness employer satisfaction and to enhance the – the “soft” skills, such as communication, college’s national reputation. The opportu- leadership, and interpersonal skills – best nity to participate in the project was seen predict business success. These same skills, as a way to help accomplish these goals however, were found to be least developed while building a deeper and stronger rela- in business programs. A targeted and effec- tionship with the Division of Professional tive FYE could emphasize and start to build Practice and the co-op employer base for these soft skills. the College of Business. The First-Year Experience has grown The College of Business sought to lever- into an international movement. It began age the FIPSE project Developing a Feed- nearly 25 years ago, as a community of back Loop for Curricular Reform to guide faculty and administrators started sharing revisions of its first-year curriculum. This resources, in hopes of solving challenges curricular revision, which began in 2002, related to orienting diverse students to was driven by two factors. First, momen- college, engaging them in learning, and tum was building nationally around the encouraging their persistence through importance of providing a cohesive “First- graduation. According to John Gardner, Year Experience” (FYE). Managing this arguably the movement’s most influential transition from high school to college was leader, interest in FYE started during the

86 Chapter 5 Case Study: Business Vietnam War, soon after the Civil Rights take the form of multiple self-report surveys Act, the Voting Rights Act, and the Higher describing student satisfaction and instruc- Education Act brought a new wave of tor approval.” students to higher education. Over time, The UC College of Business recognized the movement’s emphasis shifted from the tremendous retention and learning offering a stand-alone course—typically a opportunities and looming assessment chal- first-year seminar—as the primary vehicle lenges of FYE. As a design team began work for advancing students’ college success to on the curriculum, conversations started broader efforts seeking continuous improve- with the Division of Professional Practice ments to almost every aspect of first-year regarding their FIPSE project. The employer collegiate life, in and beyond the classroom. survey instrument offered a unique means A large body of research has amassed of evaluating students’ soft skills in practice. findings that indicate FYE interventions More specifically, the design team focused can increase retention and student engage- on students’ performance in their first co-op ment. Indeed, several national centers positions, comparing students who had par- now are devoted fully or in part to such ticipated in the new FYE with those prior to research (e.g., Policy Center on the First the curriculum revision. We now detail this Year of College, the Higher Education case, its results and future implications. Research Institute, Center for the Study of Higher Education, Washington Center for Improving the Quality of Undergraduate Description of the Education, and National Survey of Student College and the Program Engagement). Trade publications, academic The College of Business at the University of presses, and a wide spectrum of journals Cincinnati has a nationally ranked under- devote considerable coverage to this effort. graduate program. In 2007, BusinessWeek Indeed, the positive impact of FYE initia- Magazine ranked UC’s College of Business tives upon student persistence and engage- as one of the top undergraduate degree ment is now widely accepted. programs in the Midwest along with the In business programs, FYE programs likes of the University of Michigan, Indiana increasingly are initiated to enable the University, the University of Notre Dame socialization and learning that foster both and Washington University. The UC Col- retention and vital soft-skill development. lege of Business is a high impact research Cox, Schmitt, Bobrowski, and Graham college with a strong tradition of experience called for courses that use an experien- based learning including one of the largest tial approach to bridge more technical, optional business co-op programs in the discipline-specific competencies with the nation and a ground breaking mandatory broader personal and interpersonal skills co-op program that is designed to produce needed in today’s business world. The Cincinnati’s future business leaders. The University of Pennsylvania’s Wharton mandatory co-op program, Lindner Hon- School, for instance, launched a freshmen ors-PLUS, is designed for highly academi- course stressing such areas as leadership, cally motivated students and includes six teamwork, and communication skills. Yet quarters of cooperative education, a fresh- a recent study of that program noted the man year internship experience, at least one challenge of evaluating the students’ subse- quarter of study abroad experience, and a quent success in these areas. According to myriad of cohort based learning experiences Tuleja and Greenhalgh (2008), “At best, data designed to develop leadership skills. The

Chapter 5 Case Study: Business 87 college attracts a quality student body with Management Skills Practicum (MSP), and average SAT scores between 1020 and 1190. the general education requirements (which There are over 2,000 undergraduate stu- typically included English, math, psychol- dents and 500 graduate students enrolled in ogy and/or communication). Interestingly, the College of Business annually. the most valuable outcome of the pilot In 2002, the College of Business was was the development of a learning com- a first-mover in FYE on the UC campus, munity among the involved faculty, staff spurred by two factors. First, unaccept- and administrators. Ongoing conversations able retention rates highlighted the need among the design team spurred creativity to better socialize incoming students. The and higher aspirations for FYE. freshman year was seen a pivotal time for Working through that summer, this setting student expectations and building team expanded the FYE efforts dramati- curricular foundations. FYE efforts, such as cally. First, they determined that all first- Learning Communities, also offered oppor- year business students would participate tunities to engage students, building stron- in a learning community. Second, they ger links between students and between focused on revising MSP to make the most students and the institution. of freshmen’s only course within the college. Second, survey and qualitative feed- This course had been taught lecture-style, back from business faculty and employers two days a week in a large classroom. To suggested that our students lacked profes- enable both socialization and efficiency, the sional development, particularly in the team developed a creative solution. One day personal and managerial “soft skills.” That a week, the class would meet in 70-student same year the AACSB published an exten- sections (combining three LCs per sec- sive study mirroring these concerns. Spe- tion) for lectures and guest speakers. The cifically, the report identified communica- other day, students would meet with their tion, leadership, and interpersonal skills as learning community in breakouts sessions the best predictors of business success. Yet dedicated to cases, discussions and prob- targeted development of these same skills lem-solving applications. In addition, each was scarce in U.S. business programs. LC breakout session was led by a student The college turned to FYE as a means of mentor, serving as the teaching assistant. emphasizing and starting to build these The most innovative element of the core competencies. expanded FYE, however, became labeled Project Fast Track. The design team sought an experiential project that students could The Curricular Reform Effort complete with their LC and that would The College of Business’s FYE initiative demand application of their soft skills. The began with a Learning Communities (LCs) result paired each LC with a leading corpo- pilot project in 2002. A dedicated team of ration to conduct intensive research. Greater faculty and staff, university- and college- Cincinnati is home to some of the world’s level leadership, and funding from an Ohio most admired corporations from Procter & Learning Network Grant fueled the effort. Gamble and Chiquita to Kroger and Gen- Starting small, the team initiated four LCs eral Electric. A comprehensive list of all comprised of 20-25 students each taking sponsors is presented in Table 5.1. courses together throughout their freshman The opportunity to learn about and year. The LCs included a set of common from such business leaders offered a courses: the lone business course labeled tremendous learning collaboration.

88 Chapter 5 Case Study: Business American Financial Cincinnati Bell Cinergy Cintas Convergys Delta E.W. Scripps Federated Fifth Third Bank Frisch’s General Electric Johnson & Johnson Kroger Procter & Gamble Toyota US Bank

Table 5.1 Corporate partners for Project Fast Track.

Participating firms provided freshmen a 83 percent. Such data encouraged the team, “live” context for exploring key business while spurring them to continue examin- concepts and best practices, and oppor- ing areas for improvements. tunities to interact professionally with the The much more challenging goal was firms’ managers in different business func- evaluating the development of students’ tions. The first class of Project Fast Track soft skills. Thankfully, the College had a teams is shown Figure 5.1. unique resource at their disposal: cooperative education. Managed by UC’s Division of Professional Practice, co-op is central Results and Impact of the Project to the University and strongly encouraged Raising student retention between their within the College of Business. Indeed, first and second year, was a central goal of cooperative education was founded at the the College’s FYE initiative. More specifi- University of Cincinnati in 1906. Involv- cally, the development team sought to raise ing alternating quarters of coursework retention dramatically from 78 percent and professional experience, co-op offered in 2002, to a rate more in line with the the potential for employers to evaluate most selective and prestigious colleges on students’ skills. At the end of each co-op campus (Design, Art, Architecture and rotation, students’ supervisors complete Planning at 89 percent, and Engineering at an extensive survey, gauging the student 86 percent). Monitoring College of Busi- performance. ness retention rates closely, the team saw a Recognizing the potential wealth in slow but steady increase. Four years after this employer feedback, the Division of its launch (entering class of 2006, return- Professional Practice had launched an ing autumn 2007), retention had risen to extensive project (funded by FIPSE – Fund

Figure 5.1 The 2003 Business Freshmen in their Project Fast Track Teams.

Chapter 5 Case Study: Business 89 for the Improvement of Postsecondary Edu- dents’ performance on their initial co-op cation) to systematize co-op evaluations so experience? Business students’ first co-op that they might guide ongoing curriculum occurs in their sophomore year, offering development. The employer evaluation form excellent timing to address this question. was revised using employer focus groups to The design team compared the perfor- evaluate the most critical skills. Conducting mance of students who had no First Year an exploratory factor analysis on resulting Experience to those who had completed survey questions, these items loaded onto the new program. The goal of this prelimi- five factors often noted in management nary study was to identify early indica- education research: professionalism, expres- tors of program effectiveness and/or areas sive communication, work management, needing further revision. leadership and teamwork. Factor loadings Analyzing the results, the design team are presented in Table 5.2. Not surprisingly, hoped to see student means increase, the final co-op evaluation survey included indicating improvements in the skill areas, items assessing personal and managerial while standard deviations decrease, indi- soft skills. cating greater consistency among student Leveraging this instrument, the design performance. Table 5.3 addresses the team explored the question: does the new development of the measured factors on a first-year experience positively impact stu- parameter level. Development of the under-

Table 5.2. Exploratory Factor Analysis for Soft Skill Items (F = factor). F1 F2 F3 F4 F5

Shows initiative/is self-motivated 0.87 0.18 0.11 0.04 0.25 Demonstrates a positive attitude toward change 0.83 0.16 0.27 0.19 0.14 Exhibits good listening and questioning skills 0.77 0.29 0.34 0.17 -0.04 Assumes responsibility/accountable for actions 0.74 0.34 0.16 0.24 0.17 Demonstrates flexibility/adaptability 0.68 0.00 0.23 0.11 0.54 Possesses honesty/integrity/personal ethics 0.64 0.36 0.12 0.01 0.43 Understands/contributes to the organization’s goals 0.54 0.28 0.29 0.21 0.41 Allocates time to meet deadlines 0.23 0.87 0.09 0.10 0.18 Sets goals and prioritizes 0.25 0.75 0.28 0.34 0.15 Manages several tasks at once 0.32 0.71 0.44 0.15 0.11 Manages projects and/or other resources effectively 0.29 0.69 0.16 0.32 0.37 Writes clearly and concisely 0.33 0.25 0.83 0.15 -0.04 Makes effective presentations 0.25 0.09 0.73 0.25 0.21 Speaks with clarity and confidence 0.11 0.17 0.66 0.18 0.50 Exhibits self-confidence 0.27 0.39 0.63 0.26 0.32 Motivates others to succeed 0.08 0.28 0.19 0.86 0.05 Gives direction, guidance and training 0.12 0.00 0.16 0.85 0.06 Manages conflict effectively 0.20 0.26 0.17 0.76 0.04 Works effectively with others 0.36 0.31 0.07 -0.14 0.73 Functions well on multidisciplinary team 0.24 0.26 0.38 0.30 0.70 Extraction Method: Principal Component Analysis. Rotation Method Varimax with Kaiser Normalization. Rotation converged in 7 iterations.

90 Chapter 5 Case Study: Business Table 5.3: Measurement results pertaining to all underlying parameters. MEAN STDEV n MEAN STDV Measured parameter GR1 GR2 GR1 GR2 GR1 GR2 ∆ Cnf. ∆ Conf. PROFESSIONALISM [Cronbach Alpha = 0.935] P1 Exhibits good listening and questioning skills 4.29 4.15 0.71 0.99 28 20 -0.14 0.28 > 90% P2 Assumes responsibility/accountable for actions 4.50 4.35 0.65 0.75 26 20 -0.15 0.10 P3 Possesses honesty/integrity/personal ethics 4.59 4.70 0.75 0.57 27 20 0.11 -0.18 P4 Shows initiative/is self-motivated 4.41 4.30 0.80 1.08 27 20 -0.11 0.28 > 90% P5 Demonstrates a positive attitude toward change 4.35 4.30 0.63 0.86 26 20 -0.05 0.24 > 90% P6 Understands/contributes to organization goals 4.27 4.42 0.60 0.84 26 19 0.15 0.23 > 90% P7 Demonstrates flexibility/adaptability 4.48 4.60 0.64 0.60 27 20 0.12 -0.04 exPRESSIVE COMMUNICATION [Cronbach Alpha = 0.926] EC1 Writes clearly and concisely 4.00 4.11 0.76 0.76 25 18 0.11 -0.01 EC3 Speaks with clarity and confidence 3.96 4.20 0.84 0.70 28 20 0.24 -0.14 EC3 Makes effective presentations 4.06 4.36 0.94 0.67 18 11 0.31 -0.26 EC4 Exhibits self-confidence 3.96 4.32 0.85 0.75 27 19 0.35 >90% -0.10 WORK MANAGEMENT [Cronbach = 0.923] OP1 Manages projects and/or resources effectively 4.08 4.37 0.84 0.60 26 19 0.29 -0.25 > 90% OP2 Sets goals and prioritizes 4.04 4.06 0.89 0.73 25 18 0.02 -0.16 OP3 Manages several tasks at once 4.27 4.22 0.83 0.73 26 18 -0.05 -0.10 OP4 Allocates time to meet deadlines 4.23 4.26 0.95 0.73 26 19 0.03 -0.22 leADERSHIP [Cronbach Alpha = 0.893] L1 Gives direction, guidance and training 4.08 4.00 0.86 0.50 13 9 -0.08 -0.36 > 90% L2 Motivates others to succeed 3.93 4.00 0.88 0.60 15 12 0.07 -0.28 > 90% L3 Manages conflict effectively 4.06 4.25 0.83 0.75 17 12 0.19 -0.07 teAMWORK [Cronbach Alpha = 0.782] T1 Works effectively with others 4.41 4.75 0.75 0.44 27 20 0.34 >90% -0.30 > 90% T2 Functions well on multidisciplinary team 4.28 4.65 0.61 0.61 25 17 0.37 >90% -0.01 Group 1: no First Year Experience (control group) Group 2: Integrative First Year Experience lying parameters is further illustrated factor ability to work on a multidisciplinary team by factor in Figures 5.2–5.6. improved with a 90 percent confidence Although the sample was small, the level. This is certainly no surprise, consid- results provided insightful direction to the ering that the pedagogic improvement is design team. The items indicating expres- focusing on teamwork abilities. The con- sive communication, work management, fidence level of the development of means leadership and teamwork showed nearly can be considered conservative as the cal- unanimous improvement. Regarding culations account for the bias uncertainty teamwork, Figure 5.6 conveys that both the of different assessor populations through an ability to work with others, as well as the expanded t-test as described in Chapter 3.

Chapter 5 Case Study: Business 91 MEAN PROFESSIONALISM 4.80

4.70 GR II P3

4.60 GR II P7 GR I P3

4.50 GR I P2 GR II P6 GR I P7 GR I P4 4.40

GR I P5 GR II P2 GR II P4 4.30 GR II P5 GR I P1 GR I P6

4.20

GR II P1 4.10 0.55 0.65 0.75 0.85 0.95 1.05 STDEV

Figure 5.2 Changes in the underlying parameters of professionalism. Red = neg. change; Green = pos. change; Circle = significant change. GRI = Group 1 GRII = Group 2 P1 = Exhibits good listening and questioning skills P2 = Assumes responsibility/accountable for actions P3 = Possesses honesty/integrity/personal ethics P4 = Shows initiative/is self-motivated P5 = Demonstrates a positive attitude toward change P6 = Understands/contributes to organization goals P7 = Demonstrates flexibility/adaptability

MEAN EXPRESSIVE COMMUNICAT ION 4.40

GR II EC3 4.35 GR II EC4 4.30

4.25

4.20 GR II EC2

4.15

GR II EC1 4.10

4.05 GR I EC3

4.00 GR I EC1

GR I EC4 3.95 GR I EC2

3.90 0.65 0.70 0.75 0.80 0.85 0.90 0.95 1.00 STDEV Figure 5.3 Changes in the underlying parameters of expressive communication. Red = neg. change; Green = pos. change; Circle = significant change. GRI = Group 1 GRII = Group 2 EC1 = Writes clearly and concisely EC3 = Speaks with clarity and confidence EC3 = Makes effective presentations EC4 = Exhibits self-confidence

92 Chapter 5 Case Study: Business MEAN ORGANIZATIONALWORK MANA PLANNINGGEMENT 4.40

GR II O1 4.35

4.30

GR II O4 GR I O3 4.25 GR I O4 GR II O3 4.20

4.15

4.10 GR I O1 4.05 GR II O2

GR I O2 4.00 0.55 0.60 0.65 0.70 0.75 0.80 0.85 0.90 0.95 1.00 STDEV

Figure 5.4 Changes in the underlying parameters of work management. Green = pos. change; Circle = significant change. GRI = Group 1 GRII = Group 2 OP1 = Manages projects and/or resources effectively OP2 = Sets goals and prioritizes OP3 = Manages several tasks at once OP4 = Allocates time to meet deadlines

MEAN LEADERSHIP 4.30

4.25 GR II L3

4.20

4.15

4.10 GR I L1

4.05 GR I L3 4.00 GR II L1 GR II L2

3.95 GR I L2

3.90 0.45 0.50 0.55 0.60 0.65 0.70 0.75 0.80 0.85 0.90 STDEV Figure 5.5 Changes in the underlying parameters of leadership. Green = pos. change; Circle = significant change GRI = Group 1 GRII = Group 2 L1 = Gives direction, guidance and training L2 = Motivates others to succeed L3 = Manages conflict effectively

Chapter 5 Case Study: Business 93 MEAN TEAMWORK 4.80

GR II T1

4.70

GR II T2

4.60

4.50

4.40 GR I T1

4.30 GR I T2

4.20 0.40 0.45 0.50 0.55 0.60 0.65 0.70 0.75 0.80 STDEV Figure 5.6 Changes in the underlying parameters of teamwork. Green = pos. change; Circle = significant change. GRI = Group 1 GRII = Group 2 T1 = Works effectively with others T2 = Functions well on multidisciplinary team

Similar positive trends can be observed and Questioning Skills; Understands/Con- with regard to leadership and work man- tributes to Organization Goals over two agement. All underlying parameters show factors. The first four parameters (writing, an increase in mean and a decrease in speaking, presenting, and exhibiting self standard deviation. The curricular change confidence) are explicitly student driven, a appears to not only produce better results, fact that caused the authors of this chapter but the student performance is also more to name the factor Expressive Communi- homogeneous. Here again Gives Direction, cation. These parameters showed a positive Guidance and Training; Motivates Others development both with regard to standard to Succeed; and Manages Projects and/or deviations and means. The reminder of Resources Effectively; show a statistically communications related parameters, such significant improvement in the homogene- as Exhibits Good Listening and Question- ity of outcomes. ing Skills and Understands/Contributes to Before moving to discuss profession- Organization Goals, in contrast, showed alism and expressive communication, a negative trend. One can only speculate it might be interesting to reflect on the regarding reasons for this development. outcome of the factor analysis. The factor It is tempting to hypothesize that an analysis splits issues related to commu- enhanced opportunity to practice team- nication such as Writes Clearly and Con- work encourages communication between cisely; Speaks with Clarity and Confidence; team members; the situation is particu- Makes Effective Presentations; Exhibits larly suited for training students in lateral Self-confidence; Exhibits Good Listening communication. Because of the large team

94 Chapter 5 Case Study: Business size, the educational situation fosters an Future Directions atmosphere where only some portion of As FYE efforts within the College of students is provided an opportunity to Business gathered momentum, the FIPSE develop leadership. The students assuming project provided valuable direction. This these leadership roles will have opportuni- early study, combined with rising retention ties to practice vertical communication. rates, encouraged broadened support for The role that each student played in the the freshmen curriculum. Since its ini- team could certainly be reflected in their tial, college-wide launch, the design team subsequent work performance. has added a second course with elements It is further important to remember that targeted explicitly at enhancing profession- the assessment data at hand solely reflects alism. This one-credit course for each of the aggregated opinion of supervisors. In a three quarters, titled, “Pathways to Busi- work situation, students that have not had ness,” includes class sessions addressing leadership roles on campus might be more image management, networking, profes- comfortable in taking guidance from their sional etiquette, and goal setting. team, rather than from a supervisor. As The potential for the FIPSE project, the voice of the supervisor is the only one however, goes well beyond the depicted reflected in the data, it should be no surprise FYE initiative. The College of Business has that Exhibits Good Listening and Question- recently revised its junior- and senior-level ing Skills, Shows Initiative/is Self-motivated, curriculum, adding substantial depth to its Demonstrates a Positive attitude Toward varied majors. Again, employer feedback Change, and Understands/Contributes to will enable comparisons of student per- Organization Goals show an increased het- formance before and after the revision. As erogeneity in behavior. It is further impor- reporting of the co-op data becomes insti- tant to recognize that there is no significant tutionalized, the opportunities will mag- difference in the means between the control nify. Similar to any curriculum preparing and the treated group. The difference is students for a dynamic workplace, continu- solely reflected in behavioral heterogeneity. ous improvement and change are essential. Based on the data, the design team noted the Future uses may be proactive, signaling the importance of rotating leadership responsi- need for curricular, as well as reactive, test- bilities within the group, making sure that ing the impact of revisions. In combination, all students get exposed to both lateral and co-op data may become part of an ongoing vertical communication. cycle of institutional learning.

Chapter 5 Case Study: Business 95 Chapter Six Case Study: Civil and Environmental Engineering By Richard Miller, T. Michael Baseheart, Catherine Maltbie, Thomas Newbold

“If the only tool you have is a hammer, all of your problems look like nails.” — An Engineering Maxim

Motivations for Joining the Project program, a constant and reliable feedback loop is available. In the past, co-op data Engineering programs are required to con- was used to improve co-op, but never to tinuously assess curriculum as part of the improve the curriculum. accreditation process. This is done though a continuous feedback cycle. Part of this There should be a continuous effort cycle consists of surveys of recent gradu- to ask co-op employers questions about ates and the employers of these graduates. how the academic unit can help meet their Unfortunately, it is very difficult to get changing needs; both needs as a co-op people to respond to these surveys. It is employer and future needs as a potential not uncommon to “lose track” of gradu- employer of engineering graduates. By ates, making it impossible to get data from providing a vehicle that encourages this either former students or their employers. dialog, continuous improvement can be In other cases, people simply do not wish achieved. The feedback loop will have the to, or do not have time to respond. The effect of changing the curriculum to meet largest drawback to the post-graduate sur- the changing needs of industry. Addition- veys is that any data they provide on the ally co-op employers will be able to see curriculum is often out of date. Engineer- tangible proof that their feedback matters ing curricula change as does the content and has been taken seriously by the aca- of courses. Thus, it is not uncommon for demic unit. Students will be better pre- a former student to comment on a course pared to meet the needs of industry and which either no longer exists or has under- there will be a shared vision of what needs gone extensive changes. Even if the course to done to prepare students for the world has not changed, it may be several years of work. between the time the student takes the class and the time the survey feedback is received. A more immediate feedback loop Description of the is desirable. Department and the Program Co-op employer data on student work The University of Cincinnati (UC) is a com- performance provides a more immedi- prehensive state university with more than ate feedback. It is possible to assess the 37,000 students. Among many remark- effectiveness of parts of the curriculum as able innovations to emanate from UC was soon as the student completes the material. the introduction of the first program of Since completion of student and employer cooperative education in 1906. Initiated by evaluations is a mandatory part of the Dean Herman Schneider in UC’s College of

96 Chapter 6 Case Study: Civil and Environmental Engineering Engineering, cooperative education was a includes the Integrated Design Sequence new teaching methodology which changed (IDS) courses spanning three quarters. 20th century education. The IDS courses are offered in conjunc- The Department of Civil and Environ- tion with a practicing professional engi- mental Engineering (CEE) at the Univer- neer from industry and faculty members sity of Cincinnati, with 26 tenure track fac- acting as mentors. Thus, graduates of the ulty distributed equally between the Civil program acquire an integrated theoretical, Program and the Environmental program, design, and practical background, and are is the largest CEE Department in Ohio. highly sought after by industry. The stated mission of the department is “… At UC, undergraduate study in civil to educate civil and environmental engi- engineering includes five specialty areas. neers through a stimulating and support- Seniors select between construction engi- ive academic environment, reinforced with neering (35 percent of enrollment); struc- meaningful graduate research and under- tural engineering (30 percent of enroll- graduate cooperative work experience.” ment), geotechnical engineering (15 percent CEE students are well versed in fun- of enrollment), transportation engineering damental knowledge and problem solving (10 percent of enrollment) and environmen- abilities, and possess the technical, com- tal engineering (10 percent of enrollment). munication and interpersonal skills neces- The Civil and Environmental Engi- sary for lifelong learning and productive neering co-op program has 803 place- careers in a rapidly changing world. The ments from the fall of 2004 through the department has a budget of about $8 mil- fall of 2007. During this period of time lion per year, including over $5.2 million the civil and environmental engineering ($200,000/faculty) in grants and contracts. co-op students have earned approximately The typical enrollment is 300 undergradu- $6,000,000. Students are placed with com- ate and 160 graduate students, although panies throughout the United States and undergraduate enrollment has been the world. For the last 17 years the Civil increasing in recent years and now stands and Environmental Engineering program at about 350 students. has had a 100 percent success rate in plac- The undergraduate program in civil ing students in career related positions. engineering and our graduate program in environmental engineering are both fully accredited by ABET, the national The Curricular Reform Effort Accreditation Board for Engineering Traditionally, engineering curricula at the and Technology. The graduate program University of Cincinnati have exceeded 200 in environmental engineering and sci- quarter credit hours. As most other degree ence is ranked second in the nation by programs in the university required only the Gourman Report (1995), and is listed 180-190 hours, the Office of the Provost consistently among the top 20 U.S. envi- requested that future changes in engineer- ronmental engineering programs by the ing curricula should include a reduction in U.S. News and World Report rankings. The credit hours, if this could be accomplished undergraduate program incorporates a without a sacrifice in quality. In year 2000, cooperative education component in which CEE reduced their curriculum from 203 students complement their academic cur- to 197 credit hours though realignment of riculum with on-the-job training for six courses needed to meet general education quarters before graduation. The senior year requirements.

Chapter 6 Case Study: Civil and Environmental Engineering 97 In 2004, the CEE Department was dent co-op evaluation. When the number reviewed for accreditation by the Accredi- of credits of Structural Analysis was tation Board for Engineering and Technol- reduced, certain subject areas received far ogy (ABET). The rules in effect at the time less coverage. Members of the structural required each civil engineering program to engineering faculty identified eight such demonstrate proficiency in a minimum of areas. Co-op employers were asked to four of the possible 11 areas of civil engi- evaluate the proficiency of the students in neering. The CEE Department has chosen these eight areas. Data from the class of structural engineering, geotechnical engi- 2007 (with seven credit hours of analysis) neering, water resources engineering/fluid was compared with data from the class mechanics and environmental engineer- 2008 (with four credit hours of analysis). ing as the four areas. One outcome of this The results are shown in Table 6.1. and review was a criticism that there was not Figure 6.1. enough design in the curriculum in these The data indicates that there is a mar- four areas to constitute proficiency. Adding ginal increase or no change in the mean design courses would have increased credit value, but an increase in standard devia- hours and gone against the desire of the tion. This could be explained two ways. university to reduce rather than increase The first is that by reducing the coverage, credit hour requirements. the poorer students don’t have as much CEE met the ABET requirement by time to “get it” and their performance suf- requiring that four of the eight available fers. The other possibility is that devoting CEE electives be design electives; one in less time to the subject does not allow cov- each proficiency area. This change made erage of the finer points. So while the stu- the curriculum overly heavy in structural dent’s proficiency with average problems is engineering, so it was decided that Struc- unchanged, their performance suffers with tural Analysis would reduced from seven more difficult or esoteric problems. credit hours to four. This was effective with However, only seven to 15 employ- the graduating class of 2008. ers responded to each area. The problem In order to assess this change, co-op is that only a portion of the CEE co-op employers were surveyed by adding ques- employers are engaged in structural analy- tions in the form of a dedicated assess- sis and only a portion of the co-op stu- ment instrument II to the quarterly students working for those employers actually

Class of 2007 Class of 2008 ∆ n Mean S n Mean S Mean S 8. Bending stresses 15 4.20 0.41 11 4.27 0.79 0.07 0.37 9. Shear stresses 14 4.14 0.36 10 4.30 0.82 0.16 0.46 10. Combined stress 9 4.22 0.44 12 4.25 0.75 0.03 0.31 11. Eccentric loading 8 4.00 0.53 9 4.22 0.67 0.22 0.13 12. Column buckling 10 4.00 0.67 10 4.10 0.74 0.10 0.07 13. Determinate structures 8 4.00 0.53 8 4.13 0.83 0.13 0.30 14 Indeterminate structures 7 3.86 0.69 7 4.00 0.82 0.14 0.13 15. Influence line 7 4.00 0.58 7 3.71 0.76 -0.29 0.18 Table 6.1 Development in Means and Standard deviations. Only parameters 8 and 9 show statistical significance with regard to standard deviation development (Conf. > 95%).

98 Chapter 6 Case Study: Civil and Environmental Engineering Mean Exit 2007 2008 4.35 809 808 4.25 810 710 811 708 ∆STDEV conf > 94% 4.15 709 813 812

4.05

711 715 712 814 713 3.95

Bending stresses 08 3.85 714 Shear stresses 09 Combined stress 10 3.75Eccentric loading 11 Column buckling 12 n 2007 = 7 - 15 815 Determinate structures 13 n 2008 = 7 - 12 3.65Indeterminate structures 14 Influence0.30 line 0.40 0.5015 0.60 0.70 0.80 0.90

STDEV

Figure 6.1 Mean Standard Deviation Matrix that demonstrates the development of means and standard deviations of different measured parameters. AI II parameters of the CEE program. do analysis related work. It was also found 2. Each engineering company has its own that some supervisors were not able to standards and methods of design. They respond to the survey because the person expect to teach the students these stan- filling out the student evaluation was an dards on the job. Thus, the employer overall supervisor who was not familiar wants the student to have an excellent with the student’s work at a detailed level. understanding of the basic concepts Subsequent investigation of this area, rather than being able to perform a though examination of other data (see the task a particular way. following section) and discussion with 3. In many cases, employers are looking employers through focus groups led to the for students to produce a design and following conclusions: they are not particular about which of 1. Employers are interested in broad skills many valid methods a student may use. sets. Thus, an employer is interested in For example, there are several equally whether a student can communicate, valid ways to find the deflection of a not whether they write using a specific structure. As long as the student knows style or use a specific presentation some of them, the employer is particu- program. lar as to which one the student knows.

Chapter 6 Case Study: Civil and Environmental Engineering 99 The general conclusion of this effort was Another difficulty occurs when a that co-op employer data has a limited use- student has an irregular co-op schedule. fulness for evaluating specific subject matter Transfer students may graduate with as within a course. few as four co-op quarters if they transfer after sophomore year. Students with academic difficulties may be delayed in start- Results and Impact of the Project ing co-op and may graduate with only four It became clear that the greatest use of data or five quarters of co-op. However, what is from co-op employers was to look at broad, more likely is that a student with academic general skill sets, which in the context of difficulties is demoted and each demo- the program is best done using Assessment tion adds one or two co-op quarters (The Instrument I (see Chapter 3 & Appendix 1). college record is 13 total co-op quarters, There are important points that need to the department record is 11). Data from be made before any data is subject to analy- students with irregular co-op schedules sis. Engineering programs are notorious for were eliminated. high attrition rates. The attrition rate in the As longitudinal data was not yet avail- College of Engineering is given as approxi- able at the time of the study, the analysis mately 35 percent. However, this is based on is based on comparing two snapshots in a simple head count. If a given class starts time. The investigators chose to compare with 100 freshmen and, when that same co-op quarter three with an aggregate of class graduates, there are 65 graduates; the data pertaining to quarters five and six, attrition rate is given as 35 percent. How- as this to a large extent eliminated leak- ever, the 65 seniors who graduate did not age of students in and out of the program. necessarily start as freshmen. Anecdotal The analysis also excluded students who evidence and some spot checks of civil engi- did not follow the co-op curriculum in a neering class indicate that only about 40-50 predetermined manner. percent of a given graduating class started Figure 6.2 shows the evaluated areas as freshmen with that class. This means a plotted for their change in standard devia- given graduating class is approximately 65 tion vs. change in mean value (∆MSM). percent students who started as freshmen The plot compares the performance of and 35 percent transfers. quarters three and the aggregate of quar- The attrition rate and the number of ters five and six. The selection was made in transfers have an impact on the data. Dur- order to minimize the relatively high attri- ing the first few quarters a given class co- tion rate of the program. The plot further ops, there are a number of students being indicates the measurement significance of evaluated who will not graduate. There the change through five different symbols. are also students transferring in and out Data in the upper left hand corner of the of programs. All of this stabilizes after the graph is the most desirable as this indicates sophomore year. For this reason, comparing an increased mean (employers are more the first co-op quarter with the final co-op satisfied with student performance) and a quarter is of questionable validity since it decreased standard deviation (students are is not really the same group of students. more uniform in their abilities). Data in For this reason, some of the data presented the lower right had corner is not desirable compares the third co-op quarter, when as it indicates less satisfaction with student enrollments stabilize, to the sixth and final performance and a greater difference in co-op quarter. individual student ability.

100 Chapter 6 Case Study: Civil and Environmental Engineering Delta MSM Civil and Environmetal Eng Quarter 3 / Quarter 5 & 6

0.80

0.70

H1 0.60 B4 F2 H2

J3 F3 0.50 B3 K3 C3 K2 J1 C2 A1 J4 A4 B1 0.40 G2A3 D4 D2 E2 A2J2 H3 B2 G3 E4

Delta MEAN 0.30

I1 C1 D1 K1 I3 E3 G1 I2 0.20 D5 E1

D3 0.10 K5

0.00 -0.35 -0.30 -0.25 -0.20 -0.15 -0.10 -0.05 0.00 0.05 0.10 0.15 K4

-0.10 Delta STDEV

= ∆ A and ∆ S significant (conf. 90 percent) = ∆ S significant (conf. 90 percent) = ∆ A significant (conf. 90 percent) = ∆ A significant (conf. 80 percent) ∆ S not significant = ∆ A and ∆S not significant

Figure 6.2 ∆MSM of Civil and Environmental Engineering Program, comparing co-op quarter 3 to quarters 5 and 6. See Table 6.2 (on next page) for Abbreviation explanations.

Chapter 6 Case Study: Civil and Environmental Engineering 101 Table 6.2 Key for AI I abbreviations pertaining to Figure 6.2.

A1 Speaking F1 Gives Direction A2 Writing F2 Motivates Others A3 Presenting F3 Conflict Management A4 Listening G1 Use of Technology B1 Evaluates Situations G2 Systems Understanding B2 Problem solving G3 Understanding of Technology B3 Creative Thinking H1 Comp. Design Ability B4 Idea Generation H2 Experimental Design Ability C1 Learning H3 Data Analysis C2 Applies Specialized Knowledge I1 Work Culture Understanding C3 Applies Classroom Learning I2 Respects Diversity D1 Accountability I3 Rec. Political Implications D2 Self Confidence J1 Project Management D3 Integrity J2 Goal Setting D4 Self Motivation J3 Task Management D5 Positive Attitude J4 Time Allocation E1 Works with others K1 Professional Attitude E2 Goal Orientation K2 Work Quality E3 Flexibility K3 Work Volume E4 Multi-disciplinary Team Functioning K4 Attendance K5 Punctuality

Some of the areas which show no experimental design skills. The high marks improvement are not surprising. K4 and in the H categories supports the idea that K5 hardly change and are “attendance” the current curriculum does an adequate and “punctuality.” These speak more to the job in teaching design concepts. student’s personal qualities, and are hard The “A” categories (speaking, writ- to impact by curricular revisions. ing, presenting) show increases in mean From the graph, the “H” category and decreases in standard deviation. The shows a statistically significant increase last major curricular reform focused on in mean value. This category deals with improving these skills and this appears to design and data analysis. One specific have succeeded. Courses on public speak- example is H2 Experimental Design. In the ing and technical writing were added and last curricular reform, a conscious effort communication skills were to be empha- was made to include more material on sized in the senior capstone courses. experimental design and data analysis in The “B” and “C” categories (Problem the CEE laboratory classes; especially CEE Solving and Application of Learning) show 474, Construction Materials Lab. CEE 474 increases in mean and decreases in stan- is the first CEE lab the students take, so dard deviation, but to a lesser extent than this class was chosen as place to emphasize other categories. These areas could use

102 Chapter 6 Case Study: Civil and Environmental Engineering improvement. To address this, the fresh- students were docked points for being late, man Introduction to CEE course is adding but that is not a curriculum issue. material. The largest addition is material The research shows that the feedback which will attempt to show the students loop is most effective when measuring gen- how the esoteric material they learn in eral work related competencies, measured math courses is applied to practical engi- by Assessment Instrument I, as the instru- neering problems. Additional material on ment yields sufficient n-values, to result problem solving is also being added. in significant measurements, and as the One area where the students are weak assessment of these traits are within the is the “E” category. This category addresses competency level of all supervisors. When the ability to work in teams, especially measuring detailed analysis abilities, such multi-disciplinary teams. The “I” category, as those related to the dimensioning of which involves respecting diversity, under- structures, the methodology yields low standing work cultures and understanding n-values both as a result of a low num- political implications is also weak. This ber of jobs where these skills are directly seems to indicate that the student have applied, and as a result of these skills not difficulty dealing with people and issues necessarily being assessable by the supervi- outside of engineering. Because of this, the sor. Still, even with the very low n-values, Senior Capstone courses are being redone. the study supports, that as the investment Instead of working on teams which are into instruction decreased, the diversity of multi-disciplinary within civil engineer- student performance increased. ing (e.g. structural students working with environmental students), the CEE students will be working with students from Future Directions architecture, planning and/or political Based upon the initial results and suc- science. The courses will still emphasize cesses shown by the research, the civil and engineering design, but will require stu- environmental engineering department dents to look at the broader implication will continue to use the co-op data for of their work. They will also have to deal evaluation of curriculum. Periodic updat- with input from non-engineering students. ing of the employer and student evalua- It is hoped that this collaboration will tions, especially to be able to add questions broaden the engineering student’s ability to gage the effect of curricular reform, will to see entire systems rather than individual be vital particularly as impending organi- elements (e.g. see the entire transportation zational changes will force the institution system, not the bridge or pavement.) to evaluate a large number of changes. In The “G” category on use of technology the particular case of the change in struc- shows an increased mean, but an increased tural analysis, the team recognizes that standard deviation. This indicates that the the assessment instrument (Assessment ability to use technology is not uniform Instrument II) should be reworked. When among the students. This issue needs to be the content is highly specific and applies addressed in future curriculum reforms. to only a subset of employers this instru- Clearly, some of the areas cannot be ment is not the correct tool. The use of affected by curricular reform. No change focus groups of key employers (Assessment in the formal curriculum can make a stu- Instrument III) is perhaps a better meth- dent more punctual. A change in depart- odology which allows the team to focus on ment policy could affect punctuality if more qualitative information such as why

Chapter 6 Case Study: Civil and Environmental Engineering 103 the employers think a specific skill is or is not important. Using the developed PAL assessment database (Assessments I and II) is most effective when gathering information from “big picture” questions from which skill sets can be inferred. The research team spent considerable time looking at data, processing data and creating instruments to fill in gaps, which has significant resource implications that must be addressed. The research team included diverse departments involved with co-op and provided the opportunity for cross fertilization which could perhaps lead to a synergistic use of resources. Change is always slow, but now there is a system in place that will keep up with the changing environment in business. Faculty members are intrigued to see the loop closed and continuous improvement taking place. This continuous loop of feedback can affect the curriculum and keep students up to date with changes in the field. Now that a system is in place continued work will be easier because of the work already done. The developed mechanism for curricular change is now a working model of success.

104 Chapter 6 Case Study: Civil and Environmental Engineering Chapter Seven Lessons Learned By Kettil Cedercreutz and Cheryl Cates

The corporate feedback system developed academic fields. The steering committee through the Fund for Improvement of soon grew into a coherent research team Postsecondary Education (FIPSE) project at that was well able to articulate each other’s hand identifies discrepancies between cur- perspective. In retrospect it is obvious that ricular offerings and stakeholder needs. The the lessons learned by the team clustered recurring statistically-validated feedback around two main areas; systems oriented process that the project set out to develop issues (related to the actual measurement will hopefully be a viable asset for any insti- and analysis of data), and the human tution that desires to grow its operation to interaction oriented issues (related to the meet both academic criteria and employer presentation of the constituencies involved requirements. The goals of the project in curricular reform). The section below “Developing a Corporate Feedback System analyzes the systems oriented lessons for Curricular Reform” included: learned, as they form a foundation for the 1) the creation of a process that aggregates presentation of the data to the involved employer input of co-op student work constituencies. The human interaction performance (The process was based on oriented lessons are more subtle, but in fact the examination of data collected using far more critical. They can, however, only an existing and tested work perfor- be covered once a solid foundation is laid mance assessment instrument, with the with regard to a systems understanding. ultimate objective of getting a sense of The chapter is concluded with an analysis student learning and development on a of future opportunities. programmatic level); 2) the development of additional assess- Systems Oriented Lessons Learned ment instruments to be used for a specific period of time, and targeted The project team rapidly learned that when- towards a mapping a variety of issues ever it was on a mission to deliver a positive important to the academic departments message, the data was seldom questioned. involved in the project; and On the other hand, whenever the data showed a decline in student performance, 3) fine tuning of the focus of tailored the scrutiny of data collection principles instruments through focus group inter- and relevant analysis methods would views of employers. become endless. This called for the develop- In accomplishing these goals, the proj- ment of an analysis model that left no doubt ect steering committee learned a number in the mind of the target audience regarding of lessons that can be helpful to future the applicability of the analysis to a specific adopters of the concept. The grant brought situation. The topics presented on the next together faculty from seven departments, pages summarize the findings related to multiple institutions and a variety of data collection, analysis and presentation.

Chapter 7 Lessons Learned 105 Samples and sample sizes typically require that data be manually The team found that longitudinal studies combined for analysis. While it is obvious form the most reliable basis for data analy- that web-based systems are effective, the full sis, as the study eliminates any variability potential can only be realized if systems feed caused by comparing the performance of into a well designed database that allows for different individuals to one another. Lon- efficient analysis of data. In order to be cost gitudinal studies also help eliminate the effective a system needs to provide a multi- effect of students leaving and entering the tude of features. A quality system supports program midstream. The problem with lon- efficient student placement; allows for com- gitudinal data is, however, the time needed munication with students and employers to accumulate it. The time lag can be very during the co-op assignment; administers cumbersome to manage whenever we need student assignments; collects qualitative and to act quickly and do not have the luxury to quantitative data from employers, students wait for longitudinal data to accumulate. In and faculty; and stores the data in perpe- the beginning of a pilot project, when lon- tuity. The work pursued during the grant gitudinal data is unavailable, the research revealed that labeling each work assignments team will be forced to work with the epi- with regard to; academic program identifier; sodic data available. The small size of stu- student identifier; assessor identifier; section dent cohorts does often add to the measure- identifier (I or II); time of year (fall, winter, ment challenge. There simply is not always spring, or summer); position in curriculum enough data to draw conclusions having any (freshmen, sophomore, pre-junior, junior, or measurement significance. In some cases senior); work term sequence number (1, 2, the situation can be addressed by combin- 3, 4, 5 or 6); and expected graduation year ing several years’ worth of data through makes the information required for analysis the use of rolling averages. The methodol- effectively accessible. A strong labeling sys- ogy boosts n-values, at the cost of accuracy tem within the data repository is an invalu- pertaining to specific student cohorts. The able asset for future research and curriculum findings of the project, however, support the development efforts. notion that the behavior of different class years enrolled in mature co-op programs is Understanding what the system can measure relatively uniform. Regardless of the sampling methodology used, it is important to The feedback loop was created to analyze fully internalize what a specific analysis of curricular efficacy based on supervisor any data set conveys. All methods will have evaluations. The data can thus only be as their pitfalls. Only by understanding these relevant as the aggregate understanding of pitfalls will we be able to present our case. the pool of work supervisors the program relies upon. The system can only be used to evaluate behavior that is both detectable Data Collection and Storage by the supervisor, and that falls within the Data collection and storage must be made professional competency of the supervi- effective in order to make the development sor. Additionally, the measured behavior of a feedback system worthwhile. Relying on must occur with a high enough frequency paper instruments is doomed to fail, as the to allow a quantitative analysis. Behavior data entry required prior to analysis becomes measured with Assessment Instrument I cost prohibitive. It is equally unwieldy to (communication, conceptual and analyti- base the data collection on a mix of ad hoc cal ability, learning/theory and practice, data collection systems, as these solutions professional qualities, team work, leader-

106 Chapter 7 Lessons Learned ship, technology, design and experimental measure that is typically elicited from the skills, work culture, organization plan- input of relatively few faculty members). ning, evaluation of work habits) were Surprisingly the measurement uncertainty found to be both within the realm of the proved to be one of the least problematic competency of a typical supervisor as well areas of the study. Whenever n-values as occurring with appropriate frequency exceed 30 in each group the enhanced t-test for efficient measurement. Questions that derived by the project seems to be able to focused upon measuring details of techni- demonstrate a significant change, whenever cal skills often times were not matched one is truly present. to the competency of a particular set of supervisors (i.e. web designers are often Reporting hired by individuals that know little about the actual technology required to build The development of reports was one of the web pages). Measuring technical skills most exhilarating aspects of the project. The proved problematic, even when the ques- challenge was to develop a reporting instru- tions hit supervisor core competency. This ment that was both succinct and expressive. was the case whenever the researchers After extensive research the project team tried to measure the development of skills found it important to map both student that, however important, are seldom used cohort performance level, and performance in an industrial setting. uniformity. For the purpose of formative assessment Mean Standard Deviation The fact that the data is based on super- Matrixes (MSMs) and Delta Mean Standard visor observations brings an additional layer Deviation Matrixes (∆MSMs) were found of complexity into the mix. As the data is to be most effective. For summative assess- based on an observation it does not neces- ment, that often addresses the needs of the sarily constitute an objective benchmark. general public, tables comparing means The data seems to reflect how effectively were found to be the easiest to comprehend. the student meets the objectives of the Automatic generation of effective reports supervisor, but may not necessarily reflect proved, however, to be even more chal- how effectively the work is actually being lenging to develop due in part to the rapid executed. As an example, enhanced team- growth of data housed in the newly devel- work skills can have negative implications oped system. As the project matured the with respect to supervisor assessment, as access to more data points caused members group dependent students are less prone to of the research team to request increasingly follow supervisor directives and more likely more sophisticated reports. When trying to take the lead from the group. to answer specific research questions in a curricular change situation, the research Measurement uncertainty team found that all data had to be filtered The large quantity of co-op supervisors on a very detailed level. This elimination of (typically as much as a factor of 10 more misleading base data (i.e. elimination of stu- than faculty), and the relatively low level of dents that had not pursued a specific cur- variability of supervisor systematic uncer- ricular sequence) was far more labor inten- tainty or bias brings down the uncertainty sive than the actual data analysis. While of typical measurements to a manage- it is certainly possible to develop canned able level. The work performance data is reporting algorithms, researchers must also typically far more stable than the aggregate analyze data on a case by case basis within student grade point average (an aggregate any effective reporting system.

Chapter 7 Lessons Learned 107 Data Sharing opportunity for organizational develop- Data sharing is a very delicate matter. ment in a wide range of areas. These areas Work assessment data is very sensitive, as are described in the sections below: it holds information on individuals that can be potentially damaging even decades Team Work, Academic Culture, and Trust after graduation. Needless to say, the access to the data must be strictly moni- The developed feedback loop relies on tored. For integrity reasons the data must a large co-op program that is strongly be kept fully confidential. After careful aligned with the identity of the institution. scrutiny, however, reports based on this The University of Cincinnati is a research data can be distributed both campus wide university highly engaged in experiential and to external constituencies. Care must learning. As a result the opportunity to be given to honoring the privacy of all contribute to a prestigious grant, having assessed individuals. Additional problems significant external funding, was well in arise whenever the institution wants to harmony with the institution’s academic report data that is homogeneous in nature, culture. The success of the grant can yet shows low performance of all students largely be attributed to the pilot project in a specific program. Fortunately the design of the team. It brought together a project team found no such program, but steering group of committed individuals this does not relieve the institution from who served as champions for the project liability would such a case be reported. within their own departments and colleges; it established a strong partnership between cooperative education practitio- Organizational Lessons Learned ners and classroom instructors that went far beyond student placement; it focused The human interaction oriented lessons on student learning as a collective charge; are important and more difficult to mas- it fostered the development of a research ter than the systems oriented lessons. The assessment methodologies developed by mindset within faculty and cooperative the project demonstrated that learning education practitioners as it pertains to can be verified through the aggregation of student work performance; and it devel- employer feedback. This feedback creates a oped a mindset within the academic units program fingerprint that reveals the proof cooperative education as a contributor gression of student skill development over to student learning. the course of the program. The reporting Making changes in the curriculum is tools make it easy to identify topic areas in dependent upon the culture of a specific which students are systematically progress- academic unit which cautioned the team to ing as well as areas in which learning is be sensitive to a variety of cultural implica- merely serendipitous. The methodology tions. During the project the leader of each allows departments to focus on the qual- program specific team had to be sensitive ity of instruction as the measurement is to the culture of his/her academic unit and conducted in a real world context. Com- modify the approach accordingly. A posi- bining the analysis of generic assessment tive attitude toward change can only be instruments (Assessment Instrument I), built in an atmosphere of trust. No change tailored assessment instruments (Assess- is possible if the faculty does not trust ment Instrument II), and focus groups the data, the analysis, and the messenger. (Assessment Instrument III) provided the The culture must evolve from having the

108 Chapter 7 Lessons Learned mandate of a research project to becoming team felt that the data might be most effec- an all encompassing culture of continu- tive if faculty came to experience the data ous improvement. In the early stage the on their own terms and in their own time discussion tends to focus on the validity frame without the guiding influence of of the data, as well as the analysis. When individuals perceived as outsiders. organizations mature, more time can be Conclusions from this data are being used analyzing the actual meaning of the used to shape future curricular reform findings. in many departments. The project had a The project brought faculty together secondary benefit of improving commu- around an atmosphere of exploration and nication between the professional practice excitement. The team, however, quickly faculty and the college faculty. Both sides realized that trust was going to be the key developed an appreciation of how changes element in project success. The first action to the curriculum affect or do not affect in establishing trust was to recognize that student performance on the job and, indi- comparisons across programs would in rectly, student learning. Through the pilot essence be comparing apples to oranges. projects the team learned that concentrat- Though there may be trends that cross ing on broad skills, not individual subjects, programs, the goal is not to determine held the most promise to impact student any program’s superiority, but to aid all performance evaluations. The pilot proj- programs in their individual improvement ects demonstrated that the departments efforts. need to provide identifiable and usable The team also had to come to grips skills as early in the curriculum as possible with conflicting feedback, stemming from to maximize employer satisfaction. Those a variety of sources. As an example, indus- involved in the pilot projects could clearly trial advisory boards have often been used see employer satisfaction increases when to generate employer insights into areas of the students acquire certain critical skills. improvement which might be in conflict with the voice of co-op supervisors. The Importance of Resources researchers focused on co-op supervisors The project was enabled by external and to provide insights on the operational per- institutional funding of almost one mil- formance of students while understanding lion dollars. This funding was critical that this information did not necessarily to the development of infrastructure to have to be in harmony with the more stra- allow the research to take place. Lack- tegic vision of industrial advisory boards. ing resources in terms of infrastructure, As a final challenge to the development funds, or time, can form a significant of trust, the team discussed the issue of barrier for project implementation. When presenting reports at departmental meet- implementing the system in a new envi- ings. Committee members expressed some ronment an institution would need to concern that the faculty might not be allocate resources and provide colleges receptive to the information because of the and departments incentives to participate perception that they were being coerced to (i.e. additional staff or graduate assis- make decisions based on an outside man- tants to analyze data). Institutions would date. In an effort to help establish trust also need to have an effective evaluation beyond the research team itself, data was system in place and recognize that it will provided to departments in an electronic require substantial time and effort to format to be used at their discretion. The analyze the results. Co-op evaluations

Chapter 7 Lessons Learned 109 generate an abundance of data that needs idly changing society. The next challenge to be analyzed with patience. The project is to develop a cost effective way to transfer team spent a lot of time figuring out how the knowledge and the system generated to most effectively analyze and report data. through the project to other institutions. The wisest use of resources turned out to be the development of a data entry system that Moving from Pilot to Institutionalization allowed individual employers and students In retrospect, it is easy to see that starting to enter data directly into the data base by out the development in the form of a pilot completing web-based assessment instru- project formed an effective approach. The ments. Without this initial investment, the excitement drew in change agents from all time and money required to complete the participating colleges. The members of the project would have been cost prohibitive. team contributed to forming a full picture of the mosaic, as each team member had Benefits of the Corporate Feedback System a different piece of the story to tell. A pilot To ensure that faculty members are will- project was very conducive to developing to invest their time in the process, ing a spirit of ownership and excitement. the benefits of the corporate feedback Structuring the project in multiple cycles system must be well established. In addi- of curricular reform maximized oppor- tion to the intrinsic benefit of reforming tunities for success. Establishing a regular curricula because it aids student learning meeting schedule further allowed depart- and provides a more relevant education ments engaged in various curricular reform for graduates, faculty are also required to projects to exchange information with one continuously improve the curriculum for another. The project demonstrated that accreditation purposes. In this process cooperative education can regain its posi- employer data can be extremely valuable. tion as a pioneer in experiential learning by Another aspect of the corporate feedback focusing on assessment of student learning system that could entice faculty to invest as part of a continuous improvement pro- their time in it, is the ability to generate cess. Modifications to expand the concept credible marketing material targeting all beyond cooperative education are fully pos- major stakeholders (i.e. employers, high sible and already under development. school guidance counselors, students, and As the systems development moves parents). The process further solidifies the into an institutionalization phase, the position of cooperative education as not dissemination of systems understand- only a learning methodology but also an ing will become crucial. Members of the all encompassing assessment methodology. pilot project will have an opportunity to The fact that work performance data has serve as ambassadors of change in their high credibility tends to increase the esteem individual academic environments. In the of the academic process in the eyes of the particular case of UC the process is sup- community, industry, and government. The ported by the decision of the Office of corporate feedback system significantly the Provost to administer the feedback increases the added value of cooperative to individual departments and to require education as it moves the methodology to a individual colleges to use assessment data strategic asset of the institution. An insti- in their continuous improvement processes. tution monitoring its academic excellence Programs that rely on alternative forms of through contextual work assessment will experiential learning will equally benefit continue to have an upper hand in a rap- from the institutionalization as the system

110 Chapter 7 Lessons Learned will be expanded to facilitate these forms feedback data must be complemented by of experiential learning. The development strategic thinking, provided by executives of a convenient user interface that makes on industrial advisory boards, think tanks reports accessible to academic departments etc. Still, as an operational tool, the meth- will became the focus for development as odology is strong in dovetailing education the project moves from a pilot stage towards with current industry-based demand. institutionalization. Whenever the operational nature of the methodology is understood, one can Generation of Discussion move on to dream about its future use. In a The reaction to work performance based university that pursues 5,000 student place- assessment data tends to be coupled with ments a year the system generates up to half strong emotions. These emotions can create a million data points annually. Assuming heated dialogue, the energy of which needs that we make no systemic changes, we can, to be channeled effectively. The emotional within 10 years, pursue longitudinal studies reaction that comes forth in these dialogues based on five million observations. This will tends to be extremely positive as one of give pedagogic research in an employment pride, satisfaction and awakening to the oriented setting a totally new meaning. fact that the data shows the impact of the Institutions having the capacity to assess education provided by the faculty. It also student performance in the context of the tends to cause defensive reactions that need real world, can translate their knowhow to be managed with care. This tension does, into a competitive advantage. They can be however, form a good beginning. It fosters on the cutting edge in the development of an atmosphere of dialogue that can be used the understanding of the cause an effect of to benefit all stakeholders. At the end of the programmatic changes. These institutions day there are no simple solutions. Decisions can turn their organizations into laborato- must be based on an aggregate of strategic, ries where different pedagogic approaches operational, demographic, and economi- can be contrasted for best practices. They cal information. Shaping the future is not can cross link institutional and work assess- a science but an art. In this art, work based ment data in order to understand the rela- assessment forms an important ingredient. tion between classroom learning and work It is important to remember, that in con- performance. They can also go beyond that tinuous improvement, the journey is some- pedagogic link into other areas of interest such as economic impact on the local area times more important than the destination. or economic impact on the institution.

Future Opportunities Contextual assessment of student work performance forms a strong competitive advantage for an institution. It can be built into a tool that on an operational level keeps the methodologies applied by an institution abreast with industry-based needs. The shortcoming of the system is that it relates curricular efficacy only to the industry of today. The methodology is not conducive to mapping the future. The

Chapter 7 Lessons Learned 111

Index

A, Average, Mean 40 IDS, see Integrated Design Sequence ABET, see Accreditation Board of Engineering and Technology Information Systems 15 ACCE, see Accreditation Council for Cooperative Education Integrated Reliability 48 Accounting 15 Internal Consistency (Cronbach Alpha) 15, 61, 62, 91 Accreditation Board of Engineering and Technology 17, 29, 97, 98 k 44

Accreditation Council for Cooperative Education 5, 11, 26, 27 kEFFECT 44 AI I, see Assessment Instrument I (General) LC, see Learning Communities AI II, see Assessment Instrument II (Situation specific) Leakage 58-60 AI III, see Assessment Instrument III (Focus group) Learning Communities 88 Architecture 12, 15, 16 Longitudinal study 100, 106, 111 ASEE, see American Society of Engineering Education 24 Management Skills Practicum 88 Assessment Instrument I (General) 34, 37-39, 41, 45, 61, 100, 106, 108 Materials Science Engineering 24 Assessment Instrument II (Situation specific) 14-15, 17, 24, 34, 39-40, 73, 95, 99, 108 Mean Standard Deviation Matrix 14, 42, 65, 77, 92-94, 99, 107 Assessment Instrument III (Focus group) 15, 34 Mean, Average 44 Assessment, Levels, Multiple 34 Mechanical Engineering Technology 25 Average reading 40 Miami University 5 Bias 40 MSM, see Mean Standard Deviation Matrix Business Administration 12 MSP, see Management Skills Practicum CAFCE, see Canadian Association for Cooperative Education n, Amount of returned assessments 44 Canadian Association for Cooperative Education 120 N, Total amount of students 44 CAS, see OMI College of Applied Science NAAB, see National Architecture Accreditation Board Center for Cooperative Education Research and Innovation National Architecture Accreditation Board 79 CEE, see Civil and Environmental Engineering NCAA, see North Central Accreditation Agency CEIA, see Cooperative Education and Internship Association Noise 44 CHEA, see Council for Higher Education Accreditation n-values 44 Chi Square Test 50 OMI College of Applied Science (University of Cincinnati) 27 Civil and Environmental Engineering 5, 15, 17 PAL, see Professional Assessment and Learning System CoB, see College of Business PP, see Division of Professional Practice CoE, see College of Engineering Professional Assessment and Learning System 6, 35-36 College of Business (University of Cincinnati) 27 Random error 40 College of Design, Architecture, Art and Planning 27 Reading 40 College of Education Evaluation Services Center 5, 11, 27 Research Team Design 51 College of Engineering (University of Cincinnati) 27 Resolution 40 Construction Management 12 Rolling average 56 Co-op Calendar 23 S, see Standard deviation Co-op Program Characteristics 15 SAID, see School of Architecture and Interior Design Co-op, see Cooperative Education School of Architecture and Interior Design 6-7 Cooperative Education Section 23 Cooperative Education and Internship Association 120, 121 Signal, the value of the actual property being measured 44

Council for Higher Education Accreditation 29 SRAND , Standard deviation of random error 44

Cronbach Alpha, see Internal Consistency SREAD 44

DAAP, see College of Design, Architecture, Art and Planning SSIGN 44

Delta Standard Deviation Matrix 14, 43, 66, 101 SSYST 44 Division of Professional Practice 5, 11, 12 Standard Deviation 44 EAC, see Evaluation and Assessment Center for 5, 11 Standard deviation of reading 44 Mathematics and Science Education at Miami University Standard deviation of signal 44 Effective assessor amount 40 Standard deviation of systematic error 44 Engineering, see College of Engineering Statistical Uncertainty 44 Environmental Technology one, two, and three STDEV, see Standard Deviation 44 ESC, see Evaluation Services Center 5, 11 Systematic error 44 ET1–ET3, see Environmental Technology one, two, and three 79-83 t, t-factor 44 Evaluation and Assessment Center for Mathematics 5, 11 t-test 52-54 and Science Education at Miami University Type A error 44 Evaluation Services Center 5, 11 Type B error 44 Expanded t-test 53 U, Uncertainty 44 Exposure time 56, 57 USTAT, Statistical Uncertainty 44 Expressive Communication 91, 92 V, Percentage of unique assessors 44 F, F-factor for specific confidence level and specific 55 VI, VII, VIII,… Assessors filing 1,2,3,… assessments each 44 Factor Analysis 61, 62, 90, 91 WACE, see World Association for Cooperative Education FIPSE, see Fund for the Improvement of Postsecondary Education WIL, see Work Integrated Learning First Year Experience 86-95 Work Integrated Learning 28 F-test 55 ΔA, Difference in Average, Difference in Mean 44 Fund for the Improvement of Postsecondary Education 5, 11, 12 ΔMSM, see Delta Standard Deviation Matrix 44 FYE, see First Year Experience ΔS, Difference in Standard Deviation 44

Index 113

Appendix

115 116 117 118 Bibliography

AACSB. (2002). Management education at risk. Report of the Management Education Task Force to the AACSB International Board of Directors. (p. 19). ABRAHAMSON, E. (2000). “Change without Pain.” Harvard Business Review, July-August. ACCREDITATION BOARD FOR ENGINEERING AND TECHNOLOGY, [Engineering Accreditation Commis- sion ABET/EAC]. (2005). Criteria For Accrediting Engineering Programs-Effective for Evaluations During the 2005–2006 Accreditation Cycle, Atlanta, GA, Accreditation Board for Engineering and Technology. ACCREDITATION BOARD FOR ENGINEERING AND TECHNOLOGY, [Engineering Accreditation Commis- sion, ABET/EAC]. (1997). Engineering Criteria 2000, 3rd ed., Atlanta, GA, Accreditation Board for Engineering and Technology. ACCREDITATION COUNCIL FOR COOPERATIVE EDUCATION. (1996). The Attributes of Cooperative Edu- cation Programs, Atlanta GA, Accreditation Council for Cooperative Education. ALBRIGHT, S. (2001). VBA for modelers: developing decision support systems with Microsoft Excel, Pacific Grove, CA, Duxbury/Thomson Learning. ALTBACH, P. G, BERDAHL, R. O., and GUMPORT, P. J. (1999) American Higher Education in the Twenty First Century: Social, Political and Economic Challenges. Baltimore, MD, The Johns Hopkins University Press ALVESSON, M. (1993). Cultural Perspectives on Organizations. New York, NY: Cambridge University Press. ANDERSON, D. R., SWEENEY D. J. and WILLIAMS T. A. (1981). Introduction to statistics: an applications approach, St. Paul, MN, West Pub. Co. ARGYRIS, C. (1999). On Organizational Learning (2nd ed). Malden, MA, Blackwell. ARGYRIS, C. (1990). Overcoming Organizational Defenses: Facilitating Organizational Learning. Boston, MA, Allyn and Bacom APOSTOLIDES, V. (1994). Cooperative Education Association 31st Annual Conference, Models of Cooperative Education Curricula. ASSOCIATION FOR INSTITUTIONAL RESEARCH [AIR]. (2006). Annual Report 2005-2006, Tallahassee, FL, Association for Institutional Research. ASTIN, A. (2002). Assessment for excellence: the philosophy and practice of assessment and evaluation in higher education, Westport, CT, Oryx Press. AYYUB, B. M. and MCCUEN, R. H. (2003). Probability, statistics, and reliability for engineers and scientists, 2nd ed, Boca Raton, FL, Chapman & Hall/CRC. BADIRU, A. B. and AYENI, B. J. (1993). Practitioner’s guide to quality and process improvement, London, UK, New York, NY, Chapman & Hall. BAHR, N. J. (1997). System safety engineering and risk assessment: a practical approach, Washington DC, Taylor & Francis. BANTA, T. W. (2002). Characteristics of Effective Outcomes assessment, Foundations and Examples. In Banta, T. W. [Ed.] and Associates, Building a scholarship of assessment, San Francisco, CA, Jossey-Bass. BANTA, T., BLACK, K., LUND, J. and OBLANDER, F. (1996). Assessment in Practice: putting principle to work on college campuses, San Francisco, CA, Jossey-Bass Publishers. BANTA, TRUDY W. (1993). Making a Difference: Outcomes of a Decade of Assessment in Higher Education. San Francisco, CA, Jossey-Bass Publishers. BARTKUS, K. and STULL, W. Supervisor/Manager Perceptions of Cooperative Education/Internship Students: Implications for the Development of a Needs-based Program, Journal of Cooperative Education, Volume XXVI, No. 3. BARTLETT, P. M. and GILBY, S.W. (2006). Discovering New Opportunities, a Value Analysis In Process Confer- ence, Proceedings, CEIA 2006 Annual Conference, Defining the Next Century, Walnut Creek, CA, Cooper- ative Education and Internship Association.

Bibliography 119 BASS, B. (1985). Leadership and Performance Beyond Expectations. New York, NY, The Free Press. BELASEN, A.T. (2000). Leading the Learning Organization, Albany, NY, State University of New York Press. BENNIS, W. and NANUS, B. (1985). Leaders: Strategies for Taking Charge. New York, NY, Harper & Row. BENNIS, W. (1994). On Becoming a Leader. Reading, MA, Preseus Books. BHOTE, K. R. (2003). The power of ultimate Six Sigma: Keki Bhote’s proven system for moving beyond quality excellence to total business excellence, New York, NY, AMACOM. BOONSTRA, J.J. (2004). Dynamics of Organizational Change and Learning. West Sussex, England: John Wiley and Sons, Ltd. BOYCE, M. E. (2003). Organizational Learning is Essential to Achieving and Sustaining Change in Higher Educa- tion. Innovative Higher Education, 28(2), p. 119-133. BORICH, G. D. (1992). Effective Teaching Methods. New York, NY, Macmillian. BRANTON, G., VAN GYN, G., CUTT, J., LOKEN M., NEW, T., and RICKS, F. (1990). A Model for Assessing the Learning Benefits in Cooperative Education. Journal of Cooperative Education, No. XXVI, No. 3. BROWN, M. (1997). Measuring up against the 1997 Baldrige Criteria, In Journal for Quality and Participation, Vol: 20, Iss: 4, Date: Sep 1997, p. 22-28. BROWN, M. (1994). Baldrige award winning quality: How to inter-pret the Malcolm Baldrige Award Criteria, Mil- waukee, WI, ASQC Quality Press, 4th ed. BRUMM, T. J., HANNEMAN, L. F. and MICKELSON, S. K. (2006). Assessing and Developing Program Out- comes through Workplace Competencies, In International Journal of Engineering Education, Vol: 22, Iss: 1., p. 121-129, Great Britain. CANADIAN ASSOCIATION FOR CO-OPERATIVE EDUCATION [CAFCE]. (1996). Accreditation Guide and Application, Toronto, ON, Canadian Association for Co-operative Education. CANALE, R., and DUWART, E. (1997).“Re-engineering Cooperative Education, A Call to Action” Proceedings of the Conference for Industry Education Collaboration. Tampa, FL, American Society of Engineering Edu- cation. CANALE, R., and DUWART, E. (2000). Internet Based Reflective Learning for Cooperative Education Students During Co-op Work Periods. Journal of Cooperative Education, Vol. XXXIV, No 2, Spring, 2000 Wash- ington D.C. CARDER, B. and RAGAN, P. (2004). Measurement matters: how effective assessment drives business and safety performance, Milwaukee, WI, ASQ Quality Press. CATES, C. and LANGFORD, D. (2006). The University of Cincinnati Program Based on Core Principles of Coop- erative Education, In Conference Proceedings, CEIA Annual Conference, Defining the Next Century, Wal- nut Creek, CA, Cooperative Education and Internship Association. CATES, C. and JONES, P. (1999). Learning Outcomes, the Educational Value of Cooperative Education, Columbia, MD, The Cooperative Education Association. CATES, C., CEDERCREUTZ, K., and TRENT, L. (2003). “The Current Evolution of Cooperative Education at the University of Cincinnati,” Proceedings of the 2003 Conference for Industry and Education Collaboration. CATTELL, R. B. (1946). Description and measurement of personality, Yonkers-on-Hudson, NY, World book company. CATTELL, R. B. (1973). Factor analysis; an introduction and manual for the psychologist and social scientist, West- port, CT, Greenwood Press. CATTELL, R. B. (1976). The dimensions of groups and their relations to the behavior of members : a large-scale experimental study and a theoretical model, Ann Arbor, MI, Xerox University Microfilms. CEDERCREUTZ, K. and CATES, C. (2006). Using Statistical Analysis of Co-op Student Work Term Performance in the Assessment of Co-op Curricula, Proof of Principle. In Conference Proceedings, CEIA Annual Confer- ence, Defining the Next Century, Walnut Creek, CA, Cooperative Education and Internship Association.

120 Bibliography CEDERCREUTZ, K., CATES, C., ECKART, R. and TRENT, L. (2002). Impact of Outcome Based Accreditation Standards on the Cooperation Between the College of Engineering and the Division of Professional Prac- tice at the University of Cincinnati. In proceedings for Accreditation Board for Engineering and Technology ABET Annual Meeting, 2nd National Conference on Outcomes Assessment for Program Improvement, Balti- more, MD, Accreditation Board for Engineering and Technology. CEDERCREUTZ, K., CATES, C., MALTBIE, C., MILLER, C. and UWAKWEH, B. (2005). The Development of a Corporate Feedback Loop for Use in Curricular Reform. In Proceedings for Accreditation Board for Engi- neering and Technology ABET Annual Meeting, Baltimore MD, Accreditation Board for Engineering and Technology. CEDERCREUTZ, K. (2007), Calibration of a Work Performance Assessment Instrument to Support Continuous Improvement of Cooperative Education Curricula, PhD Dissertation, Cincinnati, OH, Union Institute & University/ProQuest. CHEN, HUEY-TSYH. (2005). Practical Program Evaluation: Assessing and Improving Planning, Implementation, and Effectiveness, Thousand Oaks, CA, Sage Publications, Inc. CONFERENCE BOARD OF CANADA’S CORPORATE COUNCIL ON EDUCATION. (1992). Employability Skills Profile, Canada, 1992, Conference Board of Canada’s Corporate Council on Education. CONTOMANOLIS, EMANUEL. (2005). “Integrating Cooperative Education Based Student Learning in the Col- lege Classroom: A Study of Engineering Faculty Attitudes and Activities,” Journal of Cooperative Educa- tion and Internships. Volume 39. COUNCIL FOR HIGHER EDUCATION ACCREDITATION [CHEA]. (2006). Directory of CHEA Recognized Organizations 2006-2007, www.chea.org/Directories/index.asp [Data retrieved November 30, 2006]. COX, PAMELA L.; SCHMITT, ELIZABETH DUNNE; BOBROWSKI, PAULA E.; GRAHAM, GLENN. (2005). Enhancing the First-Year Experience for Business Students: Student Retention and Academic Success. Journal of Behavioral & Applied Management, 7/1: 40-68. CRONBACH, L. J. (1946). A case study of the split-half reliability coefficient. In The Journal of Educational Psy- chology, Issue: 37, p. 473-480. CRONBACH, L. J. (1943). On estimates of test reliability. In The Journal of Educational In Psychology, Issue: 34, p. 485-494. CRONBACH, L. J. (1951). Coefficient alpha and the internal structure of tests. In Psychometrika, Issue: 16, p. 297-334. CRONBACH, L. J. (1990). Essentials of psychological testing. New York, NY, Harper & Row, 5th edition CRONBACH, L. J. (1998). Internal consistency of tests: Analyses old and new. In Psychometrika, Issue: 53 p. 63-70. CUMMINS, T. and WORLEY, C. (2001). Organization Development and Change (7th ed) Cincinnati, OH, South- Western College Publishing. DE VAUS, D. A. (2002). Surveys in social research, London, UK, Routledge. DEAN T.E. and KENNEDY, A.A. (2000). The New Corporate Cultures. London, Great Britain, Texere. DEMING, W. E and KILIAN C. S. [compiled by]. (1992). The world of W. Edwards Deming, Knoxville, TN, SPC Press. DEMING, W. E. (1982). Quality, productivity, and competitive position, Cambridge, MA, Massachusetts Institute of Technology, Center for Advanced Engineering Study. DEMING, W. E. (1943). Statistical adjustment of data, New York, NY, J. Wiley & Sons, inc. London, UK, Chap- man & Hall, ltd. DEMING, W. E. (1948). A brief statement on the uses of sampling in censuses of population, agriculture, public health, and commerce, New York, NY, Lake Success. DEMING, W. E. (2000). The new economics: for industry, government, education, 2nd ed., Cambridge, MA, MIT Press. DEWEY, JOHN, (1938). Experience and Education. New York, NY, Macmillan Publishing Company. DEW, J. and NEARING, M. (2004). Continuous Quality Improvement in Higher Education, Westport, CT, ACE/ Praeger Series on Higher Education.

Bibliography 121 DIETRICH, C. F. (1991). Uncertainty, calibration and probability. Bristol, UK, Adam Hilger. DOEBELIN, E. O. (1990). Measurement systems: application and design, New York, NY, McGraw-Hill, 4th ed. DRUCKER, P.F. (1970) Technology, management and society. New York, NY, Harper. EATON, J. S. (2003). Is Accreditation Accountable? The Continuing Conversation Between Accreditation and the Federal Government, In CHEA Monograph Series, Number 1, Washington, DC, Council for Higher Edu- cation Accreditation. EDDY, J. P. and MURPHY, S. D. (2000). International higher education, Lanham, MD, University Press of America. EVERS, T., RUSH, J.C. and BERDROW, I. (1998). The Bases of Competences: Skills for life long learning and employability, San Francisco, CA, Jossey-Bash Publishers. EWELL, P. T. (1984). Conducting student retention studies, New York, NY, National Center for Higher Education Management Systems, College Board. EWELL, P. T. (1983). Student-outcomes questionnaires: an implementation handbook, New York, NY, National Center for Higher Education Management Systems, College Board. EWELL, P.T. (2002). An Emerging Scholarship: A Brief History of Assessment, In T W Banta and Associates, Building a scholarship of assessment, San Francisco, CA, Jossey-Bass. FIEDLER, F. and GARCIA, J. (1987). New Approaches to Effective Leadership. New York, NY, Wiley. FISHER, D. and TOLBERT, W. (1995). Personal and Organizational Transformations: The True Challenge of Con- tinual Quality Improvement. New York, NY, McGraw Hill. FREUND, C. J. (1946). The Co-operative System – A Manifesto. Journal of Engineering Education, Volume 37, Number 2, October, 1946. GAGLIARDI, P. (1986). “The Creation and Change of Organizational Cultures: A Conceptual Framework,” Orga- nizational Studies, 7. GARDNER, J. N. (2006). The First-Year Experience’s First 25 Years—The Jury is In, But Out on the Next: You Can Determine the Verdict. Plenary address presented at the 25th Anniversary First-Year Experience Confer- ence and Gala Celebration, Atlanta, GA. Retrieved May 22, 2006, from www.sc.edu/fye/events/presentation/FYEAnnualConf06PlenarySpeech.pdf. GITLOW, H. S. and LEVINE, D. L. (2005). Six sigma for green belts and champions: foundations, DMAIC, tools, cases, and certification, Up-per Saddle River, NJ, Pearson/Prentice Hall. GREENWOOD, R., SUDDABY, R., and HININGS, C.R. (2002). “Theorizing Change: The Role of Professional Associations in the Transformation of Institutional Fields.” Academy of Management Journal, 45. GRUBB, W. N. and MARVIN, A.L. (2005). Vocationalism in Higher Education: The Triumph of the Education Gospel. Journal of Higher Education, 76(1), p. 1-25. HANDEL, M. J. (2003). The Sociology of Organizations: Classic, Contemporary and Critical Readings. Thousand Oaks, CA, Sage Publications. HANNEMAN, L. F., MICKELSON, S. K., PRINGNITZ, L. K. and LEHMAN, M. (2002). Constituted-Created, Competency Based ABET Aligned Assessment Tools for the Engineering Experiential Education Work Place. In Proceedings for Accreditation Board for Engineering and Technology ABET Annual Meeting, 2nd National Conference on Outcomes Assessment for Program Improvement, Baltimore, MD, Accreditation Board for Engineering and Technology. HAYLER, R. and NICHOLS, M. (2005). What is six sigma process management? New York, NY, London, Great Britain, McGraw-Hill. HATCH, M. J. (1993). “The Dynamics of Organizational Culture” Academy of Management Review, 18. HENDRY, C. (1996). “Understanding and Creating Whole Organizational Change Through Learning Theory” Human Relations, 49. HEWETT, D. (1998). “Cooperative Education and EC 2000” Proceedings of the Conference for Industry Education Collaboration. Palm Springs, CA, American Society of Engineering Education.

122 Bibliography HIAM, A. (1993). Does Quality Work: A Review of the Relevant Studies. New York, NY, The Conference Board. HOEY, J. and NAULT, E. W. (2002). Progress in Assessment Obstacle Analysis, Proceedings for Accreditation Board for Engineering and Technology ABET Annual Meeting, Baltimore MD, 2nd National Conference on Out- comes Assessment for Program Improvement, Accreditation Board for Engineering and Technology ABET. HOEY, J., MARR, J. and GARDENER, D.C. (2002). Multiple Vantage Points for Employment Related Feedback: Some Vantage Points, In Proceedings for Accreditation Board for Engineering and Technology ABET Annual Meeting, 2nd National Conference on Outcomes Assessment for Program Improvement, Baltimore, MD, Accreditation Board for Engineering and Technology. HOLBECHE, LINDA. (2006). Understanding Change: Theory, Implementation and Success. Oxford: Elseiver/But- terworth-Heinemann. HOUSE, M. (1996). Confessions of a quality expert: a systematic approach to implementing total quality, Eastham, UK, Tudor. HOUSE, R. and BAETZ, M. (1979). “Leadership: Some Generalizations and New Research Directions” in Research in Organizational Behavior, B.M Shaw editor. Greenwich, CT, JAI Press. HOWARD, A. (1986). “College Experiences and Managerial Performance.” Journal of Applied Psychology. HUBA, M. and FREED, J. (2000). Learner-Centered Assessment on College Campuses: Shifting the Focus from Teaching to Learning, Needham Heights, MA, Allyn & Bacon, A Pearson Education Company. IRESON, W. G., COOMBS, C. F. and MOSS R. Y. [Eds]. (1996). Handbook of reliability engineering and management, 2nd ed., New York, NY, McGraw Hill. ISO [International Organization for Standardization]. (1993). (A). Guide to the expression of uncertainty in measurement, Geneva, Switzerland, International Organization for Standardization. ISO [International Organization for Standardization]. (1993). (B). International Vocabulary of Basic and General Terms in Metrology, 2nd ed., Geneva, Switzerland, International Organization for Standardization. JENKINS, L. (2003). Improving student learning: applying Deming’s quality principles in classrooms, 2nd ed., Mil- waukee, WI, ASQ Quality Press. JOHNS, G. (1983). Organizational Behavior: Understanding Life at Work. Glenview, IL: Scott Foresman. KENNEDY, D. (1997). Academic Duty. Cambridge Mass: Harvard University Press. KING, B. (2001). The Co-op Supervisor’s Role as Teacher in the Workplace, Journal of Cooperative Education, Vol- ume XXVI, No. 3. KOLB, D., RUBIN, I., MCINTYRE, J., (1984). Organizational Psychology: An Experiential Approach to Organiza- tional Behavior. Englewood Cliffs, NJ, Prentice-Hall. KUH, G.D. (2001). The National Survey of Student Engagement: Conceptual Framework and Overview of Psycho- metric Properties. Bloomington, IN: Indiana University Center for Postsecondary Research. Retrieved May 21, 2006 from http://nsse.iub.edu/pdf/conceptual_framework_2003.pdf. KUH, G. D., KINZIE, J., SCHUH, J. H., and WHITT, E. (2005). Student success in college: Creating conditions that matter. San Francisco, CA, Jossey-Bass. LANGFORD, D. and CATES, C. (1995). Developing General Education skills through co-operative education, In The Proceedings of the World Conference on Cooperative Education, Jamaica, Boston, MA, World Associa- tion for Cooperative Education Inc. LATZKO, W.J. and SAUNDERS, D. (1995). Four days with Dr. Deming: a strategy for modern methods of management, Reading, MA, Addison-Wesley Pub. Co. LESKES, A. and MILLER, R. (2006). Purposeful pathways: helping students achieve key learning outcomes. Wash- ington, DC, Association of American Colleges and Universities. LEWIS, R.G., and SMITH, D. H. (1994). Total Quality in Higher Education. Delray Beach, FL, St. Lucie Press. LOCKHART, R.S. (1998). Introduction to statistics and data analysis for the behavioral sciences New York, NY, W.H. Freeman. LUCAS, A. (2000). Leading Academic Change. San Francisco, CA, Jossey Bass.

Bibliography 123 MADU, C. N. [Ed].(1998). Handbook of Total Quality Management, Dortrect, The Netherlands, Kluwer Academic Publishers. MALTBIE, C, MATILUS, J. K. and CATES, C. (2006). Employers Expectations for Co-op Students, Ways to Iden- tify Expectations and Incorporate them into Curriculum Decisions, In Conference Proceedings, CEIA Annual Conference, Defining the Next Century, Walnut Creek, CA, Cooperative Education and Internship Association. MARINI, ROBERT C. (1995), 9th World Conference on Cooperative Education, Conference Proceedings. Giving Graduates Worldwide the Business Skills They Need Through Cooperative Education. MARSIC, V. (1987). Learning in the Workplace. New York, NY, Croom Helm. MCKENDRICK, N. (1987). Learning and Apperception through Cooperative Education. Fifty Views of Cooperative Education. Detroit, MI, University of Detroit. MCNIFF, J. (2000). Action Research in Organizations, London, UK, Routledge. MIDDLE STATES ASSOCIATION OF COLLEGES AND SCHOOLS [Middle States Commission on Higher Edu- cation]. (2005). Assessing Student Learning and Institutional Effectiveness: Understanding Middle States Expectation, Philadelphia, PA, Middle States Association of Colleges and Schools, Middle States Commis- sion on Higher Education. MINK, O., OWEN, K., and MINK, B. (1993). Developing High Performance People. Reading, MA, Addison-Wesley Publishing Company, Inc. NOLAN, T. D. (1988). Task Analysis of the Cooperative Education Coordinator, Twenty-fifth Anniversary Confer- ence Proceedings, March 1988, Cooperative Education Association, Inc. ONWUBOLU, G. C. (2005). Mechatronics: principles and applications/Oxford, Great Britain, Burlington, MA, Elsevier Butterworth-Heinemann. PALOMBA, C. A. and BANTA, T. W. (1999). Assessment essentials: planning, implementing, and improving assessment in higher education, San Francisco, CA, Jossey-Bass Publishers. PANDE, P. S., NEUMAN, R. P., and CAVANAGH, R. R. (2000). The Six Sigma way: how GE, Motorola, and other top companies are honing their performance, New York, NY, McGraw-Hill. PARK, C. W. (1916). The cooperative system of education: an account of Cooperative Education as developed in the College of Engineering, University of Cincinnati, Washington DC, Govt. Printing Office. PARK, C. W. (1943). Ambassador to industry; the idea and life of Herman Schneider, Indianapolis IA, New York, NY, The Bobbs-Merrill Company. PHANZAGL, J., BAUMANN, V. and HUBER, H. (1971). Theory of measurement, Würzburg–Vienna, Physica Verlag. PFEFFER, J. and SALANCIK, G. (1978), The External Control of Organizations. New York, NY, Harper and Row. PIERCE, J. February, (1998). “Developing a Framework for the Assessment of Engineering Education: The Role of Cooperative Education” Proceedings of the Conference for Industry Education Collaboration. Palm Springs, CA: American Society of Engineering Education. POSKANZER, S. G. (2002). Higher education law: the faculty, Baltimore, MD, Johns Hopkins University Press. POOLE, M. S. and VAN DE VEN, A. H. (2004). Handbook of Organizational Change and Innovation. New York, NY, Oxford University Press. REILLY, M. B. (2006). The Ivory Tower and the Smokestack: 100 Years of Cooperative Education at the University of Cincinnati. Cincinnati, OH, Emmis Books. RHODES, F. H. (2001). The Creation of the Future: The Role of the American University. Ithaca, NY, Cornell Uni- versity Press. ROWDEN, R. (2001). “The Learning Organization and Strategic Change” Society for the Advancement of Manage- ment Journal 66(3). RUBEN, B. D. (2001). Excellence in higher education 2001-2002: a Baldrige-based guide to organizational assessment, planning and improvement, Washington, DC, NACUBO.

124 Bibliography RUBEN, BRENT D. ET AL. (1995). Quality in Higher Education. New Brunswick, NJ, Transaction Publishers RUBEN, BRENT D. (2004). Pursuing Excellence in Higher Education: Eight Fundamental Challenges (1st ed.). San Francisco, CA, Jossey-Bass. SCHEIN, E. H. (1992). Organizational Culture and Leadership. San Francisco, CA, Jossey-Bass. SCOTT, W. R. (2001). Institutions and Organizations. Thousand Oaks, CA, Sage. SECRETARY’S COMMISSION ON ACHIEVING NECESSARY SKILLS.(1991). What work requires in schools, A SCANS Report for America 2000, Washington, DC, US Department of Labor. SESSA, V. and LONDON, M. (2006). Continuous Learning in Organizations, Mahwah, NJ, Lawrence Erlbaum Associates. SENGE, P. M. (2006). The Fifth Discipline: The Art and Practice of the Learning Organization. New York, NY, Cur- rency Doubleday. SENGE, P., ART, M., ROBERTS, C., ROSS, R., ROTH, G., and SMITH, B. (1999). The Dance of Change: The Chal- lenges to Sustaining Momentum in Learning Organizations. New York, NY, Currency Doubleday. SHELDRAKE, J. (1996). Management theory: from Taylorism to Japanization, Boston, MA, International Thomson Business Press. SHIBA, S., WALDEN D., GRAHAM A. and PETROLINI J. (2001). Four practical revolutions in manage-ment: systems for creating unique organizational capability, Productivity Press; Cambridge, MA, Center for Quality of Management. SOKAL, R. and ROHLF R. (1981). Biometry: the principles and practice of statistics in biological research,2nd ed., San Francisco, CA, W. H. Freeman. SPEARMAN, C. E. (1923). The nature of “intelligence” and the principles of cognition, London, UK, Macmillan & Co., Limited. SPEARMAN, C. E. (1927). The abilities of man: their nature and measurement, New York, NY, Macmillan. SPEARMAN, C. E. (1931). Creative mind, New York, NY, D. Appleton, Cambridge, UK, The University Press. STEERS, R.M and BLACK, J.S. (1994), Organizational Behavior, 5th Ed. New York, NY, Harper Collins College Publishers. SUSKIE, L. (2000). Fair Assessment Practices: Giving Students Equitable Opportunities to Demonstrate Learning, AAHE Bulletin, Vol: 52, No: 9, p. 7-9. SUSKIE, L. (2004). Assessing student learning: a common sense guide, Bolton, MA, Anker Pub. Co. SUSKIE, L. (2006). Accountability and Quality Improvement, In: Hernon P., Dugan R. E. and Schwartz C. [Eds] Revisiting outcomes assessment in higher education, Westport CT, Libraries Unlimited. TABACHNICK, B. G. and FIDELL, L. (2001). Using multivariate statistics, Boston, MA: Allyn and Bacon, 4th ed. TARKKONEN, L.(1987). On Reliability of Composite Scales, In Statistical studies 7, Helsinki, Finland, Finnish Sta- tistical Society. TAYLOR, F. W. (1915). Principles of scientific management, New York, NY, London, UK, Harper & Brothers. TAYLOR, K., with MOORE, W. S., MACGREGOR, J. and LINDBLAD, J. (2003). Learning community research and assessment: What we know now. Olympia, WA: The Evergreen State College, Washington Center for Improving the Quality of Undergraduate Education, in cooperation with the American Association for Higher Education. TAYLOR, V. (1995), Cooperative Education as a Delivery System for the SCANS Workplace Competencies and Foundation. Journal of Cooperative Education, No. XXX, Vol. 2. THE HIGHER LEARNING COMMISSION. (2003). Handbook of Accreditation, 3rd ed., Chicago, IL, The Higher Learning Commission. THE HIGHER LEARNING COMMISSION. (1997). Handbook of Accreditation, Chicago, IL, The Higher Learning Commission, 1st ed. TIERNEY, WILLIAM G. (1998). The Responsive University: Restructuring for High Performance. Baltimore, MD, Johns Hopkins University Press.

Bibliography 125 TINTO, V. (1997). Classrooms as communities: Exploring the educational character of student persistence. The Journal of Higher Education, p. 68, 599-623. TOBOLOWSKY, B. F. (2005). The 2003 national survey on first-year seminars: Continuing innovations in the collegiate curriculum (monograph no. 41). Columbia, SC, University of South Carolina, National Resource Center for First-Year Experience and Students in Transition. TROMP, S A. and RUBEN, B. D. (2004). Strategic Planning in Higher Education: A Guide for Leaders. Washington DC: NACUBO. TSUTSUI, W. M. (1998). Manufacturing ideology: scientific management in 20th-century Japan, Princeton, NJ, Princeton University Press. TULEJA, ELIZABETH A. and GREENHALGH, ANNE M. (2008). Communicating across the Curriculum in an Undergraduate Business Program: Management 100—Leadership and Communication in Groups. Busi- ness Communication Quarterly, 71/1: 27-43.” (p. 40). UNIVERSITY OF CINCINNATI, (2001). Co-op Curriculum Materials, Professional Practice, McGraw-Hill Primis Custom Publishing. UNIVERSITY OF CINCINNATI. (2006). Student Handbook, Division of Professional Practice, Cincinnati, OH, University of Cincinnati. U.S. DEPARTMENT OF EDUCATION. (2006). A Test Of Leadership, Charting the Future of U.S. Higher Educa- tion, A Report of the Commission Appointed by Secretary of Education Margaret Spellings, Pre-Publication Copy, Washington, DC, U.S. Department of Education. VAN DER VORM, P. (1995). Cooperative Education and General Education: A Partnership with Potential. Journal of Cooperative Education, Vol. XXX, No. 1. VAN GYN, G. (1994). The Educational Orientation of Cooperative Education: A Critical Variable in Effectiveness. Journal of Cooperative Education, Vol XXX, No. 1. VEHKALAHTI, K. (2000). Reliability of Measurement Scales; Tarkkonen’s general method super-sedes Cronbach’s alpha, Litentiate Dissertation, PDF version Helsinki, Finland, Helsingin Yliopiston Verkkojulkaisut. WANG, J .X. and ROUSH, M. L. (2000). What every engineer should know about risk engineering and management, New York, NY, Marcel Dekker. WANG, J. X. (2005). Engineering robust designs with six sigma, Upper Saddle River, NJ, Prentice Hall PTR. WEISBORD, M. R. (2004). Productive workplaces revisited: dignity, meaning, and community in the 21st century, 2nd ed., San Francisco, CA, Jossey-Bass. WEISZ, M. and CHAPMAN, R. (2004). Benefits of Cooperative Education for Educational Institutions, In Coll R. K. and Eames C. [Editors] International Handbook for Cooperative Education, An International Perspec- tive of the Theory Research and Practice of Cooperative Education, Boston, MA, World Association for Cooperative Education Inc. WILSON, J., STULL, W. and VISONHALER, J. (1996). Rethinking Cooperative Education. Journal of Cooperative Education. Volume XXXI, No. 2-3. WILSON, J. (1997). On the Questions Asked about Cooperative Education. Journal of Cooperative Education. Vol- ume XXXII, No. 2. WOOLSON, R. F. and CLARKE, W. R. (2002). Statistical methods for the analysis of biomedical data, 2nd ed., New York, NY, Wiley-Interscience. WORLD COMMISSION ON ENVIRONMENT AND DEVELOPMENT, (1987). Our Common Future, Report of the World Commission on Environment and Development,. Published as Annex to General Assembly document A/42/427, Development and International Cooperation: Environment August 2, 1987. WREN, D. A. and GREENWOOD, R. G. (1998). Management innovators, the people and ideas that have shaped modern business, New York, NY, Oxford University Press. ZINK, K. J. (1998). Total Quality Management as a holistic management concept: the European model for business excellence, Berlin, Germany, New York, NY, Springer.

126 Bibliography

Leveraging Cooperative Education to Guide Curricular Innovation The Development of a Corporate Feedback System for Continuous Improvement

Have you ever thought of implementing employment oriented assessment on your campus?

What would it take to make it happen?

This book walks you through the process in light of real world examples and case studies. It covers the process all the way from pilot to the gates of institutionalization. Cheryl Cates, MBA, ABD Kettil Cedercreutz, PhD

“I am delighted to have the opportunity to comment on this most impressive research project and the very comprehensive publication– Leveraging Cooperative Education to Guide Curricular Innovation. I have witnessed the development of this innovative corporate feedback system and have been so impressed with the creative and very purposeful work of Cheryl Cates and Kettil Cedercreutz. I believe this feedback model will be useful to the many co-op professionals who are committed to improving the educational impact of cooperative education.” — Dr. Paul J. Stonely CEO, World Association for Cooperative Education (WACE) President, National Commission for Cooperative Education (NCCE)

“The development of a corporate feedback system as described in this FIPSE study goes to the very heart of Cooperative Education, which is to partner industry and academics for the improvement of education. Two thumbs up! UC has hit a home run with this one.” — Tom Akins Executive Director, Division of Professional Practice, Georgia Institute of Technology Past President, Accreditation Council for Cooperative Education

“The University of Cincinnati has emerged as the world leader as it delves into the impact of cooperative education on curricula development and/or enhancement. As a professional with 37 active years of experience in cooperative education, UC’s research is the most significant of which I am aware. The empirical data collected, analyzed, and applied by UC has profound implications to document the value and further link academia and structured cooperative education work experiences. The international academic community will highly value the methodology and findings of the University of Cincinnati’s work.” — Dr. Luther Epting Director of Career Services, Mississippi State University (retired) Fellow, American Society for Engineering Education

Center for Cooperative Education Research and Innovation University of Cincinnati P.O. Box 210115 Cincinnati, Ohio 45221

ISBN: 978-0-615-25136-3