Executive Summary of Assessment of Student Learning and Development

Executive Summary Number 5 Assessment of Student Learning and Development

2001-2002

Concordia University, St. Paul

Submitted by

Miriam Luebke Associate Dean for Academic Affairs

David Stueber Director of Institutional Research Executive Assessment Summary, 2001-2002

Executive Summary of Assessment of Student Learning and Development

Number 5

2001-2002

Introduction

Assessment in Higher Education Today

Assessment in higher education began as a response in the 1980s to public insistence on educational accountability. Since then, regional accreditation agencies as well as various discipline and professional accreditation groups have included demonstrations of student learning outcomes in their requirements. Performance-based or outcomes-based budgeting is part of spending formulas for public colleges and universities in at least half the states (Lopez, 1999). Lopez, of the Higher Learning Commission, predicts that the “public demand for educational accountability will continue to require colleges, universities and systems of higher education to document and improve student learning” (1999, p. 6).

Even more important than accountability mandates, the essential goal of assessment is to improve the quality of student learning by improving the practices that impact learning. The goal of the assessment process is the improvement of all institutional practices that affect student learning and development, from the classroom to the athletic field. It begins with setting clear objectives for student learning and development that are consonant with the University’s mission statement. It continues with the gathering of meaningful information to measure the accomplishment of institutional, departmental, and classroom objectives. The assessment “loop” is complete only when the information gained from this process informs planning and decision-making in each and every department and results in constructive change.

Relationship of Assessment of Learning to Other Measures of Institutional Effectiveness

The assessment of student learning and development, which is covered in this document, is only one aspect of the evaluation of institutional effectiveness (Lopez, March 1996). The evaluation of other aspects of institutional effectiveness is carried out by department chairs, deans, program directors, and vice-presidents as they complete program evaluations, annual reports, and other activities to evaluate the achievement of Concordia University’s strategic goals. The following sections summarize the ongoing departmental and institutional activities that assess student learning and development and the activities that impact them.

Summary

The 2001-2002 reports submitted by faculty assessing student learning in their academic majors continue to demonstrate that most faculty at Concordia University are fully engaged in reflecting on their practice as teachers and on improving the curriculum and the quality of student learning. Academic majors and graduate programs at Concordia University continue to include a wide range of assessment activities, for example, capstone courses, internships, portfolios, performances, exhibitions, case studies, business plans, comprehensive exams, research projects, and conference presentations.

Most departments were able to document their intentional efforts to review evidence of the learning that has taken place, and then, where there are disappointments or concerns about the outcomes, to honestly deliberate about how to improve the depth, quality, or other aspects of that learning. Some departments

2 Concordia University, St. Paul Executive Assessment Summary, 2001-2002 indicate that they are making or considering changes in pedagogy, course content, learning objectives, or assessment measures in response to assessment results. Changes made previously in response to past assessment results have been deemed effective in addressing areas of concern. Above all, it appears that, where assessment has seemed to make the greatest impact, faculty members have engaged in discussions around the common goal of improving the quality of their students’ learning,

Assessment of learning in the academic major or graduate program continues to be a strength for Concordia University. Systematic assessment of learning outcomes in general education, however, is still in the developmental stages. One assessment plan that was first introduced in fall 2000 was revisited by a task force in June 2002, and will be further developed and implemented in 2002-03.

Despite the challenges of limited personnel and other resources, faculty overall are finding ways to review evidence of student learning and development, analyze and discuss its implications, and make changes intended to improve the quality of that learning. For the most part, faculty reported that students are satisfactorily meeting the goals and objectives set for academic programs, despite continuing significant variability in the quality of students’ performances.

Learning and Development in General Education

In fall 2001, the general education committee conducted an audit of all the general education offerings using the mission and criteria for general education courses that they had developed during 2000-01, and which were approved by Faculty Senate the previous May. Faculty teaching general education courses were asked to submit syllabi along with a form where they would indicate how the course fit the mission and criteria for a general education offering.

The plan for assessing general education learning outcomes that was developed during the summer of 2000 by the University assessment team and the general education committee was put on hold during this time. (For a brief summary of the plan, see the Executive Assessment Summary for 2001.) This plan was initiated in August 2000, focusing on course-level assessment, but documentation of these assessment activities has not occurred and the continued implementation of the assessment plan has not taken place. The focus of the GE Committee this year was the course audit (see above) and the development of a proposal for a reduction in the number of credits required for general education.

In March 2002, the University assessment team proposed to the general education committee that it would take responsibility for designing the next phase in the assessment of general education, which it would implement the following year (2002-03), and then turn over to the general education committee for continued implementation and oversight (2003-04). A task force, made up of University assessment team members and a member of the general education committee, convened in June 2002 to design a method for assessing learning outcomes in general education courses.

During the 2001-2002 academic year, faculty members who have received funds from the Bush Foundation faculty development grant focused on the creation or modification of general education courses to enhance students’ critical thinking skills. Nine “mini-grant” projects were completed, and eight of those reported that their goal of increasing students’ abilities to apply, synthesize, analyze, and evaluate were achieved or achieved in part. All grant participants indicated that they had learned methods of improving students’ critical thinking skills that they would apply in other courses that they taught.

Although learning outcomes such as information literacy and technology competency may be addressed in individual courses or programs, they have not been intentionally assessed across colleges, departments, or courses. During 2001-2002, library faculty and staff had two opportunities to informally assess the level of information literacy and technology competency of CU students. One such opportunity came in the form of a survey completed by the acting library director, which revealed, based on experiential evidence, that: 1) the campus has not developed a common definition of information

Concordia University, St. Paul 3 Executive Assessment Summary, 2001-2002 literacy; 2) our institution does not gather evidence that students are information literate when they graduate; and 3) “only a few” departments/programs have a technology competency requirement or information literacy requirement. Based on faculty collaborations with library staff, the acting library director indicated that “more than half” include course assignments that require the development of information literacy; “about half” assess information literacy as part of course grades; and “less than half” work with library professionals to outline key information literacy learning outcomes. These observations suggest that information literacy is a priority for some programs or departments, but that it is not yet seen as a primary learning outcome across disciplines and departments.

During one Bibliographic Instruction session, library staff surveyed a group of 18 undergraduate students in their third year about their bibliographic skills. More than half indicated that they had not known before this session how to search the library’s most basic resources (e.g., CLICnet, Infotrac, weblinks, etc.). Librarians expressed concern that students are not building these skills early enough in their college experience in an intentional enough way, and that the quality of student research would be improved by teaching them these skills explicitly. As a result, library staff are increasing efforts to make faculty and staff aware of these resources and providing workshops and in-class sessions on how to access them.

Again this year, many assessment reports for the academic majors and graduate programs noted that students’ writing skills needed improvement. In previous executive summaries, intentions were expressed to assess student writing skills across the institution, but none of these intentions have yet been carried out.

Learning and Development in the Academic Major

Academic departments document their assessment activities and use of assessment results for improvement in an annual assessment report for each major or graduate program. Twenty-six assessment reports for 2001-02 were completed for undergraduate academic majors, and reports were received for six graduate programs. As a follow-up activity, assessment strategies in the majors, along with their successes and challenges, were reviewed in small group settings with faculty peers, facilitated by a member of the university assessment team. Each group discussed the following questions regarding their own majors/programs: 1) “What assessment activity from this past year gave you information you’re excited about or concerned about? 2) “What kinds of changes are you making in response to what you’ve learned from assessment activities? 3) “What processes have you developed for gathering information from individual assessments for a ‘second look’ for program assessment?” The faculty were fairly evenly split regarding their preference for this kind of interactive discussion of their assessment activities versus the written feedback they were given on their reports in previous years.

Brief summaries of 2001-2002 assessment activities for majors and programs, as reported in the annual assessment reports from academic departments, follow.

College of Arts and Sciences

Department of Art

Similar assessment reports were completed for studio art, community arts, and K-12 art education majors. Procedures for assessment of student learning include portfolio review in course work, periodic critiques of student art in class, and comparisons of student success in off-campus evaluations or their art. The senior capstone experience includes a portfolio review and exhibition and the development of an Artist’s Statement (analysis of the student’s work and linkage to other art works and historical influences). It was found that Concordia students produced more works of art than students at comparison schools and work of similar quality, all current Concordia art graduates produced at least one major senior exhibition with multiple studio disciplines and one area of strength, graduates developed adequate Artist’s Statements, and advanced students were more adept at making historical allusions in work and discussions. Faculty

4 Concordia University, St. Paul Executive Assessment Summary, 2001-2002 indicated that their discussions of outcomes and student feedback led to improvement in course content and the development of a more rigorous and useful capstone experience. Professors also mentored each other more frequently when teaching advanced students. Revisions have been made in art education coursework to comply with state licensure guidelines.

Department of Business

The department of business provided assessment reports for the accounting, marketing, and finance majors. In all three majors, multiple activities at the course level were used to assess student learning. A standardized exam in business (Major Field Test in Business) was also administered. The median score for Concordia’s business students taking the test during 2001-02 was slightly higher than the national median and the Concordia mean slightly lower, indicating competitive performance by CSP students.

Assessment procedures used to assess students’ learning in the finance major included a financial analysis of a major corporation and a formal public presentation of the analysis, income tax research problems, and a paper analyzing the airline industry. The students did well on the oral presentation but not as well on the related financial analysis. Faculty noted improvement on students’ abstract thinking and analytical skills, but that writing skills needed improvement. Based on assessment results and department discussions, it was decided to cover less material in major courses, but in greater depth. Some classroom routines were also modified.

In the marketing major, performance assessments were emphasized as on ongoing part of coursework. Assessment procedures included the development of a business marketing plan, case analyses assessing whether or not an organization’s mission is consistent with its culture, a SWOT analysis, analysis of data sets, the preparation of personal interview and focus group material and the conduction of interviews and focus group meetings, the preparation and administration of questionnaires, observation studies, experimental design, and the analysis of secondary data sources. Student performance gave direct evidence of their learning of marketing principles and their ability to apply these principles to actual marketing practices. Based on student success and student feedback, similar assessment procedures will continue to be utilized.

Faculty indicated in the assessment report for the accounted major that primary assessment procedures included weekly quizzes, homework problems, and semester exams in individual courses. An additional assessment procedure was added for 2001-02: a comprehensive problem designed to tie together several different accounting concepts on which the student shows how the concepts are interrelated. The problem is designed to approximate an actual business problem or situation. The assessment report did not provide details about student performance strengths and weaknesses in particular areas.

Department of Communication Studies

Faculty developed a grid indicating where/how the six objectives for the communication major are assessed in individual courses. Internship portfolios, culminating course projects, and senior capstone activities were used to assess student understanding of the theory and process of communication. As part of the senior capstone experience, students continued to present well-received papers at the annual Undergraduate Communication Research Conference at the University of St. Thomas (St. Paul). Department discussions are listed as an assessment procedure for several objectives. Other procedures to assess student learning noted in the report include small group projects, journals, an oral history analysis, reflective writings, a video production project, media analysis paper, speeches, and rhetorical analysis. Considerable detail is provided in the report about the quality of results. Concordia students at the St. Thomas conference were praised for presenting papers that were “well grounded in theory, well presented, written, and using creative instrumentation and operating at a high level of research.” A number of revisions in curriculum and practice are being implemented or discussed based on assessment results. These changes include the team-teaching of one course in the senior capstone sequence to bring in more quantitative elements and the introduction of a communication ethics course.

Concordia University, St. Paul 5 Executive Assessment Summary, 2001-2002

Department of English and Modern Languages

The department of English and modern languages provided an assessment report for the English major. Procedures in the required senior capstone course for majors, ENG499 Framing the Literary Tradition, are used to assess students’ learning. Department faculty team-teach this course. The primary assessment procedures are class discussion and weekly writing, a written examination, and an oral examination. Faculty noticed continued improvement in student learning. They attributed it, in part, to the improved clarity with which expectations in all courses in the major have been communicated to students. The use of a reordered reading list, rather than a chronological list, continues to be effective. In order to improve performance on the oral examination, a preliminary oral exam was included in an earlier required seminar course. This “practice” exam improved performance on the capstone oral exam and will continue to be included in the earlier course. The department is discussing whether or not the capstone course is a sufficient tool to assess student writing. Alternative forms, such as portfolios and/or grammar exams, are being considered. The department continues to make refinements in curriculum and course materials based on assessment results.

Department of History

The department of history report indicated that the goals and objectives of the history major are assessed primarily in course level activities. The assessment procedures include class discussions, written assignments based of selected primary/secondary source documents, test instruments that require students to integrate documents in essay form, research papers and presentations, and creative writing assignments. Students are encouraged to consider internships, and more students chose to take advantage of those experiences this year. Further attempts to promote internship opportunities are being planned. Assignments/discussions related to the oral history project and similar projects continue to provide students with real-life learning opportunities. Students have responded favorably to these learning opportunities. There continues to be a wide range of students’ achievement of learning outcomes. It was found that student performance on use of primary sources improved when written assignments were followed up with class discussions. Faculty agreed that an increasing number of research papers were assessed in the average to above-average range. Information technology is incorporated into all classes, but great differences between student expectations and abilities concerning technology and its classroom applications still exist. Students generally performed well on assignments focusing on the skills of inquiry and critical thinking and on structured readings and in-class discussions. Students’ creative writing skills varied greatly. Faculty noted that lack of strong writing skills holds back many students.

Department of Mathematical and Natural Sciences

The department of mathematical and natural sciences did not submit assessment reports for their individual majors, but instead provided an assessment plan to be implemented for all department majors. The main assessment tool is the development of student portfolios. The level of a student’s progress toward meeting the objectives of a major will be determined in periodic meetings between the student and a departmental advisor and based primarily on the quality of the work documented in the student’s portfolio. Students are also expected to join the science club and make a club presentation during their last year. Students will also take an exit exam during their last year and must achieve a set score or retake the exam. The departmental advisor and other department members assess the achievement level of each graduating student in a major, based on documentation in the student’s portfolio and faculty input. The assessment plan also includes the use of a survey of the prior year’s graduates to determine their opinions about the strengths and weaknesses of their major as it relates to their current position or course of further study. Assessment results will be discussed in departmental meetings and will be used to make adjustments in the majors.

Department of Music

The department of music provided assessment reports for the music, church music, and music education (vocal/instrumental, K-12) majors. The Major Field Test in Music is one of the procedures used for

6 Concordia University, St. Paul Executive Assessment Summary, 2001-2002 assessment in all three majors. The church music test results are described as unimpressive. Scores for music majors were slightly higher than the previous year, and scores were the highest since the tests were first used for music education majors.

Faculty involved in the music major continue to seek an assessment tool to be used in addition to the major field exam to more accurately assess students’ success. Senior recitals presented as part of the senior seminar were consistent with ability. The effect of the addition of a junior recital requirement on the quality of the senior recital will be evaluated next year. The department wants to establish an assessment tool for performance, but it has not yet been created. The audition process for the major is still being used and is deemed as vital to the student’s success in the major. The piano proficiency exam will also continue to be used. The department will seek ways to improve students’ writing skills. The senior seminar requires students to develop a personal philosophy informed by the Framework for Learning.

Students who complete the Director of Parish Music (DPM) program must complete a church music major. Students can, however, complete this major without pursuing DPM certification. In addition to the major field exam, assessment procedures include evaluations of field experiences by music faculty and parish musicians, faculty evaluation of senior projects, and department tracking of the careers of church music graduates. Concern was expressed about allowing students to go into a parish field experience having completed the music major but not the DPM program. Senior projects were satisfactory. Church music alumni were surveyed, but the results were not yet analyzed by the time of the submission of the assessment report. Results of the survey will be used to help shape curriculum.

Since there are many common assessment procedures, the assessment report for the K-12 music education major also contains comments about student progress in the other department majors. In addition to the major field test required of all department majors and the field experiences required of all teacher education majors, assessment procedures include individual course requirements and course evaluations, juried public performances in conjunction with courses and lessons, recital performances, a sophomore year public performance and program application, and a keyboard proficiency exam. Faculty report that student performance varies based on their ability, their work ethic, and the quality of their instruction. Faculty assessment of student performance is shared with students and adjustments made in teaching to meet students needs. The music department is exploring a pre-test to help students prepare for the major field test. The need for more and better communication with students was noted in regard to the keyboard exam. The department is working with students to implement a four-year portfolio process.

Department of Religion and Theology

The department of religion and theology provided assessment reports for the major in theology/minor in confessional Lutheranism and for the Christian outreach major.

To assess student learning in the major in theology and minor in confessional Lutheranism, the department used pre-test and post-test essay questions in the areas of Bible content and Lutheran doctrine. Pre-tests were administered in lower level required courses and the post-tests in an upper level course, THY420 Church and Ministry. Results showed that growth from pre-test to post-test was, on average, less than the previous year on both Bible content and doctrine. Results may lack validity since the post-test essays were not required course components and, therefore, not taken as seriously by students as they should have been. Faculty also expressed concern that post-test essay results were below what the department expects of graduates. To further address the question of accurately assessing graduates’ knowledge of Lutheran doctrine, several department members will construct a common graded assignment to be used in THY420. To further assess knowledge of Bible content, all church work and pre-seminary graduates will take the seminary entrance exams and the results analyzed in light of departmental objectives.

Assessment procedures for the Christian outreach major include a senior capstone course, graduation portfolios, exit interviews, and annual discussions with practitioners in the field. Structured evaluations of interns are conducted early in the internship and again at the end of the experience by the on-site supervisor. The OHSCO (Oswald Hoffmann School of Christian Outreach) faculty meets weekly to

Concordia University, St. Paul 7 Executive Assessment Summary, 2001-2002 discuss program status and meets in May to discuss final assessment results. Faculty reported that all students in the senior capstone course were able to satisfactorily articulate an understanding of their prospective roles as Christian outreach workers. In exit interviews, all graduates expressed satisfaction that the stated goals of the major were attained. Internship supervisors expressed satisfaction with the interns’ academic training. Based on assessment results and OHSCO discussions, the portfolio and entrance and exit interview processes can be improved, a more varied field experience should be developed to better prepare students for internships, technology needs to be more deliberately integrated into the curriculum, and an internship between the junior and senior years is being considered.

Department of Social and Behavioral Sciences

The department of social and behavioral sciences submitted assessment reports for their majors in psychology, sociology, and criminal justice.

Assessment procedures for the psychology major include the Major Field Test in Psychology, reviews of internship experiences, a graduating senior program evaluation, and review of student presentations at the Minnesota Psychology Undergraduate Psychology Conference. Thirteen students completed the Major Field Test in Psychology in the fall semester and five in the spring semester. The mean for the total score was higher than for last year’s students but slightly below the national mean. The Concordia means were within one standard deviation of the national mean on all subscores. The department will encourage students to take the exam during their last semester of college to take full advantage of course experiences and will continue to use test scores as a factor in the determination of graduation honors in psychology. Other standardized exams will be examined to possibly replace the major field exam. An internship database will be compiled to evaluate the merit of each site for future use. Based on student input, a psychology club was started and an honors program was developed. In the graduating seniors’ program evaluation, professors, curriculum, small class size, student involvement in classes, and the variety of course topics were identified as strengths. Graduates would have liked to see a greater number of courses offered each term. The means on the individual evaluation items were similar to the means in the previous three years. Criteria for graduation with honors were identified. Application will be made to the national psychology honors association. It was noted that eleven past and present students were accepted into graduate/professional programs for 2002-03.

In the sociology major, assessment procedures include activities in required core courses, research papers/projects/discussions in SOC453 Social Theory, SOC454 Sociological Research Methods and Statistics, and SOC456 Seminar in Sociology, the Major Field Test in Sociology, internship evaluations and summary papers, and a graduating senior program evaluation. The mean scores of the eight students taking the major field exam in 2001 and the three taking the test in 2002 were higher than the national mean in 1999 (most recent national mean available). Faculty report that students in research and seminar classes met or exceeded expectations in their ability to connect social theory and practice, process material and lead colleagues in discussions, and design and complete research projects. Based on the seminar experience outcomes, faculty identified the need to keep the seminar in the course rotation and encourage students to enroll. Field supervisors indicated that they were extremely pleased with their interns. Students showed the ability to connect theory and practice in their internship summary papers. Internship sites were evaluated and ranked, and a database is being constructed in order to ensure that students experience appropriate academic and community opportunities. The results of the graduating senior program evaluation showed that all graduating seniors had favorable ratings of the knowledge, teaching ability, and enthusiasm of sociology instructors. They said that they would like to see more sociology professors and a greater number of sociology courses offered.

Assessment procedures in the criminal science major include the Major Field Test in Criminal Justice, internship journals and summary papers, course requirements in SOC354 Sociology of Law and SOC453 Social Theory, and a graduating senior program evaluation. Students demonstrated their understanding of the philosophy of the American criminal justice system and their ability to connect social theory to sociological phenomena and issues in criminal justice in internship papers and in the completion of course assignments. The mean score for students taking the major field exam was 152.8 (range 141- 162). No national means are as yet published for this exam. Faculty have asked for a review copy of the

8 Concordia University, St. Paul Executive Assessment Summary, 2001-2002 exam to compare major objectives with exam content. Field supervisors indicated that they were extremely pleased with interns. Interns showed that they could apply criminal justice tools to community issues. The program earned POST certification (Minnesota Peace Officer Standards and Training Board). Graduates noted POST certification, the professors, and small class sizes as strengths. Instructors were rated highly for their enthusiasm, capability to offer varied views, knowledge, and teaching ability. Students were least satisfied with the variety and elective/required distribution of courses. Since only a few students have completed this relatively new major, faculty have decided to wait for further assessment results before making curriculum changes.

Department of Theatre and Dance

For the theatre major, assessment activities continue to occur primarily in required courses and participation in theatre productions. Faculty indicated that students showed a high degree of knowledge of basic theatre crafts and that the year’s productions showed advancement in student technical abilities. In the area of scholarly research, students showed a high level of proficiency in computer usage. Faculty expressed concern that web-based sources might be over-used and will ask the librarian to visit classes to provide additional instruction in the use of library materials. Students improved in their abilities to analyze texts from a variety of perspectives and in their ability to work collaboratively on projects and presentations. Group projects, script analyses, and analysis papers will continue to be used. It was felt that early exposure to script analysis would benefit students in more advanced courses. Students’ design projects in lighting, scenery, and costumes are described by faculty as being of generally high quality. The assessment results of student direction projects were mixed. Consistency is needed regarding the duties of a student director. The varied acting roles in productions continue to provide students with appropriate challenges. Many actors demonstrated significant growth. Writing assignments related to theatre history were of varied quality. Faculty noted that some “high-end writers” possessed graduate level ability. Based on results of assessment activities involving the evaluation of a theatre going experience from various points of views, faculty would like to encourage greater attendance at and critical discussion of off campus productions.

College of Education

Early Childhood, Elementary, and Secondary Education

The dean of the College of Education provided a unified assessment report for all the teacher education licensure programs. The following assessment procedures are being implemented or in the development stage. At entrance to professional program: At completion of program: 1) Admission to Program portfolio; 1) Professional portfolio; 2) content examination (pre-test) – multiple choice; 2) content examination (post-test) – multiple choice; 3) pedagogy examination (pre-test) – essay; 3) pedagogy examination (post-test) – essay; 4) admission interview; 4) presentation of action-research project; 5) Pre-Professional Skills Test in Reading, Writing, and 5) Principles of Learning and Teaching test (PLT); Mathematics (PPST). 6) standardized content area examination (different for different fields); 7) Student Teaching evaluations by University Supervisor and Cooperating Teacher.

Eighty students were interviewed and admitted into the program during the year. Twenty students submitted content exams to the state and all surpassed mandated cut-off scores. Ratings based on composite scores of university supervisors and cooperating teachers showed that high percentages of student teachers were rated as either proficient or distinguished in the areas of subject matter, learning and behavior, teaching strategies, and personal qualities fostering learning. Some variation in scores, perhaps related to the understanding of the rubric, were found between university supervisors and

Concordia University, St. Paul 9 Executive Assessment Summary, 2001-2002 cooperating teachers. In-house content and pedagogy examinations (pre-tests and post-tests) have not yet been developed but are being considered. Content area focus groups (including students) annually review program content and make suggestions. Topics for further discussion and review based on assessment results include clarity in expectations of licensure candidates and curriculum involving learning and development, how/where candidates show that they use assessment to inform teaching practices, the differences in levels of professionalism found between K-6 and 7-12 teacher education students, and candidate concerns about their action research projects. This last concern resulted in the expectation that university supervisors must directly work with students on these projects.

K-12 Art Education See department of art comments.

K-12 Music Education See department of music report.

K-12 Physical Education See department of kinesiology and health science comments below.

Department of Kinesiology and Health Science

The department provided a combined assessment report for the kinesiology and K-12 physical and health education majors. Assessment procedures are mainly found in individual courses and include lesson plans, presentations, papers, and career interviews. An organization for students with these majors was formed. Faculty reported that students understand theories but that results of assessments of transference from theory to practice show mixed levels of success. The application of theory to practice in the field will continue to be a focus. Students are expected to become proficient in the appropriate use of technology in coursework. Students improved in the technological skills through the year, and this skill development will continue as an emphasis. The department will build on the results of career interviews and presentations to better prepare students for career opportunities.

College of Graduate and Continuing Studies

Concordia School of Accelerated Learning (CSAL)

Organizational Management The director of the BAOM program (Bachelor of Arts in Organizational Management) provided the assessment report. Assessment procedures for the organizational management behavior continue to be found primarily in individual courses and include instructor evaluations of student performance, applied summary projects, student self-evaluations, and exercises in critical thinking. A final research project is required. Case studies and reaction papers have been added in several courses to enhance student learning. The director reports that an employer survey was also used in one course, and employer feedback is being analyzed. Faculty meet informally, on a regular basis, to discuss student progress. Students’ evaluations of instructors indicate that instructors who emphasize small group activities and actively work to develop good writing habits are appreciated. The director reports that standards for evaluating writing projects are not consistent among instructors and recommends further examination of the issue of writing skill development in the program. The determination of whether or not the final research project is achieving program goals remains a topic of ongoing discussion.

Innovation and Marketing Management The program chair for the major in marketing management and innovation indicates that each course in the major contains at least three types of student assessment; weekly readings and assignments, expectations of a high level of student class participation, and a major course paper. Many short papers are included in each course. Case studies are utilized to improve students’ problem solving and decision making skills. During the program, each student designs and completes a “professional grade business

10 Concordia University, St. Paul Executive Assessment Summary, 2001-2002 plan.” The students are also required to give a formal, multimedia presentation of the plan. The business plans are evaluated by program advisors and the program chair. Results of these evaluations indicate that about six of ten plans submitted “could be taken to the market as written” and that three of ten are academically sound, but have at least one serious detriment (usually the financial component). One of ten is deficient in two or more critical areas. One area of concern is the financial statement, and changes in the financial course in the program have been made. The chair estimates that each student writes about 55 papers during the program and reports that writing improves remarkably. He reports that ongoing efforts are being made to ensure consistency in writing standards among all the instructors in the program. Students continue to report their high level of enjoyment with the learning process in the program. An important program goal is to develop team-learning abilities, and students report that their cohort team experiences are positive. Evaluation of the effectiveness of the functioning of a team is improving.

Information Technology in Management Student progress in the information technology in management major is assessed in individual courses through papers, presentations, and other requirements. A final applied research project is also required of all students. In addition to the ongoing evaluation in each course, a focus group of six faculty and one alumnus was formed to assess program objectives by studying portions of six selected student applied research projects. The papers were selected by the project coordinator to include both A and B/C type papers. The focus group used the following grading rubric: 1=doesn’t meet norm, 2=meets norm, 3=exceeds norm. The group averages ranged from 1.9 to 2.2 on the five program goals assessed in this way. The lowest average was 1.9 on the goal related to written and oral communication skills. The highest average was 2.2 on the goal related to the application of technology to a business setting. Instructors are being asked to encourage students to seek writing mentoring or tutorial help in their first courses to start improving writing skills early in the program. Changes have been made in curriculum and course materials to improve students’ understanding of the “convergence of learning technology in the context of the business.” Other changes were made to enhance students’ abilities to go beyond technical expertise to use their expertise in real world situations. Overall quality of student work on the combined research project is improving, in part because of the years of experience of project coordinators. Two program goals are being revised and one added for the 2002-03 year. A new goal involves growth in independent critical thinking skills.

Human Resource Management Classes in this major have now been offered in all discipline areas, and assessment processes are being refined. Assessment procedures in individual courses include writing, speaking, and critical thinking and problem solving exercises. The final project is a major synthesis and analysis paper focusing on a specific discipline within the field. Information from a certification exam offered by the Society for Human Resource Management (SHRM) is being utilized in course construction and assessments. Some instructors are planning to seek SHRM certification, and SHRM certification is a goal for graduates of the program. The department chair reports that many changes in curriculum and the structure of the final project have been made due to feedback from instructors and students. A general weakness in professional writing skills has been found. Adjunct instructors are being encouraged to mentor students in writing skills and/or encourage students to seek help in the Writing Center. Guidelines for evaluating written material have been developed. Exit feedback is being solicited from students in the first cohort nearing completion of the program.

Concordia School of Human Services (CSHS)

School-Age Care The school-age care program is offered in a distance learning format. Assessment procedures for the major in school-age care (SAC) continue to include course synthesis reflections (CSR), professional portfolios, individual course projects, and exit interviews. CSR’s are used to assess student progress and results show that students are meeting goals in many areas. For example, the department chair reports that successes are seen in “positive gains in the learners ability to articulate the role/purpose of SAC, problem solve within their agencies, and advocate for the programs and the needs of children and youth.” CSR information is shared with course instructors before they teach a course a second time. In portfolio

Concordia University, St. Paul 11 Executive Assessment Summary, 2001-2002 and other course work, students demonstrated the “ability to analyze situations and make decisions about intentional-healthy interaction with each child.” Modifications have been made in the curriculum to move learners beyond the term “role model” in discussions of their adult roles. Exit interviews continue to show growth in students’ understanding and confidence. After analyzing exit interviews, the department chair sends a letter to all program faculty containing identified areas of strength and needed improvement. The chair meets with faculty of individual courses to ensure that course content and assessments match program objectives. Changes in curriculum and course materials have been made and continue to be planned based on assessment results. Adjustments made based on past input have led to the use of a new textbook and other individual course materials, have positively impacted portfolio work, and have led to the introduction of a new and well-received course, SAC456 Communicating for Advocacy.

Youth Development This major is also offered in a distance learning format. Assessment procedures for the major in youth development include multiple course assignments, student course evaluations, student email reflections, and a final portfolio. In addition to ongoing course evaluations by students, an “adjunct reflection” will be introduced in 2002-03 to compare student and instructor comments. The department chair reports that students show that they are developing a better understanding of young people, are better prepared to work with young people, are able to articulate the goals of “out of school” programs, and have a sound research base in youth development. Students indicated the desire to have a closer connection made between course and portfolio work. They also indicate that they want to learn more about program advocacy and have a better understanding of how national and community policy impacts youth work. Instructors will be notified of assessment results so that curriculum can be refined to better meet student needs and course and portfolio work can be better aligned.

Graduate Studies

Master of Arts in Organizational Management The Master of Arts program in organizational management (MAOM) is offered through the School of Accelerated Learning. Assessment reports were provided for the ongoing program and also for a new MAOM program with an emphasis in human resources.

Assessment procedures in MAOM include instructor evaluations of student performance, a thesis project, student self-evaluations, a Self-Directed Learning Perception Sale (SDLPS), portfolios, case studies, and summary papers. Faculty noted that SDLPS research conducted in 2001 and other assessment procedures indicate that students are achieving program goals. Students value small group work and appreciate instructors who emphasize and help develop good writing skills. It is reported that writing standards are not consistent among instructors, and related grading policies must be further examined. Thesis chairs must be diligent in working with students on their writing skills. Department discussion continues concerning the effectiveness of the thesis project in demonstrating a student’s depth of learning and critical thinking skills. Case studies and additional summary papers have been added to the curriculum to assess the development of higher order thinking skills. The faculty meets informally but regularly to discuss student progress. A new portfolio process was implemented in 2001. Results of an employer survey will be shared with faculty. Collaboration between CSAL and CSHS faculty will be encouraged in areas of common concern.

A new emphasis in human resources within the MAOM program is designed to meet the needs of students interested in professional enhancement in the area of human resources management. Assessment procedures include entrance and exit portfolios, human resources journals and summary papers, multiple course assessments, and a master’s thesis. Assessment results and implications will be shared in departmental meetings.

Master of Arts in Education with an Emphasis in Early Childhood Education The Master’s degree program in early childhood education is a collaborative effort of the College of Education and the School of Human Services offered in distance learning and face-to-face formats. A major assessment is the field experience paper. These major papers are revisited by students three times during their program; in two synthesizing seminars and in the final capstone course. The early

12 Concordia University, St. Paul Executive Assessment Summary, 2001-2002 childhood education coordinator compiles results from these papers and shares those results with program faculty. The coordinator reports that the addition of a portfolio assignment and other changes in procedures based on previous assessment results have increased the rate of successful completion of the capstone papers. A new assessment procedure planned is the Assessment of Professional Practice in Early Childhood. “Learners will be asked to have a colleague, supervisor, or professional partner complete this short questionnaire to assess the effect that graduate level learning has had on practices of the learner. We expect to see the learner’s self-assessment from their field experience paper supported in the responses on this questionnaire.” Program faculty continue their commitment to well-respected, up-to-date textbooks and readings.

Master of Arts in Education with an Emphasis in Youth Development The Master’s program in youth development is offered through the School of Human Services in a distance learning format. Many courses in the program incorporate multiple assignments. Other assessments include students’ reflective emails and students’ course evaluations. Each student is required to conduct a thesis/research project on a chosen topic and produce a capstone paper on this topic. Planned assessment procedures include periodic reflection assignments and instructor reflections. Assessment results indicate that students have a deeper understanding of the complexity of youth development issues, are able to conduct research, are able to identify key concepts in systems thinking, have developed reflective thinking abilities, and have shown the ability to apply leadership theory to their work situations. Students have indicated that they want more time to discuss field-related issues, have a desire to practice reflection, and want more time to study the topic of leadership. Assessment results will be shared with program faculty. The department chair indicates that assessment results were used in a number of ways. One course was expanded, another one moved to an earlier position in the program, and the expansion of a third course is being considered. Instructors in appropriate courses have been asked to integrate youth development issues, encourage students to apply systems thinking throughout their program, and link leadership and systems coursework. Students will be required to submit periodic reflections in the synthesizing and capstone courses. The department chair will share areas of strength and needed improvement gathered from these reflections with instructors.

Master of Arts in Human Services with an Emphasis in Family Studies This program is offered through the School of Human Services in the distance learning format. Assessment activities included in the comprehensive assessment plan are formative program and objectives evaluations, a summative capstone program and objectives evaluation, course evaluations, and reviews of course assignments. Each individual course in the program has multiple assessment activities. The department chair reported that assessment findings show that students expanded their understanding of the professional role of the family life educator, were able to conceptualize current research trends in the field, and expanded their understanding of cultural factors influencing family life. Students also gained an understanding of how families adapt to stress and crises, demonstrated systemic thinking, expanded their understanding of the stages in family development, deepened their understanding of family communication, were challenged to consider personal biases, developed a “multifaceted understanding” of parenting and parent education, gained a solid foundation in the area of family law and public policy, and clarified their understanding of the principles and philosophy of family life education. Changes are being made in the capstone process to meet changing professional standards and “the student’s ability to negotiate” the process. Refinements in some courses are being made to provide greater clarity and provide more practical application of material. Student feedback has also reinforced earlier decisions made about textbooks and assignments.

Master of Arts in Human Services with an Emphasis in Criminal Justice This program is also offered through the School of Human Services in a distance learning format. The first cohort only recently started, so no assessment results are yet available. Assessment procedures in the assessment plan provided by the department chair include evaluations from students, faculty input and discussion, mid-program cohort meetings, assignments in two synthesizing seminars, and a capstone project. Program faculty are being kept informed of the progress of the first cohort in order to help shape the development of the program.

Student and Alumni Surveys

Student Satisfaction Inventory 2002

A sample of students completed the Noel-Levitz Student Satisfaction Inventory in February 2002. Based on the demographic and academic breakdown of the student sample supplied in the Noel-Levitz Campus Report, the population of traditional students (i. e., students in or intending to complete programs in the College of Education or College of Arts and Sciences) appeared to be reasonably well represented. Many of the results reinforced findings found by Concordia’s Retention Task Force in their analysis of retention data in recent years and in the comments made in focus groups conducted in fall 2001. The strengths noted below also coincide with strengths found in the National Survey of Student Engagement completed in spring 2001.

Areas of Ongoing Strength Students generally appear to be reasonably satisfied with the number and quality of their interactions with faculty. There were 18 items on the inventory related to instruction, advising, and curriculum. When compared to the national private college sample, Concordia students expressed significantly higher levels of satisfaction on 6 of these items. Four of these 6 items related to academic advising. The only significantly negative result in comparing these items concerned the variety of courses offered on campus. When compared with the Concordia sample on the 1999 Student Satisfaction Inventory on these items, there was significant improvement on one item and some improvement on 10 others.

Students expressed high levels of satisfaction with laptop technology at Concordia. The levels of satisfaction improved significantly on 8 of the 9 common items compared to 1999 results, and these levels were already quite high at that time. This ongoing satisfaction should continue to be emphasized in student recruitment. Future improvements in the utilization of laptop technology across the university will build on this strength.

Areas of Ongoing Concern Perhaps some good news from the survey findings is that no new major problems were uncovered. The areas of concern identified in the Summary of Findings have already been discovered in previous research and supported by anecdotal evidence. The statement “Tuition paid is a worthwhile investment” received a significantly lower satisfaction rating by current Concordia students compared to the national sample and to the 1999 Concordia sample. The item “Adequate financial aid is available for most students” was rated significantly lower this time when compared to the national sample and was slightly down from the 1999 Concordia sample. The 2002 survey was completed shortly after the announcement of Concordia’s largest tuition increase in recent years, so this may have skewed the financial responses somewhat. Retention research has shown that inability or unwillingness to pay the cost of education at Concordia is a common reason for leaving listed by students. Continued large increases in student costs must be carefully evaluated. The quality of education needs to be perceived by students as worth the cost.

The fact that the statement “Students are made to feel welcome on this campus” was rated significantly lower in satisfaction by current students than by both 1999 Concordia students and the current national sample is a cause for concern. Service concerns about some university offices are also found in other research by the retention committee, and some actions are being taken and will be taken to improve in these areas. The lower satisfaction ratings from students of color on a few key items is also of concern and has been corroborated in other research findings. Problems with campus maintenance found in the survey had also been identified in previous research, and again solutions are being studied. Follow-up is essential for improvement.

Findings from the Student Satisfaction Inventory have been shared with the Strategic Planning Steering Committee and with the Retention Task Force. Survey findings are being utilized in conjunction with other research findings by these and other groups as strategies for improvement in institutional quality are developed. Alumni Survey 2002

14 Concordia University, St. Paul Executive Assessment Summary, 2001-2002

Surveys were mailed to 2000-01 Concordia graduates (those recorded as having graduated in December 2000, May 2001, or August 2001) during May and June 2002. Eighty-one alumni returned completed surveys, a return rate of 18%. Because of the low return rate, results cannot be expected to reflect the opinions of all 2000-01 graduates. Nevertheless, the results provide useful information when viewed in conjunction with other sources of information. Overall, the comments and ratings of respondents were generally favorable. Even the few people who submitted what would be characterized as generally negative surveys did have some positive comments about their Concordia experiences.

Overall Institutional Reputation and Satisfaction The results indicate that respondents were generally satisfied with their Concordia experiences and consider the academic quality as good. Ratings from graduates of the College of Graduate and Continuing Studies (CGCS) were somewhat higher than the combined ratings of graduates from the College of Arts and Sciences (CAS) and the College of Education (COE).

What made you decide to come to Concordia? Most the CGCS graduates mentioned the length, convenience, and flexibility of the programs and the program fit to their careers. A few mentioned the affordability of the programs, the availability of courses at off campus locations, the Lutheran/Christian setting, and recommendations from others. The reasons mentioned by CAS/COE graduates included the following: the desired program (DCE, Education, Outreach, Pre-Seminary, etc.), location/size/religious orientation of the institution, athletics, availability of scholarships, and institutional ties with family or friends.

What led you to stay at Concordia? What was the most meaningful part of your Concordia experience? Most CGCS respondents mentioned their cohorts and the friendships developed in response to these questions. Other responses included the strong desire to complete what they had started and relationships with instructors. The formation of friendships was the most frequent response by CAS/COE respondents. Other responses included the quality and care of faculty and staff, campus community and spiritual life, the quality of their academic programs, and athletics.

Satisfaction with Concordia Experiences and Services The overall satisfaction average with a wide range of services and experiences was slightly higher for CGCS respondents than for COE/CAS respondents. Highest ratings for CGCS respondents were for the availability of required courses and the variety of course offerings. The highest COE/CAS ratings were for the amount of contact with the faculty and having someone on the faculty or staff with whom to share concerns.

Additional Comments Thirty-two respondents supplied additional comments. Most of the comments were of a positive nature, some expressing gratitude for good educational experiences. Several mentioned that they recommend Concordia to friends. Some negative comments were made about contacts with a few university offices. Several CAS/COE graduates noted some concern about the decline in campus spirituality during their time at Concordia, while others expressed the concern that Concordia was too focused on Lutheran students and their interests.

Submitted by:

Miriam Luebke, Associate Dean for Academic Affairs David Stueber, Director of Institutional Research

References

Indiana University Center for Postsecondary Research & Planning (2001). National survey of student engagement: Overview. Indianapolis, IN: Author.

Lopez, C.L. (1999). Assessing student learning: Why we need to succeed. Assessment and Accountability Forum, Summer1999, 5-7/18.

Lopez, C.L. (March 1996). Opportunities for improvement: Advice from consultant-evaluators on programs to assess student learning. Chicago: North Central Association of Colleges and Schools Commission on Institutions of Higher Education.

16 Concordia University, St. Paul