AOL Retreat

Faculty Feedback Summary Report

Summary

The goals of the Assurance of Learning (AOL) WSBE Faculty Retreat held on September 28th were to: (1) inform the faculty of the AOL process and its role in AACSB re-accreditation; (2) inform faculty of the Undergraduate and Graduate Curriculum and Assessment Committees’ AOL efforts to date; (3) discuss ideas to improve student performance based on current metrics; and (4) receive feedback on current AOL goals and measures. This report is a summary of the faculty input collected and discussed, as well as a discussion of how both committees are planning on using the information moving forward.

During the Retreat faculty broke out into small groups to discuss 7six different content areas, each tied to the learning goals set out for the AOL and AACSB accreditation. These groups discussed their content area in terms of both graduate and undergraduate programs (see Table below). The small group discussion topics were:

AOL Learning Goal(s) Undergraduate Graduate (MBA) 1.1 Graduate will be critical thinkers and Qualitative Problem Solving proficient problem solvers 1.2 Graduates will be effective team Leadership/Teamwork members and leaders 1.3 Graduates will be effective communicators to a variety of business Communicators audiences  Oral  Written 1.4 graduates will be prepared for a career Global Awareness in a global economy 1.5 Graduates will have developed an Ethical Awareness ethical awareness and perspective Sustainability No goal set at this time

Overall three common themes that emerged during multiple small group discussions are listed below, along with responses from the Committees, Venky, Roger, and Cynthia.

 The benefits of multiple evaluators

o We agree that, given unlimited resources, having more than one faculty member assess each piece of work would produce more robust results. However, we also note that we do have resource constraints and that AACSB does not require multiple raters. So we are going to be happy to have multiple raters when we can get enough faculty members to volunteer for this to be possible, but our primary goal is to make sure that everything gets assessed once. Note that just to get everything assessed once, we are going to need faculty beyond the AOL committees to help out. Please keep this in mind when we ask for volunteers. As our AOL program reaches a steady state, we will be in a better position to use multiple assessors for each objective. Can we afford not to do it the other way around? That is, to have multiple evaluators validate the assessment tool and once calibrated, go the single evaluator route?

 Measuring learning goals within options and electives as opposed to core courses

o This is an issue that both committees started discussing years ago. There are a number of cases where the easiest place to get measurement data would be in an elective or option course. The problem is that AACSB requires that we present data that represent our entire student population for an entire degree program. So if we wanted to gather data on a particular goal (say writing in the BSBA) in option-required courses, we’d have to gather data from at least one required course in EACH of our seven options. This presents additional challenges of consistency and the further spreading of limited faculty resources. For this reason, collecting data in core courses is really our only practical method (again, keeping in mind resource constraints).

 The need for periodic summary reports on assessments for all programs and goals to be provided to faculty and discussed at faculty/department meetings and through other forums (i.e., a website, e-mails)

In addition, I suggest bringing to awareness the state of where we are right now and where we end up providing we follow current course and pace.

A list of specific comments (by content) area are listed below, along with committee responses and plans for future work.

1. Comments and Suggestions on Ethics

Suggestions for all programs

 Student/faculty statement of ethical commitment

 Guest speaker series

 Long-Term - Consider a core course (credit or no credit) that sets the foundation for what ethics are, what they mean, etc. using any or all in combination – practitioner, academic, or UNH Law.

 Long-Term – create core content that should be taught in all courses, drawing from this core course and allowing faculty to tailor to their unique course content (e.g., accounting, finance, HR, etc.)

 Legal vs. Ethical issues  All disciplines have codes of professional and ethical behavior; incorporate these into individual core courses.

Suggestions for BSBA

 Course structure/design (assignments/exams)

 Improve civility in classroom

 ECON – Misconduct in business. Current examples presented in class.

 OB – Case studies containing ethical dilemma

Suggestions for BS in HMGT

 HR Case - whistleblowers

 ID (un)ethical behavior

 Self-test

Suggestions (?) for MSA

 Course (ethics and professional practice)

Suggestions for MBA

 Guest speakers on ethics topics

 Add cases covering ethical situations in business settings

Committee Discussion/Action:

For both the Undergraduate and Graduate programs where, when and how to measure ethics is a topic of concern. Based on conversations at the retreat, faculty cover ethics in a variety of courses. In the short term, both committees will be talking with individual faculty members to gain a better understanding of how ethics-related topics are taught as well as what samples of student work we can readily collect and use to assess our students understanding and awareness of ethics. Long-term suggestions included developing a possible business ethics course and/or better core content to be covered in appropriate core courses. Both committees believe this is an important step to discuss and begin development with WSBE faculty, so it may be included in our report to AACSB.

2. Comments and Suggestions on Sustainability  Needs to be defined: “green”/environmental and/or Corporate Social Responsibility or triple bottom line?) This needs to then be reflected in revised rubric.

 Consistent case/rubric used across all sections in which we teach/assess sustainability.

 Rubric needs to be re-written for non-expert evaluator ease of use.

 Assessment needs multiple evaluators.

 Document sustainability in the curriculum (what, when, how).

 If it is in the mission statement, needs to be integrated across curriculum.

 Sustainability needs to be a MBA learning goal.

Committee Discussion/Action:

Undergraduate Curriculum Committee: In the short term, actions will be taken to involve multiple evaluators for the next round of assessments, as well as a rubric revision to simplify use. Over the longer term, discussion will begin in the committee and amongst faculty as to how sustainability should be defined and how it can be integrated across the curriculum.

Graduate Curriculum Committee: Sustainability is not currently a learning goal for the graduate programs. This was brought up among the faculty as an area of concern. In the short term, the committee is going to focus efforts on our existing learning goals. For the long term, and to include in our report to AACSB, the committee plans on having discussions with faculty about sustainability, how the graduate programs will define it, what the learning goal should be, how it can be integrated across the curriculum, where and when we will be able to assess it. In November of 2011, there was faculty approval of “Leading & Managing Sustainable Enterprises” as a common theme across all WSBE MBA programs, however at this point in time this is not well articulated or documented in terms of curricula or learning goals.

3. Comments and Suggestions on Global Awareness

 Measure and measurement tool refinement (group presentations/rubric)

i. When will the assessment be done over for undergraduate? (stratified sampling)

 Rubric should be clearly stated on course documents (i.e. consider embedding in syllabus for ADMN 703 and other courses)

 Refine rubric and align with program content

i. No one teaches history (needs to be removed from rubric) ii. “CAGE” model: Cultural, administrative, geographic, economical

 Single vs. multiple evaluators

 ADMN 703 as a potential measurement site Mike suggested this is a good course and he can get buy-in from the different instructors to conduct the assessment

 Individual-level course assessment could be done with one integrated “essay-style” question

Committee Discussion/Action:

Undergraduate Curriculum Committee: In the short term, a new place to measure Global awareness in the undergraduate programs needed to be identified. We have now identified ADMN 703 written case studies as our measure for this goal; assessment will be done for Spring 2013 assignments. The Global Awareness rubric will be revised, and given to faculty to include on course documents. This is another area in which faculty would like to have multiple evaluators.

Graduate Curriculum Committee: In the short term, a new measurement tool, as well as a larger sample size must be identified and collected. The rubric will be revised to better suit graduate rigor and what is being covered in the courses. For the long term, making the international business course required across the MBA, ExMBA and Part-time MBA is up for discussion, which would come at the expense of another course. I do not recall this to be a major point of discussion. Discussions revolved around the need for an assignment, a target course, if possible, and a useful rubric (CAGE was suggested), but an assignment was not nor was a course.

4. Comments and Suggestions for Oral Communication

 Communicate student strengths and weaknesses identified by assessments to all faculty and suggest ways to improve

 Measure seniors (measure early in program and end of program)

 Consider using students as peer evaluators

 Set student expectations early in courses that have presentations

 Provide rubric to students

 Expand assessment beyond student presentations (e.g., in-class contributions to discussion)

 Involve faculty more directly in each learning goal

 Share effective presentations with students (either student presentations, or have an outside presenter share tips)  Connecting the “closing the loop” items to the results report

 External measurement (non-wsbe) of presentations (in-person or videos)

 On-line student accessible FAQ (include videos/presentation issues/communication issues)

Committee Discussion/Action:

Undergraduate/ Graduate Curriculum Committees: This particular learning goal for the Graduate and Undergraduate programs is an area in which our students do well; short-term efforts will focus on other areas. However, many great ideas came out of the retreat for long term consideration including, external evaluators, creating online and possibly video materials on tips, examples and FAQ’s, pre/post measurements to track improvement and where else this skill can be measured outside of oral presentations. This will continue to be discussed within the committee with the intention of including any future changes in our long term plans section of the AACSB report.

Specifically for the Graduate programs in the short-term, the data we have is only for the full time MBA program. We have since identified courses in which we will be able to assess the part-time MBA and are working on where we can measure the ExMBA. Another issue which must be raised at the Graduate level is how we measure oral communication within the online MBA format. If oral communication remains a learning goal, we must be able to show that all of our students (regardless of program or delivery method) are being evaluated and that these evaluations are being used to improve programs and curricula. For the short-term goal of meeting the AACSB deadline of August 27, 2012, this may be placed in the “parking lot” with expressed plan of action and goals set out for the AACSB reviewers. This will let them know we are serious and working toward a measurable goal.

5. Comments and Suggestions on Teamwork/Leadership

Undergraduate Programs

 Include assessment rubric or expectations for teamwork in syllabus in any class with team projects

 Teach teamwork in class

 Give faculty ½ pager on “team norms” (how to teach students to work in teams, what is expected, how to evaluate team members)

 Create written ground rules early to operationalize  Standardized rating system in classes

MSA

 Tweak the tool :

-Rating scale changed to top 5%, top 10% ect. (find a new rating tool)

-Add questions about the team, currently only questions about individual performance…

-Allocate points or rankings or percents?

 Pre-program survey (with the same questions as the post course survey)

 Changing the course where the assessment takes place

 Do not connect student feedback to student grades so students will be more apt to assess critically

MBA

 Identify all courses that use teams

 Increase MBA target levels to 80%

 Better align objective with measure

 FAQ’s

Committee Discussion/Action:

Undergraduate Curriculum Committee: In the short term, we will work with instructors to see if the assessment rubric can be provided to students in any course containing team projects and, with help from faculty outside of the committee, we will be providing faculty of these courses with information on how to teach students to work in teams.

Graduate Curriculum Committee: The MSA program in particular had trouble with their measurement tool, this has since been adjusted based on all the feedback obtained from the retreat. The revised scoring tool will be used this semester. For the MBA programs for our next round of assessments, the target level will be raised from 75% to 80%.

6. Comments and Suggestions on Quantitative Skills (?!) problem solving?

 MBA (short term) – places to measure:

 Courses:: measure across courses, ask for test questions  Operations Management

 Quantitative methods:

 MIS

 Look at 3 to 5 courses, have instructors provide 3 to 5 questions per course, collect data that way.

 MBA (long term): include other core courses:

 Finance

 Marketing

 Accounting

 Economics

 Organizational Behavior

 Use ETS exam (full-time MBA) to baseline math skills

 Undergraduate:

 Consider using senior or junior level courses for measurement

 Bring finite math into WSBE

 Use ETS exam to check for improvement post-ADMN 410

Committee Discussion/Action:

Undergraduate Curriculum Committee: In the short term, math skills test will be used to measure improvement over the course of the program. We will repeat internal mathematics skills testing of our 1st year economics students in January to try to tease out the effect of fall math courses (including Finite Math). We are also likely to start longer term discussions with the math department about math requirements for our majors.

Graduate Curriculum Committee: It was decided the instrument used to measure quantitative skills within the part-time and EMBA programs is not providing an accurate picture of the students’ quantitative problem solving skills. The committee will be speaking with faculty teaching in the MBA programs to collect homework assignments/test questions/cases/projects that will provide more accurate data. 7. Comments and Suggestions on Written Communication

The issues discussed revolved around the undergraduate programs. At the time of the AOL meeting, graduate assessments had not been completed.

 Short-Term

 Rethink the use of graduate students from the English Department OR communicate to them any uniqueness that the business curriculum has (point 3 in action plan)

 Rethink the blanket “Advised to work with section instructor and writing center” as this seems too loose of a remedy, particularly with 25 different instructors reviewing/assessing the material. Rather, have the writing center provide material that can be used for the section instructors for consistency and as a “Coaching” tool.

 Long-Term

 Work with the Writing Center to place a satellite center in the new building. Given the size of the cohort (+-600) they could tailor their guidance to our students on site.

 Have a conversation with the English Department about what our needs and or expectations are and possibly have ENGL 401 sub-sections for business students only.

 For those students that do not make the grade in ADMN 400, a required no credit “Remedial” writing course, fee based, that they have to complete. This could be done on-line (less resources) or face-to-face.

 For the ADMN 575 course, Behavior in Organizations, it was suggested to assess the goal/objective after the first assignment. Many students appear to be putting in relatively low effort for the first assignment and this tends to change in later assignments.

 Members of this group thought the rubric used was good.

Committee Discussion/Action:

Undergraduate Curriculum Committee: In the short term, it was suggested that all undergraduate faculty be reminded the writing book provided in ADMN 400 should be what students are using throughout the program as a resource when writing papers, as well as having the ADMN 400 instructors remind students this book should be going with them through their entire college career (e.g. used as a resource for all courses). We will also make sure that ADMN 400 instructors and instructors in other writing-intensive courses are familiar with basic writing references available on the Writing Center website. Finally, we note that the most important deficiencies we have identified in student writing so far, and the ones we’re trying to address in ADMN 400 with the outsider-grader program, are fundamental writing errors, such as spelling, punctuation, syntax and grammar. We have specifically chosen basic assignments for which grader knowledge of business writing conventions is relatively unimportant. For the long term one of our priorities will be discussions with faculty and the English department on what processes and procedures can be put in place regarding remediation.

Graduate Curriculum Committee: This learning goal has been measured for the graduate programs in full-time, part-time (Durham and Manchester), and on-line programs for Admin 912 – Organization Behavior. The determination of whether the students have met the targets set out for this goal is in process. A plan has been established for term 4 to reassess (close the loop) using Admin 940 (Tech Operations) in the full-time program and Admin 960 (marketing) in the two part-time programs. Once the assessment of Admin 912 has been completed, the targets for the reassessment will be reviewed and modified accordingly. The reassessment for on-line program is under discussion.