What is the Value of Course-Specific Learning Goals?

By Beth Simon and Jared Taylor The authors examined and faculty opinions regarding the use of detailed learning goals in three courses. reported the use of learning goals to be very positive, aiding them with studying, in , and in determining the important material to learn. Likewise, faculty indicated that using learning goals was a positive experience, especially for communicating course material to students and other faculty and for creating course assessments.

xtensive research has demon- Holliday 2006; Kiewra 1985). We specific, course-level learning goals strated that experts in a subject hypothesized that these problems in improved the student’s interaction have a mental organizational teaching might be addressed by hav- with the course, we looked specifi- structure for knowledge that ing explicit learning goals or objec- cally at the following questions: novicesE lack (Ross 2006; Ericsson tives for lectures. Such goals would 2006). Experts recognize patterns and provide learners (students) with orga- • Did students perceive learning goals classify material on the basis of this nizational scaffolding, and thus help as being valuable in the course? organizational system, but the orga- students understand the perspective • What did students report about nizational system is deeply ingrained being taken by the instructor. how they used learning goals and and rarely consciously recognized. A range of work has discussed how was this different across sev- The subconscious nature of this orga- both the results and impact of learn- eral instructors and courses? nization impacts the ability of instruc- ing goals (as we will refer to them) • Did the instructors perceive the tors to recognize that an explanation for programmatic assessment (Marsh value of learning goals both for they find compelling may be incoher- 2007; Adam 2004; Harden 2002). themselves and their students? ent to students. Many studies have However, in this work, we explored provided examples of instructors and the impact of learning goals on the As discussed in this article, our re- students perceiving an explanation individual student in a course and, sults indicate that explicit learning profoundly differently, with resulting to a lesser degree, on the instructors goals provide a valuable aid to guide negative consequences for student teaching the courses. We studied three students in their learning. These re- learning. Examples have been shown courses in which instructors had de- sults give instructors a glimpse into in -scenario studies (Hrepic, tailed, course-specific learning goals how students use learning goals and Zollman, and Rebello 2007) and in that were integrated into their classes. suggest best practices for the use of studies of note taking (Bonner and To explore the general hypothesis that learning goals.

52 Journal of College Science Teaching The setting for this study was three although the spring computing course 44 were not included in this analysis courses in which the three instructors inherited and adopted some of the because they were clearly describing used learning goals on a daily basis learning goals from the fall course the value of the class, not the value in their classrooms. These courses (39 of 75 spring learning goals were of learning goals. A content analysis– are part of the computer science and repeated from fall). All three instruc- based coding process was developed microbiology and immunology de- tors presented learning goals as part for the responses from Course A. We partments at the of British of their lectures, two always presented organized the codes into a number of Columbia (UBC). The three classes them at the beginning of the lecture categories such as Study, Exams, Lec- were the fall 2007 (Course A) and (Course A and Course C), and one ture, Focus, Guide, General Positive, spring 2008 (Course B) offerings presented them at the beginning of and Negative (see Table 2). We also of a computing literacy course and each unit (Course B). grouped some related code categories an upper-level microbiology course into larger groupings: Knowing What (Course C). Both departments have Study methods I Need to Know and Comprehension. been granted funding through the In the last week of the term, we asked All positive comments fit within one UBC Carl Wieman Science Educa- students (during class) to complete or more of the first six categories. We tion Initiative to (1) establish what up to five copies of the sentence, independently coded the data in an students should learn, (2) scientifi- “For me, the use of learning goals in iterative process by which categories cally measure what students are actu- this course is . . .” Students were giv- were refined. We achieved an aver- ally learning, (3) adapt instructional en examples to help them feel more age intercoder reliability of 95% (29 methods and pedagogical research to comfortable about what they were out of 553 student comments lacked achieve desired learning outcomes, being asked to do. These examples complete coding agreement; Course A and (4) disseminate and adopt what were “helpful because . . .” and “not = 97%, Course B = 93%, Course C = works. Two of the three something I’ve really noticed.” We 95%).The three instructors were not involved in the study had attended performed interviews with the three involved in the coding or the analysis workshops at UBC on creating learn- instructors to characterize the differ- of the comments. ing goals, and all three were indepen- ences in development, intent, and use dently interested in considering the of learning goals for the courses. Results and discussion impacts of learning goals on student We collected 597 responses (225 We analyzed responses across the learning. The instructors all consid- from Course A, 252 from Course B, classes first by the total number of ered the development of the learning 120 from Course C) from 186 stu- comments in each of the coding cat- goals to be an ongoing process, and dents (59 in Course A, 76 in Course B, egories and then by the percentage of they continue to invest time and effort 51 in Course C). Of the 597 responses, students who made a comment that to improve and refine them. The three 13-week courses each Table 1 had between 57 and 75 learning Examples of learning goals. goals that reflected single-lecture or multiple-lecture content. Examples Class Learning goals of learning goals from these three classes are shown in Table 1. In gen- After the web unit/HTML lab you will be able to: eral, each of the goals completed the • Briefly describe the parts of a web search engine. sentence, “At the end of this lecture/ • Explain how a search engine finds and indexes web pages. • Predict how and whether a page will be found by a web topic students will be able to . . .” Course A and B The goals are primarily written in crawler, given the link structure around the page and this form so that students are able to which pages the crawler already knows. identify whether they can accomplish • Construct HTML to present the structure and visual ap- what is described, even if they have pearance you intend for an HTML web page. not yet mastered the terminology • Comment on the strengths and weaknesses of classifi- of the subject. Vague terms such as cation systems used to organize diversity in the world understand are (usually) absent. around us. Instead, specifics on how a student Course C • Describe the relationships between point mutation and would operationally demonstrate a gene transfer in the context of evolution. requisite level of understanding are • Evaluate the concept of microbial species using compara- used. The instructors began to use tive genomics. learning goals of their own accord,

November/December 2009 53 fell into a category. Learning goals FIGURE 1 were seen as very valuable by nearly Percentages of students and valuations. all students in total and consistently in the three classes. Of the 553 total 70% comments, 471 were positive (Course A, 85%; Course B, 84%; Course C, 60% n Course A n Course B n Course C 88%). Figure 1 shows that less than 10% of the students in each class 50% made only negative comments, and 40% nearly two-thirds made only positive comments. The fact that 85% of the 30% comments were positive also indi- cated that those students who made 20% mixed comments gave significantly of students Percentage more positive than negative com- 10% ments. Of the negative comments, the majority fell into the categories 0% of finding the learning goals unhelp- Students with only Students with both Students with only positive valuations positive and negative negative valuations ful or used unclear wording (Figure valuations 2). Students also provided some recommendations to make learning

Table 2 Examples of codes. Groupings Codes Example

Study, Prepare, Review “improves my studying habits” (Course A)

Exam, , Quiz “important for exam review” (Course B)

Lecture, Class, Course “helpful to connect lectures” (Course C)

Focus “helpful because it tells me what I need to focus on” (Course A)

Implied focus “a good thing to highlight important concepts” (Course B)

Summary, Outline “a good outline of the topics” (Course C)

“useful because it guides me through the progression of Knowing What I Guide the class throughout the term” (Course A) Need to Know “helpful for organizing my notes with similar topics” Organize (Course B)

Track “good for keeping me on track” (Course A)

Preview, Introduction, Expectation “gives an idea what the lecture will be about” (Course C)

“well-organized and helpful in understanding the material Understand during lectures” (Course A) Comprehension “nice because they are a list of all the new things I can say Learn I’ve learned in this class” (Course B)

General positive “a good idea” (Course C)

“sometimes confusing when I don’t know what the learn- Negative ing goal(s) is/are referring to” (Course A)

54 Journal of College Science Teaching What is the Value of Course-Specific Learning Goals? goals more effective (by improving wording, for example). FIGURE 2 Breakdown of valuation of learning goals, by percentage of total What is the perceived value of relevant comments. Totals for each course can be greater than 100% learning goals? due to cross-coding of comments. We identified the most commonly list- 50% ed values of goals by examining the distribution of comments across the 45% six categories (Figure 2). There was n Course A n Course B n Course C striking similarity among the three 40% courses. The most frequently men- tioned value for learning goals was 35% that it helped students “know what I need to know” for a course. The next 30% most common valuations referred to lecture or study. Notably, valuation 25% for exams is not terribly prevalent (though reference to “study” implies 20% preparation for exams). The exam Students of Relevant Percent 15% category was the only category that was noticeably different between the 10% courses. We believe this is due to the Course A instructor making an ex- 5% plicit and repeated promise that all

exam questions would come from 0% learning goals, and that the Course C Study instructor had not yet given an exam Exams Lecture

at the time of the survey. Unhelpful Knowing what Knowing What I Need to Know knowI need to Comprehension General positive wording Unclear

The dominant value identified was Recommendations clearly Knowing What I Need to Positive comments Negative comments Know. Students were very specific as to the ways that learning goals helped them: “focus,” “guided me,” learning goals help them recognize of every lecture period, whereas in “kept me on track,” “summarized,” this structure and approach their Course B learning goals were shown and “outlined the course.” Overall, learning more effectively. at the beginning of a unit or topic we found that these comments indi- Many comments were about how area. There is no obvious impact of cated that students valued additional students valued applying Knowing this difference. information or structure, which al- What I Need to Know. Of the com- lowed them to organize the informa- ments in that category, 53% were Instructor impressions tion more effectively and be more also coded in the Lecture, Exam, or We interviewed the instructors to ex- expertlike in their approach to the Study categories. The percentages of amine their views of learning goals, class. The comments indicated that the cross-coded comments for each both for themselves and for their students appreciated that instructors course are shown in Figure 3. This students. By comparing notes and were being explicit about what was shows that students most commonly interview recordings, we identified of the most importance in the class, mentioned learning goals as helping several broad themes, most notably and that learning goals helped them them know what they needed to know communication with students and narrow down and carefully apply in the lecture setting. Learning goals other instructors, structured learning, their efforts to those topics, skills, or are often seen by students as helping and assessment development. concepts. This supports our hypoth- them get more out of a lecture by fo- The most common point made esis that students struggle to see the cusing and guiding their attention. In by the instructors was that learning organization and relative importance Course A and Course C, learning goals goals enhanced communication, of material in courses, and explicit were always shown at the beginning both with students and other fac-

November/December 2009 55 ulty members. The instructors felt that the learning goals provided a FIGURE 3 method to very clearly outline for Comments coded both as Knowing What I Need to Know and either students the important concepts and Lecture, Study, or Exams. material for the class. The learning 45% goals also provided a very easy way 40% for instructors to communicate with n Course A n Course B n Course C other instructors about what is cov- ered in a course, something that is 35% needed when organizing material in a sequence of classes. For example, 30% Course A instructor commented: 25%

“The learning goals are kind of a 20% contract between me and the stu- dents so we can all know what 15%

the foundation is that we should of students Percentage be building a class on and they 10% can call me out and say does this have anything to do with learn- 5% ing goals? . . . It’s a shared struc- ture by which I and students and 0% other instructors can know what Lecture Study Exams the course is about.”

Course B instructor commented: The instructors also commented Course C instructor commented: on what they perceived to be the “When learning goals weren’t value of using learning goals from “A learning goal primes [the a primary focus, it was hard to a student’s point of view: provid- students] to think about the in- establish expectations . . . even ing structure for learning. This was formation and organize it . . . when I really emphasize things closely related to the ideas they ex- We can tell students that these verbally, [the students] miss pressed about how communication learning goals are important these sorts of things. But to was enhanced with students. The in structuring your knowledge have something written down instructors observed that students . . . Students are hungry for that you can point to and say were able to both use the learning anything that structures their that here are my expectations goals to determine the important learning.” and here is what we are intend- material in the course and use the ing to give you as a skill set learning goals as a measuring stick Finally, the instructors mentioned when you walk out of the class to gauge their own progress. This that the learning goals streamlined . . . it gives instructors greater seems very similar to the students’ the process of writing exam ques- confidence in interacting with comments coded as “on track.” tions and improved assessment. students and other faculty Course A instructor commented: members.” Course A instructor commented: “[Learning goals] will save you Course C instructor commented: “It’s a question of organiza- time in the end, because instruc- tion as in knowing what I am tors will want to put together a “I would emphasize the need to doing and the students know- high-quality exam, and you share our learning goals with ing what I am doing as well . . . will put together a high-quality one another [other faculty] and They had a goal for what to exam much more easily with to communicate what is being learn. They didn’t just learn the learning goals. Everyone I taught. Learning goals are a as much as they could in this have talked to that uses learn- parsed way to look at what a course, but they had some- ing goals has talked about this. course is about and what the thing to work towards and Your exam writes itself . . . I can core concepts are.” measure themselves against.” check to see if each question ad-

56 Journal of College Science Teaching What is the Value of Course-Specific Learning Goals?

dresses a learning goal and if it orientation of their goals, which were Herriot-Watt University. Edinburgh, doesn’t I will throw it out.” similar across the three instructors. Scotland. Students of a discipline may lack Bonner, J.M., and W.G. Holliday. 2006. Course B instructor commented: the intellectual scaffolding and orga- How college science students engage nizational structure taken for granted in note-taking strategies. Journal of “The learning goals were not by experts instructing them. The Research in Science Teaching 43 (8): the primary source for exam primary value students expressed 786–818. questions, but one notch down. regarding learning goals in this study Ericsson, K.A. 2006. The influence of They would drive the initial is that they provided them with struc- experience and deliberate practice on structure of the exam, and then ture in their efforts to know what it the development of superior expert I would fill in the remainder.” is they need to know. The fact that performance. In The Cambridge many students indicated that learn- Handbook of Expertise and Expert Summary ing goals helped them scaffold their Performance, eds. K.A. Ericsson, N. lecture experience provides hope that Charness, P.J. Feltovich, and R.R. Returning to our original research learning goals reduce some of the Hoffman, 683–703. Cambridge, UK: questions, we found that students well-known difficulties students have Cambridge University Press. overwhelmingly found value in in understanding lecture and in taking Harden R.M. 2002. Developments in the use of detailed, course-specific notes and therefore support more ef- outcome-based . Medical learning goals. Specifically, students fective learning. 24 (2): 117–120. found that the learning goals helped We urge others to consider devel- Hrepic, Z., D.A. Zollman, and N.S. them determine what they needed to oping learning goals at the course Rebello. 2007. Comparing students’ know. Student responses were con- level that complete the sentence, “By and experts’ understanding of the sistent across all three courses in two the end of today, students will be able content of a lecture. Journal of Sci- very different academic disciplines. to . . .” and to report both on the con- ence Education and Technology 16 Upon repeated reading and coding tent of those goals and their valuation (3): 213–224. of the comments, a general sentiment by students. We are encouraged by Kiewra, K.A. 1985. Providing the of the students emerged. Students the positive, shared valuation by stu- instructor’s notes: An effective addi- expressed relief and gratitude at be- dents across one computing and one tion to student notetaking. Educa- ing given clear direction as to how to microbiology class and are interested tional Psychologist 20 (1): 33–39. focus their efforts, most notably in the to see if others (and their students) Marsh, P. 2007. What is known about stu- lectures, and also in organizing their find similar value. Exemplar learning dent learning outcomes and how does it studying, reviewing, and preparing goals of this style can be found at the relate to the scholarship of teaching and for exams. Interviews indicated that University of British Columbia’s Carl learning? International Journal for the instructors found learning goals help- Wieman Science Education Initiative Scholarship of Teaching and Learning ful in keeping their lecture prepara- website (http://cwsei.ubc.ca). n 1 (2). http://academics.georgiasouth- tions on track. This corresponded to ern.edu/ijsotl. student responses that indicated that Acknowledgments Ross, P.E. 2006. The expert mind. Sci- daily, explicit presentation of learning We thank Steve Wolfman, Kim Voll, and entific American 295 (2): 64–71. goals made the instructor’s organiza- Steven Hallam for their cooperation and tion and focus more clear. Students participation in this work, as well as Carl Beth Simon ([email protected]) is found learning goals to be a clear, Wieman for his invaluable input and feed- a faculty member in the Computer Sci- valuable form of communication from back. This work was funded by the Carl ence and Engineering Department at the the instructor that was missing in their Wieman Science Education Initiative at University of California, San Diego, and a other courses. As one student stated, the University of British Columbia. former Science Teaching and Learning Fel- learning goals are “useful in that they low in the Carl Wieman Science Education show me what the prof wants us to References Initiative and the Department of Computer learn” (Course A). Adam, S. 2004. Using learning Science at the University of British Colum- Although the instructors were not outcomes: A consideration of the bia in Vancouver, British Columbia. Jared explicitly directed in their develop- nature, role, application and impli- Taylor ([email protected]) is a Sci- ment of goals for this study, we found cations for European education of ence Teaching and Learning Fellow in the that their respective student popula- employing learning outcomes at the Carl Wieman Science Education Initiative tions valued learning goals similarly. local, national and international and the Department of Microbiology and This suggests that what is important levels. Report on the United King- Immunology at the University of British Co- is the detailed nature and student- dom Bologna Seminar, July 2004, at lumbia in Vancouver, British Columbia.

November/December 2009 57