Boosting Complex Learning by Strategic Assessment and Course Design

Manuscript number # 2002-0851

Boosting Complex Learning by Strategic Assessment and Course Design. Christina Bergendahl, Department of Chemistry, Umeå University, 90187 Umeå, Sweden. Lena Tibell, Department of Biomedicine and Surgery, Linköping University, 581 85 Linköping, Sweden.

Abstract: Learning quality depends on, among others, the assessment methods. Therefore, alignment of the objectives with appropriate assessment and instructions become important. Our focus is to study this relation, and whether the choice of assessment methods can promote students’ acquisition of higher order cognitive skills. The subject of the study is a practical biochemistry course, at the university level. We will present the results from a course design containing five different assessment methods; written examination, laboratory work, seminar, grant proposal and poster. Multiple types of data are collected; observations, interviews, questionnaires, written exam questions/answers, objectives, students’ written reports/products and judgements made by the teachers. As evaluation tools we have used quantitative as well as qualitative methods; analysis of inquiries and interviews, a simplified variant of the Bloom taxonomy, and statistical analysis with Spearman ranking and Principle Component Analysis. We conclude that a strategic choice of assessments and instructional design can be used as a driving force to more complex learning. Cognitive, affective and psychomotor domains appear to be important for internalisation, and the change in perception of learning seems to be related to the students’ control over their learning process. Difficulties in advancing to the synthesis category were observed. Correlations between the 48 students’ grades on the five different assessment methods are presented.

Keywords: Biochemistry, Chemical education research (CER), CER Student-Centered Learning, CER Qualitative methods, CER Quantitative methods, Testing/Assessment.

1 Manuscript number # 2002-0851

Introduction The choice of teaching methods exerts a major influence on the development of different types of knowledge and skills, and the relationships between assessment, teaching and learning have been explored extensively in the literature (1, 2, 3, 4, 5, 6, 7, 8). For instance, Snyder (1) showed that students constructed their own understanding of the curriculum from messages, explicit and implicit, about what counts in assessment (the “hidden curriculum”), and Brown (2) argues that if teachers “get right” the processes and practice of assessment, then learning of the appropriate type will ensue. Therefore, in addition to its roles in evaluating the learning outcome and in the quality assurance process, assessment can provide a powerful tool to improve the effectiveness of learning (9,10). The aim of education in general, and higher education in particular, is to produce independent, responsible and capable people. Other educational goals that are often highly ranked include strengthening “the desire to learn”, fostering “a scientific attitude” together with the capacity to “organize, prioritize, plan activities”, and “cooperate”. Interesting issues, therefore, are the extent to which these types of educational goals are being fulfilled, and how they are being assessed. Traditional types of assessment often focus on knowledge that can be easily tested, but they sometimes fail to measure the difference between what is learned and what should be learned. To paraphrase Bodner et al. (11), a serious problem with traditional assessment might be understood in terms of the metaphor of a drunk searching for a coin beneath a lamp post – not because this is where the coin was dropped, but because it is where the light is. When new examination techniques are introduced research is rarely conducted to investigate whether or not they lead to valuable improvements in students´ understanding of knowledge. Our main focus in this study is to investigate whether a strategic course and assessment design promote the students’ capacity for higher order cognitive thinking. We also believe that it is essential to find out more about the students’ views of their own learning process.

Research / Experimental design The sample The subject of our study is a 10-week biochemistry course entitled “Strategic protein separation and analysis techniques”, which has been taught 15 times since 1993 at Umeå University, Sweden. The test population in this study consisted of 48 students (34 female and 14 male) with educational backgrounds in chemistry, molecular biology or biotechnological engineering. The overall course aim is to provide students with the knowledge and abilities needed for solving problems involving protein purification and analysis. Approximately one third of the course, mainly concentrated in the first three weeks, is geared towards fostering understanding of the principles of protein separation and methods of protein analysis, stressing their application, requirements and limitations. This knowledge is intended to deepen during the practical work, which constitutes the main part of the course. In the practical projects the key objectives of the students (working in pairs) are to develop a purification method for their specific protein, to determine its purity, and to characterize it. The tasks are often parts of ongoing research projects and mimic authentic situations that the students are likely to encounter in their future careers. The students are trained in skills such as decision-making, critical thinking, problem solving and ability to communicate, especially in relation to practical laboratory work.

Arguments for change and the educational intervention In the original course design the assessment method for the entire course was a written examination taken at the end of the course in which the questions mainly focused on the theory underlying methods of protein purification and analysis. This had two adverse consequences. First, the theoretical knowledge was generally not acquired until the end of the course, although the students most urgently needed it during the project work. Second, even though the course centered on the application of techniques used in protein separation and analysis, the sole basis for grading the students was their results in the written exam, and their practical work was never assessed. Thus, the major learning goals and course objectives were not assessed, and the opportunity for developing higher- order cognitive skills was not maximized. The resulting dissatisfaction prompted a program of continuous course development resulting, after several trials, in a revised design for the course. The general goals in the new course design were to align the assessment strategies with the aims of the course to promote higher-order cognitive thinking. The instructions and working methods were broadly the same in the old and new course designs, but more formative types of assessment, including continuous, diversified kinds, were

2 Manuscript number # 2002-0851 introduced into the new format. In addition, the course objectives and criteria for grading were carefully formulated and communicated to the students. Feedback was given to the students after each assessment.

Strategic assessment A strategic choice of assessment methods means, in this case, choosing appropriate assessment methods and aligning them to ensure a progression of higher order thinking. The goals have to be reflected in the assessment methods. For instance, the degree to which the students have fulfilled the goals “to organize and plan an experiment” should involve assessing the corresponding part of the laboratory work, and “to communicate results” should involve assessment of corresponding skills as displayed in for example written and oral presentations. Our idea was that the discrepancy between assessment and objectives in the (old) course could be overcome by using formative and authentic assessment. Authentic assessment should reflect what practicing professionals do (12), and in our case relating the assessment methods to professional research activities was also a strategic choice. In our study this strategy also gave us several opportunities to collect and analyze data on the learning outcomes during the course. Our choice of assessment methods included two types of written examinations, but in addition the students’ laboratory work, their performance in a seminar, and their preparation of a grant proposal and a poster1 were also assessed. In the written examinations the students’ theoretical understanding was assessed by continuous examination during the first weeks of the course, and by a take-home examination at the end of the theoretical part of the course. In the continuous written examination, the questions were focused on facts and simple relationships, while the take-home examination included more extensive theoretical tasks1. Since the practical laboratory work is considered a crucial part of the course, we believe it is very important to assess it. However, the analysis is extensive, so the criteria and methodology involved will be discussed in a future paper. After three weeks of experimental work, the students presented their preliminary results in a seminar, assessed by two teachers. The criteria for grading the seminar included: knowledge of the subject, usage of relevant terminology, ability to select and explain relevant material, and capacity to evaluate the results and conclusions. The ability to discuss, respond to questions and to engage others in the discussion were also assessed. About two weeks before the end of the course the students each wrote a grant proposal. This had to be written according to formal, written requirements, and had to include a summary of previously published information, a description of their own results, conclusions based on knowledge from both of these sources, and a proposal for a project to test a hypothesis based on the integrated knowledge1. The students were also expected to propose ways to test their new hypothesis, to present a time plan, and a budget. An outline of their project, the final results and conclusions were also presented in a poster session at the end of the course. Each poster was assigned an external examiner who assessed the poster, together with the oral presentation and defense of it. The external examiner, one for each poster, was either a professor from the chemistry department or a researcher from the biotechnology industry.

Collection and analysis of data As teachers and researchers involved in chemistry education, we have been able to gather multiple types of evidence to address our research question. Written notes related to the observations, interviews, questionnaires, written exam questions/students’ answers, formulated objectives for the different types of assessment and the students´ written reports/other products, together with the grades and judgments awarded by the teachers, provided the evidence base for our analysis.1 One of the earliest and most widely used systems for categorizing learning outcomes and communicating results of educational evaluations is the Bloom taxonomy (13, 14). In this scheme, three major learning domains are defined: the cognitive (intellectual abilities and skills), the affective, and the psychomotor. In our case, much of the laboratory work, and some aspects of the seminar and poster presentations, falls into the affective and psychomotoric domains, whereas most traditional educational objectives fall into the cognitive domain (13). “Bloom taxonomy” includes a hierarchical ordering of the cognitive domain into six different levels, starting with knowledge per se as the “lowest” cognitive level followed by comprehension, application, analysis, synthesis and evaluation. A simplified variant of Bloom’s Taxonomy, without subcategories, was used for the analysis and categorization of objectives, written exam questions and assessment products. We sometimes refer to the categories knowledge, comprehension and application as 3 Manuscript number # 2002-0851

“lower Bloom” and analyses, synthesis, evaluation as “higher Bloom” categories. When classifying tasks according to their learning objectives we focused on the most important objectives, rather than identifying all possible objectives. Two to three persons in the teaching staff classified the questions, objectives, and products according to Bloom Taxonomy. Examples of the Bloom classifications of two written exam questions and the objectives for the grant proposal are given in supplemental material 1. To investigate possible connections or correlations between the different types of assessment the students’ grades1 were analyzed by two different methods. The Spearman rank correlation coefficients (15) for the students’ grades obtained in the different types of assessment were calculated. To further analyze the grades obtained in each assessment, a multivariate data analysis method, Principal Component Analysis (PCA), was applied (16,17,18). To investigate qualitative changes in students’ perceptions of learning during their education, Johnstone (19) developed a questionnaire, based on the work of Perry (20). In this questionnaire the student’s perceptions of five key educational parameters were probed: knowledge, the role of the teacher, their own role, assessment, and the value of experiments. We used an adapted form of Johnstone’s questionnaire (21) 1, which the students answered twice, at the introductory event and prior to the interview at the end of the course. Students’ mean values (21), before and after the course, were calculated from their responses to the statements, and paired T- Tests were used to analyze the data1. After three weeks of the course, the students responded to a questionnaire containing open-ended questions regarding their experience of the written exams. At the end of the course the students were interviewed individually, for about 30 minutes each, in a semi-structured interview (22). The students were asked, “What did you learn during this course?” then questioned about what they learned from each type of assessment. The interviews were tape-recorded and the audiotapes were transcribed for analysis. Interview data, as well as questionnaire data, were analyzed using the technique of analytical induction, based on Abell and Smith (23). This process involved repeated readings of the respondents’ comments

Results and Discussion The analysis of our results aims to identify patterns in the outcomes of the assessments, factors promoting higher order cognitive skills, and objectives that the students find difficult to achieve. Analysis of objectives and assessment tasks Questions from six individual exams from the original course design were analyzed, with respect to Bloom classification, and the results are presented in Figure 1. Approximately 25% of the questions required higher cognitive skills (analysis and synthesis), and none of the questions were of the categories knowledge or evaluation. Questions from the two different types of written exam in the new course design were analyzed in a similar fashion, and the mean values derived are presented in Figure 1. Approximately 50% of the questions in the continuous examinations, and 70% in the take-home exam were of higher Bloom categories.

Figure 1 Frequency of questions classified in different Bloom categories, expressed as percentages of the total number of questions. The questions analyzed for the old variant of the course came from six written exams, while those for the new design came from written continuous examinations administered in four of the courses and take-home exams from two of the courses.

60 Written exam (old) 50 Continuous exam (new) 40 Take-home exam (new)

% 30 20 10 0 Comp. Appl. Anal. Synt.

The objectives of the other assessment methods used in the “new” course were also analyzed (Figure 2). The proportion of objectives at higher Bloom-levels ranged between 50 to 70%, with the grant proposal and take- home exam scoring highest.

Figure 2 Proportion of lower Bloom level (knowledge, comprehension, application) and higher Bloom level (analysis, synthesis, evaluation) questions/objectives in the different assessment methods.

100 80 60 High Bloom % 40 Low Bloom 20

0 ) l r ) r ) d a a e l w w t n s o e i e s ( o n n o p m ( (

m e o P r a m m S p x a a t e x x

n e e n a

r e e s t t u G i m r o o u h W n - i e t k n o a T C

In summary, the assessments chosen for the new model are clearly set at higher Bloom levels than the written exam questions in the old variant of the course. However, this prompts the question: Do the students really achieve these higher goals? In an attempt to answer this question we have used both qualitative and quantitative methods to analyze our data.

Qualitative analysis of the assessment methods When analyzing the assessment methods in the new course design the students, not surprisingly, appeared to find it more difficult to do well at the higher than at the lower Bloom-levels. Analysis of our material shows that it is considerably harder to attain analysis, synthesis and evaluation levels at an early stage in the course. The outcomes from the take-home exam show that the students had problems in combining facts to present a coherent, integrated account (synthesis). Many of their answers emanated from their lecture notes. The results from the questionnaires and interviews support this conclusion, clearly indicating that students did not know at this stage how to tackle the questions at higher Bloom-levels. The seminar, given halfway through the students’ practical period, aimed to provide an opportunity to explain the rationale underlying their experiments, to organize and analyze their material and results, and to discuss their research objectives and results with their peers. The feedback from the other students and the teacher was intended to stimulate higher order cognitive thinking. However, even though most of the students succeeded in summarizing the theoretical background of their work, and organizing their results, only about 50% of the students appeared to have reached a deeper understanding of their project (according to their teachers’ judgments), and the intended discussion rarely developed during the seminars. “The disadvantage with the seminar is that you only learn and understand what you are talking about yourself. Before your own seminar you are so nervous that you only think about what you are going to say and afterward you are exhausted: in both cases it is impossible to listen carefully to the other seminars. You therefore do not learn so much about the other projects”. On the other hand, the preparation for the seminar did offer an opportunity for the students to organize and analyze their material. Since this in itself is a desirable activity we suggest that the seminar should be replaced by a short report, which could be discussed with peers and the supervisor and thus provide a formative learning assessment. As an assessment tool, the grant proposal, in particular, provides an opportunity to evaluate such high-level cognitive skills as the ability to select relevant facts and organize arguments, fluency, and creativity. Many students testified, in the interviews, that they experienced difficulties in writing the grant proposal. Nevertheless, 5 Manuscript number # 2002-0851 many also reported that once they had started it became more fun, they felt more motivated, and it seemed an important, useful exercise for the future. “Fantastic, because many will have to do this kind of task in the future. It's good to have a chance to test things out before doing them for real. The discussion afterwards was good - it's not often that one has the chance to discuss things with a supervisor and investigate and clarify different issues. Lacking information on how to structure the first page, I was forced to call my father to ask him for help.” Making a poster is the final activity of the course. The students have worked on their projects for nine weeks at this point and, in a sense, they have become experts in their specific area. Nevertheless, they found tasks requiring analysis and synthesis levels of cognitive skill difficult to accomplish. Only about 30% of the students succeeded in making an optimal selection of their material, connecting their own results to appropriate results from the literature, and basing their conclusions on the combined material in the poster. However, the majority of the students found it very stimulating, and some of them became almost completely absorbed in their task when creating a poster. In addition, the “external evaluators” generally rated the students’ defense of their posters highly. The effectiveness of this assessment is confirmed in the interviews. ”Great fun. It was good to summarize everything that had been covered during the ten weeks. It forces you to think through what to highlight in order to communicate what you have done. Reflection and consideration are the most important components of what we have done. It was both useful and good fun to make use of alternative approaches.” In agreement with Ramsden (24) we concluded that creative activities stimulate the affective domain in the process of internalization. Tasks like this, requiring the use of a combination of cognitive and affective skills, are in our experience very effective for deepening students’ understanding of theory, and provide excellent opportunities to develop higher order cognitive skills. Our conclusion is, therefore, that all three domains appear to be important components for internalization and have to be considered when designing a course to foster higher order cognitive skills. Following Mills et al. (25), we argue that the poster exam provides a valuable and visible alternative means of assessment to the traditional, written exam.

The problematic synthesis category Difficulties in advancing to the synthesis category were observed for all assessment methods (see above). However, in the discussions with the evaluators and visitors during the poster session, advancement to the synthesis level was often stimulated. This interaction appeared to help the students in their learning process, or to display their learning more effectively. The difficulty of developing good synthesizing ability was most clearly displayed in the students’ attempts to write the grant proposal. Here, the main task was to relate the proposed idea or hypothesis to relevant theory, background literature, and their own experimental results. The students were often able to make some of the connections, and perform a few synthesis processes, but failed to exploit the full potential. This inductive process seems to be a very complex cognitive process requiring experience, imagination and creativity. Interpretation of our data suggests that synthesizing is harder than evaluating, or constitutes a threshold in the student’s cognitive development. This observation is in accordance with the recently published revision of Bloom’s taxonomy (26) in which the authors changed the name of the category “synthesize” to “create” and interchanged the order of synthesize/create and evaluation/evaluate. Synthesis probably requires a rather complex, creative ability. Identification of thresholds in the cognitive process, and methods to enhance these types of skills need to be developed. Quantitative analysis and correlations between the five different assessment methods To identify possible correlations between the different assessments, Spearman rank correlation coefficients for the student grades obtained in the different types of assessment were calculated (Table 1). Interestingly, there appears to be no (0) correlation between the grades for the grant proposal and the seminar and almost no correlation (0.1) between the grant proposal and poster grades. The correlations between the other assessments are generally low. However, the grades for experimental (laboratory) work appear to be most highly correlated to grades from all the other assessments, and also the grades from the theory exams are significantly correlated. The low correlation values (0.3-0.5) could be partly attributable to the low resolution of the grading scale, since a random change of a single unit in one of the grades in a pair of assessment methods will give a large apparent increase in the variation and, hence, reduce the apparent correlation.

Table 1. Spearman rank correlations for the different assessment methods Seminar Lab.work Grant proposal Poster Theory 0.309* 0.523* 0.497* 0.338* Seminar 0.379* 0.000 0.337* Lab.work 0.405* 0.419* Grant proposal 0.100 *Correlations significant at the 0.05 probability level

To interpret and evaluate the results more fully, Principal Component Analysis (PCA) was applied (Figure 3 A and B).

Figure 3 A and B PCA analysis of the grades of the 48 students on the five assessment methods: theory, seminar, lab work, grant proposal, and poster. A) Score plot. B) Loading plot.

2 26 0. 60 Sem. 1518 33 2 10 1 1 28 3 37 17 14 0. 40 Poster 25 12 20 34 45 4 35 22 7 41 48 t[2] 11 23 0 8 13 36 44 942 27 0. 20 39 5 6 16 3847 19 -1 30 24 p[2] 0. 00 Lab. 40 work 46 32 31 29 -2 43 -0 . 20 Written 21 exam. -4 -3 -2 -1 0 1 2 3 4 -0 . 40 t[1] Grant -0 . 60 proposal

0. 0 0 0. 10 0. 2 0 0. 30 0. 4 0 0. 50 p[1]

A PCA model describes the maximum variance in the collected data. If there is a difference between students´ performances in the different types of assessment it should be detected. The score plot (Figure 3A) describes a two-dimensional projection of the students’ grades awarded for all five assessments. The score plot is a representation of the students’ grades where “good students”, with high grades for all five assessments, are located to the right and students with lower grades to the left. No clear clusters are seen. The loading plot (Figure 3B) describes the relationship between the students’ grades for the five assessments. To identify variables that are important for the separation, the loading p-vectors are plotted. A high loading value (the first dimension = p(1)) indicates a high contribution to the distribution of grades seen in the score plot, and the second dimension (p(2)) describes the residual variation. Since all assessments are assembled at high loading values (in the first dimension) they all appear to be important for the distribution of the grades i.e. all of them are good for measuring capacity. In the second dimension, the seminar and poster grades are grouped together (positively correlated), but they are far apart from the grant proposal grades. Thus, the seminar and the grant proposal seem to be the most differentiating types of assessment, measuring quite different abilities and skills. A similar conclusion was drawn from the Spearman rank correlation analysis (see above). It is important to consider why the seminar and poster grades are grouped together, and far apart from the grant proposal grades. Note that the seminar is an oral, the poster a combination of written and oral, and the grant proposal a written examination. Only a few investigations of this type have been published as yet. However, Ben-Zvi et al. (27) show that performance in practical exercises is only weakly correlated with performance in paper-and-pencil tests and Tamir (28) showed that even when the same skills are apparently being assessed (e.g. planning experiments), achievement in laboratory examinations differs from achievements in paper-and-pencil tests.

Individual variations in performance 7 Manuscript number # 2002-0851

In Figure 4 the grades obtained by five students, selected to demonstrate individual variations in outcome, are summarized.

Figure 4 Grades that five different students (10, 13, 15, 16, and 21) achieved in the different modes of assessment. The maximum grade is 5.

6 5 Written exam 4 Sem inar e d

a 3 Lab.work r

G 2 Grant proposal 1 Poster 0 10 13 15 16 21 Student

There are several possible reasons for the variability in grades for specific students, for instance: (a) different methods may emphasize different capabilities and skills; (b) the students may concentrate their efforts on a random selection of assessments; (c) motivation may vary for the different assessments; (d) the teachers’ judgements may vary between assessments; (e) performance may vary due to personal factors. Explanation (d) is not likely since more than one teacher grades each assessment. Of course, the different types of assessment emphasize different skills, but in addition to assessing the different course objectives they also give each student a fair chance to demonstrate their personal abilities.

Change in perception of learning related to the students’ control over their own learning process The students were obliged to be active since we assessed them several times with methods that required higher order cognitive thinking. However, this approach could also, conceivably, produce the opposite effect. Although the course design was communicated to the students at the start of the course, the high demands and sometimes almost continuous assessment might push the students too hard, and thus reduce their capacity and the learning outcome. Thus, it was important to gauge their perceptions of the assessment process. When summarizing and analyzing the interview responses, in which the students described their learning outcomes, almost all of them initially testified to a gain in self-confidence. Secondly, they said they had learned to purify proteins and analyze their properties by different biochemical methods. They usually mentioned features in the affective domain, and after a while gradually progressed to issues related more to the cognitive domain. Despite our guidance and control, the students felt they had become more independent during this course. The experimental period was regarded as important, effectively promoting learning in an active and creative way. Most of them also emphasized that during the course they learned how to work, plan and think more independently. “I have learned to work independently and to plan, to read instructions and assimilate information. I have realized that in the real world things don’t come prepared and ready-made.” ”You feel that you’ve learned a lot of the sort of things requested in job advertisements.” When asked, in the interviews, what they learned from the different assessments, many of the students referred to the feedback and its importance for their learning process, and some students referred to metacognitive phenomena when asked if they had changed in any way during the course. “I have gained a new perspective on how to learn things. For the last three years the routine has been the same - theory, lab work and then an exam. Now we are learning in new ways, and must tackle tasks ourselves." “Lots of assignments to be handed in. I have previously looked down on that sort of exam and mocked my neighbors, saying that such things are simple and facile. But they are not.”

When analyzing the responses to the “Perry questionnaires” (about the students’ perception of learning) before and after the course, a qualitative change during the course was noted, from a largely dualistic view towards a more contextual relativism (Figure 5). In answers to questions concerning experimental work the change is statistically significant 1. About 2/3 of the students shifted towards a more contextual, relativistic view concerning laboratory work.

Figure 5 Change in attitude values during the course. The figure shows the number of students (frequency), with calculated differences in responses to the attitude inquiries conducted before and after the course. Difference in calculated mean value (for each student) is shown.

7 General statements 6 5 Statements about y c

n 4 laboratory work e u

q 3 e r

F 2 1 0 5 1 5 7 3 1 3 9 , , , , , , , , 0 0 0 0 0 0 0 0 - - - Change in attitude value during the course

This change towards a more contextual, relativistic perspective of laboratory work fits nicely with the responses in the interviews, where the students testified that they grew in self-confidence and became more independent during the course, thanks to the experimental work.

Conclusions We conclude from our study that a strategic choice of assessments and instruction set at higher Bloom categories can be used as a driving force to foster more complex learning. The students’ outcome shows that they are capable of achieving these objectives, however the synthesis category constitutes a threshold in the student’s cognitive development. The importance of all three domains for internalization is supported in our results, and the students’ perception of their own learning appears to be related to their control over their learning process. The different types of assessment evaluate different capabilities and skills, approaching the theoretical content from different perspectives. We did not find a single assessment method to be clearly best, but by carefully choosing methods that enhance higher order thinking (i.e. making a strategic choice), we found a combination of assessments that met our objectives.

Implications for teachers To develop and implement a new course design of the type described above involves a high initial workload for the teachers concerned, but once the initial investment has been made it will be labor-saving for future courses. The main goal of our research project was to develop an understanding of the relationships between teaching, assessment and objectives. Our results suggest the following for a strategic course design: . Identify the goals of the course and consider them carefully. . Make the goals explicit to both the teachers and students. . Design the course so that the objectives, instructions and assessment methods are aligned. . Include teaching and assessment methods that introduce a range of knowledge, abilities and skills in a logical, coherent sequence. . Optimize the different assessment methods, and set their time and content in a way that is helpful to the students’ further development. . Progressively assess the students’ knowledge and skills, with constructive feedback, as the students’ advance through the course, to optimize their learning.

Acknowledgements

The authors wish to thank the students on the course “Strategic protein purification and characterization”, years 1998-2000, and the staff teaching these courses at the Department of Biochemistry, Umeå University, for their kind and enthusiastic participation and the latter also for their help with cross-checking categorizations of questions, objectives and grading. We also would like to thank Dr Peter Anton for advice and assistance with the statistical treatment of the data, Professor Uri Zoller for discussions of an early draft of this paper, Professor Henry Heikkinen for extensive comments on the manuscript, Professor Lisbeth Lundahl and Professor Hans- Jürgen Schmidt for valuable discussions, and Professor Nalle (B-H) Jonsson for critical comments. This work was supported by grants from the Swedish National Agency for Higher Education. Supplemental material 1. Tables displaying a “summary of the teaching and assessment methods/data collected during the course”, “examples of Bloom classification of exam questions and objectives for the grant proposal”, “examples of responses to the questionnaire about students’ perception of learning”, “all the 48 students’ grades obtained in the different assessments and mean values”, and “standard deviation for the students’ responses to the attitude statements” are included in this issue of JCE Online. References 1. Snyder, B. The hidden curriculum; MIT Press:Cambridge, MA, 1971. 2. Brown, S. In Assessment matters in higher education; Brown, S ; Glasner, A., Eds.; St Edmunsbury Press Ltd: Suffolk, 1999; pp 3-13 3. Ramsden, P. In The Experience of Learning; Marton, F.; Hounsell, D.; Entwistle, N.J., Eds.; Scottish Academic Press:Edinburgh, 1984; pp 144-164. 4. Watkins, D.; Hattie, J. A. Human Learning 1985, 4, 127-41. 5. Boud, D. In P. Knight (Ed.), Assessment for learning in higher education. Birmingham:Kogan page, 1995 6. Harlen, W.; James, M. Creating a positive impact of assessment on learning. Paper presented at the American educational research association annual conference, (1996) 7. Bowden, J. ; Marton, F. The university of learning beyond quality and competence. London: Kogan page, 1998 8. Resnick, L.B. ; Resnick, D.P. In B.R.Gifford and M.C. O’Connor (Eds.), Changing assessments: Alternative views of aptitude, achievement and instruction. Boston:Kluwer, 1992; pp37-75 9. Brown, S.; Knight, P. Assessing Learners in Higher Education; Kogan Page: London, 1994. 10. Brown, G.; Bull, J.; Pendelbury, M. Assessing Student Learning in Higher Education; Routledge:New York, 1997. 11. Bodner, G.; MacIsaac, D.; White, S. University Chemistry Education 1999, 3(1), 31-34. 12. Emery, D. In Assessment in science. A guide to professional development and classroom practice; Shepardsson, D.P., Ed.; Kluwer:Dordrecht, 2001; pp 227-247. 13. Bloom, B.S. (Ed.); Engelhart, M.D.; Furst, E.J.; Walker, H.H.; Krathwohl, D.R. Taxonomy of educational objectives. Handbook 1 Cognitive domains; David McKay Company, Inc.:New York, 1956. 14. Krathwohl, D.R.; Bloom, B.S.; Masia, B.B. Taxonomy of educational objectives, the Classification of Educational Goals. Handbook II: The affective domain; David McKay:New York, 1964. 15. Zar, J. H. Biostatistical Analysis; Prentice-Hall:New Jersey, 1996; pp 389-392. 16. Jackson, J.E. A Users Guide to Principle Components; Wiley, New York 1991. 17. Joliffe, I.T., Principle component analysis; Springer, New York, 1986. 18. Eriksson, L.; Johansson, E.; Kettaneh-Wold, N.; Wold, S.; Multi- and Megavariate Data analysis. Principles and Applications. Umetrics, Umeå, 2001. 19. Johnstone, A.H. http://science.ntu.ac.uk/chph/improve.html (improve projects, (iv)case studies, Evaluation of innovation) (accessed Aug 2001). 20. Perry, G.W.Jr. Forms of intellectual and ethical development in the college year; Holt:New York, 1970. 21. Berg, A.; Bergendahl, C.; Lundberg, B.; Tibell, L.E.A. International Journal of Science Education 2003, 25, 351-372. 22. Kvale, S. Interviews: an introduction to qualitative research interviewing; Sage:Thousand Oaks, 1996. 23. Abell, S.; Smith, D. International Journal of Science Education 1994, 16, 473-487. 24. Ramsden, P. Learning to teach in higher education; Routledge:London, 1992.

25. Mills, P.; Sweeney, W.; DeMeo, S.; Marino, R.; Clarkson, S. Journal of Chemical Education 2000, 77(9), 1158-1161. 26. Anderson, L.; Krathwohl, D. A Taxonomy for learning, teaching and assessing. A Revision of Bloom’s Taxonomy of Educational Objectives; Longman Inc.:New York, 2001. 27. Ben-Zvi, R.; Hofstein, A.; Samuel, D.; Kempa, R.F. Journal of Research in Science Teaching 1977, 14, 433-439. 28. Tamir, P. In The student laboratory and the science curriculum; Hegarty-Hazel, E., Ed.; Rutledge:London, 1990; pp 242-266.