Statement 1: Compared to Written Laboratory Reports, I Enjoy Doing Chemical Post-Labs

The ChemiCAL Post-lab System* Evaluation of level 1 PHC student attitudes

March 2007

Foreword

This evaluation took place between 5th and 19th March 2007 using an on-line anonymous questionnaire embedded into the ChemiCAL post-lab menu. The questionnaire was set to activate for students registered on module PHCMC1022 (Pharmaceutical Chemistry) immediately on entry to their last post-lab session of the academic year. This point was chosen particularly to maximise engagement with the questionnaire after the completion of five (MPharm students) or seven (ACAPS students) post-lab sessions. Whilst the completion of the questionnaire was voluntary, some pressure was placed on the students to engage; i.e. to avoid the questionnaire, the students had to email a request to deactivate it within their account. No requests were submitted and 166 students engaged (~67%). This is a very large sample, and thus the findings have corresponding significance.

Reasons for this evaluation

The programming of ChemiCAL software started in January 92. Its development since has been dynamic and continues to be so. The author of the system was awarded the Royal Society of Chemistry Higher Education Teaching Award in 1998 for the design and implementation of the software. Due to costs, the interface has changed little over the years and thus its appearance can justifiably be described as “dated” or (more politely) as “retro”. However, the underlying code has advanced considerably in terms of what it does for both staff and students. Consequently, this evaluation was not aimed to gather opinions on the “quality” of the software, but rather to ascertain student attitudes to the provision and use of the system.

This is not the first evaluation of the software. Many such evaluations were done in the late 1990s on each aspect of ChemiCAL provision (virtual lectures, pre- and post-labs and virtual assessments); always the software has evaluated well. However, these evaluations clearly indicated a small but significant number of students (20% or less) who generally despised the software. As in all walks of life, dissenters shout louder and longer than others, and thus the pressure from this group was disproportionately high, damaging the perceived integrity of the software within the department. This situation has been exacerbated by the recent shift in use of the system to the MPharm and ACAPS programs for two reasons: firstly, the much larger size of the cohort (and a correspondingly high number of dissenters) and, secondly, the nature of the cohort (largely being pharmacy based with a disinterest towards chemistry).

Summary of findings

Students lodged 249 comments in the questionnaire. Pleasingly, there were no derogatory comments, and a good balance of positive, negative and neutral comments were offered. The questionnaire comprised ten positive statements, and students were asked to strongly agree, agree, neither agree nor disagree, disagree or strongly disagree. Respectively, these were scored as +2, +1, 0, -1 and -2. Unfortunately (but expectedly), there was a high correlation between the laboratory

* The Royal Society of Chemistry National HE Teaching Award 1998 marks awarded by the software to individual students and their overall opinion of ChemiCAL post- labs; those who generally scored less well gave significantly lower ratings than those who scored better. For example, those students who felt they scored poorly in the software gave it an overall average rating of –1.2 (on a scale of -20 to +20 over the ten statements). Those who felt they scored normally gave an average rating of +0.1, whereas those who felt they scored well gave an average rating of +3.5.

In all, the student class as a whole enjoyed using the post-labs (see statement 1) compared to writing a report, but also found them more stressful (statement 2). They thought the software helped them with their data manipulative ability (statement 3) and produced a better marks breakdown (statement 4). Whilst they found it more difficult to work with friends while working with the software (statement 5), they found the post-labs a little less academically demanding than writing a lab. report (statement 6). They also thought the software was no more accurate in marking than the lecturers themselves (statement 7); this is an astonishing finding – one of the main features of the software is accuracy and fairness in marking, but this has been lost on the student cohort as a whole. However, the class thought the instant feedback of marks was very useful (statement 8), but just about disliked being marked by a computer rather than a tutor (statement 9). Pleasingly, the class thought the post-lab software and its implementation into their course was a good overall thing (statement 10).

Overall on these 10 points, the students rated the software on average at +1.70 on a scale of -20 to +20.

General conclusion

These findings show that the students appreciate the system, but work needs to be done on convincing them of the accuracy of the software when dealing with their marks. Also, it needs to be emphasised that they should take their time over completion of each post-lab to reduce the stress they feel in undertaking their work. On the staffing side, the post-lab software was estimated to have saved over 500 hours of staff marking workload, making this time available for research and other school developments. Consequently, the advantages of this software are truly double-edged.

The following pages give the information summarised above in terms of charts and student comments on the ten statements. Responses have been made to a selection of comments. The comments made by students were generated electronically, and thus they have been replicated “as is” i.e. typographical, grammatical and spelling errors have remained. A separate Excel document (postlabsurvey_level1.xls - appended) holds the numerical data as generated chronologically, and a text file (postlabsurvey.txt) holds the comments (not appended).

Dr. Barry Nicholls School of Pharmacy and Chemistry April 2007

ChemiCAL post-lab survey 2007 2 Statement 1: “Compared to written lab reporting, I enjoy ChemiCAL post-lab reporting”

Question 1 - Enjoyable - %?

45.0 40.0 35.0 30.0 25.0 20.0 15.0 10.0 5.0 0.0 S.agree Agree Neutral Disagree S.disagree

Scale +2 to –2: Strongly agree = +2, Agree = +1, Neutral = 0, Disagree = -1, Strongly disagree = -2 Count: Strongly agree = 18, Agree = 66, Neutral = 28, Disagree = 37, Strongly disagree = 17

Outcome: 166 students gave average result = +0.19 (agreement) This is a positive evaluation of the post-lab system.

50.6% of students enjoyed using the post-labs (compared to writing a report) and 32.3% did not.

Student comments:

1.1 “more challenging” 1.2 “its a good method for certain practicals that we do” 1.3 “Its quicker to do than a written write up” 1.4 “I like the fact that if you make a small mistake chemiCAL lets you know and you can correct it instead of continuing through a written report unaware.” 1.5 “ChemiCAL is very imortant way of sending, but I don't think it can be a replacement for writing report. Written reports are very good because they build us up in writing good reports” 1.6 “Can be a lot more stressful thsn writing lab reports but its less time consuming.” 1.7 “save time to write the report and also can receive the mark after finish the ChemiCAL post-lab report.” 1.8 “time saving and do not need to refer more books as information present on chemical” 1.9 “it is a stressful program.” 1.10 “Chemical is less inpersonal and the setup is slightly daunting.” 1.11 “Not as much workload” 1.12 “It feels like the lecturer/tutor is standing beside you tutting at the work you do. I feel very pressured when doing chemical” 1.13 “I feel so much pressure when attempting the questions and what makes it worse is the negative marking.” 1.14 “I feel too much pressure when doing chemiCAL due to the negative marking etc. It doesn't help!” 1.15 “The chemiCAL program can ask fairly vague questions, but overall was very slightly more enjoyable than written reports...” 1.16 “Sometimes I'd rather be able to go through the calculations with a tutor rather than alone on the computer.” 1.17 “ChemiCAL effectively tests your knowledge of the procedure you carried out and the calculations involved without wasting time writing out methods and procedures, but i feel that it is harder to do well in chemiCAL due to the negetive marking system”

ChemiCAL post-lab survey 2007 3 1.18 “it`s too difficult and sometimes the result are correct but the computer doesn`t take it in account so then we can loose mark even if our work is correct” 1.19 “i dont think that there is a point for chemical ...” 1.20 “most of the times we put the right answer but it appears as it's wrong according to the system. while its 100% right by thae calculations even the simple ones as 2*2=4 there is no other answer but the computer takes it as a wrong answer.” 1.21 “ChemCal does not give you any feedback on what you did wrong or how to correct it. If you get a good grade on your claculations, your grade then comes down because of precision and accuracy. So despite understanding the experiment, you end up with a bad grade becuase you need to work on your experimental technique, not a good way of grading.” 1.22 “the results are much more instant and shows you clearly where you gained and lost marks” 1.23 “well if you have done ur calculation properly, but when you entered ur data wrongly, still u wud lose mark. with a written report one can check its answer.” 1.24 “Its more stressfull but takes less time to complete” 1.25 “Its hard to get marks for accuracy and precision in chemical” 1.26 “i find it easier completing lab reports in which i have to go and resaerch the work i do not consider maths to be my strong point” 1.27 “I prefer to have time to go back and correct my work without negative marking.” 1.28 “I think chemical is rather stressful as it puts you under pressure and has negative marking. i prefer to have a lot of time to sit down and work things out with the written lab reports” 1.29 “its easier for me to work through the calculations on chemical than on paper, and it helps me understand where to get the numbers from” 1.30 “Written lab reports allow us to recalculate until we are happy with out answer, the pressure is less due to the fact that we can see what we have to do in front of us. ChemiCAL adds pressure, a simple typing error can cause us to lose marks, and when we know that a question is incorrect, it disheartens us and generally we work less hard for the rest of the program” 1.31 “Interactive interface is easy to use and does put pressure on students to thoroughly prepare for their labs.” 1.32 “they are both very stressful but in there own respects” 1.33 “it takes shorter time to complete chemical rather than lab report however through the computer can be intimidating” 1.34 “i think thers too much emphasis on accuracy and precision which isn't exactly fair” 1.35 “ChemiCal allows you to gain a full understanding of how much accuracy and precision is important.” 1.36 “there is obviously less written work but i probably learn more about the topic doing a written report” 1.37 “i prefer writing a lab report rather than submitting a chemiCAL report as it is much more stress to submit a chemiCAL.” 1.38 “no need to do write up and save time to revise.” 1.39 “Received feedback immediately.” 1.40 “too messy” 1.41 “chemiCAL does not help you find where you went wrong” 1.42 “you are being tested as you go along and can see your marks. One little mistake can cost you quite a few marks which may not have happened if a person was marking it and understood where you had gone wrong.” 1.43 “In one sense , I enjoy using chemiCal as it can be more time efficient and faster than writing out a full report. However , at times I find my self somewhat frustrated with the weay the program calculates Precision marks. I think a precision mark of absoute zero is somewhat unfair for a small measurement error.” 1.44 “there are more calculations and no need necessarily for words” 1.45 “It takes an equal amount of effort to doing written lab reporting and doing ChemiCAL post-lab reporting.” 1.46 “I like to receive feedback from tutors, I find it more informative than the comments on ChemiCAL.” 1.47 “It is quicker to do, but I feel that you don't learn much from it. You're basically just entering in your results from the lab and are being tested on accuracy and precision, but not the material itself. First year MPharm is about learning theory.”

ChemiCAL post-lab survey 2007 4 1.48 “Get a longer deadline and not a lot of writing, like the calculations.” 1.49 “I dislike chemical as much as I dislike writing a lab report” 1.50 “questions are stated clearly” 1.51 “It's quite intimidating” 1.52 “It is shorter and more interactive learning” 1.53 “I disagree to enjoy doing a ChemiCAL since I don't find it actually aids me in understanding anything since I am preoccupied with achieving a good mark. Also, the marking scheme does not seem fair that even if you do well or tired to do well in the lab due to few errors you might achieve a low grade.” 1.54 “ChemiCAL is directed and easy to follow.” 1.55 “Not accurate marking of one's abilities as a scientist”

BSN response to selected comments:

1.5 ChemiCAL post-labs are only designed to operate for practicals where the overall result, and the student’s ability to arrive at that result, comprise the assessment criteria. In these cases, a written report is unnecessary. Practicals where report writing is important (such as those practicals that require a good deal of interpretation) will never be supported by post-labs. 1.13 Negative marking is present to encourage the student to concentrate on the work, by discouraging multiple random entries within calculations. The records show this works very well, with relatively few instances of the award of negative marks. 1.16 Going through the calculations with a tutor is not a viable option – in general. 1.18 There have been no instances where students have lost marks for correct work. 1.20 There were two particular problems with computer code affecting a number of student calculations – these were all identified and the code corrected – no student was penalised. 1.21 Marks awarded are for experimental technique and data manipulative ability. The programs are not designed to test other “understanding”, as the experiments to which they apply are all very simple in theory. 1.23 There is no reason why students cannot check their answers before input within ChemiCAL. 1.25 Accuracy and precision marks are ChemiCAL independent. 1.30 There is always an opportunity to correct typing errors without loss of marks. 1.34 There is actually too little emphasis on accuracy and precision! 1.36 There is little theory to learn in the practicals supported. 1.41 The post-labs are laid out in such a way where the students need to spend little time in working out where an error has occurred. Supported experiments are simple ones. 1.42 Each mistake carries a maximum penalty of only 1% of the practical mark! 1.43 The precision marks are set very generously indeed, and have not changed with the introduction of post- labs. 1.47 There is little theory to learn in supported experiments. Accuracy, precision and data manipulation is all-important in these experiments. 1.55 The post-labs are ultimately accurate in assessing laboratory results.

ChemiCAL post-lab survey 2007 5 Statement 2: “Compared to written lab reporting, I find ChemiCAL post-lab reporting less stressful.” *

Que stion 2 - les s stress ful - %?

0.0 S.agree Agree Neutral Disagree S.disagree

*this statement/analysis is a mirror of the actual statement raised in the questionnaire

Outcome: 166 students gave average result = -0.39 (disagreement) This is a negative evaluation of the post-lab system.

51.8% of students disagreed, with 25.9% agreeing. This is an anticipated result – many of the reasons for this are outlined in the student comments below. Marking occurs at every stage throughout the post-labs, and those marks are displayed instantly to the student. This is bound to cause an increased stress level! However, is this really a bad thing? Graduates in this discipline will need to get used to situations were their decisions and their effects are immediately assessed.

Student comments:

2.1 “If you do something wrong then you cannot just cross it out and do it again, as it will already have saved.” 2.2 “i feel i am more relaxed when i do chemical than when i am writing a report” 2.3 “saves having to search for non useful information” 2.4 “There is no way back..” 2.5 “Must work out the answer immediatly not much time to think” 2.6 “it is quite stressful for the first time users as we don't know the clearly procedure how it works. Using ChemiCAL achieved lower marks than writing a normal report. e.g. for calculate the result which cannot round up too early.” 2.7 “because in specially lab report writing the presentation calculation using microsoftword is hard.” 2.8 “Quick and not as time consuming” 2.9 “It feels like someone is standing over you.” 2.10 “Because of the negative marking your under more stress to get the answer right the first time.” 2.11 “Because the mark for the practical was givin there and then, it was an advantage of the program. However, making mistakes in the decimal places of questions was easier to make I felt than in an exam or when handing in a paper.” 2.12 “I think its frustrating which makes it more stressful. If the system crashes, and you do not get credit for your caluations and having error carried forward.” 2.13 “you cannot go back and change anything”

ChemiCAL post-lab survey 2007 6 2.14 “once u have done a mistake, its done.” 2.15 “you feel you have to get it right there and then whereas with written reports you can think more about it and change parts” 2.16 “as above, it is hard to get marks for accuracy in chemical” 2.17 “its hard to get marks for accuracy” 2.18 “I think it is quite scary and has negative marking” 2.19 “When we know an answer is incorrect, we lose heart and dont really want to carry on with the rest of the exercise as we are afraid to lose more marks.” 2.20 “A little more stress, since incorrect answers are penalised and cannot always be changed.” 2.21 “I think its more pressure” 2.22 “genrally tends to be straightforward” 2.23 “ChemiCal is a much more simplier and less time consuming layout.” 2.24 “it can be stressful if you leave it to the last minute but you can also save and quit at any time.” 2.25 “Just feel more pressure when doing it through ChemiCAL” 2.26 “Both Ok” 2.27 “there is confusion when filling out the chemical program.” 2.28 “a bit hard to understand it” 2.29 “less time is spent doing the work” 2.30 “Even if you did the lab properly, slight variations can end up getting you a fail mark.” 2.31 “I did in the begining but not any more.” 2.32 “I find both stressful” 2.33 “You can't make mistakes and then correct them later” 2.34 “I do agree the chemiCAL can be much more stressful since there are always space for error even though one may have tired in the lab.” 2.35 “ChemiCAL is directed and easy to follow” 2.36 “I dont feel stressed, I see this all as a challenge”

2.1 There is always an opportunity to correct mistakes without loss of marks. This opportunity is obviously denied once the student has formally submitted the answer and computer has revealed the correct answer. 2.5 There are no time limits within post-labs, and they can be completed in many re-opened sessions. 2.10 Exactly! Negative marking concentrates minds! 2.12 ChemiCAL allows for crashes without loss of marks. Students need to “save” more regularly. There are very few instances of crashes. 2.13 You also cannot go back and change a report after it has been marked! 2.14 There is always an opportunity to correct mistakes without loss of marks.

ChemiCAL post-lab survey 2007 7 Statement 3: “Compared to written lab reporting, I find that ChemiCAL post-lab reporting increases my ability to do calculations successfully, and I learn more from them.”

Ques tion 3 - helps w ith calculations - %?

50.0 45.0 40.0 35.0 30.0 25.0 20.0 15.0 10.0 5.0 0.0 S.agree Agree Neutral Disagree S.disagree

54.2% agreed and 18.6% disagreed. This is an anticipated result. It is known that some students fail to engage with the post-lab programs appropriately. Such students will learn little from the process.

Student comments:

3.1 “gets the head thinking” 3.2 “With chemiCAL you know whether you've done the calculations right straight away rather than having to wait ages to get lab reports back.” 3.3 “if calculations are incorrect i have the chance to try them again or the system will give the correct answer and i can work from there. with chemical i am able to note down how to work out particular calculations for future reference” 3.4 “Alot of the time my method of doing the calculation is correct, i jst make small mistakes that produce the wrong answer. I do not get marks for using the right method you just get marks for the right answer.” 3.5 “its good practice I agree.” 3.6 “There is no feedback as to why i have got it wrong just that it is wrong” 3.7 “ChemiCAL doesn't inform you of what you have done wrong when you make a mistake. However lecturers usually write on practical reports where you have gone wrong.” 3.8 “This mightn't be appropriate to say, but the amount and ease of cheating in the ChemiCAL program is very high, and sometimes feels unfair when we realise some people attain very high marks by using calculations made by other students.” 3.9 “It does not show you where you went wrong or how to fix it, no feedback.” 3.10 “Its assessed there and then so you can identify mistakes quicker” 3.11 “it does help with calculations” 3.12 “i think i will eventually become more confident with maths” 3.13 “I dont feel it has helped my calculations at all” 3.14 “i get to see where the numbers come from and it helps me through and to understand what is going on”

ChemiCAL post-lab survey 2007 8 3.15 “seeing it written out in front of me on paper helps me to remember and learn more as i have written it out myself” 3.16 “as you can recalculate what you did wrong learning from your mistakes” 3.17 “ChemiCal increases my practice at calcultions and so improves my ability to do such calculations correctly.” 3.18 “my calculations have improved more with chemical.” 3.19 “Immediate feedback usefulin noting errors and derermining mistakes made.” 3.20 “there is no help when doing the calculations” 3.21 “The same calculations can be done whilst doing a written report. It just makes it more stressful if you can't do a calculation because you lose marks!” 3.22 “I wouldnt say its a case of learning the more about calculation.” 3.23 “I don't think it helps because it really just provides u with the answer” 3.24 “same for both written and chemiCAL" 3.25 “I do agree it does help with learning to do calculations but those can be done and shown in a written lab report. Also, because one is worried about the marks they will achieve if the calculation is wrong it doesn't really aid in learning.” 3.26 “I believe chemiCAL to be limited and too short”

3.6 Such feedback on simple calculations is not necessary; students should always be prepared to work out themselves where they have gone wrong. All calculations are broken down to the simplest levels in the post-labs. 3.7 Such feedback on simple calculations is not necessary; students should always be prepared to work out themselves where they have gone wrong. All calculations are broken down to the simplest levels in the post-labs. 3.8 This problem is ChemiCAL independent. 3.9 Such feedback on simple calculations is not necessary; students should always be prepared to work out themselves where they have gone wrong. All calculations are broken down to the simplest levels in the post-labs. 3.20 All calculations are broken down to the simplest levels in the post-labs.

ChemiCAL post-lab survey 2007 9 Statement 4: “Compared to written lab reporting, ChemiCAL post-labs are better at informing me of where I gain and lose marks.”

Ques tion 4 - bette r m arks break dow n - %?

50.6% agreed and 23.5% disagreed. Judging by the comments below, many students have unrealistic expectations of the lab marking system. In the practicals supported by post-labs, it is not possible to state why a student’s accuracy and/or precision is poor, unless they have been closely observed in the lab.

Student comments:

4.1 “i seem to do poorly in the accuracy in my lab performance and chemical doesn't tell me why” 4.2 “too many marks lost for accuracy and precision” 4.3 “if an incorrect answer is entered, the system will give the correct answer. this can be used as reference for future assignments” 4.4 “ChemiCAL makes me realise instantly where I have gone wrong and I can learn from this much more easily.” 4.5 “ChemiCAL is ok in this respect as it provides me with this information immediately, whereas getting a report back takes a while.” 4.6 “I agree that I can see where my error come from. It suggest that can know the acutal data be better.” 4.7 “Knows where the problem is at during the report” 4.8 “ChemiCAL nor written lab reports point out where you have gone wrong.” 4.9 “There is no difference really.” 4.10 “written reports do not have a clear enough mark scheme” 4.11 “it clearly states how you loose marks” 4.12 “lecturers clearly state were i went wrong in my written lab reports” 4.13 “Its hard to know what is lo0st for accuracy and precision” 4.14 “I do not think the program gives me any feedback on where i am going wrong” 4.15 “half the time when the marked write ups come back they simply have the mark on, and no breakdown of where the marks were lost” 4.16 “written reports with feedback written on them i feel are alot better” 4.17 “immediate feedback is available” 4.18 “Lab reports give greater and more in depth feedback.”

ChemiCAL post-lab survey 2007 10 4.19 “of course because it's marking you as progress.” 4.20 “it shows me where i lose the marks but it does not help me to solve the problem” 4.21 “I tend to finish working with chemical and wonder where some of my marks have been lost.In this sense i would prefer a lab report.” 4.22 “The rersults are given straight away.” 4.23 “I know where I lose marks, but tutors can inform me on how I've lost them, where I've gone wrong, this isn't done on chemiCAL.” 4.24 “I do not agree to it since it only shows a certain mark but one does not really know how and in what they have errored for the future chemiCAL labs to improve, nor is there any feedback.” 4.25 “ChemiCAL does not offer the degree of accuracy that a lecturer does”

4.1 It is not possible for anybody (let alone a computer) to identify why a student has lost accuracy/precision marks without actually observing the student in the laboratory. 4.14 The post-labs clearly show where calculations are incorrect. However, it is not possible to detail where a student has poor technique in the laboratory. 4.20 Showing a student where marks are lost for calculations should automatically help the student correct his procedure – with a little bit of thought. 4.23 It is not possible for anybody (let alone a computer) to identify why a student has lost accuracy/precision marks without actually observing the student in the laboratory. 4.25 Computer marking is ultimately accurate given correct coding!

ChemiCAL post-lab survey 2007 11 Statement 5: “Compared to written lab. reporting, it is more difficult to liaise with colleagues when reporting via ChemiCAL post-labs”

Question 5 - m ore difficult w orking w ith friends - %?

45.0 40.0

35.0 30.0 25.0

20.0 15.0 10.0

5.0 0.0 S.agree Agree Neutral Disagree S.disagree

39.7% agreed and 19.3% disagreed. An expected result. Anecdotal evidence suggests that those who have difficulty with calculations can fairly easily copy routes from the written reports of others, amending for their own data. With computer based post-labs, this is much more difficult.

Student comments:

5.1 “both of them can disscuss with colleages” 5.2 “As I've already said, it was much easier to liaise or 'cheat' during the ChemiCAL pre- and post- labs.” 5.3 “Which is also a problem. By its designed to prevent the few bad students from cheating, but it does not take into account that students benefit when they interact with each other, and might learn things they might have missed, which in the end enhances everyones learning experience.” 5.4 “I always complete chemical on my own as its harder to consult other people when you have different titre values” 5.5 “lab reports can be checked by a colleage to detect errors. Chemical cant be checked” 5.6 “ChemiCAL does a better job at ensuring that students are not sharing work and liasing results.” 5.7 “theres not much opporortunity to task for help” 5.8 “I think there is always an element of colaboration between colleages when using chemiCal. In some sense it is easier to collaborate in chemicak than in written work as chemical only requires the imput 9of numerical data. This data can be easily copied.” 5.9 “I don't get help from friends anyway.” 5.10 “because u can exit chemical and re-enter it again” 5.11 “when you dont understand something using ChemiCAL and you ask a friend they usually tell you the answer when trying to explain to you.”

5.8 This problem can be identified by data-sorting – this is not possible with written reports.

ChemiCAL post-lab survey 2007 12 Statement 6: “Compared to written lab. reporting, ChemiCAL post-labs are more intellectually demanding”

Question 6 - m ore dem anding - %?

45.0 40.0

35.0 30.0

25.0 20.0

15.0 10.0

Outcome: 166 students gave average result = -0.09 (disagreement) This is a negative evaluation of the post-lab system.

25.9% agreed and 34.3% disagreed. This is an unexpected result, as it was anticipated that students would find the post-labs a little bit more difficult. However, the overall response is close to neutral – no action required.

Student comments:

6.1 “No they are not as with lab reports you often have to do a full write up, whereas chemical is just compromised of calculations and questions, thus being less time consuming.” 6.2 “i find them enjoyable” 6.3 “chemICAL are more calculation based questions, lab report ask much more and require more detail.” 6.4 “Written lab reports involve calculations like ChemiCAL, whereas written lab reportingalso involves research into the scientific area that the practical covered.” 6.5 “good for students to learn.” 6.6 “No as it is mainly calcualation based” 6.7 “Not as much work” 6.8 “ChemiCAL reports are more concerned with the calculations and not really about the reasoning behind the experiment” 6.9 “You are able to research lab reports to help you along but with ChemiCAL you are expected to know the answrers.” 6.10 “Lab reports usually ask for more than just calculations as in chemiCAL. I learn more from lab reporting” 6.11 “Written lab reporting to me is more academically demanding, as reports should be tidy and easy to read, aswell as correct.” 6.12 “Written lab make you go out and research the subject and actually learn!! While ChemCal concentrates on whether or not you can do calculations only.” 6.13 “it depends on the content” 6.14 “its more about precision and accuracy” 6.15 “They take less time than other written reports”

ChemiCAL post-lab survey 2007 13 6.16 “the program talks you through the calculations and so on, so you have to do less work and research yourself” 6.17 “wirrten labs are more demanding however i feel i learn more from them than chemical. Chemical just adds unnessesary stress” 6.18 “I find Written lab reporting more demanding, as thorough explanations are discussions are required.” 6.19 “Lab reporting requires more thought into structure, layout and content.” 6.20 “there is much more work that goes into a written report.” 6.21 “They involve more calculating.” 6.22 “they are just as demanding just in different ways” 6.23 “Lab reports require more background than the basic”

6.12 The practicals supported by post-labs are simple and do not require any research.

ChemiCAL post-lab survey 2007 14 Statement 7: “Compared to marks in written lab. reports, ChemiCAL post-lab marks reflect attainment more accurately”

Question 7 - accurate m arking - %?

45.0 40.0

35.0 30.0

25.0 20.0

15.0 10.0

Outcome: 166 students gave average result = -0.02 (disagreement/neutral) This is a statistically neutral evaluation of the post-lab system.

30.7% agreed and 26.5% disagreed. This is an entirely unexpected result as we know that marking by computer code is ultimately precise and accurate (given correct coding). It was anticipated that the response to this statement would be highly positive!

Student comments:

7.1 “you know what you've got right there and then, sometimes we don't get lab reports back” 7.2 “some people can change the values therefore cannot reflect the actually they achieved.” 7.3 “It is calculation based rather than understanding what might be happening” 7.4 “I feel that the accuracy of my results can not be assessed by a computer” 7.5 “in chemical the results that are given take no account of error, the results are compared to the stock answers, and i feel that the margins for error are too strict” 7.6 “chemical is PICKY! a simple mistype or clicking the wrong thing or mis spelling can affect marks. Chemical is not always clear of the answer it requires” 7.7 “it could be luck from the practical that you attained accurate results.” 7.8 “writen lab reports have inconsistent marking procedures.” 7.9 “no work is shown and our marks are awarded by only a number instead of our actual procedure.” 7.10 “I think chemical would be a better reflection of attainment , if and when the system could show where marks are lost. There seems to be element of " hmmm chemical just takes your scores and adds an element of randomness to them."” 7.11 “[disagree] because sometimes the practicals do not work right and chemical does not take this into consideration because it is a computer!” 7.12 “no room to explain subjective errors made during lab. no discussion of sources of error.” 7.13 “Computer generated so must be more accurate”

7.2 The changing of values is to be addressed within the lab via “receipts” from next year 7.4 Accuracy is easily assessed by a computer

ChemiCAL post-lab survey 2007 15 7.5 The margins to allow for error are too lenient 7.6 ChemiCAL allows the correction of mistakes without loss of marks. 7.10 Computer code does the opposite of adding “randomness”!! 7.11 ChemiCAL marks are always subject to ratification by a tutor. 7.13 Absolutely!

ChemiCAL post-lab survey 2007 16 Statement 8: “I prefer instant feedback of marks rather than waiting weeks, because it helps me to timely assess where I need to improve my work.”

Question 8 - instant feedback - %?

45.0 40.0

35.0 30.0

25.0 20.0

15.0 10.0

Outcome: 166 students gave average result = +0.81 (agreement) This is a positive evaluation for the post-lab system.

68.1% agreed and 12.0% disagreed. Expected result. Instant feedback of marks is one of the main anchors of ChemiCAL.

Student comments:

8.1 “proper feedhead whilst its still fresh in the memory and easier to remember your mistakes” 8.2 “chemical gives an indication of standard of lab work” 8.3 “its good reason why ChemiCAL can save time.” 8.4 “The feedback from chemical is not as useful as the feedback from a written report” 8.5 “Having the mark straight away does help me asses where I need to improve my work there and then.” 8.6 “Yes this is useful as I have not had some reports back from sep/ oct.” 8.7 “i can also remember what work i got the marks for wheras for written work you've forgotten what you did when the results come back” 8.8 “i dont mind how long i wait” 8.9 “Its good and helpful to get results immediately” 8.10 “Waiting longer for reports i feel is better because you do not feel that you have let yourself down. Getting a bad chemical score does not push you to improve those areas, it just devolps a sever disliking for chemical” 8.11 “end the end of the day both will give me a feedback.” 8.12 “I think that instant feedback is definitly very useful. However just because feed back is instant does not nescessarily mean it is of high quality, or that useful.” 8.13 “That is the best part of ChemicCal, you know your results straight away.” 8.14 “the feedback given from the lab reports is more worthwhile because it doesn't say "you failed do better". The lab reports suggest how improvements should be made” 8.15 “the feedback is not very informative; but it is much faster” 8.16 “I haven't seen much improvement in my chemiCAL labs since I haven't recieved feedback that really helps”

ChemiCAL post-lab survey 2007 17 BSN response to selected comments:

8.10 It is known that poor marks issued by ChemiCAL can lead to a severe resentment of the system. This is possibly due to the instant nature of the feedback – the heat of the moment, so to speak! Also, the fact that a computer rather than a tutor is informing on poor marks makes some people very resentful indeed.

ChemiCAL post-lab survey 2007 18 Statement 9: “I like a computer being involved in marking my laboratory data and calculations; I would not prefer a lecturer to do this alone.”*

Question 9 - like com puter m arking - %?

*this statement/analysis is a mirror of the actual statement raised in the questionnaire

Outcome: 166 students gave average result = -0.13 (disagreement) This is a negative evaluation for the post-lab system.

27.7% agreed and 36.1% disagreed. Expected result.

Student comments:

9.1 “both computer and lecturer to mark could be better” 9.2 “The marks i lost because of the difference in number rounding is annoying” 9.3 “Computers don't have minds of their own HOWEVER sometimes it can be fairer. It is either right or wrong. Whereas lecturers can have differing opinions and may mark work in a way that they think the answer is right.” 9.4 “A lecture can appreciate the students work and can give partiall credit, while a computer can not do that. Plus a lecturer is able to point out your mistakes better and what to do about them.” 9.5 “i like both ways” 9.6 “i dont mind” 9.7 “i feel like a computer can make tonnes of errors” 9.8 “a lecturer produces constructive feedback. A computer, not so much.” 9.9 “Depending on the lab that is being marked. If it is simple calculations, then I dont mind the computer marking the work However, sometime there are technical errors that cannot be reversed when using ChemiCAL and that gets a little frustrating sometimes.” 9.10 “Computer is more time effiicient and accurate” 9.11 “if the answers are right, i dont mind.” 9.12 “i learn whiles i using the program therefore it does make a difference to me.” 9.13 “The lecture can deem that there may have been a problem with the solutions involved if a number of people find the result to be in a different range than that entered into chemical” 9.14 “they can reflect more on human error than a computer!” 9.15 “this would depend on whether the answers supplied do not need to be explained in words as to how they were achieved then a lecturer is preferred otherwise, the computer will do.”

ChemiCAL post-lab survey 2007 19 9.16 “I like doing the calculations, and having those maked by a computer, but I'd prfer a lecturer to leave a comment reflecting on how I have done.” 9.17 “it doesn't bother me either way” 9.18 “Since I do my chemiCAL labs from my home laptop if for some reason it does turn off or some technical error occurs then my grades reflect it which isn't fair.” 9.19 “Computers are prone to error, but so are lecturers, but I can ask questions of my feedback to a lecturer”

9.3 This is precisely one problem the post-labs seek to eliminate! 9.7 This statement is completely the opposite of the actual situation. 9.9 There are no technical errors that have resulted in loss of marks. All errors have been addressed. 9.13 This problem is always attended to by a tutor. 9.18 Technical problems do not lead to a loss in marks if reported.

ChemiCAL post-lab survey 2007 20 Statement 10: “All things considered, it is a good thing that ChemiCAL post-labs form part of my overall laboratory studies.”

Question 10 - overall good thing - %?

50.0 45.0 40.0 35.0 30.0 25.0 20.0 15.0 10.0 5.0 0.0 S.agree Agree Neutral Disagree S.disagree

Outcome: 166 students gave average result = +0.39 (agreement) This is a positive evaluation for the post-lab system.

54.8% agreed and 18.0% disagreed. This is an unexpectedly high result for those in agreement.

Student comments:

10.1 “yes, its good to use ChemiCAL.” 10.2 “Easier to hand in and complete the assignments” 10.3 “I once got 100% in the questions part of a chemiCAL report and ending up doing badly due to the way the final mark is found.” 10.4 “If chemiCAL was not a part of the course, i would not feel i was missing out. I woudn't say that chemiCAL has improved my working and I prefer doing written reports (find these more useful).” 10.5 “Its fun hearing how bad some students perform in the exercises...” 10.6 “it helps me improve calculations and shows a different style of questions which will benefit me in different ways” 10.7 “i find them my weak point” 10.8 “I do not like chemical as it is stressful and breaks down frequently” 10.9 “they add unessasary stress to the constant load” 10.10 “It gives a broader range of skills when reporting” 10.11 “it is not really clear how much the ChemiCAL post-lab grades contribute to the overall grade for each module.” 10.12 “i prefer feedback from my lecturers” 10.13 “It is a good instrument used once in awhile but I wouldn't want to use it all the time. A different format but I don't really think it helps learning or understanding the information.” 10.14 “Too basic and not testing enough”

10.8 There is no evidence of any breakdowns due to ChemiCAL

ChemiCAL post-lab survey 2007 21 11. Overall results

ChemiCAL survey - level 1 March 2007 - entire cohort result 166 respondents

0.40 l e

v 0.20 e l

e 0.00 m

e 1 2 3 4 5 6 7 8 9 10 e

r -0.20 g A -0.40

-1.00 Question number

Further general student comments:

11.1 “at the start of the year i diliked ChemiCAL but as time went on i found it better and learned from it.” 11.2 “I do like to use ChemiCAL, e.g. virtual learning and pre-lab. give me chance to pre-study. therefore I agree to use ChemiCAL can improve my study.” 11.3 “ChemiCAL is completely new teaching method by introduction of new technology and its very informative and time saving method.” 11.4 “I think in theory the ChemiCAL post-labs are a good idea I personally prefer the feedback that i get from a written report” 11.5 “only smarties have the answers.” 11.6 “I do not like chemiCAL as it is very easy to make accidently mistakes by pressing the wrong buttons and lose unnecessary marks which would be easily noticed and rectified if it were done by hand but it cannot be rectified on chemiCAL as you only get 1 attempt” 11.7 “I don't like entering data and calculations into chemical. would much rather write a lab report with introduction, method, conclusion etc. This helps my understanding in the subject and critical feedback from a lecturer is appreciated. Feedback and being told you are wrong by a computer is not an appropriate way to be told of your current level.” 11.8 “I think that the chemical environment creates more stress on the student and causes them to make silly errors unlike how they would if they were just given the chance to write a full report.” 11.9 “i dont like chemical, personally i find it to be in inaccurate way of marking because there is more to a awarded grade than a specific number. we should be given marks based on our work process not just the single answer. also, chemical is not a very good program in general. i dont like using it.”

ChemiCAL post-lab survey 2007 22 11.10 “sometimes, the computer doesn't give good accuracu and precison marks” 11.11 “chemiCal no doubt saves lecturers time and saves me time writing up practical reports.”

11.6 There is always an opportunity to correct mistakes without loss of marks.

ChemiCAL post-lab survey 2007 23